This may have some huge impact on how we will work with e. g. Android in the future. Xen may run on Android devices (especially smartphones and tablets), and it might even be the case that you won’t boot into your “normal” Android OS when switching on your device. Instead you will boot into a virtual machine and may switch between various instances of Android. This makes multiple user profiles and even isolation of apps possible.

Furthermore this might lead to forging a solution which creates backups from your Android VMs. Later, you might be able to restore your Android within a few seconds in case of emergency.

This is definitely some great news for the community, and in the future every ARM device will profit from it. Thanks to the individuals which made this possible!

It is a great honor to write for such a well-known magazine, and I hope that there might be other occasions for me to contribute an article (or even a part of it) in the future.

In addition, the German Android User, a new magazine about Android smartphones and tablets, published a short Asus Transformer review from me. You can read the full article here: “Im Test: Das Asus Eee Pad Transformer TF101” The German Android User version is also available since today.

I really enjoyed working together with those very talented and skilled people.

Support those magazines and buy them!

One of the probably most anticipated enhancements is the guest support for Red Hat Enterprise Linux 6. Since XCP is “the Xen version for the cloud”, better RHEL support might be very interesting for companies and hosters.

Click here to view the full list of new features.

You can download XCP 1.1 here.

If you want to try out XCP, you can either read my tutorial or visit the recently updated Xen wiki.

Being written in 2004 and heaving heavily evolved ever since, dstat has become a replacement for most of the tools being mentioned above (at least for me). dstat is very flexible and shows you the information you want in real time, e.g. RAM and CPU usage, bandwith, I/O in total, MySQL-stats, interrupts and performance bottlenecks.

Introduction to dstat
In this blog post, the latest version of dstat (0.7.2, 03/09/2010) on Debian 6 will be covered. Please notice that I will only cover the real-time monitoring features of this tool, but not the “export data to a csv-file and use it for my own charts”-stuff.

For the start let’s simply install and call dstat:
apt-get install dstat
dstat

As you can see, dstat starts with some default parameters and shows the CPU, net, disk and paging usage. Furthermore it indicates the current amount of hardware interrupts and context switches. If you already looked at man dstat, you might guess that calling “dstat” equals to “dstat -c -d -n -g -y”:

• -c = total CPU usage
• -d = disk total
• -n = net total
• -g = paging system
• -y = system stats, such as the hw interrupts and the context switches

For most ambitious sys admins, this is not enough. So let’s take a look at some of the most useful parameters:

• -l = shows load statistics
• -m = shows the memory usage (used, buffer, cache, free)
• -r = displays I/O statistics,
• -s = shows the swap usage
• -t = puts the current time in the first column
• --fs = displays file system stats (includes amount of files and used inodes)
• --nocolor = sometimes very useful…
• --socket = shows interesting network statistics
• --tcp = displays common TCP stats
• --udp = shows you the listen and active figures for the UDP usage

The parameters mentioned above already make dstat a very powerful monitoring tool, and we haven’t covered the external plugins yet Have a look at /usr/share/dstat in order to view them. Some of them are:

• --disk-util = shows how much the disks are busy at the moment
• --freespace = shows the current disk usage
• --proc-count = displays the number of running processes
• --top-bio = points to the most expensive block I/O process
• --top-cpu = draws the attention on the most expensive CPU process
• --top-io = shows the most expensive “normal” I/O process
• --top-mem = displays the process using the most memory

Simply call “dstat –plugin name” in order to use them.
Example:
dstat --proc-count

There are also many MySQL and NFS plugins, simply have a look at them and try them out (do “dstat –list” to view the list of available plugins) In addition there is an wifi extension which displays the signal power and noise.

Some useful examples
After having looked at dstat’s potential, we are now ready to use it.

In the first example we want to cover everything which is related to the disk:
dstat -d --disk-util --freespace

In the second example we simply want to see who is eating all the memory:
dstat -g -l -m -s --top-mem

The third example shows some stats to the CPU resource consumption:
dstat -c -y -l --proc-count --top-cpu

Example number four will show some beautiful network statistics:
dstat -n --socket --tcp --udp

As you can see, there are many things you can observe with dstat. I highly recommend to have a look at dstat’s manpage and get in touch with all the other parameters.

Writing your own dstat plugin
Writing a new dstat plugin should be easy. I have not tried it yet, but /usr/share/dstat/dstat_helloworld.py shows how simple this should be:

### Author: Dag Wieers

class dstat_plugin(dstat):
"""
Example "Hello world!" output plugin for aspiring Dstat developers.
"""

def __init__(self):
self.name = 'plugin title'
self.nick = ('counter',)
self.vars = ('text',)
self.type = 's'
self.width = 12
self.scale = 0

def extract(self):
self.val['text'] = 'Hello world!'


Final words
I think Dag did great work by writing this tool really saving time by summarizing many tools into one.
You can visit his website here and view the latest changelog for dstat here.

For me, my Asus Transformer has become a Netbook replacement. For those who have not decided yet if to buy a tablet or not, the following list should help to determine if you really could have use for one or not (in no particular order):

Reason #1: Because you want to have it
Reason #2: Because you are a geek/nerd
Reason #3: Awesome mobile Internet access
Reason #4: Reading books, no matter where you are
Reason #5: Mobile gaming
Reason #6: Working/learning from everywhere
Reason #7: SSHing to your server from everywhere
Reason #8: Comfortable Netbook replacement for your couch sessions
Reason #9: Something to give your little sister/brother for introducing them to computers
Reason #10: Watch movies comfortably, no matter where you are
Reason #11: Mobile file sharing with your friends
Reason #12: Paper replacement (e.g. writing down notes everywhere you are)
Reason #13: Great meeting/presentation device
Reason #14: Stable operating system (iOS, Android), so (almost) no crashes
Reason #15: Faster working, more productivity (if you know how to use your tablet)
Reason #16: Better than most Netbooks or cheap Notebooks
Reason #17: Long Battery life in comparison to Net- and Notebooks
Reason #18: More Fun!
Reason #19: Better photo sharing
Reason #20: Camera, large screen and touch screen input interface on the go

Jesus, what mess! But there is a solution. In this tutorial I try to show you how to plan everything, make the right decisions and finally come to the conclusion that a VPS with a high-performance setup is more than enough for your purpose.

Choosing the right hosting plan
Running a forum is a real challenge and can drain heavily from your financial reserves. Many people use a small shared hosting offer because of the costs, others want to rent a dedicated machine and think this is necessary because of the thousands of people who could come and visit every month.

But in fact running a good forum only takes a small, but powerful virtual server. A shared hosting plan often is not enough since the provider sets strict limits on performance and traffic. As soon as you come close to those limits, you either pay or get kicked out.

On the other side, there are the people who have some money and decided themselves for a dedicated server. This is not a bad choice, of course. But is this really necessary? Unless you expect something like 10 000 members for your forum or less, a VPS may still serve the purpose.

So, I highly recommend to grab a small VPS server and upgrade the server/transfer your community later (if needed). If you have enough money you might even consider renting a cloud instance which can be scaled without blowing RAM or CPU limits.

I personally would stick with a 13 Euro VPS (approx. 18 Dollar), which leads you to 150 Euros or less server costs per year. And there is still enough room for a cheap domain

Which operating system?
You have the server? You already purchased a cool domain? Now you are ready to get started? Great! Now we should make our minds up and select a decent OS for you.

Of course there is Windows and Win2008 Web Edition. This operating system is expensive, but works and is supported by a large company. But on the other hand, we don’t want to spend a few hundred Euro just for licensing and stuff like this. So clearly we decide ourselves for a Unix or Linux.

Unix is great, especially the *BSD variants. But if you were able to do your daily work on a BSD machine, you wouldn’t read this article and have already setup your high availability cluster with a self-compiled and optimized kernel

So, there is only one type of operating systems left: Linux. Linux comes along with a huge variety of distributions, and obviously we need something which is stable, fast, easy to handle, supported by many software vendors and has a large community behind it. This leads us to either CentOS as a RedHat clone, Ubuntu Server or Debian (this list is based on my own experience, maybe you are missing OpenSuse or something else here..). Since we don’t want to pay for our OS, this list is very obvious for me.

Let’s look at the operating systems: CentOS sometimes lacks of easy handling (have you upgraded a machine from CentOS X to CentOS Y yet? Jesus, what a *****!), but profits from its big brother RHEL. Ubuntu Server depends on Canonical, but is very easy to be configured etc. Debian sometimes is too conservative, but always stable and runs on almost every hardware on the planet. Well, the differences are not always clear to see and you should select the distribution you are more familiar with. For me, this is either Ubuntu or Debian, and chose Debian. Better wait longer for updates than have new, but rarely bugged software on my system.

Which software packages?
When thinking of web server software, there are many applications jumping to my mind. There are Apache, Cherokee, Lighthttpd, nginx, Tornado…. All of them are great, but Apache is not that good scalable for huge loads. Cherokee and Tornado are both very innovative, but are the wrong choice for a normal forum software (at least that is my opinion – feel free to share your own by dropping me a mail!). This leads us to decide between either Lighthttpd or nginx. Both of them do great in large setups, both of them have their flaws and both of them are already used by large companies. When it comes down to documentation, the Lighthttpd server clearly is the leader here. On the other hand, there are statistics showing that nginx is more stable (Lighthttpd had so many memory leaks issues during the last 3 years), more popular (this changed just a few months ago, I think) and consumes less CPU resources than the other favorite.

So, let’s go with nginx! Furthermore I used php-fpm, which is far better than the mod_php of Apache you might be familiar with and also slightly better than spawn-fcgi, according to this blog entry. What impressed me the most was the fact that php-fpm has the decent “process management ability  to “graceful” stop and start php workers without losing any queries.” In addition it comes with the “possibility of gradually update the configuration and binary without losing any queries.” (Source: http://adityo.blog.binusian.org/?p=428) Furthermore we will use php-apc for caching the content. This helps us to keep the memory consumption of our small VPS very low.

Last but not least is the database software. Which one to choose? Oracle, Postgres, MySQL .. Oracle is too expensive and clearly not necessary here, which leaves Postgres and MySQL on the list. As far as I know, both database systems do great in heave-load situations. However, Postgres beats MySQL by a better handling for sub queries and joins (at least they are faster). This is why we will choose Postgres for our small VPS setup.

Forum software – cheap, but many features plz!
When thinking of forums, WBB, PHPBB, vBulletin, SMF and Invision Power Board come to my mind. vBulletin and WBB are both commercial forum packages, so that’s a “No way!” from me (although both of them are great, though).  Invision Power Board and SMF are not widely spread, but have their advantages. But for me this is not enough to stand up against PHPBB3, probably one of the most successful PHP applications of all times. There are so many free mods and styles for this internet forum software and it is so simple to install and use – so, why not?

Let’s go! Preparing the VPS
I assume that you installed Debian 6 64 Bit (I tested the following steps with also with a 32 Bit Debian, btw), setup your network interface, have a SSH server and DNS + the other basic stuff works. You are now logged in, located at /root/ and ready to go.

At first we install some useful packages (since I wrote this tutorial for my ego I only install software which I find useful ):

apt-get update && apt-get upgrade
apt-get install htop cbm atop dstat dvtm ssh vim bzip2 unzip mc vim lsof

Now let’s extend your sources.list in order to gain access to more packages:

echo deb http://packages.dotdeb.org stable all >> /etc/apt/sources.list
wget http://www.dotdeb.org/dotdeb.gpg
cat dotdeb.gpg | apt-key add -
rm dotdeb.gpg
apt-get update

Installing nginx and co
Now let’s get started with the fancy stuff:

apt-get install nginx php5-fpm php5-pgsql php5-gd php5-curl php-apc postgresql imagemagick

Configuring nginx and our vhost
Now open the configuration file of nginx:

vim /etc/nginx/sites-available/www.domain.tld

Edit this file until it looks like the one below. Please remember to replace www.domain.tld with your own domain name:

server {
listen 80;
server_name domain.tld www.domain.tld;

access_log /var/log/nginx/domain.access_log;
error_log /var/log/nginx/domain.error_log;

root /var/www/www.domain.tld;
index index.php index.htm index.html;

location ~ .php$ {
fastcgi_pass   127.0.0.1:9000;
fastcgi_index  index.php;
fastcgi_param  SCRIPT_FILENAME /var/www/www.domain.tld$fastcgi_script_name;
include fastcgi_params;
try_files $uri =404;
}
}

Attention: Don’t forget to add “try_files $uri =404;”! This one closes a security vulnerability which lets attackers parse normal image and text files as PHP scripts.

Ok, let’s continue by doing the necessary stuff on the filesystem layer:

ln -s /etc/nginx/sites-available/www.domain.tld /etc/nginx/sites-enabled/www.domain.tld
mkdir -p /var/www/www.domain.tld

Configure php-fpm to work with Postgres
We need to edit the right php.ini in order to let PHP talk with Postgres and vice versa:

vim /etc/php5/fpm/php.ini

Search the file for [Postgres] and add a new line directly below:

extension=pgsql.so

Now restart php-fpm and start nginx itself:

/etc/init.d/php5-fpm restart
/etc/init.d/nginx start

Downloading PHPBB3
Now let us take care of the forum itself:

cd /var/www/www.domain.tld
wget http://www.phpbb.com/files/release/phpBB-3.0.9.tar.bz2
tar jxf phpBB-3.0.9.tar.bz2
rm phpBB-3.0.9.tar.bz2
mv phpBB3/* . && mv phpBB3/.htaccess .
rmdir phpBB3/

Prepare Postgres for PHPBB3
Before installing PHPBB3 we should prepare Postgres:

su postgres -c psql
\connect template1
alter user postgres with password 'yourSecretPasswordHere';
CREATE USER "www-data" WITH PASSWORD 'yourOtherSecretPasswordHere';
CREATE DATABASE phpbb3;
GRANT ALL PRIVILEGES ON DATABASE phpbb3 TO "www-data";
ALTER DATABASE phpbb3 OWNER TO "www-data";
\q
/etc/init.d/postgresql restart

Optional: Installing phppgadmin
If you have used MySQL before, there might be a high chance that you already know PHPMyAdmin, the great web-based tool for managing MySQL databases. There is a direct equivalent for Postgres, although it lacks some features of its bigger brother:

mkdir /var/www/www.domain.tld/db_admin
cd db_admin
wget http://downloads.sourceforge.net/phppgadmin/phpPgAdmin-5.0.2.tar.bz2?download
tar jxf *
rm phpPgAdmin-5.0.2.tar.bz2\?download
mv phpPgAdmin-5.0.2/* .
rmdir phpPgAdmin-5.0.2/

If you want to you can now open up your browser, visit www.domain.tld/db_admin, login with the www-data user and view your two databases. From now on, you can manage all the database stuff with this web application. Secure the db_admin directory with chmod 000 or use a htpasswd file in order to protect it from unauthorized access.

Installing PHPBB3 – finaly!

cd /var/www/www.domain.tld
chmod 777 config.php

Now open your browser, visit www.domain.tld and follow the installation instructions. Make sure to perform the following actions when you completed all the steps from the PHPBB3 wizard:

rm /var/www/www.domain.tld/install -r
chmod 640 config.php

Now go and browse your awesome new forums on your high-performance monster!

Bonus: View apc statistics
If you want to check if apc is really working you could..

gunzip /usr/share/doc/php-apc/apc.php.gz
cp /usr/share/doc/php-apc/apc.php /var/www/www.domain.tld/db_admin/

Now call www.domain.tld/db_admin/apc.php with your browser and view the statistics

That’s it!
That’s it, we made it! We just installed a high-performing monster in a matter of minutes by using free open source software. And it was so easy!
Of course we missed a view things, such as setting up the rewriting stuff for PHPBB3 and do some performance tuning. However, you are now ready to go and have fun with your low-budget setup.

PS: The missing stuff will be added in future tutorials. So be sure to visit this blog from time to time

Since more than a year, one of the most anticipated games in the industry is Star Wars: The old Republic. “ToR is KoToR 3, 4 and 5″, Bioware claims. This sentence and other, surely provocative statements, rise the hype around the game.

Until today nobody knows the exact release date. Beta testers “have a feeling” that it will be on October the 4th 2011, others say that it will be christmas, others say it will be during spring next year. For me, this is pure speculation, but the truth seems to be that the game is in a really good state now. Today there are several game testing groups, playing different builds of the game. We haven’t seen a decent Fan Friday update for weeks now and no new information were released to the public for a while now.

My opinion? I think that the game will be going live soon. According to Bioware, there won’t be an open beta, like RIFT had just 4 months ago. Beta testers are saying that the game is in a really good state, but still needs adjustments and some love. The spell animations are not the best (like in WoW, e.g., unspectacular, but full of love) and some planets are not playable fully – but that’s about it. No other issues were brought up during discussions in various boards or in the comment sections of blogs publishing leaked SW:ToR material.

As a true Star Wars fan, I could not stop myself watching most of the beta videos which were released by the testers. It is remarkable that the graphics are quite good when you consider that the textures and models are all built in in a low resolution. You can see that when zooming into the pictures: It seems to be the case that the amount of polygons is very limited currently. This is properly the case because the beta wouldn’t be playable with “ultra” graphics – I guess.

However, a few months ago I feared that SW:ToR will become a flop and won’t draw enough attention in order to be the next WoW. But the fact that this new MMO will run on every machine which is capable of running WoW makes me hope for a hit and a for it to become a very time-consuming activity during my spare time (while I am not doing research for vulnerabilities or Linux stuff).

I think there is no better time to release such a game. As a matter of fact, WoW achieved something spectacular: World of Warcraft was able to “collect” millions of players worldwide from many online games and brought them all together. People with different backgrounds played together, raided, PvEed and organized themselves. Later on, many players left for masochistic reasons like Aion or Lord of the Rings online and therefore were spread all over the gaming world. True, WoW still has constantly more than 10 million active players, but most of them are playing other games, too.

When RIFT was released this spring, there was hope for a new hit. Unfortunately this game lacked love in animations, character design and the raid content. PvP is not that bad, but still boring since world PvP doesn’t seem to happen at all. In addition, the game had very strange graphics demands.. I failed to understand why I was able to play Star Craft 2 in a full HD resolution with high-ultra settings while RIFT only ran on low details. One month after release, the hype around RIFT was blown away, leaving only a dying community and a disappointed community of around 1-2 million players (well, potential players).

Now, with RIFT and WoW failing, with Guild Wars 2 not being released yet or too late, with Star Wars Galaxy being shut down soon, Bioware and Lucas Arts both got the chance to achieve something which was missing since 2004: Creating a game which attracts the masses for a longer period. Star Wars: The old Republic – we are ready for you. Now it’s your turn!

Today I read an interesting blog post about how RAM is used in Linux, furthermore the comments on that article really gave me some additional information about that topic.

For those still learning, just like me, my notes might be somehow interesting. Please feel free to mail me if you find any wrong information.

Understanding Linux RAM Usage

cat /proc/<pid>/smaps or calling pmaps
cat /proc/<pid>/pagemap (binary)
cat /proc/<pid>/statm: memory status information
cat /proc/<pid>/status: same as above, but human readable
cat /proc/<pid>stack
cat /proc/slabinfo
cat /proc/
$ free | awk '/^\-\/\+/ {print $3 }'

Linux-Tool ps and shared libraries
==========================================================
- Two possible outputs regarding RAM:
  * VSZ = Virtual Set Size
  * RSS = Resident Set Size
- Does not always show real RAM usage
- Instead, ps shows how much real memory each process would take up
  if it were the only process running
- The reason for this: shared libraries and how Linux uses them
- Example: KDE text editing software will use several shared libraries,
  e.g. X libraries and general generic system libraries, such as libc
  due to this sharing, Linux is using a trick: it loads a single copy
  of the shared libraries into memory and use that one copy for every
  program that references it
  so ps shows shared memory usage + process usage
  this means, that the one time RAM consumption of a single shared lib
  will be displayed in many processes as "consumpted" ram, but in fact
  the value gets only consumed one time
  so the real one time consumption of a shared lib is reported several times,
  which is of course wrong
- ps -lyu <user> shows in the SZ column the writable/private data, like pmap

Understanding the output of pmap
==========================================================
- Each shared library is listed twice
  * a) one time for its code segment, modus: r-x--
  * b) one time for its data segment, modus: rw---
- When a shared lib is listed three times, e.g. with r----,
   is it protected memory space?
- segments with large memory consumptions are usually
  the code segments of the included shared libs
- those code segments can be shared between processes
- if you factor out all the parts that are shared between
  processes, you end up with the "writeable/private" total,
  which is showed at the bottom of the output
- so like 2 MB is more realisting than like 24
  24 = with shared libs
  2  = without, so the "real" consumption
- so 2 mb are more realistic than what ps shows
- rwxps = read write execute private shared
- column major_minor (major device number, minor number, has to do
  with drivers)
- colum afterwards: inode number of the file from which the
  memory is being mapped
- anon = anonymous memory = private memory mapped to
  the process = data segment??

misc
==========================================================
- each of those shared libraries take up a number of virtual to
  physical page mappings, these, unlike memory, are a very
  precious resource
- consequence: try running something like KDE on non x86 hardware that
  doesn't have huge translation tables; on mibs hardware, he has seen
  apps spend over 30 percent of their time updating tlb entries
- same principal of shared lib memory consumption applies to executables
  as well; e.g. if you have several xterms running, you will only have
  one copy of the xterm executable program text in memory, which will be
  shared between all instances, but each instance ofc has its own stack and
  heap
- shared libs share backing store as long as the code is not modified,
  but if the code segment needs to be modified as it's loaded, it will
  requre its own (probably anonymous memory) swap pages

Fragmentation
==========================================================
- an app can still end up eating memory that it isn't actually using
- e.g. the app needs to make a number of large allocations for some
  temporary working space for some operation; it also needs to allocate some
  book keeping information (undo history, e.g.) and other bits of
  info to store/utilize the computed data
  when ram is allocated, the heap grows
  heap can only shrink by truncation
  this only happens when the program can  only release memory back to the
  OS by shrinking the heap, not by cutting it into pieces
  so if the app makes many allocations, and frees only some of them,
  the heap may end up like UUUXXXXXXXXXXXXXXXXXXXXXUU
  where X is free and U is used memory
  so fragmentation occurs
  that's four pages of used and 12 pages of unused memory that can't be freed
  when new allocations are made, it can use all of that unused heap, but if
  it isn't making any more allocations anytime soon, it'll appear to be
  gobbling up memory, which it is.

  with linux -> no problem, unused pages will be swapped out to disk so you
  won't run out of memory
  -> bad performance
  and when the app needs to reallocate memory it'll swap back in a bunch of
  gargabe/unused data

  this phenomen is one of the many reasons why good modern garbage collectors
  are far more efficient than manual memory management with
  malloc/free (c functions)
  a compacting collector will completely negate the above problem
- the linux allocator manages large objects with mmap, which can free
  memory chunks no matter where they are in the heap
- garbage collection requires more space and always swaps far more than
  malloc; why? because it has to touch every page periodically, including
  those swapped to disk, while it looks for garbage
- lsof shows memory/VFS ?

sources:

http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html

The goal of the discussion is to improve the protocol and to involve all people who have an interest in the WHOIS service.
Well, I also do have an opinion here.

But let’s first call the WHOIS database entry for my domain, so we have a common ground for four own little discussion (entry was shortened):

Domain ID:
Domain Name:XENUSER.ORG
Created On:06-Feb-2010 15:56:01 UTC
Last Updated On:07-Feb-2011 01:26:14 UTC
Expiration Date:06-Feb-2012 15:56:01 UTC
Sponsoring Registrar:Key-Systems GmbH (R51-LROR)
Status:OK
Registrant ID:
Registrant Name:Valentin Hoebel
Registrant Street1:
Registrant City:
Registrant Postal Code:
Registrant Country:DE
Registrant Phone:
Registrant Email:valentin@xenuser.org
Admin ID:
Admin Name:Valentin Hoebel
DNSSEC:Unsigned

As you can see, my personal data (street, phone number etc.) is published. Everyone being able to use the WHOIS service is able to view this data and to use it (e.g. for automatized processing, abusing it and so on).

In my eyes, this is neither fair to the domain owner, nor it has a signification for something since the data maybe outdated or simply invalid. What happens if I provide my registrar with a wrong name and someone else is in trouble when I cause it?

Well, many local authorities use the WHOIS information for prosecuting people having illegal content on their website, others use this data to gain more spam addresses.

The ICANN asked for participation in the discussion about WHOIS. My contribution would be to remove all personal data from the published WHOIS entry. The reasons for this proposal are that abusing the data is possible, and you may even provide your domain registrar with wrong information or you are also often able to use a proxy service.

For clarification please see my mail to the ICANN, regarding the WHOIS debate.