Being written in 2004 and heaving heavily evolved ever since, dstat has become a replacement for most of the tools being mentioned above (at least for me). dstat is very flexible and shows you the information you want in real time, e.g. RAM and CPU usage, bandwith, I/O in total, MySQL-stats, interrupts and performance bottlenecks.

Introduction to dstat
In this blog post, the latest version of dstat (0.7.2, 03/09/2010) on Debian 6 will be covered. Please notice that I will only cover the real-time monitoring features of this tool, but not the “export data to a csv-file and use it for my own charts”-stuff.

For the start let’s simply install and call dstat:
apt-get install dstat
dstat

As you can see, dstat starts with some default parameters and shows the CPU, net, disk and paging usage. Furthermore it indicates the current amount of hardware interrupts and context switches. If you already looked at man dstat, you might guess that calling “dstat” equals to “dstat -c -d -n -g -y”:

• -c = total CPU usage
• -d = disk total
• -n = net total
• -g = paging system
• -y = system stats, such as the hw interrupts and the context switches

For most ambitious sys admins, this is not enough. So let’s take a look at some of the most useful parameters:

• -l = shows load statistics
• -m = shows the memory usage (used, buffer, cache, free)
• -r = displays I/O statistics,
• -s = shows the swap usage
• -t = puts the current time in the first column
• --fs = displays file system stats (includes amount of files and used inodes)
• --nocolor = sometimes very useful…
• --socket = shows interesting network statistics
• --tcp = displays common TCP stats
• --udp = shows you the listen and active figures for the UDP usage

The parameters mentioned above already make dstat a very powerful monitoring tool, and we haven’t covered the external plugins yet Have a look at /usr/share/dstat in order to view them. Some of them are:

• --disk-util = shows how much the disks are busy at the moment
• --freespace = shows the current disk usage
• --proc-count = displays the number of running processes
• --top-bio = points to the most expensive block I/O process
• --top-cpu = draws the attention on the most expensive CPU process
• --top-io = shows the most expensive “normal” I/O process
• --top-mem = displays the process using the most memory

Simply call “dstat –plugin name” in order to use them.
Example:
dstat --proc-count

There are also many MySQL and NFS plugins, simply have a look at them and try them out (do “dstat –list” to view the list of available plugins) In addition there is an wifi extension which displays the signal power and noise.

Some useful examples
After having looked at dstat’s potential, we are now ready to use it.

In the first example we want to cover everything which is related to the disk:
dstat -d --disk-util --freespace

In the second example we simply want to see who is eating all the memory:
dstat -g -l -m -s --top-mem

The third example shows some stats to the CPU resource consumption:
dstat -c -y -l --proc-count --top-cpu

Example number four will show some beautiful network statistics:
dstat -n --socket --tcp --udp

As you can see, there are many things you can observe with dstat. I highly recommend to have a look at dstat’s manpage and get in touch with all the other parameters.

Writing your own dstat plugin
Writing a new dstat plugin should be easy. I have not tried it yet, but /usr/share/dstat/dstat_helloworld.py shows how simple this should be:

### Author: Dag Wieers

class dstat_plugin(dstat):
"""
Example "Hello world!" output plugin for aspiring Dstat developers.
"""

def __init__(self):
self.name = 'plugin title'
self.nick = ('counter',)
self.vars = ('text',)
self.type = 's'
self.width = 12
self.scale = 0

def extract(self):
self.val['text'] = 'Hello world!'


Final words
I think Dag did great work by writing this tool really saving time by summarizing many tools into one.
You can visit his website here and view the latest changelog for dstat here.

Jesus, what mess! But there is a solution. In this tutorial I try to show you how to plan everything, make the right decisions and finally come to the conclusion that a VPS with a high-performance setup is more than enough for your purpose.

Choosing the right hosting plan
Running a forum is a real challenge and can drain heavily from your financial reserves. Many people use a small shared hosting offer because of the costs, others want to rent a dedicated machine and think this is necessary because of the thousands of people who could come and visit every month.

But in fact running a good forum only takes a small, but powerful virtual server. A shared hosting plan often is not enough since the provider sets strict limits on performance and traffic. As soon as you come close to those limits, you either pay or get kicked out.

On the other side, there are the people who have some money and decided themselves for a dedicated server. This is not a bad choice, of course. But is this really necessary? Unless you expect something like 10 000 members for your forum or less, a VPS may still serve the purpose.

So, I highly recommend to grab a small VPS server and upgrade the server/transfer your community later (if needed). If you have enough money you might even consider renting a cloud instance which can be scaled without blowing RAM or CPU limits.

I personally would stick with a 13 Euro VPS (approx. 18 Dollar), which leads you to 150 Euros or less server costs per year. And there is still enough room for a cheap domain

Which operating system?
You have the server? You already purchased a cool domain? Now you are ready to get started? Great! Now we should make our minds up and select a decent OS for you.

Of course there is Windows and Win2008 Web Edition. This operating system is expensive, but works and is supported by a large company. But on the other hand, we don’t want to spend a few hundred Euro just for licensing and stuff like this. So clearly we decide ourselves for a Unix or Linux.

Unix is great, especially the *BSD variants. But if you were able to do your daily work on a BSD machine, you wouldn’t read this article and have already setup your high availability cluster with a self-compiled and optimized kernel

So, there is only one type of operating systems left: Linux. Linux comes along with a huge variety of distributions, and obviously we need something which is stable, fast, easy to handle, supported by many software vendors and has a large community behind it. This leads us to either CentOS as a RedHat clone, Ubuntu Server or Debian (this list is based on my own experience, maybe you are missing OpenSuse or something else here..). Since we don’t want to pay for our OS, this list is very obvious for me.

Let’s look at the operating systems: CentOS sometimes lacks of easy handling (have you upgraded a machine from CentOS X to CentOS Y yet? Jesus, what a *****!), but profits from its big brother RHEL. Ubuntu Server depends on Canonical, but is very easy to be configured etc. Debian sometimes is too conservative, but always stable and runs on almost every hardware on the planet. Well, the differences are not always clear to see and you should select the distribution you are more familiar with. For me, this is either Ubuntu or Debian, and chose Debian. Better wait longer for updates than have new, but rarely bugged software on my system.

Which software packages?
When thinking of web server software, there are many applications jumping to my mind. There are Apache, Cherokee, Lighthttpd, nginx, Tornado…. All of them are great, but Apache is not that good scalable for huge loads. Cherokee and Tornado are both very innovative, but are the wrong choice for a normal forum software (at least that is my opinion – feel free to share your own by dropping me a mail!). This leads us to decide between either Lighthttpd or nginx. Both of them do great in large setups, both of them have their flaws and both of them are already used by large companies. When it comes down to documentation, the Lighthttpd server clearly is the leader here. On the other hand, there are statistics showing that nginx is more stable (Lighthttpd had so many memory leaks issues during the last 3 years), more popular (this changed just a few months ago, I think) and consumes less CPU resources than the other favorite.

So, let’s go with nginx! Furthermore I used php-fpm, which is far better than the mod_php of Apache you might be familiar with and also slightly better than spawn-fcgi, according to this blog entry. What impressed me the most was the fact that php-fpm has the decent “process management ability  to “graceful” stop and start php workers without losing any queries.” In addition it comes with the “possibility of gradually update the configuration and binary without losing any queries.” (Source: http://adityo.blog.binusian.org/?p=428) Furthermore we will use php-apc for caching the content. This helps us to keep the memory consumption of our small VPS very low.

Last but not least is the database software. Which one to choose? Oracle, Postgres, MySQL .. Oracle is too expensive and clearly not necessary here, which leaves Postgres and MySQL on the list. As far as I know, both database systems do great in heave-load situations. However, Postgres beats MySQL by a better handling for sub queries and joins (at least they are faster). This is why we will choose Postgres for our small VPS setup.

Forum software – cheap, but many features plz!
When thinking of forums, WBB, PHPBB, vBulletin, SMF and Invision Power Board come to my mind. vBulletin and WBB are both commercial forum packages, so that’s a “No way!” from me (although both of them are great, though).  Invision Power Board and SMF are not widely spread, but have their advantages. But for me this is not enough to stand up against PHPBB3, probably one of the most successful PHP applications of all times. There are so many free mods and styles for this internet forum software and it is so simple to install and use – so, why not?

Let’s go! Preparing the VPS
I assume that you installed Debian 6 64 Bit (I tested the following steps with also with a 32 Bit Debian, btw), setup your network interface, have a SSH server and DNS + the other basic stuff works. You are now logged in, located at /root/ and ready to go.

At first we install some useful packages (since I wrote this tutorial for my ego I only install software which I find useful ):

apt-get update && apt-get upgrade
apt-get install htop cbm atop dstat dvtm ssh vim bzip2 unzip mc vim lsof

Now let’s extend your sources.list in order to gain access to more packages:

echo deb http://packages.dotdeb.org stable all >> /etc/apt/sources.list
wget http://www.dotdeb.org/dotdeb.gpg
cat dotdeb.gpg | apt-key add -
rm dotdeb.gpg
apt-get update

Installing nginx and co
Now let’s get started with the fancy stuff:

apt-get install nginx php5-fpm php5-pgsql php5-gd php5-curl php-apc postgresql imagemagick

Configuring nginx and our vhost
Now open the configuration file of nginx:

vim /etc/nginx/sites-available/www.domain.tld

Edit this file until it looks like the one below. Please remember to replace www.domain.tld with your own domain name:

server {
listen 80;
server_name domain.tld www.domain.tld;

access_log /var/log/nginx/domain.access_log;
error_log /var/log/nginx/domain.error_log;

root /var/www/www.domain.tld;
index index.php index.htm index.html;

location ~ .php$ {
fastcgi_pass   127.0.0.1:9000;
fastcgi_index  index.php;
fastcgi_param  SCRIPT_FILENAME /var/www/www.domain.tld$fastcgi_script_name;
include fastcgi_params;
try_files $uri =404;
}
}

Attention: Don’t forget to add “try_files $uri =404;”! This one closes a security vulnerability which lets attackers parse normal image and text files as PHP scripts.

Ok, let’s continue by doing the necessary stuff on the filesystem layer:

ln -s /etc/nginx/sites-available/www.domain.tld /etc/nginx/sites-enabled/www.domain.tld
mkdir -p /var/www/www.domain.tld

Configure php-fpm to work with Postgres
We need to edit the right php.ini in order to let PHP talk with Postgres and vice versa:

vim /etc/php5/fpm/php.ini

Search the file for [Postgres] and add a new line directly below:

extension=pgsql.so

Now restart php-fpm and start nginx itself:

/etc/init.d/php5-fpm restart
/etc/init.d/nginx start

Downloading PHPBB3
Now let us take care of the forum itself:

cd /var/www/www.domain.tld
wget http://www.phpbb.com/files/release/phpBB-3.0.9.tar.bz2
tar jxf phpBB-3.0.9.tar.bz2
rm phpBB-3.0.9.tar.bz2
mv phpBB3/* . && mv phpBB3/.htaccess .
rmdir phpBB3/

Prepare Postgres for PHPBB3
Before installing PHPBB3 we should prepare Postgres:

su postgres -c psql
\connect template1
alter user postgres with password 'yourSecretPasswordHere';
CREATE USER "www-data" WITH PASSWORD 'yourOtherSecretPasswordHere';
CREATE DATABASE phpbb3;
GRANT ALL PRIVILEGES ON DATABASE phpbb3 TO "www-data";
ALTER DATABASE phpbb3 OWNER TO "www-data";
\q
/etc/init.d/postgresql restart

Optional: Installing phppgadmin
If you have used MySQL before, there might be a high chance that you already know PHPMyAdmin, the great web-based tool for managing MySQL databases. There is a direct equivalent for Postgres, although it lacks some features of its bigger brother:

mkdir /var/www/www.domain.tld/db_admin
cd db_admin
wget http://downloads.sourceforge.net/phppgadmin/phpPgAdmin-5.0.2.tar.bz2?download
tar jxf *
rm phpPgAdmin-5.0.2.tar.bz2\?download
mv phpPgAdmin-5.0.2/* .
rmdir phpPgAdmin-5.0.2/

If you want to you can now open up your browser, visit www.domain.tld/db_admin, login with the www-data user and view your two databases. From now on, you can manage all the database stuff with this web application. Secure the db_admin directory with chmod 000 or use a htpasswd file in order to protect it from unauthorized access.

Installing PHPBB3 – finaly!

cd /var/www/www.domain.tld
chmod 777 config.php

Now open your browser, visit www.domain.tld and follow the installation instructions. Make sure to perform the following actions when you completed all the steps from the PHPBB3 wizard:

rm /var/www/www.domain.tld/install -r
chmod 640 config.php

Now go and browse your awesome new forums on your high-performance monster!

Bonus: View apc statistics
If you want to check if apc is really working you could..

gunzip /usr/share/doc/php-apc/apc.php.gz
cp /usr/share/doc/php-apc/apc.php /var/www/www.domain.tld/db_admin/

Now call www.domain.tld/db_admin/apc.php with your browser and view the statistics

That’s it!
That’s it, we made it! We just installed a high-performing monster in a matter of minutes by using free open source software. And it was so easy!
Of course we missed a view things, such as setting up the rewriting stuff for PHPBB3 and do some performance tuning. However, you are now ready to go and have fun with your low-budget setup.

PS: The missing stuff will be added in future tutorials. So be sure to visit this blog from time to time

Being written in Perl, shelldap tries to emulate a shell-like environment while you browse through a LDAP database. And yes, it supports [Tab] auto completion!

Here are some examples:

% shelldap
~ > [ tab ]
~ >
cat     clear   cp      delete  env     grep    id      ls      move    passwd  read    search  touch   whoami
cd      copy    create  edit    exit    help    list    mkdir   mv      pwd     rm      setenv  vi
~ > cd ou=People
ou=People,~ > cat uid=ma[ tab ]
ou=People,~ > cat uid=mahlon

dn: uid=mahlon,ou=People,dc=laika,dc=com
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
objectClass: posixAccount
objectClass: shadowAccount
objectClass: apple-user
cn: Mahlon E. Smith
departmentNumber: Technology - IT
displayName: Mahlon
gecos: Mahlon E. Smith
gidNumber: 200
givenName: Mahlon
homeDirectory: /home/m/mahlon
l: Ghetto
loginShell: /bin/tcsh
mail: mahlon@martini.nu
mobile: 1+8829999005747
sn: Smith
title: Manager, Information Systems
uid: mahlon
uidNumber: 20933
userPassword: *

ou=People,~ >

Visit the shelldap website here.

Description
The Simple LAN Scanner is a very simple LAN scanner written in Python. It scans the local network and tries to give you the MAC and IP addresses of the discovered running systems. Furthermore it creates a small log file at the end of the scan.

Usage
sudo ./simple_lan_scan.py –network=<your network>

Usage example
sudo ./simple_lan_scan.py –network=192.168.1.0/24

Installation
Make sure you install the package python-scapy before you run the Simple LAN Scanner.

Feature list
- Tries to give you the MACs and IPs of discovered running systems.
- Creates a small log file.

Some notes
- Tested with Python 2.6.5.
- Modify, distribute, share and copy the code in any way you like!
- Please note that this tool was created for educational purposes only.
- Power to teh c0ws!

Description
The Simple Local File Inclusion Vulnerability Scanner helps you to find LFI vulnerabilities.

Usage
./lfi_scanner.py –url=

Usage example
./lfi_scanner.py –url=”http://www.example.com/page.php?file=main”

Usage notes
- Always use http://….
- This tool does not work with SEO URLs, such as http://www.example.com/news-about-the-internet/.
- If you only have a SEO URL, try to find out the real URL which contents parameters.

Feature list
- Provides a random user agent for the connection.
- Checks if a connection to the target can be established.
- Tries to catch most errors with error handling.
- Contains a LFI vulnerability scanner.
- Finds out how a possible LFI vulnerability can be exploited (e.g. directory depth).
- Supports nullbytes!
- Supports common *nix targets, but no Windows systems.

Known issues
- This tool is only able to handle “simple” LFI vulnerabilities, but not complex ones.
- Like most other LFI scanners, this tool here also has trouble with handling certain server responses.

Some notes
- Tested with Python 2.6.5.
- Modify, distribute, share and copy the code in any way you like!
- Please note that this tool was created for educational purposes only.
- Do not use this tool in an illegal way. Know and respect your local laws.
- Only use this tool for legal purposes, such as pentesting your own website
- I am not responsible if you cause any damage or break the law.
- Power to teh c0ws!

Screenshot

Simple Local File Inclusion Vulnerability Scanner screenshot

With great pleasure I hereby announce the availability of the new “The Joomla Hacking Compendium”. It contains almost 1000 lines of pure knowledge and shows you the way to hack and protect Joomla.

It contains the following chapters:

0x01 - Purpose of this document
0x02 - Introduction
0x03 - The Basics of Joomla
0x04 - The Joomla core
0x05 - Joomla extensions
0x06 - Hacking Joomla
0x07 - SEO, our strongest enemy
0x08 - Examples for Joomla SQL injections
0x09 - Examples for Joomla local file inclusions
0x10 - Examples for Joomla remote file inclusions
0x11 - Examples for Joomla XSSs/CSRFs
0x12 - How to protect your Joomla
0x13 - Conclusion and a look at Joomla's feature
0x14 - How to stay informed (or: the latest vulnerabilities)
0x15 - Useful tools
0x16 - Greetings and THX

Please find an excerpt below:

::
:: 0x04 - The Joomla core
::

Before inspecting the Joomla component attack vendors we first have a
look at the core.

Download Joomla somewhere and extract all files. Open the file
libraries/phpinputfilter/inputfilter.php
and look at the code:
----------------------------------------
var $tagsArray; // default = empty array
var $attrArray; // default = empty array

var $tagsMethod; // default = 0
var $attrMethod; // default = 0

var $xssAuto; // default = 1
var $tagBlacklist = array ('applet', 'body', 'bgsound' [...]
var $attrBlacklist = array ('action', 'background'     [...]
----------------------------------------

As you can see, some filter methods of Joomla are based on blacklisting.
This knowledge can be used later to exploit potential vulnerabilities in
a better way. I find this method not very effective, btw.

While HTML tags containing "body" or "bgsound" will be filtered out
at input fields or URL parameters, they can be written in many ways,
e.g. like "bOdY" or "b o DY" etc. You are only limited by your
creativity and will find ways for tricking the blacklist of the
Joomla framework.

Another interesting part is this one (same file):
----------------------------------------
/*
* Is there a tag? If so it will certainly start with a '<'
*/
$tagOpen_start  = strpos($source, '<');
while ($tagOpen_start !== false)
{
/*
* Get some information about the tag we are processing
*/
$preTag            .= substr($postTag, 0, $tagOpen_start);
$postTag                = substr($postTag, $tagOpen_start);
----------------------------------------

As you can see they assume that an HTML tag being used in XSS attacks
starts with a "<". In fact, I never use this character and many
XSS cheatsheets suggest this, too. With this information in mind,
you can most likely avoid being detected by the filters. You can start
your XSS string with "><tag... for example.

If you want to you can continue looking. You will find other filter
methods and, at the end of the file, there are also built in
mechanics which should help to prevent SQL injection vulnerabilities:
[...]

Just visit the “My Tools” section for the download link.

Description
The Simple Local File Inclusion Exploiter helps you to exploit LFI vulnerabilities. After you found one, simply pass the URL of the affected website and the vulnerable parameter to this tool. You can also use this tool to scan a parameter of an ULR for a LFI vulnerability.

Usage
./lfi_sploiter.py –exploit-url= –vulnerable-parameter=

Usage example
./lfi_sploiter.py –exploit-url=http://www.example.com/page.php?file=main –vulnerable-parameter=file

Usage notes
- Always use http://….
- When you pass a vulnerable parameter, this tool assumes that it is really vulnerable.
- If you do not know if a parameter is vulnerable, simply pass it to this script and let the scanner have a look.
- Only use one vulnerable parameter at once.
- This tool does not work with SEO URLs, such as http://www.example.com/news-about-the-internet/.
- If you only have a SEO URL, try to find out the real URL which contents parameters.

Feature list
- Provides a random user agent for the connection.
- Checks if a connection to the target can be established.
- Tries catch most errors with error handling.
- Contains a LFI scanner (only scans one parameter at once).
- Finds out how a LFI vulnerability can be exploited (e.g. directory depth).
- Supports nullbytes!
- Exploit features: Dumps a list of interesting files to your hard disk.
- Supports common *nix targets, but no Windows systems.

Known issues
- I know there is more about LFI than it is covered in this tool. But this is the first release,
and more features will be implemented in future versions.
- This tool is only able to handle “simple” LFI vulnerabilities, but not complex ones. For example: Some LFI vulnerabilities consist of two URL parameters or require to find a way around filters. In those cases, this tool unfortunately does not work.
- Like most other LFI exploiter / scanner, this tool here also has problems with handling certain server responses. So this tool does not work with every website.

Some notes
- Tested with Python 2.6.5.
- Modify, distribute, share and copy the code in any way you like!
- Please note that this tool was created for educational purposes only.
- Do not use this tool in an illegal way. Know and respect your local laws.
- Only use this tool for legal purposes, such as pentesting your own website
- I am not responsible if you cause any damage or break the law.
- Power to teh c0ws!

Screenshot

Simple Local File Inclusion Exploiter screenshot

Currently, the tool is still beta (version 0.8). I guess that it will be released during the next 48 hours.

It contains all the features from the first released version 0.3 and now contains – in addition – a column fuzzer. Simply start a scan by using python sqli_scanner.py -u “target” and then start fuzzing by using the parameter -fuzz “exploit url”. The exploit url will be provided by the scanner (when a vulnerability was found).