2006-01-01 hunting the initial vector - Chas Tomlin (cet@ecs.soton.ac.uk)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The first hack of the year came earlier than expected for 2006, only hours
into the year and a host fell victim to attackers. The network under attack is
laid out something like below. The firewall/snort box is a Sguil sensor that
runs snort, captures full content and also TCP session data. The sensor has
around 200GB of disk space which is normally good to keep full pcap data for
about two weeks. Alerts and session data are pushed up to a central Sguil
database server.
 

        +----------+        +----------+         +----------+
        |   WAN    |        | Firewall |         |  10/100  |
    2Mb |  ROUTER  |  100Mb |          |  100Mb  |  Switch  |
        |          |        |  Snort   |         |          |
        +----------+        +----------+         +----------+

                                                    100Mb

                                                 +----------+
                                                 |          |
                                                 |  Victim  |
                                                 |          |
                                                 +----------+


The first indication that something was amiss was 100% line utilization on
the 2Mb WAN interface, the ISP who supplies the leased line informed us of this.
The 10/100 switch is a managed device that runs SNMP, the switch is queried
periodically by MRTG to produce traffic graphs for each connected host. From
the MRTG graphs we could easily determine which host was hogging all the
bandwidth. After logging into the offending Linux host a quick glance at the
process list showed an unusual script called cw running, the host was then
powered off.

Before the server underwent physical analysis I reviewed the Sguil data to see
if I could determine the initial vector of the attack. To my surprise there
were no alerts relating to this host. So I turned to the full content data,
however there was none of any use, only around 190GB of UDP junk. The host had
been used in a denial of service attack against a host on the internet. Under
normal conditions the Sguil sensor would be capable of storing two weeks of
full content data before it would be overwritten this is due to traffic being
limited by the 2Mb leased line. In this scenario the compromised host had a
100Mb path to the WAN router which passes the Sguil sensor. This meant that
the full content data was over written in a matter of hours rather than weeks
losing any full content record of the initial attack vector.

The last type of data that the Sguil sensor stores is session. On this
particular sensor session statistics are collected via a patch to Snort which
only logs TCP sessions. This is not the preferred method for collecting session
stats and usually sancp is used which collects ICMP, UDP and TCP data. I just
haven't got round to installing sancp on this sensor, this might not have been
a bad thing though. The DOS attack used UDP packets which weren't logged
by the Sguil sensor, if I had been running sancp I suspect it would have
generated a very large amount of session stats that might have made it
difficult to analyse. 

So I began to look at the TCP sessions to and from the server around the time
of the attack. The first sign of anything malicious was a session that looked
like a backdoor to a remote server on port 6667 (IRC). So I decided to see if
I could establish where the first backdoor session took place and look at
the surrounding sessions for something that resembled an exploit.  Again from
web application exploit experience I knew I was looking for something  that
would look like a normal inbound connection to port 80 followed by the web
server initiating an outbound session probably to port 80 to download some
malicious code. I found a sequence of sessions that followed this exact pattern
and determined that this was the initial vector. The server was exploited
through a web application of some sort but without any further data
available from Sguil it was necessary to turn to host based forensics.

The server was removed from the rack in the data centre and moved to a
different network for analysis. The single IDE hard drive that was in the
server was removed from the chassis and placed into an analysis PC. The
analysis PC runs a version of Linux developed in house that boots via PXE and
runs from RAM. At this stage normally the disk is cloned and analysis takes
place on the cloned disk, however time was at a premium so analysis took place
on the original disk in read only mode. 

From experience of investigating compromised Linux web hosts I first looked in
the /tmp directory for any files owned by the apache user. Unsurprisingly
there were three files that fitted the bill. These were called cw, httpd and
udp.pl. From the analysis PC the three files were copied to a remote file
server for further investigation later.  

I then decided to see if the web server logs would give any indication of the
initial vector. Apache log files on Red Hat Linux are typically found in 
/var/log/httpd, a quick look in this directory showed that the attacker had
removed any access logs to hide their activity. However they didn't delete any
of the error logs and one contained the following.

httpd(1028): Operation not permitted
--23:00:39--  http://members.lycos.co.uk/img0d/httpd
           => `httpd'
Resolving members.lycos.co.uk... done.
Connecting to members.lycos.co.uk[212.78.204.20]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 29,124 [text/plain]

    0K .......... .......... ........                        100%  148.91 KB/s

23:00:39 (148.91 KB/s) - `httpd' saved [29124/29124]


At this point I could have worked through the web servers content and guessed
what was used as the initial vector, but this seemed to inaccurate for my
liking. So I turned to the files that the attacker had downloaded onto the
server. Two of the files were binaries the other a perl script, the script was
just a DOS tool for crafting UDP packets. Running strings over the two
binaries revealed lots of useful information. The cw binary was pretty small
and contained amongst other things the string;

Cock whipping host : %s with leet packets

Probably a DOS tool of some sort. The httpd binary was a little more
interesting and running strings over it revealed many things including  
IRC commands and IP addresses. This looked like the backdoor that was being
used to control the host. I still needed more information to determine the
initial attack vector so I decided that I would try and connect up to the IRC
server to see if this would reveal any more info. I found strings in the
binary that looked like the IRC server IP address (which I already had from
the session data) and the channel that the botnet was running on. I had
connected up to botnets before using a simple perl script which would appear
like the real thing but be completely dumb, and allow me to log any activity
on the channel. So I altered my script to look like what I thought the
backdoor would look like from analysing the binary and tried to hook up to the
server. It didn't work, I tried a few more times but no luck. So I thought I
would have to run the real binary on a test network to see how it would
connect. I ran up VMWare and installed a basic version of Red Hat Linux 9 with
no patches. I made sure I had Sguil monitoring the network and I started the
backdoor, let it connect then suspended the virtual machine. I pulled up the
full content data in ethereal and noted the exact details of how the client
connected. I then altered my IRC perl script to mimic the client as closely as
possible and was able to hook up to the IRC server. 

It wasn't long before I noticed some operators on the channel actually
chatting to each other. I was amazed that they were doing this so openly and I
followed their conversation. Much to my surprise one operator revealed how
he was exploiting all the hosts, using the old xmlrpc vulnerability.

I checked back on the server and sure enough found the vulnerable xmlrpc.php
file. I decided that I would try and recover the deleted log files to see if
this would confirm what the 'hackers' claimed. I used netcat and dd to make an
image of the / partition and store it on a remote machine. This took a while
but once done I had a single file that held the contents of the entire root
partition. I ran strings over the file searching for what I thought to be the
initial attack source IP address. After a while even I was surprised when my
terminal revealed;

# strings hda2 |grep "81.218.24.251"
81.218.24.251 - - [01/Jan/2006:22:07:27 +0000] "POST /phpAdsNew/adxmlrpc.php
HTTP/1.1" 200 375 "-" "Internet Explorer 6.0"
81.218.24.251 - - [01/Jan/2006:22:54:14 +0000] "POST /phpAdsNew/adxmlrpc.php
HTTP/1.1" 200 375 "-" "Internet Explorer 6.0"
81.218.24.251 - - [01/Jan/2006:22:55:09 +0000] "POST /phpAdsNew/adxmlrpc.php
HTTP/1.1" 200 375 "-" "Internet Explorer 6.0"
81.218.24.251 - - [01/Jan/2006:22:56:03 +0000] "POST /phpAdsNew/adxmlrpc.php
HTTP/1.1" 200 375 "-" "Internet Explorer 6.0"
81.218.24.251 - - [01/Jan/2006:22:57:06 +0000] "POST /phpAdsNew/adxmlrpc.php
HTTP/1.1" 200 375 "-" "Internet Explorer 6.0"
81.218.24.251 - - [01/Jan/2006:22:57:59 +0000] "POST /phpAdsNew/adxmlrpc.php
HTTP/1.1" 200 375 "-" "Internet Explorer 6.0"
81.218.24.251 - - [01/Jan/2006:22:58:54 +0000] "POST /phpAdsNew/adxmlrpc.php
HTTP/1.1" 200 375 "-" "Internet Explorer 6.0"
81.218.24.251 - - [01/Jan/2006:22:59:46 +0000] "POST /phpAdsNew/adxmlrpc.php
HTTP/1.1" 200 375 "-" "Internet Explorer 6.0"

Bingo! 


Resources
~~~~~~~~~

the simple perl IRC client
IRC conversation
PXE forensic client
Sguil
Snort

Questions and Answers
~~~~~~~~~~~~~~~~~~~~~

Q.)Why was there no alert for the initial vector from Snort?  

A.)This sensor runs the official Snort ruleset which currently conatins the
following rule for detecting the xmlrpc.php exploit;

alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"WEB-PHP xmlrpc.php
post attempt"; flow:to_server,established; uricontent:"/xmlrpc.php"; nocase;
pcre:"/^POST\s/smi"; reference:bugtraq,14088; reference:cve,2005-1921;
classtype:web-application-attack; sid:3827; rev:1;)

the phpAdsNew install prefixes the xmlrpc.php file with ad (adxmlrpc.php) which
will not be matched by the official Snort rule :(

Q.)Did the attackers do anything else on the host? 

A.)The attackers did not install any further rootkit or escalate privileges
past that of the apache user. The apache user provided them with enough
privileges to DOS other hosts on the internet which was enough for them in the
short time that the machine was compromised. Later monitoring of the IRC
channel revealed they would install rootkits on the hosts manually picking
those hosts with older kernels probably so they can use older do_brk and mmap
exploits to escalate privileges.