Months ago I wrote a serie of articles (Italian only) about why relying on an AntiVirus only is far from being an effective approach to network safety nowadays. Today, I stumbled upon this piece, where Brian Dye, Symantec’s senior Vice President for Information Security apparently says «AntiVirus is dead».
To quote Mark Twain, I think the report of AV death is an exaggeration: nobody should -in my opinion- turn their AV off because it’s not effective anymore. It is certainly true, however, that this approach cannot be the only one in place if you plan to combat malware on your network effectively.
In the fourth part of the series above I already suggested using lists of Command & Control IPs to create nullrouting or firewall entries to inhibit network traffic trying to reach “bad resources”. I also said how one of these lists is available from Spamhaus (as that’s the one I’ve been using) and how they provide this list in the form of a BGP feed you can configure directly in your border router(s).
Whatever the list you chose and however you’re feeding it to your router, you’re going to face a problem: how to monitor what is being nullrouted and what the supposedly infected system is trying to do?
Here is what I did and how you can use a normal linux system to dump and log the blocked traffic and hijack the HTTP sessions (that are by far the most interesting ones) to obtain more intel about the infections.
Designing the landscape
Understanding where our linux system needs to be placed is important and may be less trivial than you’d think.
This is an oversimplified picture of how my network is structured:
What is labeled as “LAN” is actually much more than a single VLAN with lots of machines: it has its internal routing structure, but this is not important here.
The important things, instead, are:
- The border routers lie on the same (internal) network segment as the system we’re using as nullroute target. This means we don’t need to care about the route between the router receiving the nullrouted traffic and the nullroute target. If this were not the case, we’d need to ensure that the path followed by this traffic only traversed BGP-aware routers.
- The IPs to nullroute come in through an eBGP session. This means we can manage the destinations to be blocked as “normal routes“, something we can’t do with ACLs.
- The iBGP session between the routers redistributes the nullrouted entries to every border router, with no need to replicate the ruleset to every border router in the network.
BGP session
When you configure a BGP session aimed to nullroute traffic with a security information provider (Spamhaus in this case, but any similar service would just be the same) what you usually do is set the next-hop for the route entries announced by the peer to the Null interface:
ip route 192.0.2.1 255.255.255.255 Null0
route-map Spamhaus-BGPf permit 1000
description BGPCC
match community SH-BGPCC
set ip next-hop 192.0.2.1
In our case, however, we don’t want the traffic to be dropped by the router directly, but want to divert it to our “Nullroute Target” (I’ll simply refer to it as NRT, below) instead. So, the above becomes as follows:
route-map Spamhaus-BGPf permit 1000
description BGPCC
match community SH-BGPCC
set ip next-hop 172.23.6.100
Now our bad traffic is going to be delivered to the NIC of our NRT.
Nullroute Target network configuration
What is this machine going to receive?
The traffic is being delivered here due to a “next-hop” clause, like it was a router.
From the technical point of view this means it’s going to receive
- Ethernet frames with the NRT’s MAC address as L2 destination, containing
- IP packets with the nullrouted IP as L3 destination
Assuming our NRT is a freshly installed linux machine, what is going to happen, then?
The packets are going to be accepted by the Ethernet stack and passed to the IP stack. Since their IP destination is not our NRT machine, each packet is going to be rejected with an ICMP Destination Unreachable, ’cause the system is not a router.
That’s something, but we’d really prefer to simply drop most of the traffic while hijacking a portion of it instead, after logging it as extensively as we can.
So, as a first step we’ll tell the host it’s expected to route the traffic:
sysctl -w net.ipv4.ip_forward=1
Then we log and drop such traffic using iptables and ULOG, with a ruleset like the following:
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A PREROUTING ! -d 172.23.6.100/32 -i eth0 -j ULOG
COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A FORWARD ! -d 172.23.6.100/32 -i eth0 -j DROP
COMMIT
As you can see, we’re logging packets in the PREROUTING chain, before any decision about the whether the destination IP is local or not has been taken. We’ll see the reason for this later.
NRT logging
As said, the reason for all this mess is logging what is going to be dropped as extensively as possible. That’s why we created the ULOG rule in iptables’ ruleset above.
ULOG has several features that will greatly help us inspect the traffic we’re interested in. I strongly suggest taking your time to look at it as some of them may be interesting for you, but for now I’m sticking with my basic setup: I installed ulogd and its pcap extension, and configured my /etc/ulogd.conf as follows (most are default values, that I’m omitting):
######################################################################
# PLUGIN OPTIONS
######################################################################plugin=”/usr/lib/ulogd/ulogd_LOGEMU.so”
plugin=”/usr/lib/ulogd/ulogd_PCAP.so”[LOGEMU]
file=”/var/log/ulog/syslogemu.log”
sync=1[PCAP]
file=”/var/log/ulog/pcap.log”
sync=1
The above logs each packet hitting the ULOG iptables rule in 2 ways:
- syslog emulation in /var/log/ulog/syslogemu.log using the same format used by plain iptables LOG directive
- pcap dump of dropped packets, stored in /var/log/ulog/pcap.log
This second format is particularly interesting, as you can use tools like tcpdump or wireshark to inspect the content of the packets being dropped. This can be extremely useful when the dropped traffic is UDP, like DNS traffic.
Sinkholing HTTP traffic
Of all the traffic we’re dropping, there’s one type that is particularly interesting: HTTP traffic.
Given what we’re blocking (mostly Command&Control servers, used to instruct bots in a botnet) most if not all of the HTTP traffic we’re blocking will be caused by bots on our network trying to register and/or obtain orders from the botnet controllers. This kind of traffic will reveal which IPs on our network are infected and (by looking at the “botnet family” of the C&C) by what.
However, since we’re dropping traffic between the bot and the C&C, all we have in our logs is a bunch of SYN packets; while they can be enough to start investigating the health status of the originating system, having better info about what the supposedly compromised machine is trying to do would be extremely helpful.
We can do that, of course: we can hijack the HTTP traffic using a NAT rule, thus redirecting the traffic to a local HTTP server. This server will send every request to a script, that will capture the relevant informations and log them, while providing an innocuous page back to the client.
First of all, we’ll add an additional IP to our system’s loopback interface:
auto lo:sink
iface lo:sink inet static
address 192.0.2.254
netmask 255.255.255.255
We’ll then install our preferred HTTP server (I’ll use apache here) on the system, and configure it to bind to this IP only.
Then, we’ll modify our iptables ruleset to change the destination of every packet trying to reach port 80/TCP of a nullrouted resource
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A PREROUTING ! -d 172.23.6.100/32 -i eth0 -j ULOG
-A PREROUTING ! -d 172.23.6.100/32 -i eth0 -p tcp -m tcp –dport 80 -j DNAT –to-destination 192.0.2.254:80
COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -d 192.0.2.254/32 -i eth0 -j ACCEPT
-A FORWARD ! -d 172.23.6.100/32 -i eth0 -j DROP
COMMIT
As you can see, the DNAT takes place after the original packet has been logged.
The trick, here, is applying the DNAT to packets going to any destination except our local NRT IP address: these are all packets trying to use the NTR as a router, but this machine does not announce itself as a router, and the only traffic of this kind reaching it is the traffic we’re explicitly sending to it as a result of our BGP next-hop definition. So, we don’t need to instruct the NRT with every single nullrouted IP and apply the DNAT to those destinations only.
At this point, we need to instruct the HTTP server to send every request to a single page/script, where we’ll be doing all the real work. In order to do this, we’ll rely on apache’s mod_rewrite: assuming /var/www is our DocumentRoot, we’re going to place in it a .htaccess file containing the following:
RewriteEngine on
RewriteRule .* index.php [L]
Then we’ll create, in the same location, an index.php containing something like this:
<?php
$handlebrief = fopen(“/var/log/sinkhole/httpsink.log”, “a”);
$output = array();
$command = “sudo /usr/local/sbin/getnatIP “.$_SERVER[‘REMOTE_ADDR’].” “.
$_SERVER[‘REMOTE_PORT’].” “.$_SERVER[‘SERVER_PORT’];
exec($command,$output);
fwrite($handlebrief,$_SERVER[‘REQUEST_TIME’].” “.$_SERVER[‘REMOTE_ADDR’].”:”.$_SERVER[‘REMOTE_PORT’].
” -> “.$output[0].”:”.$_SERVER[‘SERVER_PORT’].” \””.$_SERVER[‘REQUEST_METHOD’].” “.
($_SERVER[‘SERVER_NAME’] === ‘default’ ? ‘-‘ : $_SERVER[‘SERVER_NAME’]).” “.
$_SERVER[‘REQUEST_URI’].”\”\n”);
fclose($handlebrief);
?>
As you can easily guess from the above, I’m far from being a PHP coder (let alone a good one), but the above works well enough for now.
All it does is opening a logfile and write down the main info about each connection it was called for, like the following:
1399384107 172.23.42.54:12193 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
The fields are, respectively
- Timestamp
- Source IP
- Source Port
- Destination (nullrouted) IP
- Destination port (always 80, of course)
- HTTP method (mainly POST or GET)
- Host header sent by the client
- URL requested by the client
In the case above, you can see how the client (the bot, in fact) tried a POST request to 109.120.190.24 for hxxp://application-watcher.com/server/controlsource.php
According to http://www.spamhaus.org/sbl/query/SBL220903 this IP is hosting a KINS botnet controller:
We can safely assume, then, that the machine on that IP is compromised by this Trojan.
You may have noticed how the PHP page calls an external script (through sudo) named /usr/local/sbin/getnatIP, passing the source IP and port of the current HTTP request as argument. This is done because the PHP script can’t see the original IP the HTTP client was trying to reach, since the destination IP has been rewritten by iptables. We can still recover that information, however, looking at the conntrack table used by netfilter/iptables (in /proc/net/ip_conntrack), except this resource is not accessible to apache but to administrators only.
For this reason, I created a simple script (/usr/local/sbin/getnatIP, indeed):
#!/usr/bin/perl
die “Source address is not an IP\n” if ( $ARGV[0] !~ /^\d+\.\d+\.\d+\.\d+$/);
die “Source port is not valid\n” if ( ($ARGV[1] !~ /^\d+$/) or (0 > $ARGV[1]) or ($ARGV[1] > 65535) );
die “Destination port is not valid\n” if ( ($ARGV[2] !~ /^\d+$/) or (0 > $ARGV[2]) or ($ARGV[2] > 65535) );my $SAddr = $ARGV[0];
my $SPort = $ARGV[1];
my $DPort = $ARGV[2];
my $DAddr;open CONNTR, “</proc/net/ip_conntrack”;
while (<CONNTR>) {
if ($_ =~ / ESTABLISHED src=(\d+\.\d+\.\d+\.\d+) dst=(\d+\.\d+\.\d+\.\d+) sport=(\d+) dport=(\d+) /) {
if ( ( $1 eq $SAddr )
and ( $3 eq $SPort )
and ( $4 eq $DPort ) ) {
$DAddr = $2.”\n”;
last;
}
}
}
close CONNTR;
print “$DAddr”;
This script expects -as seen above- the current session IP and ports as arguments, then opens /proc/net/ip_conntrack, searches for the conntrack entry matching them and exits returning the original IP before the NAT.
Make the script executable by root only, then add an entry to your sudoers file, allowing www-data (or the user your webserver runs as) to run it through sudo without providing any password:
www-data ALL=(ALL)NOPASSWD:/usr/local/sbin/getnatIP
And you’re done. The resulting log file will be something like the following:
1399385849 172.23.42.54:12315 -> 109.120.190.24:80 “POST service-controller.com /server/controlsource.php”
1399385850 172.23.42.54:12315 -> 109.120.190.24:80 “POST service-controller.com /server/controlsource.php”
1399385850 172.23.42.54:12319 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
1399385850 172.23.42.54:12321 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
1399385855 172.23.42.54:12321 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
1399385855 172.23.42.54:12319 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
1399385860 172.23.42.54:12319 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
1399385860 172.23.42.54:12321 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
1399385865 172.23.42.54:12321 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
1399385865 172.23.42.54:12319 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
1399385871 172.23.42.54:12319 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
1399385871 172.23.42.54:12321 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
1399385899 172.23.17.200:26536 -> 92.53.119.248:80 “POST 92.53.119.248 /ssdc32716372/login.php”
1399386030 172.23.17.200:36238 -> 188.225.38.251:80 “POST 188.225.38.251 /ssdc32716372/file.php”
1399386199 172.23.17.200:22385 -> 92.53.119.248:80 “POST 92.53.119.248 /ssdc32716372/login.php”
1399386499 172.23.17.200:34712 -> 92.53.119.248:80 “POST 92.53.119.248 /ssdc32716372/login.php”
1399386800 172.23.17.200:56181 -> 92.53.119.248:80 “POST 92.53.119.248 /ssdc32716372/login.php”
1399386891 172.23.17.200:31117 -> 188.225.38.251:80 “POST 188.225.38.251 /ssdc32716372/file.php”
1399387023 172.23.42.54:12378 -> 109.120.190.24:80 “POST service-controller.com /server/controlsource.php”
1399387029 172.23.42.54:12378 -> 109.120.190.24:80 “POST service-controller.com /server/controlsource.php”
1399387034 172.23.42.54:12378 -> 109.120.190.24:80 “POST service-controller.com /server/controlsource.php”
1399387039 172.23.42.54:12378 -> 109.120.190.24:80 “POST service-controller.com /server/controlsource.php”
1399387044 172.23.42.54:12378 -> 109.120.190.24:80 “POST service-controller.com /server/controlsource.php”
1399387045 172.23.42.54:12380 -> 109.120.190.24:80 “POST application-watcher.com /server/controlsource.php”
Feel free to look up the Spamhaus website to see what is what (if the related SBL pages are still up when you read this).
Of course, the above is only an example: you may want to change the PHP script to do other things, like dumping all the request details to a file, or write everything into a database, or send an alert email, or…
My next goal, for sure, will be integrating it with the data provided by Spamhaus about the reasons for the listing, trying to extract info from the SBL page. My first attempts at doing that are not exactly a success, but as said I’m a terrible coder, particularly wrt web-related stuff and PHP particularly…
I’ll probably need to switch to a language I’m more comfortable with (another name for “perl”) and perhaps ask SH people if they are providing the same informations in an easier-to-parse format (through an API, maybe?).
Having the listing details stored with the record entry will result in a much more useful set of info available to the internal NOC when a botnet alert is triggered.
Just a final note. If you are familiar with routing, you’ve certainly realised how the above doesn’t really require your network to run BGP.
As a matter of fact, for most corporate networks doing the above using BGP is a problem: BGP is not always available or used and, when it’s used, it’s used on the border routers, outside the firewall and therefore “after source NAT has been applied”. If we run our NullRouting+Hijacking there we lose an extremely interesting information: the IP of the compromised client, unless we peek into the firewall’s translation tables or logs and correlate. On the other side, on many firewall platforms, running BGP is simply not supported, so running a BGP session there with the sole purpose of nullrouting this bad traffic is not an option.
However, firewall appliances (at least serious ones) usually support an IGP of some kind (like OSFP).
If you are in a situation like this one, then you can replace the BGP config with
- a script retrieving the list of resources to nullroute (HTTP/FTP/rsync/whatever)
- a script pushing and pulling such IPs into a route server configuration (somehow)
- a route server announcing these IPs in your network through your IGP of choice
Your NRT will then divert all the traffic trying to reach C&C servers to itself, and hijack the HTTP portion of such traffic, with exactly the same results as above.
Perhaps I’ll get on this another day and describe how to make it work.
Skull's blog 
Leave a Reply