Tons of Empty HTTP POST Requests to Your Web Server? It's the Pushdo Botnet.

You're checking your Apache logs one morning and you're finding WebLogExpert taking a bit longer than usual to create your activity report.

That's strange. Maybe the log file was corrupted?

Checking your day's log, you find the file is over a dozen times larger than usual. Multiple megabytes over the usual size. After opening the file in your text editor, you find it full of nonsensical blank entries. Tens of thousands of lines of this:

[IP Address Removed] - - [21/Sep/2013:00:01:08 +0000] "POST / HTTP/1.1" 301 499 "-"
 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
[IP Address Removed] - - [21/Sep/2013:00:01:08 +0000] "POST / HTTP/1.1" 301 499 "-"
 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
[IP Address Removed] - - [21/Sep/2013:00:01:09 +0000] "POST / HTTP/1.1" 301 499 "-"
 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
[IP Address Removed] - - [21/Sep/2013:00:01:10 +0000] "POST / HTTP/1.1" 301 499 "-"
 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
[IP Address Removed] - - [21/Sep/2013:00:01:10 +0000] "POST / HTTP/1.1" 301 499 "-"
 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
[IP Address Removed] - - [21/Sep/2013:00:01:11 +0000] "POST / HTTP/1.1" 301 499 "-"
 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
[IP Address Removed] - - [21/Sep/2013:00:01:12 +0000] "POST / HTTP/1.1" 301 499 "-"
 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

Uh, oh. For some reason, out of nowhere, a huge amount of garbage traffic has been arriving at your web site. Each an empty POST request, with the requests arriving from hundreds of different IP addresses.

Checking your web server's status through your control panel, you see a spike in CPU activity showed up along with those empty POST requests, making your report look something like this:

And your web server's disk i/o has also spiked:

It's not noticeably slowing your web site down yet, but it looks like that spike in activity is going to continue growing and it will soon affect your web site's performance. One way or another, seeing your web server spike in activity for absolutely no reason can't be good.

Is someone attacking? Is this the start of a denial of service? A prelude to some kind of cyber-extortion?

Congratulations. You've just found yourself the latest 'winner' of the Pushdo lottery.

Huh? Pushdo?

Before we start explaining how to fix things, let's take a look at the problem's cause.

You've probably heard of botnets: Networks of 'zombie' computers taken control of by viruses and joined together to do a variety of nasty things, from sending spam email to attacking web sites and taking them offline with huge bursts of traffic. The traffic you're seeing on your web site now is coming from one of those botnets. This one's referred to as Pushdo.

As explained through that link, the Pushdo botnet is an infrastructure of sorts to handle various computer crime tasks and, to hide its activity, it constantly blasts out traffic to make it difficult to spot the center of command for its work. Your web server is now receiving one of those bursts of network traffic.

For one analogy, you can think of Pushdo acting like a thief who has zombified the population of a small town to help hide his crimes. Using his zombie powers, the thief sets up thousands of people to constantly walk a city's streets and stumble into its buildings and houses. The zombies continue to stumble along, day and night, so that whenever the thief wants to rob a bank, he can just stumble off and quickly blend in with the crowd. Sure, the city streets would grind to a halt as those thousands of zombies milled about 24 hours a day, but it would be reasonably easy for the thief to perform whatever crime he'd like and then slip away in the crowds, stumbling along with the rest of the zombies.

That 'needle in a haystack' situation is similar to what you're experiencing with Pushdo. Your web server has become a house filled with zombies, with Pushdo stuffing it full of web site traffic arriving from thousands of computers to hide its activity from anyone who might try to figure out which computer was sending meaningful criminal information to the other.

Pushdo isn't attacking your web site directly, and your web server (probably) isn't infected with Pushdo itself. Your site is just an innocent bystander being used to hide activity.

So, why was your site chosen? Simple bad luck, really. Your site was one of hundreds picked at random from the world's DNS records, and now you're the target of traffic from thousands of botnet computers.

How Do I Stop This Surge in Zombie Traffic?

The bad news is, you can't stop the surge itself and it's probably going to continue hammering your web site for the next few months -- possibly the next few years. Pushdo will continue to hide its activity by using your web site as a random target and it will only stop once your site rotates off the list of zombie destinations used for camouflage.

However, while you can't stop the traffic at its source, you can configure your web server to ignore the traffic as it arrives. The tens of thousands of zombie messages will still come knocking at your web site every day, but you can set your computer to block those messages the moment they arrive on your network and keep them from reaching your web server.

To do this, you'll need to set up a firewall rule to block all traffic which fits the Pushdo message signature.

Using iptables to Block Pushdo

We're assuming here that you're running your own web server on Linux, either as a stand-alone computer or through a hosted VPS package with Linode, Slicehost or another provider. If your web site is instead being managed by a full-service hosting provider, your best action would be to contact your server support team and have them apply the appropriate traffic-blocking actions for you.

Continuing with the assumption that you're running your own web server on Linux and have full control over that computer, here's an overview of how to fix your Pushdo problem:

To block the zombie network traffic bombarding your web server, you're going to use iptables. The iptables utility is a network packet filter which you can use to regulate internet traffic. Linode provides an excellent overview of the package in their documentation, and the raw help text for the tool can also be reviewed direct through the "man iptables" command on your server computer.

Watch out! Since a mis-configuration of iptables can also block legitimate traffic (even your own login), you will want to review the documents linked above to gain a better understanding of exactly how the utility works before applying the examples given here. It's really easy to accidentally set iptables to block all network traffic arriving on your web server and, since such a mistake would block any attempts you might make to fix things, that could lead to irreparable disaster. Trust me: It's worth the extra bit of effort to look over the documentation and understand at least the basics before proceeding.

Have you read those guides over? Good. Let's put them to work.

In reviewing those documents, we can see that it's very easy to configure iptables to block individual IP addresses. That's really helpful if there's a particular nuisance source of traffic you'd like to block but, in this case, the Pushdo traffic is arriving from many thousands of different IP addresses and it would be extremely tedious to enter a blocking rule for each of them every time a new address showed up.

What we're going to do instead is block the arriving traffic based upon its contents. As you might have noticed when examining your Apache logs, all of the Pushdo traffic has the same general signature of:

[IP Address Removed] - - [21/Sep/2013:00:01:08 +0000] "POST / HTTP/1.1" 301 499 "-"
 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

For all of the messages received, there's a consistent POST command of either:

So, if we can detect that particular bit of text in a received packet, and that text is not used in any legitimate traffic accepted by your web server, we can safely set up a rule to block anyone sending a message which contains that particular blurb of text.

Ah! But your own web site might have legitimate traffic with that same signature! Before implementing this particular fix (which worked for us, as there were no legitimate POST requests on our own web server which were blank), you will want to search through your past Apache logs to confirm the text being scanned for is not used by legitimate visitors to your site. If it is, you would best select additional criteria to examine for blocking the message (for example, adding in the 'MSIE 6.0; Windows NT 5.1; SV1' bit, since blocking users of IE 6 who send along POST requests is probably not going to nab legitimate traffic).

The iptables Commands to Use For Blocking Pushdo

To block the two snippets of text identified above, you will want to apply the following iptables rules as root (preferably, using sudo):

iptables -I INPUT -d [your web server IP address] -p tcp --dport 80 -m string --string
 'POST / HTTP/1.1' --algo bm -j DROP
iptables -I INPUT -d [your web server IP address] -p tcp --dport 80 -m string --string
 'POST / HTTP/1.0' --algo bm -j DROP

Where [your web server IP address] is the IP address of the web server computer on which iptables is running.

Taking a look at the contents of that command, this is what it is doing:

iptables : running iptables
-I : inserting a rule
INPUT : for packets going to local sockets
-d [your web server IP address] : heading to the computer with this IP address
-p : now, here comes the rule:
tcp -dport 80 : for tcp traffic arriving on port 80
-m string : use the string module
-string 'POST / HTTP/1.1' : so that any traffic matching the string 'POST / HTTP/1.1'
-algo bm : which we're identifying using the Boyer-Moore (bm) string-matching algorithm
-j DROP : will immediately jump to the DROP rule the moment it's spotted.

After applying those rules to iptables, any network activity spotted which contains the text 'POST / HTTP/1.1' will be dropped immediately, closing your web site's door to those zombies.

So, What Next?

With the iptables rules applied, you're done. Pushdo traffic will still come knocking at your web server with zombie traffic, but iptables will prevent that traffic from entering your web site. You can confirm that by checking your web server's log files a few minutes after applying those rules: You should see the blank POST entries suddenly stop.

Your computer will still show higher CPU and disk I/O usage than usual, it just won't be as bad as it could have been and (with luck) won't noticeably affect the performance of your web site. For one example, the charts shown in the earlier part of this article display the result of applying those iptables rules the day Pushdo traffic first arrived on our site. In our case, there was an initial dip in CPU usage as the first round of Pushdo traffic was handled, followed by a surge as the real torrent of zombies arrived. Our iptables utility was kept pretty busy that month, but website performance was not affected.

Remember: Your iptables rules will be cleared out after every reboot of your web server, so you must be sure to apply those rules after every reboot. If you don't yet have a checklist of steps to follow whenever rebooting your web server, now would be a good time to create one.

Your iptables rules will need to remain in place until Pushdo moves on from your web site, and that will probably be quite some time. I'd recommend adding a note to your server reboot checklist reminding you to leave out your Pushdo iptables rules every three months to see if that unwanted traffic has disappeared. After rebooting your web server with those rules removed, you can then check your Apache logs after an hour to see if those telltale blank POST messages are still appearing.

You should find that the Pushdo traffic drops off in intensity after about six months but still remains in place for a bit over a year after its first appearance. However, one day it will disappear completely and, when that day comes, you can stop applying those iptables rules to your web server.

Be patient. Being a target of Pushdo is a pain, but there's nothing that can be done about it but to batten down your web server's ports and leave iptables to do its filtering job.