Summary: how to diagnose and eliminate the xmlrpc.php WordPress exploit when it’s clobbering your web server.
You’ve been running a public Ubuntu 12+ and Apache/LAMP web server for a while now. Until recently everything has worked fine, but suddenly your server is struggling to display even static pages. Sometimes content takes forever to load, and occasionally pages fail to load at all, with Apache showing the 503/Service Unavailable error. You haven’t made any recent changes to your applications, server software, or databases, so what’s going on?
Diagnosing the xmlrpc.php Attack
Logging onto your server via SSH, or loading your admin pages if possible, you note a number of symptoms consistent with an HTTP flood:
- The htop utility shows heavy server load and multiple forks of mysqld and apache2 with massive memory and CPU utilization, sometimes reaching 100% per core.
- The iotop utility shows mysqld, apache2, and php-fpm hammering the disk.
- The nload eth0 command shows suddenly increased network traffic, upwards of 1 or more megabits per second.
- Other monitoring utilities you may be using, such as Smokeping, RRDTool, or remOcular, are showing large jumps in network traffic, disk I/O, and CPU utilization.
Digging deeper, the tcpflow -i eth0 -c port 80 command shows a lot of traffic like this:
185.081.157.024.33571-198.058.103.067.00080: POST /xmlrpc.php HTTP/1.1
Host: myserver.com
Content-type: text/xml
Content-length: 280
User-agent: Googlebot/2.1 (+http://www.google.com/bot.html)
Connection: close
<?xmlversion="1.0"?><methodCall><methodName>pingback.ping</methodName><params><param><value><string>http://sportsaccess.se</string></value></param><param><value><string>http://mywebsite.com/some-url</string></value></param></params></methodCall>
198.058.103.067.00080-185.081.157.024.33571: HTTP/1.1 200 OK
Date: Wed, 24 Aug 2016 19:31:26 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.24
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: no-cache, must-revalidate
X-Content-Type-Options: nosniff
Content-Length: 377
Vary: Accept-Encoding
Connection: close
Content-Type: text/xml
<?xml version="1.0"?>
<methodResponse>
<fault>
<value>
<struct>
<member>
<name>faultCode</name>
<value><int>-32601</int></value>
</member>
<member>
<name>faultString</name>
<value><string>Server error. Requested method pingback.ping not specified.</string></value>
</member>
</struct>
</value>
</fault>
</methodResponse>
Finally, tail -f /var/log/apache2/website-access.log shows massive numbers of identical entries:
185.81.157.204 - - [21/Aug/2016:07:34:38 +0000] "POST /xmlrpc.php HTTP/1.1" 200 705 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)" 185.81.157.204 - - [21/Aug/2016:07:34:38 +0000] "POST /xmlrpc.php HTTP/1.1" 200 705 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
By now you can safely conclude that you’re getting clobbered by the well-known xmlrpc.php WordPress attack. In the next section we’ll look at mitigating this problem. But first, several points:
- The fact that the attacking source claims to be “Googlebot” is meaningless, since a bot or attack script can report anything as its user-agent. Other versions of the attack may claim to be MSIE or almost anything else.
- xmlrpc.php attacks are automated and their appearance doesn’t mean someone is intentionally DDOSing you. The problem is worldwide.
- The fact that you may not be running WordPress, or may have disabled the XML-RPC service if you are, doesn’t make you immune to xmlrpc.php attacks. Your web server still has to do something with these requests, even if that “something” is only to throw a 403 error. If your server receives enough of these it will start to choke and throw up 50x/Service Unavailable errors.
Now, let’s stop the attack.
Blocking the xmlrpc.php Attack
I’m going to focus on two solutions here, both using the iptables firewall. There are many other possible approaches, like blocking access to xmlrpc.php in .htaccess or an Apache VirtualHost or Nginx server block, but I prefer to prevent known attacks from reaching the web server level at all. Other solutions such as deploying an IPS/IDS like OSSEC or Snort will be covered at another time.
Solution #1: Add an xmlrpc.php Filter to fail2ban
fail2ban is a utility that monitors your server logs and temporarily adjusts your iptables rules to block malicious activity. This is likely preferable to solution #2, below, since our fail2ban approach targets the xmlrpc.php attack specifically. This method also lets you allow in legitimate xmlrpc.php requests if you’re using the service.
To proceed, go into your fail2ban filters directory (in a default Ubuntu install, /etc/fail2ban/filters.d) and create a new file called “apache-xmlrpc.conf” or something meaningful. Add some logic to block xmlrpc.php:
# Fail2Ban configuration file # # Author: Geoff Stratton # # Version 1.0 # # This tells fail2ban to block xmlrpc.php requests and where to find the originating IP address. # Assumes your log files are formatted as above, with the IP address appearing first on each line. [Definition] failregex = ^<HOST> .*POST .*xmlrpc.php.* ignoreregex =
Next we tell fail2ban how to use our new filter. In your /etc/fail2ban/jail.local file, add this:
[apache-xmlrpc] enabled = true port = http,https filter = apache-xmlrpc logpath = /var/log/apache*/*access.log maxretry = 3
And restart fail2ban:
root@ubuntu:/# service fail2ban restart
Now any source that sends more than three xmlrpc.php requests will be blocked by your server firewall for the default bantime specified in jail.local.
Solution #2: Use ufw to Block xmlrpc.php Source IPs
This method uses the Uncomplicated Firewall, or ufw for short, a useful front end provided with Ubuntu for managing iptables. Here the IP addresses sending xmlrpc.php requests are blocked permanently until you override the block.
Since all the xmlrpc.php attacks are conveniently listed in our server logs, a text processing utility like awk is excellent for analyzing them. First, let’s count the number of xmlrpc.php requests we’ve seen lately:
root@ubuntu:/# awk '/xmlrpc.php/ {count++} END {print count}' /var/log/apache2/website-access.log
24006
Quite a few. Let’s list out the source IP addresses, sorted by number of requests:
root@ubuntu:/# awk '/xmlrpc.php/ {count[$1]++} END {for (ip in count) print ip, "-", count[ip]}' /var/log/apache2/website-access.log | sort -k3nr
185.81.157.187 - 6449
185.81.157.204 - 5121
185.81.157.24 - 3847
185.81.158.135 - 3575
185.81.158.122 - 3307
185.81.157.57 - 1962
173.208.241.106 - 47
46.119.112.23 - 2
138.201.194.73 - 1
208.109.52.29 - 1
45.119.208.100 - 1
Now that we know who the troublemakers are, blocking them is fairly easy. Since a lot of them originate from one IP range, we can block the whole range:
root@ubuntu:/# ufw deny from 185.81.0.0/24
If blocking the whole IP range seems like overkill, awk can call shell commands:
root@ubuntu:/# awk '/xmlrpc.php/ {count[$1]++} END {for (ip in count) system("ufw insert 1 deny from " ip)}' /var/log/apache2/website-access.log
This one-liner iterates over the list of XML-RPC attackers and issues the ufw deny from [ip address] command to block each of them.
root@ubuntu:/lib/ufw# ufw status Status: active To Action From -- ------ ---- Anywhere DENY 45.119.208.100 Anywhere DENY 46.119.112.23 Anywhere DENY 185.81.157.57 Anywhere DENY 185.81.157.24 Anywhere DENY 185.81.157.204 Anywhere DENY 208.109.52.29 Anywhere DENY 185.81.157.187 Anywhere DENY 138.201.194.73 Anywhere DENY 185.81.158.135 Anywhere DENY 185.81.158.122 etc.
After changing your firewall rules, remember to restart iptables:
root@ubuntu:/# service ufw restart
A couple of points here. The order of iptables rules in /lib/ufw/user.rules is important, which is why we use insert 1 in our awk one-liner above: this inserts the new deny rule at the beginning of the rules listing. If the deny rules are listed after the 80/tcp ALLOW IN Anywhere rule, iptables will let in packets from the “blocked” IPs before it applies the blocking rules.
Furthermore, the awk system() function uses .sh instead of bash, which isn’t particularly important here but might be relevant in other cases. You can specify bash with system("bash -c '''ufw insert 1 deny from "ip"'''").
Last but not least, if blocking IP addresses manually seems like a pain, you can always set this up as a cron job.
