For anyone curious, here are some details about the compromise.

I use Dreamhost shared hosting for my blog hosting. I realize this is strictly amateur-hour by geek standards, but it’s cheap and spacious and it has met my needs. And with the right caching in place it doesn’t fall over when it gets Reddited, unlike what a similarly-priced VPS might do. Also, they make it dirt-simple to deploy WordPress and keep it up-to-date. I realize WordPress is old-and-busted in this day of Octopress, but I like the plugin ecosystem and being able to schedule posts into the future.

Just before I gave my talk at CVREG, someone tweeted to me that they were being redirected to a Bing search whenever they tried to view one of my posts. As soon as I got home, I investigated. I discovered that my site had indeed been cracked, and that the compromise was extensive.

The initial exploit was via world-writable directories in two of my blogs. One directory was in a plugin (Extended Comment Options), and one was a cache directory inside a theme. I’m not sure why either directory was world-writable; I assume because of badly-written PHP.

You can all laugh at me for using WordPress now.

Another account on the same shared server wrote backdoor PHP files to those world-writable directories. This account was probably just another in a chain of compromised accounts, not that of the original attacker. In all cases the files were named “r.php”. They contained obfuscated PHP code, but from looking at the logs and from researching related exploits I surmise that they enabled arbitrary commands to be executed in response to POST requests from the web.

You can all laugh at me for using a shared host now.

These backdoors were set up on the 14th of February. On the 21st, an unknown attacker used one or more of the backdoors to rewrite every .php file in my account. Each file had an obfuscated block of PHP inserted at the beginning. The block’s purpose is to redirect visitors to malware/scam sites. Here’s a post on the attack, which apparently originated back in 2010.

Dreamhost makes it possible to create multiple users, but I had lazily run most of my sites under my main login. As a result, the attack infected this blog, wideteams.com, and other personal sites. Once I notified Dreamhost of the issue their automated scan was able to furnish me with a handy list of all the affected files, which clued me in to the extent of the attack.

You can all laugh at me for using a single user for most of my sites now.

The good news was, the inserted code was pretty obvious once I knew what to look for. In addition, PHP files were apparently the only assets targeted. I found no altered HTML or JavaScript files, and the database tables were apparently unaffected.

The first thing I did was set up site-wide redirects to a maintenance page.

A review of access logs around the time that the files were modified, as well as the Dreamhost security report, clued me in to the backdoor files. I deleted them, after tarballing them up in case I wanted to take a closer look at them later.

I also tarballed up all the compromised sites, so that I could safely make modifications without accidentally losing any important data.

I changed all of my Dreamhost user account passwords, including my Dreamhost panel password, although I had no reason to think that had been compromised. I changed every MySQL user password as well. I wiped out and re-created my .ssh- in case that had been tampered with.

While it had become clear that the attack probably had nothing to do with weak or stolen passwords, I decided to take the opportunity to finally go through and change every single one of my online accounts to use a unique, random password, managed by LastPass. I had already done this for the really important stuff like email and banking accounts, but I still had quite a few accounts that I’d used one of my “standard” passwords on. I used the LastPass Security Challenge to hunt down the accounts with duplicate and/or weak passwords.

I wrote a trivial script to clean the affected PHP files, and ran it on the cracked site directories. In theory, I could have just returned the sites to service at this point. But since I had no pristine copy to compare them with (silly me), I wasn’t comfortable doing that. And in any case, I wanted to set things up Right, or at least better, this time.

Up until this happened, I had been running this blog in a sub-directory of avdi.org. I did this because I read somewhere that it was better Google juice to have a blog as a sub-directory of your main site than as a subdomain. However, running it out of a sub-directory severely limited my options for isolating the blog hosting, or, for that matter, for moving the blog to a separate host like WPEngine. I decided that flexibility trumps Google juice. Formerly http://devblog.avdi.org and http://virtuouscode.com had simply redirected to http://avdi.org/devblog. I switched my setup to have Dreamhost fully host http://virtuouscode.com separately from http://avdi.org, with http://devblog.avdi.org as a mirror domain.

In addition, I created a new user dedicated to the virtuouscode.com domain. All PHP code in this domain runs under the dedicated user account, so that any future attacks of this nature will be isolated to a single blog.

I used the Dreamhost one-click installer to set up a bare-bones WordPress installation in the newly hosted domain. As soon as I was done doing the initial set-up, I initialized a git repo for the site and added everything—PHP, HTML, etc.—to it. Then I set up a private GitHub repo and pushed the files to it.

I wrote a simple script, suitable for automation with cron, to bundle up any changes to the site and push them to GitHub. I was careful to make the script log everything it did locally, and then email the log to me when finished.

I had initially hoped that I would be able to simply point the new blog at the existing database tables. This turned out to be nontrivial. I had recent backups of the DB, so I then thought I would import the backups. However, the tool that I’d used to back up the tables did simple SQL dumps, which the “official” WordPress import/export tool didn’t understand. And for various reasons it wasn’t as simple as just importing the old tables into the new ones with MySQL admin tools.

Rather than spend all night fiddling with MySQL imports, I opted to fire up the old (cleaned) site long enough to do a “proper” WordPress export. I then imported the tables into the new installation, which worked out quite nicely, even setting up missing users so the posts wouldn’t be “orphaned”.

Then it was a matter of reinstalling and configuring my theme, as well as a base set of plugins. Notably, I wasn’t about to go live again without WP SuperCache configured. While I was at it, I added some new optimization plugins: Use Google Libraries, which substitutes the Google hosted versions of various popular JS libs like jQuery; and Better WordPress Minify, which bundles and minifies CSS and JavaScript assets. With these plugins in place, the front page gets a respectable 81 on Google Page Speed Online.

I reinstalled various other plugins, like the Embed GitHub Gist plugin (which does exactly what you’d expect), and the Clicky plugin (for live site stats). After each plugin, I committed and pushed the changes to the Git repo. If there’s one thing this whole fiasco has taught me, it’s the importance of being able to roll back my WordPress install to a previous state.

Finally, I added a rewrite rule in the old avdi.org site to permanently redirect requests for anything formerly hosted at avdi.org/devblog to the devblog.avdi.org domain. Aside: nothing makes me feel like a complete n00b again quite like Apache rewrite rules.

What have I learned?

• It can never be said too many times: backup, backup, backup. And in the context of WordPress, backup not only the data tables, but the site files. I’ll be adding my GitHub push script to my daily crontab shortly.
• A backup is useless unless you’ve gone through a restore scenario and verified it works. I was lucky that I didn’t have to use my SQL-dump backups. I’m going to have to look into a better solution for automated WordPress data backups, one that dumps to a format which can be trivially re-imported. Perhaps something like VaultPress is a good investment.
• Separate accounts for separate sites, always.
• World-writable directories are very bad news on a shared host. Shortly I’ll be writing a cron job to scan for them, fix them, and notify me.
• WordPress has a big red target painted on it as a result of its ubiquity. I still like the WordPress ecosystem, but I’ll continue to consider moving to static blog hosting software like Octopress. For now, though, I have a publishing setup which works well for me, and I’m not quite ready to give that up, despite the recent difficulties.

I don’t expect many of you have read this far, but I figured there might be one or two people interested in the details of what happened and how I dealt with it. If nothing else, maybe someone in a similar boat will google this some day and find some useful information here.

If you have any recommendations for better WordPress security – plugins, hosts, tools, scripts, backup services, anything—please feel free to post them in the comments. I’m always interested in improving my setup. Now more than ever.