Back in business

If you tried to read one of my posts in the last few days and got a maintenance page instead, sorry about that. While I was down in Virginia visiting CVREG, my hosting was cracked, and I’ve been doing damage control and recovery every since. I haven’t finished rebuilding the site, but all the old posts should be available once again.

Incident report

For anyone curious, here are some details about the compromise.

I use Dreamhost shared hosting for my blog hosting. I realize this is strictly amateur-hour by geek standards, but it’s cheap and spacious and it has met my needs. And with the right caching in place it doesn’t fall over when it gets Reddited, unlike what a similarly-priced VPS might do. Also, they make it dirt-simple to deploy WordPress and keep it up-to-date. I realize WordPress is old-and-busted in this day of Octopress, but I like the plugin ecosystem and being able to schedule posts into the future.

Just before I gave my talk at CVREG, someone tweeted to me that they were being redirected to a Bing search whenever they tried to view one of my posts. As soon as I got home, I investigated. I discovered that my site had indeed been cracked, and that the compromise was extensive.

The initial exploit was via world-writable directories in two of my blogs. One directory was in a plugin (Extended Comment Options), and one was a cache directory inside a theme. I’m not sure why either directory was world-writable; I assume because of badly-written PHP.

You can all laugh at me for using WordPress now.

Another account on the same shared server wrote backdoor PHP files to those world-writable directories. This account was probably just another in a chain of compromised accounts, not that of the original attacker. In all cases the files were named “r.php”. They contained obfuscated PHP code, but from looking at the logs and from researching related exploits I surmise that they enabled arbitrary commands to be executed in response to POST requests from the web.

You can all laugh at me for using a shared host now.

These backdoors were set up on the 14th of February. On the 21st, an unknown attacker used one or more of the backdoors to rewrite every .php file in my account. Each file had an obfuscated block of PHP inserted at the beginning. The block’s purpose is to redirect visitors to malware/scam sites. Here’s a post on the attack, which apparently originated back in 2010.

Dreamhost makes it possible to create multiple users, but I had lazily run most of my sites under my main login. As a result, the attack infected this blog, wideteams.com, and other personal sites. Once I notified Dreamhost of the issue their automated scan was able to furnish me with a handy list of all the affected files, which clued me in to the extent of the attack.

You can all laugh at me for using a single user for most of my sites now.

The good news was, the inserted code was pretty obvious once I knew what to look for. In addition, PHP files were apparently the only assets targeted. I found no altered HTML or JavaScript files, and the database tables were apparently unaffected.

Cleanup

The first thing I did was set up site-wide redirects to a maintenance page.

A review of access logs around the time that the files were modified, as well as the Dreamhost security report, clued me in to the backdoor files. I deleted them, after tarballing them up in case I wanted to take a closer look at them later.

I also tarballed up all the compromised sites, so that I could safely make modifications without accidentally losing any important data.

I changed all of my Dreamhost user account passwords, including my Dreamhost panel password, although I had no reason to think that had been compromised. I changed every MySQL user password as well. I wiped out and re-created my .ssh- in case that had been tampered with.

While it had become clear that the attack probably had nothing to do with weak or stolen passwords, I decided to take the opportunity to finally go through and change every single one of my online accounts to use a unique, random password, managed by LastPass. I had already done this for the really important stuff like email and banking accounts, but I still had quite a few accounts that I’d used one of my “standard” passwords on. I used the LastPass Security Challenge to hunt down the accounts with duplicate and/or weak passwords.

I wrote a trivial script to clean the affected PHP files, and ran it on the cracked site directories. In theory, I could have just returned the sites to service at this point. But since I had no pristine copy to compare them with (silly me), I wasn’t comfortable doing that. And in any case, I wanted to set things up Right, or at least better, this time.

Up until this happened, I had been running this blog in a sub-directory of avdi.org. I did this because I read somewhere that it was better Google juice to have a blog as a sub-directory of your main site than as a subdomain. However, running it out of a sub-directory severely limited my options for isolating the blog hosting, or, for that matter, for moving the blog to a separate host like WPEngine. I decided that flexibility trumps Google juice. Formerly http://devblog.avdi.org and http://virtuouscode.com had simply redirected to http://avdi.org/devblog. I switched my setup to have Dreamhost fully host http://virtuouscode.com separately from http://avdi.org, with http://devblog.avdi.org as a mirror domain.

In addition, I created a new user dedicated to the virtuouscode.com domain. All PHP code in this domain runs under the dedicated user account, so that any future attacks of this nature will be isolated to a single blog.

I used the Dreamhost one-click installer to set up a bare-bones WordPress installation in the newly hosted domain. As soon as I was done doing the initial set-up, I initialized a git repo for the site and added everything—PHP, HTML, etc.—to it. Then I set up a private GitHub repo and pushed the files to it.

I wrote a simple script, suitable for automation with cron, to bundle up any changes to the site and push them to GitHub. I was careful to make the script log everything it did locally, and then email the log to me when finished.

I had initially hoped that I would be able to simply point the new blog at the existing database tables. This turned out to be nontrivial. I had recent backups of the DB, so I then thought I would import the backups. However, the tool that I’d used to back up the tables did simple SQL dumps, which the “official” WordPress import/export tool didn’t understand. And for various reasons it wasn’t as simple as just importing the old tables into the new ones with MySQL admin tools.

Rather than spend all night fiddling with MySQL imports, I opted to fire up the old (cleaned) site long enough to do a “proper” WordPress export. I then imported the tables into the new installation, which worked out quite nicely, even setting up missing users so the posts wouldn’t be “orphaned”.

Then it was a matter of reinstalling and configuring my theme, as well as a base set of plugins. Notably, I wasn’t about to go live again without WP SuperCache configured. While I was at it, I added some new optimization plugins: Use Google Libraries, which substitutes the Google hosted versions of various popular JS libs like jQuery; and Better WordPress Minify, which bundles and minifies CSS and JavaScript assets. With these plugins in place, the front page gets a respectable 81 on Google Page Speed Online.

I reinstalled various other plugins, like the Embed GitHub Gist plugin (which does exactly what you’d expect), and the Clicky plugin (for live site stats). After each plugin, I committed and pushed the changes to the Git repo. If there’s one thing this whole fiasco has taught me, it’s the importance of being able to roll back my WordPress install to a previous state.

Finally, I added a rewrite rule in the old avdi.org site to permanently redirect requests for anything formerly hosted at avdi.org/devblog to the devblog.avdi.org domain. Aside: nothing makes me feel like a complete n00b again quite like Apache rewrite rules.

Lessons

What have I learned?

  • It can never be said too many times: backup, backup, backup. And in the context of WordPress, backup not only the data tables, but the site files. I’ll be adding my GitHub push script to my daily crontab shortly.
  • A backup is useless unless you’ve gone through a restore scenario and verified it works. I was lucky that I didn’t have to use my SQL-dump backups. I’m going to have to look into a better solution for automated WordPress data backups, one that dumps to a format which can be trivially re-imported. Perhaps something like VaultPress is a good investment.
  • Separate accounts for separate sites, always.
  • World-writable directories are very bad news on a shared host. Shortly I’ll be writing a cron job to scan for them, fix them, and notify me.
  • WordPress has a big red target painted on it as a result of its ubiquity. I still like the WordPress ecosystem, but I’ll continue to consider moving to static blog hosting software like Octopress. For now, though, I have a publishing setup which works well for me, and I’m not quite ready to give that up, despite the recent difficulties.

Conclusion

I don’t expect many of you have read this far, but I figured there might be one or two people interested in the details of what happened and how I dealt with it. If nothing else, maybe someone in a similar boat will google this some day and find some useful information here.

If you have any recommendations for better WordPress security – plugins, hosts, tools, scripts, backup services, anything—please feel free to post them in the comments. I’m always interested in improving my setup. Now more than ever.

This entry was posted in Announcements and tagged , , . Bookmark the permalink.
  • http://twitter.com/nlsmith Nathan L Smith

    Thanks for the report. Don’t feel bad for using WP; It’s not the new sexy thing on the block, but it’s a good blogging platform. http://wordpress.org/extend/plugins/wp-security-scan/ is a decent plugin that can check for open permissions. And I’d recommend http://wordpress.org/extend/plugins/w3-total-cache/ over super-cache for caching. 

    As someone who hosts many wordpress sites, I’ll attest to the difficulty of keeping secure, backed-up, and up to date across multiple sites. If you’re proficient with git and those kind of tools, a static site with octopress or something like it might be a better fit for you. I’d also recommend http://www.cloudflare.com/ for an extra layer of security and cache.

    The web is a dangerous place. You’ve been a great resource for ruby knowledge, so don’t be discouraged or embarrassed when these things happen.

    • http://website-in-a-weekend.net/ Dave Doolin

      Good advice.

      I have found a good middle ground by using the default WordPress installation, but keeping my theme (and custom plugin) files under git control. I use a bare repo on the host and a post-receive hook to checkout the latest push automatically. It’s been working well.

  • http://rakeroutes.com/ Stephen Ball

    Hey Avdi, thanks for the write up. I too use Dreamhost shared hosting and have one account that holds just about everything. Hmm, I should look into improving that.

    Glad the blogs are all safe and sound and back online.

  • http://trevorbramble.com/ Trevor Bramble

    Great write-up, Avdi.

    Nathan’s right. You are way too hard on yourself about using what really are the appropriate choices. The only serious misstep was lax permissions. Otherwise, it’s perfectly reasonable to have all of the sites that only one person maintains structured under one user, and the case of this attack would not have provided any additional protection, only a slightly lower chance of complete corruption.

    The WordPress extension ecosystem is absolutely worth staying with WP, so long as those extensions are being used and aren’t easily replicated on a less problematic platform.

    Likewise, DreamHost is still a reasonable choice. They occupy a certain horizontal slice of the hosting market where their only considerable competition is either comparatively expensive or comparatively incompetent, so they’re often the best choice for what they offer.

    Your approach diverged from my own when cleaning up ninebullets.net last year (while I host my stuff on VPSs, I maintain that one WP site, on DH, so like I assume you have, I’ve run the numbers and figured it’s best to keep it there) but only in small ways. Diagnosis and reconstruction were similar. I did start afresh with clean WP, theme, and plugins, and then manually verified the non-WP files that were mixed in by the site owner (mostly images).

    The only truly exasperating part of it all was watching Google ever so slowly drop the malware-stained cached pages, even while prodding it with their webmaster tools. Oh and by the way, I wouldn’t worry about the Google juice so much with regard to sub-domain and sub-directory. If I recall correctly that doesn’t make as much of a difference anymore. And in any case, SEO is generally not worth the attention. Just write good content and let the web do its thing. =^)

    I don’t have any suggestions to offer above what you’re already covering, but if I think of any I’ll bring them up.

    Cheers,Trevor

    • http://website-in-a-weekend.net/ Dave Doolin

      I second this, decouple the 1-click.

  • Than

    Actually, it shouldn’t be to hard to restore a site from from the SQL dumps. I find myself moving and backing up sites pretty regularly with nothing more sophisticated than mysqldump. The structure that WP uses isn’t all that complex, as long as you know the couple of values in the options table you typically need to fiddle with whenever you move stuff around.

    Good writeup, and glad you got the blog back up.

  • http://twitter.com/nickcoynephoto Nick Coyne

    Thanks Avdi. One of my wordpress blogs was hacked on Dreamhost too, on the 21st. It sounds similar to your scenario, although you’ve highlighted a few things I didn’t check, eg the world-writeable directories – I’ll get on that now…

  • http://www.celticwolf.com/ Charles Calvert

    Like Than, I’ve had no trouble using mysqldump to move WordPress databases from one host to another.  As he said, the trick is knowing which values in wp-config.php you need to modify.

  • http://twitter.com/marksim Mark Sim

    Would you publish via a gist the php script you used to clean up your files?

    • http://avdi.org Avdi Grimm

      Here ya go: https://gist.github.com/1926842

      I used it in conjunction with find:

      find . -name “*.php” -type f -print0 | xargs -0 ~/bin/cleanup