Let’s get started recovering the site. See Part 1 for the background. Note that I actually did this recovery in February 2015, and some software may have changed since then.
1. Dump the DB of the infected site in the test SQL dump format. This creates a human readable (and editable) file on my laptop.
There are all kinds of tutorials out there on dumping a SQL DB using phpMyAdmin. They are all better than I could write. This one, for example.
2. Examine and edit the DB dump file to remove any obvious damage. Is it worthwhile to continue?
For this I used Emacs. Yes, Emacs. You can use any test editor that you understand well, that has a “repeat this edit” or a general “search and replace” function. It must handle long lines, as each DB record is on a single loooong line. It helps if the editor can deal with escape characters. To make a long story short, the damage was almost immediately obvious. I was able to find the suspect lines and ^K (kill) them pretty quickly. For large values of “quickly”. There were about 1500 damaged or bogs records. Using search/replace and making a “fine pattern and kill line” worked wonders.
OK, after about 45 minutes of editing, I’ve got a clean database. All the records that I see are (probably) valid WordPress code/values or (probably) valid user records, or image pointers. It’s worthwhile to continue.
However, there’s still some cleanup, and this is a raw mySQL dump. I can’t import this into WordPress.com, yet. For that I need a WXR format dump, and this WordPress version was so old, that WXR isn’t even supported. I need a modern WordPress install somewhere that will accept the old MySQL dump and then allow a WXR export.
3. Install stand-alone WordPress somewhere (but how, and where?)
I’m going to use this new environment to examine the site in a sandboxed environment and get a chance at some forensics and to more completely assess the damage. This will also be the bridge between the raw mySQL dump and the WXR file that I import into WordPress.com later.
I expected that installing a new host and WordPress to take the most time of the entire process. In the olden days I would start with a physical host, do a full Linux install, add mySQL, Apache, etc and eventually WordPress. I don’t want to take this much time.
What’s the fastest, easiest way to get a full-blown WordPress setup? Turns out, the cloud is a pretty good place to start.
This series is about how I recovered a compromised three+ year old WordPress site, from a medium/bad compromise with no backups. Using Bitnami’s WordPress package and AWS saved me countless hours and I got all the data back. It cost me a few hours of my time and $0.50 of AWS EC2 usage.
Part I – the plan
Recovering a compromised WordPress site can be either very easy, or very difficult depending on the version of WordPress that you have installed, and the severity of the compromise. At the easy end of the spectrum is recent WordPress and good backups. At the harder end is very old versions of WordPress, no recent backups and a site at a shared hosting provider (no shell access).
This particular site was for a class reunion from several years ago. It was left running because it was hosting some pictures from the event, but was rarely visited. Based on PHP file modification dates, the site was compromised in November, and probably again in December, using two different WP plug-ins. This left behind lots and lots of modified PHP code. There were multiple kinds of PHP damage, which looks like it was done by two different attackers.
In other words, a mess.
Using PhpMyAdmin, I was able to determine that in general the database was touched very lightly, if at all. But the PHP code was unrecoverable. The best plan was to pick the site up and move it to WordPress.com, and abandon the old site (and hosting provider). I decided to pick up the site and move it to WordPress.com so that I wouldn’t have to deal with WP software updates and incompatible plug-ins. Also, the managed hosting company had bumped its rates from around $50/year to over $150/year in the 9 years the site has been running.
But WordPress.com only accepts a WordPress Export File (WXR) as the import source. While stand-alone installs of WP offer many import options, WXR is the only option for WP.com without engaging their $ervices. This broken WordPress was old enough that it didn’t even offer the WXR as an export option. And, I wasn’t sure that I wanted to import the site wholesale until I had a chance to examine it.
This was my plan.
- Dump the DB of the infected site in the test SQL dump format. This creates a human readable (and editable) file on my laptop.
- Examine and edit the DB dump file to remove any obvious damage. Is it worthwhile to continue?
- Install stand-alone WP somewhere. I could use this to examine the site in a sandboxed environment and get a chance at some forensics and to more completely assess the damage. I expected installing a new host and WordPress to take the most time of the entire process.
- Import the database ONLY into the sandbox WP install.
- Test the sandbox WP site to see if there are any remaining landmines.
- Decide if it is still worthwhile to continue.
- If there are repairable problems, fix them in the original DB dump file (on the laptop) and clear the sandbox DB.
- Lather, rinse, repeat until the site looks OK
- When the sandbox site looks OK – save it as a WXR file
- Import WXR file into WP.com
- Test site.
- Drink beer.
Three years ago I wondered when wordpress.com would begin to support IPv6. The irony of hosting a blog that talks a lot about IPv6 on an IPv4-only platform is not lost on me 😦
This is not an OS issue, a load balancer issue, a PHP language issue, or even a WordPress software issue. I’ve seen WP software running full dual-stack, no problems. Five years ago. I’ve done it myself, I just don’t want to run WP myself anymore.
For whatever reason, WordPress.com has not decided to commit to the future. yet.
Seriously. Just stop that.
Stop disabling IPv6 as part of your standard OS install and network configurations.
If you’re like a lot of IT shops, you’ve probably been building “golden images” of your operating systems to use as the template for OS installation. While these images are (hopefully) on a regular patch cycle after installation, the basic configurations and options can remain unchanged for years.
The upshot of this is that there are a lot of operating system images out there that were initially created around the time that the base OS was released, and which have had minimal changes since then, other than mandatory patches.
Windows 7 and Server 2008R2 were released in 2009. Centos 5 was released in 2007. Both are still in very wide use. Even if you’ve moved up to Windows Server 2012 or Centos 6 (both released in 2011), it is not uncommon for golden images of these to retain the network and other configurations such as IPv6 from prior versions.
In other words, it is quite likely that your brand new OS install is using assumptions and configurations from 2009 or even 2007, when it was still considered good practice to disable IPv6 at every opportunity. We’re beginning to see new OS features, such as DirectAccess, that require functioning IPv6, either native or tunneled.
I have yet to find any service that’s available in the MacOS X, Centos or Ubuntu systems that can’t make use of IPv6, or is negatively impacted in any way by dual-stacking the host. I have also not found any instance where taking a dual-stack-capable host onto an IPv4-only network has caused an issue, in at least 2 years.
So just quit disabling IPv6 by default, mmmkay?
They all have their uses, but seem to be just too shallow for tech, and life.
Face it, when you need an answer to a technical question or learn about something that isn’t in Wikipedia, chances are that Google will lead you to a blog post. Not a Facebook page (not indexed, and rarely technical). Not Twitter (how much can you explain in 140 characters?) And probably not Google+, either (although there is sometimes good discussion there).
Nope, you’re going to end up at someone’s blog post. Someone who faced the same problem, did their homework, pulled together from other sources, and solved the problem.
Go to Twitter for breaking news, Facebook for your friends, and Google+ for interesting discussions.
But the next time you solve a problem, how about you contribute to the world-wide-knowledgebase via a blog post somewhere?
I’m taking part in the “99 days of freedom”, by leaving Facebook for 99 days. I was never a big FB person anyway, this is just a convenient excuse.
If you’re really a FB-only person, here’s the countdown until I return 🙂
I’m just getting ready to lab some MS Server 2012 and Active Directory stuff to see how it works with IPv6, so that should begin to show up here in a few weeks.
Or how about we connect using the original “social media”, face to face? I’ll be at DEF CON next week.
Work projects have kept me super-busy the past month. I’ll be back with more IPv6 (and sysadmin) posts in a few weeks. Thanks for stopping by.
This is encouraging news: Google reports that 3% of its traffic is now IPv6. This means that IPv6 traffic has more than doubled, each of the past four years.
A quick back of the envelope shows that that’s about 5.7 million IPv6 visitors each month, and most of that comes from US carrier Verizon wireless. That’s not surprising, considering that 39% of VZW’s traffic is IPv6.
So while 3% isn’t much as a percentage, it is a lot of traffic, and the rate of change is very good. All this bodes well for IPv6 adoption in the coming 24 months.
No surprises here. US ISPs and cable companies (among many other industries) continue to rock record profits, and instead of investing, just buy back their stock, or sit on the cash.
On the technology front, this means that instead of upgrading backbones, or delivering native IPv6, or a higher quality of service, they are deploying stopgap measures. Some examples of this are Carrier Grade NAT (CGN) instead of native IPv6. “Dumb” DVRs that are less programmable, and less usable than some home grown solutions. No investment in technical support. Man-in-the-middle ad networks, DNS hijacking, abusive legislation, and other interference with their customers’ data.
As long as the last mile is a de-facto monopoly, that’s just what we’re stuck with.
Since my first trip to Europe 5 years ago, I’ve been trying to get a chip-and-pin credit/debit card. As far as I have been able to find out, other than a single credit union in DC, there is no way to get a chip-and-pin card in the US. American Express and others have chip-and-signature, but that’s not the same, even if they try to tell you that it is. For example, you can’t use chip-and-signature at unattended gas stations, vending machines or many other places in Europe.
It looks like, finally, the American card industry is willing to truly join the EMV card world, and issue chip-and-pin by 2015. It only took 10s of millions of credit cards numbers being stolen within a single month or so, to get them to move.
Almost all of our credit and debit cards were re-issued to us in January, by several credit unions and other financial institutions. That had to be expensive for all of them, and there is talk of the banks suing Target over their breach.
While this won’t end credit card fraud completely, it will definitely make it more difficult.
Just one more thing to think about as I work on my personal privacy…