Recovering a compromised WordPress site – Part 2 (SQL dump and edit)

Let’s get started recovering the site. See Part 1 for the background. Note that I actually did this recovery in February 2015, and some software may have changed since then.

1. Dump the DB of the infected site in the test SQL dump format. This creates a human readable (and editable) file on my laptop.

There are all kinds of tutorials out there on dumping a SQL DB using phpMyAdmin. They are all better than I could write. This one, for example.

2. Examine and edit the DB dump file to remove any obvious damage. Is it worthwhile to continue?

For this I used Emacs. Yes, Emacs. You can use any test editor that you understand well, that has a “repeat this edit” or a general “search and replace” function. It must handle long lines, as each DB record is on a single loooong line. It helps if the editor can deal with escape characters. To make a long story short, the damage was almost immediately obvious. I was able to find the suspect lines and ^K (kill) them pretty quickly. For large values of “quickly”. There were about 1500 damaged or bogs records. Using search/replace and making a “fine pattern and kill line” worked wonders.

OK, after about 45 minutes of editing, I’ve got a clean database. All the records that I see are (probably) valid WordPress code/values or (probably) valid user records, or image pointers. It’s worthwhile to continue.

However, there’s still some cleanup, and this is a raw mySQL dump. I can’t import this into, yet. For that I need a WXR format dump, and this WordPress version was so old, that WXR isn’t even supported. I need a modern WordPress install somewhere that will accept the old MySQL dump and then allow a WXR export.

3. Install stand-alone WordPress somewhere (but how, and where?)

I’m going to use this new environment to examine the site in a sandboxed environment and get a chance at some forensics and to more completely assess the damage. This will also be the bridge between the raw mySQL dump and the WXR file that I import into later.

I expected that installing a new host and WordPress to take the most time of the entire process. In the olden days I would start with a physical host, do a full Linux install, add mySQL, Apache, etc and eventually WordPress. I don’t want to take this much time.

What’s the fastest, easiest way to get a full-blown WordPress setup? Turns out, the cloud is a pretty good place to start.





Recovering a compromised WordPress site using AWS and Bitnami – Part 1

This series is about how I recovered a compromised three+ year old WordPress site, from a medium/bad compromise with no backups. Using Bitnami’s WordPress package and AWS saved me countless hours and I got all the data back. It cost me a few hours of my time and $0.50 of AWS EC2 usage.

Part I – the plan

Recovering a compromised WordPress site can be either very easy, or very difficult depending on the version of WordPress that you have installed, and the severity of the compromise. At the easy end of the spectrum is recent WordPress and good backups. At the harder end is very old versions of WordPress, no  recent backups and a site at a shared hosting provider (no shell access).

This particular site was for a class reunion from several years ago. It was left running because it was hosting some pictures from the event, but was rarely visited. Based on PHP file modification dates, the site was compromised in November, and probably again in December, using two different WP plug-ins. This left behind lots and lots of modified PHP code. There were multiple kinds of PHP damage, which looks like it was done by two different attackers.

In other words, a mess.

Using PhpMyAdmin, I was able to determine that in general the database was touched very lightly, if at all. But the PHP code was unrecoverable. The best plan was to pick the site up and move it to, and abandon the old site (and hosting provider). I decided to pick up the site and move it to so that I wouldn’t have to deal with WP software updates and incompatible plug-ins. Also, the managed hosting company had bumped its rates from around $50/year to over $150/year in the 9 years the site has been running.

But only accepts a WordPress Export File (WXR) as the import source. While stand-alone installs of WP offer many import options, WXR is the only option for without engaging their $ervices. This broken WordPress was old enough that it didn’t even offer the WXR as an export option. And, I wasn’t sure that I wanted to import the site wholesale until I had a chance to examine it.

This was my plan.

  1. Dump the DB of the infected site in the test SQL dump format. This creates a human readable (and editable) file on my laptop.
  2. Examine and edit the DB dump file to remove any obvious damage. Is it worthwhile to continue?
  3. Install stand-alone WP somewhere. I could use this to examine the site in a sandboxed environment and get a chance at some forensics and to more completely assess the damage. I expected installing a new host and WordPress to take the most time of the entire process.
    1. Import the database ONLY into the sandbox WP install.
    2. Test the sandbox WP site to see if there are any remaining landmines.
    3. Decide if it is still worthwhile to continue.
    4. If there are repairable problems, fix them in the original DB dump file (on the laptop) and clear the sandbox DB.
    5. Lather, rinse, repeat until the site looks OK
  4. When the sandbox site looks OK – save it as a WXR file
  5. Import WXR file into
  6. Test site.
  7. Drink beer.


2 Comments – still no IPv6

Three years ago I wondered when would begin to support IPv6. The irony of hosting a blog that talks a lot about IPv6 on an IPv4-only platform is not lost on me 😦

Here we are in 2016 with Google, Akamai and pretty much all content providers are reporting more IPv6 traffic, and is still stuck in 1983.

This is not an OS issue, a load balancer issue, a PHP language issue, or even a WordPress software issue. I’ve seen WP software running full dual-stack, no problems. Five years ago. I’ve done it myself, I just don’t want to run WP myself anymore.

For whatever reason, has not decided to commit to the future. yet.


Stop disabling IPv6 in your system images

Seriously.  Just stop that.

Stop disabling IPv6 as part of your standard OS install and network configurations.

If you’re like a lot of IT shops, you’ve probably been building “golden images” of your operating systems to use as the template for OS installation. While these images are (hopefully) on a regular patch cycle after installation, the basic configurations and options can remain unchanged for years.

The upshot of this is that there are a lot of operating system images out there that were initially created around the time that the base OS was released, and which have had minimal changes since then, other than mandatory patches.

Windows 7 and Server 2008R2 were released in 2009. Centos 5 was released in 2007. Both are still in very wide use. Even if you’ve moved up to Windows Server 2012 or Centos 6 (both released in 2011), it is not uncommon for golden images of these to retain the network and other configurations such as IPv6 from prior versions.

In other words, it is quite likely that your brand new OS install is using assumptions and configurations from 2009 or even 2007, when it was still considered good practice to disable IPv6 at every opportunity. We’re beginning to see new OS features, such as DirectAccess, that require functioning IPv6, either native or tunneled.

I have yet to find any service that’s available in the MacOS X, Centos or Ubuntu systems that can’t make use of IPv6, or is negatively impacted in any way by dual-stacking the host. I have also not found any instance where taking a dual-stack-capable host onto an IPv4-only network has caused an issue, in at least 2 years.

Here’s some more info for you Windows folks, including a list of MS services that do, or don’t use IPv6.

So just quit disabling IPv6 by default, mmmkay?

1 Comment

life (and tech) is not soundbytes

Google+, Twitter and Facebook just aren’t suitable for technical topics.

They all have their uses, but seem to be just too shallow for tech, and life.

Face it, when you need an answer to a technical question or learn about something that isn’t in Wikipedia, chances are that Google will lead you to a blog post. Not a Facebook page (not indexed, and rarely technical). Not Twitter (how much can you explain in 140 characters?) And probably not Google+, either (although there is sometimes good discussion there).

Nope, you’re going to end up at someone’s blog post.  Someone who faced the same problem, did their homework, pulled together from other sources, and solved the problem.

Go to Twitter for breaking news, Facebook for your friends, and Google+ for interesting discussions.

But the next time you solve a problem, how about you contribute to the world-wide-knowledgebase via a blog post somewhere?

Leave a comment

99 days of freedom

I’m taking part in the “99 days of freedom”, by leaving Facebook for 99 days. I was never a big FB person anyway, this is just a convenient excuse.

I’m actually doing more (useful) things using Google+ and you can still find my #craftbeer posts at Untappd

If you’re really a FB-only person, here’s the countdown until I return 🙂

I’m just getting ready to lab some MS Server 2012 and Active Directory stuff to see how it works with IPv6, so that should begin to show up here in a few weeks.

Or how about we connect using the original “social media”, face to face? I’ll be at DEF CON next week.


, , ,

Leave a comment

Back soon…

Work projects have kept me super-busy the past month. I’ll be back with more IPv6 (and sysadmin) posts in a few weeks. Thanks for stopping by.

Leave a comment

IPv6 traffic doubles, again

This is encouraging news: Google reports that 3% of its traffic is now IPv6. This means that IPv6 traffic has more than doubled, each of the past four years.

Even if 3% seems small, that’s huge in terms of the number of people involved. Google sees about 12 billion searches each day, and has about 191 million unique visitors each month.

A quick back of the envelope shows that that’s about 5.7 million IPv6 visitors each month, and most of that comes from US carrier Verizon wireless. That’s not surprising, considering that 39% of VZW’s traffic is IPv6.

So while 3% isn’t much as a percentage, it is a lot of traffic, and the rate of change is very good. All this bodes well for IPv6 adoption in the coming 24 months.

Leave a comment

USA! USA! (unless you want affordable high-speed Internet service)

“In comparison with the rest of the developed world, the US has slower broadband speeds and higher broadband prices than just about anybody.”

No surprises here.  US ISPs and cable companies (among many other industries) continue to rock record profits, and instead of investing, just buy back their stock, or sit on the cash.

On the technology front, this means that instead of upgrading backbones, or delivering native IPv6, or a higher quality of service, they are deploying stopgap measures. Some examples of this are Carrier Grade NAT (CGN) instead of native IPv6. “Dumb” DVRs that are less programmable, and less usable than some home grown solutions. No investment in technical support.  Man-in-the-middle ad networks, DNS hijacking, abusive legislation, and other interference with their customers’ data.

As long as the last mile is a de-facto monopoly, that’s just what we’re stuck with.

, , , , ,

Leave a comment

chip and pin! Finally! (maybe)

Since my first trip to Europe 5 years ago, I’ve been trying to get a chip-and-pin credit/debit card. As far as I have been able to find out, other than a single credit union in DC, there is no way to get a chip-and-pin card in the US. American Express and others have chip-and-signature, but that’s not the same, even if they try to tell you that it is. For example, you can’t use chip-and-signature at unattended gas stations, vending machines or many other places in Europe.

It looks like, finally, the American card industry is willing to truly join the EMV card world, and issue chip-and-pin by 2015. It only took 10s of millions of credit cards numbers being stolen within a single month or so, to get them to move.

Almost all of our credit and debit cards were re-issued to us in January, by several credit unions and other financial institutions. That had to be expensive for all of them, and there is talk of the banks suing Target over their breach.

While this won’t end credit card fraud completely, it will definitely make it more difficult.

Just one more thing to think about as I work on my personal privacy…

, ,


%d bloggers like this: