A windsurfing, CSS-grudging, IE-hating, web-developing, gigantic-machine-puzzling blog

Bad Web Crawlers

Bad Crawler Bots:

Bad Web CrawlersFound this bot accessing the site via lots of different 202.46.* IPs. Reverse DNS points to

The range of IPs for to is associated with ShenZhen Sunrise Technology Co., Ltd.

This is how to ban via .htaccess RewriteRule:

## ban
RewriteCond %{REMOTE_ADDR} ^202\.46\.(3[2-9]|[4-5][0-9]|6[0-3])\.
RewriteRule !^robots\.txt - [F]

Optionally, you can add this RewriteCond for the useragent they happen to be using at the moment:

RewriteCond %{HTTP_USER_AGENT} ^Mozilla/5\.0\ \(Windows\ NT\ 5\.1;\ rv:6\.0\.2\)\ Gecko/20100101\ Firefox/6\.0\.2

However, the IP ban is specific to the range owned by the company, so personally I wouldn’t bother using that useragent criteria. They could just change it at any time.

I did see they made several requests to robots.txt, but without a proper user agent identifying this bot as a crawler, your guess is as good as mine how to ban it in robots.txt, perhaps:

Disallow: /
Plesk Apache 404 File Does Not Exist errors

Plesk Apache 404 error: File does not exist: /var/www/vhosts/default/htdocs/

You’ve no doubt discovered massive amounts of 404 errors in your main Apache error log that go something like:

File does not exist: /var/www/vhosts/default/htdocs/....

The requests may appear to be legitimate requests for page on the the primary virtualhost, but are returning 404 errors. Or, they may be crap requests to /var/www/vhosts/default/htdocs/phpMyAdmin etc made by script kiddies looking for vulnerabilities. Sound familiar?

Chances are you have SSL disabled for the domain in Plesk, & these requests to vhosts/default/htdocs/ are from HTTPS requests.

Plesk handles this use case in the most asinine way possible.

Since you have SSL disabled for your virtualhost, Plesk doesn’t route HTTPS requests to any virtualhost. Instead, it’s using the default host settings in /etc/httpd/conf/plesk.conf.d which can be something like:

<VirtualHost your_ip_here:7081>
    ServerName "default-your_underscored_ip_here"
    DocumentRoot "/var/www/vhosts/default/htdocs"

Little-known (to me) Plesk fact: For SSL requests, Apache listens to port 7081 when it’s running behind nginx, per /etc/httpd/conf.d/ssl.conf

How do you know this is going on? Enable servername & port logging in access_log so you can tell exactly what these requests are coming in as. To set that up, in /etc/httpd/conf/httpd.conf look for where your main access_log is defined, like:

 CustomLog logs/access_log combined

Then look for a LogFormat line that defines the log format nickname “combined”:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Add %v %V %p in there — right after %t is a good spot. Doing this adds the servername in two flavors, & the port of the original request. The servername helps you to determine which section of your Apache config is getting used, if you aren’t sure. The port shows the original — not mapped — port of the request. HTTPS starts out as a port 443 request so you’ll see that in the access log, not port 7081.

Restart Apache, either through Plesk, or apachectl restart. Then go tail -f access_log  to watch the log with that additional data.

How do you fix how Plesk handles these SSL requests? In Plesk…

  • In Hosting Settings for your domain, check the box to enable (…yep) SSL.
  • In the Apache & nginx settings for your domain, under “Additional directives for HTTPS “, add this RewriteRule to redirect HTTPS requests to HTTP:
RewriteEngine On
RewriteRule .* http://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

That’s the best way I know how to fix this, anyway.

Any other suggestions? Normally you could set up a “black hole” entry, but I’m not sure how to overwrite the default Apache server settings, since server.conf is auto-generated by Plesk.

I’m off to bang some rocks together.

MediaTemple TrueSpeed CDN Control Panel

MediaTemple TrueSpeed CDN Control Panel Is Not What You Think

This is MediaTemple’s sales page for their TrueSpeed™ CDN, which is a rebranded product operated by

MediaTemple TrueSpeed CDN Sales Page

Sounds great & what a deal, right? I thought so. Note the “integrated control panel”. Here’s what you actually get for the control panel:

Only having on/off & purge-all controls is not the way to manage a CDN, in the same way that using only the ignition switch & yelling “get the fuck out!” to your passengers is not a good way to drive a car.

Hopefully this guy from the TrueSpeed CDN sales page is designing a new control panel.

This guy on MediaTemple’s CDN sales page is no doubt hard at work designing a real control panel.

With MediaTemple’s TrueSpeed CDN there are no traffic stats or reporting, & no asset purge controls except for all-or-nothing.

I had been using Edgecast ProCDN (through MediaTemple) which has a real control panel with nice purge controls & very detailed traffic stats/reporting.

If you sign up for the same CDN through & not via MediaTemple, you get a much more capable control panel with all the features you’d expect for managing a CDN, but you’ll probably pay more than $30/month.

I think MediaTemple’s TrueSpeed CDN is still a good deal & no doubt they have plenty of customers who don’t need CDN stats or any real controls.

But MediaTemple should be far more upfront about what customers actually get for the “integrated control panel”. Maybe they should call it what it is: “integrated on/off/purge buttons”.

In lieu of MediaTemple being upfront about it, now you know!

MediaTemple GridServer to DV Migration

Migrating MediaTemple’s GridServer (GS) to Dedicated Virtual (DV) VPS

I recently moved lots of websites off MediaTemple’s GridServer (gs) platform to their Dedicated Virtual (dv) platform. I’ve kind of abused Grid Server for the past 12 years, but finally the overage fees caught up.

This post is a quickly written stub but I wanted to write stuff down before I blissfully forget the migration headaches.

I went with the Plesk 12.5/CentOS 6 hosting option. The standalone DV server was tempting but I don’t quite know enough about Linux admin to go that route.

Here were some of the bigger migration issues going from MediaTemple’s GridServer (gs) to Dedicated Virtual/VPS (dv) service.

Plesk doesn’t come with root enabled.

Chances are you’ll need to enable root, which is done through AccountCenter. I tried not to & there’s just too many fixes/workarounds that you can’t do without root access.

The MT sales rep set up the (dv) account with a made-up domain.

The issue here is they/you need to put down something temporary while you get started with your site migration. Whether or not a fake domain is the best option, I don’t know, but it caused problems. The fake domain isn’t just for a name in AccountCenter, which is what tech support first told me.

The fake domain gets set as Reverse DNS for your new (dv) service IP address, which can cause your IP to get blacklisted for email. Minor detail, yep.

THE FIX: As soon as you get your primary site migrated over, fix the Reverse DNS (AccoutCenter, DNS section). Then change the primary AccountCenter domain for your (dv) account to your real domain – that’s hidden in AccountCenter under Server Guide.

Plesk creates websites with the web directory set to httpdocs/

…Whereas of course GridServer uses html/ for the top-level web directory. Aside from that I prefer the shorter “html”, I really don’t like changing things like this in Git which we use for development. Luckily it’s easy to change how Plesk works here.

THE FIX: In Plesk, click on your domain & click Hosting Settings. Change Document Root to html/. Plesk will create html/ but leaves httpdocs/ so you’ll have to delete httpdocs/ manually.

Note: for the rest of this post I’ll continue to reference “httpdocs/” for consistency.

Plesk puts cgi-bin/ as a subdirectory of httpdocs/.

In other words, Plesk uses httpdocs/cgi-bin/. GridServer had cgi-bin/ at the same level as html/. So basically if you’ve used Git for years like we have, you can either move the folder in Git & hope the history stays or change how Plesk works. Moving the folder & keeping the Git history is possible, but messing with Git gives me the creeps.

THE FIX: Create cgi-bin/ where you want it. You may need to set permissions using chmod 755 cgi-bin/. It’s probably good to follow Plesk convention where top-level web directories are assigned to the psacserv group so chgrp psaserv cgi-bin/ too. Then in Plesk under the domain, click Apache & nginx Settings. Scroll to the “Additional Apache Directives” section and add:

ScriptAlias "/cgi-bin/" "/var/www/vhosts/"

Then in Plesk under the domain, click Hosting Settings & disable CGI, Perl etc. Or keep it, I don’t think it matters at this point. I also deleted the now-defunct httpdocs/cgi-bin/ directory.

Plesk creates websites directories with a bunch of default & testing files.

THE FIX: SSH to your account, change to the website directory & rm -rf html/*. I’m assuming you know enough about Linux to realize this deletes everything in html/ so hopefully you haven’t uploaded the website files you want to keep, yet.

Plesk sets up subdomains as subdirectories of the parent website.

In other words, Plesk creates This sucks. Luckily, easy fix. When adding the subdomain site in Plesk, pick the primary domain option rather than the subdomain option. Ignore the “www” prefix. The subdomain site will work fine & you’ll just have an extra domain alias for in nginx & Apache that won’t be used.

Plesk forces you to create a different user for each web site.

The files all get assigned to that user, & group psacln. The httpdocs/ directory is also assigned to the same user, & group psaserv. So, users can’t browse each others’ web folders. Through Plesk you also can’t have an FTP account (i.e. for BeanStalk) that has access to multiple web sites, which is what I needed since we use one Git repository to manage ~30 very similar websites with shared resources.

THE FIX: I think what I did was su root & then grant the psacserv group to ssh/ftp users that I wanted to have access to the full range of web directories (but not root privileges): usermod -a -G psaserv username That way the group privileges work nicely. Just make sure you set the correct group privileges on folders/files.

I set up one user for SSH & FTP with access to all website directories. If you go that one-user route rather than the Plesk-created users for each website, reset directory & file ownership to your super-web-user with: chown -R username:psacln * That chown operates recursively starting from the current directory, so run that only from within html/ and cgi-bin/ because otherwise it will try to reset ownership on your system files, log files & other non-public stuff that should probably stay assigned to root.

Similarly depending on how you upload/migrate your web files, you may need to set correct permissions on files & directories within html/:

find . -type f -exec chmod 644 {} +
find . -type d -exec chmod 755 {} +

(Execute these commands only from within the directories you want affected.)

For cgi-bin/ you’ll probably want to chmod 755 script files rather than 644, so your scripts (Perl in the example below) have world-read/execute permissions:

find . -type f -name '*.pl' -exec chmod 755 {} +

Final step is have each web property execute CGI scripts as your one user. For each domain’s Apache & nginx settings, in “Additional directives for HTTP “, add:

SuexecUserGroup "username" "psacln"

Plesk sets each FTP user’s home directory to within a web directory.

Again we use Git/BeanStalk with several repos that manage groups of similar websites, so I needed an FTP login for BeanStalk to have access to the vhosts/ directory where all the website directories are located.

THE FIX: You can change the home directory for your FTP user in bash through the normal way, & Plesk doesn’t care:

usermod -d /var/www/vhosts/ username

Plesk runs all cron jobs as root.

Any files that your cron job creates get root user permissions & are not available to the web server users. Plesk shows the cron user as root, but it’s not anything you can change. Yes, this is lame.

THE FIX: Have each cron run a shell script that uses su, sudo or runuser to switch to the web-level user first. For example, have the cron run a shell script with:

/sbin/runuser username -s /var/www/vhosts/

Or you could have each cron job command (each entry in Plesk) start with one of the user-switching methods. But since cron commands become the email subject for status notifications, the subject line would start with “runuser” etc all the time.

Plesk doesn’t serve web font files correctly by default.

We serve css & web fonts from a different domain than the main website, so we need to have an access control header to allow that. The standard code is:

<FilesMatch "\.(ttf|otf|eot|woff)$">
<IfModule mod_headers.c>
    Header set Access-Control-Allow-Origin "*"

This worked great on GridServer in .htdocs, but didn’t have any effect after we migrated to Plesk. As far as I can tell, mod_headers is enabled by default so that’s not the problem. Eventually I noticed the response header for the web font files was nginx & not Apache. I got it working by going into Apache & nginx Settings for the domain & disabling “Smart static files processing”. I think what’s happening is without the second box checked that defines specific extensions, nginx serves font files by default so the request never makes it to Apache.

Even if it works for you to have nginx serve your font files, I found nginx serves them as text/plain so under the MIME types section at the top of the same screen, it might help to add:

font/ttf .ttf
font/opentype .otf
application/font-woff .woff
application/ .eot

MediaTemple warns that you’ll lose all your (gs) IMAP email as soon as you click Point At Another Server in AccountCenter.

FIX: True, but you can still switch your site over to (dv) without losing old email. Instead of clicking Point At Another Server, go about adding/migrating your site & ignore Plesk’s DNS warning. Then when you’re ready to switch web traffic over, in AccountCenter, edit DNS & change the A records from your (gs) IP to your (dv) IP.

Make sure you have email set up in Plesk before switching A records, because although you won’t lose your old email, new email will start going to your (dv) accounts.

This way although your domain gets switched over to (dv), you won’t lose your old (gs) email because you can still access it through your gridserver domain ( or If you’re not sure what these are, in AccountCenter, click on Server Guide for your grid server & look at the Email section.

Incidentally migrating IMAP email is really easy with Thunderbird. Add your new IMAP account, select all the folders in your old account, drag over to new & you’re done. Then do the Point At Another Server thing.

Plesk doesn’t come with CPAN or (m)any Perl modules.

I’m a dinosaur, I guess.

Drobo5N fan replacement

Replace Noisy Drobo5N Fan

Drobo5N fan replacementDrobo makes a nice backup system, but the cooling fans are crap. The fan in my Drobo 5N was no exception & failed after 3 years, most of which it spent on standby with the drives spun down. As far as I know, the Drobo cooling fan runs constantly even when the drives are spun down.

For the cost of a $15 fan & maybe an hour of your time, you can replace the fan in your $500 Drobo5N backup system.

You’ll need a replacement 120x25mm 12VDC 1.9W 2-wire fan, soldering iron & solder (or very small wire nuts), phillips screwdriver, 1/16″ shrink tubing (and lighter) or electrical tape, & a paperclip.

  1. Shut down & unplug the Drobo. Remove the hard drives & remember the drive order in the bays. It might not matter, but why tempt fate?
  2. Remove the magnetic faceplace & pull out the rubber gasket from the groove behind the faceplate.
  3. Unscrew the 4 rubber feet on the bottom of the unit. Also remove the small hatch on the bottom that covers the mSATA bay.
  4. Slide the metal case off the chassis. I don’t think it matters which end you slide it off. The case is split in half on the bottom. You’ll need to lift up one side just a bit to get the case to slide past a few items in the mSATA bay.
  5. Push in the plastic tabs to remove the fan backplate on one end & the drive bays on the other end. I think there’s 5 tabs for each. This part was easy.
  6. Unscrew 4 screws total, on the sides at the fan end: 1 upper & 1 lower screw. Leave all the other screws in place. Trust me on this.
  7. Slide the top of the chassis back & up to separate it from the bottom/sides. This can be tricky – use a flat screwdriver to pry the lower slot closest to the back (see photo) if it’s stuck. Unplug the white plastic fan & power switch connectors when you can get to them – they just pull, no clips.
  8. Drobo5N fan anchor

    Don’t pry off from the head end!

    Finally, easy access to the fan! Not so fast, Batman. The fan is held in place by 4 plastic anchors that work like drywall anchors – there’s a center pin which spreads out the tip of the anchor when it’s pushed in all the way. Best method for removal is push the center pin from the fan side with a paperclip or small nail until the anchor pops out. Don’t try prying the anchors from the cap end or you’ll likely end up breaking them off.

  9. Cut the fan wire close to the old fan so you have decently long leads on the connector plug end.
  10. Solder (or use wire nuts) the connector onto the new fan wires. Use shrink tubing! It’s awesome. Remember to slide the tubing on, before you solder.

Now you’re ready to put everything back together. Really this is just an excuse to start the instruction numbering over at #1. Also now that you’re in this deep, let’s make putting your Drobo back together look like 4 steps:

  1. Reattach the fan to the chassis with the plastic anchors you didn’t break.
  2. Slide the chassis back together, reconnect the fan & power switch connectors, & screw the 4 screws back in. PRO TIP: make sure the fan wires aren’t in the way of the fan blades.
  3. Snap the fan backplate & the drive bays back in. It can be a little tricky to get the drive bays seated all the way. Make sure you line up the many clear plastic nubs along the bottom edge with the all holes (these are the blue lights that indicate capacity).
  4. Slide the chassis back into the case. Reinstall the mSATA hatch cover, rubber feet, rubber gasket, drives, & magnetic faceplate. Plug in & turn on.

Nice work. You saved $500 on a new Drobo.

Hope the new fan lasts longer than the old one.

MediaTemple Cron Job Restrictions

Workaround For MediaTemple’s Lame Gridserver Cron Job Limitation

MediaTemple limits their GridServer (GS) customers to only 5 cron jobs.

Some restrictions make sense … some don’t!

This makes absolutely no sense.

MediaTemple allows cron jobs to run as often as every 5 minutes … I needed more than 5 weekly processes, which put no strain on the server compared to someone who sets up 5 jobs to repeat every 5 minutes.

Limits on cron jobs by run frequency would be far too logical.

As a workaround, combine all your weekly crons into one daily job, your daily crons into an hourly job, etc. Then create a shell script that uses date logic tests to branch out to different jobs, based on the day of the week or hour of the day.

For example, here’s a daily cron script that branches out into weekly jobs:

if [ $(date +%u) -eq 1 ]; then ./
elif [ $(date +%u) -eq 2 ]; then ./
elif [ $(date +%u) -eq 3 ]; then ./
elif [ $(date +%u) -eq 7 ]; then ./

If you don’t have a job for every day of the week, just leave out the logic test for that day & the cron script will exit without running anything further.

Similarly with a cron script set to run hourly, you can test the hour to run up to 24 different daily jobs (one for each hour):

if [ $(date +%k) -eq 0 ]; then ./
elif [ $(date +%k) -eq 1 ]; then ./
elif [ $(date +%k) -eq 2 ]; then ./
elif [ $(date +%k) -eq 23 ]; then ./

Taking this to the extreme, in theory you could use an every-5-minutes cron job to test $(date +%M) and run up to 12 different jobs per hour. And it goes without saying you can combine these day/hour/minute logic tests to create 2,016 possible combinations, or throw in some $(date +%d) day-of-the-month tests for 62,496 possible cron jobs. Take that, MediaTemple-cron-limitation-type people!

Obviously the script names in the examples above (,, etc) can be changed to anything you want.

I’m not on MediaTemple’s GridServer platform anymore, but hope this helps someone. I switched to a their DV/VPS platform & promptly discovered they assigned a blacklisted IP to my account. So far they are refusing to fix it. Good thing MediaTemple doesn’t sell cars.

UPDATE: To their credit, MediaTemple finally saw the logic in providing new accounts with non-blacklisted IPs, so that’s good.

Old fridge

Beer & Seltzer Kegerator Fridge Conversion Guide

Preface: if you already have a kegerator & want to add seltzer, all that’s stopping you is $60 in parts. See the section at the bottom of this post.

Stupid kegerator

Dumb idea. Who wants 15 gallons of just one beer?

Most beer drinkers want a kegerator, which has obvious benefits: cheep beer, always cold, rarely runs out, no empties, access to beers you can only get by keg… bonus freezer space …

You buy beer in half-barrel kegs because per drink, it’s half the cost of bottled or canned beer. But there’s no sense having a huge half-barrel keg taking up your entire your kegerator, especially as it gets toward empty. Also there’s having to drink 15 gallons of just one type of beer & getting sick of it.

What you need is 4 soda or corny (originally, “Cornelius”) kegs. And a few other things. We’ll get to that. What you may not know is:

  • For some variety, trade corny kegs with a neighbor/co-worker/relative, so you don’t have to drink 15 gallons of one beer yourself, thereby ruining your favorite beer.
  • You can easily use the same CO2 system to make an endless supply of seltzer or soda. This blows away a Sodastream in all respects, especially price & carbonation.

The idea here is get a half-barrel keg, but transfer it off into three 5-gallon corny kegs. Use a 4th corny keg for seltzer. Corny kegs take up far less room in your kegerator fridge & are easier to manage.

The fridge

Old fridge

Use an energy-star rated fridge, from 2001 & newer if you can.

Chances are you already have an old fridge, so you’re not choosing what fridge to convert into your kegerator. If you do have a fridge choice, get a model with the freezer on the bottom, & one that maximizes corny keg space (see keg layout below). You can store beer kegs for a year or more, so some extra space for reserve kegs in your fridge is a good thing.

Some people don’t like the idea of an extra fridge because they think there’s a high electricity cost. That’s only true for refrigerators over 25 years old. Any fridge made this century is an energy-efficient model that will add $6/month. Older models from 15-25 years ago might burn through $10/month. The Energy Star website has a fridge energy cost calculator.

Keg layout

Best choice: a bottom-freezer fridge w/ room for 6+ soda kegs.

Best choice: a bottom-freezer fridge w/ room for 6+ corny kegs.

Figure out how many corny kegs will fit in your fridge. Take out all unnecessary bins, shelves, etc. Keep the fridge door bins if you can, since it’s nice to have storage for some bottled beer too. The door needs to close, so adjust your depth measurement for that. The plastic floor of your fridge is not meant for 50-lb point loads as you roll kegs around, so make a plywood floor insert to distribute the keg weight.

There are two types of corny keg connections: ball- & pin-lock. They’re equally good but the kegs are slightly different dimensions:
Keg couplers

  • Ball-lock: 8.5″ diameter, 25.75″ tall w/ couplers
  • Pin-lock: 9″ diameter, 24″ tall w/ couplers

That minor difference can matter if space in your keg fridge is tight. Pin-lock kegs by themselves are shorter, but their couplers are taller. You can get a pin-lock keg down to 23.5″ tall (again, with couplers) using a ball-lock conversion kit.

Tap layout

Figure out how many taps you want, & the best place to locate them: the side, or the door. Few things to consider here…

Locating the taps on the sides:

  • Coolant tubes. Find out where the coolant tubes are by running the fridge with the door open for a bit. Condensation will form on the interior walls showing you where the tubes are. On most fridges, the tubes are only in the back of the fridge & the sides are just foam, meaning you can put your taps pretty much anywhere. That said, don’t mess up. Hitting a coolant tube will instantly ruin your fridge.
  • The taps will extend into the fridge compartment by an inch or two, so make sure there’s still room for your keg layout. If space is tight, the best place could be for the taps to go on the fridge door in between the bins.
  • Determine where you’ll be drilling through on the inside & then mark the layout on the outside, or vice versa. Especially for taps located toward the back of the fridge, make sure you’ll be drilling into the fridge compartment & not into the radiator space.

Locating the taps on the fridge door:

  • Fridge door link for long tap handles.

    Fridge door link for long tap handles.

    If you have a top-freezer fridge, either make a fridge door link (see photo) or locate the taps low enough on the fridge door so there’s enough room to open the freezer without hitting the tap handles. Short tap handles extend 5″ from the center of the tap holes to the top of the tap handle. Long tap handles can extend upwards of 12-18″, so in terms of pour height, long taps work better located on the sides of the fridge, or use the door link. For a bottom-freezer fridge, tap handle height isn’t a problem.

  • Open the fridge door to see what you’ll be drilling through on the inside.

Regardless of the tap locations, you’ll need a flat or mostly flat surface on the inside of about 1.5″ diameter to tighten down each tap shank nut, which keeps your taps from spinning.

Tap details

Who knows, you might get this tap handle someday?

Next, tap shank length. The shank goes from the outside of the fridge to the inside. The tap faucet should stick out from the fridge so the tap handle (especially tall ones) can tilt back closed without hitting the fridge. I’d recommend getting at least 5″ long tap shanks. Whether or not you end up using long taps, it’s nice to have some space around tap handles, & it’s easy to make a wood spacer. More on that later.

For horizontal tap spacing, I’d wouldn’t go tighter than 3″ on center. Consider the available width on your fridge, your drip tray width, & future tap expansion plans. If you’re not ready to not put in your maximum number of taps right away, with tap spacing over 6″ apart, you can always add a tap in between later on.

Drip trays

There are two types, drain hole & no drain hole. Trays with a hole have to drain somewhere, like into a bucket — get these if your pouring style is to dump the first foamy half-pint down the drain. Drain trays are also good for amateur guest pourers who frequently overfill. Be warned though — with either style, mold will grow after a few weeks, & it stinks. No drain means limited spill capacity, & you’ll need to rinse it out more often — not necessarily a bad thing — and it looks cleaner, less like a science project.

Trays are available starting at 6″ wide & go up from there. Tap faucet spouts extend down a couple inches, so to fit most beer glasses, plan on locating the drip tray top at least 9″ below the tap holes’ center line … 12″ below is pretty common.

Minor detail: the drip trays will need to be mounted out from the fridge wall the same distance as the wood spacer you make for the taps.

No gaugesA caveat for beer super-snobs

This setup assumes you’re good with keeping your beers at one common pressure. Most beer drinkers are okay with this. If you seriously need your beers kept at simultaneously different pressures, this setup is not for you.

I’m buying what now?

Here’s the shopping list. Don’t get overwhelmed. You save $1 with each beer you drink! IMPORTANT: Kegs & couplers linked below are ball-lock.

  • CO2 tank – $65 for 5-gal tank (empty), or rent from your local homebrew shop.
  • Dual-pressure regulator (60 PSI max) w/ shutoff & check valves – $103
  • 4-way manifold with shutoff & check valves – $50 (less & more valves available)
  • Drip tray – $24 for 6″ no drain wall-mount, or $24 or $16 for a 19″ no drain stainless that requires shelf or glue mount. eBay, & are good sources for more drip trays.
  • D-style keg coupler – $26
  • Air tubing, 12′ of clear 5/16″ ID – $10
  • Beer tubing, 100′ of clear 3/16″ ID – $55 – each tap requires ~15′, & with this large roll you can just replace tubing rather than using strong chemical beer line cleaners.
  • Beer line (5′) with ball-lock coupler & party tap – $14
  • Beer line tail piece fitting – $3
  • Quick disconnect set – $15 each, get two
  • Male quick disconnect – $7 each, get two
  • 11 hose clamps (1/4″ to 5/8″ range) – $4 for a 10-pack. Get two packs since you’ll need more for your taps – see below.
  • Spray bottle – $3
  • Sanitizer, 32oz – $16 for iodine-based or $24 for acid-based Star San
  • PBW cleaner, 1lb – $9
  • Keg lube – $5
  • O-ring kit for corny kegs – $3 – just in case.

If you aren’t putting in your maximum possible number of taps right away, get a manifold with more valves than you might use at first. This makes future expansion easy, & just leave the extra valves shut off. With a 4-valve manifold you can have 4 beer taps with a shared line for keg transfers, or 3 beer taps plus a dedicated transfer line. Also, most manifolds have a pass-through port on the end so you can always add another manifold later on.

For each tap, you’ll need:

A new keg. So pretty.

A new keg. My precious.

  • Ball-lock corny keg – $120 new, or ~$60 (used) on eBay, or try Craigslist.
  • Perlick 630SS stainless tap faucet – $40
  • 5″ stainless tap shank w/ nipple assembly – $22
  • Air tubing assembly (5/16″ ID clear) w/ ball-lock air coupler & hose clamps – $9
  • Ball-lock beer coupler – $5
  • 2 hose clamps (for beer line)
  • Tap handle – $2 for a plastic handle, or the sky is the limit on eBay, buy them direct from your favorite breweries, or make your own.

The new kegs from are amazing, made in Italy & shipping is free! Granted you can save a lot with used kegs, but be prepared to deal with cleaning soda syrup residue, leaky O-rings & other light keg maintenance — no big deal & O-rings are cheap, but used kegs are just not as easy, or as shiny.

If you need to cut costs, you can find a chrome-plated tap & shank combo for half the price of stainless, like this one (2 taps with 2 shanks for $57).

Kegerator Beer Taps

Side taps. The wood spacer allows for taller tap handles.

The 5″ tap shanks are likely much longer than necessary to get through your fridge wall, so you’ll need to make a wood spacer for the outside of the fridge. The benefit is you can use long tap handles — the spacer creates enough room so the tap handles can reach shut-off position without hitting the fridge. Cut & stack thin boards together until you get the thickness you need. Hardwood scraps work great, especially flooring, or pine 1x4s. If you don’t want the option of using fun long tap handles (what’s wrong with you!), get shorter shanks — whatever length is just long enough to get through your fridge wall.

A 5-gallon CO2 tank should last around 6 months, depending on how much you drink. I usually go through something like 2 half-barrel keg transfers & serve 4 kegs of beer & 4 kegs of seltzer. Seltzer is at a higher pressure, so it takes more CO2 per keg. Eventually consider getting a spare CO2 tank, because it sucks when you run out & can’t pour beer.

Get the hole saw set. Works much better than ruining spade bits.

Tools required

  • Saw (for wood – jigsaw, handsaw, etc)
  • Knife
  • Power drill
  • Tape measure
  • Crescent wrench
  • Regular screwdriver
  • 3/4″ & 1″ hole saws – $13 for a 13-piece set
  • Tap wrench – $3
  • Round metal file (optional)

When pushing beer or air tubing onto a barbed fitting, first dunk the tubing in a cup of very hot water for a minute or two to soften the tubing. Otherwise it’s basically impossible.

Setting up the CO2

  1. Get your CO2 tank filled at your local homebrew supply, welding supply, etc.
  2. Determine a good place for the CO2 tank outside the fridge & make a bracket or strap for it so it can’t fall over. Also decide where to put the air manifold, probably outside the fridge again. Mounting it on a board makes things easy. If you do put the manifold inside the fridge, that means less drilling (only requires 1 hole for the CO2 supply hose) but definitely put the manifold in front near the door, not behind any kegs.
  3. Screw the regulator onto the CO2 tank. It’s a rubber gasket, so don’t overtighten & don’t use teflon tape. Close the inline shutoff valves below the regulators, then open the main tank valve. Turn the pressure adjustment knobs on each regulator to set one at ~12 PSI (beer), & the other at ~40 PSI (seltzer). If you turn a knob to decrease pressure, bleed off the excess by opening the inline shutoffs very briefly to bleed off the excess pressure & see the reading change.
  4. CO2 manifold shutoff valves

    CO2 tanks & manifold with shutoff valves.

    Cut & use ~3′ from the 12′ of air tubing, & run it from the CO2 regulator you set at 12 PSI to the air manifold’s supply end. Save the other ~9′ for the CO2 supply to the D-style coupler for keg transfers.
  5. Find a good spot for the air hoses for each tap to pass into the fridge, where hoses won’t get in the way of your keg layout. Drill holes for the air hoses. Don’t hit any coolant lines. Use a 5/8″ drill bit or hole saw, or a 3/4″ hole saw. 3/4″ is larger than the air tubing, but it won’t matter after you seal them. Dull any razor-sharp edges with a metal file or screwdriver.
  6. Run air hoses for each tap through the holes you drilled. These air lines run from the manifold valves for as many beer taps you’re putting in, & also one air line directly from the seltzer regulator. Tape the ends of the hoses as you push them through so you don’t get fridge foam crud in them. Seal around the holes with silicon sealant.
  7. Cut air hoses to an appropriate length, & attach the air couplers to the hose ends inside the fridge (if they didn’t come preassembled) — just don’t make the air tubing too short. The air couplers are harder to push onto the kegs once the air lines & kegs are pressurized. If vertical space is tight, leave enough tubing so you can push couplers onto kegs just outside the fridge where it’s easy & then set the kegs back into the fridge. Label the air hoses #1 #2 etc at the valve end & also at the coupler end.
  8. Check for leaks — pressurize the system with the regulator & manifold valves open, then turn off the tank valve & check the pressure gauge after ~15 minutes. It should not read zero.

If you have a leak, spray soapy water on all the CO2 fittings. Try around the regulators first since that’s high-pressure, especially the threaded metal fittings on the regulators & manifold. Carbon dioxideIf threaded fittings are leaking, try tightening them & if that doesn’t work, you’ll need to take them apart, clean off the old teflon tape, wrap on new teflon tape & retighten. Also repeat the pressure test with the manifold valves closed. If that fixes the leak, it’s either the check valves directly below the shutoff valves (retighten or re-teflon & tighten), or the keg couplers are leaking around the pin that’s up inside the coupler. That’s fine since they will be on beer kegs, & what matters is the O-ring seal on the keg poppet. Remember to shut off manifold valves for any CO2 supply lines you aren’t using.

If you have an extra valve at the manifold that you’re not using for beer taps, save it for a dedicated air line to the D-style coupler for transfers — that one doesn’t go into the fridge. Or if you’d rather use all your manifold outlets for taps, that’s okay — you’ll set up one tap air line as a dual-purpose keg transfer air line, using a quick disconnect. More on that later.

Setting up the taps

Shiny Perlick taps.

So pretty. So shiny. So much potential.

  1. Drill your tap holes. A few simple words, but such a big step. There are many things to consider here … intrusion into your keg layout, faucet spacing, pour height, tap handle clearance, drip tray … By now you’ve worked all this out, right? Right. Once again, don’t hit any fridge coolant lines. Use the 1″ hole saw. Or you can use (ruin) a 1″ spade bit, but it’s not pretty. The tap shanks are actually 7/8″ diameter but if you drill the holes at 1″, you’ll have less trouble getting the faucets aligned nicely.
  2. Install the tap shanks through the holes. They should extend into the fridge interior as little as possible — which means creating a wood spacer for the outside of your fridge & drilling the tap shank holes through that as well. This will help get the taps away from your fridge enough so you’ll have room for long tap handles. Tighten the shank nuts.
  3. Put the taps on! Use the tap spanner wrench. If you didn’t get one, use vice grips with leather or thick cloth pieces in the jaws so it doesn’t scratch your precious shiny taps. Don’t overtighten.
  4. Coil your beer tubing so it doesn't turn into a giant unruly mess.

    Coil your beer tubing or it’s a giant unruly mess.

  5. Cut your beer tubing to length. There’s a whole science devoted to determining beer line length to get the correct PSI at the tap, which affects foam. Ignore anything that says 3/16″ tubing drops 2-3 PSI per foot — it’s not a linear equation & far more complicated than that. Plan on 12-15′ of beer tubing per tap or if you want to get exact about it, smart physics-PhD-type people have you covered.
  6. Connect your beer tubing from the tap shanks to your (beer) keg couplers.

Setting up the transfer system

The finished product.

The transfer system (short air hose for photo purposes only).

American beers typically come in kegs with a Sankey “D”-system valve. To transfer it you’ll need a beer line that goes from a D-style coupler to a corny keg coupler. You’ll also need a CO2 line to the D-style coupler, either directly from the manifold, or you can set up a dual-purpose CO2 supply using one of the tap air lines. Either way, use the leftover 9′ section of air tubing — it’s probably longer than you need but you can cut it down once you figure out what length is convenient.

    1. If you are going with a dedicated air transfer valve from the manifold, attach the 9′ air hose on to the manifold valve. Cut the air hose a few inches away & install a quick disconnect where you made the cut, with the female end toward the manifold & the male end on the long section of hose.
    2. If you’re dual-purposing a tap air line as the transfer air supply, cut one of the 5′ air line assemblies about a foot from the keg coupler end. Put a quick disconnect on that, with the female end toward the manifold & the male end toward the keg coupler. Then put a male quick disconnect on the 9′ section of air hose.
  1. Attach the other end of the 9′ air hose to the D-style coupler to the barbed fitting, usually labelled “GAS IN”, that’s at an angle.
  2. Take your “Beer line with ball-lock coupler & party tap” & cut the line in half. Remove the beer tubing on the party tap end & save it for the next step — even at 5′ length, it’s far too short to prevent foam. Instead, cut ~12′ of beer line from your bulk roll & put that onto the party tap instead. Put male quick disconnects on both the party tap hose & the keg coupler hose.
  3. Take the short leftover beer line from the last step & attach one end to the D-style coupler using the beer line tail piece fitting, & put a female quick disconnect on the other end.

Sanitize your kegs

If your kegs are really dirty, scrub with dish soap & rinse. For used kegs, you may also need to take off the poppets with a socket wrench to clean them too (search YouTube for help). Add cleaner per the instructions on the container & let sit. This can take 24 hours with some cleaners. Rinse when done. Lube the gaskets with keg lube.

Mix appropriate amounts of sanitizer & shake for an appropriate amount of time. I do about 2 minutes but read the instructions. I prefer iodine-based sanitizer because it’s very effective but not harmful chemicals. Some people prefer Star San. To each his own.

Pro tip: Do not spray sanitizer in your eyes.

Pro tip: Do not spray sanitizer in your eyes.

Do sanitizing with the keg slightly pressurized — hot tap water will heat up the air inside the keg & usually does the trick, or add a bit of CO2 — that lets you run sanitizer out the IN (with keg upside down) & OUT (keg upright) poppets by pushing the pin down with a highly technical tool, like a fork. Do not spray sanitizer in your eyes. Run some through the pressure relief valve (keg upside down) too.

Most people don’t open their kegs after the sanitizer step, & both Star San & iodine-based sanitizers don’t need a rinse — read the labels. Watch YouTube videos if you need more help. Don’t leave the keg open if you rinse it, so bacteria & other contaminants can’t get in.

Partially pressurize your empty sterilized keg, but not all the way or you’ll waste a lot of CO2. This will help you check the seals & also slows down the beer as it starts transferring, which prevents foaming.

Last step is to mix up some sanitizer solution in a spray bottle. Spray all the couplings & keg poppets.

Do whatever it takes to get the keg home safely.

Do whatever it takes to get the keg home safely.

The first beer transfer

Get a half-barrel keg. Treat it like a sleeping baby, no shaking, bumping or dropping it. That creates foam, ENEMY OF BEER. Also, keep it cold, because again with the foam.

Ready for a drink? Okay, but pay attention here because the order matters. Do it wrong & the beer can backflow up the air line.

  1. Clean the half-barrel keg’s D-fitting with your sanitizer spray bottle.
  2. Connect your CO2 supply line to the D-coupler.
  3. Turn on the air valve at the manifold — either the dedicated line, or the dual-purpose tap line depending on how you set it up.
  4. Connect the D-coupler to the party tap — press the quick disconnect until it clicks. Make sure the party tap is closed. If it’s open, beer will start pouring out in the next step.
  5. Connect the D-coupler to the half-barrel keg — twist clockwise until it stops (turn firmly but not hard), then push down the handle until it clicks.

Cheers. Nice work.

Pour off a pint or two. Cheers. The half-barrel is 15.5 gallons & 3 corny kegs fit 15 gallons. So you have at least 4 pints to drink, or store in a growler if you must.

Okay, ready for the keg transfer.

  1. Disconnect the party tap from the D-coupler, & connect the ball-lock coupler. Have some paper towels handy, because the quick disconnects will drip a bit.
  2. Connect the ball lock coupler to your sanitized keg’s OUT poppet.

Beer will start flowing from the half-barrel keg to your corny keg, backfeeding down the OUT tube into the bottom of your keg. Every few minutes, pull the pressure relief valve for a second or two, not too much or the beer will foam & filling your keg full will take MUCH longer. Keep going until foam starts shooting out the relief valve when you pull it.

Get a bathroom scale & put the keg on it. When it hits ~51.5 lbs, it’s full. Anything less & you need to wait 20-30 minutes. This is an excellent time to have another pint, & start filling another keg. By then the foam will settle & you can try topping it off more.

Eventually you’ll hear the bubbling sound of your half-barrel keg going empty.

Clean up

  1. Sterile areaRemove the D-coupler from the half-barrel keg & the corny keg coupler from the corny keg.
  2. Disconnect the transfer air & beer lines at the quick disconnects.
  3. Connect the party tap to the D-coupler & open the party tap to bleed off the residual pressure in the beer line.
  4. With the party tap still open, push the D-coupler handle down, turn it upside down & run tap water through the D-coupler & out the party tap, then run some sanitizer solution through it, then drain it.
  5. Disconnect the party tap & attach the ball-lock coupler. Push in the bottom of the ball-lock coupler — this can be a little hard — use the sanitized head of a small screwdriver. Run more water & sanitizer through the D-coupler, & drain the lines again.
  6. Put the whole transfer setup away somewhere clean, ready for your next keg transfer.
  7. Tag your corny kegs with the beer details & transfer date.


Final beer setup

Make sure your air lines are pressurized. Connect your CO2 & beer lines to the kegs you’re tapping (close taps first) & put everything into the fridge. Check for beer leaks & make sure the fridge door shuts all the way. Pour beer out the taps. Drink. Be merry. Have parties. Enjoy your awesome new beer drinking experience.

Check for beer leaks again in a few hours, just in case.

ONLY CONNECT A CO2 LINE TO A KEG WITH THE MANIFOLD VALVE SWITCHED ON, SO THE CO2 LINE IS PRESSURIZED. Better yet, connect the liquid line first & pour a glass, then connect the pressurized CO2 line.  Kegs can build up pressure while sitting. Pouring off beer before hooking up the CO2 reduces keg pressure & prevents backflow up the air line. Some keg couplers don’t have an integrated check valve (backflow preventer). Even when they do, they don’t always work.


What about the seltzer?

Oh, right. The seltzer. Fill a sanitized keg with tap water. Hook up your 40 PSI CO2 line, shake the keg for 10 minutes, & put it in the fridge. Come back & start drinking seltzer in a few days. For faster carbonation, get a carbonation stone ($6).


Go buy a 6-pack.

I’m bored, what else can I do?

Build a keezer. It’s a kegerator made from a chest freezer. More room for more kegs!

Already own a kegerator & just want to add a second seltzer regulator?

1/4" MPT regulator coupler LHTGet another primary regulator ($55) that can handle up to 60 PSI & has a shutoff AND a check valve. Then buy a 1/4″ MPT LHT coupler ($5) or if that’s out of stock, try here. This setup works better than secondary regulators with no chance for cross contamination.

Threading info on the regulator housing.

Threading info on the regulator housing.

The LHT (left-hand thread) adapter will handle the typical CO2 regulator thread setup. Check your regulator threads to be sure — check the housing, or you can tell by looking at it: which way would something twist when being screwed in? Clockwise = RHT, Counterclockwise = LHT.

Remove the pressure gauge from your old regulator & remove the tank inlet from your new regulator. Thread the two regulators together using the adapter with teflon tape. Set the new regulator at 40 PSI, clean & fill a keg with water, hook it up to your seltzer CO2 line, shake the keg for 10 minutes to give it a head start, & wait a few days for full seltzer carbonation. For faster carbonation, get a carbonation stone ($6).

Fox and baby chicken

Heavy Duty Solar Powered Automatic Chicken Coop Door

Fox and baby chickenWhen we first got our chickens, each night I’d walk up to the coop & close them in. That worked great until the night I’d fall asleep putting our 3 kids to bed, or start watching a late-night movie, & suddenly OOOOHH SHIT, THE CHICKENS!!! …followed by a guilty run to the coop, wondering if I was about to find sleepy hens or a poultry massacre.

Chickens are a tasty snack for lots of predators. It’s a tough spot in the food chain. Locally we have raccoons, possums, weasels, foxes, coyotes, neighborhood dogs, hawks, eagles, owls… my friend Chris who loves fried chicken a little too much… Raccoons at night were my main concern.

A few months of this started to feel like Russian roulette. Like so many other pet chicken owners, I decided to try building an automatic chicken coop door.

On Youtube there are plenty of automatic chicken coop doors that use string to raise/lower a guillotine-style door, sliding vertically in a track. It’s a safe design — it won’t kill chickens if one gets in the way while closing — but I was worried the door would get jammed from ice & snow.

I wanted something with a direct drive to use with a door that swung up on hinges to open. Here’s what I ended up using:

  • 12V linear actuator, 8″ extension, IP65 rated w/ built-in limit switches & mounting brackets: ~$55
  • (2) 12V programmable digital timers: $5 each
  • automatic chicken coop door12V DPDT relay w/ base: $3
  • Wiring, inline fuse holder/fuse, terminals: $6

A/C dedicated power option:

  • 12V 6-amp power adapter: $7

A/C with power outage protection:

  • 12V 7-amp battery: $17
  • battery maintainer: $20

D/C solar power option:

  • 12V 7-amp battery: $17
  • Low-watt solar panel: $32
  • 12V solar charge controller: $15 (optional)

Total cost: $81 dedicated A/C, $111 A/C battery backup, or $123 solar D/C ($138 w/ charge controller)

Linear actuators use a small motor to move an extendable/retractable arm. The arm moves very slowly with 50 to 200 pounds of force. Get one with built-in limit switches & an IP65 rating so dust/water/ice/snow is no problem — pretty great for chicken coops. There are various lengths for the arm travel distance. I got the 8″ model & it takes about 20 seconds to extend/retract the arm … plenty of time for chickens to move out of the way. Small 12V actuators like these usually have a rating of around ~5 amps, so make sure to use a relay, fuse & wiring that’s appropriate. Also make sure the actuator comes with mounting brackets, or you’ll need to come up with something.

Typically, actuators with higher force ratings mean slower movement. Same goes for the arm extension length — longer extension means your door closes more slowly — more time for chickens to get out of the way.

Next, how to power it. If your coop is near A/C power, you could use a 12V power supply instead of the battery/solar panel. Just make sure the power supply is rated for enough amps to reliably drive the linear actuator motor. Better yet, use a 12V battery permanently hooked up to a battery maintainer & you won’t ever have to worry about power outages.

Our coop is on wheels & we move it around our field far away from A/C power, so I needed it to be self-powered. Linear actuators only draw a few amps so a small 12V battery will do the trick — I had an old one lying around that wouldn’t start the lawn mower anymore, but worked great for the coop door.

Solar panelTo recharge the battery, I used a small 1.25-watt 12V solar panel. Since the panel’s power output is so low, it acts as a trickle charger, & that way you may not need a solar charge controller as long as the panel is in direct sunlight for most of the day. I’d still recommend a charge controller to make sure the panel doesn’t have a net drain effect on the battery in winter or other low-light conditions.

Last challenge was for the door to open in the morning & close in the evening. I went with a simple setup with very low power draw: two programmable 12V timers. The first timer (the “power timer”) switches on twice a day for 1 minute each to provide power to the actuator. The second timer (the “reversing timer”) energizes a DPDT relay concurrently during one of the power timer events to reverse polarity to the actuator. That opens & closes the coop door.

The last piece is a 12V DPDT relay wired as an H-bridge. This relay has 4 sets of +/- pins: normally closed (NC), normally open (NO), common, & coil. The coil switches the common between the NC pins to the NO pins. For the H-bridge setup:

  • connect your power source (+/-) to the timers’ power inputs. Fuse on the (+) wire.
  • both timers: jump power (+) over to the 1st switch pin.
  • power timer: connect 2nd switch pin (+) to a NC pin on the relay.
  • jump that same NC pin (+) to a NO pin, but with opposite polarity.
  • reversing timer: connect 2nd switch pin (+) to a coil pin (doesn’t matter which one).
  • connect the linear actuator (+/-) to the common pins.
  • connect ground (-) to the remaining open pins on NC, NO, & coil.

If when you’re all done the actuator operates the opposite from what you want, just flip the actuator’s connections to the relay’s common pins.

Automatic chicken coop door wiring diagram

Next, program the timers so their clocks are set identically. Let them sit for a few days & figure out which timer is faster than the other. Use the faster timer for the reversing timer.

Power timer: set for two daily events (morning & night) of 1 minute each. For example, 6:30AM- 6:31AM and 9:00PM – 9:01PM.

Reversing timer: set to run concurrently with the morning power timer event, so it comes on sooner & stays on longer than the power timer. For example, 6:30AM – 6:35AM. I prefer the morning run so if anything goes wrong it only means the door won’t open (no big deal).

Whenever you change the time, make sure the reversing timer is always just a bit ahead. This way you can have the morning event start at the same time on both timers.

Check the timers after a month. I was surprised to find my timers get about 20 seconds off from each other. To compensate, I set my reversing timer event to stay on for 5 minutes — energizing the relay coil is a very minor drain on the battery. That way my system can run for over a year before I’d have to resync the timer clocks. I change the timer settings 3-4 times a year anyway, to adjust for daylight.

Here’s the whole system in action:

UPDATE: In the video I mention as a good source for the actuator, but they don’t include mounting brackets. Lately you can find IP65-rated actuators on eBay with mounting brackets included for the same $60 price, with free shipping.

Bad Web Crawlers

Bad Crawler Bots: Proximic, CrystalSemantics, Grapeshot, Gigavenue

Bad Web CrawlersEvery so often I go through the error logs & watch for server abuse. The latest review found a few new players: Crystal Semantics, Grapeshot, Gigavenue & Mangoway.


Crystal Semantics does after-the-fact contextual advertising. They crawl your pages after an ad is shown. Risky Internet covers this topic well:

Since we do not need a whole series of Ad crawlers making a business out of stealing bandwidth and each on their own reloading pages, the ONLY valid solution is that the seller of the ad-space (whether they are Google Ads or other) deliver the valid classification, since they are the first to crawl the page.No need to have a whole series of separate companies scrape off the same page, and adding more load to all sites, just to make their own business out of it.

Amen to that. Normally I wouldn’t mind so much, but in all their HTTP requests they’ve been accessing the path portion of the URL in all-lowercase. We use mixed case so they’ve been getting gazillion 404 Page Not Found errors. Probably sloppy coding somewhere between their ad agency partner & their service — but after months of 404 errors, they’ve had plenty of opportunity to discover the problem through self-monitoring & fix it.


Basically a repeat of above, except they apparently & rather arrogantly don’t comply with robots.txt. Not quite as many 404 errors as Crystal Semantics had, but I don’t agree with the whole post-ad-serving contextual value added crawl business model.


Evil. They’re crawling the site like crazy from multiple IPs but don’t use a unique user-agent. Zero information about their crawler. Emails to all three email addresses listed on bounce (,, I tried contacting Adam D. Binder via LinkedIn & we’ll see how it goes.

So the changes to robots.txt:

User-agent: crystalsemantics
Disallow: /
User-agent: grapeshot
Disallow: /

Gigavenue doesn’t publish robots.txt info so your guess is as good as mine what robots.txt useragent to use for them.

For good measure, ban them in .htaccess too:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Ruby|proximic|CrystalSemanticsBot|GrapeshotCrawler|Mangoway) [NC,OR]
RewriteCond %{REMOTE_ADDR} ^(208\.78\.85|208\.66\.97|208\.66\.100)
RewriteRule !^robots\.txt - [F]

This bans them by UserAgent for the better-behaved crawlers that have one, and by IP for the evil services that don’t, & sends them all to a 403 Forbidden response, except they can access robots.txt to find out the nice way they are disallowed from crawling the site.

NOTE: These IPs in the example code are now several years old & probably aren’t correct anymore. They are only meant to serve as an example of how to ban these & similar services, if you choose to do that.

United MileagePlus Expired Miles Scam

Onepass United MileagePlus Expired Miles Are A Scam

United MileagePlus Expired Miles ScamFor years I had a Continental Onepass account. It was great. My miles never expired.

Then one day Continental merged with United, & Onepass merged into MileagePlus. United quietly changed the terms & conditions, no doubt to “benefit their customers”.

On September 30 2013, my ~150,000 coveted mileage points expired. I didn’t notice until a few weeks later & quickly saw United’s convenient offer to “reinstate” them for $300. What a silly mistake, I thought. I called the MileagePlus hotline & spent about an hour talking to various levels of customer disservice. In the end, there were three options:

  • MileagePlus Expired Miles Reinstatement FeesPay $300 to have my miles “reinstated”.
  • Find any points that should have been applied within the last 18 months, which would magically reactivate my expired miles.
  • Get the United MileagePlus Explorer credit card, pay $200.

The United phone rep claimed that I’d received warning notices about my points expiring at least three times but couldn’t provide specifics. I didn’t get the memo. It’s odd — and by “odd” I mean complete bullshit — because the whole time I received all the usual account updates & other promotional United junkmail. I even went back & searched through 3 years of Continental/United emails since the merger… nothing about my miles expiring.

Budget Car Rental to the Rescue … ?

I’ll play their little game, I thought. I remembered I’d rented a car from Budget, one of their MileagePlus points partners, within their 18-month period. I even had the receipt with the rental agreement number. So I called Budget, gave them my MileagePlus account number & waited the requisite 8 weeks for my points to show back up. Nothing.

SuperBudgetManI called United MileagePlus support & was assured I needed to go through Budget.

Budget customer service was far more helpful & verified that my MileagePlus number they had used was correct & that the problem was on United’s end.

I sent all that information to United MileagePlus support.

Another reply from United indicated they might deal with the issue if I got confirmation from Budget.

Budget sent me confirmation, which I forwarded to United MileagePlus support.

Waited another week with no reply from MileagePlus.

Called United back & they said it might be because I had rented the car through Priceline. It hadn’t been a “name your own price”, just a normal package deal. But United couldn’t tell me for sure, they had to check with supervisors & they’d let me know.

That’s where I’m at so far. One thing is for sure, I’m not paying to “reinstate” my miles.

UPDATE 1/25/14: Got the reply from United:

After reviewing your MileagePlus account, an inquiry was sent to Budget regarding the missing mileage request. Budget advise that rental did not qualify for mileage credit due to an invalid rate code. If you have any questions, please contact Budget.

So apparently booking through Priceline (not even “name your own price”) means no mileage credit. No reinstated miles.

I went with the F@#$%^&*! credit card option. That way it’s $200 to recover the expired miles, but you get 30,000 miles once you spent $1,000 within 3 months. I waited for the new card to arrive before calling back to charge the $200 expired miles fee onto their card. The 30,000 points is worth $200 & I get my ~140k expired miles back.

And one cancelled credit card on my credit report — If you also go this route, remember to cancel the card after you get the 3-month points benefit or you’ll get hit with their annual fee a year later.

Page 1 of 5

Powered by WordPress & Theme by Anders Norén