Netscraps – A windsurfing, CSS-grudging, IE-hating, web-developing, gigantic-machine-puzzling blog

In DIY

Honeywell makes a programmable digital “solar time table” switch that adjusts events automatically throughout the year based on daylight hours.

It’s far more reliable than a photocell, but one caveat — it runs on AC power. If you only have 12V DC power at the coop, use one of the other automatic chicken coop door methods I’ve written about: 12V timers or photocell. For those of you with AC power available at your chicken coop, read on!

On the Honeywell switch you’ll set the current date, your latitude & longitude & your event times, & the switch does the rest. The switch also has an override button that cycles the switch on & off manually.

The other component that makes the whole setup work is the relay. You’ll need a DPDT relay with a coil rated for 120VAC, but with 12VDC contacts. See the full product lists with links below for each relay type.

Option 1: Heavy Duty Functional Devices Relay

Heavy duty 120VAC/12VDC DPDT relay from Functional Devices.

The heavy-duty Functional Devices relay costs a little more (~$12) mostly because it’s rated for continuous duty at 10 million cycles. It will be on for hours at a time with this timer setup, so continuous duty is a good thing.

Also if you mount the Honeywell Solar Timer switch into an outdoor-rated junction box, the Functional Devices relay screws into the top of the box & you end up with a nice compact dustproof timer setup that will last a long time, even in a chicken coop.

Here’s the equipment list:

If the linear actuator runs backwards from how you’d like it to work, reverse the 2 actuator leads where they connect to the relay’s yellow & purple wires.

If YouTube is your thing, here’s the how-to video.

For setup tips & troubleshooting, scroll down below the “standard-duty relay” section below.

Option 2: Standard-duty relay

A standard-duty relay works fine but the relay coil may fail over time, since these relays typically are made with pretty cheap components. However if it fails, you can just unplug the bad relay & plug in a new one.

Here’s the full product list:

The blue & white wires run to pins (screw terminals) 7 & 8. The actuator connects to pins 5 & 6. Connect 12V power to pins 1/2 & 3/4 with each pair having opposite polarity from each other — so if the bottom pins have red/black connected, the top pins have black/red.

DO NOT GUESS WHICH PINS THE AC POWER (BLUE & WHITE WIRES) RUN TO ON THE RELAY BASE! If you can’t determine which pins 7 & 8 are from the tiny writing on the base, they are the outer/lower set of pins on the end of the relay base that has the “nub” that sticks out. Don’t confuse pins 7/8 with pins 1/2 on the opposite end or very bad things will happen.

If the linear actuator runs backwards from how you’d like it to operate, reverse the actuator leads where they connect to the relay base (pins 5/6).

Make sure you wrap the exposed screw terminals with electrical tape where the white & blue wires from the timer connect to the relay base (pins 7 & 8). Remember, those carry AC power.

Setup Tips & Troubleshooting (both relay types)

Safety first: Always work on the wiring with the AC power cord unplugged.

I recommend putting a fuse on the positive 12V lead. The appropriate fuse size is typically 7.5 amps for most linear actuators.

For 12V power supply ideas, see this post & scroll down to the “power supply options” section.

In a power outage, the Honeywell switch retains the programmed events. When the power comes back on, the door will catch up on whatever state it’s supposed to be in. One downside to this setup is there’s no easy battery backup to make this system work during a power outage, like there is for the other all-12VDC automatic chicken coop door methods.

I recommend placing everything in a weatherproof junction box & cover since the Honeywell switch isn’t really meant for a chicken coop environment.

Timer setup: use Automatic mode, & make sure for the event times you pick either the sunrise or sunset option. The timer lets you adjust the event time up to 70 minutes before or after sunrise/sunset.

If you don’t know your latitude & longitude, go to Google Maps, right-click near where you live, click “What’s here?” from the menu & use the numbers in light grey text (round to the nearest whole number).

Disclaimer: I’ve only used this system for a short time so far, & I’m using the heavy duty Functional Devices relay. If you went with the standard-duty relay, I don’t know how long it will last.

Any questions or comments, leave ’em below!

Heavy Duty Automatic Chicken Coop Door – Photocell

By Wick

On April 19, 2018

In DIY

UPDATE: If you have AC power in your coop, a more reliable method uses a Solar Time Table Switch which adjusts automatically throughout the year for daylight hours based on your location.

A question that comes up a lot in my Automatic Chicken Coop Door posts is what’s the wiring diagram with a photocell? Ask & ye shall receive!

One thing I’ll say up front while I still have your attention — the wiring diagram above is NOT WRONG in terms of the photocell wire colors. It’s not what you’d expect. The black wire is positive, white is ground, & red is the switched output (+). Also, the pin wiring for the relay may change slightly depending on which relay you use. For instance, here’s the wiring diagram for the “heavy duty” relay option in the parts list below:

This is a “dusk till dawn” photocell which means the photocell switches on at night. It’s advertised as waterproof & the light sensitivity is adjustable. It also has fairly mediocre reviews on Amazon so if anyone has a better photocell to recommend, please do!

Here’s the parts list:

12V waterproof adjustable photocell
12V DPDT relay with base or spend a bit more and get a heavy duty relay
12V linear actuator, 8″ extension, IP65 rated w/ built-in limit switches & mounting brackets from eBay (or get it on Amazon)
12V programmable digital timer ** OPTIONAL — see timer override below.

The DPDT relay is wired as an H-bridge. This means you make an “X” with power to the normally open (NO) & normally closed (NC) terminals so they have reverse polarity from each other. The “common” terminals connect to the linear actuator.

The photocell controls the coil, the coil switches the relay & that reverses the motor. The linear actuator’s built-in limit switches take care of the rest.

IF IT OPENS AT NIGHT & CLOSES IN DAYLIGHT: Flip the leads to the linear actuator where they plug into the relay.

POWER DRAIN: Although daytime power usage is minimal (0.004 amps) when the photocell is only monitoring the light level, the photocell & relay have a constant power drain at night of 0.12 amps when both the photocell & DPDT relay coils are energized. So I recommend only using this photocell system on dedicated power.

You can still use the photocell system with a solar panel/battery setup, but you would need enough capacity to handle the power drain (12 hours night @ 0.12 amps = ~1.5 amp-hours just for operating the photocell). If you need to conserve power, use this timer-driven system instead.

Here’s a video explaining the wiring:

OPTIONAL TIMER OVERRIDE: Don’t trust the photocell? Add a timer so that closing the door happens regardless of whether the photocell works. Here’s a wiring diagram for that:

Set up one timer event:

start time = forces the door to close
end time = door available for photocell to open

So for example, start time of 10pm & end time of 5am.

Keep in mind the timer only overrides closing — if you have a bad photocell that doesn’t recognize daylight, the timer won’t force the door to open.

I use a system of timers instead of this photocell system, because where I live has high winds & I’m concerned the photocell wouldn’t be reliable with blowing dust & snow. The timer system also has far less power drain which is useful if you have a solar-powered coop. I change the open & close times every few months to keep pace with daylight. But the photocell method is pretty slick & if it works for you, great.

Any questions, post in the comments below. Hope this helps!

Ruben Harris TrekkSoft Spam

By Wick

On April 4, 2018

In abuse, awful customer service

Back in September 2017, an email address at my small scuba diving company Hero Divers began receiving unsolicited marketing emails from Ruben Harris, ruben.h@trekksoft.io, Product Marketing Manager at TrekkSoft AG.

As far as I can tell, he’s a legitimate employee & has a ZoomInfo profile.

Most of Ruben’s emails have a customized subject (i.e. “About Hero Divers”) & he mentions Hero Divers throughout the email, so it looks like a personalized email marketing campaign. The links in Ruben’s emails redirect through a link-tracking service & end up here:

https://www.trekksoft.com/en/lp/simple-sp-demo-request?utm_campaign=scubadiving2018&utm_source=email

Here’s the first email I received from Ruben Harris from 9/20/17:

Hi,

I’m writing you in the hopes of finding the person that is responsible for online marketing at Hero Divers. How are things going at Hero Divers? Do you find you are stretched this season?

I work with TrekkSoft, a booking solution for companies like yours. We provide you an integrated booking solution that automates your web, front desk and partner bookings. In plain English, we cut down on phone calls and emails while driving you more bookings through our distribution channels.

Have a look at what we can offer Hero Divers. Now is a great time of year to understand where you can make improvements. I’d be delighted to review your website and your operations to see if we can come up with some improvements together.

If there’s anyone else at Hero Divers that would be better to talk to, I would greatly appreciate it if you could point me in the right direction.

Kind regards,
Ruben Harris
Product Marketing Manager at TrekkSoft

PS: We’ve got a large amount of clients worldwide, which would be ideal to cross sell with through the partner network we’ve set up.

There’s no remove link on any of his emails. I continued to receive similar TrekkSoft emails from Ruben at about one per week: on 9/25, 10/1, 10/6, & 10/12.

I just assume Ruben will get bored after awhile & stop emailing. Sure enough, the weekly barrage stopped after 10/12. But get an axe, it was a trick.

March 9th, what pops up? Another fucking email from Ruben Harris at TrekkSoft. He has the gall to pretend like we don’t have any prior history: “Hi, I’m Ruben from TrekkSoft. I came across Hero Divers online and wanted to get in touch to see if your team is thinking about options for an all-in-one booking system ….”

March 13th, it’s Ruben again: “Hi, just checking in to see if you wanted to ask any questions about TrekkSoft booking software ….”

I reply that I’m not interested.

March 18th, more from Ruben, “Just checking in to see if you had any questions about TrekkSoft for Hero Divers ….”

I reply again asking to be removed.

March 22, Ruben Harris at it again: “Quick question: as scuba diving company, what’s the biggest challenge for Hero Divers right now? ….”

March 26, yet another email from Ruben: “It’s Ruben from TrekkSoft. I’m writing to see if you’d seen my last message and to quickly pass on our Travel Trends Report 2018 ….”

March 30, yep: “Hi, I hope you found my last few emails useful. I also wanted to offer you a time slot to discuss Hero Divers properly and find out whether TrekkSoft’s all-in-one booking software could be a good fit for you ….”

Replied yet again asking to be removed.

April 3, “Hello, just wondering if you found the time to read through my emails over the past few weeks. If you are not interested in what our software can offer Hero Divers then I won’t mail you again ….”

Yes Ruben, I read your fucking emails. And if it’s true what you say that you won’t email me again, thank fucking god. Although by now, I’m mad.

I would never do business with TrekkSoft, even if I was interested. Ruben, either run a legitimate mass-marketing campaign & include remove links, or run a personal one & remove people when they ask.

I do a little research. Apparently I’m not alone. I find this from “The Stranger” on Twitter from 2016:

https://twitter.com/rivenworld/status/691033252908371968?lang=en

April 3, I post on the TrekkSoft Facebook page complaining about all these Ruben Harris emails & asking if he really worked as Marketing Manager for the company.

Next day I go back to the TrekkSoft Facebook page & my post is gone. Looks like they’ve hidden my post with no reply. Now I’m the angry non-customer.

April 4, I post again on their Facebook page:

April 5, no reply from TrekkSoft. At least my comment is still up on their Facebook page. Facebook indicates TrekkSoft “usually replies within an hour”, but it’s been 2 days & counting with no reply to my Ruben Harris spam complaint.

I did some more digging & found out Quickmail.io is the link tracking service Ruben is using. They have a very helpful abuse reporting page, so I sent in a spam abuse complaint about TrekkSoft with copies of all the spam emails & my requests back to be removed. Really hoping they shut Ruben down.

TrekkSoft is a marketing company. It’s ironic how badly they are screwing up their own marketing & reputation with this shitty Ruben Harris email campaign. I also sent a direct message to VP of Sales at TrekkSoft with my complaint & a link to this blog. Maybe he’ll care.

Stay tuned.

DJI Mavic Air vs Mavic Pro vs Spark Specs Comparison

By Wick

On January 23, 2018

In tips

It’s been a whole entire day since the Mavic Air was unveiled & the internet hasn’t put together a specs comparison of DJI’s newest drone with their other lower-end models. To make things worse, the DJI website doesn’t use consistent units.

TL;DR: If you want 4K video & don’t mind the shorter range & somewhat shorter flight time, get the Mavic Air. Otherwise get the Mavic Pro. Get the Spark if you don’t have loads of money.

So I’ve compiled & converted all that for you. Here goes:

DRONE STATS	Spark	Mavic Air	Mavic Pro
Flight time	16 mins	21 mins	27 mins
Range	1.2 mi (2 km)	2.5 mi (4 km)	4.3 mi (7 km)
Top Speed	31 mph (50 kph)	42 mph (68 kph)	40 mph (65 kph)
Ascent	9.8 ft/s (3 m/s)	9.8 ft/s (3 m/s)	16.4 ft/s (5 m/s)
Descent	9.8 ft/s (3 m/s)	6.6 ft/s (2 m/s)	9.8 ft/s (3 m/s)

Notes: “Range” is the transmitter range, not total theoretical travel distance. Top speeds are in sport mode. Some stats were rounded to whole numbers because as you learn in school, sometimes decimals are ridiculous.

CAMERA STATS	Spark	Mavic Air	Mavic Pro
Max resolution	HD (1080p)	4K (2160p)	4K (2160p)
Max HD framerate	30 fps	120 fps	96 fps
Max 4K framerate	N/A	30 fps	30 fps

SCAM: Web Content Scraping in Realtime

By Wick

On April 5, 2017

In abuse

I discovered several illicit websites have been scraping, reprocessing & re-serving copyright web content from CarComplaints.com in real-time.

It’s an assholish way to do business.

Here’s how the scam works:

An unsuspecting visitor to one of these illicit websites requests a web page.
The web server passes the request to the content scraper bot.
The scraper bot script makes a web request to the legitimate website & reprocesses (steals) the content.
The scraper bot transmits the stolen content back to the illicit web server.
The web server serves the stolen content back to the site visitor.

This content-scraping happens in realtime, in the background over a few seconds as the visitor’s browser sits there waiting.

The first content scraper site I discovered was replacing “CarComplaints.com” anywhere it appeared in the HTML code with the name of the illicit website, & also replaced advertising so that the scammers earned the ad revenue instead of my company. Evil!

The largest offender so far was the website carcomplaints.xyz, which has since been shut down after I filed complaints with their ISP. They had managed to get ~9,150 pages indexed by Google, which are (hopefully) in the process of being removed sometime soon. Their entire site was a duplicate of mine with all pages scraped from my site & returned to their visitors in realtime. The scam website was hosted on a different IP & service from the content scraper, but it was easy to track down by requesting a bogus page on the scam website & then watching the content scrape request hit my site by tailing the Apache access log.

Once I identified these content scrape requests, I reviewed my access log & found many similar requests being made from other IPs, but I couldn’t find the corresponding scam websites. It’s impossible to track down which website these requests were originating from, but you can still go after the ISP that’s hosting the content scrapers.

For now the scraper bots are using the useragent “Go-http-client/1.1“.

Many of the scraper bots use Amazon AWS as the host. To file a complaint, email details including log files to abuse@amazonaws.com — generally AWS is pretty good about taking care of it, but you will need to prove there’s been an AWS Acceptable Use Policy violation or else AWS simply passes your complaint on to their customer.

To establish an AUP violation, ban the Go-http-client useragent in your robots.txt file. AWS requires any clients operating web crawlers to follow the robots.txt convention. I couldn’t find where any of these IPs had tried to access robots.txt but I did it anyway so AWS could take further steps against their client.

Until the scammers change the useragent, you can also ban that traffic by returning a 403 Forbidden response using a RewriteRule in .htaccess:

RewriteCond %{HTTP_USER_AGENT} (Go-http-client) [NC]
RewriteRule !^robots\.txt - [F]

Or have a bit more fun with the scammers & redirect their content scraper requests to a copyright violation notice page:

RewriteCond %{HTTP_USER_AGENT} (Go-http-client) [NC]
RewriteRule !^(robots\.txt|copyright_violation\.html) /copyright_violation.html [R,L]

Or the FBI Cyber Crime page:

RewriteCond %{HTTP_USER_AGENT} (Go-http-client) [NC]
RewriteRule !^robots\.txt https://www.fbi.gov/investigate/cyber/ [R,L]

NOTE: These examples assume you already have mod_rewrite enabled & “RewriteEngine On”.

Heavy Duty Automatic Chicken Coop Door – Easier Timers

By Wick

On August 14, 2016

In DIY

UPDATE: I’ve also posted instructions for an automatic chicken coop door using a photocell with an optional timer override. However it consumes more power & the photocell may not be as reliable, so if that’s a concern, use the two-timers method below.

UPDATE II: For coops with AC power, there’s a 3rd method where the automatic coop door uses a Solar Time Table switch.

In March 2015 I posted a method for making an automatic chicken coop door using two timers & a DPDT relay, but the timer setup was complicated — one timer provided power, while the 2nd timer controlled reversing polarity & had to turn on simultaneously with the power timer. Not easy.

Here’s another method of wiring the timers that’s more straightforward. One timer opens the door & the 2nd timer closes the door. One event per timer … Simple, easy, inexpensive, & as reliable as the old way.

NOTE: Older CN101A timers may need the timer power wires swapped (reverse polarity).

This new system uses a dual-SPDT relay module which replaces the DPDT relay in the old design.

Any 2-channel SPDT relay module with a high-level trigger should work. Typically there are 6 terminals on one side: NO/NC/COM for each relay, & 4 terminals on the other side: signal inputs for each relay (IN1/IN2), & power for the module (marked as +/-, or VCC/GND). There’s a jumper block to select the trigger type.

The timer wiring is the same as before — daisy chain power to each timer, & then to the module. Jump (+) to both NO terminals, and (-) to both NC terminals. Connect the actuator leads to the COM terminals. Run the output from each timer to the module’s IN1/IN2 terminals.

NOTE: Both trigger jumpers must be set to HIGH (outward setting). Apparently this relay is occasionally shipped with the jumpers set to LOW (inward), which would require different wiring from what I’ve shown.

YET ANOTHER NOTE: Sometime in 2016, these CN101A digital timers changed so the two power leads are reversed from how earlier CN101A timers work. I’ve updated the wiring diagram above to reflect this change, so now looking at the timers from the front, (-) is connected at the far left & (+) is 2nd in from the left.

Parts list:

If you want a guillotine door instead of a swing door, get a 12″ extension linear actuator instead. Although around your chickens, maybe call it a “vertically sliding door”…

Power Supply Options

I’ve received several questions about my wiring diagram’s purposefully ambiguous “power supply”, so here are some different options.

Solar panel: You can use a very low-watt solar panel connected directly to the battery with a fuse, so that the solar panel acts as a trickle-charger. The problem is the solar panel also slowly discharges the battery at night*, & so this system relies on whether the solar panel can generate more power during the day than it uses at night – normally not a problem, except if you live somewhere like I do without much sunshine in the winter.

* To prevent discharge you can add a blocking diode, but I’m not going to get into that (Google has your back) — the solar charge controller below is a better method for about the same price.

Solar panel w/ controller: This uses a solar charge controller which regulates power to the battery & automatically disconnects the solar panel at night. You can use any size solar panel, although panels over 20W are probably not necessary unless you are using a different system with a higher constant power draw (like a photocell) rather than the two timers.

Dedicated power: If you have A/C power to your coop, you can use a 12V power adapter with an amp rating higher than the power draw of the linear actuator. This method is by far the least expensive, but if the power goes out, your chicken coop door won’t open/close.

Dedicated power with battery backup: Nice method that handles power outages. With this system you need a trickle charger (“battery maintainer”), and a 12V battery with an amp rating higher than the power draw of the linear actuator. As with any battery, put a fuse on the positive lead coming off the battery.

So that’s it for power supply options. Here are some other useful notes:

Fuse sizing: Typically the fuse is rated 50% more than the maximum power draw of the linear actuator, so for instance if your linear actuator is rated for 5 amps max, use a 7.5-amp fuse. For a 6-amp linear actuator, use a 10-amp fuse.

Wire gauge: 16-gauge or 18-gauge wire should be fine, unless you are using more than a few feet of wire for some reason.

Wire connectors: I used spade terminals to connect wires to the timers & battery tabs. Keep in mind you’ll need to use a larger size terminal (than your wiring) when you splice 2 wires into one terminal. You can order a nice assortment of terminals on Amazon, or your local hardware store typically sells individual spade terminals from the small parts drawers.

Timer setup: First, set the time. Hold down the “clock” button & (still holding down “clock”) press D/H/M buttons to set day of the week, hour & minute.

Then press & release the “P” button. The number in the lower left shows the timer event number (1, 2, 3 etc) & whether you are setting the ON or OFF time for each event. So the first time you press “P” the timer shows “1” and “ON” in the corner — you are setting the start time for the first event. Press the H/M buttons to set the event start info. To have the event occur every day, make sure the display indicates “MO TU WE TH FR SA SU”. To change it, push the “D” button. When you’re done setting the event start info, press “P” again & set the same info for the event’s end time. Press the clock button when you’re done.

Example timer settings:

Door open timer: start event 6:30AM, end event 6:31AM.
Door close timer: start event 8:30PM, end event 8:31PM.

Final step is press the “Manual” button until you see “AUTO”. That means the timer is ready to be triggered by the events that you set up.

Press “Manual” whenever you need to override the timer. It cycles through AUTO -> ON -> AUTO -> OFF, so you may need to push the manual button several times to trigger ON. Remember to set the mode back to AUTO when you’re done — otherwise the events won’t trigger the timer.

The “C/R” button resets the time if you make a mistake setting up an event.

Manufacturer instructions for the CN101A timer are here.

Troubleshooting: If the actuator runs backwards, switch the actuator leads where they plug into the COM terminals. If the wrong timer controls the wrong event, switch the timer output leads either where they plug into the IN1/IN2 terminals or at the timers (doesn’t matter, same result). If a timer doesn’t switch at all, reverse power polarity to the timer (swap positive & ground). Also make sure the power supply has sufficient amps because otherwise the red light will come on but the timers won’t actually switch the circuit. If the timers don’t work when an event occurs (no red light & no “click” sound), make sure it’s set to AUTO mode — push the MANUAL button until you see AUTO on the display.

Wiring & testing the prototype.

Circuit Details: With neither timer activated, both motor leads are (-). With one timer/relay pair switched on, one lead switches to (+), the other stays (-) & the motor either runs forward or reverse. With both timers activated, both motor leads are (+) … that shouldn’t happen with your timers set properly, but it’s fine if it does (not a short circuit).

Don’t shoot the hobbyist: So far this design seems reliable. I’ve only had to replace 1 timer that stopped switching after 5 years of use.

Troubleshooting: See below for two videos that demonstrate normal operation of the relay & tips such as how to set the relay module trigger & test the relay.

Questions for electrical engineers:

Does this module handle EMP from the actuator motor being switched off, or ideally should I add something to manage that? There are a bunch of other components on the circuit board in addition to the two relays, not sure what it’s designed to handle.
I’ve come across this relay module used with IN1/IN2, NO1/NC2, & NC1/NO2 each jumped together, like this, which seems to provide the same exact function as a single DPDT relay wired as an H-bridge. To me this makes very little sense — essentially using two SPDT relays to accomplish the same function as one DPDT relay, but with more complicated wiring & greater possibility of component failure. Are there any benefits to this setup over a single DPDT relay?
Is there any benefit to using a motor reversing solenoid over this 10-amp relay module (perhaps built-in handling of EMP)? Or are those solenoids just primarily designed to handle more amps & a longer duty cycle?

Happy chicken coop dooring. Any questions or comments, let me know!

If you use this automatic chicken coop door design in a video or blog post, please give a link or mention this blog post. Much appreciated.

Bad Crawler Bots: ptr.cnsat.com.cn

By Wick

On April 9, 2016

In abuse

Found this bot accessing the site via lots of different 202.46.* IPs. Reverse DNS points to ptr.cnsat.com.cn.

The range of IPs for 202.46.32.0 to 202.46.63.255 is associated with ShenZhen Sunrise Technology Co., Ltd.

This is how to ban via .htaccess RewriteRule:

## ban ptr.cnsat.com.cn
RewriteCond %{REMOTE_ADDR} ^202\.46\.(3[2-9]|[4-5][0-9]|6[0-3])\.
RewriteRule !^robots\.txt - [F]

Optionally, you can add this RewriteCond for the useragent they happen to be using at the moment:

RewriteCond %{HTTP_USER_AGENT} ^Mozilla/5\.0\ \(Windows\ NT\ 5\.1;\ rv:6\.0\.2\)\ Gecko/20100101\ Firefox/6\.0\.2

However, the IP ban is specific to the range owned by the company, so personally I wouldn’t bother using that useragent criteria. They could just change it at any time.

I did see they made several requests to robots.txt, but without a proper user agent identifying this bot as a crawler, your guess is as good as mine how to ban it in robots.txt, perhaps:

User-Agent: ptr.cnsat.com.cn
Disallow: /

Plesk Apache 404 error: File does not exist: /var/www/vhosts/default/htdocs/

By Wick

On April 6, 2016

In awful usability, Plesk, tips

You’ve no doubt discovered massive amounts of 404 errors in your main Apache error log that go something like:

File does not exist: /var/www/vhosts/default/htdocs/....

The requests may appear to be legitimate requests for page on the the primary virtualhost, but are returning 404 errors. Or, they may be crap requests to /var/www/vhosts/default/htdocs/phpMyAdmin etc made by script kiddies looking for vulnerabilities. Sound familiar?

Chances are you have SSL disabled for the domain in Plesk, & these requests to vhosts/default/htdocs/ are from HTTPS requests.

Plesk handles this use case in the most asinine way possible.

Since you have SSL disabled for your virtualhost, Plesk doesn’t route HTTPS requests to any virtualhost. Instead, it’s using the default host settings in /etc/httpd/conf/plesk.conf.d which can be something like:

<VirtualHost your_ip_here:7081 127.0.0.1:7081>
    ServerName "default-your_underscored_ip_here"
    DocumentRoot "/var/www/vhosts/default/htdocs"
    ....
</VirtualHost>

Little-known (to me) Plesk fact: For SSL requests, Apache listens to port 7081 when it’s running behind nginx, per /etc/httpd/conf.d/ssl.conf

How do you know this is going on? Enable servername & port logging in access_log so you can tell exactly what these requests are coming in as. To set that up, in /etc/httpd/conf/httpd.conf look for where your main access_log is defined, like:

 CustomLog logs/access_log combined

Then look for a LogFormat line that defines the log format nickname “combined”:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Add %v %V %p in there — right after %t is a good spot. Doing this adds the servername in two flavors, & the port of the original request. The servername helps you to determine which section of your Apache config is getting used, if you aren’t sure. The port shows the original — not mapped — port of the request. HTTPS starts out as a port 443 request so you’ll see that in the access log, not port 7081.

Restart Apache, either through Plesk, or apachectl restart. Then go tail -f access_log to watch the log with that additional data.

How do you fix how Plesk handles these SSL requests? In Plesk…

In Hosting Settings for your domain, check the box to enable (…yep) SSL.
In the Apache & nginx settings for your domain, under “Additional directives for HTTPS “, add this RewriteRule to redirect HTTPS requests to HTTP:

RewriteEngine On
RewriteRule .* http://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

That’s the best way I know how to fix this, anyway.

Any other suggestions? Normally you could set up a “black hole” entry, but I’m not sure how to overwrite the default Apache server settings, since server.conf is auto-generated by Plesk.

I’m off to bang some rocks together.

MediaTemple TrueSpeed CDN Control Panel Is Not What You Think

By Wick

On March 30, 2016

In awful usability, MediaTemple

This is MediaTemple’s sales page for their TrueSpeed™ CDN, which is a rebranded product operated by SiteLock.com:

Sounds great & what a deal, right? I thought so. Note the “integrated control panel”. Here’s what you actually get for the control panel:

Only having on/off & purge-all controls is not the right way to manage a CDN, in the same way that driving using only the ignition switch & yelling “get the fuck out!” to all passengers is not the right way to drive a bus.

Hopefully this guy from the TrueSpeed CDN sales page is designing a new control panel.

This guy on MediaTemple’s CDN sales page is no doubt hard at work designing a real control panel.

With MediaTemple’s TrueSpeed CDN there are no traffic stats or reporting, & no asset purge controls except for all-or-nothing.

I had been using Edgecast ProCDN (through MediaTemple) which has a real control panel with nice purge controls & very detailed traffic stats/reporting.

If you sign up for the same CDN through SiteLock.com & not via MediaTemple, you get a much more capable control panel with all the features you’d expect for managing a CDN, but you’ll probably pay more than $30/month.

I think MediaTemple’s TrueSpeed CDN is still a good deal & no doubt they have plenty of customers who don’t need CDN stats or any real controls.

But MediaTemple should be far more upfront about what customers actually get for the “integrated control panel”. Maybe they should call it what it is: “integrated on/off/purge buttons”.

In lieu of MediaTemple being upfront about it, now you know!

Migrating MediaTemple’s GridServer (GS) to Dedicated Virtual (DV) VPS

By Wick

On March 28, 2016

In MediaTemple, tips

I recently moved lots of websites off MediaTemple’s GridServer (gs) platform to their Dedicated Virtual (dv) platform. I’ve kind of abused Grid Server for the past 12 years, but finally the overage fees caught up.

I went with the ~~Plesk 12.5/CentOS6 hosting option~~ one year later, I upgraded DV accounts & did this all over again for Plesk Onyx 17/CentOS7. The standalone DV server was tempting but I don’t quite know enough about Linux admin to go that route.

Here were some of the bigger migration issues going from MediaTemple’s GridServer (gs) to Dedicated Virtual/VPS (dv) service.

Plesk doesn’t come with root enabled.

Chances are you’ll need to enable root, which is done through AccountCenter. At first I tried not to enable root but there’s just too many fixes/workarounds that you can’t do without root access.

Plesk creates websites with the web directory set to httpdocs/

…Whereas GridServer uses html/ for the top-level web directory. I prefer the shorter “html”, & I really don’t like changing things like this in Git which we use for development. Luckily it’s easy to change how Plesk works here.

THE FIX: In Plesk, click on your domain & click Hosting Settings. Change Document Root to html/. Plesk will create html/ but leaves httpdocs/ so you’ll have to delete httpdocs/ manually.

Note: for the rest of this post I’ll continue to reference “httpdocs/” for consistency.

Plesk puts cgi-bin/ as a subdirectory of httpdocs/

In other words, Plesk uses httpdocs/cgi-bin/. GridServer had cgi-bin/ at the same level as html/. So basically if you’ve used Git for years like we have, you can either move the folder in Git & hope the history stays or change how Plesk works. Moving the folder & keeping the Git history is possible, but messing with Git gives me the creeps.

THE FIX: Create cgi-bin/ where you want it & set permissions using chmod 755 cgi-bin/. ~~It’s probably good to follow Plesk convention where top-level web directories are assigned to the psacserv group so chgrp psaserv cgi-bin/ too.~~ This worked with Plesk 12 under CentOS6, but Plesk Onyx under CentOS7 requires cgi-bin/ to stay assigned to the psacln group or else Perl scripts running under Apache mod_cgi will return the 500 error “End of script output before headers” (thank you MediaTemple CloudTech Supervisor Gary R for figuring that out).

~~Then in Plesk under the domain, click Apache & nginx Settings. Scroll to the “Additional Apache Directives” section and add:~~

ScriptAlias "/cgi-bin/" "/var/www/vhosts/domain.com/cgi-bin/"

I found out Plesk directly supports moving cgi-bin to the www-root level & doesn’t need a ScriptAlias added manually. Run these commands:

/usr/local/psa/bin/domain -u domain.com -cgi-mode www-root
/usr/local/psa/admin/bin/httpdmng --reconfigure-domain domain.com

In Plesk under Hosting Settings, now you’ll see a select box:

Finally click Hosting Settings & disable “Perl”. It’s not what you think. This Plesk option actually disables mod_perl, & does not disable “regular” mod_cgi Perl.

Mod_perl is very efficient but typically requires porting over your Perl scripts, or else it can wreak havoc, as described in pretty much the entire porting guide.

Fun fact: The Plesk “Perl” option isn’t tied to the cgi-bin/ location. It only updates <Directory> options for the httpdocs/ folder — you can watch Plesk change the setting in vhosts/system/domain.com/conf/httpd_ip_default.conf — so if you’ve moved cgi-bin/ out of httpdocs/ to the www-root level, it won’t actually matter whether or not you disable the “Perl” option in Plesk.

Last step is to delete the now-defunct httpdocs/cgi-bin/ directory.

Plesk creates websites directories with a bunch of default & testing files.

THE FIX: SSH to your account, change to the website directory & rm -rf html/*. I’m assuming you know enough about Linux to realize this deletes everything in html/ so make sure you haven’t uploaded the website files you want to keep, yet.

Plesk sets up subdomains as subdirectories of the parent website.

In other words, Plesk creates domain.com/httpdocs/subdomain.com/. This sucks. Luckily, easy fix. When adding the subdomain site in Plesk, pick the primary domain option rather than the subdomain option. Ignore the “www” prefix. The subdomain site will work fine & you’ll just have an extra domain alias for www.subdomain.domain.com in nginx & Apache that won’t be used.

Plesk forces you to create a different user for each web site.

The files all get assigned to that user, & group psacln. The httpdocs/ directory is also assigned to the same user, & group psaserv. So, users can’t browse each others’ web folders. By default, with Plesk you can’t have an FTP account that has access to multiple web sites, which is what I needed since we use one Git repository deployed via BeanStalk to manage ~30 very similar websites with shared resources.

THE FIX: First su root & grant the psacserv group to ssh/ftp users that you want to have access to the full range of web directories (but not root privileges): usermod -a -G psaserv username — this just adds psacserv as a secondary group & the primary group stays psacln — so any new files created via these accounts will still fit the Plesk convention.

I set up one user for SSH & FTP with access to all website directories. If you go that one-user route rather than the Plesk-created users for each website, reset directory & file ownership to your super-web-user with: chown -R username:psacln * That chown operates recursively starting from the current directory, so run that only from within html/ and cgi-bin/ because otherwise it will try to reset ownership on your system files, log files & other non-public stuff that should probably stay assigned to root.

Similarly depending on how you upload/migrate your web files, you may need to set correct permissions on files & directories within html/:

find . -type f -exec chmod 644 {} +
find . -type d -exec chmod 755 {} +

(Execute these commands only from within the directories you want affected.)

For cgi-bin/ you’ll probably want to chmod 755 script files rather than 644, so your scripts (Perl in the example below) have world-read/execute permissions:

find . -type f -name '*.pl' -exec chmod 755 {} +

Final step is have each web property execute CGI scripts as your one user. For each domain’s Apache & nginx settings, in “Additional directives for HTTP “, add:

SuexecUserGroup "username" "psacln"

Plesk sets each FTP user’s home directory to within a web directory.

Again we use Git/BeanStalk with several repos that manage groups of similar websites, so I needed an FTP login for BeanStalk to have access to the vhosts/ directory where all the website directories are located.

THE FIX: You can change the home directory for your FTP user in bash through the normal way, & Plesk doesn’t care:

usermod -d /var/www/vhosts/ username

Plesk runs all cron jobs as root.

Any files that your cron job creates get root user permissions & are not available to the web server users. Plesk shows the cron user as root, but it’s not anything you can change. Yes, this is lame.

THE FIX: Have each cron run a shell script that uses su, sudo or runuser to switch to the web-level user first. For example, have the cron run a shell script with:

/sbin/runuser username -s /var/www/vhosts/domain.com/cgi-bin/somescript.pl

Or you could have each cron job command (each entry in Plesk) start with one of the user-switching methods. But since cron commands become the email subject for status notifications, the subject line would start with “runuser” etc all the time.

Plesk doesn’t serve web font files correctly by default.

We serve css & web fonts from a different domain than the main website, so we need to have an access control header to allow that. The standard code is:

<FilesMatch "\.(ttf|otf|eot|woff)$">
<IfModule mod_headers.c>
    Header set Access-Control-Allow-Origin "*"
</IfModule>
</FilesMatch>

This worked great on GridServer in .htdocs, but didn’t have any effect after we migrated to Plesk. As far as I can tell, mod_headers is enabled by default so that’s not the problem. Eventually I noticed the response header for the web font files was nginx & not Apache. I got it working by going into Apache & nginx Settings for the domain & disabling “Smart static files processing”. I think what’s happening is without the second box checked that defines specific extensions, nginx serves font files by default so the request never makes it to Apache.

Even if it works for you to have nginx serve your font files, I found nginx serves them as text/plain so under the MIME types section at the top of the same screen, it might help to add:

font/ttf .ttf
font/opentype .otf
application/font-woff .woff
application/vnd.ms-fontobject .eot

MediaTemple warns that you’ll lose all your (gs) IMAP email as soon as you click Point At Another Server in AccountCenter.

FIX: True, but you can still switch your site over to (dv) without losing old email. Instead of clicking Point At Another Server, go about adding/migrating your site & ignore Plesk’s DNS warning. Then when you’re ready to switch web traffic over, in AccountCenter, edit DNS & change the A records from your (gs) IP to your (dv) IP.

Make sure you have email set up in Plesk before switching A records, because although you won’t lose your old email, new email will start going to your (dv) accounts.

This way although your domain gets switched over to (dv), you won’t lose your old (gs) email because you can still access it through your gridserver domain (sXXXXXX.gridserver.com) or xxxx-xxxx.accessdomain.com. If you’re not sure what these are, in AccountCenter, click on Server Guide for your grid server & look at the Email section.

Incidentally migrating IMAP email is really easy with Thunderbird. Add your new IMAP account, select all the folders in your old account, drag over to new & you’re done. Then do the Point At Another Server thing.

Plesk doesn’t come with cpan or any Perl modules (or gcc).

I’m a dinosaur, I guess. The “yum” installer does have a bunch of Perl modules available but is missing lots of the common modules I use. Cpan works fine for me. Luckily you can use yum install install CPAN & gcc:

yum install perl-CPAN
yum install gcc

BEFORE YOU RUN CPAN: the yum-installed CPAN adds some environment variables to ~root/.bashrc which sets cpan to install Perl modules under root rather than in one of the @INC locations, so fix that by deleting the added lines in .bashrc & also delete ~root/perl5/ and ~root/.cpan/. Then run cpan setup & picking “lib::local” should put Perl modules into a web-accessible lib/ directory.

Using cpan to install GD doesn’t work.

Cpan aborts with an error message: Could not find gdlib-config in the search path. Please install libgd 2.0.28 or higher. I’m not a sysadmin & didn’t want to mess with this. Easy fix — use “yum” to install GD instead:

yum install gd

Mod_perl under CentOS7 behaves inconsistently compared to the CentOS6 Perl CGI environment.

With CentOS7 the only way I could get Perl scripts working via Apache was to use mod_perl, & with mod_perl the output/success of Perl scripts is inconsistent due to the way mod_perl compiles our Perl scripts that weren’t specifically coded for mod_perl. Generally the first request is successful while subsequent requests fail.

The big problem I ran into was when global subroutines defined in our custom modules are loaded via require("somemodule.pm"), the error log shows “undefined subroutine”. I believe this is why (in my case, it’s scenario 3).

Added bonus, under mod_perl, relative paths don’t work. I could work around that if I had to, but the custom module problems are a deal-killer.

Still troubleshooting. This sucks.

The MT sales rep set up the (dv) account with a made-up domain.

The issue here is my rep felt they needed to put down something temporary while I got started with the site migration. I don’t think the fake domain was a good idea because it caused problems. The second time I went through this DV account switch, the rep used my primary domain for setup & everything worked just fine. The fake domain isn’t just for a name in AccountCenter, which is what tech support first told me.

The fake domain gets set as Reverse DNS for your new (dv) service IP address, which can cause your IP to get blacklisted for email. Minor detail, yep.

THE FIX: As soon as you get your primary site migrated over, fix the Reverse DNS (AccoutCenter, DNS section). Then change the primary AccountCenter domain for your (dv) account to your real domain – that’s hidden in AccountCenter under Server Guide.