Read more

Exclude your staging site from Google with robots.txt and not shoot yourself in the foot

Henning Koch
September 09, 2010Software engineer at makandra GmbH

If you want to exclude your staging site from Google using robots.txt without running the risk to forget deleting the file once you go live, name the file robots.exclude.txt instead.

Illustration online protection

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

  • Prevents you from data breaches and liability risks
  • Upgrade at your own pace
  • Works with modern Rubies
Read more Show archive.org snapshot

In your Apache Vhost config, rewrite requests for the staging server only:

RewriteEngine On
RewriteCond %{HTTP_HOST} ^staging\.project\.com$
RewriteRule ^robots\.txt$ /robots.exclude.txt

Your robots.exclude.txt looks like this:

# This file is returned for /robots.txt on staging servers
User-agent: *
Disallow: /

Important Note: If your setup is incorrect and /robots.txt is not accessible, it means there is no protection at all!

Posted by Henning Koch to makandra dev (2010-09-09 09:51)