Posted over 11 years ago. Visible to the public. Linked content.

Exclude your staging site from Google with robots.txt and not shoot yourself in the foot

If you want to exclude your staging site from Google using robots.txt without running the risk to forget deleting the file once you go live, name the file robots.exclude.txt instead.

In your Apache Vhost config, rewrite requests for the staging server only:

RewriteEngine On RewriteCond %{HTTP_HOST} ^staging\.project\.com$ RewriteRule ^robots\.txt$ /robots.exclude.txt

Your robots.exclude.txt looks like this:

# This file is returned for /robots.txt on staging servers User-agent: * Disallow: /

Important Note: If your setup is incorrect and /robots.txt is not accessible, it means there is no protection at all!

Flaky tests are tests that sometimes fail for no obvious reason. They are the plague of many end-to-end (E2E) test suites that automate the browser through tools like Capybara and Selenium.

Join our free training event and learn to fix any flaky test suite, even in large legacy applications.

Owner of this card:

Henning Koch
Last edit:
9 months ago
by Henning Koch
google, robots.beta.txt
About this deck:
We are makandra and do test-driven, agile Ruby on Rails software development.
License for source code
Posted by Henning Koch to makandra dev
This website uses short-lived cookies to improve usability.
Accept or learn more