Posted 19 days ago. Visible to the public. Linked content.

SEO: The subtle differences of robots.txt disallow vs meta robots no-index

The robots.txt file and <meta name="robots"> HTML tag can be used to control the behavior of search engine crawlers. Both have different effects.

robots.txt

Marking a URL path as "disallowed" in robots.txt Archive tells crawlers to not access that path.

robots.txt is not a guarantee for exclusion from search engine results.

A "disallowed" URL might be known from an external link, and can still be displayed for a matching search.
Example: even if /admin is disallowed in robots.txt, /admin/some-page.html might turn up when searching for "some page".

robots meta tag

Setting a robots meta tag Archive as "noindex" tells search engines to not list that path in their search results.

Crawlers can only see meta tags on paths that they can access.

If a path is "disallowed" in robots.txt, crawlers will not access it and and never see its robots meta tags.

What does this mean?

  • URLs from "disallowed" paths may appear on search results.
    • Crawlers will not show their contents, only that the URL exists.
    • This is usually not an issue, since such paths need to be referenced from somewhere first. It might be undesirable for them to appear in search results, though.
  • If you want to avoid that, allow (or rather: don't disallow) that path in robots.txt and use a noindex robots meta tag.
    • You can also use that to (at the next crawling cycle) remove an indexed URL from a crawler's index since it will then see meta tags.

Your development team has a full backlog of feature requests, chores and refactoring coupled with deadlines? We are familiar with that. With our "DevOps as a Service" offering, we support developer teams with infrastructure and operations expertise.

Owner of this card:

Avatar
Dominik Schöler
Last edit:
15 days ago
by Arne Hartherz
About this deck:
We are makandra and do test-driven, agile Ruby on Rails software development.
License for source code
Posted by Dominik Schöler to makandra dev
This website uses short-lived cookies to improve usability.
Accept or learn more