SEO: The subtle differences of robots.txt disallow vs meta robots no-index

The robots.txt file and <meta name="robots"> HTML tag can be used to control the behavior of search engine crawlers. Both have different effects.

robots.txt

Marking a URL path as "disallowed" in robots.txt Show archive.org snapshot tells crawlers to not access that path.

robots.txt is not a guarantee for exclusion from search engine results.

A "disallowed" URL might be known from an external link, and can still be displayed for a matching search.
Example: even if /admin is disallowed in robots.txt, /admin/some-page.html might turn up when searching for "some page".

robots meta tag

Setting a robots meta tag Show archive.org snapshot as "noindex" tells search engines to not list that path in their search results.

Crawlers can only see meta tags on paths that they can access.

If a path is "disallowed" in robots.txt, crawlers will not access it and and never see its robots meta tags.

What does this mean?

• URLs from "disallowed" paths may appear on search results.
  • Crawlers will not show their contents, only that the URL exists.
  • This is usually not an issue, since such paths need to be referenced from somewhere first. It might be undesirable for them to appear in search results, though.
• If you want to avoid that, allow (or rather: don't disallow) that path in robots.txt and use a noindex robots meta tag.
  • You can also use that to (at the next crawling cycle) remove an indexed URL from a crawler's index since it will then see meta tags.