The robots.txt file and <meta name="robots">
HTML tag can be used to control the behavior of search engine crawlers. Both have different effects.
Marking a URL path as "disallowed" in robots.txt Show archive.org snapshot tells crawlers to not access that path.
robots.txt is not a guarantee for exclusion from search engine results.
A "disallowed" URL might be known from an external link, and can still be displayed for a matching search.
Example: even if/admin
is disallowed in robots.txt,/admin/some-page.html
might turn up when searching for "some page".
Setting a robots meta tag Show archive.org snapshot as "noindex" tells search engines to not list that path in their search results.
Crawlers can only see meta tags on paths that they can access.
If a path is "disallowed" in robots.txt, crawlers will not access it and and never see its robots meta tags.
robots.txt
and use a noindex
robots meta tag.