What is a robots meta tag? Controlling indexing and previews
Explain noindex and preview controls, distinguish page tags, headers and robots.txt and verify crawler access.
Understand the concept
Robots meta tags express indexing or preview rules; HTTP headers can carry related controls. noindex differs from robots.txt crawling rules. Crawlers must read directives, so blocking access can prevent discovery of tags. Check live responses and processing, not only edited source.
Indexing and crawling controls have different roles
Google documents noindex and previews separately from robots.txt paths. Blocked URLs may still be known through links. Allowing fetches can let noindex be read. Support varies by crawler. Search directives are not universal permissions or security boundaries; protect sensitive content through access control.
Inspect live tags and headers on the same page
For a public test page intended to be unindexed, inspect tags, X-Robots-Tag and accidental crawl blocks. Check live caches, and where available use index tools for processed state. One response does not prove instant index changes. No controls or removals were changed here. Verify preview semantics to avoid accidental noindex.
Check accidental inheritance and conflicts after template updates
Templates can spread test noindex to production. Version rules, keep recovery and inspect added server or CDN headers. Recheck live and search states. Align withdrawals, links and sitemaps with intended controls. Report configured versus processed status separately rather than guaranteeing immediate appearance or removal.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.