What is robots.txt? Crawl rules are not confidentiality controls
Explain robots exclusion versus indexing and inspect host scope, path matching and responses.
Understand the concept
robots.txt communicates crawl rules to compliant crawlers. It is not authentication and does not guarantee blocked URLs never appear in results. Crawl blocking differs from noindex. Check target agents, hosts and live responses; a local Disallow does not prove compliance.
Protocol directives do not stop arbitrary visitors
RFC 9309 defines the protocol and Google describes Search behavior. Rules do not delete pages everywhere and noncompliant visitors can ignore them. Use actual access controls for secrets. Blocking can prevent reading noindex. Check agent groups and paths instead of copying broad rules across a site.
Test allowed and blocked scope for specific URLs
For a test directory, inspect the live root file and both allowed and blocked public URLs. Check path and case matching and server-error behavior under the crawler’s rules. No robots file or crawl configuration was changed here. For indexing goals, inspect readable noindex rather than piling on Disallow rules.
Retain recovery and external observation for rule updates
Recheck scope after templates or domain changes, retain old versions and important allowed paths and observe actual crawling. Uploading is not external processing. Include server and CDN diagnosis. Sitemaps do not override crawl blocks. State verified rules and unobserved behavior; robots.txt is a cooperative protocol, not permissions or ranking assurance.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.