ROBOTS.TXT VALIDATOR / A PRACTICAL GUIDE
Test robots.txt rules without guessing
Check syntax and test specific paths against the crawler groups you intend to use.
Use the root file
A site’s robots.txt belongs at the root of its origin. Fetch the public site or paste your draft. Different schemes, hostnames and ports can have different files.
Test important paths
Try a public content page and a path you intend to disallow. Select the relevant user agent. Review the matching rule, rather than interpreting the list of directives in isolation.
Keep access control separate
Robots.txt is public guidance for crawlers. It does not protect documents, authenticate users or reliably remove a URL from search results. Protect private pages with real access controls.
An example to work from
User-agent: * Allow: / Disallow: /internal/ Sitemap: https://example.com/sitemap.xml
Before you finish
Test /articles/example and /internal/example. A sitemap declaration is checked as a URL; the validator does not fetch the sitemap itself.