Rules are case-sensitive.
Directive names and crawler tokens ignore case, but /private/ and /Private/ are different paths. The tester includes query strings when matching.
Robots.txt Validator
Free toolReview your crawler instructions and test the paths they allow or block.
Specific rules take priority.
The most specific matching crawler groups are combined. The longest matching rule wins; Allow takes precedence when matching rules have equal length.
Crawling isn’t indexing.
Blocking a path doesn’t guarantee its URL disappears from search. Robots.txt is public crawler guidance, not a way to protect private data.
Tests use Google-style group selection and common robots.txt path rules, informed by RFC 9309 ↗ and Google’s documentation ↗. Other crawlers may interpret extensions differently. A test result describes parsed rules; it does not fetch the tested page, verify indexing, or model cached rules and HTTP error policies.
ABOUT THIS FREE TOOL
How to validate and test robots.txt
Use this robots.txt validator to inspect crawler instructions and test how they apply to a path. Paste your rules or fetch a website’s root robots.txt, review the findings, and try a URL against a crawler’s matching rules.
Paste your robots.txt content, or enter a public website URL to fetch its root robots.txt file.
Review syntax findings, crawler groups, and sitemap declarations.
Enter a URL and user agent in the access tester to see whether the parsed rules allow or block that path.
COMMON QUESTIONS
Robots.txt Validator: useful answers
What does a robots.txt validator check?
It checks common directive syntax, user-agent groups, Allow and Disallow rules, and sitemap declaration URLs. The access tester identifies the matching rule for the URL and user agent you enter. Sitemap declarations are checked for syntax; their files are not fetched.
Can I test whether Googlebot is blocked?
You can test a URL using a Googlebot user agent against the parsed rules. The result models Google-style group selection and path matching. It does not contact Googlebot or account for cached rules, file-fetching failures, or actual indexing status.
Does Disallow remove a page from Google?
Not necessarily. Disallow controls crawling for matching crawlers; a blocked URL may still be indexed. Robots.txt is also public and should not be used to protect private information.
Where should my robots.txt file be placed?
For a website origin such as https://example.com, the file belongs at https://example.com/robots.txt. Rules are scoped to their protocol, hostname and port. A file under a subdirectory does not serve as that origin’s root robots.txt.