How to use
- Build rules with the form, or paste an existing file (open yoursite.com/robots.txt and copy it).
- Enter a URL path and a user agent to test.
- The verdict shows the rule that decided it.
Worked example
With Disallow: /private and Allow: /private/press, the path /private/press/kit.pdf is allowed (the longer match wins).
Supported formats and limits
| Input | Builder form with presets (allow all, block all, block AI crawlers), robots.txt text or file |
|---|---|
| Output | robots.txt, Allow/Disallow verdict with the deciding rule and line, Line-numbered warnings |
| Engine | RFC 9309 parser and longest-match evaluator |
Limitations
- robots.txt controls crawling, not indexing; use noindex to keep pages out of results.
- It is a request, not access control: crawlers that ignore it are not blocked. The AI-crawler preset lists tokens published by their operators as of October 2026; new ones appear.
- Crawl-delay and other non-standard lines are reported but not used in the verdict; individual crawlers may still differ from RFC 9309 in edge cases.
Questions
Which rule wins when Allow and Disallow both match?
The longest matching rule wins, and Allow wins a tie, as RFC 9309 specifies. With Disallow: /private and Allow: /private/press, the path /private/press/kit.pdf is allowed.
Does Disallow keep a page out of search results?
No. robots.txt controls crawling, not indexing, and it is a request rather than access control. Use a noindex directive to keep a page out of results, and protect private content with authentication.
Guides
Privacy
Runs on your device. Files and text are processed in this browser tab and are not uploaded.
See the privacy policy for how toolsdocks handles data.