robots.txt generator & tester
Build a robots.txt or paste one to test whether a path is allowed for a given user-agent. Runs entirely in your browser — no live fetches.
~~~
Honest note: robots.txt is advisory. Well-behaved crawlers respect it; malicious bots ignore it. It is not access control — use auth and firewall rules for that.
AI crawler list (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, etc.) — verify vendor docs periodically.
Disallow paths
Allow paths
Generated robots.txt
robots.txt
~~~
About this tool
robots.txt tells crawlers which paths they may fetch. The tester implements longest-match precedence with * wildcards and $ end anchors per RFC 9309 / Google interpretation. Allow wins on equal-length ties.
~~~