Free robots.txt Tester
Test URLs against a live or pasted robots.txt for Googlebot, Bingbot, GPTBot, ClaudeBot and other crawlers. See allowed or blocked and the line that decides it.
Loading tool…
How it works
Fetch or paste
Enter a site to fetch its /robots.txt, or paste a draft to test it before it goes live.
Add paths, pick a crawler
Up to 20 paths or full URLs. Test one crawler, your own product token, or all of them at once.
Read the result
Each row shows allowed or blocked, the deciding line and the group that applied. Lines that decide a result are marked in the file.
How are rules matched?
As RFC 9309 says. The crawler's group is the one whose user-agent matches its product token, case-insensitive; groups with the same token are merged, and the * group applies only when none match. Within the group the longest matching path wins, and Allow wins a tie. * matches any characters; $ marks the end of the URL.
What if robots.txt is missing or fails?
A 4xx response other than 429, including 401 and 403, means no robots.txt: everything is allowed. A 5xx response or no response means everything is disallowed. Google treats 429 as a server error and more than 5 redirects as a 404. After 12 hours of server errors Google uses its last good copy for up to 30 days, or allows everything if it has none.
Is my robots.txt sent anywhere?
Pasted text is tested in this tab and not sent anywhere. In fetch mode our server downloads the site's /robots.txt (up to 512 KiB) and the test runs in this tab.
Crawl the pages you are allowed to.
Browser Use agents open pages in a real browser, including dynamic ones, and return structured data.