BROWSER USE

- Browser Use Agents: give Browser Use a task and receive completed work. API V4 is current for new integrations.
- Browser Infrastructure: connect your agent or automation to managed browsers through SDK, REST, or CDP. Starts at $0.02/browser-hour.
- Developer tools: Open Source, Browser Harness, SDK, and MCP support the two products above.

[Developer Index](https://browser-use.com/index.md)
[Product Map](https://browser-use.com/llms.txt)
[Full Product Context](https://browser-use.com/llms-full.txt)
[Pricing](https://browser-use.com/pricing.md)
[Cloud Docs](https://docs.browser-use.com/cloud/quickstart)
[Open Source Docs](https://docs.browser-use.com/open-source/introduction)

---

# robots.txt Tester, Checker and Validator

## Free robots.txt Tester

Test URLs against a live or pasted robots.txt for Googlebot, Bingbot, GPTBot, ClaudeBot and other crawlers. See allowed or blocked and the line that decides it.

## How it works

### Fetch or paste

Enter a site to fetch its /robots.txt, or paste a draft to test it before it goes live.

### Add paths, pick a crawler

Up to 20 paths or full URLs. Test one crawler, your own product token, or all of them at once.

### Read the result

Each row shows allowed or blocked, the deciding line and the group that applied. Lines that decide a result are marked in the file.

### How are rules matched?

As RFC 9309 says. The crawler's group is the one whose user-agent matches its product token, case-insensitive; groups with the same token are merged, and the \* group applies only when none match. Within the group the longest matching path wins, and Allow wins a tie. \* matches any characters; $ marks the end of the URL.

Source: [RFC 9309, section 2.2](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2) · [Google: how Google interprets robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)

### What if robots.txt is missing or fails?

A 4xx response other than 429, including 401 and 403, means no robots.txt: everything is allowed. A 5xx response or no response means everything is disallowed. Google treats 429 as a server error and more than 5 redirects as a 404. After 12 hours of server errors Google uses its last good copy for up to 30 days, or allows everything if it has none.

Source: [RFC 9309, section 2.3.1](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1) · [Google: robots.txt status codes](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#http-status-codes) · [Google: 429](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes)

### Is my robots.txt sent anywhere?

Pasted text is tested in this tab and not sent anywhere. In fetch mode our server downloads the site's /robots.txt (up to 512 KiB) and the test runs in this tab.

## Crawl the pages you are allowed to.

Browser Use agents open pages in a real browser, including dynamic ones, and return structured data.

[See AI web scraping](https://browser-use.com/web-scraping)

SourceFetch from URLPaste robots.txtCrawlerAll crawlersGooglebotBingbotGPTBotClaudeBotPerplexityBotGoogle-ExtendedCustom product token

Website URLPaths or URLs to test, one per line (up to 20)

TestLoad example