Correct matching rules
The longest matching pattern wins, Allow breaks a tie against Disallow, and both wildcard and end-anchor syntax behave the way the protocol specifies.
Robots.txt Tester
Load a live robots.txt or paste your own, pick a crawler, and check a batch of URLs against the real matching rules Google uses.
Group selection, longest-rule-wins, wildcards, and end anchors follow the published protocol.
Paste mode needs no request, so unpublished rules can be tested before they go live.
- Batch test up to fifty paths.
- Deciding rule named with its line.
- Syntax problems flagged per line.
How to test URLs against robots.txt
Load or paste the file, choose the crawler, list the paths you care about, and read the deciding rule.
- 1
Load a live file or paste your own
Enter any URL from a domain to pull the file from its root, since robots.txt only ever lives there. Paste mode is the better choice while you are still drafting rules, because it lets you edit the file and retest without publishing anything.
- 2
Choose the crawler to simulate
Googlebot, Bingbot, GPTBot, ClaudeBot, Applebot, and others each pick their own group, and a site can grant one crawler access it denies another. Custom lets you type any token when you need to check a niche or internal bot.
- 3
List the URLs to test
Add one path or full URL per line. Full URLs are reduced to their path and query automatically, which makes it easy to paste straight from a crawl export, a sitemap, or a Search Console coverage report.
- 4
Read the deciding rule
Each result says allowed or blocked and names the rule and line number that produced it. The file panel highlights those lines, so a rule that blocks more than intended is easy to spot and correct.
Twelve crawler presets
Googlebot and its image and news variants, AdsBot, Bingbot, DuckDuckBot, Applebot, GPTBot, ClaudeBot, PerplexityBot, and the Facebook fetcher are all one click away.
Batch URL testing
Paste up to fifty paths at once to check a whole section, a sitemap sample, or a list of pages flagged as excluded in a coverage report.
Paste mode with no request
Draft rules can be pasted and tested entirely in the browser, which means a blocking mistake gets caught before the file is ever deployed.
Syntax and policy linting
Missing colons, rules declared before any user-agent line, relative sitemap URLs, unsupported directives, byte order marks, and the 500 KB ceiling are all reported per line.
Groups, sitemaps, and report
Every user-agent group is summarised with its rule counts, sitemap entries are listed as links, and the whole audit downloads as a plain text report.
Robots.txt Tester privacy and processing
These notes separate the one server request used to load a live file from the matching work that happens locally, and cover what remains after the tab closes.
Server fetch, browser matching
Loading a live file needs one server request because browsers cannot read another origin directly. Parsing, group selection, wildcard expansion, and every allow or block decision then run inside the open page with no further requests.
No stored files
Pasted rules, tested paths, and results stay in current tab state, and fetched files sit only in a short shared cache that clears itself. Nothing is written to an account.
Robots.txt Tester account access
Loading a file, pasting your own rules, switching crawlers, and downloading the report all work without an account or any sign-in prompt.
Before You Use Robots.txt Tester
Blocking a URL from crawling is not the same as keeping it out of search results, and the difference matters.
Crawling
A disallowed URL can still be indexed when other pages link to it, usually with no description shown.
To keep a page out of results, allow the crawl and serve a noindex robots meta tag on the page itself.
Compliance
Only well behaved crawlers read robots.txt at all, and nothing in the file enforces anything.
Never rely on a disallow rule to protect private data, admin paths, or anything that needs real access control.
Blocking
Disallowing scripts, stylesheets, or images can stop Google rendering a page the way visitors see it.
A blocked page also cannot show its noindex tag, which is why the two directives should never be combined.
Scope
Rules apply per protocol, host, and port, so the file on a subdomain governs only that subdomain.
Paths are matched case sensitively, and the file itself must sit at the site root to be read at all.
Limits
Robots.txt Tester loads files up to 512 KB from a site root, and pasted files are matched entirely in the browser with no request leaving the tab.
Loading a live file is rate limited per address, while pasted content can be edited and re-tested as often as you like because nothing is fetched.
Questions about Robots.txt Tester
Does a Disallow rule remove a page from Google?
No. It stops the page being crawled, but the URL can still appear in results when other sites link to it. Removing a page from search requires a noindex tag on a page Google is still permitted to crawl.
Which rule wins when Allow and Disallow both match?
The rule with the longest path pattern wins. When two matching rules are exactly the same length, Allow takes precedence over Disallow. This tool applies that order and names the winning rule on every result.
What do the asterisk and dollar characters do?
An asterisk matches any run of characters, so Disallow: /*.pdf covers every PDF path. A dollar anchors the end of the URL, so Disallow: /*.php$ matches a path ending in .php but not one with a query string after it.
Can I block AI crawlers with robots.txt?
You can add groups for tokens such as GPTBot, ClaudeBot, and PerplexityBot, and the operators that publish those names state they honour the rules. Enforcement still depends entirely on the crawler choosing to comply.
Why does Googlebot ignore my Crawl-delay line?
Google has never supported Crawl-delay and ignores it completely. Bing and Yandex do read it. To influence how fast Google crawls, use the crawl rate settings in Search Console instead.
Can I test rules that are not published yet?
Yes. Switch to paste mode, drop in your draft file, and everything is parsed and matched inside your browser with no request sent anywhere. It is the safest way to catch a rule that blocks more than you intended.
4.3 (409 ratings)