All tools

robots.txt tester

Enter a domain and a path, and see whether a crawler is allowed to fetch it, with the rule that decided it and the whole file it came from.

example.com or https://example.com. We add /robots.txt, so any path you paste here is ignored.
A leading slash is added if you leave it off.
Pick one from the list or type any crawler name.

We fetch robots.txt from our server and nothing else. The path you test is never requested.

Fetching the file from the other site can take a few seconds.

How a crawler decides

A crawler does not stop at the first matching line. It compares every rule in its group and takes the one with the longest path pattern, so Allow /blog/public beats Disallow /blog for a URL that starts with both. When two matching patterns are exactly the same length, Allow wins.

A crawler reads exactly one group. It takes the group whose user agent names it most specifically, falls back to the group named with an asterisk, and follows nothing at all if neither is there. Rules in the other groups are invisible to it, which is why a rule that looks right often turns out to sit in a group the crawler never reads.

Why this page exists

Google removed the robots.txt tester from Search Console, and what replaced it reports far less, saying only whether one URL is blocked, on a property you have verified, without naming the rule. Most of the time the question is not whether a path is blocked but which line blocked it, and on a site somebody else runs.

A rule in robots.txt stops a crawler fetching a URL. It does not remove that URL from a search result, and it does not hide the page from anyone who has the address. Anything that must stay private needs authentication, and anything that must stay out of the results needs a noindex tag on a page a crawler is still allowed to read.

Questions about robots.txt

Everything is allowed. A crawler that gets a 404 for robots.txt treats the site as having no restrictions at all. A server error is different, because while robots.txt cannot be fetched a crawler holds off entirely.

Google retired the robots.txt tester that used to live in Search Console. The URL Inspection tool still says whether a single URL is blocked, but only for a site you have verified, and it does not show you the rule that decided it. This page works on any public site and shows the rule.

Yes. An asterisk stands for any run of characters and a dollar anchors the pattern to the end of the URL, so Disallow /*.pdf$ blocks every PDF. Both are extensions to the original standard, and the major crawlers support them.

No. A disallowed URL can still be indexed if other pages link to it, because the block stops the crawl rather than the listing. Use a noindex meta tag or an HTTP header on a page that must stay out of the results, and leave it crawlable so the tag can be read.

Report the bug, not a description of it

Session Replay captures the page, the console, the network log and the browser details in one link, so the next person to look at a broken page already has the whole thing in front of them.

Install ExtensionHow It Works