Robots.txt Generator and Tester

How a crawler reads robots.txt

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules. A crawler picks exactly one group: the one whose user agent matches its own name most specifically. Only when no group names it does it fall back to the User-agent: * group.

This is the part people get wrong most often. If you write a User-agent: Googlebot group with one rule, Googlebot ignores everything under *, including your /wp-admin/ block. Groups do not add up across names. They only merge when the same name appears in two groups, in which case Google combines their rules.

Some Google crawlers have a fallback chain. Googlebot-Image and Googlebot-News obey their own group if it exists and the Googlebot group otherwise. Directive names and user agent names are case-insensitive. Paths are not: /Admin/ and /admin/ are different rules.

Which rule wins when several match

Google does not use the first matching rule. It uses the most specific one, measured by the length of the rule path. When an Allow and a Disallow rule are equally long, Allow wins. Two characters have special meaning: * matches any sequence of characters, and $ at the end anchors the rule to the end of the URL.

The last row catches people out. Disallow: /blog also blocks /blog-tips/ and /blogroll. Write /blog/ when you mean the folder. The tester above shows every rule that matches a path, longest first, so you can see why the winner won.

Blocking AI crawlers without leaving AI search

AI companies now run separate crawlers for training and for search. Blocking the training crawler keeps your pages out of future model training. Blocking the search crawler removes you from that product's answers and links, which is usually not what a business wants.

Google-Extended does not change how Googlebot crawls or ranks you, and blocking it does not remove you from AI Overviews in Search. The "Block AI training bots" preset blocks the training crawlers and leaves the search ones on Default. Robots.txt is a request, not a wall: reputable companies honour it, and anything that ignores it needs blocking at the server or firewall.

"Blocked by robots.txt" and "Indexed, though blocked by robots.txt"

Both statuses appear in the Search Console Page indexing report, and they mean different things.

Blocked by robots.txt is a not-indexed reason. Googlebot found the URL, your robots.txt told it not to crawl, and the page is not in the index. If the page should rank, find the rule with the tester above, remove or narrow it, then use URL Inspection to request indexing. If the URL is meant to stay private, this status is fine and needs no action.

Indexed, though blocked by robots.txt means Google indexed the URL without crawling it, usually because other pages link to it. It can show up in results with no description. The common mistake is trying to remove such a page by adding noindex while the robots.txt block stays. Google cannot crawl the page, so it never sees the noindex. To take it out of the index, allow crawling, add noindex, wait for Google to recrawl, and only then block it again if you still want to.

What robots.txt cannot do

• It does not remove pages from Google. It controls crawling, not indexing. Use noindex or the Removals tool for that.
• Noindex: lines inside robots.txt do nothing. Google stopped supporting them on 1 September 2019.
• Crawl-delay is ignored by Google. Googlebot adjusts its crawl rate to how fast your server responds.
• It does not protect anything. The file is public, so listing /secret-admin/ tells everyone where it is. Protect staging sites with a password, not only with Disallow: /.

How Google handles a missing or broken file

Robots.txt lives at the root of each host, for each protocol and subdomain separately: https://www.example.com/robots.txt does not cover https://shop.example.com. Google caches the file for up to 24 hours, so changes take up to a day to apply.

The Test mode shows the HTTP status of the fetched file and applies these rules, so a 404 shows every URL as allowed and a 503 shows every URL as blocked.