robots.txt for AI crawlers, line by line

Longest match wins

A crawler looks for the User-agent group that best matches its token, and only that group applies. If OAI-SearchBot has its own group, the star group is ignored for it. If it has none, the star group applies. Inside a group, the most specific path rule wins, and Allow beats Disallow at equal length. This is RFC 9309 and the large vendors follow it.

Two consequences. A blanket "User-agent: * Disallow: /" blocks every crawler with no group of its own, including every AI search crawler, and stays in force on the production host if a preview robots.txt was copied over. And a group written for one token, say GPTBot, does nothing for the other tokens of the same vendor.

What to allow, what to decide

For a site that wants to be found through assistants, the search-index and live-fetch tokens should be allowed, either by having no group of their own under a permissive star, or by an explicit Allow. Training tokens are a policy decision with no effect on whether assistants can answer about you today; opting out of training does not remove you from answers, and opting in does not put you in them.

Keep the decisions separate in the file: one group per token you actively manage, comments explaining why, and no wildcard tricks. A robots.txt that a person can read is one a crawler reads correctly.

What the repair does

When your crawler policy allows search and live-fetch crawlers, a repair turns a "Disallow: /" in their own group into "Allow: /", or adds an explicit Allow group where the star rule was catching them. It never edits training tokens, and it never opens a blanket Disallow on a preview host, because that rule is usually there on purpose.