What AI systems actually fetch from your site

Four classes of crawler

Search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot) build the index an assistant searches when it answers. Live-fetch agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because someone asked about it right now. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider) collect text for model training. Googlebot is its own case: AI Overviews and AI Mode use the ordinary Google index, so Googlebot decides both.

These classes behave differently and are controlled separately. Blocking GPTBot does nothing to ChatGPT search, which uses OAI-SearchBot. Allowing Googlebot but blocking Google-Extended keeps you in AI Overviews while opting out of training. Treating "AI crawlers" as one switch is the most common mistake we see.

Who honours robots.txt

Search-index and training crawlers from the large vendors state that they honour robots.txt. Live-fetch agents are less uniform: Perplexity-User documents that it fetches on a user request regardless of robots rules, and others behave the same in practice. That means a Disallow line stops the index from being built, but does not reliably stop a single fetch when someone pastes your link into an assistant.

The practical consequence: what a fetcher receives on that visit matters as much as whether it is allowed. An allowed crawler that receives an empty page is worse than a blocked one, because it records that your site has nothing to say.

What the audit does

The audit evaluates each named token in a versioned catalog against your robots.txt with the same longest-match rules a crawler applies, and quotes the line that decided the verdict. A missing robots.txt means every crawler is allowed. A group you never wrote means unspecified, which also allows. Nothing is inferred from a generic "AI crawlers" setting, because no crawler reads one.