Documentation
AI Search Readiness
AI Search Readiness checks whether AI systems can access and understand your website. It does not guarantee citations or recommendations.
The module answers one question: can AI systems reach this site, receive real content, and understand what the business is. It is not a citation tracker. It does not ask ChatGPT questions or report how often you are mentioned.
Three layers
- 01
Access
Can the named AI bots fetch this origin? robots.txt per token, noindex directives, sitemap, canonicals on the production host, no challenge wall.
- 02
Receipt
Do they receive real content if they fetch? The raw HTML, not the page after JavaScript. An empty application shell fails here even when the site looks finished in a browser.
- 03
Meaning
Can they tell what the business and the page are? Valid JSON-LD with an Organization or WebSite, a coherent name across title, heading and markup, one main heading, an optional llms.txt.
Each layer is reported as blocked, partial or ready. A site can pass Access and fail Receipt; that is the usual story for a site built with an AI builder. The layers are never collapsed into one score in the report; a composite number exists only for trends.
Bot classes
- Search index
- Builds an AI search index used for live answers. Recommended: allow, unless you have opted out on purpose.
- Live fetch
- Fetches a page because a user asked about it. Recommended: allow for a public marketing site.
- Google search
- Googlebot feeds Google Search and, with it, AI Overviews and AI Mode. Blocking it is usually accidental.
- Training
- Used to train future models. A policy choice; BulkAudits never changes it for you.
Google-Extended is not Googlebot. Blocking Google-Extended withholds content from Gemini training and grounding; it does not remove a site from Google Search, AI Overviews or AI Mode. Blocking Googlebot does.
Bot catalog v2026-09-16
| Token | Operator | Class | Honours robots.txt | Note |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Search index | yes | Builds the index behind ChatGPT search. Sites that block it do not appear in ChatGPT search answers. |
| ChatGPT-User | OpenAI | Live fetch | partial | Fetches a page when a ChatGPT user asks about it. OpenAI says it honours robots.txt; treat Allow as desired. |
| GPTBot | OpenAI | Training | yes | Collects pages for model training. Blocking it does not remove the site from ChatGPT answers. |
| Claude-SearchBot | Anthropic | Search index | yes | Indexes pages to improve Claude search results. |
| Claude-User | Anthropic | Live fetch | yes | Fetches a page when a Claude user asks about it. |
| ClaudeBot | Anthropic | Training | yes | Collects pages that may contribute to model training. |
| anthropic-ai | Anthropic | Training | yes | Older Anthropic training token still present in many blocklists. |
| PerplexityBot | Perplexity | Search index | yes | Surfaces and links sites in Perplexity answers. Not used for training. |
| Perplexity-User | Perplexity | Live fetch | no | Fetches a page on a user request. Perplexity documents that it generally ignores robots.txt, so an Allow here is not what decides access. |
| Google-Extended | Training | yes | Controls Gemini training and grounding only. Blocking it does not remove the site from Google Search, AI Overviews or AI Mode. | |
| Googlebot | Google search | yes | Google Search, and with it AI Overviews and AI Mode. Blocking it is almost always accidental. | |
| Applebot-Extended | Apple | Training | yes | Apple Intelligence training. Applebot itself serves Siri and Spotlight. |
| CCBot | Common Crawl | Training | yes | Common Crawl, the open dataset many models are trained on. |
| Amazonbot | Amazon | Search index | yes | Indexes pages for Alexa and Amazon answers. |
| Meta-ExternalAgent | Meta | Training | yes | Meta AI training. Meta-ExternalFetcher handles user-triggered fetches. |
| Bytespider | ByteDance | Training | yes | ByteDance model training. |
What we fetch
- robots.txt (up to 5 redirects, 256 KB), evaluated per token with longest-match rules; the exact line that decided each verdict is kept as evidence.
- llms.txt (status and body) and sitemap.xml.
- The home page and a sample of internal pages: raw HTML, headers including X-Robots-Tag, JSON-LD blocks, headings, visible text.
- One request identifying as Googlebot to detect hosting rules that reject crawlers. We never identify as the AI bot of a vendor. See our crawler.
What we never change automatically
- Training-crawler policy. A repair may allow search and live-fetch tokens that a simple rule blocked; it never flips GPTBot, ClaudeBot, Google-Extended, CCBot or Bytespider to Allow, and never deletes a Disallow you wrote for them.
- A Disallow: / on a preview or staging host.
- Business facts we did not observe: no invented sameAs, phone numbers, logos, prices or FAQs.
- Rendering. A single-page app is not switched to server rendering; the report says which framework setting does that.
- Body copy and headings.
About llms.txt
An optional map for coding agents and documentation tools, checked as one low-severity signal inside the Meaning layer. It is not confirmed as a citation or ranking signal for ChatGPT or Google, and it can never turn a blocked layer into a ready one. The repair writes a stub from the title and pages of the site itself when the file is missing; it never dumps the whole site.