AI visibility

Anthropic

Can Claude read your site?

ClaudeBot gathers pages; Claude-User fetches a page the moment someone asks about it.

Reads robots.txt and llms.txt. Nothing stored.

How it reads the web

Anthropic documents ClaudeBot as the crawler that collects public pages, plus user-triggered agents that fetch a page on request. Claude reads the served HTML and follows robots.txt. Claude does not execute scripts to reveal content, so a page that only exists after hydration is a blank page to it.

Documented user agents

  • ClaudeBotchecked
  • Claude-Userdocumented
  • Claude-SearchBotdocumented

What your site needs

Four things that decide whether Claude can use your pages

  1. 01

    Allow ClaudeBot in robots.txt

    Most builder templates ship a permissive robots.txt, but an "AI blocklist" snippet copied from a blog post often includes it.

  2. 02

    Server-visible content

    Static export or server rendering for the pages you want quoted.

  3. 03

    One main heading and real sections

    Extraction follows the document outline. A page of divs with no headings has no shape.

Why builder sites fail here

  • SPA routers that return the same shell for every URL
  • Meta robots noindex left on from the preview build
  • Open Graph and canonical tags pointing at *.lovable.app or *.vercel.app

BulkAudits finds these on a live site, then opens a pull request on a new branch with the fix. You merge.

Audit my site

Questions about Claude

Is there a Claude search index?
Anthropic documents Claude-SearchBot alongside ClaudeBot and Claude-User. BulkAudits checks ClaudeBot today and reports the others as they become common in robots.txt files.
Will llms.txt make Claude cite me?
No file guarantees a citation. llms.txt is a courtesy index for assistants; BulkAudits counts it as one of nine readiness checks, and the other eight matter more.

Other assistants

Every check above is part of the nine-point AI readiness score in a full audit.