Glossary

AI crawler

A bot operated by an AI company to collect, index or fetch web pages for training or answers.

The ten BulkAudits checks

Reported per site from the live robots.txt.

GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI)
ClaudeBot (Anthropic)
PerplexityBot (Perplexity)
Google-Extended (Google)
Applebot-Extended (Apple)
CCBot (Common Crawl)
Bytespider (ByteDance)
meta-externalagent (Meta)

Blocking wisely

Blocking a training crawler withholds content from training. Blocking a search indexer or live fetcher removes you from answers. Decide per crawler rather than with a copied blocklist.

Key Takeaways

1.AI crawler is important for website performance and SEO.
2.You can measure it using tools like BulkAudit and Google Lighthouse.
3.Improving this metric can lead to better user experience and search rankings.

Learn More

Google Web Fundamentals

Related Terms

Browse Full Glossary

Explore all SEO and web performance terms

Test Your AI crawler Score

Use BulkAudit to measure ai crawler and other key metrics for your website.

Audit my site