AI crawlers
What GPTBot, ClaudeBot, PerplexityBot and Googlebot fetch, what they skip, and how to let the right ones in.
The free tools that answer "is my site blocking AI crawlers" read robots.txt. A block can also sit at the edge, in an X-Robots-Tag header, or in a meta tag you never wrote. We failed all four on our own site.
Cloudflare Radar put Anthropic at 47,779 pages crawled per referral in January 2026 and 549 in September. Perplexity moved the other way, from 112 to 2,373. A single ratio is a dated sample, not a property of a crawler.
Since 15 September 2026, Cloudflare sorts crawlers into Search, Training and Agent. Disallow AI Training keeps Googlebot in search; the hard Block setting now stops it, search included. Here is how to tell which one your zone uses.
GPTBot, ClaudeBot and PerplexityBot read the raw HTML response and don't execute JavaScript. If your content only appears after rendering, answer engines can't retrieve it or cite it.
What does AI say about you?
We ask ChatGPT, Gemini, Google AI Mode and AI Overviews the questions your customers ask, and write down every answer.