How to check if your site blocks AI crawlers (all four layers)

Search for a way to check whether your site blocks AI crawlers and you get a page of free checkers. We read what they say they do. Almost all of them read your robots.txt and report which bots it allows.

That is one layer. A block can sit in four places, and a request meets them in this order: your CDN's firewall, then robots.txt, then an X-Robots-Tag response header, then a meta robots tag. The most restrictive one decides the outcome. A clean robots.txt tells you very little on its own.

Where can an AI crawler block sit?

In four layers that control different things. The firewall and robots.txt govern whether the page gets fetched. Headers and meta tags govern what happens after it is fetched. A noindex header will remove you from Google completely, and practitioner guidance is not to treat it as a control over what an LLM does with a page it has already read.

A missing sitemap belongs in the same audit even though it blocks nothing. It removes the discovery path, and on a database-backed site that path is often the only way a new URL gets found. It is also more common than the rules people worry about: Cloudflare's 5 October 2026 scan of 111,711 top domains found explicit AI-crawler rules in 80% of robots.txt files, and an XML sitemap on only 68.5% of sites.

Why do checkers miss the firewall layer?

Because the block lives in your CDN, not in any file a checker can read. Cloudflare's AI Crawl Control implements a block by writing a WAF custom rule onto your zone. The rule runs before anything in your application, so a blocked crawler collects a 403 and never sees your robots.txt, your headers or your content. And because it is one WAF rule among others, order decides the result: a crawler you set to Allow in the dashboard can still be stopped by a rule that runs earlier.

Cloudflare also ships a button that translates the AI directives in your robots.txt into a firewall rule. Press it and a polite request becomes enforcement, which means a file someone wrote months ago may now be operating as a hard block.

Since 15 September 2026 the Training control has two ways to say no, and they treat Googlebot differently. Disallow AI Training writes a no-training preference into robots.txt, blocks training-only crawlers, and keeps mixed-use crawlers such as Googlebot and Applebot crawling for search. Block stops mixed-use crawlers outright, search included. Existing training blocks, the legacy "Block AI bots" toggle among them, were migrated to Disallow AI Training. Anyone who has since picked Block has taken their zone out of Google, and a robots.txt checker will still report Googlebot as allowed.

On Cloudflare's network, the share of crawler requests answered with a 4xx rose from 33.1% in January 2026 to 38.1% in September.

What is the difference between robots.txt and X-Robots-Tag?

robots.txt stops the fetch; X-Robots-Tag lets the fetch happen and asks for the page not to be indexed. People reach for these two as though they were the same lever at different altitudes.

Because robots.txt stops the fetch, the content never gets read. That makes it the stronger control, though compliance is voluntary. Training and search crawlers honour it per their vendors' documentation. User-initiated fetchers like ChatGPT-User and Perplexity-User do not honour it reliably, and controlling those needs a server or edge rule rather than a file. TollBit's August 2026 measurement found 15% of AI page fetchers in Europe reaching URLs that robots.txt had disallowed.

For Google, an X-Robots-Tag: noindex is decisive. For an LLM that has already read the page, much less so.

Does blocking one bot block the whole vendor?

No. Anthropic runs three crawlers and each needs its own directive:

Most robots.txt files haven't caught up with the split. In the same October scan, ClaudeBot was disallowed by 473 top-domain files and Claude-SearchBot by 184.

The same trap runs in reverse with Google. Blocking Google-Extended opts you out of Gemini training and grounding and leaves Google Search untouched, which is what most teams want and what most of them never configure. It does not take you out of AI Overviews or AI Mode, which Googlebot feeds and which Google controls through a separate Search Console setting. Google added a fourth name in March 2026: Google-Agent, a user-triggered fetcher for AI systems browsing on someone's behalf.

What did the check find on this site?

All four layers failed. One constant in the Nuxt config, a launch hold left switched on longer than intended, produced every failure below.

• robots.txt served User-agent: * / Disallow: /, closing the site to every compliant crawler.
• The response header carried X-Robots-Tag: noindex, nofollow on every route.
• A meta robots tag carried the same directive. The config only mentioned the header, so we found this one on the wire, not in the source.
• The sitemap returned 404, because the same flag disables sitemap generation.

A robots.txt checker would have caught our first problem. It would have missed the other three, and the fix for all four is a single line.

The same request turned up something unrelated to blocking. Our pages sent no Last-Modified and no ETag while setting cache-control: must-revalidate. With no validator to compare against, every revalidation returns a full 200 and a 304 is impossible, so a crawler re-checking an unchanged page downloads all of it again. We had been recommending the fix for it in another post while shipping the problem here.

How do you check all four layers?

Run these five checks in order. They take about a minute.

1. Fetch /robots.txt and read it literally. Look for a wildcard Disallow: /, and for any group that overrides your specific rules.
2. Read the response headers of a real content page. Look for X-Robots-Tag, and note whether Last-Modified and ETag are there.
3. Read the rendered page for <meta name="robots">. A module can emit it even when your template does not.
4. Request /sitemap.xml and confirm it returns XML rather than a styled 404 page.
5. Open your CDN's bot controls and its WAF rule list, in rule order, and see which AI crawlers get stopped before your application is involved. On Cloudflare, check what Training is set to: Disallow AI Training keeps Googlebot in search, and Block takes it out.

If you only have time for one of them, read the response headers. That is the layer least likely to get checked and most likely to have been set by something you did not write.