Set Up Cloudflare AI Crawl Control: Allow, Block, Verify

If your site sits behind Cloudflare, you can allow or block each AI crawler from one table in the dashboard, and Cloudflare turns every block into a single WAF custom rule on your zone. The setting takes about five minutes. The part that trips people up is not the toggle but the order of things around it: a robots.txt you wrote, a managed robots.txt Cloudflare can prepend to it, an older WAF rule that fires first, and a free-plan detector that reads user-agent strings only.

This guide walks the dashboard path as Cloudflare's documentation describes it on 21 October 2026, then shows how to prove the setting did what you meant. It stays on AI Crawl Control. For the September change to Cloudflare's training-bot toggle and what it did to Googlebot, read Cloudflare's AI bot block and Googlebot instead.

Quick summary

What the experts say: Cloudflare's own documentation says robots.txt is a stated preference and AI Crawl Control is the enforcement. Use both, and test the result.

How AI Crawl Control works

Think of your site as a building with two signs and one guard. The robots.txt file is the sign at the door: polite visitors read it, others walk past. The guard is a WAF rule that checks who is at the door and refuses entry. AI Crawl Control is the console that tells the guard which visitors to stop.

Three facts from Cloudflare's documentation shape every decision below:

• Each crawler gets one action. In the crawler table you choose Allow or Block per crawler. A third action, Charge, belongs to pay per crawl, which is in closed beta, so most sites will not see it. See pay per crawl for that decision.
• A block is a WAF custom rule. Cloudflare says blocking some or all AI crawlers uses one WAF custom rule. If you allow every crawler, AI Crawl Control uses none. Plans have limits on custom rules, so it matters that this takes only one.
• Detection depends on your plan. On the free plan, AI Crawl Control identifies crawlers by their user-agent strings, which finds well-known, self-identifying crawlers. Paid plans can use Bot Management's detection ID for deeper detection.

The consequence: a crawler that lies about its user agent can slip past a free-plan block, and a robots.txt line cannot stop it either. RFC 9309 is explicit that robots.txt rules are not access authorization.

Before you start: decide what you want

Decide per crawler category, not per brand. A sensible starting split:

If you are unsure, allow search and assistant bots first, and block nothing until you have looked at a week of the Requests column. Cloudflare's guidance for Allow is to choose crawlers that give you value through citations, referrals or agreements.

Path A: set per-crawler actions in AI Crawl Control

1. Log in to the Cloudflare dashboard and select your account and domain.
2. Go to AI Crawl Control.
3. Open the tab Cloudflare's docs name for this task. The current page says Security; the table it describes is headed Crawlers. Tab labels have moved between documentation revisions, so look for the table with a Crawler, Category, Requests, Robots.txt violations and Action column.
4. Review a week of activity first. The Requests column counts allowed and unsuccessful requests, and unsuccessful ones can come from any rule or error, not only from AI Crawl Control.
5. In the Action column, select Allow or Block for each crawler.
6. If you block anything, note which crawlers you chose. The next section is how you confirm each one.

You can still enforce robots.txt on a crawler you allow. Allow is not a promise that every path is open.

Path B: also state it in robots.txt

A block stops a crawler at the edge. A robots.txt line tells every well-behaved crawler your preference, including ones that never reach Cloudflare's detection. Cloudflare can write these lines for you.

1. In the Cloudflare dashboard, go to the Security Settings page.
2. Filter by Bot traffic.
3. Select Configure AI bot policies.
4. For Training, select Disallow.
5. Turn on Enable Bot Preference Sync.
6. Select Save.

Cloudflare's managed robots.txt page says Cloudflare prepends its managed block ahead of your existing file when your origin returns 200 for robots.txt, and creates a file when you have none. The block carries a Content-signal line and Disallow: / groups for training crawlers. The feature is available on all plans. The same page notes that robots.txt compliance is voluntary.

If you run WordPress, keep one owner for robots.txt. ThinkRank edits it in one place alongside your sitemap and llms.txt settings, and it imports from Rank Math, Yoast, AIOSEO and SEOPress, so you re-enter nothing. Cloudflare then prepends its block ahead of what the plugin serves. Check the combined file with curl rather than assuming. For hand-written files, the robots.txt starter covers the rules, and declaring a content-use policy covers the Content-Signal syntax.

Path C: change what a blocked crawler receives (paid plans)

1. Go to AI Crawl Control.
2. Open the Settings tab.
3. Under Block response, select Edit.
4. Choose the response code: 403 Forbidden to say no, or 402 Payment Required to say the crawler must pay.
5. Edit the response body if you want to point the crawler's owner at a licensing page.
6. Select Save.

You must have blocked at least one crawler first. If you hand-edited the AI Crawl Control WAF rule to return some other code, the dropdown appears blank until you restore 403 or 402. If you want path-based exceptions, extend the rule in WAF, as the docs describe.

What we measured on 21 October 2026

We read robots.txt from 98 well-known hosts and requested the homepage with three user agents on the 45 that sit behind Cloudflare. One datacenter IP, one pass, homepage and robots.txt only.

Read this carefully. Zero managed blocks means the feature is opt-in and rarely switched on among large sites we sampled, not that it fails. Zero user-agent differences means none of the 45 refused a self-declared GPTBot or ClaudeBot at the homepage. It does not prove those sites allow the real crawlers: real crawlers arrive from published IP ranges, and a block that verifies identity treats our single-IP forged user agent differently from the real thing. The sample is also tilted toward large, well-known sites. Treat the numbers as a starting point for your own check, not a market figure.

Verify the setting worked

Start with the scanner, then check by hand if you would rather.

With AIScan. Run npx aiscan-cli yoursite.com or paste the URL at aiscan.site. check:B1 looks for a Content-Signal line in robots.txt (informational, it cannot lower your score) and check:B2 looks for explicit rules for AI bots. Both read the file, so they confirm Path B, not the edge block; the curl test below confirms the block. It is free, with no account. Details are on the bot access checks page.

By hand. These commands work from any terminal:

# 1. Is robots.txt served, and by whom?
curl -sI https://yoursite.com/robots.txt | grep -iE '^(HTTP|server|content-type)'

# 2. Does the managed block appear?
curl -s https://yoursite.com/robots.txt | grep -n -iE 'cloudflare managed|content-signal'

# 3. What does a self-declared crawler get? Baseline first.
curl -s -o /dev/null -w '%{http_code}\n' -A 'Mozilla/5.0' https://yoursite.com/
curl -s -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.2' https://yoursite.com/

Pass: robots.txt returns 200 with text/plain, the lines you expect appear, and a crawler you blocked gets 403 (or 402 if you chose that) while the baseline gets 200. Failure looks like the same status for both, or a 403 on the baseline too. On a free plan, remember detection is by user agent, so a forged GPTBot string is a fair test of the block. For the full method, including the control path and rate-limit repeat, see fetching your site as an AI agent.

What AIScan cannot see

AIScan fetches your site from outside, like any visitor. It cannot read your dashboard, so it cannot tell whether a 403 came from AI Crawl Control, a country rule or a rate limit. It also cannot show you the real GPTBot arriving from its published IPs. The Requests and Robots.txt violations columns in Cloudflare are the only view of that.

Common mistakes

• Blocking before looking. Read a week of traffic first. A block on an assistant bot removes you from the answers it writes.
• Forgetting upstream rules. AI Crawl Control's rule contains only the bots you blocked. If an allowed bot is still refused, an earlier WAF custom rule is the usual cause. Cloudflare says to review your custom rules in order.
• Trusting a violation count right after an edit. Violations are computed against your current robots.txt, so adding a new Disallow makes past requests look like violations.
• Rate limiting the discovery files. See rate limiting AI crawlers without blocking.

Maintenance

Reopen the crawler table monthly. New crawlers appear, and the docs page for managing crawlers was last updated on 28 July 2026, so recheck the path before you follow a screenshot from last year. After any change, rerun the three curl commands.

Your next step

Run the scan on your site now and read the B1 and B2 results, then set one crawler's action and re-test it. Every AI readiness guide, by check, is in the guides index.