---
title: "Set Up Cloudflare AI Crawl Control: Allow, Block, Verify"
slug: cloudflare-ai-crawl-control-setup
published: 2026-10-21T03:10:01.671669+00:00
updated: 2026-10-21T03:10:01.671669+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: how-to, cloudflare, ai crawl control, ai crawlers, robots.txt, waf, check:B1, check:B2, platform:cloudflare
description: "Set Cloudflare AI Crawl Control step by step: allow or block each AI crawler, sync robots.txt, set the block response, then verify it with curl and a free scan."
url: https://aiscan.site/blog/cloudflare-ai-crawl-control-setup
---

If your site sits behind Cloudflare, you can allow or block each AI crawler from one table in the dashboard, and Cloudflare turns every block into a single WAF custom rule on your zone. The setting takes about five minutes. The part that trips people up is not the toggle but the order of things around it: a robots.txt you wrote, a managed robots.txt Cloudflare can prepend to it, an older WAF rule that fires first, and a free-plan detector that reads user-agent strings only.

This guide walks the dashboard path as Cloudflare's documentation describes it on 21 October 2026, then shows how to prove the setting did what you meant. It stays on AI Crawl Control. For the September change to Cloudflare's training-bot toggle and what it did to Googlebot, read [Cloudflare's AI bot block and Googlebot](/blog/cloudflare-ai-bot-block-googlebot-2026) instead.

## Quick summary

| If you want to... | Do this | Plan | Time |
|---|---|---|---|
| Let AI search and assistant bots in, keep training bots out | Set **Allow** or **Block** per crawler in **AI Crawl Control** | Free and up | 5 min |
| Tell crawlers your preference in robots.txt too | Turn on **Bot Preference Sync**, set **Training** to **Disallow** | All plans | 3 min |
| Change what a blocked crawler receives | **Settings** tab, **Block response**, pick 403 or 402 | Paid plans | 2 min |
| Find out why an allowed crawler is still refused | Read your WAF custom rules in order | Any | 10 min |
| Check the result | Run `npx aiscan-cli yoursite.com` (checks `check:B1`, `check:B2`), or curl by hand | Free | 1 min |

**What the experts say:** [Cloudflare's own documentation](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) says robots.txt is a stated preference and AI Crawl Control is the enforcement. Use both, and test the result.

## How AI Crawl Control works

Think of your site as a building with two signs and one guard. The robots.txt file is the sign at the door: polite visitors read it, others walk past. The guard is a WAF rule that checks who is at the door and refuses entry. AI Crawl Control is the console that tells the guard which visitors to stop.

Three facts from Cloudflare's documentation shape every decision below:

- **Each crawler gets one action.** In the crawler table you choose **Allow** or **Block** per crawler. A third action, **Charge**, belongs to pay per crawl, which is in closed beta, so most sites will not see it. See [pay per crawl](/blog/cloudflare-pay-per-crawl-should-you-charge) for that decision.
- **A block is a WAF custom rule.** [Cloudflare says](https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/) blocking some or all AI crawlers uses one WAF custom rule. If you allow every crawler, AI Crawl Control uses none. Plans have limits on custom rules, so it matters that this takes only one.
- **Detection depends on your plan.** On the free plan, AI Crawl Control identifies crawlers by their user-agent strings, which finds well-known, self-identifying crawlers. Paid plans can use Bot Management's detection ID for deeper detection.

The consequence: a crawler that lies about its user agent can slip past a free-plan block, and a robots.txt line cannot stop it either. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) is explicit that robots.txt rules are not access authorization.

## Before you start: decide what you want

Decide per crawler category, not per brand. A sensible starting split:

| Category | Examples | Typical choice | Why |
|---|---|---|---|
| AI search and assistant fetchers | the bots behind ChatGPT search, Claude and Perplexity answers | Allow | They fetch pages to cite you in answers |
| Training crawlers | GPTBot, ClaudeBot, CCBot | Your call | They collect text for model training |
| Archivers | Internet Archive style bots | Allow | Low load, public record |

If you are unsure, allow search and assistant bots first, and block nothing until you have looked at a week of the **Requests** column. Cloudflare's guidance for **Allow** is to choose crawlers that give you value through citations, referrals or agreements.

## Path A: set per-crawler actions in AI Crawl Control

1. Log in to the Cloudflare dashboard and select your account and domain.
2. Go to **AI Crawl Control**.
3. Open the tab Cloudflare's docs name for this task. The current page says **Security**; the table it describes is headed **Crawlers**. Tab labels have moved between documentation revisions, so look for the table with a **Crawler**, **Category**, **Requests**, **Robots.txt violations** and **Action** column.
4. Review a week of activity first. The **Requests** column counts allowed and unsuccessful requests, and unsuccessful ones can come from any rule or error, not only from AI Crawl Control.
5. In the **Action** column, select **Allow** or **Block** for each crawler.
6. If you block anything, note which crawlers you chose. The next section is how you confirm each one.

You can still enforce robots.txt on a crawler you allow. **Allow** is not a promise that every path is open.

## Path B: also state it in robots.txt

A block stops a crawler at the edge. A robots.txt line tells every well-behaved crawler your preference, including ones that never reach Cloudflare's detection. Cloudflare can write these lines for you.

1. In the Cloudflare dashboard, go to the **Security Settings** page.
2. Filter by **Bot traffic**.
3. Select **Configure AI bot policies**.
4. For **Training**, select **Disallow**.
5. Turn on **Enable Bot Preference Sync**.
6. Select **Save**.

[Cloudflare's managed robots.txt page](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) says Cloudflare prepends its managed block ahead of your existing file when your origin returns 200 for robots.txt, and creates a file when you have none. The block carries a `Content-signal` line and `Disallow: /` groups for training crawlers. The feature is available on all plans. The same page notes that robots.txt compliance is voluntary.

If you run WordPress, keep one owner for robots.txt. [ThinkRank](https://thinkrank.ai) edits it in one place alongside your sitemap and llms.txt settings, and it imports from Rank Math, Yoast, AIOSEO and SEOPress, so you re-enter nothing. Cloudflare then prepends its block ahead of what the plugin serves. Check the combined file with curl rather than assuming. For hand-written files, [the robots.txt starter](/blog/robots-txt-starter-for-ai-crawlers) covers the rules, and [declaring a content-use policy](/blog/declare-content-use-policy-to-ai) covers the `Content-Signal` syntax.

## Path C: change what a blocked crawler receives (paid plans)

1. Go to **AI Crawl Control**.
2. Open the **Settings** tab.
3. Under **Block response**, select **Edit**.
4. Choose the response code: `403 Forbidden` to say no, or `402 Payment Required` to say the crawler must pay.
5. Edit the response body if you want to point the crawler's owner at a licensing page.
6. Select **Save**.

You must have blocked at least one crawler first. If you hand-edited the AI Crawl Control WAF rule to return some other code, the dropdown appears blank until you restore 403 or 402. If you want path-based exceptions, extend the rule in WAF, as the docs describe.

## What we measured on 21 October 2026

We read robots.txt from 98 well-known hosts and requested the homepage with three user agents on the 45 that sit behind Cloudflare. One datacenter IP, one pass, homepage and robots.txt only.

| Measurement | Result |
|---|---|
| Hosts probed | 98 |
| robots.txt answered 200 | 91 (7 returned 403, 418 or no response) |
| Of those 91, served through Cloudflare (`server: cloudflare` or `cf-ray`) | 45 |
| Hosts with Cloudflare's managed block (`BEGIN Cloudflare Managed content`) | 0 of 91 |
| Hosts with any `Content-Signal` line | 11 of 91 |
| Hosts with a `GPTBot` group | 21 of 91 |
| Cloudflare hosts that answered a GPTBot or ClaudeBot user agent differently from a Chrome user agent on a page that Chrome could load | 0 of 45 |

Read this carefully. Zero managed blocks means the feature is opt-in and rarely switched on among large sites we sampled, not that it fails. Zero user-agent differences means none of the 45 refused a self-declared GPTBot or ClaudeBot at the homepage. It does not prove those sites allow the real crawlers: real crawlers arrive from published IP ranges, and a block that verifies identity treats our single-IP forged user agent differently from the real thing. The sample is also tilted toward large, well-known sites. Treat the numbers as a starting point for your own check, not a market figure.

## Verify the setting worked

Start with the scanner, then check by hand if you would rather.

**With AIScan.** Run `npx aiscan-cli yoursite.com` or paste the URL at [aiscan.site](https://aiscan.site/). `check:B1` looks for a `Content-Signal` line in robots.txt (informational, it cannot lower your score) and `check:B2` looks for explicit rules for AI bots. Both read the file, so they confirm Path B, not the edge block; the curl test below confirms the block. It is free, with no account. Details are on the [bot access checks page](/docs/checks/bot-access).

**By hand.** These commands work from any terminal:

```bash
# 1. Is robots.txt served, and by whom?
curl -sI https://yoursite.com/robots.txt | grep -iE '^(HTTP|server|content-type)'

# 2. Does the managed block appear?
curl -s https://yoursite.com/robots.txt | grep -n -iE 'cloudflare managed|content-signal'

# 3. What does a self-declared crawler get? Baseline first.
curl -s -o /dev/null -w '%{http_code}\n' -A 'Mozilla/5.0' https://yoursite.com/
curl -s -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.2' https://yoursite.com/
```

Pass: robots.txt returns 200 with `text/plain`, the lines you expect appear, and a crawler you blocked gets 403 (or 402 if you chose that) while the baseline gets 200. Failure looks like the same status for both, or a 403 on the baseline too. On a free plan, remember detection is by user agent, so a forged `GPTBot` string is a fair test of the block. For the full method, including the control path and rate-limit repeat, see [fetching your site as an AI agent](/blog/fetch-site-as-ai-agent-curl).

## What AIScan cannot see

AIScan fetches your site from outside, like any visitor. It cannot read your dashboard, so it cannot tell whether a 403 came from AI Crawl Control, a country rule or a rate limit. It also cannot show you the real GPTBot arriving from its published IPs. The **Requests** and **Robots.txt violations** columns in Cloudflare are the only view of that.

## Common mistakes

- **Blocking before looking.** Read a week of traffic first. A block on an assistant bot removes you from the answers it writes.
- **Forgetting upstream rules.** AI Crawl Control's rule contains only the bots you blocked. If an allowed bot is still refused, an earlier WAF custom rule is the usual cause. Cloudflare says to review your custom rules in order.
- **Trusting a violation count right after an edit.** Violations are computed against your current robots.txt, so adding a new `Disallow` makes past requests look like violations.
- **Rate limiting the discovery files.** See [rate limiting AI crawlers without blocking](/blog/rate-limit-ai-crawlers-without-blocking).

## Maintenance

Reopen the crawler table monthly. New crawlers appear, and the docs page for managing crawlers was last updated on 28 July 2026, so recheck the path before you follow a screenshot from last year. After any change, rerun the three curl commands.

## Your next step

Run the scan on your site now and read the B1 and B2 results, then set one crawler's action and re-test it. Every AI readiness guide, by check, is in [the guides index](/guides).