Search, Agent, Training: What Cloudflare's Three AI Crawler Categories Actually Control

Updated 25 August 2026. Cloudflare now sorts crawler behaviour into three categories you can configure — Search, Agent, and Training — and the setting you pick for each decides whether Google indexes you, whether ChatGPT can answer a question about your page, and whether your content reaches a training run. Get one wrong and you can lose organic traffic without losing a single AI citation, or the reverse. This is the reference for what each category actually controls, which crawlers sit in it, and what blocking it costs.

If you only want to know whether your own site is currently blocking something, start with the 15 September checklist instead — it is the short version with the curl commands. This page is the underlying map.

Quick summary

Short version: Search feeds both your Google rankings and your AI citations, and it is allowed by default. Agent is the one with a real person waiting. Training is the one that quietly takes Googlebot with it if you reach for Block instead of Disallow.

The rule that catches people: a crawler can carry more than one tag, and the most restrictive matching rule wins. Googlebot is tagged Search and Training, which is why blocking Training blocks Googlebot.

The one-line test: curl -sI -A "<user-agent>" https://yoursite.com/ | head -1 — 200 allowed, 403 blocked, 402 pay-per-crawl.

Why the categories exist

Before July 2026, "AI bot" was a single bucket. That was always wrong, because the bots inside it do unrelated jobs with opposite consequences. GPTBot collects text for a future model. OAI-SearchBot builds the index that decides whether ChatGPT cites you today. ChatGPT-User fetches your page right now because a person pasted the link. Blocking all three with one switch is like refusing all post because you dislike junk mail.

Cloudflare's 1 July 2026 taxonomy splits crawler behaviour into eleven types — Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social/Link Preview, Feed Fetching, and Monitoring & Operations. Three of them are configurable by every customer on every plan, including Free. Those three are the ones below.

One rule governs how they combine, and it catches people out: a crawler can belong to more than one category, and the most restrictive matching rule wins. That is the mechanism behind the Googlebot problem covered in the companion guide.

Enforcement happens at the Cloudflare edge through WAF rules. robots.txt is a separate advisory layer that records preferences. The two can disagree, and until Bot Preference Sync rolls out, they frequently do.

Search

What it covers: crawling to build search indexes or RAG databases.

That second half is the part most people miss. Cloudflare's verified bots documentation states it directly: "Under the taxonomy introduced on July 1, 2026, there is no longer a meaningful distinction between 'AI Search' and traditional search — both are treated as Search behavior. The AI Search category value is retained for backward compatibility."

Crawlers in this category:

Default: allowed. It was never part of the 15 September block, in any version of the policy.

What blocking it costs: your rankings and your AI citations, together. This is the category that produces the outcome most people think they are avoiding when they block "AI bots." Cloudflare's own docs carry the warning in the pay-per-crawl flow: setting these crawlers to Block or Charge "may negatively impact your site's SEO performance, as search engines may not be able to properly index your content."

Who should block it: almost nobody. If you are considering it, you want noindex on specific pages, not an edge-level block on the category.

Agent

What it covers: a bot fetching your page live because a person asked it to. Cloudflare's description: "chat fetch bots (e.g., ChatGPT-User) and browser-use agents (e.g., Gemini or Claude driving Chrome)."

Crawlers in this category:

Default: for new publisher domains that tick "I monetize from pages with ads," blocked on pages that display ads. Everyone else, allowed.

What blocking it costs: the reader who is asking about you right now. There is a person on the other end of an Agent request, waiting. Block it and they get told the page could not be accessed. That is a worse outcome than not being cited, because the intent was already yours to lose.

Who should block it: ad-funded publishers, and only after deciding deliberately. The trade is real — an Agent fetch reads your page without loading your ads. Cloudflare built the ad-page-scoped option specifically for this, so the block applies where the revenue is and nowhere else. Note that Cloudflare uses automated detection to decide which pages display ads, and publishes no accuracy figures for that detection.

Who should definitely not block it: e-commerce, SaaS, documentation, and anyone whose pages are the destination rather than the inventory. Blocking Agent breaks agentic shopping and breaks an agent's ability to read your docs.

This category is also the unresolved argument in the whole space. When Cloudflare de-listed Perplexity as a verified bot in August 2025 for what it called stealth crawling, Perplexity's defence was that "the difference between automated crawling and user-driven fetching isn't just technical — it's about who gets to access information on the open web." The Agent category is Cloudflare's answer to that question, and the answer is that a user-driven fetch is still something a site owner may refuse.

Training

What it covers: collecting content to train models.

Crawlers in this category — this is the list Cloudflare's managed robots.txt disallows, verified live on 25 August 2026:

Amazonbot · Applebot-Extended · Bytespider · CCBot · ClaudeBot · CloudflareBrowserRenderingCrawler · Google-Extended · GPTBot · meta-externalagent

The list is now generated dynamically through Cloudflare's BotBase rather than hard-coded, so treat any published copy of it — including this one — as a snapshot. CloudflareBrowserRenderingCrawler already appears in live output without being in the documented example.

Default: for new publisher domains, Disallow. For everyone else, no change.

What blocking it costs: inclusion in future training runs, which is genuinely hard to value. And, from 15 September, mixed-purpose crawlers — because Googlebot, Applebot and Bingbot all carry a Training tag alongside their Search tag.

That last point is the single most consequential detail in this taxonomy, and it is why the 21 August update added a Disallow option distinct from Block. Cloudflare's description: a "no training" preference is written to your robots.txt "so that cooperating mixed-use crawlers who take the extra Transparency step can still access your content for search indexing." Disallow states the preference without dropping a 403 on the search crawler.

Who should block it: sites with high-value proprietary content and low dependence on search. Everyone else should prefer Disallow, or accept training as the price of being findable.

What none of this controls

Two limits, because a lot of coverage implies otherwise.

Google AI Overviews. Google-Extended governs Gemini model training only. Google's documentation says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode are fed by ordinary Googlebot. The only levers are nosnippet, data-nosnippet, max-snippet and noindex, all of which also cut your regular search snippets. There is no AI-Overviews-specific opt-out available from Cloudflare or anyone else.

Whether anyone honours your robots.txt preferences. Edge blocks are enforced regardless. Preferences are not. On Content Signals specifically, Google's John Mueller said in July 2026: "none of the crawlers / llms use the 'content-signal' robots.txt directives. It was made up by a CDN, afaik it has no effects whatsoever for any crawler or llm. Using it just adds bloat & future maintenance to your robots.txt file."

Cloudflare does not really dispute the mechanism. Its own wording is that the signals "signal a website owner's preference, rather than issuing blocks directly." Worth knowing before you treat a Content-Signal line as protection.

Reading the signals in robots.txt

If Cloudflare manages your robots.txt, you will find a line like this:

Content-Signal: search=yes,ai-train=no,use=reference

The use= field arrived with the July 2026 update and has three values:

Cloudflare is also experimenting with transitive trust via the RFC 7239 Forwarded header, so an agent acting for a named principal can declare it: Forwarded: for="openai";use="reference". That is early and not widely implemented.

Matching categories to your site

The pattern is consistent: Search stays allowed in every row. The real decisions are Agent, which trades reader access against ad revenue, and Training, where Disallow is almost always the better instrument than Block.

Setting these categories on WordPress

The three categories are configured in Cloudflare, not in your CMS — but the robots.txt half of the picture is usually your CMS's job, and on WordPress that means a plugin.

This causes a specific confusion worth naming. Cloudflare's managed robots.txt prepends its block to whatever your site already serves, so a WordPress robots.txt and a Cloudflare block can coexist and contradict each other. Cloudflare acknowledged the problem when it shipped Bot Preference Sync on 21 August: when stated preferences and enforced rules disagree, "some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules."

So keep the WordPress side coherent and let Cloudflare own enforcement. ThinkRank handles that side — robots.txt and robots meta from one place, llms.txt generation, and schema and Q&A structuring for answer engines, with migration from Rank Math, Yoast, All in One SEO and SEOPress so existing directives carry over rather than being retyped. AIScan detects ThinkRank during a scan and adapts its fix steps accordingly.

The division of labour is simple: your plugin decides what your site says, Cloudflare decides what your site does. When they disagree, Cloudflare wins.

Common mistakes

Assuming Search means "traditional search only". Cloudflare's own definition includes RAG databases, and its documentation names OAI-SearchBot as the example. Retrieval for AI answers is Search.

Blocking Training to protect content, and losing rankings instead. Use Disallow. Block is for content you genuinely will not let search engines index either.

Reading robots.txt to find out whether you are blocking something. It records a preference. The edge decides. These regularly disagree.

Blocking Agent on an e-commerce site. That is the category that carries a buyer. Cloudflare's own worked example is a shopper asking a chatbot for a sofa.

Copying a crawler list from a blog post without checking it. They rot fast, and anthropic.com/claudebot.json has been dead for a while yet still appears in plenty of them. Cloudflare's own list is now generated dynamically through BotBase.

Verifying what a crawler claims to be

Every user-agent string in this article can be forged in one curl flag. If you are making decisions from access logs, match the source IP against the operator's published list. All live on 25 August 2026:

anthropic.com/claudebot.json returns 404. The live location is on claude.com, per Anthropic's documentation. A lot of published crawler lists still carry the dead URL, which is a decent test of whether a list is maintained.

Cloudflare's own bar for a Verified bot is worth borrowing as a mental model: "Honest self-identification — it declares who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS," plus non-abusive behaviour.

Checking your own configuration

Test enforcement rather than intent, because the two diverge:

# Search — your rankings and your AI citations
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/ | head -1
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://example.com/ | head -1

# Agent — a person is waiting on this one
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" https://example.com/ | head -1

# Training
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" https://example.com/ | head -1

200 allowed, 403 blocked, 402 pay-per-crawl. Run it against a real content URL rather than your homepage — the Agent default is scoped to pages that display ads.

In the Cloudflare dashboard the controls live at Security Settings > Configure AI bot policies, with the deprecating legacy switch at Security Settings > Block AI bots. For what the mixed-purpose rule does to Googlebot on 15 September, see the 15 September checklist.

Where AIScan fits, and where it doesn't

The three categories live in your Cloudflare dashboard, and nothing outside your origin can read that state. What is externally visible is the declaration layer — and that is what AIScan grades:

That last check is the one that decides whether any of this mattered. Allowing every category and then serving an empty DOM is a more common failure than being blocked, and it is invisible in a browser.

npx aiscan-cli example.com

Free for 20 scans a month, no account needed for the commands in this article. For the enforcement question — is my edge returning 403 — use curl against a real content URL. A scanner reads what your site declares; only a request reveals what it does.

The short version

Search is what feeds both your rankings and your AI citations, and it has never been blocked by default. Agent is the one with a human waiting. Training is the one that quietly takes Googlebot with it if you reach for Block instead of Disallow.

Almost every mistake in this area comes from treating those three as one thing.