Cloudflare's AI Bot Block Is About to Catch Googlebot: What Changes on 15 September 2026

Updated 25 August 2026. On 15 September, Cloudflare changes how it enforces AI crawler rules. If you have ever ticked "Block AI bots" on a Cloudflare zone and left it on, Googlebot and Bingbot start receiving HTTP 403 responses from your site on that date. Checking takes about sixty seconds with curl, and fixing it is one setting. This guide covers both, plus the three categories Cloudflare now sorts crawlers into, because that taxonomy is what decides which bots reach you.

The headline you have probably seen — "Cloudflare blocks AI crawlers by default on September 15" — is wrong on two counts, and one of them matters a great deal. Cloudflare softened the default on 21 August. The part that survived is the part almost nobody is writing about, and it puts organic search traffic at risk rather than AI traffic.

Quick summary

Short version: Cloudflare is not blocking AI crawlers by default on 15 September. The change that will actually cost you traffic is that mixed-purpose crawlers stop being exempt — so an old "Block AI bots" toggle starts returning 403 to Googlebot.

The 60-second test: curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://yoursite.com/ | head -1. A 200 means you are fine. A 403 means fix it today.

What actually changes on 15 September

Two separate things, and they apply to different people.

Change one: new defaults for new domains. In the 1 July 2026 announcement, Cloudflare said that for domains newly onboarding to Cloudflare, the Training and Agent categories would be blocked by default on pages that display ads, while Search stayed allowed. Then on 21 August Cloudflare revised it: new non-publisher customers now get no blocks at all by default, and publishers self-select at onboarding by ticking "I monetize from pages with ads on this domain," which sets Training to Disallow rather than Block. Cloudflare's own words: "new customers will not have any blocks or disallows added by default when they onboard a domain: the choice is up to the customer."

If you are an existing Cloudflare customer, none of that touches you.

Change two: mixed-purpose crawlers get judged on all their behaviours. This one does touch existing zones, and it is the reason this article exists. From the July announcement:

"multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)."

Cloudflare classifies Googlebot as both a Search crawler and a Training crawler, because Google uses the same fetch for both. Until now the legacy "Block AI bots" toggle carved out an exception for mixed-purpose bots. On 15 September that exception ends, and the most restrictive matching rule wins. Block Training, and you block Googlebot with it.

This is not hypothetical. Search Engine Journal reported on 4 August that site owners were already seeing it, quoting one: "When I set AI Training = Block, both Googlebot and Bingbot start receiving HTTP 403 responses." Google's John Mueller asked for details to investigate.

Cloudflare sits in front of roughly a fifth of the web, so the population of sites with a forgotten toggle from the 2025 "Content Independence Day" push is not small.

The date held. The content of the change did not.

The mechanism: three categories, one restrictive rule

Before the steps, it helps to know why a single toggle can produce this result.

Cloudflare's July 2026 taxonomy sorts crawler behaviour into eleven types, three of which you can configure: Search, Agent, and Training. A crawler can belong to more than one. Enforcement happens at the Cloudflare edge through WAF rules, not in robots.txt — robots.txt is a separate, advisory layer.

Think of it as a bouncer with three lists rather than a sign on the door. The sign asks politely. The bouncer checks every arrival against every list, and if the visitor appears on any list marked "blocked," they do not get in. Googlebot carries two badges, Search and Training. Blocking one badge is enough.

Googlebot carries two behaviour tags. From 15 September, the most restrictive matching rule wins.

Here is what each category covers and what it costs you to block it.

Two things in that table surprise most people.

Search was never blocked, and Search is what feeds AI citations. Cloudflare's verified bots documentation is explicit: "Under the taxonomy introduced on July 1, 2026, there is no longer a meaningful distinction between 'AI Search' and traditional search — both are treated as Search behavior." OAI-SearchBot and PerplexityBot are Search crawlers. If you were worried this change would cut you out of ChatGPT results, it does not — unless you block Search yourself.

Agent is the category people underestimate. Cloudflare describes it as "chat fetch bots (e.g., ChatGPT-User) and browser-use agents (e.g., Gemini or Claude driving Chrome)." These fire when a real person pastes your URL into ChatGPT and asks about it. Block them and that person gets "I couldn't access that page." It is the one category with a human waiting on the other end.

For the full reference on the three categories — which crawlers sit in each, and what blocking each one costs — see Search, Agent, Training: what each Cloudflare category actually controls.

How to check whether your site is blocking anything

Three methods, easiest first. Do at least the first two — they answer different questions.

Method 1: curl, the enforcement test

This is the test that matters, because it measures what actually happens rather than what you intended. Run it against a real content URL, not just your homepage, and include one page that carries ads if you have any.

# Baseline — should be 200
curl -sI -A "Mozilla/5.0" https://example.com/ | head -1

# Search engines — check these before 15 September
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/ | head -1
curl -sI -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" https://example.com/ | head -1

# Retrieval crawlers — these feed your AI search citations
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://example.com/ | head -1
curl -sI -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://example.com/ | head -1

# Live agents — a person is waiting on these
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" https://example.com/ | head -1
curl -sI -A "Mozilla/5.0 (compatible; Claude-User/1.0; +Claude-User@anthropic.com)" https://example.com/ | head -1

# Training crawlers
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" https://example.com/ | head -1
curl -sI -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://example.com/ | head -1

Reading the results:

• HTTP/2 200 — allowed
• HTTP/2 403 — blocked at the edge
• HTTP/2 402 — pay-per-crawl is set on this zone

Cloudflare's block fires on the user-agent string alone, so a plain curl with a spoofed header gets treated exactly like the real crawler. Running this against Cloudflare's own test zone crawltest.com on 25 August 2026 returned 403 for GPTBot and ClaudeBot, and 200 for a plain browser string — which is what a correctly configured training block looks like.

A 403 on the Googlebot line is the finding this article is about. Fix it today, not on 14 September.

Method 2: read your robots.txt

curl -s https://example.com/robots.txt | grep -iE "Content-Signal|BEGIN Cloudflare|GPTBot|ClaudeBot"

Look for a block that Cloudflare injected, marked # BEGIN Cloudflare Managed content or, since late August, # BEGIN Cloudflare Bot Preference Sync, and a line resembling:

Content-Signal: search=yes,ai-train=no,use=reference

That use= field is new as of July 2026. It has three values: immediate (interact but store nothing), reference (index, excerpt, link back — the default), and full (summarize and reproduce).

Treat this method as informational only. robots.txt records a preference; it does not prove enforcement. Cloudflare says so plainly: the values "signal a website owner's preference, rather than issuing blocks directly." A clean robots.txt tells you nothing about whether you are returning 403s at the edge. That is why Method 1 comes first.

Worth knowing before you rely on Content Signals: Google's John Mueller said in July that "none of the crawlers / llms use the 'content-signal' robots.txt directives. It was made up by a CDN, afaik it has no effects whatsoever for any crawler or llm."

Method 3: the Cloudflare dashboard

Exact paths, current as of August 2026:

1. Security Settings > Configure AI bot policies — the Search / Agent / Training controls. Each has three options: Block (on all pages), Block on pages with ads, and Allow (do not block).
2. Security Settings > Block AI bots — the legacy toggle, marked "[Deprecating on September 15, 2026]". If this is on, you are in the affected group.
3. AI Crawl Control — per-crawler control and metrics. The Metrics tab has a status-code distribution chart and a CSV export, which is the fastest way to see whether you have been returning 403s to search engines already.
4. Security Analytics — filter by blocked bot traffic.

One thing to know about AI Crawl Control: blocking a crawler there "creates or updates a WAF custom rule on your zone." It is not a soft preference. It also supports only two response codes, 403 and 402 — anything else and the block will not be enforced.

What to do, by site type

The right answer depends on how you make money, so pick your branch. To work out which one you are on, ask: does someone pay me because a person landed on this page?

Ad-funded publisher. Allow Search. Set Training to Disallow rather than Block — that is exactly what the 21 August change added, and Cloudflare's description is that "a 'no training' preference is written to your robots.txt, so that cooperating mixed-use crawlers who take the extra Transparency step can still access your content for search indexing." Decide Agent deliberately rather than by default; blocking it on ad pages protects ad impressions but costs you the reader who asked ChatGPT about your article.

E-commerce, SaaS, or lead generation. Allow all three. Cloudflare's own example makes the case: "an e-commerce store may want everything crawled and trained on, so its products surface when a shopper asks a chatbot for 'the best sofa for a small apartment.'" Blocking Agent here also breaks agentic shopping, which is the traffic source everyone is trying to win.

Anyone who depends on Google organic. Do not leave Training set to Block past 15 September without explicitly opting out of the mixed-use default. Cloudflare added an opt-out for this: "if a website owner wants to opt out of these new default configurations, they can easily mark this in their Security settings any time leading up to September 15, which will confirm that they want no changes on Training crawlers that also crawl for Search purposes."

High-value proprietary content with low SEO dependence. Block Training everywhere. Consider serving 402 instead of 403 through pay-per-crawl, which signals a possible commercial relationship rather than a flat refusal. It remains in private beta, with a minimum price of $0.001 per crawl, and requires the crawler to have a Cloudflare billing relationship and Web Bot Auth signing.

One caution that applies to every branch. Cloudflare's own documentation warns in the pay-per-crawl flow that setting search crawlers to Block or Charge "may negatively impact your site's SEO performance, as search engines may not be able to properly index your content." Take that literally.

If your site runs on WordPress

WordPress adds a wrinkle, because robots.txt is usually not a file — it is generated, and often by a plugin. So you can have three systems with an opinion about it at once: WordPress core's virtual robots.txt, your SEO plugin, and Cloudflare's managed robots.txt prepending its own block. That is the most common reason someone edits robots.txt, sees no change, and concludes the edit did not work.

Work out which one is authoritative before you change anything:

curl -s https://yoursite.com/robots.txt

If you see # BEGIN Cloudflare Managed content or # BEGIN Cloudflare Bot Preference Sync, Cloudflare is prepending to whatever WordPress serves. Your plugin controls the rest of the file, and Cloudflare's edge rules override both regardless of what the file says.

For the WordPress half, ThinkRank manages robots.txt and robots meta directives from one place, and generates llms.txt rather than leaving you to hand-maintain it — which matters here because llms.txt and the Content-Signal line live in the same discovery layer that Cloudflare is now writing to. It migrates existing settings from Rank Math, Yoast, All in One SEO and SEOPress, so you are not re-entering directives by hand. AIScan detects ThinkRank as an installed plugin during a scan and tailors its fix steps to it, the same way it does for Rank Math and Shopify.

What no plugin can do is change Cloudflare's edge behaviour. If Cloudflare is returning 403 to Googlebot, that is fixed in the Cloudflare dashboard and nowhere else — no WordPress setting will override it.

What this change cannot do for you

Two limits worth stating plainly, because a lot of coverage implies otherwise.

You cannot opt out of Google AI Overviews this way. Google-Extended controls Gemini model training only. Google's documentation says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." AI Overviews and AI Mode are fed by ordinary Googlebot, and the only levers are nosnippet, data-nosnippet, max-snippet and noindex — all of which also cut your normal search snippets. There is no AI-Overviews-specific opt-out, from Cloudflare or anyone else.

Blocking training does not stop AI search citations, and allowing it does not guarantee them. Those are separate crawlers doing separate jobs. Conflating them is the single most common mistake in this topic.

Verifying a claimed AI crawler is real

User-agent strings are trivially forged — as the curl test above demonstrates, plain curl can impersonate GPTBot well enough to trip Cloudflare's rules. If you are making allow/block decisions from your access logs, match the source IP against the operator's published list rather than trusting the string.

All of these were live on 25 August 2026:

Note the Anthropic host. anthropic.com/claudebot.json returns 404; the correct location is on claude.com, per Anthropic's support documentation. Plenty of published crawler lists still carry the dead URL.

For Google and Bing you can also use reverse DNS, which OpenAI and Anthropic do not support because they crawl from cloud IP ranges:

host 66.249.66.1                        # → crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com    # must resolve back to the same IP

Five mistakes people are making with this

1. Treating "AI bots" as one thing. Search, Agent and Training do unrelated jobs with opposite consequences. Blocking all three with one switch is the root of nearly every problem in this article.

2. Using Block where Disallow is the right instrument. Block drops a 403 at the edge on everything carrying a Training tag, Googlebot included. Disallow writes the preference and leaves search indexing alone. If you need organic traffic, Disallow is almost always what you meant.

3. Trusting robots.txt as evidence. It records a preference. It proves nothing about enforcement. Test with curl.

4. Believing Content Signals protects anything. Google has said plainly that it ignores the directive. Publish it if you want your position on record, but do not treat it as a control.

5. Testing only the homepage. The Agent default is scoped to pages that display ads, and Cloudflare decides which pages those are by automated detection with no published accuracy figures. Test an actual article URL, not /.

Maintenance

This is not a set-and-forget configuration.

• Re-test after any Cloudflare security change. Bot rules interact with WAF custom rules and rate limiting, and a rule added for an unrelated reason can catch a crawler.
• Watch the deprecation. The legacy "Block AI bots" toggle goes away on 15 September. If your runbook references it, update the runbook.
• Re-check the user-agent list quarterly. It changes. Cloudflare has moved to a dynamic list via BotBase, and CloudflareBrowserRenderingCrawler already appears in live managed robots.txt output without being in the documented example.
• Watch for the Bot Preference Sync rollout. Announced 21 August as shipping "in the coming week," it generates robots.txt from your Search/Agent/Training configuration so the two layers stop contradicting each other. Cloudflare's reasoning is worth noting: when stated preferences and enforced rules disagree, "some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules."

Where AIScan fits, and where it doesn't

Being straight about this, because the boundary matters.

The curl test above measures enforcement, and it is the authoritative test. Nothing can replace it, because only a request to your origin reveals what your edge actually does.

What AIScan reads is the declaration layer — the part crawlers consult before they ever hit your server — and grades it:

That last row is the one people skip. Winning the access argument is pointless if the page returns an empty DOM to anything that does not execute JavaScript — a failure mode this site had until recently, on this very blog.

A scan is free for 20 a month, and there is a CLI if you would rather gate it in CI so a robots.txt regression fails a build instead of surfacing in a traffic report six weeks later:

npx aiscan-cli example.com

What AIScan cannot tell you: whether your Cloudflare dashboard has Training set to Block. That state is not visible from outside your origin — it only shows up as a 403 in response to a specific user-agent. So run the curl test for enforcement, and scan for everything downstream of it. The two answer different questions and you need both.

The honest summary

The story that spread is that Cloudflare is blocking AI crawlers by default on 15 September. That version was already an overstatement on 1 July, and after the 21 August revision it is simply not what the policy says. New non-publisher domains get no blocks. Search was never blocked. AI search citations are not at risk from this change.

The real risk runs the other way, and it is duller: an old toggle, ticked during a news cycle in 2025, that quietly starts returning 403 to Googlebot. Sixty seconds of curl settles it.