Does Blocking AI Training Protect Your Content?

Quick summary

Too long, didn't read? Here's the honest version.

Every AI crawler operator now says roughly the same thing in different words: honor robots.txt. Anthropic goes further and admits the honest limit of the whole system, in a passage quoted below. The question worth asking isn't "should I block AI training crawlers." It's "what does blocking actually change, and what doesn't it touch."

What "blocking" actually promises, and what it doesn't

A Disallow: / line, an explicit deny rule in robots.txt, or a Content-Signal header asking crawlers not to train on a page are all requests. They tell a well-behaved crawler where not to go. None of them are access control in the way a password or a firewall rule is access control: nothing stops an unauthenticated request from reaching the page unless the server itself refuses it.

That distinction matters because the two things get conflated constantly in how site owners talk about "protecting" content. There are really three separate claims bundled into the word "blocking," and they hold up very differently:

1. Stop future crawls of this content. Robots.txt does this reasonably well, when the operator honors it.
2. Stop the content already crawled from being used in a training run that hasn't happened yet. This depends entirely on the operator's own retraining schedule and whether they re-crawl or reuse a stale corpus.
3. Undo the effect of training that already happened. Nothing does this. Not a robots.txt change, not an opt-out form, not a takedown request to the model provider.

Site owners who add a block after finding their content referenced by a chatbot are usually trying to accomplish (3) with a tool built for (1). That's the gap this article is about.

The sequence usually runs the same way. Someone asks a chatbot a question, recognizes a sentence or a structure that looks like it came from their own site, and reaches for robots.txt the same afternoon. The instinct is reasonable. The file just can't do what the moment is asking of it. A Disallow rule added today changes what happens on the next crawl. It has no effect on a model that already finished a training run using a page fetched months earlier, and it has no effect on outputs the model can already produce from what it learned in that run.

The part that can't be undone

This is the uncomfortable half, and it's worth saying plainly instead of burying it under caveats: once a page has been fetched and folded into a training run, disallowing the crawler afterward does not remove that page's influence from the resulting model. There is no retroactive deletion mechanism, no crawler recall, no way to identify and subtract one page's contribution from billions of trained parameters after the fact.

Anthropic's own crawler documentation states this about as directly as a vendor ever does about their own product. According to Anthropic's crawler support article, fetched and verified on 23 September 2026, blocking by IP address "may not work correctly or persistently guarantee an opt-out," specifically because doing so also stops the crawler from reading the robots.txt file that carries the actual opt-out instruction. Block the wrong way and you don't just fail to opt out. You make the one channel that would have told the crawler to leave unreadable.

The practical corollary is about identity, not intent. A crawler can only honor a block if it identifies itself honestly. According to HUMAN Security's Satori Threat Intelligence team, which measured an average spoof ratio of 1:17 across a two-week study published 9 September 2025, roughly one genuine AI-crawler request arrives for every seventeen requests merely claiming to be one. ai-crawler-user-agent-list-2026 on this blog has the full per-operator breakdown and the verification method that actually distinguishes a real crawler from a spoofed one, in its own words. A block that only checks the User-agent string is a block a spoofed request walks straight past, whatever training question is or isn't at stake.

None of this is an argument against blocking. It's an argument for being precise about which problem it solves. Blocking is a forward-looking control on a specific channel. It is not a rollback button, and treating it like one sets an expectation the mechanism was never built to meet.

Advisory, not a lock: what enforcement data shows

Even limited to its forward-looking job, a block only works if the crawler chooses to respect it. robots-txt-ai-bots-docusaurus and the wider crawler-compliance literature both describe this as advisory rather than access control, and a direct measurement backs it up. The distinction matters legally as much as technically: a robots.txt rule is a published preference, not a locked door, and treating it as the latter overstates what any site can actually guarantee to a content owner asking whether their material is protected.

According to stealth-crawling-user-driven-fetching-debate's measurement on this blog, which ran 525 identity-varied probes across 75 hosts and re-probed every differential to confirm it wasn't noise, a full Disallow: / was backed by an actual server-side refusal only 45 to 67 percent of the time, depending on the operator.

That table describes sites that had already written the rule. Roughly half the time, the file said one thing and the server did another, not because the crawler was misbehaving, but because nobody had wired the declared policy to an actual edge-level refusal. Writing Disallow: / and never testing whether it's backed by anything is functionally the same as not writing it, for a meaningful share of the population.

If you want to know which side of that gap your own site is on, that's a server-configuration question, not a crawler-behavior one, and it's exactly the kind of thing worth confirming with an actual probe rather than reading the file and assuming. Run a scan at AIScan and check B2 (explicit AI-bot rules) for what your robots.txt declares. Read the check's evidence string, not just its pass/fail, since a passing B2 today only confirms the rule exists. It doesn't yet confirm a crawler was refused when it ignored that rule. That's a genuine gap in this rubric and in every scanner in the category; see the section below on what AIScan can and can't tell you here.

Blocking and charging are the same lever with different signage

The clean split between "sites that block AI crawlers" and "sites that charge them" doesn't survive contact with how large publishers actually configure their infrastructure. Several run both policies from the same Cloudflare dashboard, aimed at different crawlers, in the same week.

stackoverflow.com is the clearest example, and it's easy to verify yourself. As of 23 September 2026, verified directly for this article, its robots.txt is served with an HTTP 418 status (technically telling any strictly compliant crawler it may discard the file entirely), while the file's own content carries Content-signal: search=no, ai-train=no and a blanket Disallow: /. Fetching the site as CCBot returns an HTTP 403 challenge. Fetching /llms.txt as ClaudeBot returns an HTTP 403 as well. Meanwhile, the same operator's crawlers that Stack Overflow has licensing arrangements with are billed, not blocked, per cloudflare-pay-per-crawl-should-you-charge's measurement of the same host.

www.theatlantic.com shows the third variant. As of 23 September 2026, also verified directly, fetching its homepage as ClaudeBot returns an HTTP 402 with the plain JSON body {"message":"Please contact the site owner for access."}, a payment-required status attached to no visible price at all. That's a refusal wearing a billing status code, functionally identical to a block for any crawler that can't negotiate a manual deal, but invisible to any automated check that only asks "is this site blocking AI crawlers" and treats 402 as evidence of a licensing program rather than a wall.

The lesson isn't that any one of these choices is wrong. It's that "blocking" and "monetizing" sit on the same policy spectrum rather than being opposites, and a status code alone doesn't tell you which one you're looking at without checking the body.

Who actually blocks, and why the answer depends on your business

The base rate of AI-crawler blocking is not one number. It moves by an order of magnitude depending on what kind of site is being measured, and treating "everyone is blocking AI now" or "almost nobody blocks AI" as a universal fact means picking whichever sample happens to support the point.

That's roughly a 14-to-1 spread between the most defensive population (publishers) and the most permissive one (documentation sites), on a question that gets discussed as if it had one answer. And 35.4% of the same publisher sample named no AI crawler at all in their robots.txt, which is its own data point: even inside the most block-happy population measured, a third had made no explicit decision either way.

The reason isn't mysterious once you separate the populations by what they're protecting. A newsroom's entire product is the text itself, sold once as a subscription and again, potentially, as a training license. Every free read by a crawler is a read the paywall didn't collect. A documentation site's entire product is downstream: developers using the tool correctly, which an AI assistant citing the docs accurately makes more likely, not less. Blocking training crawlers on a docs site protects nothing the site is trying to sell, and actively works against the thing it's trying to achieve, which is getting the reference material read wherever the developer happens to be asking.

If you're deciding a policy, start from that question rather than from what a competitor did: what exactly does a crawl of this content cost you, and does an AI answer built from it help or hurt the outcome you're paid on?

What blocking does achieve

None of the above is a case against blocking. It's a case for being specific about what it's good for:

• It stops honest crawlers going forward. Every operator's own documentation says they check robots.txt before crawling, and the enforcement data above shows a meaningful share of declared blocks are backed by an actual refusal, just not all of them. For the operators and crawlers that do honor the file, every future fetch that would have happened doesn't.
• It creates a documented record. If a dispute over crawling ever needs a paper trail, "we declared this in robots.txt on this date" is evidence a verbal preference is not. That record also has a date attached, which matters if a licensing negotiation or a legal question ever turns on when a site's position changed.
• It separates search visibility from training use. OpenAI, for instance, lets a site allow OAI-SearchBot for search results while disallowing GPTBot for training, independently. That distinction only exists because sites bother to write it down; a site that never configures either token gets whatever the operator's default treatment is, which is not necessarily the split it would have chosen.
• It's the precondition for every enforcement mechanism that comes after it. Cloudflare's Pay Per Crawl, Content-Signal headers, and edge-level bot management all start from a declared preference. You can't enforce a policy you haven't stated, and a site that skips the declaration step has nothing for a bot-management product to act on later.
• It costs almost nothing to maintain. Compared with every other lever discussed in this article, editing a text file is the cheapest possible move, which is itself a reason to get it right rather than skip it because it "won't really work anyway."

Treat a block as what it is, a declared, forward-looking, imperfectly-honored preference, and it does real work. Treat it as a wall and you'll be disappointed by results it was never designed to produce.

When blocking is the right call, and when it's theatre

Blocking is the right call when the goal matches what the mechanism actually does: stopping future crawls by operators who honor robots.txt, documenting a policy for later reference, or separating a training use from a search-visibility use you still want. It's closer to theatre when the goal is "stop AI companies from having already used my content," which the block cannot touch, or when it's added without ever confirming the rule is backed by anything at the server level, in which case it may not even be stopping future crawls from the crawlers it names.

The middle case is the common one: a site adds the rule, feels like it has addressed the concern, and never checks whether the declaration and the server's actual behavior agree. Given that a fully declared block is only enforced 45%–67% of the time across the operators measured above, that gap is worth closing before assuming the policy is doing its job.

There's a fourth situation worth naming separately: a site that blocks reflexively, without checking whether the crawler it's caught is one it would rather allow. ChatGPT-User, Claude-User, and Perplexity-User fetch pages in response to a specific person's question, not as part of a bulk training run, and each operator documents that distinction in its own crawler policy. A wildcard rule that catches all six major tokens under one Disallow: / also blocks the user-driven fetchers, which means the site stops appearing in the exact assistant answers where a real visitor was asking about it by name. Reflexive blocking and deliberate blocking produce the same-looking robots.txt file and very different outcomes.

How to declare the block you actually mean

Most of the confusion in this area comes from writing one blanket rule when the underlying goal is more specific than "keep AI out." OpenAI's own crawler documentation makes the point directly: disallowing GPTBot while allowing OAI-SearchBot lets a site keep search visibility and still opt out of training, because the two crawlers serve different purposes and can be governed by separate lines in the same file.

Two practical notes worth acting on immediately. First, a rule aimed at one crawler's training use should live in that crawler's own User-agent: group, not in the wildcard * group, because a wildcard rule catches every crawler that doesn't have its own more specific group, including ones you may not have intended to affect. Second, write the rule, then test it the way this article's enforcement section describes: send an actual request carrying the blocked identity and confirm the server refuses it, rather than trusting that a file existing means a file is being read.

If your site runs on WordPress and you'd rather not hand-edit robots.txt directly, this is exactly the configuration ThinkRank manages from one settings screen, alongside the robots meta tags, schema markup, and llms.txt file a training-crawler policy usually needs to stay consistent with. Changing crawler policy from one place cuts the risk of a rule written there getting silently overridden by a setting somewhere else in the stack.

Where AIScan fits, and where it doesn't

AIScan checks B1 (Content Signals adoption) and B2 (explicit AI-bot rules in robots.txt) against your live site, which answers the "did you declare a policy" half of this question in under a minute. Paste a URL at aiscan.site or run npx aiscan-cli yoursite.com, no account required.

What it can't yet tell you: whether a declared block is actually enforced at the edge for a given crawler identity. That requires sending real, identity-varied requests the way stealth-crawling-user-driven-fetching-debate's research did, and no scanner in this category, AIScan included, currently automates that probe as part of a routine check. A passing B2 confirms the file exists and names the right tokens. It doesn't confirm the server refuses the crawlers it names. Read that as a known limitation rather than a claim we're not making: you can finish verifying enforcement yourself with the identity-varied curl requests shown above, with or without running our scan first.

Common mistakes about AI blocking

• Treating a robots.txt edit as a takedown request. It changes nothing about content already used in a completed training run.
• Blocking by IP address instead of, or on top of, robots.txt. Anthropic's own documentation warns this can break the crawler's ability to read the actual opt-out file.
• Never testing whether the block is enforced. Writing the rule and reading it back confirms the file exists. It doesn't confirm the server acts on it.
• Reading a 402 as proof of a licensing program. Some 402 responses carry a real price. Others, like www.theatlantic.com's, carry a contact-us message and nothing else; check the body, not just the status code.
• Applying a publisher's blocking posture to a site with a different business model. A documentation site protecting nothing it sells directly gains little from a policy built for subscription media.
• Assuming one crawler identity speaks for an operator. ChatGPT-User, Claude-User, and Perplexity-User are described by their own operators as user-initiated fetches distinct from training crawls, and several sites in the enforcement data above treat them differently from GPTBot, ClaudeBot, and PerplexityBot for that reason.
• Copying a competitor's robots.txt wholesale. A rule written for a subscription newsroom's business model doesn't automatically fit a documentation site, a marketplace, or a personal blog, and pasting it in without checking which tokens it actually names can block crawlers you never meant to affect.
• Never revisiting the rule as crawlers change. New tokens appear, existing ones split into training and user-driven variants, and an operator can rename or relocate its published crawler list, as Google has done with its IP-range files. A robots.txt written once in 2024 is unlikely to still name every crawler worth a decision today.

Test what your block actually does, not what you wrote

Start by confirming what your site has actually declared: scan it at AIScan and read B1 and B2's evidence strings, not just their pass/fail status. Then decide, deliberately, which of the three claims from the top of this article you're actually trying to make (stopping future crawls, separating search from training, or something a robots.txt rule was never going to deliver) and pick the mechanism that matches. For the full walk-through of every check in this rubric, including the ones this article flags as incomplete, see AIScan's guides.