---
title: "Stealth Crawling vs User-Driven Fetching in 2026: What 75 Sites Actually Do"
slug: stealth-crawling-user-driven-fetching-debate
published: 2026-09-09T03:26:43.931838+00:00
updated: 2026-09-09T03:26:43.931838+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: check:B1, check:B2, check:B3, check:D1, AI crawlers, robots.txt, stealth crawling, Perplexity, Web Bot Auth, bot access
description: "Cloudflare de-listed Perplexity for stealth crawling. We sent 7 identities to 75 sites: robots.txt splits crawling from user fetching, 40 of 44 servers do not."
url: https://aiscan.site/blog/stealth-crawling-user-driven-fetching-debate
---

On 4 August 2025 Cloudflare removed Perplexity from its verified bot list and published the evidence. Perplexity's answer was that fetching a page because a person asked for it is not the same act as crawling, and that the distinction "isn't just technical". Thirteen months later the argument is still open, and almost nobody has checked what websites themselves do about it.

So we checked. On 9 September 2026 we sent seven different identities to the same 75 sites: an ordinary desktop browser, and the training crawler and user-driven fetcher published by each of OpenAI, Anthropic and Perplexity. Then we read the same 75 robots.txt files and compared what each site says with what it does.

The two halves disagree, and the disagreement is the story.

## Quick summary

| Question | What we measured, 9 September 2026 | Number |
|---|---|---|
| Do publishers write the distinction into robots.txt? | Sites fully blocking the training crawler but not the user-driven fetcher, across three operators | **26 declarations** on 64 readable files |
| Do publishers enforce the distinction at the edge? | Hosts giving an operator's crawler and its user-driven fetcher the same HTTP answer | **40 of 44** |
| Is a `Disallow: /` backed by an actual refusal? | Share of robots.txt full blocks that the edge also enforced | **45% to 67%**, depending on the token |
| Do the three operators agree on whether robots.txt binds a user-driven fetch? | Positions in their own documentation | **No. Three different answers** |
| Does the edge sort by purpose or by company? | Hosts refusing Anthropic or Perplexity but no OpenAI identity | **9**, against 2 the other way |
| Can anyone prove who they are yet? | AIScan check B3, Web Bot Auth key directory, across 499 scanned sites | **3.8% pass** |
| Does AIScan see any of this today? | Every check fetches with our own identity | **No, and that is our open bug #6** |

Every figure below carries the date it was measured. The raw method is in the "how we measured" section so you can re-run it.

## The argument, and why it is still open

Cloudflare's post, published 4 August 2025, reports that Perplexity kept fetching after a block by switching away from its published user agent to one shaped like ordinary desktop Chrome. The trigger is the part that matters, and Cloudflare states it verbatim: the swap happened *"when their declared crawler was blocked"*.

The test was clean. Cloudflare bought brand-new domains that no search engine had indexed, put a blanket `Disallow` in robots.txt, added WAF rules against both declared Perplexity crawlers, then asked Perplexity questions about those domains and got detailed answers back. It measured the declared agent at 20 to 25 million daily requests and the undeclared one at 3 to 6 million.

The same post names the control condition, and this is the part worth keeping. Cloudflare ran the identical test against ChatGPT and reports that the fetcher read the robots file first and then, verbatim, *"stopped crawling when it was disallowed"*, with no follow-up attempts from any other agent.

Perplexity published a response. We could not retrieve it: fetching that post on 9 September 2026 returned HTTP 403 from a live host that resolves normally, so we are not quoting it and you should not trust anyone who quotes it without saying where they got it. What we can source first-party is Perplexity's own crawler documentation, which states the position plainly. More on that in a moment.

Two things have changed since August 2025, and both make the question sharper rather than softer. Identity stopped being a courtesy and became a billing input. And the operators themselves stopped agreeing with each other in writing.

## What the standard actually says about an automated client

Everyone in this argument cites RFC 9309. Very few people quote its opening section, which is where the definitions live. Fetched from the RFC Editor as plain text on 9 September 2026:

> "This document applies to services that provide resources that clients can access through URIs as defined in [RFC3986]. For example, in the context of HTTP, a browser is a client that displays the content of a web page."

> "Crawlers are automated clients."

And then, four paragraphs later, the sentence that decides more arguments than any other line in the document:

> "These rules are not a form of access authorization."

Read together, those three sentences do something people rarely notice. The standard draws its line between a browser and a crawler on *automation*, not on who benefits. A fetch that happens because a person typed a question is, by that definition, genuinely ambiguous: it is automated in mechanism and human in origin. The RFC never resolves it, because in 2022 the case barely existed.

And the third sentence says robots.txt was never the enforcement layer. It is a request. Which raises an empirical question nobody had answered: when a site writes `Disallow: /` for an AI crawler, is anything behind it?

## The three operators do not agree with each other

All three positions below were fetched from each operator's own documentation on 9 September 2026 and are quoted verbatim.

| Operator | User-driven token | What its own documentation says about robots.txt |
|---|---|---|
| **OpenAI** | `ChatGPT-User` | *"ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply."* |
| **Anthropic** | `Claude-User` | *"Claude-User allows site owners to control which sites can be accessed through these user-initiated requests."* |
| **Perplexity** | `Perplexity-User` | *"Since a user requested the fetch, this fetcher generally ignores robots.txt rules."* |

That table is the single most useful thing we found, because of who is in which column.

The position Cloudflare de-listed Perplexity over is written, in slightly gentler words, in OpenAI's own bot documentation. OpenAI says robots.txt "may not apply" to a user-initiated fetch. Perplexity says the same fetcher "generally ignores" robots.txt. Those are the same claim at different volumes, published by the company Cloudflare held up as the good example and the company it removed from its verified list.

The difference between them is not the doctrine. It is the behaviour Cloudflare observed when a block was applied anyway: one operator stopped, the other rotated user agents and ASNs. That distinction is real and it is the one worth arguing about. The doctrine is common ground.

Anthropic sits in the third position, and it is the only one of the three that tells site owners the control is theirs. Its documentation spells out the cost of switching `Claude-User` off, in its own words: the content stops being retrievable when a reader asks, which *"may reduce your site's visibility"*. That is a trade offered honestly, not a claim of exemption.

## How we measured it

Seventy-five hosts: news and reference publishers, technical documentation, health, personal finance, food, travel, retail and three storefronts. Each got seven `GET /` requests, one per identity, from a datacentre address on 9 September 2026. All 525 probes were recorded, and every response body was written to disk so a status code was never read on its own.

The identities were the full published user-agent strings for `GPTBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-User`, `PerplexityBot` and `Perplexity-User`, plus a desktop Chrome control.

Three rules governed the reading, and they matter more than the sweep:

**The browser control decides what a refusal means.** Of the 75 hosts, only 44 returned HTTP 200 to an ordinary browser from this address. The rest returned 10 refusals at 403, five at 402, five 302 redirects, two 301, two 401, and one each of 303, 307, 406, 429 and 202, with two connection failures. **A 403 from a datacentre IP is not evidence of a bot decision**, so every finding below is computed on the 44 hosts we know serve us normally, and that denominator is printed every time.

**Every differential was re-probed twice.** Sixteen host-and-operator differences appeared in the first pass. We fetched all twelve affected hosts again, twice, spaced apart. **All sixteen reproduced in three rounds of three.** These are deterministic policies, not noise.

**And one uncomfortable note we are printing rather than hiding.** The stealth agent Cloudflare fingerprinted was `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ... Chrome/124.0.0.0`. Our browser control is the same string at Chrome 140. The measurement in this article is only distinguishable from the behaviour it describes by intent, which is exactly the problem the article is about. We declare it here because there is no header that declares it for us.

## What publishers write down: robots.txt does distinguish

Sixty-four of the 75 robots.txt files were usable. The eleven exclusions are themselves worth publishing: five Stack Exchange properties serve robots.txt with **HTTP 418**, two hosts returned 403 with an HTML page, one returned 406, one failed to connect, and **two returned HTTP 200 with an HTML error page instead of a file**, which is the soft-404 pattern that quietly corrupts every well-known-path check in every scanner including ours.

Across the 64 readable files, here is how often each token gets named in a `User-agent` line at all:

| Token | Role | Named | Share |
|---|---|---|---|
| `ClaudeBot` | training | 31 | 48.4% |
| `CCBot` | training | 31 | 48.4% |
| `GPTBot` | training | 28 | 43.8% |
| `PerplexityBot` | search crawler | 27 | 42.2% |
| `Google-Extended` | training token | 27 | 42.2% |
| `ChatGPT-User` | user-driven | 24 | 37.5% |
| `Claude-User` | user-driven | 20 | 31.2% |
| `Claude-SearchBot` | search index | 20 | 31.2% |
| `OAI-SearchBot` | search index | 18 | 28.1% |
| `Perplexity-User` | user-driven | 18 | 28.1% |

The user-driven token is named less often than its sibling crawler in all three cases, but not by much, and the gap is smaller than we expected. Between 28% and 38% of these publishers have gone to the trouble of writing a rule for a fetcher that only exists because a person asked a question.

Then we resolved each token to its effective policy, taking the token's own group where one exists and falling back to the `*` group, and counted only a full `Disallow: /` with no overriding `Allow`:

| Operator | Crawler fully blocked | User-driven fully blocked | Both | **Crawler only** | User-driven only |
|---|---|---|---|---|---|
| OpenAI | 22 | 16 | 14 | **8** | 2 |
| Anthropic | 35 | 23 | 23 | **12** | 0 |
| Perplexity | 28 | 22 | 22 | **6** | 0 |

**Twenty-six declarations across 64 files deliberately block the automated crawler and let the user-driven fetch through.** That is not an accident of the `*` group: in every one of those cases the site wrote a named group for the crawler and either wrote a permissive one for the fetcher or left it to a permissive default.

So the distinction Perplexity's defence rests on is not a lawyer's invention. A real and consistent minority of publishers have decided that a machine reading their site to build a model is a different act from a machine reading their site because a reader asked, and they have written it down.

## What publishers enforce: the edge does not

Now the same 44 hosts at the HTTP layer.

| Pattern across the six AI identities | Hosts |
|---|---|
| Refused none of them | 26 |
| Refused all six | 3 |
| Refused some and allowed others | 15 |
| **Gave an operator's own two tokens different answers** | **4** |

![Two panels comparing AI crawler policy on 75 sites: robots.txt full blocks per token against what servers actually returned, measured 9 September 2026.](https://gqdxuwsuuasfjeeqafyq.supabase.co/storage/v1/object/public/blog-covers/stealth-crawling-user-driven-fetching-debate-figure.jpg)

**Forty of 44 hosts treat a company's crawler and that company's user-driven fetcher identically.** The four that split are the BBC, Slate, Reddit and WebMD, and each split reproduced three times out of three, so they are real. Four.

Put the two halves side by side. Twenty-six declarations in the text file say the two acts are different. Four servers behave as though they are.

This is what RFC 9309 section 1 means in production. The `Disallow` line is a request, the edge is where authorization actually happens, and on 40 of 44 sites the edge has never been told there is a difference. A publisher can spend an afternoon writing a careful, purpose-aware robots.txt and ship a WAF rule that matches on the word "Claude".

## Half of every Disallow is not backed by anything

Forty hosts gave us both a usable robots.txt and a browser-200 control. For each token we asked one question: this file says `Disallow: /`, so does the server actually refuse?

| Token | robots.txt blocks | Edge also refuses | Enforced | robots.txt allows | Edge refuses anyway |
|---|---|---|---|---|---|
| `GPTBot` | 14 | 7 | **50%** | 26 | 1 |
| `ChatGPT-User` | 11 | 5 | **45%** | 29 | 4 |
| `ClaudeBot` | 19 | 9 | **47%** | 21 | 0 |
| `Claude-User` | 12 | 7 | **58%** | 28 | 0 |
| `PerplexityBot` | 14 | 8 | **57%** | 26 | 2 |
| `Perplexity-User` | 12 | 8 | **67%** | 28 | 3 |

Between a third and a half of every full block in this sample is a sign on a door that opens anyway.

We want to be careful about what that does and does not say. It is not a finding that publishers are careless, and it is certainly not an argument that a crawler may ignore a `Disallow` because nothing stops it. The RFC is explicit that these rules were never access control, and a publisher who writes one is expressing a preference exactly as designed. What the number says is narrower and more useful: **if your AI policy lives only in robots.txt, roughly half the operators who choose to ignore it will succeed, and you will never see it in a log you look at.** That is the gap Cloudflare's test was designed to expose, and it exists on ordinary sites too.

The right-hand columns are the quieter half of the result. Between 0 and 4 hosts refuse a token they never disallowed. Enforcement and declaration are two systems that mostly do not talk to each other, in both directions.

## The edge sorts by company, not by purpose

The 15 hosts that refuse some identities and allow others were the interesting group, so we looked at the shape of each pattern rather than the count.

Eight of them do the same thing: refuse both Anthropic tokens and both Perplexity tokens, allow both OpenAI tokens. **Nine hosts refused at least one Anthropic or Perplexity identity while refusing no OpenAI identity, against two the other way round.**

The payment-required responses split along the same seam. Over all 75 hosts, counting only HTTP 402:

| Identity | 402 responses |
|---|---|
| `GPTBot` | 6 |
| `ChatGPT-User` | 6 |
| `ClaudeBot` | 15 |
| `Claude-User` | 13 |
| `PerplexityBot` | 14 |
| `Perplexity-User` | 14 |

This reproduces, on a different host list three days later, what we measured on [6 September across 119 publishers](https://aiscan.site/blog/cloudflare-pay-per-crawl-should-you-charge), where the Stack Exchange network charged all three Anthropic tokens and PerplexityBot and let all three OpenAI tokens through free. That result was re-verified on 9 September 2026 and nothing has moved: Stack Overflow answers a ClaudeBot user agent with `HTTP 402`, `crawler-error: MissingCrawlerPrice` and `crawler-price: USD 0.5`, byte-identical to the 6 and 8 September readings.

We are not going to speculate about why. A commercial agreement, a differing view of each operator's behaviour, a default in somebody's bot management dashboard and plain inertia would all produce this picture, and none of them is visible from outside. The point is structural: **the layer that actually decides sorts requests by which company sent them, and the crawl-versus-fetch distinction that both sides of the argument care about is not a variable it reads.**

Which tokens belong to which operator, and how many there now are, is the subject of our [AI crawler user-agent list](https://aiscan.site/blog/ai-crawler-user-agent-list-2026). Thirty-three documented crawlers, five operators running a three-way split, and reverse DNS verification that works for four of them and fails for the three most impersonated.

## Identity became a billing question

Here is what has genuinely changed since August 2025, and it is the reason this argument now has an ending nobody planned.

Cloudflare's Pay Per Crawl bills on a Web Bot Auth signature. The payment header has to sit inside the signed components of the request. We tested the failure mode in an earlier run and it is unambiguous: **an unsigned request offering the exact asking price returns `403 PaymentFailed`.** Money is not enough. You have to be able to prove who you are before anyone will take it.

The register of who charges what is readable by verified crawlers and by nobody else. Verified on 9 September 2026, fetching Cloudflare's Pay Per Crawl discovery endpoint returns, verbatim:

```
HTTP/2 403
{"error":"Only verified bots can use this endpoint"}
```

So the answer to "who gets to fetch a page for a human" is quietly resolving into "whoever can be identified, because that is who can be invoiced or refused." Not whoever is honest about their purpose. Whoever carries a signature.

And publishers have built almost none of the receiving end. Verified on 9 September 2026 against our own corpus of 499 scanned sites, one latest scan per host, **check B3, the Web Bot Auth key directory, passes on 19 sites. That is 3.8%.** Content Signals in robots.txt (B1) passes on 110, or 22.0%. Explicit AI bot rules in robots.txt (B2) pass on 38.7%.

Meanwhile publishers already enforce identity in the crude way, and correctly. In a [110-publisher sweep on 7 September 2026](https://aiscan.site/blog/google-ai-overviews-what-you-can-control) we found that **six of 91 hosts served an ordinary browser a real robots.txt and refused a self-declared Googlebot outright**, because they verify by IP range and reverse DNS rather than trusting a header. They are right to. A user-agent string is a claim, and the whole stealth-crawling argument is downstream of the fact that a claim is all it has ever been.

## Where AIScan fits, and where it doesn't

The honest answer is that AIScan grades the declaration layer of this article and none of the enforcement layer.

**What it does check.** Run a scan and you get **B2** (are any AI crawlers addressed in robots.txt at all), **B1** (is a Content Signals directive present), **B3** (is a Web Bot Auth key directory published) and **D1** (is robots.txt reachable at the top-level path where RFC 9309 requires it, graded under [discoverability](https://aiscan.site/docs/checks/discoverability)). Those four are the whole of what a site declares about bot access.

**What it does not check, and this is our open product bug number 6.** Every AIScan check fetches with our own identity. **We cannot see per-crawler policy at all.** A site can hold 100 out of 100 on our rubric and answer `402` to every AI crawler on the internet, and our report will say nothing about it, because we never asked as one. The proposed fix is a candidate check we call B4: one extra request per scan with a documented AI crawler user agent, compared against the browser fetch, with a browser control built in. It is the cheapest item on our backlog and it is not shipped.

**And a defect this run found in B3 itself, which we are publishing rather than filing quietly.** Grouping the evidence strings across all 499 sites returns exactly three values: *"not published (optional / experimental)"* on 439 sites, and the literal string `HTTP 200` on **both** the 41 sites scored `partial` **and** the 19 scored `pass`. The check reports two different verdicts with one identical piece of evidence, and never says what the key directory actually contained. That is the same one-signal failure we have already published against C1, C2, P2, P3, M1 and C3, and it takes the family to ten. The fix is one line: assert the key, not the status code.

**A reporting note on B1 while we are here.** B1 never fails. All 499 results are `pass` or `info`, so "Content Signals adoption is 22%" means 110 passes and 389 informational results, not 389 failures. Anyone quoting our numbers, us included, should say so.

If you want the full picture of what the rubric does and does not reach, the [state of AI agent readiness study](https://aiscan.site/blog/state-of-ai-agent-readiness-2026) publishes every check's pass rate and every known defect against the same population definition.

## What to do about it on your own site

Start with the scan, because it takes one command and tells you which of the four declaration checks you currently pass:

```
npx aiscan-cli yoursite.com
```

No terminal? The same scan runs from the box on [our home page](https://aiscan.site/), free and without an account. The four rows this article is about are **B1**, **B2**, **B3** and **D1**, and each one is documented under [bot access](https://aiscan.site/docs/checks/bot-access).

Then close the gap the scan cannot see for you, in this order:

**1. Decide the crawl-versus-fetch question deliberately, and write both tokens.** If you want to stay retrievable when a reader asks an assistant about you while staying out of training corpora, that is a coherent position and 26 declarations in our sample take it. It requires naming the user-driven token explicitly, because a named group for the crawler does not cover the fetcher.

**2. Check whether your edge agrees with your file.** This is the step almost everyone skips, and it is two commands. Fetch your own homepage as a browser and again as a crawler you have disallowed, and compare:

```
curl -sI -o /dev/null -w '%{http_code}\n' \
  -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36' \
  https://yoursite.com/

curl -sI -o /dev/null -w '%{http_code}\n' \
  -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)' \
  https://yoursite.com/
```

Two 200s mean your `Disallow` is a preference and nothing more, which is exactly what the standard says it is. Whether that is acceptable is your call, but it should be a call rather than a surprise.

**3. On WordPress, put robots.txt, robots meta, schema, sitemaps and llms.txt behind one plugin.** The reason this section exists is that the commonest cause of a robots.txt that does not say what its owner thinks it says is three plugins each generating a version of the same file. [ThinkRank](https://thinkrank.ai) handles all five surfaces from a single settings screen, so there is one source of truth for the file, and it migrates from Rank Math, Yoast, All in One SEO and SEOPress, so adopting it costs nothing in re-entered settings. Rank Math and Yoast are both stronger than ThinkRank on classic on-page SEO workflow and content analysis, and if that is the work you spend your week on, they are the better fit; neither manages the AI-crawler surfaces in one place. Pick on that basis rather than on ours.

**4. If you publish an llms.txt, remember it is not on anybody's free list.** Cloudflare's always-free Pay Per Crawl paths are `/robots.txt`, `/sitemap.xml`, `/security.txt`, `/.well-known/security.txt` and `/crawlers.json`. `llms.txt` is not among them, which is why Stack Overflow's returns `402` to a crawler and a file to a browser. If you need one built, our [llms.txt generator](https://aiscan.site/llms-txt-generator) writes a spec-shaped file from your site.

**5. If the traffic is the actual problem, read the cost side before the identity side.** Roughly half the web cannot answer "nothing changed" to a conditional request, which we measured at [51.8% across 240 probes](https://aiscan.site/blog/ai-crawler-traffic-waste-2026), and a re-crawl of unchanged content is billable under Pay Per Crawl. Fixing your cache validators removes more crawler load than any robots.txt rule will.

## The part that is still genuinely unresolved

Here is the case for each side, as fairly as we can put it.

**For the publishers.** A preference expressed in the place the standard designates should be honoured by anyone who claims to be a good citizen, whatever their reading of who benefits. Cloudflare's test was not a trap; it was a blank domain with a blanket disallow and a WAF block, which is about as clear as a signal gets. Rotating user agents and ASNs in response to a block is not a philosophical position about user agency, it is evasion, and the fact that the doctrine underneath it is shared with OpenAI does not make the evasion shared too.

**For the operators.** RFC 9309 defines a crawler as an automated client and says nothing about a client that acts once, for one person, on one URL, and does not recursively traverse anything. That gap is real. A browser extension that fetches a page for you is not a crawler; a model that does the identical fetch on the identical trigger arguably is not either. And the standard itself says these rules are not access authorization, which cuts both ways: a publisher who wants a fetch stopped has a mechanism for that, and it is not this file.

**What our data adds.** Both sides are arguing about a distinction that the enforcement layer does not implement. Twenty-six declarations in our sample take a purpose-aware position, and 40 of 44 servers ignore it. If the question ever gets settled it will not be settled by robots.txt, because robots.txt is where the distinction goes to be written down and not to be enforced.

**And the direction it is actually resolving in is worth naming, because it is not the one either side is arguing for.** Signed identity is becoming the price of admission, not because anyone won the ethical argument but because you cannot bill an anonymous request. Cloudflare's BotBase now validates Web Bot Auth signatures as part of operator onboarding. Pay Per Crawl refuses an unsigned request that offers the full price. The price register is readable by verified bots only. In that world "was this fetch for a human" stops being the question anyone asks, because "who signed it" is answerable and "why did you want it" never will be. Whether a web where only cryptographically identified clients can transact is better than one where a text file is politely obeyed is a real question, and 3.8% of sites are ready for it either way.

## Run both layers against your own site

Find out what you currently declare before deciding what you should. One command returns B1, B2, B3 and D1 with the exact evidence string behind each verdict:

```
npx aiscan-cli yoursite.com
```

The same scan runs from [our home page](https://aiscan.site/) if you prefer. [Bot access](https://aiscan.site/docs/checks/bot-access) explains what each of those four rows means and how to fix it. Then run the two-curl edge test from the section above, because that is the layer we cannot reach for you and it is the layer that decides. Everything we have published on crawler policy, robots.txt and agent readiness is indexed at [aiscan.site/guides](https://aiscan.site/guides).