Cloudflare Pay Per Crawl in 2026: Should You Charge AI Crawlers? 119 Sites Measured

Stack Overflow will sell you its homepage for fifty cents. Point an AI crawler at it and the server answers HTTP 402 Payment Required, crawler-price: USD 0.5, and a JSON body naming Cloudflare's Pay Per Crawl by name. The status code that RFC 9110 still describes as "reserved for future use" is now a live price tag on one of the twenty most-visited sites on the web.

That changes the question this article was going to ask. "Should you charge AI crawlers" stopped being hypothetical some time before 6 September 2026, and the useful version now is narrower: what does charging actually get you, what does it cost you, and what should you do first instead.

To answer it we probed 119 publisher and reference sites with an AI crawler user agent on 6 September 2026 and recorded exactly what came back. Sixteen asked for money. Only six named a price.

Quick summary

The honest answer for most readers is still "not yet". Pay Per Crawl remains a closed beta, according to Cloudflare's own documentation as fetched on 6 September 2026, and it only bills crawlers that have already onboarded to Cloudflare and identified themselves with Web Bot Auth. The people it works for today are the ones with content valuable enough that Anthropic, Perplexity and Meta would rather pay than skip.

Best for large publishers: Pay Per Crawl, because you have the traffic and the bargaining power. Best for everyone else: a 402 block response with a licensing address in it, which costs nothing to run and opens a conversation. Best first move for every site: conditional requests, because they reduce the bill you are already paying.

What a 402 actually looks like on the wire

Here is the whole thing, fetched from stackoverflow.com on 6 September 2026 by a request carrying a ClaudeBot user agent:

HTTP/2 402
date: Sun, 06 Sep 2026 03:10:56 GMT
content-type: application/json
crawler-error: MissingCrawlerPrice
crawler-price: USD 0.5
server: cloudflare

And the body:

{"x402Version":0,
 "accepts":[{"scheme":"deferred","network":"cloudflare","resource":"/",
             "maxTimeoutSeconds":30,"maxAmountRequired":"0.5","asset":"USD"}],
 "error":"Payment is required to access content. Please refer to the Cloudflare Pay Per Crawl documentation for more information."}

Two things in there are worth stopping on.

The crawler-price and crawler-error headers are the ones named, verbatim, on Cloudflare's Crawl pages reference, down to the error code. This is not a lookalike. It is the product.

The body is an x402 payload. Cloudflare's Pay Per Crawl documentation never uses the word x402 anywhere, but the wire format announces itself as x402Version: 0 with scheme: "deferred" and network: "cloudflare". The protocol everyone associates with crypto micropayments is also the envelope Cloudflare uses for a Stripe-settled invoice. Nobody appears to have published that before, and it matters for anyone building a crawler: the same parser handles both.

RFC 9110 §15.5.3, the current HTTP semantics standard, still says of this status code: "The 402 (Payment Required) status code is reserved for future use." The future turned up without an errata.

We probed 119 publishers as an AI crawler. Sixteen asked for money.

Method. 119 news, magazine, reference, recipe, finance and Q&A sites, chosen by hand from public brands and kept entirely separate from our scan corpus. One GET / each, user agent ClaudeBot/1.0, no Web Bot Auth signature, 6 September 2026. We recorded the status code, the crawler-price and crawler-error headers, the server header and the response body. A second pass sent fourteen different crawler user agents at the two most interesting hosts.

Sixteen of 119 is higher than we expected and much higher than any published figure. But the split inside those sixteen is the real finding.

Only 6 of 119 sites, 5.0%, quoted a machine-readable price. All six are Stack Exchange properties: stackoverflow.com, stackexchange.com, superuser.com, serverfault.com, askubuntu.com and math.stackexchange.com, all at USD 0.5, all with identical headers. One publisher, one decision, six domains.

Three caveats travel with these numbers. The sample is a hand-built list of large English-language publishers and is not representative of the web. An unsigned request is not what a real crawler sends, so a site could behave differently for a verified bot. And a 402 seen from one datacentre IP on one day is a snapshot, not a policy.

Two ways to say "pay me", and only one of them has a price

The ten priceless 402s are not failures. They are a different product being used deliberately.

Cloudflare's Manage AI crawlers page, last updated 28 July 2026, lets any paid-plan customer choose the status code returned to a blocked crawler from exactly two values, verbatim: "403 Forbidden: Use this option if you wish to indicate that you do not want the AI crawler to access your content. 402 Payment Required: Use this option if you wish to indicate that the AI crawler must pay to access your content." You can also write the response body, and Cloudflare says why: "This provides you with a channel to open dialogue with the AI crawler owner, and to inform the AI crawler how to properly license their content, thereby creating a direct path from crawling attempt to commercial agreement."

Six sites in our sample are running that feature word for word. allrecipes.com, investopedia.com, realsimple.com, seriouseats.com, thespruce.com and verywellhealth.com all return 402 to an AI crawler with a byte-identical body:

Payment Required - If you wish to license content from People Inc,
please contact contentlicensing@people.inc

That is a licensing desk implemented as an HTTP status code, and it needs no beta invitation. www.theatlantic.com runs a plainer version, {"message":"Please contact the site owner for access."}. www.usatoday.com returns 402 with a message pointing at a third-party toll service rather than at Cloudflare, which is worth knowing before anyone reads a 402 as proof of Pay Per Crawl. And www.telegraph.co.uk returns 402 attached to an unusual-activity security notice, which reads more like a misconfigured challenge than a commercial position.

So the practical taxonomy for a 402 in 2026 is three ways, not one: a real per-crawl price, a licensing invitation, or a bot challenge wearing the wrong number.

Who Stack Overflow charges, and who it walks straight in

Cloudflare lets a site owner pick an action per crawler. We sent fourteen crawler user agents at stackoverflow.com and at www.theatlantic.com on 6 September 2026 to see how that looks from outside.

Both publishers let all three OpenAI tokens through free and charge all three Anthropic tokens. Both let the two classic search engines through untouched, which matches Cloudflare's own warning on the Select crawlers to charge page that setting a Search Engine Crawler to Block or Charge "may negatively impact your site's SEO performance".

Two things this table does not prove. It does not prove any commercial agreement exists behind the OpenAI allowance, only that the allowance exists. And a user agent string is a claim, not an identity, which is the whole reason Pay Per Crawl bills on Web Bot Auth signatures rather than on the User-Agent header. If you want to know which tokens exist and how far you can trust one, our AI crawler user agent list has the verified inventory and the reverse-DNS tests that show where verification breaks.

One correction we owe the reader, because we nearly published it wrong. stackoverflow.com/questions returns 403 to an AI crawler, which looks like a targeted block on the answers. It is not. The response carries cf-mitigated: challenge and a "Just a moment…" interstitial, and an ordinary desktop browser user agent from the same address gets the same 403. That is a managed challenge on a path, not a pricing decision. The short answer URLs /q/<id> and /a/<id> do return 402 at USD 0.5.

What the docs say it costs, and what the dashboard does not show you

All of this is fetched from Cloudflare's own documentation on 6 September 2026, and quoted in its own words where the wording matters.

And the sentence a publisher should read twice, from the Manage payouts page: "Your accrued balance is not currently visible in the dashboard. You can request balance updates from your Cloudflare team." Payouts are also "subject to settlement periods and minimum payout thresholds" that the documentation does not quantify.

That is a real limitation and it deserves to be said plainly. You can set a price today and not be able to see what you have earned without emailing someone.

The buyer side is gated just as hard. According to the same documentation set, Cloudflare publishes a Discovery API at crawlers-api.ai-audit.cfdata.org/charged_zones that lets a crawler list every domain currently charging. We called it on 6 September 2026 and got HTTP 403 with {"error":"Only verified bots can use this endpoint"}. So there is a directory of who is charging, and publishers cannot read it. Neither can we, which is why the sweep above had to be done the slow way.

The error codes have drifted from the documentation

Because Stack Overflow is live, the whole payment negotiation can be walked through with curl. We did, unsigned, so every attempt fails at the authentication step, which is exactly what makes the error codes readable.

Two of six responses do not match the published error code reference, which was last updated 23 April 2026 while the pages around it moved on 28 July. If you are writing a crawler that pays, do not switch on crawler-error string equality against that table alone.

The 403 PaymentFailed result is the important one for publishers. An unsigned request offering the full asking price still gets nothing. Pay Per Crawl bills identity, not intent, and the identity is a Web Bot Auth signature over signed components that must include the payment header. That is why you cannot audit anyone's crawl prices at scale, and why a search for "how many sites charge for crawls" returns nothing useful.

The file you publish for agents is billable. robots.txt is not.

Look again at Cloudflare's always-free list: /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt, /crawlers.json.

llms.txt is not on it. Neither is agents.md, nor /.well-known/mcp/server-card.json, nor any of the other discovery files the agentic web has invented since 2024.

We checked on the one site where it can be checked. stackoverflow.com/llms.txt returns 402 with crawler-price: USD 0.5, the same price as an article. A site that turns on Pay Per Crawl charges half a dollar for the map it wrote specifically to help agents find their way around, while handing over robots.txt for free.

For most publishers this is a small thing that will bite once. The fix is a Configuration Rule with Disable Pay Per Crawl on your discovery paths, and it takes a minute. It is also a gap Cloudflare could close by adding those paths to the free list, and until it does, anyone comparing discovery files across the web will get a distorted picture. If you are still deciding which of these files to publish at all, our breakdown of llms.txt against robots.txt and sitemap.xml covers what each one is actually for, and the llms.txt generator will write you a spec-shaped file.

One more oddity from the same host, offered with a caveat. stackoverflow.com/robots.txt answered our requests with HTTP 418 and a 113-byte body reading User-agent: *, Content-signal: search=no, ai-train=no, Disallow: /. The same 418 came back for stackexchange.com and askubuntu.com, byte-identical. We cannot tell from one network whether that status is a deliberate configuration or IP reputation, so treat it as an observation rather than a finding. What is not in doubt is the effect if it is general: Google's robots.txt handling treats most 4xx responses as "no restrictions", so a file saying Disallow: / delivered with a 418 tells a compliant crawler nothing at all.

Cloudflare's ledger and Coinbase's chain are both called x402

The queue note that started this article assumed Cloudflare Pay Per Crawl and x402 were two competing answers. They are one protocol with two settlement layers, and putting the payloads side by side is the clearest way to price the difference.

Agent Ready's own wording, verified on its pricing page on 6 September 2026: "$0.02 USDC for a 25-page scan or $0.25 for a 250-page scan, settled on Base mainnet (the facilitator pays the gas)."

The comparison that follows is uncomfortable for the per-crawl model. Agent Ready sells a result for two cents. Stack Overflow sells a page for fifty. A crawler that needs twenty-five pages of context pays $0.02 in one case and $12.50 in the other. Per-crawl pricing is only rational where the page itself is the scarce thing, which is true for a newspaper archive and false for almost everything else.

For scale on the wider protocol, x402.org reported 75.41M transactions and $24.24M of volume across 94.06K buyers and 22K sellers in the trailing 30 days as displayed on 6 September 2026. Read that as x402 payments of every kind, not as crawl payments. We found no public figure for how many crawls have ever been paid for, and Cloudflare has not published one.

Before you charge, stop paying for the crawls you do not need

There is a free lever almost nobody pulls, and a post about charging that skips it is a sales pitch.

A crawler that re-fetches an unchanged page costs you the full payload every time. The fix is thirty years old: send ETag and Last-Modified, honour If-None-Match and If-Modified-Since, and answer 304. According to Google Search Central's own crawling guidance, ETag is the recommended validator by name.

We measured how many sites can do it. Across 195 successful responses from 60 hosts on 5 September 2026, 68.2% shipped a cache validator, only 51.8% actually answered 304, and a conditional re-crawl moved 77.5% fewer bytes than a cold one. The full method and the per-surface breakdown are in AI crawler traffic in 2026.

The interaction with Pay Per Crawl is genuinely interesting and genuinely undocumented. Cloudflare, in its own FAQ, defines the charging event as a successful HTTP 200 content retrieval and says verbatim that "error responses are not billed". A 304 Not Modified is neither. The documentation does not say which side of the line it falls on, so a publisher charging per crawl cannot currently tell whether a well-behaved crawler that revalidates is a customer or a freeloader. That is a question worth asking Cloudflare before you price anything.

Either way, the sequence is the same and it is not the one most posts recommend:

1. Make each fetch cheap. Cache validators, correct 304s. Free, standards-based, works with every crawler including the ones that ignore you.
2. Say what you want. Explicit robots.txt rules for named AI agents, plus Content Signals. Our corpus says only 37.3% of sites do this at all.
3. Rate-limit the ones that misbehave. AI Crawl Control's Crawlers tab shows you who they are, free on every plan.
4. Block what you will never license. 403, or 402 with a licensing address if you would rather talk.
5. Charge. Only once steps one to four are done and the traffic still justifies it.

On WordPress, ThinkRank is the shortest route through steps two and three. One plugin owns robots.txt, robots meta, schema, sitemaps and the llms.txt file together, which matters here because the usual WordPress mess is three SEO plugins fighting over one robots.txt while you try to add a single AI directive to it. Existing configuration imports across from Rank Math, Yoast, All in One SEO or SEOPress, so nothing has to be typed twice. Yoast and Rank Math both carry larger template libraries and longer histories on classic SEO reporting; if that is your daily work they stay reasonable picks, and neither of them writes the llms.txt. On Shopify, StoreSEO covers llms.txt, agents.md, schema and the AEO side that Shopify's own agent rollout does not touch. It is rated 5.0 from 737 reviews with the Built for Shopify badge on the Shopify App Store as of 6 September 2026.

Should you charge? A threshold, not a slogan

The test is not whether AI companies should pay. It is whether your pages are worth more as revenue than as citations. For Stack Overflow, whose content is the answer a developer wanted, fifty cents is a defensible number. For a company blog whose job is to be quoted, a 402 is a hole in the funnel.

Two more constraints before anyone applies. A single zone price means you cannot charge a training crawler more than a search crawler, which is the distinction most publishers actually care about. And because the WAF overrides Charge, an existing bot rule can silently turn your revenue into a 403.

Where AIScan fits, and where it doesn't

Start with the scan. Run npx aiscan-cli yoursite.com from a terminal, or drop your address into the box on aiscan.site. No signup and no card. It grades the parts of this that a machine can check:

• M3, x402 payment signal and M4, machine-readable pricing on /docs/checks/commerce
• B1 Content Signals, B2 explicit AI bot rules and B3 Web Bot Auth key directory on /docs/checks/bot-access

Now the part we would rather not print. Across 488 sites scanned on rubric 2026.08.2 between 25 August and 6 September 2026, one latest scan per host with our own pages excluded, M3 passes on 0 of the 180 sites our classifier treats as commerce. Not one, on the third consecutive week of asking. M4 passes on 8.9% and M1 on 15.0%. On the bot access side, B2 passes on 37.3% of all 488 sites, B1 on 21.5%, and B3 on 3.9%.

AIScan does not grade a 402 at all. None of our 21 checks looks for a crawler-price header, a Pay Per Crawl block response, or a licensing message in a refusal body. The sixteen sites in the sweep above that asked an AI crawler for money would get no credit from us for it, and the six with a real price would score identically to the ten without one. That is a genuine gap, it is now on our backlog, and until it ships you cannot use our score to answer this article's question.

We also do not charge. aiscan.site returns 200 to a ClaudeBot request, and as we published last week it ships no cache validator either, so we fail our own step one before we get anywhere near step five.

Cloudflare's free scanner at isitagentready.com does grade x402 as one of five commerce protocol checks, but only on sites its classifier decides are commerce, and it misfires. Use it as a second opinion, not as an audit.

What to do this week

Run the scan and read the commerce and bot access rows. Then, in this order: check whether your origin returns 304 to a conditional request, and fix it if not; look at AI Crawl Control's Crawlers tab to see who is actually fetching you; and only then decide whether any of them should be paying. If the answer is yes, exclude your discovery files from charging before you enable anything.

The single command that tells you where you stand:

npx aiscan-cli yoursite.com

More setup walkthroughs are at aiscan.site/guides, including the WordPress and Shopify paths. If you want the wider argument about what crawlers cost you before any money changes hands, The State of AI Agent Readiness has the distribution, and Cloudflare's three crawler categories explains the Search, Agent and Training split that decides which bots a charge would even apply to.