AI Crawler Traffic in 2026: Only 51.8% of Sites Can Answer "Nothing Changed"

Verified 5 September 2026. All measurements in this article were taken on that date.

Every AI crawler that visits your site asks the same handful of questions over and over. Where is your robots.txt. Has your sitemap moved. Is there an llms.txt yet. Those files change perhaps once a month. The crawlers check them daily, and on most sites they get the entire file back every single time, because the server never told them how to ask "has this changed?"

We probed four discovery surfaces on 60 well-known sites on 5 September 2026: the homepage, /robots.txt, /sitemap.xml and /llms.txt. Of the 195 responses that came back 200, only 101 answered a conditional request with a 304. The other 94 re-sent the whole thing.

Quick summary

According to Google's Search Central blog, published 9 December 2024, the share of Googlebot fetches that could be served from cache has been falling for a decade: "10 years ago about 0.026% of the total fetches were cacheable, which is already not that impressive; today that number is 0.017%."

The finding: 68.2% of the responses we measured shipped a cache validator, but only 51.8% honoured one. The gap between those two numbers is servers that advertise a validator and then ignore it.

What this does not claim: a 304 does not reduce the number of requests a crawler makes. It reduces what each one costs.

What a wasted crawl looks like from the server side

HTTP has had an answer to this since long before AI crawlers existed. RFC 9110, the current HTTP Semantics standard, says in its own words at Section 13 that "conditional GET requests are the most efficient mechanism for HTTP cache updates."

The mechanism has two halves. On the first fetch, your server sends an ETag (an opaque version string) or a Last-Modified date. On the next fetch, a well-behaved client sends that value back in If-None-Match or If-Modified-Since. If nothing changed, the server replies 304 Not Modified with no body at all.

RFC 9110 Section 15.4.5 defines what that means: "there is no need for the server to transfer a representation of the target resource because the request indicates that the client, which made the request conditional, already has a valid representation."

Both halves have to work. A server that sends no validator gives the client nothing to send back. A server that sends one and then answers 200 anyway has wasted the round trip. In our sample, both failure modes are common, and the second one is more common than most people expect.

Google's post is unusually direct about which validator to prefer, stating verbatim that "Google's crawling infrastructure supports heuristic HTTP caching as defined by the HTTP caching standard, specifically through the ETag response- and If-None-Match request header, and the Last-Modified response- and If-Modified-Since request header. We strongly recommend using ETag because it's less prone to errors and mistakes (the value is not structured unlike the Last-Modified value). And, if you have the option, set them both."

How the 195 probes were run

The sample is 60 public sites chosen to span news, documentation, developer platforms, ecommerce storefronts and site builders. It is not random and it is not the whole web. It is deliberately weighted toward sites that AI crawlers actually fetch a lot, which makes it a generous sample rather than a harsh one.

For each site, four URLs were requested twice with a browser user agent:

1. A plain GET, recording the status, the byte count, and any ETag or Last-Modified in the response.
2. A second GET carrying whichever validators came back, in If-None-Match and If-Modified-Since.

Both passes ran within the same minute, so a 200 on the second pass means the server did not honour the condition rather than that the content genuinely changed. Redirects were followed. Anything that did not return 200 on the first pass was excluded from the percentages, which removes four homepages behind bot protection, thirteen sites with no sitemap at the conventional path, and the twenty-five sites that publish no llms.txt.

Three limits worth stating. The sample is 60 sites, so a percentage carries roughly plus or minus six points at this size. A single origin can behave differently for a crawler user agent than for a browser one. And a CDN edge node can answer differently from another edge node, so a 200 on the second pass is evidence rather than proof.

Result one: only 51.8% could say nothing changed

The homepage number is the one that surprised us. Eighteen of the 56 homepages that answered 200 carried neither an ETag nor a Last-Modified, including stripe.com, docs.stripe.com, nextjs.org, wordpress.org, notion.so, slack.com and linear.app. A crawler visiting any of those has no way to ask whether anything moved. It can only download.

Eleven more shipped a validator and then answered 200 to a request carrying it. That set includes developer.chrome.com and web.dev, both operated by Google, and www.cloudflare.com, whose own product line is built on making crawl traffic cheaper.

Result two: the files agents re-read most often are the least protected

Sitemaps are the worst-performing surface in the sample and also the largest. The 47 sitemaps that answered 200 came to 24.6 MB on the first pass, and 21 of them ship no validator at all.

That combination matters more than the homepage result, because a sitemap is the file a crawler is most entitled to poll frequently. It exists to tell a crawler what changed. The sitemaps.org protocol says in its own words that a sitemap "provides hints for web crawlers to do a better job of crawling your site", and a hint file that costs half a megabyte to re-read is working against itself.

llms.txt performs better than the homepage, at 60.0%, which is a genuinely encouraging number for a file that did not exist two years ago. The reason is mostly accidental: llms.txt is usually a static file on a CDN, and static files get validators for free. The exceptions are instructive. stripe.com/llms.txt is 65,026 bytes and ships no validator. docs.stripe.com/llms.txt is 90,052 bytes and ships none either. mailchimp.com/llms.txt is 524,818 bytes, ships a validator, and answers 200 to a conditional request anyway.

The best result in the whole sweep also came from this surface. www.twilio.com/llms.txt is 2,330,360 bytes, the largest single file we measured, and it answers 304 correctly. Every conditional re-fetch of that file transfers nothing.

Result three: 63 MB of re-crawl became 14 MB

Adding up both passes across all 195 successful responses:

One conditional re-crawl of 60 sites moved 77.5% fewer bytes than a cold one. On the homepage surface alone the figure was 34,869,193 down to 9,085,624, a 73.9% reduction.

Multiply that by the number of AI crawlers now in the field. Our own inventory of AI crawler user agents documents 33 operator-published crawler tokens, and five operators now run a three-way split between training, search indexing and live user-triggered fetching. Each of those is a separate client with its own cache, asking the same questions of the same files.

The honest caveat: a 304 still costs a TCP connection, a TLS handshake and a request. It does not stop a crawler visiting. What it removes is the payload, the origin compute that generated the payload, and in most cases the cache-fill work behind it.

Thirteen sites answer correctly everywhere, sixteen answer nowhere

Of the sites that served at least three of the four surfaces, thirteen returned 304 on every one: developer.mozilla.org, docs.python.org, docs.docker.com, kubernetes.io, about.gitlab.com, ghost.org, netlify.com, svelte.dev, vuejs.org, webflow.com, www.digitalocean.com, www.framer.com and llmstxt.org.

Sixteen returned 304 on none of them: developer.chrome.com, web.dev, docs.stripe.com, stripe.com, docusaurus.io, sentry.io, slack.com, techcrunch.com, wordpress.org, www.bbc.com, www.chubbiesshorts.com, www.cloudflare.com, www.elastic.co, www.mejuri.com, www.postgresql.org and www.shopify.com.

The split does not track company size or technical sophistication. It tracks how the site is served. The all-correct group is dominated by documentation sites and static hosts, where the origin is a file server and validators come from the filesystem. The all-wrong group is dominated by application frameworks rendering pages per request, where a validator has to be computed deliberately and usually is not.

Framer's presence in the correct column is consistent with what we measured on that platform previously. Every optimised Framer page also serves a real text/markdown twin under Accept: text/markdown, verified on 4 September 2026. Platforms that get one part of the machine-readable layer right tend to get the others right too.

What the crawler operators actually document about caching

Google documents this thoroughly. Nobody else does.

Those three zero counts were taken on 5 September 2026 by fetching each operator's crawler page and searching the served HTML for ETag, Last-Modified, If-None-Match and If-Modified-Since. Body text was present in each case, so the absence is real rather than a rendering artifact. What Anthropic's page does say, in its own words, is that "we aim for minimal disruption by being thoughtful about how quickly we crawl the same domains and respecting Crawl-delay where appropriate."

That is a rate-limiting answer to a redundancy problem. Crawl-delay slows a crawler down. A 304 makes each visit nearly free. They are not substitutes, and only one of them is in the HTTP standard.

The practical consequence: an operator that does not document conditional-request support may still send one, because most crawler frameworks do it by default. But you cannot rely on the documentation to tell you. The only way to know is to look at your own logs for If-None-Match and If-Modified-Since headers and count how many of your responses were 304.

Our own log, and why it does not let us off the hook

We log AI crawler hits to aiscan.site and we pulled the whole table for this article. Between 25 August and 4 September 2026 it holds 120 requests from 9 distinct bot names, and every one of them returned HTTP 200. Not a single 304.

Aggregate only. No individual scanned URL or visitor is named, which is the same privacy rule we adopted for our 473-site readiness study.

The breakdown is less dramatic than the headline suggests, and reporting it honestly matters more than reporting it loudly:

Fifty-eight percent of the traffic is one agent calling our MCP server, which is a tool invocation rather than a crawl, and correctly returns fresh data every time. The 42 content fetches spread across 41 distinct paths, so at our current scale AI crawlers are barely repeating themselves at all. Our own log does not demonstrate crawl waste.

What it does demonstrate is the other half of the problem. We checked our own headers on 5 September 2026:

None of those four can ever return a 304, because none of them offers anything to condition on. We score 100 on our own rubric and we are in the failing half of our own study. That is on the fix list.

One more result we are reporting with a caveat rather than a headline: our ai_referral_visits table, which is meant to record visits arriving from AI assistants, holds zero rows for the same period. An untested counter reading zero is not evidence of zero referrals, so we are treating it as an instrumentation question until it is verified.

Where AIScan fits, and where it doesn't

The fastest way to check the discovery layer this article is about is one command:

npx aiscan-cli yoursite.com

Or paste the URL at aiscan.site. Free, no account required. The checks that matter here:

AIScan does not grade cache validators at all. We ran the full rubric against our corpus of 484 sites on rubric version 2026.08.2 on 5 September 2026: all 21 checks are listed, and not one of them looks at ETag, Last-Modified or 304 behaviour. Neither does Cloudflare's 22-check agent-readiness rubric. If you want this measured today you have to measure it yourself, which is why the commands below are complete without us.

If you would rather check by hand, two commands answer it for any URL:

# 1. Does the server offer a validator?
curl -sI https://yoursite.com/robots.txt | grep -i 'etag\|last-modified'

# 2. Does it honour one? Paste the ETag value from step 1.
curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'If-None-Match: "PASTE-ETAG-HERE"' \
  https://yoursite.com/robots.txt
# 304 = correct. 200 = the validator is decorative.

Fixing it on WordPress, Shopify and the static stacks

WordPress. ThinkRank is the sensible starting point, because every file this article measures comes out of a single plugin: robots.txt, robots meta, the XML sitemap, llms.txt and schema, rather than three plugins each claiming ownership of the same robots.txt. Switching is cheap too, since it imports what you already configured in Rank Math, Yoast, AIOSEO or SEOPress. Generating the files consistently is the half that a plugin controls. The headers are the other half and belong to your host or CDN: WordPress serves a static robots.txt and sitemap.xml with filesystem validators automatically, but a plugin-generated virtual file often ships neither, so verify after you switch. Rank Math and Yoast both handle sitemaps well and have larger template libraries; neither consolidates the AEO and GEO files the way ThinkRank does. More detail on the platform is on our WordPress guides page.

Shopify. You do not control the storefront's response headers, so the validator half is not yours to fix. The file half is, and StoreSEO is the natural choice: it generates llms.txt from products, collections, pages and articles, has an agents.md editor, and covers the schema and AEO surfaces Shopify's own rollout leaves alone. It is rated 5.0 from 735 reviews on the Shopify App Store as of 5 September 2026, with the Built for Shopify badge. Yoast SEO for Shopify and Avada are reasonable alternatives if schema is your only concern; neither generates llms.txt today. Our Shopify platform page collects the store-specific guides.

Next.js and other frameworks. Route handlers return whatever headers you set and nothing more. A route.ts that generates robots.txt or llms.txt should hash its own output and return ETag, then compare against If-None-Match and return 304. Static files under public/ usually get validators from the host, which is why moving a generated file to a build artifact often fixes this by accident. Our Next.js platform page has the framework-specific notes.

Static hosts and CDNs. Cloudflare, Netlify, Vercel and Fastly all emit ETag on static assets by default. If your static file is not returning 304, the usual cause is a rewrite or a function intercepting the path. Check whether the file is genuinely static before changing any settings.

If llms.txt is the file you are fixing, our llms.txt generator writes a spec-shaped file, and the eleven mistakes we found across 39 real files covers what usually goes wrong with the content itself.

Branch on what your own headers said

Ten minutes, four commands

• Check /robots.txt for a validator, then send it back and read the status code.
• Repeat for /sitemap.xml. In our sample this is the surface most likely to fail.
• Repeat for /llms.txt if you publish one, and note its size while you are there.
• Repeat for the homepage. Expect this one to fail on any framework rendering per request.
• Grep your access log for If-None-Match and If-Modified-Since, then count what share of your responses to those requests were 304.
• Run a scan for the rest of the discovery layer and read D1, D2, C2 and E1.

The next thing to run

Start with the file that costs the most to re-send. In this sample that is almost always the sitemap, and on documentation-heavy sites it is llms.txt. Two curl commands tell you which half of the 51.8% you are in, and on most stacks the fix is a CDN toggle rather than a code change.

Then run a scan at AIScan and read the four checks this article is about: D1 robots.txt, D2 XML sitemap, C2 llms.txt and E1 correct 404 handling. Those four are on the discoverability and content dimension pages, with the fix instructions for each. The rest of the platform walkthroughs are in our guides.

And if your llms.txt turns out to be the expensive one, the six discovery surfaces and what each is actually for explains which files a crawler is entitled to poll and how often.