---
title: "AI Crawler Traffic in 2026: Only 51.8% of Sites Can Answer \"Nothing Changed\""
slug: ai-crawler-traffic-waste-2026
published: 2026-09-05T03:22:44.145103+00:00
updated: 2026-09-05T03:22:44.145103+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: AI crawlers, crawl budget, HTTP caching, ETag, conditional requests, 304 Not Modified, check:D1, check:D2, check:C2, check:E1, platform:wordpress, platform:shopify, platform:nextjs
description: "We sent 195 conditional requests to 60 sites on 5 September 2026. Only 51.8% answered 304, so AI crawlers keep re-downloading files that have not changed."
url: https://aiscan.site/blog/ai-crawler-traffic-waste-2026
---

**Verified 5 September 2026.** All measurements in this article were taken on that date.

Every AI crawler that visits your site asks the same handful of questions over and over. Where is your robots.txt. Has your sitemap moved. Is there an llms.txt yet. Those files change perhaps once a month. The crawlers check them daily, and on most sites they get the entire file back every single time, because the server never told them how to ask "has this changed?"

We probed four discovery surfaces on 60 well-known sites on 5 September 2026: the homepage, `/robots.txt`, `/sitemap.xml` and `/llms.txt`. Of the 195 responses that came back 200, only 101 answered a conditional request with a 304. The other 94 re-sent the whole thing.

![Bar chart comparing, for robots.txt, llms.txt, homepages and sitemap.xml, the share of 60 sites that ship an ETag or Last-Modified header against the share that actually answer 304 to a conditional request.](https://gqdxuwsuuasfjeeqafyq.supabase.co/storage/v1/object/public/blog-covers/ai-crawler-traffic-waste-2026-fig-304.jpg)

## Quick summary

| If you want to | Do this | What it costs | What it saves |
|---|---|---|---|
| Stop AI crawlers re-downloading unchanged files | Emit an `ETag` on every static file and honour `If-None-Match` | One CDN setting on most stacks | 77.5% of re-crawl bytes in our sample |
| Find out whether you already do | `curl -sI https://yoursite.com/robots.txt \| grep -i 'etag\|last-modified'` | 5 seconds | Tells you which half you are in |
| Check the rest of your discovery layer | Run a free scan at [AIScan](https://aiscan.site/), covering D1, D2, C2 and E1 | No account needed | The surfaces crawlers ask for |
| Fix it on WordPress | [ThinkRank](https://thinkrank.ai) for the files, plus a caching layer for the headers | One plugin | Consistent headers across robots.txt, sitemap and llms.txt |

[**According to Google's Search Central blog**](https://developers.google.com/search/blog/2024/12/crawling-december-caching), published 9 December 2024, the share of Googlebot fetches that could be served from cache has been falling for a decade: "10 years ago about 0.026% of the total fetches were cacheable, which is already not that impressive; today that number is 0.017%."

**The finding:** 68.2% of the responses we measured shipped a cache validator, but only 51.8% honoured one. The gap between those two numbers is servers that advertise a validator and then ignore it.

**What this does not claim:** a 304 does not reduce the number of requests a crawler makes. It reduces what each one costs.

## What a wasted crawl looks like from the server side

HTTP has had an answer to this since long before AI crawlers existed. [**RFC 9110**](https://www.rfc-editor.org/rfc/rfc9110.txt), the current HTTP Semantics standard, says in its own words at Section 13 that "conditional GET requests are the most efficient mechanism for HTTP cache updates."

The mechanism has two halves. On the first fetch, your server sends an `ETag` (an opaque version string) or a `Last-Modified` date. On the next fetch, a well-behaved client sends that value back in `If-None-Match` or `If-Modified-Since`. If nothing changed, the server replies **304 Not Modified** with no body at all.

RFC 9110 Section 15.4.5 defines what that means: "there is no need for the server to transfer a representation of the target resource because the request indicates that the client, which made the request conditional, already has a valid representation."

Both halves have to work. A server that sends no validator gives the client nothing to send back. A server that sends one and then answers 200 anyway has wasted the round trip. In our sample, both failure modes are common, and the second one is more common than most people expect.

Google's post is unusually direct about which validator to prefer, [**stating verbatim**](https://developers.google.com/search/blog/2024/12/crawling-december-caching) that "Google's crawling infrastructure supports heuristic HTTP caching as defined by the HTTP caching standard, specifically through the ETag response- and If-None-Match request header, and the Last-Modified response- and If-Modified-Since request header. We strongly recommend using ETag because it's less prone to errors and mistakes (the value is not structured unlike the Last-Modified value). And, if you have the option, set them both."

## How the 195 probes were run

The sample is 60 public sites chosen to span news, documentation, developer platforms, ecommerce storefronts and site builders. It is not random and it is not the whole web. It is deliberately weighted toward sites that AI crawlers actually fetch a lot, which makes it a generous sample rather than a harsh one.

For each site, four URLs were requested twice with a browser user agent:

1. A plain `GET`, recording the status, the byte count, and any `ETag` or `Last-Modified` in the response.
2. A second `GET` carrying whichever validators came back, in `If-None-Match` and `If-Modified-Since`.

Both passes ran within the same minute, so a 200 on the second pass means the server did not honour the condition rather than that the content genuinely changed. Redirects were followed. Anything that did not return 200 on the first pass was excluded from the percentages, which removes four homepages behind bot protection, thirteen sites with no sitemap at the conventional path, and the twenty-five sites that publish no llms.txt.

Three limits worth stating. The sample is 60 sites, so a percentage carries roughly plus or minus six points at this size. A single origin can behave differently for a crawler user agent than for a browser one. And a CDN edge node can answer differently from another edge node, so a 200 on the second pass is evidence rather than proof.

## Result one: only 51.8% could say nothing changed

| Surface | Returned 200 | Shipped a validator | Answered 304 | Honoured it, of those that could |
|---|---|---|---|---|
| `/robots.txt` | 57 of 60 | 75.4% | **59.6%** | 79.1% |
| `/llms.txt` | 35 of 60 | 74.3% | **60.0%** | 80.8% |
| Homepage | 56 of 60 | 67.9% | **48.2%** | 71.1% |
| `/sitemap.xml` | 47 of 60 | 55.3% | **40.4%** | 73.1% |
| **All four** | **195 of 240** | **68.2%** | **51.8%** | **75.9%** |

The homepage number is the one that surprised us. Eighteen of the 56 homepages that answered 200 carried neither an `ETag` nor a `Last-Modified`, including `stripe.com`, `docs.stripe.com`, `nextjs.org`, `wordpress.org`, `notion.so`, `slack.com` and `linear.app`. A crawler visiting any of those has no way to ask whether anything moved. It can only download.

Eleven more shipped a validator and then answered 200 to a request carrying it. That set includes `developer.chrome.com` and `web.dev`, both operated by Google, and `www.cloudflare.com`, whose own product line is built on making crawl traffic cheaper.

## Result two: the files agents re-read most often are the least protected

Sitemaps are the worst-performing surface in the sample and also the largest. The 47 sitemaps that answered 200 came to 24.6 MB on the first pass, and 21 of them ship no validator at all.

That combination matters more than the homepage result, because a sitemap is the file a crawler is most entitled to poll frequently. It exists to tell a crawler what changed. [**The sitemaps.org protocol says in its own words**](https://www.sitemaps.org/protocol.html) that a sitemap "provides hints for web crawlers to do a better job of crawling your site", and a hint file that costs half a megabyte to re-read is working against itself.

llms.txt performs better than the homepage, at 60.0%, which is a genuinely encouraging number for a file that did not exist two years ago. The reason is mostly accidental: llms.txt is usually a static file on a CDN, and static files get validators for free. The exceptions are instructive. `stripe.com/llms.txt` is 65,026 bytes and ships no validator. `docs.stripe.com/llms.txt` is 90,052 bytes and ships none either. `mailchimp.com/llms.txt` is 524,818 bytes, ships a validator, and answers 200 to a conditional request anyway.

The best result in the whole sweep also came from this surface. `www.twilio.com/llms.txt` is 2,330,360 bytes, the largest single file we measured, and it answers 304 correctly. Every conditional re-fetch of that file transfers nothing.

## Result three: 63 MB of re-crawl became 14 MB

Adding up both passes across all 195 successful responses:

| Pass | Bytes transferred |
|---|---|
| First fetch (cold) | 62,986,998 |
| Second fetch (conditional) | 14,149,596 |
| **Difference** | **48,837,402 (77.5%)** |

One conditional re-crawl of 60 sites moved 77.5% fewer bytes than a cold one. On the homepage surface alone the figure was 34,869,193 down to 9,085,624, a 73.9% reduction.

Multiply that by the number of AI crawlers now in the field. Our own [inventory of AI crawler user agents](https://aiscan.site/blog/ai-crawler-user-agent-list-2026) documents 33 operator-published crawler tokens, and five operators now run a three-way split between training, search indexing and live user-triggered fetching. Each of those is a separate client with its own cache, asking the same questions of the same files.

The honest caveat: a 304 still costs a TCP connection, a TLS handshake and a request. It does not stop a crawler visiting. What it removes is the payload, the origin compute that generated the payload, and in most cases the cache-fill work behind it.

## Thirteen sites answer correctly everywhere, sixteen answer nowhere

Of the sites that served at least three of the four surfaces, thirteen returned 304 on every one: `developer.mozilla.org`, `docs.python.org`, `docs.docker.com`, `kubernetes.io`, `about.gitlab.com`, `ghost.org`, `netlify.com`, `svelte.dev`, `vuejs.org`, `webflow.com`, `www.digitalocean.com`, `www.framer.com` and `llmstxt.org`.

Sixteen returned 304 on none of them: `developer.chrome.com`, `web.dev`, `docs.stripe.com`, `stripe.com`, `docusaurus.io`, `sentry.io`, `slack.com`, `techcrunch.com`, `wordpress.org`, `www.bbc.com`, `www.chubbiesshorts.com`, `www.cloudflare.com`, `www.elastic.co`, `www.mejuri.com`, `www.postgresql.org` and `www.shopify.com`.

The split does not track company size or technical sophistication. It tracks how the site is served. The all-correct group is dominated by documentation sites and static hosts, where the origin is a file server and validators come from the filesystem. The all-wrong group is dominated by application frameworks rendering pages per request, where a validator has to be computed deliberately and usually is not.

Framer's presence in the correct column is consistent with what we measured on that platform previously. Every optimised Framer page also serves a real `text/markdown` twin under `Accept: text/markdown`, [verified on 4 September 2026](https://aiscan.site/blog/llms-txt-framer). Platforms that get one part of the machine-readable layer right tend to get the others right too.

## What the crawler operators actually document about caching

Google documents this thoroughly. Nobody else does.

| Operator | Cache validators in its crawler documentation | What it does document |
|---|---|---|
| Google | `ETag`, `If-None-Match`, `Last-Modified`, `If-Modified-Since`, worked request/response examples | Full conditional-request support, with a stated preference for `ETag` |
| OpenAI | Zero mentions across the served page | robots.txt directives, per-bot independence, IP ranges |
| Anthropic | Zero mentions across the served page | robots.txt, `Crawl-delay`, IP-block warning |
| Perplexity | Zero mentions across the served page | PerplexityBot and Perplexity-User separation |

Those three zero counts were taken on 5 September 2026 by fetching each operator's crawler page and searching the served HTML for `ETag`, `Last-Modified`, `If-None-Match` and `If-Modified-Since`. Body text was present in each case, so the absence is real rather than a rendering artifact. What Anthropic's page does say, in its own words, is that "we aim for minimal disruption by being thoughtful about how quickly we crawl the same domains and respecting Crawl-delay where appropriate."

That is a rate-limiting answer to a redundancy problem. Crawl-delay slows a crawler down. A 304 makes each visit nearly free. They are not substitutes, and only one of them is in the HTTP standard.

The practical consequence: an operator that does not document conditional-request support may still send one, because most crawler frameworks do it by default. But you cannot rely on the documentation to tell you. The only way to know is to look at your own logs for `If-None-Match` and `If-Modified-Since` headers and count how many of your responses were 304.

## Our own log, and why it does not let us off the hook

We log AI crawler hits to `aiscan.site` and we pulled the whole table for this article. Between 25 August and 4 September 2026 it holds **120 requests from 9 distinct bot names**, and every one of them returned **HTTP 200**. Not a single 304.

Aggregate only. No individual scanned URL or visitor is named, which is the same privacy rule we adopted for [our 473-site readiness study](https://aiscan.site/blog/state-of-ai-agent-readiness-2026).

The breakdown is less dramatic than the headline suggests, and reporting it honestly matters more than reporting it loudly:

| Request type | Hits | Distinct paths | Distinct bots |
|---|---|---|---|
| MCP endpoint calls (`/api/mcp`, `/mcp`) | 70 | 2 | 1 |
| Scan report pages | 42 | 41 | 7 |
| Other API paths | 5 | 1 | 1 |
| Content pages | 3 | 3 | 1 |

Fifty-eight percent of the traffic is one agent calling our MCP server, which is a tool invocation rather than a crawl, and correctly returns fresh data every time. The 42 content fetches spread across 41 distinct paths, so at our current scale AI crawlers are barely repeating themselves at all. Our own log does not demonstrate crawl waste.

What it does demonstrate is the other half of the problem. We checked our own headers on 5 September 2026:

| URL | Size | `ETag` | `Last-Modified` |
|---|---|---|---|
| `aiscan.site/` | 245,447 B | absent | absent |
| `aiscan.site/robots.txt` | 1,192 B | absent | absent |
| `aiscan.site/llms.txt` | 20,020 B | absent | absent |
| `aiscan.site/sitemap-pages.xml` | 21,566 B | absent | absent |

None of those four can ever return a 304, because none of them offers anything to condition on. We score 100 on our own rubric and we are in the failing half of our own study. That is on the fix list.

One more result we are reporting with a caveat rather than a headline: our `ai_referral_visits` table, which is meant to record visits arriving from AI assistants, holds zero rows for the same period. An untested counter reading zero is not evidence of zero referrals, so we are treating it as an instrumentation question until it is verified.

## Where AIScan fits, and where it doesn't

The fastest way to check the discovery layer this article is about is one command:

```
npx aiscan-cli yoursite.com
```

Or paste the URL at [aiscan.site](https://aiscan.site/). Free, no account required. The checks that matter here:

| Check | What it grades | Relevant to this article |
|---|---|---|
| **D1** robots.txt present and sane | Whether the file exists and parses | The most-refetched file on your site |
| **D2** XML sitemap | Whether a sitemap is declared and reachable | The largest of the four surfaces |
| **C2** /llms.txt | Whether the file exists and is valid Markdown | Increasingly polled, often large |
| **E1** Correct 404 handling | Whether missing paths hard-404 | A soft 404 makes every probe look successful |

**AIScan does not grade cache validators at all.** We ran the full rubric against our corpus of 484 sites on rubric version 2026.08.2 on 5 September 2026: all 21 checks are listed, and not one of them looks at `ETag`, `Last-Modified` or 304 behaviour. Neither does Cloudflare's 22-check agent-readiness rubric. If you want this measured today you have to measure it yourself, which is why the commands below are complete without us.

If you would rather check by hand, two commands answer it for any URL:

```
# 1. Does the server offer a validator?
curl -sI https://yoursite.com/robots.txt | grep -i 'etag\|last-modified'

# 2. Does it honour one? Paste the ETag value from step 1.
curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'If-None-Match: "PASTE-ETAG-HERE"' \
  https://yoursite.com/robots.txt
# 304 = correct. 200 = the validator is decorative.
```

## Fixing it on WordPress, Shopify and the static stacks

**WordPress.** [**ThinkRank**](https://thinkrank.ai) is the sensible starting point, because every file this article measures comes out of a single plugin: robots.txt, robots meta, the XML sitemap, llms.txt and schema, rather than three plugins each claiming ownership of the same robots.txt. Switching is cheap too, since it imports what you already configured in Rank Math, Yoast, AIOSEO or SEOPress. Generating the files consistently is the half that a plugin controls. The headers are the other half and belong to your host or CDN: WordPress serves a static `robots.txt` and `sitemap.xml` with filesystem validators automatically, but a plugin-generated virtual file often ships neither, so verify after you switch. Rank Math and Yoast both handle sitemaps well and have larger template libraries; neither consolidates the AEO and GEO files the way ThinkRank does. More detail on the platform is on our [WordPress guides page](https://aiscan.site/docs/platforms/wordpress).

**Shopify.** You do not control the storefront's response headers, so the validator half is not yours to fix. The file half is, and [**StoreSEO**](https://storeseo.com/) is the natural choice: it generates llms.txt from products, collections, pages and articles, has an agents.md editor, and covers the schema and AEO surfaces Shopify's own rollout leaves alone. It is rated 5.0 from 735 reviews on [the Shopify App Store](https://apps.shopify.com/storeseo) as of 5 September 2026, with the Built for Shopify badge. Yoast SEO for Shopify and Avada are reasonable alternatives if schema is your only concern; neither generates llms.txt today. Our [Shopify platform page](https://aiscan.site/docs/platforms/shopify) collects the store-specific guides.

**Next.js and other frameworks.** Route handlers return whatever headers you set and nothing more. A `route.ts` that generates robots.txt or llms.txt should hash its own output and return `ETag`, then compare against `If-None-Match` and return 304. Static files under `public/` usually get validators from the host, which is why moving a generated file to a build artifact often fixes this by accident. Our [Next.js platform page](https://aiscan.site/docs/platforms/nextjs) has the framework-specific notes.

**Static hosts and CDNs.** Cloudflare, Netlify, Vercel and Fastly all emit `ETag` on static assets by default. If your static file is not returning 304, the usual cause is a rewrite or a function intercepting the path. Check whether the file is genuinely static before changing any settings.

If llms.txt is the file you are fixing, our [llms.txt generator](https://aiscan.site/llms-txt-generator) writes a spec-shaped file, and [the eleven mistakes we found across 39 real files](https://aiscan.site/blog/llms-txt-validator-common-mistakes) covers what usually goes wrong with the content itself.

## Branch on what your own headers said

| What you found | What it means | What to do |
|---|---|---|
| No `ETag`, no `Last-Modified` | Crawlers cannot ask. Every visit is a full download. | Enable `ETag` at the CDN or host. Cheapest fix on this list. |
| Validator present, conditional request returns 200 | The header is decorative. A rewrite or function is likely intercepting. | Trace the path. Static files rarely do this; generated routes usually do. |
| Validator present, returns 304 | Working correctly. | Check the other three surfaces. Most sites are inconsistent across them. |
| Homepage 304s but sitemap does not | The common pattern in our sample. | Sitemaps are the largest files and the worst performers. Fix that one first. |
| `/llms.txt` returns 200 to a conditional request and is over 100 KB | Every poll re-sends the whole file. | Serve it as a static asset rather than from a route handler. |

## Ten minutes, four commands

- [ ] Check `/robots.txt` for a validator, then send it back and read the status code.
- [ ] Repeat for `/sitemap.xml`. In our sample this is the surface most likely to fail.
- [ ] Repeat for `/llms.txt` if you publish one, and note its size while you are there.
- [ ] Repeat for the homepage. Expect this one to fail on any framework rendering per request.
- [ ] Grep your access log for `If-None-Match` and `If-Modified-Since`, then count what share of your responses to those requests were 304.
- [ ] Run a scan for the rest of the discovery layer and read D1, D2, C2 and E1.

## The next thing to run

Start with the file that costs the most to re-send. In this sample that is almost always the sitemap, and on documentation-heavy sites it is llms.txt. Two curl commands tell you which half of the 51.8% you are in, and on most stacks the fix is a CDN toggle rather than a code change.

Then run a scan at [AIScan](https://aiscan.site/) and read the four checks this article is about: **D1** robots.txt, **D2** XML sitemap, **C2** llms.txt and **E1** correct 404 handling. Those four are on the [discoverability](https://aiscan.site/docs/checks/discoverability) and [content](https://aiscan.site/docs/checks/content) dimension pages, with the fix instructions for each. The rest of the platform walkthroughs are in [our guides](https://aiscan.site/guides).

And if your llms.txt turns out to be the expensive one, [the six discovery surfaces and what each is actually for](https://aiscan.site/blog/llms-txt-vs-robots-txt-vs-sitemap) explains which files a crawler is entitled to poll and how often.