Can Your Site Say "Nothing Changed"? ETag and Last-Modified on 300 Hosts

We probed 300 popular hosts on 3 October 2026, and only 77 of the 152 usable homepages sent a cache validator at all. That is 50.7%. Until a site sends an ETag or a Last-Modified header, no crawler, AI or otherwise, can ask it the one question that makes re-crawling cheap: "has this changed since I last looked?" This post counts how many sites even get that far, and how many of those then keep the promise.

Our earlier conditional-request sweep measured the back half of the problem: of the sites that sent a validator, 51.8% of all probes answered 304 Not Modified. This study measures the front half, and joins the two into one funnel. We open on our own house, because aiscan.site fails the first stage.

Quick summary

The finding: the drop-off is steepest at the first stage on HTML pages (half send nothing) and at the second stage for validators that look present but are not honoured (35% of homepage senders). robots.txt, a static file, behaves very differently from the homepage on the same host.

What to do: run npx aiscan-cli yoursite.com or paste the URL at aiscan.site to grade the discoverability checks D1 and D2 around these files, then verify validators with the two-request test in the last section. AIScan does not grade cache validators itself, so that test is manual.

Why validators matter to an agent

A validator is a label the server attaches to a specific version of a response. RFC 9110 defines two: ETag (section 8.8.3) and Last-Modified (section 8.8.2). A client that stored the response sends the label back in If-None-Match (section 13.1.2) or If-Modified-Since (section 13.1.3). If nothing changed, the server answers 304 with no body (section 15.4.5). The RFC puts the benefit plainly: conditional GET requests are "the most efficient mechanism for HTTP cache updates."

Google's Search Central team laid out the same chain in "Crawling December: HTTP caching" on 9 December 2024. Two lines from it matter here. The share of Googlebot fetches that were cacheable fell from 0.026% ten years earlier to 0.017% at the time of writing. And Google said it strongly recommends ETag over Last-Modified because it is "less prone to errors and mistakes," and that if you can, you should set both.

None of this depends on a crawler being AI. But AI crawlers are the reason it matters more now: they fetch the same pages repeatedly, many vendors bill or throttle by request, and the site owner pays for every full 200 that a 304 would have replaced. For the effect on bytes, see our crawler traffic study, where a conditional re-crawl cut transfer by 77.5% on the sites that supported it.

Our own house comes first

On 3 October 2026 we requested five URLs on aiscan.site with a browser user-agent and read the response headers.

None of the five can ever return 304. The same was true when we first measured it in September, and our own log of AI-bot requests then showed 120 hits, all 200, none 304. A site that scores 100 on our scanner sits in the "sends nothing" half of this funnel. That is not a contradiction, because the scanner does not grade validators. It is the reason this study exists: a clean readiness score and a cacheable site are separate properties, and the second one is invisible unless you look for it.

We have logged the fix as a platform item. We are not claiming the number moved.

Method

Population. We downloaded the Tranco top-sites list on 3 October 2026, took the first 5,000 domains, sorted them by the MD5 of the hostname, and took the first 300. This is a deterministic sample of popular sites. It is not the AIScan scan corpus, and it is not a random sample of the web. Popular sites are more likely to sit behind a CDN, which skews every number below.

Probes. For each host we requested https://<host>/ and https://<host>/robots.txt with curl -L and a desktop Chrome user-agent. We recorded the final status, content type, and whether the final response carried ETag, Last-Modified, or both. If the response was 200 and carried a validator, we repeated the request with If-None-Match and If-Modified-Since set to the values we received, and recorded the status.

Counting rules. A homepage counts only if the final response was 200 with an HTML content type. A robots.txt counts only if it was 200 with a text/plain content type, which rejects the soft-200 HTML pages many sites serve in its place. We counted a host as a validator sender only when the header appeared on both passes, and as honouring it only when both passes returned 304.

Two passes. Following our standing rule to re-probe every differential, we ran the whole set twice, minutes apart, on the 209 hosts that answered 200 in the first pass. Nobody flipped between sending and not sending a validator. A few changed their conditional answer: 7 homepages and 5 robots.txt files returned 304 once and 200 once. We report those as "not stable" rather than rounding them either way.

What did not make it in. Of the 600 first-pass probes, 124 returned 502 (the proxy our sandbox runs behind could not complete the connection), 45 returned 403, and 25 returned 404. Only 156 homepages and 146 robots.txt files met the counting rules in pass one, and 152 and 144 on both passes. Hosts that block datacentre addresses are therefore under-represented, and they are exactly the hosts most likely to run strict bot management. Treat every rate below as a rate among sites that answer an unauthenticated datacentre request.

The funnel

The two files have almost the same shape on a good host and very different shapes across the population. robots.txt is a flat file, so the web server computes Last-Modified and ETag from the file system by default. The homepage is usually generated, and someone has to decide what its version label is.

Stage one: who sends a validator at all

On the homepage, 75 of 152 sites sent neither header on either pass. On robots.txt the figure was 24 of 144. The 50-point gap between a generated page and a static file is the largest number in this study, and it matches what you would expect from how stacks work, not from site quality. A static file server such as nginx emits both headers by default; a framework rendering HTML often leaves the choice to the application, and many applications choose to set Cache-Control: no-store or no-cache and send nothing to validate against.

Stage two: what kind of validator

In the first pass, among homepages that sent at least one validator (78 hosts):

For robots.txt (121 senders) the same split was 30 hosts with Last-Modified only (28 answered 304), 85 with both (77 answered), and 6 with ETag only (4 answered).

The surprise is the homepage ETag-only row. Google recommends ETag, yet in our sample the sites that sent only an ETag were the least likely to honour it: 6 of 18. The sample is small, and we did not diagnose each host, so this is a hypothesis and not a finding. The likeliest candidates are an application that generates a fresh ETag per response (a hash of per-request content such as a nonce or timestamp), or a CDN that rewrites the label when it compresses the body, so the value the server stored never matches the one it later compares. If you send an ETag, test it. A validator that changes on every request costs bytes and buys nothing.

Stage three: who then honours it

Of the 77 homepages that sent a validator on both passes, 50 answered 304 on both: 64.9%. Of the 120 robots.txt senders, 104: 86.7%. The remaining 27 homepages and 16 robots.txt files sent a validator that did not work, at least once. That group is the cheapest to fix, because the header is already there.

Against our September numbers

Our earlier sweep read 68.2% of successful responses carrying a validator and 51.8% answering 304, across four surfaces on 60 hosts. This one reads 50.7% and 32.9% on the homepage, and 83.3% and 72.2% on robots.txt. The populations are different (60 hosts there, 152 and 144 here), the surfaces are different (this study drops /llms.txt and /sitemap.xml), and the pass criteria are stricter. Do not read the gap as a trend. Both studies agree on the direction: roughly half of HTML pages cannot answer "nothing changed", and static files do better.

What a 304 saves on a homepage

For the 50 homepages that answered 304 on both passes, we ran a third pair of requests and recorded the body bytes. All 50 answered 304 again. The unconditional requests downloaded 13,853,038 bytes of body in total, with no Accept-Encoding header sent, so servers returned whatever they chose to by default. The conditional requests downloaded zero bytes of body, which is what a 304 means. Header bytes are excluded on both sides.

The spread matters more than the total. The median homepage was 97,874 bytes, the smallest 1,606 and the largest 2,145,277. A crawler that revalidates a 2 MB page saves 2 MB per visit; one that revalidates a 1.6 KB page saves almost nothing. The savings concentrate on the heavy pages, which are the ones a site owner most wants crawlers to stop re-downloading. Our September figure of 77.5% fewer bytes counted headers and mixed surfaces, so the two numbers are not interchangeable. The honest statement is narrower: when a homepage honours its validator, the body transfer disappears.

What this means for an AI crawler

A crawler deciding how often to revisit a page has two choices when a site sends no validator: refetch the full body or guess. Refetching is the safe choice, so the bytes move. The site pays for the bandwidth and the crawler pays for the parse, and neither learns anything new.

Our declaration census showed that most sites say nothing about AI crawlers in robots.txt at all. This study adds a quieter observation about the same population: on the homepage, half of them also give crawlers no way to ask cheaply whether the page changed. Neither is a defect in the strict sense. Both are defaults that nobody revisited after crawlers got more numerous.

Where this sits in your readiness work

AIScan's discoverability dimension covers the files an agent finds first. Run the scan before touching headers: npx aiscan-cli yoursite.com, or paste your URL at aiscan.site, free and without an account. Check D1 and D2 (see the discoverability checks) tell you whether your sitemap and the files around it exist and parse. They do not tell you whether those files can be revalidated, because validators are not part of the rubric today. That is a limit of the tool, and it is why the manual test below exists.

If you run WordPress, a single SEO plugin that owns robots.txt and the sitemap removes one source of conflicting headers. ThinkRank does robots.txt, robots meta, schema and llms.txt from one plugin, and migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. It does not set your page cache headers, though. That is a job for your caching layer or host, and the plugins named above do not do it either. For a broader view of how these files fit together, see llms.txt vs robots.txt vs sitemap.xml.

Check your own site by hand

If you would rather check by hand, two requests settle it. The first command prints the validators; the second sends them back.

curl -sI https://yoursite.com/ | grep -iE '^(etag|last-modified|cache-control)'
curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'If-None-Match: "<etag from above>"' \
  -H 'If-Modified-Since: <last-modified from above>' \
  https://yoursite.com/

Read the results this way:

Test the homepage and one deep URL, because your static files and your rendered pages are served by different code. Repeat the second request several times. Several of our unstable hosts passed once and failed once.

Common fixes, in the order we would try them:

1. Static files: leave etag on and the default Last-Modified in your web server. nginx does both for files on disk.
2. Rendered pages: make your framework or CDN emit a stable ETag that depends on content, not on the request. Next.js generates one for pages by default unless you turn generateEtags off.
3. Compression: check whether your CDN turns a strong ETag into a weak one when it compresses. A weak ETag still works for If-None-Match, but a mismatch between origin and edge will not.
4. Both headers: Google's own advice is to set both, so a client that stores only one can still revalidate.

What we could not measure

• Real crawler behaviour. We are a curl client. We cannot see whether GPTBot, ClaudeBot or PerplexityBot actually send conditional headers; our own request log is too small to say. The funnel's third stage tells you whether a site would answer, not whether anyone asks.
• One vantage point. All probes came from one datacentre address. Sites that vary responses by network or run bot management will look different from a residential line.
• Two surfaces. The homepage and robots.txt only. Sitemaps, llms.txt and article pages may behave differently, and our September study found the sitemap the weakest of the four.
• Popular sites. The sample is Tranco's top 5,000, so small sites are absent.
• Why a validator fails. We recorded that it did, not the cause.

Next steps

Run the scan on your own site today: npx aiscan-cli yoursite.com covers D1 and D2, the checks that decide whether an agent finds your sitemap and discovery files. Then run the two-request test above and fix whichever stage fails first. For more checks and fixes, browse the AIScan guides, or start from the State of AI Agent Readiness study to see where your score sits against 473 other sites.