---
title: "Can Your Site Say \"Nothing Changed\"? ETag and Last-Modified on 300 Hosts"
slug: etag-last-modified-adoption-corpus
published: 2026-10-03T03:20:49.003057+00:00
updated: 2026-10-04T12:06:20.081077+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: check:D2, check:D1, etag, last-modified, conditional-requests, http-caching, corpus-study
description: "We probed 300 popular hosts twice: 50.7% of homepages send an ETag or Last-Modified, and 32.9% answer 304. robots.txt does far better. Test your own site."
url: https://aiscan.site/blog/etag-last-modified-adoption-corpus
---

We probed 300 popular hosts on 3 October 2026, and only 77 of the 152 usable homepages sent a cache validator at all. That is 50.7%. Until a site sends an `ETag` or a `Last-Modified` header, no crawler, AI or otherwise, can ask it the one question that makes re-crawling cheap: "has this changed since I last looked?" This post counts how many sites even get that far, and how many of those then keep the promise.

Our [earlier conditional-request sweep](https://aiscan.site/blog/ai-crawler-traffic-waste-2026) measured the back half of the problem: of the sites that sent a validator, 51.8% of all probes answered `304 Not Modified`. This study measures the front half, and joins the two into one funnel. We open on our own house, because `aiscan.site` fails the first stage.

## Quick summary

| Question | Homepage (n = 152) | robots.txt (n = 144) |
|---|---|---|
| Sends any validator (`ETag` or `Last-Modified`) on both probes | 77 (50.7%) | 120 (83.3%) |
| Answers `304` to a conditional request on both probes | 50 (32.9%) | 104 (72.2%) |
| Of senders, share that honour the validator | 64.9% | 86.7% |
| Sends a validator on one probe and not the other | 0 | 0 |
| Sends a validator but returns `200` on at least one probe | 27 | 16 |

**The finding:** the drop-off is steepest at the first stage on HTML pages (half send nothing) and at the second stage for validators that look present but are not honoured (35% of homepage senders). `robots.txt`, a static file, behaves very differently from the homepage on the same host.

**What to do:** run `npx aiscan-cli yoursite.com` or paste the URL at [aiscan.site](https://aiscan.site/) to grade the discoverability checks D1 and D2 around these files, then verify validators with the two-request test in the last section. AIScan does not grade cache validators itself, so that test is manual.

## Why validators matter to an agent

A validator is a label the server attaches to a specific version of a response. [RFC 9110](https://datatracker.ietf.org/doc/html/rfc9110) defines two: `ETag` (section 8.8.3) and `Last-Modified` (section 8.8.2). A client that stored the response sends the label back in `If-None-Match` (section 13.1.2) or `If-Modified-Since` (section 13.1.3). If nothing changed, the server answers `304` with no body (section 15.4.5). The RFC puts the benefit plainly: conditional GET requests are "the most efficient mechanism for HTTP cache updates."

Google's Search Central team laid out the same chain in ["Crawling December: HTTP caching"](https://developers.google.com/search/blog/2024/12/crawling-december-caching) on 9 December 2024. Two lines from it matter here. The share of Googlebot fetches that were cacheable fell from 0.026% ten years earlier to 0.017% at the time of writing. And Google said it strongly recommends `ETag` over `Last-Modified` because it is "less prone to errors and mistakes," and that if you can, you should set both.

None of this depends on a crawler being AI. But AI crawlers are the reason it matters more now: they fetch the same pages repeatedly, many vendors bill or throttle by request, and the site owner pays for every full `200` that a `304` would have replaced. For the effect on bytes, see our [crawler traffic study](https://aiscan.site/blog/ai-crawler-traffic-waste-2026), where a conditional re-crawl cut transfer by 77.5% on the sites that supported it.

## Our own house comes first

On 3 October 2026 we requested five URLs on `aiscan.site` with a browser user-agent and read the response headers.

| URL | `ETag` | `Last-Modified` | `Cache-Control` |
|---|---|---|---|
| `/` | none | none | `no-cache, must-revalidate, max-age=0` |
| `/robots.txt` | none | none | not set |
| `/llms.txt` | none | none | `public, max-age=300, stale-while-revalidate=3600` |
| `/sitemap.xml` | none | none | `public, max-age=3600` |
| `/blog/ai-crawler-declaration-census` | none | none | `no-cache, must-revalidate, max-age=0` |

None of the five can ever return `304`. The same was true when we first measured it in September, and our own log of AI-bot requests then showed 120 hits, all `200`, none `304`. A site that scores 100 on our scanner sits in the "sends nothing" half of this funnel. That is not a contradiction, because the scanner does not grade validators. It is the reason this study exists: a clean readiness score and a cacheable site are separate properties, and the second one is invisible unless you look for it.

We have logged the fix as a platform item. We are not claiming the number moved.

## Method

**Population.** We downloaded the Tranco top-sites list on 3 October 2026, took the first 5,000 domains, sorted them by the MD5 of the hostname, and took the first 300. This is a deterministic sample of popular sites. It is not the AIScan scan corpus, and it is not a random sample of the web. Popular sites are more likely to sit behind a CDN, which skews every number below.

**Probes.** For each host we requested `https://<host>/` and `https://<host>/robots.txt` with `curl -L` and a desktop Chrome user-agent. We recorded the final status, content type, and whether the final response carried `ETag`, `Last-Modified`, or both. If the response was `200` and carried a validator, we repeated the request with `If-None-Match` and `If-Modified-Since` set to the values we received, and recorded the status.

**Counting rules.** A homepage counts only if the final response was `200` with an HTML content type. A `robots.txt` counts only if it was `200` with a `text/plain` content type, which rejects the soft-200 HTML pages many sites serve in its place. We counted a host as a validator sender only when the header appeared on both passes, and as honouring it only when both passes returned `304`.

**Two passes.** Following our standing rule to re-probe every differential, we ran the whole set twice, minutes apart, on the 209 hosts that answered `200` in the first pass. Nobody flipped between sending and not sending a validator. A few changed their conditional answer: 7 homepages and 5 `robots.txt` files returned `304` once and `200` once. We report those as "not stable" rather than rounding them either way.

**What did not make it in.** Of the 600 first-pass probes, 124 returned `502` (the proxy our sandbox runs behind could not complete the connection), 45 returned `403`, and 25 returned `404`. Only 156 homepages and 146 `robots.txt` files met the counting rules in pass one, and 152 and 144 on both passes. Hosts that block datacentre addresses are therefore under-represented, and they are exactly the hosts most likely to run strict bot management. Treat every rate below as a rate among sites that answer an unauthenticated datacentre request.

## The funnel

| Stage | Homepage | robots.txt |
|---|---|---|
| Usable responses (both passes) | 152 | 144 |
| Send a validator (both passes) | 77 (50.7%) | 120 (83.3%) |
| Answer `304` on both passes | 50 (32.9%) | 104 (72.2%) |
| Answer `304` on exactly one pass | 7 | 5 |
| Send a validator, never answer `304` | 20 | 11 |

The two files have almost the same shape on a good host and very different shapes across the population. `robots.txt` is a flat file, so the web server computes `Last-Modified` and `ETag` from the file system by default. The homepage is usually generated, and someone has to decide what its version label is.

### Stage one: who sends a validator at all

On the homepage, 75 of 152 sites sent neither header on either pass. On `robots.txt` the figure was 24 of 144. The 50-point gap between a generated page and a static file is the largest number in this study, and it matches what you would expect from how stacks work, not from site quality. A static file server such as nginx emits both headers by default; a framework rendering HTML often leaves the choice to the application, and many applications choose to set `Cache-Control: no-store` or `no-cache` and send nothing to validate against.

### Stage two: what kind of validator

In the first pass, among homepages that sent at least one validator (78 hosts):

| Validator sent | Hosts | Answered `304` | Rate |
|---|---|---|---|
| `Last-Modified` only | 35 | 27 | 77% |
| `ETag` and `Last-Modified` | 25 | 20 | 80% |
| `ETag` only | 18 | 6 | 33% |

For `robots.txt` (121 senders) the same split was 30 hosts with `Last-Modified` only (28 answered `304`), 85 with both (77 answered), and 6 with `ETag` only (4 answered).

The surprise is the homepage `ETag`-only row. Google recommends `ETag`, yet in our sample the sites that sent only an `ETag` were the least likely to honour it: 6 of 18. The sample is small, and we did not diagnose each host, so this is a hypothesis and not a finding. The likeliest candidates are an application that generates a fresh `ETag` per response (a hash of per-request content such as a nonce or timestamp), or a CDN that rewrites the label when it compresses the body, so the value the server stored never matches the one it later compares. If you send an `ETag`, test it. A validator that changes on every request costs bytes and buys nothing.

### Stage three: who then honours it

Of the 77 homepages that sent a validator on both passes, 50 answered `304` on both: 64.9%. Of the 120 `robots.txt` senders, 104: 86.7%. The remaining 27 homepages and 16 `robots.txt` files sent a validator that did not work, at least once. That group is the cheapest to fix, because the header is already there.

### Against our September numbers

Our [earlier sweep](https://aiscan.site/blog/ai-crawler-traffic-waste-2026) read 68.2% of successful responses carrying a validator and 51.8% answering `304`, across four surfaces on 60 hosts. This one reads 50.7% and 32.9% on the homepage, and 83.3% and 72.2% on `robots.txt`. The populations are different (60 hosts there, 152 and 144 here), the surfaces are different (this study drops `/llms.txt` and `/sitemap.xml`), and the pass criteria are stricter. Do not read the gap as a trend. Both studies agree on the direction: roughly half of HTML pages cannot answer "nothing changed", and static files do better.

### What a 304 saves on a homepage

For the 50 homepages that answered `304` on both passes, we ran a third pair of requests and recorded the body bytes. All 50 answered `304` again. The unconditional requests downloaded 13,853,038 bytes of body in total, with no `Accept-Encoding` header sent, so servers returned whatever they chose to by default. The conditional requests downloaded zero bytes of body, which is what a `304` means. Header bytes are excluded on both sides.

The spread matters more than the total. The median homepage was 97,874 bytes, the smallest 1,606 and the largest 2,145,277. A crawler that revalidates a 2 MB page saves 2 MB per visit; one that revalidates a 1.6 KB page saves almost nothing. The savings concentrate on the heavy pages, which are the ones a site owner most wants crawlers to stop re-downloading. Our September figure of 77.5% fewer bytes counted headers and mixed surfaces, so the two numbers are not interchangeable. The honest statement is narrower: when a homepage honours its validator, the body transfer disappears.

## What this means for an AI crawler

A crawler deciding how often to revisit a page has two choices when a site sends no validator: refetch the full body or guess. Refetching is the safe choice, so the bytes move. The site pays for the bandwidth and the crawler pays for the parse, and neither learns anything new.

Our [declaration census](https://aiscan.site/blog/ai-crawler-declaration-census) showed that most sites say nothing about AI crawlers in `robots.txt` at all. This study adds a quieter observation about the same population: on the homepage, half of them also give crawlers no way to ask cheaply whether the page changed. Neither is a defect in the strict sense. Both are defaults that nobody revisited after crawlers got more numerous.

## Where this sits in your readiness work

AIScan's discoverability dimension covers the files an agent finds first. Run the scan before touching headers: `npx aiscan-cli yoursite.com`, or paste your URL at [aiscan.site](https://aiscan.site/), free and without an account. Check D1 and D2 (see [the discoverability checks](https://aiscan.site/docs/checks/discoverability)) tell you whether your sitemap and the files around it exist and parse. They do not tell you whether those files can be revalidated, because validators are not part of the rubric today. That is a limit of the tool, and it is why the manual test below exists.

If you run WordPress, a single SEO plugin that owns `robots.txt` and the sitemap removes one source of conflicting headers. [ThinkRank](https://thinkrank.ai) does robots.txt, robots meta, schema and llms.txt from one plugin, and migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. It does not set your page cache headers, though. That is a job for your caching layer or host, and the plugins named above do not do it either. For a broader view of how these files fit together, see [llms.txt vs robots.txt vs sitemap.xml](https://aiscan.site/blog/llms-txt-vs-robots-txt-vs-sitemap).

## Check your own site by hand

If you would rather check by hand, two requests settle it. The first command prints the validators; the second sends them back.

```bash
curl -sI https://yoursite.com/ | grep -iE '^(etag|last-modified|cache-control)'
curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'If-None-Match: "<etag from above>"' \
  -H 'If-Modified-Since: <last-modified from above>' \
  https://yoursite.com/
```

Read the results this way:

| First request shows | Second request returns | Meaning |
|---|---|---|
| nothing | n/a | Stage one failed: no validator to send |
| a validator | `304` | Healthy |
| a validator | `200` | The header is decorative: find what rewrites or regenerates it |
| a validator | `304` once, `200` on retry | Unstable: probably several origin servers or a CDN node mismatch |

Test the homepage and one deep URL, because your static files and your rendered pages are served by different code. Repeat the second request several times. Several of our unstable hosts passed once and failed once.

Common fixes, in the order we would try them:

1. **Static files:** leave `etag on` and the default `Last-Modified` in your web server. nginx does both for files on disk.
2. **Rendered pages:** make your framework or CDN emit a stable `ETag` that depends on content, not on the request. Next.js generates one for pages by default unless you turn `generateEtags` off.
3. **Compression:** check whether your CDN turns a strong `ETag` into a weak one when it compresses. A weak `ETag` still works for `If-None-Match`, but a mismatch between origin and edge will not.
4. **Both headers:** Google's own advice is to set both, so a client that stores only one can still revalidate.

## What we could not measure

- **Real crawler behaviour.** We are a curl client. We cannot see whether GPTBot, ClaudeBot or PerplexityBot actually send conditional headers; our own request log is too small to say. The funnel's third stage tells you whether a site would answer, not whether anyone asks.
- **One vantage point.** All probes came from one datacentre address. Sites that vary responses by network or run bot management will look different from a residential line.
- **Two surfaces.** The homepage and `robots.txt` only. Sitemaps, `llms.txt` and article pages may behave differently, and our September study found the sitemap the weakest of the four.
- **Popular sites.** The sample is Tranco's top 5,000, so small sites are absent.
- **Why a validator fails.** We recorded that it did, not the cause.

## Next steps

Run the scan on your own site today: `npx aiscan-cli yoursite.com` covers D1 and D2, the checks that decide whether an agent finds your sitemap and discovery files. Then run the two-request test above and fix whichever stage fails first. For more checks and fixes, browse the [AIScan guides](https://aiscan.site/guides), or start from [the State of AI Agent Readiness study](https://aiscan.site/blog/state-of-ai-agent-readiness-2026) to see where your score sits against 473 other sites.