Add ETag and Last-Modified so crawlers can send conditional requests

To make a page answer 304 Not Modified, send an ETag or a Last-Modified header on the first response, then answer a repeat request that carries If-None-Match or If-Modified-Since with status 304 and no body. Most web servers do this for static files on their own. Dynamic pages (WordPress, Node, PHP, a framework route) usually do not, and a CDN can quietly undo it. This guide shows the headers for six stacks and a two-request test that proves it, with results from tests we ran on 27 October 2026.

Written by M Asif Rahman. The adoption numbers live in our 300-host ETag study; this post is the how-to.

Quick summary

The short version: send a validator that changes only when the content changes, answer the matching conditional request with an empty 304, and test through the same CDN your crawlers hit. Run npx aiscan-cli yoursite.com first to see how the files around this (sitemap, robots.txt) grade under check:D2 and check:D1. AIScan does not grade cache validators itself, so the two-request test below is manual.

Why a validator saves a crawler real work

A validator is a label the server attaches to one version of a response. RFC 9110 defines two: ETag (an opaque version string) and Last-Modified (a date). A client that kept the response sends the label back in If-None-Match or If-Modified-Since. If nothing changed, the server replies 304 with headers only.

Think of a courier collecting a parcel that is usually unchanged. Without a validator the courier carries the whole parcel back every visit. With one, the courier shows last week's receipt and the sender says "same parcel, keep what you have". Google's Search Central team describes the same flow in its December 2024 post on HTTP caching and recommends ETag over Last-Modified, with both set if you can.

Crawlers are not obliged to send conditional requests, and some never do. The cost of sending the headers is small and the benefit is that the ones that do can re-crawl cheaply. For measured adoption see the study linked above and the crawler traffic waste data.

What we tested

On 27 October 2026 we requested nine URLs on our four sites, then repeated each request with the validator it returned. Then we ran four small servers in a container and broke them on purpose.

Six of nine URLs answered 304. The three that did not are all on this site, which sends Cache-Control: no-cache, must-revalidate, max-age=0 on its pages and no validator at all. That is the same finding as the study, and we have not fixed it yet. A page can tell crawlers to revalidate and still give them nothing to revalidate with.

The last row is the mistake worth avoiding. A validator that changes on every request is worse than none: the header looks present, so a scan that only checks for its presence passes, and no crawler ever gets a 304.

Path 1: static files on nginx or Apache (check the defaults)

nginx generates ETag for static files by default. Per the nginx core module docs, etag is on unless you set it otherwise, and if_modified_since defaults to exact. Apache's FileETag directive defaults to MTime Size since 2.3.15. (Apache documents the older default as INode MTime Size, which gave different ETags on different servers behind a load balancer.)

1. Request a static file and read the headers: curl -sI https://example.com/robots.txt.
2. If ETag is missing on nginx, search your config for etag off; and remove it from the http, server or location block that serves the file. Reload nginx.
3. If ETag is missing on Apache, search for FileETag None and replace it with FileETag MTime Size. Reload Apache.
4. If you proxy to an application, the proxy only adds validators to files it serves itself. Your application has to send its own, which is Path 2.
5. Run the two-request test below.

nginx and Apache may turn a strong ETag into a weak one when they compress a response on the fly, and the exact behaviour varies by version and module. A weak ETag still produces 304s for If-None-Match; it only matters for byte-range requests.

Path 2: dynamic pages (Node, PHP, Python, any framework)

The server has to do two things: compute a validator that depends only on content, and short-circuit when the request matches it. The tested Node version:

const crypto = require('crypto');
// body = the final HTML string or buffer you are about to send
const etag = '"' + crypto.createHash('sha256').update(body).digest('base64url').slice(0, 16) + '"';
res.setHeader('ETag', etag);
res.setHeader('Last-Modified', new Date(updatedAt).toUTCString());
res.setHeader('Cache-Control', 'no-cache');
const inm = req.headers['if-none-match'];
if (inm && inm.split(',').map(s => s.trim().replace(/^W\//, '')).includes(etag)) {
  res.statusCode = 304;
  return res.end();
}
res.end(body);

The PHP equivalent that passed our test:

$etag = '"' . substr(hash('sha256', $body), 0, 16) . '"';
header('ETag: ' . $etag);
header('Last-Modified: ' . gmdate('D, d M Y H:i:s', $mtime) . ' GMT');
header('Cache-Control: no-cache');
if (($_SERVER['HTTP_IF_NONE_MATCH'] ?? '') === $etag) { http_response_code(304); exit; }
echo $body;

1. Build the final body first, with nothing time-dependent in it (no "generated at" stamp, no per-request nonce, no rotating ad ID).
2. Hash the body and send it as a quoted ETag. Send Last-Modified from the real last-edited time of the content, never from "now".
3. If the request has If-None-Match, compare against it and ignore If-Modified-Since. RFC 9110 gives If-None-Match priority.
4. On a match, return 304 with the same validators and no body.
5. Keep Cache-Control: no-cache if you want crawlers to ask every time. (no-cache means "revalidate before reuse", not "do not store".)

Hashing the body costs CPU on every request, so on a busy site store the hash with the page in your cache. For a Next.js app, generateEtags is on by default and you only touch it if you turned it off.

Path 3: a CDN in front (Cloudflare)

A CDN can make a good origin look broken. Cloudflare's ETag documentation says it supports strong and weak ETags set at your origin. With weak ETags you have to turn off features that rewrite HTML, such as Email Obfuscation and Automatic HTTPS Rewrites, or the edge can strip the header. The Cache Rules setting "Respect Strong ETags" controls whether Cloudflare keeps strong ETags; if it is off, Cloudflare converts them to weak ones. (Our crawl-control guide covers the dashboard side: Cloudflare AI Crawl Control setup.)

1. Run the two-request test against the public hostname, not the origin IP.
2. If the origin sends a strong ETag and the edge returns a weak one, check Caching > Cache Rules for a rule with Respect Strong ETags off.
3. If the validator vanishes only on HTML, check the HTML-rewriting features named above.
4. Repeat the test after any change to the rules.

Verify: the two-request test

Lead with the scan: npx aiscan-cli yoursite.com grades your sitemap and discovery files (check:D2, check:D1) so you know the files crawlers depend on are reachable. Then, because AIScan does not grade validators, check them by hand:

URL=https://example.com/
ET=$(curl -sI "$URL" | awk -F': ' 'tolower($1)=="etag"{print $2}' | tr -d '\r')
curl -s -o /dev/null -w '%{http_code}\n' -H "If-None-Match: $ET" "$URL"

Run it twice, a minute apart. The ET values must match when the content has not changed.

Who can skip this

A site that is mostly static and sits on nginx or Apache with defaults needs only the test. A site whose pages differ per visitor should only send validators on the public, anonymous version. And if your hosting layer already caches the page (see rate limiting without blocking crawlers for what to exempt), confirm the validators survive that layer before spending time on the origin.

Maintenance

Re-run the test after a theme or framework upgrade, a CDN rule change, or adding a plugin that injects a script with a timestamp. Any of these can silently change the body on every request. The robots.txt status and content type guide is a good companion check, since robots.txt is the first file a crawler asks for.

Next steps

Run npx aiscan-cli yoursite.com or paste your URL at aiscan.site to grade check:D1 and check:D2, run the two-request test on your homepage, sitemap and robots.txt, and read the other fixes in the guides and the discoverability checks.