---
title: "Add ETag and Last-Modified so crawlers can send conditional requests"
slug: conditional-requests-etag-last-modified
published: 2026-10-27T08:14:18.272288+00:00
updated: 2026-10-27T08:14:18.272288+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:D2, check:D1, etag, last-modified, conditional-requests, http-caching, how-to
description: "Send an ETag or Last-Modified header and answer repeat requests with a 304. Setup for nginx, Apache, Node, PHP, Next.js and Cloudflare, plus a two-request test."
url: https://aiscan.site/blog/conditional-requests-etag-last-modified
---

To make a page answer `304 Not Modified`, send an `ETag` or a `Last-Modified` header on the first response, then answer a repeat request that carries `If-None-Match` or `If-Modified-Since` with status 304 and no body. Most web servers do this for static files on their own. Dynamic pages (WordPress, Node, PHP, a framework route) usually do not, and a CDN can quietly undo it. This guide shows the headers for six stacks and a two-request test that proves it, with results from tests we ran on 27 October 2026.

Written by [M Asif Rahman](https://ar.bd). The adoption numbers live in our [300-host ETag study](https://aiscan.site/blog/etag-last-modified-adoption-corpus); this post is the how-to.

## Quick summary

| If you run | Do this | Time | Proof it worked |
|---|---|---|---|
| Static files on nginx or Apache | Check the defaults are on (`etag on;`, `FileETag MTime Size`) | 5 min | `curl -I` shows `ETag`, a repeat request returns 304 |
| A Node, PHP or Python route | Hash the body, send `ETag`, compare `If-None-Match`, return 304 | 20 min | Same two-request test |
| Next.js | Leave `generateEtags` on (the default) for pages you want revalidated | 2 min | `ETag` on the HTML response |
| Behind Cloudflare | Keep origin ETags, decide on "Respect Strong ETags" | 10 min | 304 through the edge, not just at origin |

**The short version:** send a validator that changes only when the content changes, answer the matching conditional request with an empty 304, and test through the same CDN your crawlers hit. [Run `npx aiscan-cli yoursite.com`](https://aiscan.site/) first to see how the files around this (sitemap, robots.txt) grade under `check:D2` and `check:D1`. AIScan does not grade cache validators itself, so the two-request test below is manual.

## Why a validator saves a crawler real work

A validator is a label the server attaches to one version of a response. [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html) defines two: `ETag` (an opaque version string) and `Last-Modified` (a date). A client that kept the response sends the label back in `If-None-Match` or `If-Modified-Since`. If nothing changed, the server replies 304 with headers only.

Think of a courier collecting a parcel that is usually unchanged. Without a validator the courier carries the whole parcel back every visit. With one, the courier shows last week's receipt and the sender says "same parcel, keep what you have". Google's Search Central team describes the same flow in [its December 2024 post on HTTP caching](https://developers.google.com/search/blog/2024/12/crawling-december-caching) and recommends `ETag` over `Last-Modified`, with both set if you can.

Crawlers are not obliged to send conditional requests, and some never do. The cost of sending the headers is small and the benefit is that the ones that do can re-crawl cheaply. For measured adoption see the [study linked above](https://aiscan.site/blog/etag-last-modified-adoption-corpus) and the [crawler traffic waste data](https://aiscan.site/blog/ai-crawler-traffic-waste-2026).

## What we tested

On 27 October 2026 we requested nine URLs on our four sites, then repeated each request with the validator it returned. Then we ran four small servers in a container and broke them on purpose.

| URL (n = 9) | `ETag` | `Last-Modified` | Conditional result |
|---|---|---|---|
| xspeedcache.com `/` | yes | yes | 304 on both |
| xspeedcache.com `/robots.txt` | yes | yes | 304 on both |
| storeseo.com `/` | no | yes | 304 on `If-Modified-Since` |
| storeseo.com `/robots.txt` | weak (`W/"…"`) | yes | 304 on both |
| bettershield.ai `/` | yes | no | 304 on `If-None-Match` |
| bettershield.ai `/robots.txt` | yes | no | 304 on `If-None-Match` |
| aiscan.site `/` | no | no | nothing to send back |
| aiscan.site `/robots.txt` | no | no | nothing to send back |
| aiscan.site, a blog post | no | no | nothing to send back |

Six of nine URLs answered 304. The three that did not are all on this site, which sends `Cache-Control: no-cache, must-revalidate, max-age=0` on its pages and no validator at all. That is the same finding as the study, and we have not fixed it yet. A page can tell crawlers to revalidate and still give them nothing to revalidate with.

| Local server (27 Oct 2026) | Validator sent | Conditional result |
|---|---|---|
| Node `http`, hand-written | `ETag` and `Last-Modified` | 304 on both; 200 again after we edited the file |
| Python `http.server` | `Last-Modified` only | 304 on `If-Modified-Since` |
| PHP built-in server, static file | none | no 304 possible |
| PHP script with a random `ETag` | `ETag` and `Last-Modified` | **200 every time** |

The last row is the mistake worth avoiding. A validator that changes on every request is worse than none: the header looks present, so a scan that only checks for its presence passes, and no crawler ever gets a 304.

## Path 1: static files on nginx or Apache (check the defaults)

nginx generates `ETag` for static files by default. Per the [nginx core module docs](https://nginx.org/en/docs/http/ngx_http_core_module.html), `etag` is `on` unless you set it otherwise, and `if_modified_since` defaults to `exact`. Apache's [`FileETag` directive](https://httpd.apache.org/docs/2.4/mod/core.html) defaults to `MTime Size` since 2.3.15. (Apache documents the older default as `INode MTime Size`, which gave different ETags on different servers behind a load balancer.)

1. Request a static file and read the headers: `curl -sI https://example.com/robots.txt`.
2. If `ETag` is missing on nginx, search your config for `etag off;` and remove it from the `http`, `server` or `location` block that serves the file. Reload nginx.
3. If `ETag` is missing on Apache, search for `FileETag None` and replace it with `FileETag MTime Size`. Reload Apache.
4. If you proxy to an application, the proxy only adds validators to files it serves itself. Your application has to send its own, which is Path 2.
5. Run the two-request test below.

nginx and Apache may turn a strong ETag into a weak one when they compress a response on the fly, and the exact behaviour varies by version and module. A weak ETag still produces 304s for `If-None-Match`; it only matters for byte-range requests.

## Path 2: dynamic pages (Node, PHP, Python, any framework)

The server has to do two things: compute a validator that depends only on content, and short-circuit when the request matches it. The tested Node version:

```js
const crypto = require('crypto');
// body = the final HTML string or buffer you are about to send
const etag = '"' + crypto.createHash('sha256').update(body).digest('base64url').slice(0, 16) + '"';
res.setHeader('ETag', etag);
res.setHeader('Last-Modified', new Date(updatedAt).toUTCString());
res.setHeader('Cache-Control', 'no-cache');
const inm = req.headers['if-none-match'];
if (inm && inm.split(',').map(s => s.trim().replace(/^W\//, '')).includes(etag)) {
  res.statusCode = 304;
  return res.end();
}
res.end(body);
```

The PHP equivalent that passed our test:

```php
$etag = '"' . substr(hash('sha256', $body), 0, 16) . '"';
header('ETag: ' . $etag);
header('Last-Modified: ' . gmdate('D, d M Y H:i:s', $mtime) . ' GMT');
header('Cache-Control: no-cache');
if (($_SERVER['HTTP_IF_NONE_MATCH'] ?? '') === $etag) { http_response_code(304); exit; }
echo $body;
```

1. Build the final body first, with nothing time-dependent in it (no "generated at" stamp, no per-request nonce, no rotating ad ID).
2. Hash the body and send it as a quoted `ETag`. Send `Last-Modified` from the real last-edited time of the content, never from "now".
3. If the request has `If-None-Match`, compare against it and ignore `If-Modified-Since`. RFC 9110 gives `If-None-Match` priority.
4. On a match, return 304 with the same validators and no body.
5. Keep `Cache-Control: no-cache` if you want crawlers to ask every time. (`no-cache` means "revalidate before reuse", not "do not store".)

Hashing the body costs CPU on every request, so on a busy site store the hash with the page in your cache. For a Next.js app, [`generateEtags`](https://nextjs.org/docs/app/api-reference/config/next-config-js/generateEtags) is on by default and you only touch it if you turned it off.

## Path 3: a CDN in front (Cloudflare)

A CDN can make a good origin look broken. Cloudflare's [ETag documentation](https://developers.cloudflare.com/cache/reference/etag-headers/) says it supports strong and weak ETags set at your origin. With weak ETags you have to turn off features that rewrite HTML, such as Email Obfuscation and Automatic HTTPS Rewrites, or the edge can strip the header. The [Cache Rules setting "Respect Strong ETags"](https://developers.cloudflare.com/cache/how-to/cache-rules/settings/) controls whether Cloudflare keeps strong ETags; if it is off, Cloudflare converts them to weak ones. (Our crawl-control guide covers the dashboard side: [Cloudflare AI Crawl Control setup](https://aiscan.site/blog/cloudflare-ai-crawl-control-setup).)

1. Run the two-request test against the public hostname, not the origin IP.
2. If the origin sends a strong ETag and the edge returns a weak one, check **Caching > Cache Rules** for a rule with Respect Strong ETags off.
3. If the validator vanishes only on HTML, check the HTML-rewriting features named above.
4. Repeat the test after any change to the rules.

## Verify: the two-request test

Lead with the scan: `npx aiscan-cli yoursite.com` grades your sitemap and discovery files (`check:D2`, `check:D1`) so you know the files crawlers depend on are reachable. Then, because AIScan does not grade validators, check them by hand:

```bash
URL=https://example.com/
ET=$(curl -sI "$URL" | awk -F': ' 'tolower($1)=="etag"{print $2}' | tr -d '\r')
curl -s -o /dev/null -w '%{http_code}\n' -H "If-None-Match: $ET" "$URL"
```

| Result | Meaning | First fix |
|---|---|---|
| `ET` is empty | No validator sent | Path 1 or 2 |
| `304` | Working | Repeat for `Last-Modified` with `If-Modified-Since` |
| `200` | Validator sent but not honoured | Check the validator changes per request (Path 2 step 1) or a proxy dropped the header (Path 3) |
| Two runs give different `ET` values | Validator is time-dependent | Remove the timestamp or nonce from the body |

Run it twice, a minute apart. The `ET` values must match when the content has not changed.

## Who can skip this

A site that is mostly static and sits on nginx or Apache with defaults needs only the test. A site whose pages differ per visitor should only send validators on the public, anonymous version. And if your hosting layer already caches the page (see [rate limiting without blocking crawlers](https://aiscan.site/blog/rate-limit-ai-crawlers-without-blocking) for what to exempt), confirm the validators survive that layer before spending time on the origin.

## Maintenance

Re-run the test after a theme or framework upgrade, a CDN rule change, or adding a plugin that injects a script with a timestamp. Any of these can silently change the body on every request. The [robots.txt status and content type guide](https://aiscan.site/blog/robots-txt-status-content-type) is a good companion check, since robots.txt is the first file a crawler asks for.

## Next steps

Run `npx aiscan-cli yoursite.com` or paste your URL at [aiscan.site](https://aiscan.site/) to grade `check:D1` and `check:D2`, run the two-request test on your homepage, sitemap and robots.txt, and read the other fixes in the [guides](https://aiscan.site/guides) and the [discoverability checks](https://aiscan.site/docs/checks/discoverability).