What a real 404 should return to an AI agent

Quick summary

You already know the first rule of a good 404: return status 404, not 200. That half of the check is a status line, and our own CI-gate guide already covers how to catch the "soft 404" version of getting it wrong. This post picks up where that one stops: once the status is correct, what should the body actually contain, and does it change for a request that says Accept: application/json instead of a browser's Accept: text/html? For most sites, verified against our own corpus, the answer is: nothing changes, and it should.

The status line and the body answer two different readers

A 404 response is really two documents wearing one status code. RFC 9110 §15.5.5, which AIScan's E1 check cites directly, defines 404 Not Found as nothing more than "the origin server did not find a current representation for the target resource." That's the machine-checkable half: a script, a crawler, or a scanner reads the three-digit code and moves on. The body is the other half, and it exists for whoever, or whatever, is still there after the status line: a person deciding where to click next, or an agent deciding whether the page is worth remembering at all.

Treat the two as one problem and you get exactly the failure our corpus measures. Across AIScan's own scan history, E1 currently passes on 372 of 499 sites, 74.5%, re-measured 9 September 2026. Most of the remainder aren't sites with a missing 404 handler; they're sites returning 200 for a page that plainly doesn't exist, which is the "soft 404" the CI-gate post already diagnoses in depth. What's left, once the status line is fixed, is the narrower and less-discussed failure this post is about: a correct 404 whose body is a dead end for anything that isn't a browser rendering an error page for a human to read.

What AIScan's own docs say the body needs

According to the check's own documentation, the bar is specific: "an empty or near-empty 404 is a dead end. Full credit needs a body that names the site and links back to somewhere useful," and the fix guidance names the homepage, sitemap, docs and search as the recovery links to render. That's a low, sensible bar, and it's a human-readable one: it says what an HTML page should contain, not what a non-HTML request should receive instead. Nothing in the check's own fix guide mentions the Accept header, and that gap is exactly where a genuinely useful 404 starts.

RFC 9457: the shape for the half a browser doesn't need

RFC 9457, "Problem Details for HTTP APIs," gives that machine-facing half a name and a format instead of leaving every site to invent its own. It obsoletes RFC 7807 and defines a small JSON object, served as application/problem+json, with five members an error response can carry:

For a plain "this page doesn't exist," the spec's own worked case is the simplest one: use type: "about:blank" and set title to the standard reason phrase for the status code, "Not Found" for a 404, "which MAY be localized to suit client preferences." No custom problem type is required to be useful, and RFC 9457 says so directly.

A custom type URI earns its keep only once a site has more than one kind of 404 worth telling apart, which most sites don't, and forcing one before then just adds a URI nobody dereferences. The cases where it's worth the extra work: a page that used to exist and was deliberately removed, versus a URL that never existed at all, versus a slug that changed and should carry a redirect instead of a 404 in the first place. Each of those is a different detail string at minimum, and a different type URI once an agent needs to branch on which one happened rather than just logging that something failed. Until a site has that distinction to make, about:blank plus a specific detail string naming the missing path does the job the spec describes.

The mechanism that decides which half a given request gets is ordinary HTTP, not anything AIScan-specific: proactive content negotiation on the Accept request header, per RFC 9110 §12.5.1, fetched from the same RFC series E1 already cites for the status code itself. A browser sends Accept: text/html and gets the human page with its homepage and sitemap links; a script or agent that sends Accept: application/json gets back application/problem+json with the same information in a shape it can actually parse. Same URL, same status code, two representations. That's what content negotiation means, in the RFC's own words, not a house convention this post is inventing.

Reading our own 404 honestly

This is a case where dogfooding our own product matters more than citing someone else's site: verified on 23 September 2026, directly against the live route rather than assumed from memory. A request for a path that cannot exist returns a real 404, and the body is a genuinely good instance of what the check's fix guide describes, in its own words: it names the site ("AIScan.site · HTTP 404"), explains in one sentence what AIScan does, and links six recovery routes in the rendered body: the homepage, /docs, /blog, sitemap.xml, llms.txt and openapi.json. It even carries four Link: relations on the 404 response itself, including rel="api-catalog" and rel="describedby" pointing at /llms.txt. On the human half of E1, that's close to the ceiling the check describes.

Ask the same URL for application/json and the honest result, also verified on 23 September 2026, is worse: HTTP/2 500 with {"error":"Only HTML requests are supported here"}, content-type: application/json. That's not a problem document; it's a generic API error surfacing because the request matched a catch-all route rather than the page-not-found handler at all. AIScan doesn't grade this half of the interaction yet, which is exactly why it's worth disclosing here rather than only in sites we've measured. A scanner that only checks the status line and the HTML body, ours included, will keep missing it.

Path one: a build-time 404 on a static site

Static generators serve 404.html (or an equivalent source file) as a fixed artifact from the last build, which makes the status-line half free and the body half a one-time decision.

1. Create the 404 source file at the path your framework expects: static/404.html on Hugo, src/pages/404.astro on Astro, static/404.html or a NotFound component route on Docusaurus. Check the framework's own docs for the exact path; guessing it is how a 404 template silently never ships.
2. Confirm the host serves it with status 404, not 200. A static host that can't find a matching file sometimes falls back to serving index.html with a 200, which turns your carefully built error page into a soft 404 for the one class of visitor it exists for. Check with:

curl -o /dev/null -w '%{http_code}\n' https://yoursite.com/this-cannot-exist-123
# must print: 404

1. Put real links in the body, not placeholder text: the homepage, the sitemap, and a documentation or search entry point, matching what E1's fix guide names. Six months from now those links should still resolve; check them the same way you'd check any other page's outbound links.
2. Add a machine-readable twin if your host supports an edge function or middleware route. Static hosts that can run a function at the edge (Cloudflare Pages Functions, Netlify Edge Functions, a Vercel middleware.ts) can branch on the Accept header for exactly one route, the 404, and return application/problem+json there without touching the rest of the static build.
3. Re-deploy and re-run step 2. A 404 template is only as good as the deploy that actually shipped it; verify against the live URL, not the local build output.

Path two: a server-rendered 404 with content negotiation

An application server, Express, a Next.js route handler, a WordPress theme's 404.php, has a request object to branch on, which makes true content negotiation straightforward rather than requiring a separate edge function.

1. Write a dedicated not-found handler rather than letting a wildcard route silently return 200. This is the same defect the CI-gate post measured directly on Replit's catch-all: ten of twelve tested apps returned 200 text/html for a path that had never existed, and six returned a byte-identical body for every unknown path tried. A catch-all that always resolves is the single most common way E1 fails.
2. Inspect the Accept header and branch, in Express terms:

app.use((req, res) => {
  const wantsJson = req.accepts(['html', 'json']) === 'json';
  res.status(404);
  if (wantsJson) {
    res.type('application/problem+json').json({
      type: 'about:blank',
      title: 'Not Found',
      status: 404,
      detail: `No resource exists at ${req.path}.`,
      instance: req.originalUrl,
    });
  } else {
    res.type('html').send(render404Page(req.path)); // homepage, sitemap, docs, search
  }
});

1. On WordPress, the equivalent lives in the active theme's 404.php, which by default only ever renders HTML. A theme can call wp_is_json_request() or inspect $_SERVER['HTTP_ACCEPT'] directly to serve application/problem+json from the same template before it falls through to the human page. The recovery-links half of the human page, and whether the 404 template itself carries a stray noindex or the wrong canonical, is exactly the kind of per-template robots-meta detail ThinkRank manages from one settings screen instead of a hand-edited theme file, alongside the robots.txt and schema rules a WordPress AI-readiness setup already needs to get right.
2. Verify with both Accept values against the live route, not the handler in isolation. A negotiation bug that only shows up once the route is deployed behind a CDN or reverse proxy is common enough that "it worked when I curled localhost" isn't a verification step.

Verify it actually works

Three commands, each with a concrete pass or fail signal:

# 1. The status line, for a browser-style request
curl -sI https://yoursite.com/this-page-does-not-exist-123 | head -n 1
# must read: HTTP/2 404 (or HTTP/1.1 404)

# 2. The negotiated body, for a machine-style request
curl -s -H "Accept: application/json" https://yoursite.com/this-page-does-not-exist-123 \
  | head -c 300
# must be a JSON object, ideally content-type: application/problem+json,
# never an HTML page and never a bare {"error":"..."} with no "status" or "title"

# 3. The control path, so a redirect-everything bug can't hide
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://yoursite.com/a-real-page
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://yoursite.com/this-page-does-not-exist-123
# the two lines must differ in status; identical output on both is the soft-404
# pattern the CI-gate post measured on Ghost, Replit and one Squarespace site

The third command is the one worth not skipping. A 404 handler that quietly redirects every unmatched path back to the homepage passes a lazy status-only check while still failing the actual reader, because a real page and a nonexistent one return the same thing. Compare the two responses, not just the one you're fixing.

What AIScan checks, and what it still can't see

E1 is weighted 4 and marked essential in the rubric, and it grades two things directly: the status code is a real 404 (or 410) rather than a soft 200, and the HTML body names the site and links somewhere useful. Run npx aiscan-cli yoursite.com, or paste the URL at aiscan.site, and E1's row in the report will say which half, if either, failed.

What E1 does not grade, today, is everything this post adds on top: whether a non-HTML request gets a non-HTML answer at all, whether that answer is valid application/problem+json, and whether type, title and status are populated the way RFC 9457 recommends. A site can score full marks on E1 with a 500-returning catch-all sitting one header away, the same gap aiscan.site's own 404 route has right now. Treat a clean E1 score as proof the human half is done, not proof the machine half exists, until a scanner actually tests the Accept: application/json path the way this post's second command does.

For the discovery layer the recovery links point at, the docs page at /docs/checks/discoverability covers E1 alongside the robots.txt and sitemap checks that decide whether an agent finds the real pages in the first place, and /guides collects the rest of this series, including the CI-gate guide this post builds directly on.