---
title: "What a real 404 should return to an AI agent"
slug: correct-404-for-ai-agents
published: 2026-09-23T08:22:16.247828+00:00
updated: 2026-09-23T08:22:16.247828+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: 404 handling, RFC 9457, problem details, soft 404, AI readiness, content negotiation, llms.txt, check:E1
description: "A 404 status code alone is not enough for an AI agent. What the response body should contain, RFC 9457 problem+json, and how to test both halves with curl."
url: https://aiscan.site/blog/correct-404-for-ai-agents
---

## Quick summary

You already know the first rule of a good 404: return status `404`, not `200`. That half of the check is a status line, and [our own CI-gate guide](https://aiscan.site/blog/aiscan-ci-gate-github-actions) already covers how to catch the "soft 404" version of getting it wrong. This post picks up where that one stops: once the status is correct, what should the *body* actually contain, and does it change for a request that says `Accept: application/json` instead of a browser's `Accept: text/html`? For most sites, verified against our own corpus, the answer is: nothing changes, and it should.

| Question | Short answer |
|---|---|
| Is a `404` status enough on its own? | No. AIScan's E1 check also grades the body, and a blank or near-empty error page loses credit even with the right status |
| What's missing that E1 doesn't grade? | A machine-readable twin. Ask most sites for `/nonexistent-page` with `Accept: application/json` and you still get an HTML page, or a generic server error |
| What format should the machine-readable version use? | `application/problem+json`, defined in [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457.html), with `type`, `title`, `status`, `detail` and `instance` fields |
| Does our own site do this? | Not yet. `aiscan.site`'s 404 page is a genuinely good human-facing example, but asking it for JSON returns a `500` with a generic error, not a problem document |
| How do I test both halves at once? | `curl -I` for the status line, then the same request with `-H "Accept: application/json"` for the negotiated body: two commands, covered below |

## The status line and the body answer two different readers

A 404 response is really two documents wearing one status code. [RFC 9110 §15.5.5](https://www.rfc-editor.org/rfc/rfc9110.html#name-404-not-found), which AIScan's E1 check cites directly, defines `404 Not Found` as nothing more than "the origin server did not find a current representation for the target resource." That's the machine-checkable half: a script, a crawler, or a scanner reads the three-digit code and moves on. The body is the other half, and it exists for whoever, or whatever, is still there after the status line: a person deciding where to click next, or an agent deciding whether the page is worth remembering at all.

Treat the two as one problem and you get exactly the failure our corpus measures. Across AIScan's own scan history, E1 currently passes on 372 of 499 sites, 74.5%, re-measured 9 September 2026. Most of the remainder aren't sites with a missing 404 handler; they're sites returning `200` for a page that plainly doesn't exist, which is the "soft 404" the CI-gate post already diagnoses in depth. What's left, once the status line is fixed, is the narrower and less-discussed failure this post is about: a *correct* 404 whose body is a dead end for anything that isn't a browser rendering an error page for a human to read.

## What AIScan's own docs say the body needs

According to the check's own documentation, the bar is specific: "an empty or near-empty 404 is a dead end. Full credit needs a body that names the site and links back to somewhere useful," and the fix guidance names the homepage, sitemap, docs and search as the recovery links to render. That's a low, sensible bar, and it's a human-readable one: it says what an HTML page should contain, not what a non-HTML request should receive instead. Nothing in the check's own fix guide mentions the `Accept` header, and that gap is exactly where a genuinely useful 404 starts.

## RFC 9457: the shape for the half a browser doesn't need

[RFC 9457](https://www.rfc-editor.org/rfc/rfc9457.html), "Problem Details for HTTP APIs," gives that machine-facing half a name and a format instead of leaving every site to invent its own. It obsoletes RFC 7807 and defines a small JSON object, served as `application/problem+json`, with five members an error response can carry:

| Member | What it holds |
|---|---|
| `type` | A URI identifying the problem type. Left out, it defaults to `about:blank`, meaning "nothing beyond the HTTP status code itself" |
| `status` | The same status code the response line already carries, repeated for a consumer reading the body in isolation |
| `title` | A short, human-readable summary that shouldn't change between occurrences of the same problem |
| `detail` | A human-readable explanation specific to *this* occurrence, meant to help the client fix things, not to dump debugging internals |
| `instance` | A URI identifying this specific occurrence, useful for support or logs |

For a plain "this page doesn't exist," the spec's own worked case is the simplest one: use `type: "about:blank"` and set `title` to the standard reason phrase for the status code, "Not Found" for a 404, "which MAY be localized to suit client preferences." No custom problem type is required to be useful, and RFC 9457 says so directly.

A custom `type` URI earns its keep only once a site has more than one kind of 404 worth telling apart, which most sites don't, and forcing one before then just adds a URI nobody dereferences. The cases where it's worth the extra work: a page that used to exist and was deliberately removed, versus a URL that never existed at all, versus a slug that changed and should carry a redirect instead of a 404 in the first place. Each of those is a different `detail` string at minimum, and a different `type` URI once an agent needs to branch on which one happened rather than just logging that something failed. Until a site has that distinction to make, `about:blank` plus a specific `detail` string naming the missing path does the job the spec describes.

The mechanism that decides which half a given request gets is ordinary HTTP, not anything AIScan-specific: proactive content negotiation on the `Accept` request header, per [RFC 9110 §12.5.1](https://www.rfc-editor.org/rfc/rfc9110.html#name-accept), fetched from the same RFC series E1 already cites for the status code itself. A browser sends `Accept: text/html` and gets the human page with its homepage and sitemap links; a script or agent that sends `Accept: application/json` gets back `application/problem+json` with the same information in a shape it can actually parse. Same URL, same status code, two representations. That's what content negotiation means, in the RFC's own words, not a house convention this post is inventing.

| | Status line grades a real `404` | Body names the site and links out | Negotiates a machine-readable body on `Accept: application/json` |
|---|---|---|---|
| **What AIScan's E1 checks today** | Yes | Yes | No |
| **What RFC 9457 adds on top** | n/a | n/a | Yes, as `application/problem+json` |

## Reading our own 404 honestly

This is a case where dogfooding our own product matters more than citing someone else's site: verified on 23 September 2026, directly against the live route rather than assumed from memory. A request for a path that cannot exist returns a real `404`, and the body is a genuinely good instance of what the check's fix guide describes, in its own words: it names the site ("AIScan.site · HTTP 404"), explains in one sentence what AIScan does, and links six recovery routes in the rendered body: the homepage, `/docs`, `/blog`, `sitemap.xml`, `llms.txt` and `openapi.json`. It even carries four `Link:` relations on the 404 response itself, including `rel="api-catalog"` and `rel="describedby"` pointing at `/llms.txt`. On the human half of E1, that's close to the ceiling the check describes.

Ask the same URL for `application/json` and the honest result, also verified on 23 September 2026, is worse: `HTTP/2 500` with `{"error":"Only HTML requests are supported here"}`, `content-type: application/json`. That's not a problem document; it's a generic API error surfacing because the request matched a catch-all route rather than the page-not-found handler at all. AIScan doesn't grade this half of the interaction yet, which is exactly why it's worth disclosing here rather than only in sites we've measured. A scanner that only checks the status line and the HTML body, ours included, will keep missing it.

## Path one: a build-time 404 on a static site

Static generators serve `404.html` (or an equivalent source file) as a fixed artifact from the last build, which makes the status-line half free and the body half a one-time decision.

1. **Create the 404 source file** at the path your framework expects: `static/404.html` on Hugo, `src/pages/404.astro` on Astro, `static/404.html` or a `NotFound` component route on Docusaurus. Check the framework's own docs for the exact path; guessing it is how a 404 template silently never ships.
2. **Confirm the host serves it with status `404`, not `200`.** A static host that can't find a matching file sometimes falls back to serving `index.html` with a `200`, which turns your carefully built error page into a soft 404 for the one class of visitor it exists for. Check with:

```bash
curl -o /dev/null -w '%{http_code}\n' https://yoursite.com/this-cannot-exist-123
# must print: 404
```
3. **Put real links in the body**, not placeholder text: the homepage, the sitemap, and a documentation or search entry point, matching what E1's fix guide names. Six months from now those links should still resolve; check them the same way you'd check any other page's outbound links.
4. **Add a machine-readable twin if your host supports an edge function or middleware route.** Static hosts that can run a function at the edge (Cloudflare Pages Functions, Netlify Edge Functions, a Vercel `middleware.ts`) can branch on the `Accept` header for exactly one route, the 404, and return `application/problem+json` there without touching the rest of the static build.
5. **Re-deploy and re-run step 2.** A 404 template is only as good as the deploy that actually shipped it; verify against the live URL, not the local build output.

## Path two: a server-rendered 404 with content negotiation

An application server, Express, a Next.js route handler, a WordPress theme's `404.php`, has a request object to branch on, which makes true content negotiation straightforward rather than requiring a separate edge function.

1. **Write a dedicated not-found handler** rather than letting a wildcard route silently return `200`. This is the same defect the CI-gate post measured directly on Replit's catch-all: ten of twelve tested apps returned `200 text/html` for a path that had never existed, and six returned a byte-identical body for every unknown path tried. A catch-all that always resolves is the single most common way E1 fails.
2. **Inspect the `Accept` header and branch**, in Express terms:

```js
app.use((req, res) => {
  const wantsJson = req.accepts(['html', 'json']) === 'json';
  res.status(404);
  if (wantsJson) {
    res.type('application/problem+json').json({
      type: 'about:blank',
      title: 'Not Found',
      status: 404,
      detail: `No resource exists at ${req.path}.`,
      instance: req.originalUrl,
    });
  } else {
    res.type('html').send(render404Page(req.path)); // homepage, sitemap, docs, search
  }
});
```

3. **On WordPress**, the equivalent lives in the active theme's `404.php`, which by default only ever renders HTML. A theme can call `wp_is_json_request()` or inspect `$_SERVER['HTTP_ACCEPT']` directly to serve `application/problem+json` from the same template before it falls through to the human page. The recovery-links half of the human page, and whether the 404 template itself carries a stray `noindex` or the wrong canonical, is exactly the kind of per-template robots-meta detail [ThinkRank](https://thinkrank.ai) manages from one settings screen instead of a hand-edited theme file, alongside the robots.txt and schema rules a WordPress AI-readiness setup already needs to get right.
4. **Verify with both `Accept` values against the live route**, not the handler in isolation. A negotiation bug that only shows up once the route is deployed behind a CDN or reverse proxy is common enough that "it worked when I curled localhost" isn't a verification step.

## Verify it actually works

Three commands, each with a concrete pass or fail signal:

```bash
# 1. The status line, for a browser-style request
curl -sI https://yoursite.com/this-page-does-not-exist-123 | head -n 1
# must read: HTTP/2 404 (or HTTP/1.1 404)

# 2. The negotiated body, for a machine-style request
curl -s -H "Accept: application/json" https://yoursite.com/this-page-does-not-exist-123 \
  | head -c 300
# must be a JSON object, ideally content-type: application/problem+json,
# never an HTML page and never a bare {"error":"..."} with no "status" or "title"

# 3. The control path, so a redirect-everything bug can't hide
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://yoursite.com/a-real-page
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://yoursite.com/this-page-does-not-exist-123
# the two lines must differ in status; identical output on both is the soft-404
# pattern the CI-gate post measured on Ghost, Replit and one Squarespace site
```

The third command is the one worth not skipping. A 404 handler that quietly redirects every unmatched path back to the homepage passes a lazy status-only check while still failing the actual reader, because a real page and a nonexistent one return the same thing. Compare the two responses, not just the one you're fixing.

## What AIScan checks, and what it still can't see

E1 is weighted 4 and marked essential in the rubric, and it grades two things directly: the status code is a real `404` (or `410`) rather than a soft `200`, and the HTML body names the site and links somewhere useful. Run `npx aiscan-cli yoursite.com`, or paste the URL at [aiscan.site](https://aiscan.site/), and E1's row in the report will say which half, if either, failed.

What E1 does not grade, today, is everything this post adds on top: whether a non-HTML request gets a non-HTML answer at all, whether that answer is valid `application/problem+json`, and whether `type`, `title` and `status` are populated the way RFC 9457 recommends. A site can score full marks on E1 with a 500-returning catch-all sitting one header away, the same gap `aiscan.site`'s own 404 route has right now. Treat a clean E1 score as proof the human half is done, not proof the machine half exists, until a scanner actually tests the `Accept: application/json` path the way this post's second command does.

For the discovery layer the recovery links point at, the docs page at [`/docs/checks/discoverability`](https://aiscan.site/docs/checks/discoverability) covers E1 alongside the robots.txt and sitemap checks that decide whether an agent finds the real pages in the first place, and [`/guides`](https://aiscan.site/guides) collects the rest of this series, including the CI-gate guide this post builds directly on.