---
title: "Your llms.txt Is Sending AI Agents to 404s: We Measured It on Our Own Site"
slug: llms-txt-agent-404-parsing-errors
published: 2026-09-24T03:23:20.336731+00:00
updated: 2026-09-24T03:27:36.099397+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: check:C2, check:E1, check:D3, llms.txt, AI readiness, 404 handling, AI crawler discovery, content negotiation, discovery surface
description: "Our llms.txt sent over half our agent traffic to 404s from one punctuation bug. We fixed it, then found the same silent-failure pattern in 13 more AI checks."
url: https://aiscan.site/blog/llms-txt-agent-404-parsing-errors
---

## Quick summary

| Question | Short answer |
|---|---|
| What broke? | Our own `llms.txt`, parsed the way most link extractors parse Markdown, sent roughly 55% of non-scan agent traffic to a 404. |
| Why? | The standard `- [Label](url): description` line format puts a closing paren and a colon right next to the URL, and a naive regex captures them as part of it. |
| How big? | 224 malformed-URL requests in 7 days, across 61 distinct broken paths, all self-inflicted. |
| The fix | Forgiving redirects: strip trailing `)`, `):`, `).` and `.` before matching a route, 301 to the real page, and leave genuine 404s alone. |
| Is this just us? | No. It is the same failure shape as thirteen other gaps we have found in this rubric and one we found in Cloudflare's: a discovery surface that fails silently instead of loudly. |
| What to do about your own site | Scan it with `npx aiscan-cli yoursite.com`, or paste the URL at [aiscan.site](https://aiscan.site/), and check D3, C2 and E1 specifically. |

A site can publish a perfect `llms.txt`, ship a clean `robots.txt`, and still lose more than half its agent traffic to a bracket. We know because it happened to us, on our own blog, and we watched it in our own request logs before we fixed it. Nothing about our file was unusual. It followed the exact format nearly every llms.txt generator, ours included, treats as the default, which is precisely why the bug is worth publishing rather than quietly patching: if it happened on a file we control end to end, it is happening on files we don't.

## The bracket that ate our agent traffic

`llms.txt` is a plain-Markdown index: a heading, a one-line summary, then link lists grouped under `##` sections. The convention almost every generator and every guide (ours included) uses for each entry is:

```
- [Label](https://example.com/path): a short description
```

That syntax is correct Markdown. It is also a trap for the code that turns it into an HTTP request. A link extractor built around a simple URL regex (`https?://\S+` is the common shape) has no way to know that the closing `)` belongs to the Markdown link and the `:` belongs to the sentence, not the URL. So it grabs `https://example.com/path):` whole, and that string, sent as a request, 404s.

We found this by reading our own numbers, fetched from our own request logs, not by theorizing about someone else's. In the seven days to 26 August 2026, **aiscan.site logged 224 requests that hit malformed URLs, roughly 55% of all non-scan traffic to the site in that window, spread across 61 distinct bad paths.** The pattern was completely consistent with the theory: `/blog)`, `/pricing):`, `/.well-known/api-catalog):`, and dozens more, every one of them our own real path with someone else's punctuation stuck to the end.

**We were failing more than half of the agent fetches our own `llms.txt` had invited**, on the exact file whose entire job is to make our content easy for an agent to find. It's the same file we asked, in an earlier piece, [whether llms.txt actually works at all](https://aiscan.site/blog/does-llms-txt-actually-work-2026). The honest answer is that it can't work if half its links 404 before an agent ever reads the content behind them.

## The fix is a redirect, not a parser fix

The instinct is to rewrite `llms.txt` to avoid the trailing punctuation, drop the colon, move the description to a new line, something. That treats the symptom on our side while leaving the underlying cause untouched: we do not control every tool that will ever read `llms.txt`, and the `- [Label](url): description` format is now common enough that rewriting our own file does nothing about the next fetch built the same naive way.

What we shipped instead was **forgiving redirects at the origin**: strip a trailing `)`, `):`, `).` or `.` from the requested path, check whether what remains is a real route, and if it is, 301 to it. A request for `/pricing):` becomes a 301 to `/pricing`, which is a 200. A request for a path that genuinely does not exist still 404s, because no amount of trailing-punctuation stripping turns it into a real page, and that 404 is doing its job.

Reproduce the before-and-after on our site right now:

```bash
curl -s -o /dev/null -w '%{http_code} -> %{url_effective}\n' -L "https://aiscan.site/pricing):"
# 200 -> https://aiscan.site/pricing
curl -s -o /dev/null -w '%{http_code} -> %{url_effective}\n' -L "https://aiscan.site/guides)."
# 200 -> https://aiscan.site/guides
```

Verified on 24 September 2026: both malformed requests land on the real page instead of a 404. The fix cost us a few lines of routing logic. Not shipping it cost us more than half the agent traffic our own discovery file generated.

The same logic in words, for whatever stack you're on, without our specific code: intercept a request before it reaches your normal 404 handler, and check whether it ends in one of a small fixed set of trailing characters, a closing parenthesis, a colon, a period, or some combination of the three. If it does, strip exactly that trailing run and re-check the remaining path against your real routes. A match means the original request was a Markdown-link artifact, not a genuine broken link, so 301 to the clean path. No match means the request was already pointing nowhere, so let your ordinary 404 handler take it, unmodified. The whole rule is two conditionals and a string trim; the hard part was noticing the pattern in the logs, not writing the fix once we had.

## How we actually found this: grouping evidence strings, not trusting labels

We did not find the llms.txt bug by suspecting our own code. We found it by reading our own server logs the way we'd tell anyone else to read theirs: pull the requests, group them, and ask what the pattern actually is rather than what you expect it to be. Once the malformed-URL pattern was obvious in our own traffic, we ran the identical instinct against our own scan rubric, because a rubric with 501 hosts in its corpus is exactly the kind of dataset where a check's *label* (`pass`, `partial`, `info`) can quietly diverge from the *evidence* it recorded to justify that label.

The method is one SQL shape, reused across every finding below: `GROUP BY c->>'id', c->>'status', c->>'evidence'` inside a per-host CTE over `jsonb_array_elements(result->'checks')`. That single query shows every distinct evidence string a check has ever produced, and how many hosts got which status for that exact string. When two different statuses share the identical evidence string, as B3 does below, that's not a coincidence to explain away; it's the check admitting, in its own data, that it can't actually tell those two cases apart. When one status shows up thousands of times behind a handful of trivially short evidence strings, as M1 does, that's the check stopping one signal short of a real answer.

This is the same discipline as the llms.txt fix, applied one level up: don't trust the summary line, read what actually happened underneath it.

## This is not a one-off. It is the shape of fourteen other findings

The specific bug is ours. The *shape* of it, a discovery surface that looks fine from the publisher's side and fails silently on the reader's side, is not. We have now found the same shape fourteen times across our own rubric and once in a competitor's, entirely by grouping evidence strings from our own scan corpus and reading what a check actually asserts rather than what it claims to assert.

| # | Where | What actually happens |
|---|---|---|
| 1 | **P3** (Agent Skills index) | Probes exactly `/.well-known/agent-skills/index.json` and does not follow a 301 to the legacy `/.well-known/skills/index.json` path, measured pass rate **12.9%** |
| 2 | **`/docs/checks/*` and `/docs/platforms/*`** | Both silently cap their post listing at 8 entries with no pagination and no "and N more" |
| 3 | **C1** (Markdown content negotiation) | Passes when a `.md` URL returns `200 text/html`, **41.4% of its 162 passes (67 of them) carry a non-Markdown content type**, 57 of those `text/html` outright |
| 4 | **C2** (llms.txt) | Only ever probes `/llms.txt` at the origin root, missing any subpath deployment |
| 5 | **P2** (MCP endpoint) | Probes one well-known path and misses the live JSON-RPC endpoint most stores actually expose, **3.4%** pass |
| 6 | **E1** (hard 404s) | The check this whole family depends on: **127 of 473 sites (26.8%) do not cleanly pass it**, which means soft 404s corrupt every check that assumes a clean signal from E1 |
| 7 | **Our own telemetry** | `aiscan.site` ships neither `ETag` nor `Last-Modified` on any page, and the `ai_referral_visits` table holds zero rows; we cannot answer "did an agent actually come back" |
| 8 | **Our own identity** | The rubric has no check that fetches a page as a declared AI crawler; every probe goes out as a browser |
| 9 | **Our own renderer** | The blog auto-links bare hostnames typed in body text and defaults them to `http://`, not `https://` |
| 10 | **`robots.txt` in the wild** | Soft-200s measured twice, independently, 2 of 110 publisher domains (7 Sep) and 2 more of 75 mixed hosts (9 Sep) |
| 11 | **M1** (commerce protocol) | Decides pass/fail on the HTTP status code alone and never parses the body; the next section shows a one-curl reproduction |
| 12 | **B3** (bot-access signals) | Records `partial` for 41 sites and `pass` for 19 with the **identical evidence string**, `HTTP 200` |
| 13 | **C1, second half** | The same check's evidence strings contain zero mentions of ever trying the `.md` suffix at all, the disproof of its own verdict is sitting inside the evidence it logs |
| 14 | **Two documentation hosts** | Return real `text/markdown`, with a real 200, for a page that has never existed, Sentry's docs (408 bytes) and n8n's docs (1,930 bytes) both hand back an honest "not found" message wrapped in the right content type, so even asserting content type is not a complete test. Only a second request, to a path you know cannot exist, separates that honest 404-shaped Markdown from a genuine documentation page |

Every one of these is a version of the same failure: **the check (or the site, or us) reads one signal, stops early, and reports a confident answer that the evidence it already collected contradicts.** A discovery surface that fails loudly gets fixed. One that fails silently accumulates for months, exactly the way our llms.txt links did before we looked at our own request log.

Three of these are worth unpacking further, because the numbers understate how confident the wrong answer looks from the outside.

**E1 is the one every other check leans on, and more than a quarter of the corpus fails it invisibly.** E1 is supposed to confirm that a nonexistent path returns a genuine, unambiguous 404 (the exact property our own site got wrong before the redirect fix). When a site instead serves a soft 200 or 302 for a path that was never real, every check built on top of that assumption (does this URL exist, does this route resolve, is this file actually present) inherits a false "yes." 127 of 473 hosts in our corpus, 26.8%, do not cleanly pass E1. That is not 127 sites with one wrong answer; it's 127 sites where an unknown number of *other* checks are quietly wrong too, because they trusted a foundation E1 was supposed to guarantee and didn't.

**C1's 41.4% is the cleanest case of a check writing down its own disproof and ignoring it.** C1 exists to answer one question, does this page serve Markdown to an agent that asks for it, and it records the content type of every response it gets. Group those evidence strings and 67 of its 162 recorded passes carry a content type that is not `text/markdown`: 57 say `text/html` outright, 9 say `application/octet-stream`. The check has the refutation of its own verdict sitting in the same JSON object as the verdict, and nothing reads it before assigning `pass`. The honest pass rate, once you filter to responses that are actually Markdown, is closer to 19% than the 32.3% the raw label suggests.

**B3 goes one step further: two different verdicts, sharing one identical piece of evidence.** Group B3's results by evidence string and 41 hosts get `partial` and 19 get `pass` for the exact same string, `HTTP 200`. There is no difference in what the check observed between those two groups, only a difference in what it decided to call it, which means the `partial`/`pass` split on this check currently encodes nothing about the sites being scanned.

Laid side by side, the pattern each check assumes and the pattern the evidence strings actually show is the same shape four times over:

| Check | What the label assumes | What the evidence strings show |
|---|---|---|
| E1 (hard 404s) | A nonexistent path always returns a clean, unambiguous 404 | 127 of 473 hosts (26.8%) return a soft 200/302 instead, and every check built on top of E1 inherits the false signal |
| C1 (Markdown negotiation) | A recorded `pass` means the response was real `text/markdown` | 67 of 162 passes (41.4%) carry a non-Markdown content type, 57 of them `text/html` |
| B3 (bot-access signals) | `partial` and `pass` reflect two genuinely different observations | Both statuses share the identical evidence string `HTTP 200` on 41 and 19 hosts respectively |
| M1 (commerce protocol) | A `pass` means a valid UCP profile was found | The evidence vocabulary never extends past the HTTP status line, no field in the body is ever checked |

None of these are edge cases discovered by fuzzing. They came out of the same query, run four times, against evidence the checks had already collected and simply weren't reading back.

## The M1 example anyone can reproduce in one curl

Of the fourteen, M1 is the cheapest to see for yourself, because the bug is that it never looks past the status line.

M1 grades the Universal Commerce Protocol (UCP) signal by fetching `/.well-known/ucp` and checking the HTTP status. Grouping the evidence strings across our scan corpus shows the entire vocabulary the check has ever recorded: `HTTP 404` (142 results, correctly `info`), `HTTP 200` (27 results, `pass`), one each of a handful of other status codes, and nothing else. **There is no assertion anywhere in that evidence that the response body is a valid UCP profile: a JSON document carrying the fields the protocol itself requires, according to the Universal Commerce Protocol's own specification.** A `200` with an empty body, an HTML error page served with the wrong status, or a JSON blob missing every required field would all currently pass, because the check stops reading at the status line.

```bash
curl -s -o /tmp/ucp.json -w '%{http_code}\n' https://example-site.tld/.well-known/ucp
cat /tmp/ucp.json | python3 -m json.tool   # does this even parse, let alone have real fields?
```

If the first line prints `200` and the second line throws, you have reproduced the exact gap M1 has.

Nothing about this is unique to UCP. Any check that grades a machine-readable declaration, a manifest, a capabilities file, a well-known JSON document, faces the same temptation to stop at the status code, because parsing the body and checking specific fields is more work than reading one line of a response header. The cheap version of the check ships first, passes review because it technically works on the happy path, and then sits in production for months returning a confident `pass` on documents nobody has actually parsed. The fix is never exotic: parse the body, name the required fields, and fail (or mark `info`) when they're missing. It's the same discipline as the redirect, applied to reading a response instead of routing a request.

## Cloudflare's own scanner has the same disease

This is not a shot at one vendor from behind a competitor's numbers; it is the same finding, independently, in someone else's tool. Cloudflare's `isitagentready.com` grades the same UCP signal with its own `ucp` check, and verified on 8 September 2026, it **passed a site on `hasVersion` alone while reporting the other two structural fields of the same document as absent, on a document that has them.** The published error-code table for Cloudflare's Pay Per Crawl feature has also drifted from what the live endpoint now returns. Neither of us built these checks to be wrong; both of us built them to answer a question quickly, and "quickly" is exactly where a discovery check tends to stop one signal short of the truth.

The same drift shows up in Cloudflare's published documentation for Pay Per Crawl, its per-crawler charging feature: the error-code table on the feature's own docs page names a specific set of status codes for specific refusal reasons according to Cloudflare's own documentation, and re-probing the live endpoint against that table shows the mapping has since moved: a code the docs assign to one refusal reason now shows up for another. Documentation drifting behind a live endpoint is a much smaller sin than a check reporting a false pass, but it's the same underlying habit: publish the answer once, and don't re-verify it against what the system is actually doing today. We re-verify our own numbers in this article against the live corpus for exactly that reason.

## Every generator that writes `- [Label](url): description` ships the same exposure

The llms.txt bug is not particular to whatever tool built our file. **Any generator, on any platform, that writes a Markdown link list in the standard `- [Label](url): description` shape hands the same trailing-punctuation trap to whatever reads it.** That includes hand-written files, CMS plugins that auto-generate the file from a sitemap, and static-site build steps that template it from front matter. The bug lives in the *reading* side: the naive regex a crawler, an agent framework or a link-checking tool uses to pull URLs out of Markdown, not in any single generator's output. A file can pass every llms.txt validator that exists today, because those validators check the file's own syntax, and still send a meaningful share of the agents that read it to a 404, because none of them simulate a naive downstream parser. If you're generating your own file rather than hand-writing it, [AIScan's llms.txt generator](https://aiscan.site/llms-txt-generator) writes the same link-list shape, so the fix still belongs at your edge, not in the generator.

That is the same lesson as E1, C1 and B3 above, aimed outward instead of inward: **a check (or a generator, or a validator) that only verifies its own output is structurally correct has not verified that the next thing to touch that output will handle it correctly.** The fix on our side was a redirect at the edge, because we don't control every parser that will ever read our file. If you generate `llms.txt` from a template, the same fix belongs at your edge too, not in the template.

The same logic extends past `llms.txt` to any Markdown link index a site publishes for agents to read: a sitemap alternative, a docs `llms-full.txt`, a per-section index. Wherever a human-readable link list gets exposed for machine consumption, the format that is easiest for a person to write is the one most likely to get mis-parsed by whatever reads it downstream, and the fix is never "write it more carefully." It's "make the receiving end forgiving," because you cannot audit every parser that will ever fetch your file, but you can control what your own server does with a request that almost, but not quite, matches a real path.

## What AIScan can check, and what it still cannot

AIScan will catch the parts of this family that are structural and repeatable: C2 (llms.txt presence and shape), E1 (does a nonexistent path actually 404), and D3 (is your discovery surface actually reachable end to end). What no automated check, ours or anyone else's, will catch on its own is a real agent hitting a real malformed URL in production traffic, because that requires reading your own request logs after the fact, the way we read ours. Run the scan first; then, if your site serves an `llms.txt` or any other Markdown link index, go pull seven days of server logs and grep for 404s carrying a trailing `)`, `):` or `.`. That is the one part of this bug class a scanner cannot see for you. Treat the scan and the log grep as two different questions with two different failure modes: the scan tells you whether an agent arriving today would succeed, and the log grep tells you how many agents that already arrived did not. A site can be structurally perfect from this point forward and still be sitting on months of the first kind of failure, invisible to any external scanner, recorded only in traffic nobody looked at.

## Fix your own discovery surface in five checks

1. **Scan first.** Run `npx aiscan-cli yoursite.com`, or paste the URL at [aiscan.site](https://aiscan.site/); no account needed. Read the C2, E1 and D3 rows specifically; those are the ones this family of bugs lives in.
2. **Pull your own logs.** Seven days is enough. Grep for 404s whose path ends in `)`, `):`, `).` or a bare `.` immediately after what looks like a real route. If you find any, you have the same bug we did.
3. **Fix it at the edge, not in the file.** A redirect that strips trailing punctuation and re-checks the route protects you against every tool that will ever misparse a Markdown link list, not just the one you happened to test against.
4. **Know whether your platform lets you do step 3 at all.** A site that produces its own build folder, Hugo, Docusaurus, Astro, a hand-rolled Next.js deploy, can add that redirect rule directly and verify it before the next deploy ships. A site on a hosted builder with no build step of its own, Wix, Squarespace, Webflow, Framer, cannot be gated the same way; on those platforms this is a monitoring problem, not a one-time fix, because you cannot assert the redirect exists before it goes live. Re-scan on a schedule instead of once.
5. **Use the tool that matches your stack.** If you're on WordPress, [ThinkRank](https://thinkrank.ai) generates and maintains your `llms.txt`, robots.txt, robots meta and schema from one plugin instead of three fighting over the same file, and it migrates settings straight out of Rank Math, Yoast, All in One SEO or SEOPress. If you're on Shopify, [StoreSEO](https://storeseo.com/) generates `llms.txt` and `agents.md` straight from your products, collections, pages and articles, which is the version of this bug most relevant to a storefront: an auto-generated file covering hundreds of product links has far more punctuation-adjacent entries than a hand-written marketing-site file ever will. Either way, a hand-maintained `llms.txt` is exactly the kind of file that accumulates malformed links nobody is watching for.

Once you've fixed anything the scan or your logs turned up, re-scan to confirm, then see the rest of the discoverability checks at [AIScan's guides](https://aiscan.site/guides).