Your llms.txt Is Sending AI Agents to 404s: We Measured It on Our Own Site

Quick summary

A site can publish a perfect llms.txt, ship a clean robots.txt, and still lose more than half its agent traffic to a bracket. We know because it happened to us, on our own blog, and we watched it in our own request logs before we fixed it. Nothing about our file was unusual. It followed the exact format nearly every llms.txt generator, ours included, treats as the default, which is precisely why the bug is worth publishing rather than quietly patching: if it happened on a file we control end to end, it is happening on files we don't.

The bracket that ate our agent traffic

llms.txt is a plain-Markdown index: a heading, a one-line summary, then link lists grouped under ## sections. The convention almost every generator and every guide (ours included) uses for each entry is:

- [Label](https://example.com/path): a short description

That syntax is correct Markdown. It is also a trap for the code that turns it into an HTTP request. A link extractor built around a simple URL regex (https?://\S+ is the common shape) has no way to know that the closing ) belongs to the Markdown link and the : belongs to the sentence, not the URL. So it grabs https://example.com/path): whole, and that string, sent as a request, 404s.

We found this by reading our own numbers, fetched from our own request logs, not by theorizing about someone else's. In the seven days to 26 August 2026, aiscan.site logged 224 requests that hit malformed URLs, roughly 55% of all non-scan traffic to the site in that window, spread across 61 distinct bad paths. The pattern was completely consistent with the theory: /blog), /pricing):, /.well-known/api-catalog):, and dozens more, every one of them our own real path with someone else's punctuation stuck to the end.

We were failing more than half of the agent fetches our own llms.txt had invited, on the exact file whose entire job is to make our content easy for an agent to find. It's the same file we asked, in an earlier piece, whether llms.txt actually works at all. The honest answer is that it can't work if half its links 404 before an agent ever reads the content behind them.

The fix is a redirect, not a parser fix

The instinct is to rewrite llms.txt to avoid the trailing punctuation, drop the colon, move the description to a new line, something. That treats the symptom on our side while leaving the underlying cause untouched: we do not control every tool that will ever read llms.txt, and the - [Label](url): description format is now common enough that rewriting our own file does nothing about the next fetch built the same naive way.

What we shipped instead was forgiving redirects at the origin: strip a trailing ), ):, ). or . from the requested path, check whether what remains is a real route, and if it is, 301 to it. A request for /pricing): becomes a 301 to /pricing, which is a 200. A request for a path that genuinely does not exist still 404s, because no amount of trailing-punctuation stripping turns it into a real page, and that 404 is doing its job.

Reproduce the before-and-after on our site right now:

curl -s -o /dev/null -w '%{http_code} -> %{url_effective}\n' -L "https://aiscan.site/pricing):"
# 200 -> https://aiscan.site/pricing
curl -s -o /dev/null -w '%{http_code} -> %{url_effective}\n' -L "https://aiscan.site/guides)."
# 200 -> https://aiscan.site/guides

Verified on 24 September 2026: both malformed requests land on the real page instead of a 404. The fix cost us a few lines of routing logic. Not shipping it cost us more than half the agent traffic our own discovery file generated.

The same logic in words, for whatever stack you're on, without our specific code: intercept a request before it reaches your normal 404 handler, and check whether it ends in one of a small fixed set of trailing characters, a closing parenthesis, a colon, a period, or some combination of the three. If it does, strip exactly that trailing run and re-check the remaining path against your real routes. A match means the original request was a Markdown-link artifact, not a genuine broken link, so 301 to the clean path. No match means the request was already pointing nowhere, so let your ordinary 404 handler take it, unmodified. The whole rule is two conditionals and a string trim; the hard part was noticing the pattern in the logs, not writing the fix once we had.

How we actually found this: grouping evidence strings, not trusting labels

We did not find the llms.txt bug by suspecting our own code. We found it by reading our own server logs the way we'd tell anyone else to read theirs: pull the requests, group them, and ask what the pattern actually is rather than what you expect it to be. Once the malformed-URL pattern was obvious in our own traffic, we ran the identical instinct against our own scan rubric, because a rubric with 501 hosts in its corpus is exactly the kind of dataset where a check's label (pass, partial, info) can quietly diverge from the evidence it recorded to justify that label.

The method is one SQL shape, reused across every finding below: GROUP BY c->>'id', c->>'status', c->>'evidence' inside a per-host CTE over jsonb_array_elements(result->'checks'). That single query shows every distinct evidence string a check has ever produced, and how many hosts got which status for that exact string. When two different statuses share the identical evidence string, as B3 does below, that's not a coincidence to explain away; it's the check admitting, in its own data, that it can't actually tell those two cases apart. When one status shows up thousands of times behind a handful of trivially short evidence strings, as M1 does, that's the check stopping one signal short of a real answer.

This is the same discipline as the llms.txt fix, applied one level up: don't trust the summary line, read what actually happened underneath it.

This is not a one-off. It is the shape of fourteen other findings

The specific bug is ours. The shape of it, a discovery surface that looks fine from the publisher's side and fails silently on the reader's side, is not. We have now found the same shape fourteen times across our own rubric and once in a competitor's, entirely by grouping evidence strings from our own scan corpus and reading what a check actually asserts rather than what it claims to assert.

Every one of these is a version of the same failure: the check (or the site, or us) reads one signal, stops early, and reports a confident answer that the evidence it already collected contradicts. A discovery surface that fails loudly gets fixed. One that fails silently accumulates for months, exactly the way our llms.txt links did before we looked at our own request log.

Three of these are worth unpacking further, because the numbers understate how confident the wrong answer looks from the outside.

E1 is the one every other check leans on, and more than a quarter of the corpus fails it invisibly. E1 is supposed to confirm that a nonexistent path returns a genuine, unambiguous 404 (the exact property our own site got wrong before the redirect fix). When a site instead serves a soft 200 or 302 for a path that was never real, every check built on top of that assumption (does this URL exist, does this route resolve, is this file actually present) inherits a false "yes." 127 of 473 hosts in our corpus, 26.8%, do not cleanly pass E1. That is not 127 sites with one wrong answer; it's 127 sites where an unknown number of other checks are quietly wrong too, because they trusted a foundation E1 was supposed to guarantee and didn't.

C1's 41.4% is the cleanest case of a check writing down its own disproof and ignoring it. C1 exists to answer one question, does this page serve Markdown to an agent that asks for it, and it records the content type of every response it gets. Group those evidence strings and 67 of its 162 recorded passes carry a content type that is not text/markdown: 57 say text/html outright, 9 say application/octet-stream. The check has the refutation of its own verdict sitting in the same JSON object as the verdict, and nothing reads it before assigning pass. The honest pass rate, once you filter to responses that are actually Markdown, is closer to 19% than the 32.3% the raw label suggests.

B3 goes one step further: two different verdicts, sharing one identical piece of evidence. Group B3's results by evidence string and 41 hosts get partial and 19 get pass for the exact same string, HTTP 200. There is no difference in what the check observed between those two groups, only a difference in what it decided to call it, which means the partial/pass split on this check currently encodes nothing about the sites being scanned.

Laid side by side, the pattern each check assumes and the pattern the evidence strings actually show is the same shape four times over:

None of these are edge cases discovered by fuzzing. They came out of the same query, run four times, against evidence the checks had already collected and simply weren't reading back.

The M1 example anyone can reproduce in one curl

Of the fourteen, M1 is the cheapest to see for yourself, because the bug is that it never looks past the status line.

M1 grades the Universal Commerce Protocol (UCP) signal by fetching /.well-known/ucp and checking the HTTP status. Grouping the evidence strings across our scan corpus shows the entire vocabulary the check has ever recorded: HTTP 404 (142 results, correctly info), HTTP 200 (27 results, pass), one each of a handful of other status codes, and nothing else. There is no assertion anywhere in that evidence that the response body is a valid UCP profile: a JSON document carrying the fields the protocol itself requires, according to the Universal Commerce Protocol's own specification. A 200 with an empty body, an HTML error page served with the wrong status, or a JSON blob missing every required field would all currently pass, because the check stops reading at the status line.

curl -s -o /tmp/ucp.json -w '%{http_code}\n' https://example-site.tld/.well-known/ucp
cat /tmp/ucp.json | python3 -m json.tool   # does this even parse, let alone have real fields?

If the first line prints 200 and the second line throws, you have reproduced the exact gap M1 has.

Nothing about this is unique to UCP. Any check that grades a machine-readable declaration, a manifest, a capabilities file, a well-known JSON document, faces the same temptation to stop at the status code, because parsing the body and checking specific fields is more work than reading one line of a response header. The cheap version of the check ships first, passes review because it technically works on the happy path, and then sits in production for months returning a confident pass on documents nobody has actually parsed. The fix is never exotic: parse the body, name the required fields, and fail (or mark info) when they're missing. It's the same discipline as the redirect, applied to reading a response instead of routing a request.

Cloudflare's own scanner has the same disease

This is not a shot at one vendor from behind a competitor's numbers; it is the same finding, independently, in someone else's tool. Cloudflare's isitagentready.com grades the same UCP signal with its own ucp check, and verified on 8 September 2026, it passed a site on hasVersion alone while reporting the other two structural fields of the same document as absent, on a document that has them. The published error-code table for Cloudflare's Pay Per Crawl feature has also drifted from what the live endpoint now returns. Neither of us built these checks to be wrong; both of us built them to answer a question quickly, and "quickly" is exactly where a discovery check tends to stop one signal short of the truth.

The same drift shows up in Cloudflare's published documentation for Pay Per Crawl, its per-crawler charging feature: the error-code table on the feature's own docs page names a specific set of status codes for specific refusal reasons according to Cloudflare's own documentation, and re-probing the live endpoint against that table shows the mapping has since moved: a code the docs assign to one refusal reason now shows up for another. Documentation drifting behind a live endpoint is a much smaller sin than a check reporting a false pass, but it's the same underlying habit: publish the answer once, and don't re-verify it against what the system is actually doing today. We re-verify our own numbers in this article against the live corpus for exactly that reason.

Every generator that writes - [Label](url): description ships the same exposure

The llms.txt bug is not particular to whatever tool built our file. Any generator, on any platform, that writes a Markdown link list in the standard - [Label](url): description shape hands the same trailing-punctuation trap to whatever reads it. That includes hand-written files, CMS plugins that auto-generate the file from a sitemap, and static-site build steps that template it from front matter. The bug lives in the reading side: the naive regex a crawler, an agent framework or a link-checking tool uses to pull URLs out of Markdown, not in any single generator's output. A file can pass every llms.txt validator that exists today, because those validators check the file's own syntax, and still send a meaningful share of the agents that read it to a 404, because none of them simulate a naive downstream parser. If you're generating your own file rather than hand-writing it, AIScan's llms.txt generator writes the same link-list shape, so the fix still belongs at your edge, not in the generator.

That is the same lesson as E1, C1 and B3 above, aimed outward instead of inward: a check (or a generator, or a validator) that only verifies its own output is structurally correct has not verified that the next thing to touch that output will handle it correctly. The fix on our side was a redirect at the edge, because we don't control every parser that will ever read our file. If you generate llms.txt from a template, the same fix belongs at your edge too, not in the template.

The same logic extends past llms.txt to any Markdown link index a site publishes for agents to read: a sitemap alternative, a docs llms-full.txt, a per-section index. Wherever a human-readable link list gets exposed for machine consumption, the format that is easiest for a person to write is the one most likely to get mis-parsed by whatever reads it downstream, and the fix is never "write it more carefully." It's "make the receiving end forgiving," because you cannot audit every parser that will ever fetch your file, but you can control what your own server does with a request that almost, but not quite, matches a real path.

What AIScan can check, and what it still cannot

AIScan will catch the parts of this family that are structural and repeatable: C2 (llms.txt presence and shape), E1 (does a nonexistent path actually 404), and D3 (is your discovery surface actually reachable end to end). What no automated check, ours or anyone else's, will catch on its own is a real agent hitting a real malformed URL in production traffic, because that requires reading your own request logs after the fact, the way we read ours. Run the scan first; then, if your site serves an llms.txt or any other Markdown link index, go pull seven days of server logs and grep for 404s carrying a trailing ), ): or .. That is the one part of this bug class a scanner cannot see for you. Treat the scan and the log grep as two different questions with two different failure modes: the scan tells you whether an agent arriving today would succeed, and the log grep tells you how many agents that already arrived did not. A site can be structurally perfect from this point forward and still be sitting on months of the first kind of failure, invisible to any external scanner, recorded only in traffic nobody looked at.

Fix your own discovery surface in five checks

1. Scan first. Run npx aiscan-cli yoursite.com, or paste the URL at aiscan.site; no account needed. Read the C2, E1 and D3 rows specifically; those are the ones this family of bugs lives in.
2. Pull your own logs. Seven days is enough. Grep for 404s whose path ends in ), ):, ). or a bare . immediately after what looks like a real route. If you find any, you have the same bug we did.
3. Fix it at the edge, not in the file. A redirect that strips trailing punctuation and re-checks the route protects you against every tool that will ever misparse a Markdown link list, not just the one you happened to test against.
4. Know whether your platform lets you do step 3 at all. A site that produces its own build folder, Hugo, Docusaurus, Astro, a hand-rolled Next.js deploy, can add that redirect rule directly and verify it before the next deploy ships. A site on a hosted builder with no build step of its own, Wix, Squarespace, Webflow, Framer, cannot be gated the same way; on those platforms this is a monitoring problem, not a one-time fix, because you cannot assert the redirect exists before it goes live. Re-scan on a schedule instead of once.
5. Use the tool that matches your stack. If you're on WordPress, ThinkRank generates and maintains your llms.txt, robots.txt, robots meta and schema from one plugin instead of three fighting over the same file, and it migrates settings straight out of Rank Math, Yoast, All in One SEO or SEOPress. If you're on Shopify, StoreSEO generates llms.txt and agents.md straight from your products, collections, pages and articles, which is the version of this bug most relevant to a storefront: an auto-generated file covering hundreds of product links has far more punctuation-adjacent entries than a hand-written marketing-site file ever will. Either way, a hand-maintained llms.txt is exactly the kind of file that accumulates malformed links nobody is watching for.

Once you've fixed anything the scan or your logs turned up, re-scan to confirm, then see the rest of the discoverability checks at AIScan's guides.