Docs Sites and AI Agents: Four Markdown Routes, 52 Sites Tested

Ask an AI assistant how to configure a webhook, add a database index, or set a cache header, and it does not go looking for a blog post. It goes to the vendor's documentation. Docs are the highest-value pages on the agentic web: dense, current, written by the people who built the thing, and requested constantly.

So we went and measured what documentation sites actually hand an agent. We probed 54 live documentation services on 10 September 2026, four different ways each, and the answer is stranger than "most of them are fine" or "most of them are broken".

Two thirds serve clean Markdown. Not one of them serves it by all four of the routes a client might try.

Quick summary

The short version: documentation sites are far ahead of the rest of the web on machine-readable content, and the lead is being spent on four conventions that do not interoperate. A retrieval client picks one. Which one it picks decides whether it reads your docs or your HTML.

Why the docs site is the page an agent reaches for

Every other measurement this blog has published looks at the general web: 499 scanned sites, 110 publisher domains, 30 storefronts. Documentation is a different population with a different job.

It is the population an assistant queries when a developer is stuck, which is the moment where being unreadable costs the most. It is also, unusually, a population whose owners want to be read. Nobody publishes API reference material hoping a machine will fail to parse it.

That shows up immediately in how the sites behave toward an automated request. Of 54 candidate services, 52 returned HTTP 200 to an ordinary desktop Chrome user agent from a datacentre address. That is a 3.7% exclusion rate. On publisher and retail populations the same probe from the same container has been refused 30% to 41% of the time across three independent sweeps. Documentation hosts are the friendliest population we have measured, and it is not close.

Both exclusions were plain 404s at paths that had moved, not refusals. We followed every redirect and deduplicated to distinct services, which is why one candidate collapsed into another: requests to OpenAI's older docs hostname now land on developers.openai.com, so it is counted once.

How we tested 52 documentation sites

For each service we picked one real documentation page, resolved it through its redirects, and then asked for the same content five ways plus two controls.

GET  <page>                          browser user agent, HTML baseline
GET  <page>.md                       the suffix route
GET  <page>/index.md                 the index route
GET  <page>            Accept: text/markdown
GET  <page>?md                       the query route
GET  /<path-that-does-not-exist>     control
GET  /<path-that-does-not-exist>.md  markdown control
GET  /llms.txt  /llms-full.txt  /robots.txt
GET  /.well-known/skills/index.json
GET  /.well-known/agent-skills/index.json

A route counts as working only when the response is HTTP 200, the content type is text/markdown, the body does not begin with an HTML doctype, the body is longer than 200 bytes, and the body differs from what the same host returns for a Markdown URL that does not exist. That last condition is the control path, and on this population it removed six sites that would otherwise have counted.

Then we re-probed every host that produced an interesting result twice more, spaced apart. Across 80 probe slots and three rounds, 78 were identical in status and content type. The two that were not were transient connection failures, not changed answers. That matters, because a single pass cannot distinguish a policy from a rate limit.

The first pass of this sweep had a defect worth naming: it did not follow redirects on the Markdown probes, which recorded two sites as having no Accept support when they simply answered with a 307. The re-probe round caught it and the whole population was re-run consistently. Every figure here comes from that consistent pass, verified on 10 September 2026.

Four routes to the same Markdown, and no site serves all four

The distribution of how many routes each site answers is the finding in one line: 18 sites answer none, 8 answer exactly one, 18 answer two, 8 answer three, and nobody answers four.

The ?md result deserves its zero stated plainly. That convention is real and works today on Framer-hosted sites, which is where this programme first recorded it. It appears nowhere in a documentation population. A checker that probes it will always come back empty here, and a zero from a convention nobody in the sample uses is a fact about the sample, not about the convention.

The three live routes are not variations on one idea. They are three different things pretending to be one.

Accept: text/markdown is proactive content negotiation, the mechanism HTTP has had all along. RFC 9110 defines it, in its own words, as the pattern "where the server selects the representation based upon the user agent's stated preferences". One URL, two representations, chosen by the server.

page.md is not negotiation at all. It is a second URL holding a second resource. Nothing in HTTP connects it to the first; the relationship exists only because a convention says it does.

llms.txt is closest to what RFC 9110 calls reactive negotiation, "where the server provides a list of representations for the user agent to choose from". The agent reads an index and then goes and fetches things.

The standard is unusually direct about how well the first of those works in practice. Section 12.1 says, verbatim:

"A user agent cannot rely on proactive negotiation preferences being consistently honored, since the origin server might not implement proactive negotiation for the requested resource or might decide that sending a response that doesn't conform to the user agent's preferences is better than sending a 406 (Not Acceptable) response."

That sentence was written before any of this. It describes our 2026 measurement exactly: 21 of the 52 sites received Accept: text/markdown and answered HTTP 200 with a content type that was not Markdown, 20 of them plain text/html. No 406, no Vary, no signal at all that a preference had been considered and declined.

What a single-convention client actually finds

This is the part that decides whether your docs get read.

Twenty-four sites answer both .md and the Accept header, so most of the time either convention works. The interesting sites are the ones where it does not.

Three sites answer .md and ignore the header: Bun, Deno, and Vite. Five answer the header and 404 the .md URL: Cloudflare's developer docs, Convex, Hono, Redis, and the WordPress developer reference. Two answer neither and only respond to /index.md: GitLab's docs and LlamaIndex.

None of those eight sites is broken. Each made a defensible choice, and each is invisible to a client that made a different one. There is no negotiation happening between the two decisions, because the whole point of the .md convention is that it needs no negotiation.

The platform decides, not the writer

Nobody writing a docs page chooses any of this. The docs platform does, and the split is clean enough to predict.

The generated platforms are close to unanimous and the hand-rolled ones are a coin flip. Starlight and GitBook were 100% in this sample at small n, so treat those as directional rather than settled. The pattern that holds is the one at the bottom: the single largest group of failures is sites that built their own docs stack and never added a Markdown route.

If you are on a platform in the top half of that table, the honest answer to "how do I serve Markdown to agents" is that you probably already do, and your real task is to find out which route yours picked. If you are in the bottom half, this is a routing change, not a content project.

There is a mirror of this on the publishing side. Plenty of documentation lives in WordPress rather than a docs generator, and there the machine-readable signals end up scattered across separate plugins that each want to own the same file. ThinkRank is what we point WordPress readers at, for a structural reason rather than a promotional one: one plugin holds robots.txt, robots meta, schema, sitemaps and llms.txt together, so nothing is fighting for control of them, and it reads in whatever Rank Math, Yoast, All in One SEO or SEOPress already had configured, which means adopting it does not cost you an afternoon of retyping settings. Where those plugins genuinely win is editorial: content analysis, readability scoring, and the writing workflow around a post. If that is what you need most, keep them. The gap none of them closes is single ownership of the agent-facing files, which is the subject of this article. Worth noting that the WordPress developer reference sits in our own sample, as one of the five sites an agent can reach only through the Accept header.

Shopify merchants who publish help centres or product documentation hit the same wall. StoreSEO builds llms.txt out of products, collections, pages and articles, gives you an agents.md editor, and handles the schema and answer-engine work that Shopify's own agent rollout leaves alone. Its listing showed a 5.0 rating across 742 reviews on the Shopify App Store when we read it on 10 September 2026, with the Built for Shopify badge attached. Rival Shopify SEO apps do a better job on metafields and redirect management. What none of them currently do is write the files an agent goes looking for.

The index without the content, and the content without the index

Documentation sites publish llms.txt at a rate nothing else on the web approaches. 37 of 52, or 71%. For comparison: according to Rankability's monthly tracker, run against a named Tranco list, adoption was 5.6% of the top 10,000 on June 2026 data; Ahrefs measured 28% across roughly 137,000 domains in June 2026 and warned in its own write-up that its sample "skews more technical and SEO-aware than the web at large, so treat the 28% adoption figure as an upper bound". Documentation sites clear even that upper bound by a wide margin. 22 of 52 (42%) also publish llms-full.txt.

But an index and a readable body are separate achievements, and six sites have exactly one of them.

Six publish an llms.txt whose links point only at HTML: Svelte, Vue, React, Docker's docs, Clerk, and NestJS. The agent gets a clean map of the territory and then has to parse a JavaScript-heavy page at every destination. Docker's is the sharpest version of the joke: it does answer Accept: text/markdown, with 16 bytes of Markdown, which is a content type and not a document. Three serve Markdown and publish no index at all: the WordPress developer reference, Shopify's developer docs, and Laravel. The content is right there and nothing advertises it.

The other end of the scale is its own problem. Cloudflare's llms-full.txt is 57.1 MB. Anthropic's is 42.4 MB. Twilio's llms.txt is 2.33 MB across 9,628 lines and carries 2 top-level headings, which is a correction to an earlier note in our own files that recorded zero. A file that size is not a summary an agent reads before deciding what to fetch; it is a full corpus dump wearing an index's filename, and re-fetching it costs the same bytes every time because documentation hosts are among the worst on conditional requests.

Eight sites negotiate on Accept and never say so

Of the 29 sites that change their response based on the Accept header, 22 declare it with a Vary header listing Accept. Eight do not.

developers.openai.com   (no Vary header)
docs.sentry.io          (no Vary header)
redis.io                (no Vary header)
modal.com               (no Vary header)
docs.datadoghq.com      vary: origin
www.prisma.io           vary: rsc, next-router-state-tree, next-router-prefetch
www.twilio.com          vary: accept-encoding

RFC 9110 section 12.5.5, fetched from the RFC Editor on 10 September 2026, explains what Vary is for, and the first of its two stated purposes is the one that bites here: it exists to tell caches they "MUST NOT use this response to satisfy a later request unless the later request has the same values for the listed header fields".

Without it, every shared cache between the origin and the client is entitled to treat one representation as the answer for both. A browser can be handed Markdown. An agent can be handed HTML. Neither party did anything wrong, and the bug is invisible from the origin, because a direct fetch always works.

This is a genuine defect and a cheap fix: one header on the negotiated response. It is also the single most reproducible finding in the sweep, since the absence of a header does not vary with load.

Six sites answer 200 for a Markdown page that does not exist

We asked every host for /aiscan-control-9f3d2b.md, a path that has never existed anywhere. Forty-five returned a proper 404. Six returned HTTP 200.

The first four are the familiar shape: an application shell answering everything with a 200, which this blog has documented on storefronts and deployed apps before. Vercel's is the largest instance we have recorded anywhere, at 2.5 MB.

The last two are new and they deserve a fair reading. Sentry and n8n return a real text/markdown document that says, in its own words, "Page Not Found" and then lists where to go instead. Sentry's even links its own llms.txt as a site index. As machine-readable error handling that is thoughtful work, better than most 404 pages on the web. The only thing wrong with it is the status line.

But it defeats every check in the field, ours included. A checker asserting HTTP 200 plus content-type: text/markdown counts both of those sites as serving Markdown at any URL you name. The two-request control path is what separates them, and it costs one extra fetch: ask for something that cannot exist, and compare.

Agent Skills is already split across two paths

While we were probing well-known paths, the same fragmentation showed up on a much younger convention.

Stripe's documentation publishes 8 named skills, verified on 10 September 2026, and publishes them only at the older /.well-known/skills/ path. Six sites, including Vercel, Supabase, Prisma, Resend, GitBook and Pydantic, publish only at the newer one. Four publish both.

A scanner that probes one path finds either 10 or 5 of the 11. AIScan probes the newer path, so it reports Stripe as having no skills index while Stripe serves 6,422 bytes of valid JSON describing eight of them. That is a known bug in our own product and it now has a measured population behind it rather than a single example.

Where AIScan fits, and where it doesn't

Everything above is reproducible with curl, and nothing in this article requires our scanner. But four of the things we measured map onto checks we already run, so here is what a scan tells you and, more usefully, what it gets wrong.

npx aiscan-cli yoursite.com

Free, no account, no signup. Or paste the URL at aiscan.site. The rows that matter for a documentation site:

• C1, Markdown content negotiation. The route question this whole article is about.
• C2, llms.txt. Whether an index exists and parses.
• E1, hard 404 behaviour. The control-path problem, and the check everything else depends on.
• P3, Agent Skills index. The two-path problem above.

Now the honest half, because we graded our own check against this population and it did not come out clean.

C1 probes two routes and never probes the third. Grouping the evidence strings of all 501 sites in our scan corpus shows exactly two probe shapes: Accept: text/markdown on 367 sites and /index.md on 131. The .md suffix appears in zero evidence strings. On this population those two routes would find 32 of the 34 Markdown-serving sites and miss Bun and Vite entirely. That is a modest miss rate here, and it will get worse, because the .md suffix is the route the fastest-growing docs platforms picked.

And 41% of C1's passes are on responses that are not Markdown. Across the same 501 sites, C1 records 162 passes. Sixty-seven of them, 41.4%, carry an evidence string whose content type is not text/markdown, and 57 of those say text/html outright. The check writes down its own disproof and passes anyway. Requiring the content type it already recorded would put the real C1 pass rate at 95 of 501, or 19.0%, against the 32.3% currently reported.

That 19.0% is worth sitting with, because an earlier study reached an adjusted C1 floor of about 19.2% by a completely different method, discounting passes on hosts that fail the soft-404 check. Two independent routes to the same number is the strongest evidence we have that the ceiling on C1 is real and the headline figure is inflated.

Both are open bugs on our own backlog, both are one line of code each, and both are published here rather than quietly filed. What AIScan cannot see at all: whether your server sends Vary: Accept, whether your llms.txt links point at Markdown or HTML, and what your site returns to a crawler rather than to us. Cloudflare's free scanner grades markdownNegotiation too and has the same header-only blind spot, so a second opinion does not close it either.

Fix your own docs site in four checks

Four commands, in order. Each one answers a question the next one depends on.

1. Does your site hard-404? Everything below is meaningless until this passes.

curl -s -o /dev/null -w '%{http_code} %{size_download}\n' https://yoursite.com/docs/no-such-page-xyz.md
curl -s -o /dev/null -w '%{http_code} %{size_download}\n' https://yoursite.com/docs/real-page.md

Two 200s with similar byte counts mean your .md route is a phantom and every other test lies.

2. Which route do you actually answer? Run all three and read the content type, not the status.

U=https://yoursite.com/docs/real-page
curl -sI "$U.md"                        | grep -i '^content-type'
curl -sI "$U/index.md"                  | grep -i '^content-type'
curl -sI -H 'Accept: text/markdown' "$U" | grep -i -E '^(content-type|vary)'

You want text/markdown on at least one, and if the third one is the winner you also want Vary to mention Accept. Serving two routes instead of one is a rewrite rule, not a rebuild, and it takes you from 85% client coverage to 94%.

3. Does your index point at Markdown? An llms.txt full of HTML links is a map to pages the agent still cannot read.

curl -s https://yoursite.com/llms.txt | grep -c '\.md)'
curl -s https://yoursite.com/llms.txt | wc -c

If the first number is near zero, fix the link targets before anything else. If the second is over a megabyte, you have published a corpus, not an index. Our llms.txt generator writes a spec-shaped file if you need a starting point, and the twenty-one validator checks list what usually goes wrong.

4. Then score the whole thing.

npx aiscan-cli yoursite.com

Read C1, C2, E1 and P3 first, and read C1 knowing what it does and does not probe. The full rubric is documented at /docs/checks/content for C1 and C2, /docs/checks/discoverability for E1, and /docs/checks/capabilities for P3. Every guide we publish is indexed at /guides.

One closing thought about where this goes. The four routes exist because four platforms solved the same problem in the same eighteen months without a standard to point at, and RFC 9110 section 12 has a line that reads like it was written for exactly this moment: "HTTP is not aware of the resource semantics", and the sameness of a resource's representations "is determined entirely by whatever entity or algorithm selects or generates those responses". Nothing is going to arbitrate this for you. Until something does, serving two routes instead of one is the cheapest reliability work available to a documentation site, and it is a config change.