---
title: "Markdown for AI Agents Beyond Docs: Only 19.3% of 701 Sites Serve It"
slug: markdown-twin-beyond-docs
published: 2026-10-08T03:16:32.559048+00:00
updated: 2026-10-08T03:16:32.559048+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: check:C1, check:C2, platform:astro, platform:nextjs, platform:wordpress, platform:shopify, corpus study, markdown, content negotiation, text/markdown
description: "701 sites measured on check C1: 135 (19.3%) return real Markdown, 83 pass on an HTML page. Counts by route and platform, plus the fix for each stack."
url: https://aiscan.site/blog/markdown-twin-beyond-docs
---

On 8 October 2026 we queried the AIScan database for check C1, the Markdown availability check, across **701 distinct hosts**. 135 of them (19.3%) returned a real `text/markdown` response on one of the two routes the check tries. A further 83 (11.8%) also pass C1, but the response that earned the pass is an ordinary HTML page or a binary download, not Markdown. The remaining 477 (68.0%) fail. If you have read that agents prefer Markdown and assumed most sites already offer it, the number to remember is one in five.

Our earlier study, [Docs Sites and AI Agents: Four Markdown Routes, 52 Sites Tested](https://aiscan.site/blog/documentation-sites-ai-agent-readiness), asked a friendly question: do documentation sites, the group most motivated to serve Markdown, actually do it? Of 52 services, 34 served it in some form. This post asks the counter-question. The sites in the AIScan corpus are marketing sites, blogs, stores and apps that people scanned because they wanted a grade. They are not a docs list. Their Markdown rate is the realistic baseline, and the documentation figure is the ceiling.

We state the limits of the measurement before the tables, then give the counts by route and platform, explain why 83 passes deserve a second look, and finish with the fix for each stack and a way to verify it with [AIScan](https://aiscan.site/) (`check:C1`) or by hand.

## Quick summary: Markdown availability across 701 sites

| Question | Answer from 701 hosts | What to do |
|---|---|---|
| How many sites return real Markdown? | 135 pass with a Markdown content type (19.3%) | Treat one in five as the norm, not the majority |
| How many pass C1 at all? | 218 (31.1%) | Read the content type in your evidence line, not only the verdict |
| How many pass without Markdown? | 83 (11.8% of all hosts, 38.1% of passes) | Confirm `/index.md` returns Markdown, not your homepage |
| How many honour `Accept: text/markdown`? | 50 clean passes (7.1%) | Add negotiation at the edge if your stack allows it |
| How many fail? | 477 (68.0%), of which 381 answered 200 with HTML | Serve a Markdown twin on one route first |
| Where does the probe get blocked? | 78 fails are a 403, 429 or 5xx (11.1% of hosts) | Treat the fail rate as an upper bound |
| Best platform among those with 30+ hosts | Astro, 19 of 34 (55.9%) return real Markdown | Static-site generators make a Markdown twin cheap |

**Expert recommendation:** [Cloudflare's Markdown for Agents documentation](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) (last updated 13 July 2026) describes the mechanism C1 tests. An AI system sends `Accept: text/markdown`, and an enabled zone converts the HTML to Markdown at the edge. [Cloudflare's own Agent Readiness analysis](https://blog.cloudflare.com/agent-readiness/) of the 200,000 most visited domains reported that content negotiation of this kind passed on 3.9% of sites.

**Best for a one-site owner:** publish a Markdown copy of your key pages at a predictable path and check the content type.
**Best for a WordPress owner:** use a plugin or a small rewrite rule, then confirm the header with `curl -I`.
**Best for a platform team:** negotiate on `Accept` at the CDN, so every page gets a twin without per-page work.

## What C1 tests, and what it does not

C1 sits in the content family, next to C2 (llms.txt), C3 (structured page signals), E3 and E5. You can read the family on the [content checks page](https://aiscan.site/docs/checks/content). C1 asks whether a site can hand an agent Markdown instead of HTML. The evidence line records one request and its result, in the form `<route> → <status> (<content-type>)`.

Grouping every evidence string across the corpus shows what the check actually requests. Two routes appear and no others:

1. `Accept: text/markdown` sent to the homepage. This is HTTP content negotiation.
2. `/index.md`, a Markdown file at a fixed path next to the homepage.

Every verdict in the data rests on those two probes. The evidence of a failing host shows the `Accept` request only, which means we cannot say from the stored report what `/index.md` returned on a host that failed. A `.md` suffix on a page path, which the documentation study found on several sites, is not a probe C1 sends. A site that serves `/about.md` and nothing else fails C1 here, and so does a site that serves Markdown only to named AI user agents.

That matters for how to read the 477 fails. They are hosts where neither route produced a pass, not hosts that serve no Markdown anywhere. It is also why the figures below should be taken as a floor on Markdown availability, with the pass share of 31.1% as the most generous reading the check allows.

### Limits of this measurement

- **The population is self-selected.** Each host was scanned because someone submitted it to AIScan. That skews toward people who care about readiness, and it is not a random sample of the web.
- **One scan per host.** We kept the latest scan for each host under rubric version 2026.08.2, excluded our own domain, and counted 701 hosts. Earlier scans and the older rubric are out.
- **One datacentre origin.** Every probe came from AIScan's scanner. Sites that rate-limit or challenge datacentre traffic are under-represented among the passes.
- **Association is not cause.** Any score comparison below describes who has Markdown. It does not say that adding Markdown raises a score by that amount.
- **We did not re-fetch any host for this post.** The numbers come from the stored evidence strings, so we cannot run a control path against a suspicious pass today. We say where that matters.

No individual scanned URL is named in this post.

## The adoption count: three outcomes, not two

Most posts about Markdown availability report one number. C1's evidence supports three groups, and the split between the first two is the most useful thing in the data.

| Outcome | Hosts | Share of 701 | Average overall score |
|---|---|---|---|
| Pass, response has a Markdown content type | 135 | 19.3% | 87.4 |
| Pass, response is not Markdown | 83 | 11.8% | 65.4 |
| Partial | 6 | 0.9% | 70.0 |
| Fail | 477 | 68.0% | 44.2 |

The 135 sites with real Markdown average 87.4 out of 100. The 477 that fail average 44.2. That 43-point gap is large, and it is mostly a statement about who bothers. A site that has already built a Markdown route has usually also published llms.txt, sent a clean `Link` header and fixed its canonical host. Of the 135 Markdown passes, 124 (91.9%) also pass C2, the llms.txt check. Across the other 566 hosts, 185 pass C2 (32.7%). The two files travel together, which is what you would expect from teams that take agent access seriously.

Our own [ranking of the highest-impact fix](https://aiscan.site/blog/agent-readiness-highest-impact-fix) covers the same warning from the other side: cross-tabs on observational data describe clusters of good practice. They do not rank single changes by cause.

## Two routes, two different adoption rates

Splitting the passes by the route that produced them shows that negotiation and a fixed path are not equally common.

| Route that produced the pass | Hosts | Share of 701 | Of which real Markdown |
|---|---|---|---|
| `Accept: text/markdown` on the homepage | 50 | 7.1% | 50 (all) |
| `/index.md` | 168 | 24.0% | 85 |
| Partial (Accept, Markdown content type in most) | 6 | 0.9% | not counted |

Only 50 hosts honour the `Accept` header with a Markdown response, and every one of those 50 sends the right content type. That makes header negotiation the cleanest signal in the data: a server that answers the header correctly is almost always doing something deliberate, either an edge product such as the one Cloudflare documents or a framework feature.

The fixed path produces more passes and more ambiguity. 168 hosts pass on `/index.md`, and 85 of them return Markdown. The other 83 return something else with a 200 status. The next section covers them.

For comparison, [Cloudflare's April 2026 Radar figure](https://blog.cloudflare.com/agent-readiness/) of 3.9% for negotiation across 200,000 domains sits well below our 7.1%. The gap is what you would expect from two populations. Radar measures the most visited domains on the web, and ours are sites whose owners asked for a readiness grade. The ordering, though, is the same in both: negotiation is rare, and most sites do not offer it.

## The 83 passes that are not Markdown

Of the 83 hosts that pass C1 without a Markdown response, 72 returned `text/html`, 10 returned `application/octet-stream`, and one carried another type. All 83 passed on the `/index.md` route.

There are three ways a request to `/index.md` ends in a 200 with HTML:

1. **A single-page app or catch-all route.** The server hands any unknown path the application shell, with a 200 status. The page looks like a homepage because it is the homepage.
2. **A static host that rewrites unknown paths.** Some hosting setups map every missing file to `index.html`. This is the classic soft 404, and we covered the family in [The Soft 404 Problem: Why HTTP 200 Breaks Every Agent Discovery Check](https://aiscan.site/blog/soft-404-agent-discovery-problem).
3. **A real Markdown file served with the wrong type.** A web server that does not know the `.md` extension may send `text/html` or a generic binary type for a file that really is Markdown. The 10 `application/octet-stream` responses fit this case best.

We cannot tell the three apart without fetching the pages, and the stored evidence has no body. The control-path test, which asks the same host for a nonsense path and compares the result, would separate case 1 and case 2 from case 3 in seconds. In the documentation study that test caught two hosts returning real Markdown for a page that never existed, so the reverse failure is not hypothetical. An earlier audit on 10 September 2026 found that 41.4% of C1 passes carried a non-Markdown content type. The share today is 38.1% (83 of 218). The problem has shrunk slightly as the corpus grew, and it has not gone away.

The likeliest reading, stated as a judgement and not as a measured fact: most of the 72 `text/html` passes are soft-200 pages, and an agent that fetches them receives a full HTML document where it expected Markdown. The practical consequence is the same whichever case applies. **If your C1 line says `/index.md → 200 (text/html)`, the check passed and an agent still gets no Markdown.** Fix the content type, or remove the route.

### What the score gap looks like inside the passes

The average score of the 83 non-Markdown passes is 65.4, between the 87.4 of real-Markdown sites and the 44.2 of failing ones. These sites do more than the average failing site (many have C2, sitemaps and sound titles), and they accidentally earn C1 points. That makes them a good illustration of a check that decides on a status code. Product note 1 near the end of this post asks for the content type to be part of the verdict.

## Platform breakdown

Platform detection names a platform for most hosts. The table shows the six platforms with at least 12 hosts, which together cover 674 of the 701. "Unknown" means no platform signature was detected.

| Platform | Hosts | C1 pass | Real Markdown | Pass without Markdown | Real Markdown rate |
|---|---|---|---|---|---|
| Unknown | 265 | 66 | 32 | 34 | 12.1% |
| WordPress | 185 | 44 | 29 | 15 | 15.7% |
| Next.js | 161 | 51 | 39 | 12 | 24.2% |
| Astro | 34 | 21 | 19 | 2 | 55.9% |
| Shopify | 17 | 13 | 1 | 12 | 5.9% |
| Lovable | 12 | 9 | 4 | 5 | 33.3% |

Four patterns stand out.

**Astro is the Markdown platform.** 19 of 34 hosts return real Markdown, and only 2 of its 21 passes are non-Markdown. Content collections store pages as Markdown or MDX in the first place, so a route that returns the source is a small step. Our guide, [How to serve text/markdown to AI agents on Astro](https://aiscan.site/blog/markdown-for-ai-agents-astro), shows the endpoint file.

**Next.js is second among the large platforms at 24.2%.** 39 of 161 hosts return Markdown, ahead of WordPress at 15.7% and the unknown group at 12.1%. Route handlers and rewrites make a twin practical. See [How to serve text/markdown to AI agents on Next.js](https://aiscan.site/blog/markdown-for-ai-agents-nextjs).

**Shopify is the starkest case, and the sample is small.** Of 17 hosts, 13 pass C1, and 12 of those 13 passes are not Markdown. One host returns real Markdown. A store's `/index.md` falls through to a themed page that returns 200 with HTML, so the check passes for the wrong reason. With 17 hosts this is a pattern worth testing on your own store, not a platform-wide rate. If you run a Shopify store, [StoreSEO](https://storeseo.com/) is the app we point stores to for AI readiness work, because it handles llms.txt, schema and the other discovery files in one place. It does not change what a theme returns for `/index.md`, so verify that route by hand either way.

**WordPress sits in the middle with a large non-Markdown tail.** 15 of its 44 passes return something other than Markdown. WordPress sites often rewrite unknown paths to a theme template, which is the second cause above. The remedy is a deliberate route, covered below.

The Unknown group (265 hosts) is the largest single block. These are mostly custom builds and platforms without a detectable signature. Their 12.1% real-Markdown rate is the closest thing in this study to a general baseline for the open web.

## Failure anatomy: what the 477 failing hosts did

The failing evidence strings fall into a few clear groups.

| What the Accept probe returned | Hosts | Share of 701 |
|---|---|---|
| 200 with `text/html` (header ignored) | 381 | 54.4% |
| 403 or 429 (blocked or rate limited) | 56 | 8.0% |
| 5xx (server or edge error) | 22 | 3.1% |
| No response | 5 | 0.7% |
| Other (404, 402, 202, redirects) | 13 | 1.9% |

The 381 hosts that returned HTML with a 200 are the ordinary case: the server saw the header, did not know what to do with it and sent the page anyway. That is correct HTTP behaviour. A client that asks for a representation the server does not offer may receive the default. It is also the whole reason negotiation adoption is low. Nothing breaks when a site ignores the header, so nothing prompts the owner to add it.

The 78 hosts that returned a 403, 429 or 5xx matter for a different reason. They are 11.1% of the corpus, and the probe learned nothing about their Markdown at all. Some of them probably serve it. A scanner on a datacentre address looks like the traffic that bot protection exists to challenge. This is why we describe the 19.3% as a floor.

It also connects to a separate finding. Our study of [what sites tell GPTBot](https://aiscan.site/blog/ai-crawler-declaration-census) showed that declared policy and enforced policy differ, and the same gap applies here: a site can intend to serve agents Markdown and still block the request that would show it.

## How to serve a Markdown twin

There are two ways to pass C1 properly, and the order matters. Start with the fixed path, since it works on every host, then add negotiation if your stack supports it.

### Option 1: a Markdown file at a fixed path

Publish a Markdown rendition of each key page at a predictable address, and send it with the right content type. The check tries `/index.md`, so the homepage twin goes there. For other pages, the convention that documentation sites use is the page path plus `.md`, or `index.md` inside the page directory.

```bash
# the response you want to see
curl -sI https://example.com/index.md | grep -i -E "^(HTTP|content-type)"
# HTTP/2 200
# content-type: text/markdown; charset=utf-8
```

If the status is 200 and the type is `text/html`, the route is falling through to your template. Add the explicit type for the extension in your server:

```nginx
# nginx: send .md files as Markdown
types { text/markdown md; }
```

```apache
# Apache
AddType text/markdown .md
```

On a static host with a catch-all rewrite, add a rule that serves real files first, and let the rewrite apply only when no file exists. On a framework, add a route that renders the page's source or a conversion of it. The step-by-step versions are in our guides for [Docusaurus](https://aiscan.site/blog/markdown-for-ai-agents-docusaurus), [Hugo](https://aiscan.site/blog/markdown-for-ai-agents-hugo), [Astro](https://aiscan.site/blog/markdown-for-ai-agents-astro), [Next.js](https://aiscan.site/blog/markdown-for-ai-agents-nextjs) and [WordPress](https://aiscan.site/blog/markdown-for-ai-agents-wordpress).

### Option 2: content negotiation on the Accept header

Return Markdown at the same URL when the client asks for it. The response needs `Content-Type: text/markdown` and should carry `Vary: Accept`, so that a cache does not hand the Markdown copy to a browser. On Cloudflare, the product described in the [Markdown for Agents documentation](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) does the conversion at the edge for enabled zones, so you add no routes of your own. In your own server, branch on the `Accept` header and return the twin.

Negotiation has one advantage and one risk. The advantage is that the agent needs no knowledge of your URL scheme. The risk is cache poisoning if `Vary` is missing. That is why 50 of 701 hosts is the figure to beat, and why the fixed path is the safer first step.

### A note on WordPress

[ThinkRank](https://thinkrank.ai) is the WordPress option we recommend first for the agent-facing files around this check (llms.txt, robots rules, schema and sitemaps). It is one plugin in place of several that compete over the same virtual files, and it imports settings from Rank Math, Yoast, AIOSEO and SEOPress. For Markdown output specifically, our [WordPress guide](https://aiscan.site/blog/markdown-for-ai-agents-wordpress) shows a rewrite approach that does not depend on any one plugin, and other plugins that add Markdown routes exist too. Whichever you pick, test the content type afterwards, since a plugin that registers a route can still send it as HTML.

## Verify it: AIScan first, curl second

Run the scan: `npx aiscan-cli yoursite.com`, or paste your address at [aiscan.site](https://aiscan.site/). It is free and needs no account. Read the **C1** line. The evidence tells you which route was tried and what came back. A line such as `/index.md → 200 (text/markdown; charset=utf-8)` is the result you want. If you see `text/html` on a pass, treat it as a soft-200 and fix the content type.

If you would rather check by hand, three commands cover the same ground:

```bash
# 1. negotiation: does the header change the response?
curl -sI -H "Accept: text/markdown" https://example.com/ | grep -i content-type
# 2. fixed path
curl -sI https://example.com/index.md | grep -i -E "^(HTTP|content-type)"
# 3. control path: a nonsense URL must NOT return 200 Markdown
curl -sI https://example.com/this-page-does-not-exist-9731.md | head -1
```

The third command is the one most guides leave out. If the control path returns 200 with the same size and type as `/index.md`, your server is answering every path and the first result proves nothing. The remedy for that is in [What a real 404 should return to an AI agent](https://aiscan.site/blog/correct-404-for-ai-agents).

## Reading your own C1 line

The evidence string is short, and each shape has one meaning and one fix.

| Evidence line | What it means | Fix |
|---|---|---|
| `/index.md → 200 (text/markdown…)` | Real Markdown on the fixed path | None. Add negotiation later if you want it |
| `Accept: text/markdown → 200 (text/markdown…)` | Negotiation works | Check that `Vary: Accept` is present |
| `/index.md → 200 (text/html…)` | A pass that gives an agent no Markdown | Set the content type, or fix the catch-all rewrite |
| `Accept: text/markdown → 200 (text/html…)` | The header is ignored and no fixed path passed | Publish a twin at `/index.md` |
| `Accept: text/markdown → 403` or `429` | The probe was blocked, so nothing was learned | Allow the scanner, or test from your own machine |

Start from the bottom of the table if your host blocks scanners, since a blocked probe tells you nothing about your Markdown. Start from the third row if you already pass, because that is the pass most likely to mislead you. Rescan after each change, and read the content type in the line, not only the verdict beside it.

## Product notes from the data

Grouping the evidence strings, as we do before writing about any check, turned up three things in the rubric that are worth fixing.

1. **C1 decides on the status code and ignores the content type on the fixed path.** 83 passes carry a non-Markdown type. The verdict should require a Markdown type, or at least mark the pass as partial.
2. **The stored evidence for a failure shows only the Accept probe.** We cannot see what `/index.md` returned on those hosts. Recording both probes would remove the blind spot in the failure table above.
3. **The `.md` suffix on a page path is not tried.** Several documentation sites use it, so the check misses a convention that is in real use.

We also log 403, 429 and 5xx responses as failures with the same weight as a plain "no Markdown". They are really "could not tell", and the report should say so. None of these changes the headline: Markdown is available on roughly one site in five, and more than one in ten of the passes need a second look.

## Reproduce the numbers

The query below runs against the AIScan scans table. It aggregates in SQL and never selects rows, which is why no host appears in this post.

```sql
with latest as (
  select distinct on (lower(split_part(regexp_replace(url,'^https?://(www\.)?',''),'/',1)))
         url, platform, score, result
  from scans
  where rubric_version = '2026.08.2' and scope = 'site' and url not ilike '%aiscan.site%'
  order by lower(split_part(regexp_replace(url,'^https?://(www\.)?',''),'/',1)), created_at desc
),
c1 as (
  select platform, score, c->>'status' s, c->>'evidence' ev
  from latest, jsonb_array_elements(result->'checks') c
  where c->>'id' = 'C1'
)
select count(*) hosts,
       count(*) filter (where s = 'pass') pass,
       count(*) filter (where s = 'pass' and ev like '%markdown%') pass_markdown,
       count(*) filter (where s = 'pass' and ev not like '%markdown%') pass_not_markdown,
       count(*) filter (where s = 'pass' and ev like 'Accept%') pass_accept,
       count(*) filter (where s = 'fail' and ev ~ '→ 200') fail_200,
       count(*) filter (where s = 'fail' and ev ~ '→ (403|429|5[0-9][0-9])') fail_blocked
from c1;
```

## Your next action

Scan your domain with [AIScan](https://aiscan.site/) and read the **C1** line, then the **C2** line next to it, since the two files travel together. If C1 passes on a `text/html` response, fix the content type first. If it fails, publish one Markdown twin at `/index.md`, rescan, then decide whether to add negotiation. The rest of the family, C3, E3 and E5, is on the [content checks page](https://aiscan.site/docs/checks/content), and every guide is collected at [/guides](https://aiscan.site/guides). For the wider picture, see [the state of AI agent readiness in 2026](https://aiscan.site/blog/state-of-ai-agent-readiness-2026) and [Does llms.txt actually work in 2026?](https://aiscan.site/blog/does-llms-txt-actually-work-2026).