---
title: "How to serve text/markdown to AI agents on every stack"
slug: markdown-for-ai-agents
published: 2026-11-02T12:25:43.352051+00:00
updated: 2026-11-03T12:17:23.765015+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:C1, platform:astro, platform:docusaurus, platform:hugo, platform:nextjs, platform:wordpress, Markdown, content negotiation, AI agents, nginx, WordPress
description: "Serve Markdown to AI agents that send Accept: text/markdown on Astro, Docusaurus, Hugo, Next.js and WordPress, with tested config per stack. Fixes check C1."
url: https://aiscan.site/blog/markdown-for-ai-agents
---

An AI agent that sends `Accept: text/markdown` gets your HTML unless the server answers differently. Check C1 grades that exact request, and no static-site generator or Next.js app does it by default. The fix has two halves: something must produce a Markdown copy of each page, and something that sees the request header must choose between the copies at the same URL. This guide gives the working setup for Astro, Docusaurus, Hugo, Next.js and WordPress, each built and tested by us between 30 September and 5 October 2026, and WordPress. Browsers keep getting HTML in every setup.

## Quick summary

| Your stack | Who produces the Markdown | Who picks it by `Accept` | Jump to |
|---|---|---|---|
| Astro | One on-demand endpoint returning the entry's raw `body` | `src/middleware.ts` | [Astro](#astro) |
| Docusaurus | `docusaurus-plugin-llms` writes a `.md` twin per page | An nginx `map` plus `try_files` | [Docusaurus](#docusaurus) |
| Hugo | The built-in `markdown` output format and `layouts/page.md` | An nginx `map` plus `try_files` | [Hugo](#hugo) |
| Next.js | A Route Handler under `app/md/` | A `beforeFiles` rewrite with a `has` header match | [Next.js](#next-js) |
| WordPress | One must-use plugin that converts the post body with `DOMDocument` | The same plugin, on `template_redirect` | [WordPress](#wordpress) |

Every setup must send `Vary: Accept` so a cache cannot hand the wrong copy to the next visitor, and every one needs a server that can run code or read a header. A host that only serves files by path cannot pass C1, whatever the build produces.

## What check C1 asks, and how common the answer is

C1 re-requests a page with `Accept: text/markdown` and passes when the server answers with a `text/markdown` content type. That is proactive negotiation as defined in [RFC 9110, section 12](https://www.rfc-editor.org/rfc/rfc9110.html#name-proactive-negotiation) (retrieved 2 November 2026): one URL, several representations, chosen by a request header. A `/page.md` file on its own does not satisfy it, because the scanner asks the original URL. [Cloudflare's Markdown for Agents documentation](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) (retrieved 2 November 2026) describes the same mechanism done at the edge.

Our corpus data shows where each stack stands. These counts come from the AIScan database, queried on 8 October 2026 across 701 distinct hosts, and are the source for [the full study](https://aiscan.site/blog/markdown-twin-beyond-docs):

| Platform | Hosts | Real Markdown | Real Markdown rate |
|---|---|---|---|
| Astro | 34 | 19 | 55.9% |
| Next.js | 161 | 39 | 24.2% |
| WordPress | 185 | 29 | 15.7% |
| All 701 hosts | 701 | 135 | 19.3% |

Docusaurus and Hugo did not reach the 12-host cutoff for the platform table, so they have no row. The sample is a set of sites people scanned, not a random draw of the web.

## Contents

- [Astro](#astro)
- [Docusaurus](#docusaurus)
- [Hugo](#hugo)
- [Next.js](#next-js)
- [WordPress](#wordpress)
- [How to check it](#check-it-scan-first-curl-second)

## Astro

Tested on Astro 7.3.5, `@astrojs/node` 11.1.6 and Node 22.22.0 on 2 October 2026. Adds the representation with one middleware file and one endpoint.

### Why Astro needs a server for this

Check C1 re-requests a page with `Accept: text/markdown` and passes when the server answers `text/markdown`. That is proactive negotiation as defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-proactive-negotiation): one URL, several representations, picked by a request header. A `/page.md` sidecar does not satisfy it, because the scanner asks the original URL.

According to the [Astro on-demand rendering guide](https://docs.astro.build/en/guides/on-demand-rendering/), pages are prerendered at build time by default, and rendering on the server needs an adapter. A prerendered page is a file. A file cannot look at a request header, so the negotiation has to run at request time. That is the whole reason this guide uses `output: 'server'`.

### Step 1 (Astro): switch the build to on-demand rendering

1. Run **`npx astro add node`** and accept the prompts. The command installs `@astrojs/node` and edits your config.
2. Open **`astro.config.mjs`** and confirm it matches the block below.

```js
import { defineConfig } from 'astro/config';
import node from '@astrojs/node';

export default defineConfig({
  output: 'server',
  adapter: node({ mode: 'standalone' }),
});
```

If you deploy to Netlify, Vercel or Cloudflare, install that adapter instead. The middleware and endpoint below do not change.

### Step 2 (Astro): add the Markdown endpoint

The [endpoints guide](https://docs.astro.build/en/guides/endpoints/) says a file in `src/pages` that exports a `GET` function becomes a route. Create **`src/pages/md/[...slug].ts`**. It reads from a content collection named `docs` (a `glob` loader over `src/content/docs/**/*.md`), so change the collection name to yours.

```ts
import type { APIRoute } from 'astro';
import { getEntry } from 'astro:content';

export const GET: APIRoute = async ({ params }) => {
  const entry = await getEntry('docs', params.slug!);
  if (!entry) return new Response('Not found', { status: 404 });
  const body = `# ${entry.data.title}\n\n${entry.body}\n`;
  return new Response(body, {
    headers: { 'Content-Type': 'text/markdown; charset=utf-8', Vary: 'Accept' },
  });
};
```

`entry.body` is the raw Markdown of the file without its frontmatter, so the response is your own source and no HTML-to-Markdown converter is involved. The title is prepended because it lives in frontmatter, not in the body.

### Step 3 (Astro): route by Accept header in middleware

Per the [middleware guide](https://docs.astro.build/en/guides/middleware/), `next()` accepts a path and rewrites the request in place, without a redirect and without running middleware a second time. Create **`src/middleware.ts`**:

```ts
import { defineMiddleware } from 'astro:middleware';

function prefersMarkdown(accept: string | null): boolean {
  if (!accept) return false;
  const q = (type: string) => {
    for (const part of accept.split(',')) {
      const [t, ...params] = part.trim().split(';');
      if (t.trim().toLowerCase() !== type) continue;
      const qp = params.find((p) => p.trim().startsWith('q='));
      return qp ? parseFloat(qp.trim().slice(2)) : 1;
    }
    return 0;
  };
  const md = q('text/markdown');
  return md > 0 && md >= q('text/html');
}

export const onRequest = defineMiddleware(async (context, next) => {
  const { pathname } = context.url;
  if (
    context.request.method === 'GET' &&
    pathname.startsWith('/docs/') &&
    prefersMarkdown(context.request.headers.get('accept'))
  ) {
    return next('/md' + pathname.slice('/docs'.length));
  }
  const res = await next();
  if (pathname.startsWith('/docs/')) res.headers.append('Vary', 'Accept');
  return res;
});
```

Markdown wins only when it is listed, is not `q=0`, and ranks at least as high as `text/html`. A browser sending `text/html,*/*` stays on HTML. The last three lines mark the HTML response as `Vary: Accept`, so a cache cannot hand a browser the Markdown copy. Change the `/docs/` prefix to wherever your pages live.

### Step 4 (Astro): send the same URL different Accept values

Build and start the server with `npx astro build` and `HOST=127.0.0.1 PORT=4321 node dist/server/entry.mjs`, then request one URL several ways. These are the results from the test app:

| Accept header | Status | Content-Type | Body starts with |
|---|---|---|---|
| none | 200 | `text/html` | `<!DOCTYPE html>` |
| `text/markdown` | 200 | `text/markdown; charset=utf-8` | `# Hello Astro` |
| `text/markdown, text/html;q=0.8` | 200 | `text/markdown; charset=utf-8` | `# Hello Astro` |
| `text/html, text/markdown;q=0.5` | 200 | `text/html` | `<!DOCTYPE html>` |
| `*/*` | 200 | `text/html` | `<!DOCTYPE html>` |
| `text/markdown;q=0` | 200 | `text/html` | `<!DOCTYPE html>` |

Every row carried `Vary: Accept`. An unknown slug with `Accept: text/markdown` returned a real 404 from the endpoint, so a missing page never turns into an empty 200.

### The prerender trap on a hybrid site

Pages you mark `export const prerender = true` do not reach the middleware at request time. I set one docs route to prerender, rebuilt, and requested it with `Accept: text/markdown`. The Node adapter served the built `index.html`, with `Content-Type: text/html; charset=utf-8` and no `Vary` header, for both `/docs/hello` and `/docs/hello/`. The build also logged that `Astro.request.headers` is not available on prerendered pages.

| Your setup | Does negotiation work? | What to do |
|---|---|---|
| `output: 'server'`, no `prerender = true` on docs routes | Yes | Steps 1 to 3 |
| `output: 'static'` with the docs routes opted out via `prerender = false` | Yes | Same files, plus the adapter |
| Docs routes prerendered | No | Remove `prerender = true` from them |
| Pure static site, no adapter | Not in Astro | Use a host rule (below) |

### Static hosting without an adapter

A static build has no request-time code, so Astro alone cannot satisfy C1. You can still emit the Markdown files at build time with a `src/pages/docs/[...slug].md.ts` endpoint that exports `getStaticPaths()`. I built that on a static project and it wrote `dist/docs/hello.md`. The `Accept` rewrite then has to live on your host or CDN, and each host expresses that differently. I did not test any host here, so follow your host's rewrite documentation and finish with the check below.

## Docusaurus

Tested on Docusaurus 3.10.2, `docusaurus-plugin-llms` 0.6.1 and nginx 1.24.0 on 4 October 2026.

### Why Docusaurus needs two halves

Check C1 re-requests a page with `Accept: text/markdown` and passes when the server answers `text/markdown`. That is proactive negotiation as defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-proactive-negotiation): one URL, several representations, chosen by a request header. A `/page.md` file alone does not pass, because the scanner asks the original URL.

Docusaurus can supply the first half only with help. It compiles your `.md` and `.mdx` sources into React, so the build output holds `docs/intro/index.html` and no Markdown at all. The [`docusaurus-plugin-llms`](https://www.npmjs.com/package/docusaurus-plugin-llms) package fills that gap, and the web server does the second half. Docusaurus has no request-time code in a static build.

### Step 1 (Docusaurus): install the plugin

From the project root, run **`npm install docusaurus-plugin-llms`**. It lists `@docusaurus/core` `^3.0.0` as a peer dependency.

### Step 2 (Docusaurus): write a Markdown copy of every page

Open **`docusaurus.config.js`** and add a `plugins` array next to `presets`:

```js
plugins: [
  ['docusaurus-plugin-llms', {
    generateMarkdownFiles: true,
    includeBlog: true,
    excludeImports: true,
  }],
],
```

Also set `url` to your real site address. The plugin writes absolute links into `llms.txt` from that value, and the scaffold ships with `https://your-docusaurus-site.example.com`.

Run **`npm run build`**. On the scaffold from `create-docusaurus` the log said `Generating individual markdown files...` and `Stats: 13 documents processed`. Each page now has a twin beside its folder:

```text
build/docs/intro/index.html
build/docs/intro.md
build/blog/first-blog-post/index.html
build/blog/first-blog-post.md
```

I read all three option names from the package source. `generateMarkdownFiles` and `includeBlog` both default to `false`, so with the defaults you get `llms.txt` and `llms-full.txt` but no per-page files, and no blog pages at all.

### Step 3 (Docusaurus): check what an agent will read

Open one generated file. Front matter was gone and the title became a `#` heading. Admonitions stayed as Docusaurus directive syntax (`:::tip[My tip]`), which an agent reads fine. Two things did not convert:

- `excludeImports: true` removed `import` lines, but an inline `export const Highlight = ...` block and the `<Highlight>` tags that use it stayed in the output. Pages that lean on custom MDX components will carry that markup into the Markdown.
- The home page and the `/docs` category pages have no twin. Only pages built from a source file do.

Read your most component-heavy page before you ship. If it is unreadable, the plugin's `preserveComponents` option exists for this case, but I did not test it.

### Step 4 (Docusaurus): route by Accept header in nginx

Docusaurus builds `docs/intro/index.html` for the route `/docs/intro`, and the twin sits at `/docs/intro.md`. So the rule is "append `.md` to the path when the client prefers Markdown". In the `http` block of **`/etc/nginx/nginx.conf`** or a file it includes, add:

```nginx
types { text/markdown md; }

map $http_accept $md_suffix {
    default "";
    "~*^text/markdown(?!\s*;\s*q=0(\.0+)?(,|$))" ".md";
}

map $uri $bare_uri {
    default $uri;
    "~^(?<p>.+?)/+$" $p;
}
```

Then, in your **`server`** block, point `root` at the `build` folder:

```nginx
add_header Vary Accept always;

location / {
    try_files $bare_uri$md_suffix $uri $uri/index.html =404;
}
error_page 404 /404.html;
```

Reload with **`nginx -t && nginx -s reload`**. The second `map` strips a trailing slash so `/docs/intro/` and `/docs/intro` both find `intro.md`. The first pattern matches only when the header begins with `text/markdown` and that type is not `q=0`. A browser header that starts with `text/html` never matches, and `Vary: Accept` tells caches the same URL has two answers.

### Step 5 (Docusaurus): send one URL several Accept values

I requested `/docs/intro` with different headers and read the status and content type:

| Accept header | Content-Type returned |
|---|---|
| none | `text/html` |
| `text/markdown` | `text/markdown` |
| `text/markdown, text/html;q=0.8` | `text/markdown` |
| `text/html, text/markdown;q=0.5` | `text/html` |
| `*/*` | `text/html` |
| `text/markdown;q=0` | `text/html` |
| `text/markdown;q=0, text/html` | `text/html` |

All returned 200 with `Vary: Accept`. `/docs/intro/` with a trailing slash and `/docs/tutorial-basics/create-a-page` answered Markdown too, and `/docs/nope` returned 404. The home page `/` answered `text/html` even for `Accept: text/markdown`, which is the gap from Step 3.

### Which route are you on

Pick the branch that matches how the built site is served:

- **You run nginx yourself.** Use Step 4 as written.
- **You sit behind a CDN or edge platform.** Put the same rule in its rewrite layer and keep `Vary: Accept` on both answers. Vercel documents header conditions on rewrites; I did not test it.
- **You publish to GitHub Pages.** The files exist and are linked from `llms.txt`, but C1 cannot pass because the host serves strictly by path.

To find your branch, run the `curl -sI` pair under the check section against your live URL. A `server:` header naming nginx means the first branch applies.

### Two traps I hit

The first is the MIME type. Without `types { text/markdown md; }`, nginx serves `intro.md` as `text/plain` on Ubuntu, and C1 wants `text/markdown`. The second is the host. GitHub Pages and other file-only hosts cannot read the header, so Steps 1 to 3 still produce linked `.md` files but C1 will not pass. Vercel's [`vercel.json` reference](https://vercel.com/docs/project-configuration/vercel-json) documents `has` conditions on a request header, which is the feature a rewrite like this needs. I tested only nginx, so confirm anything else with the commands below.

## Hugo

Tested on Hugo v0.167.0 and nginx 1.24.0 on 3 October 2026.

### Why Hugo needs two halves

Check C1 re-requests a page with `Accept: text/markdown` and passes when the server answers `text/markdown`. That is proactive negotiation as defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-proactive-negotiation): one URL, several representations, chosen by a request header. A `/page.md` sidecar alone does not pass, because the scanner asks the original URL.

Hugo can do the first half and cannot do the second. The [output formats documentation](https://gohugo.io/configuration/output-formats/) lists `markdown` among the built-in formats, with media type `text/markdown` and the base name `index`. Hugo has no request-time code, so the choice between the two files belongs to whatever serves them. The dev server confirmed it: `hugo server` answered `Accept: text/markdown` with `text/html; charset=utf-8`.

### Step 1 (Hugo): turn on the Markdown output format

Open **`hugo.toml`** and list `markdown` next to `html` for the page kinds you want agents to read. The home page and section lists need their own entries, otherwise they have no Markdown twin.

```toml
[outputs]
  home    = ['html', 'markdown']
  section = ['html', 'markdown']
  page    = ['html', 'markdown']
```

Hugo now writes `public/posts/hello/index.md` beside `public/posts/hello/index.html`. Change nothing else yet; the file will be empty until a template exists.

### Step 2 (Hugo): add the Markdown templates

Hugo 0.146 overhauled template lookup, and the [lookup order page](https://gohugo.io/templates/lookup-order/) now names templates by kind, so a single page is `page.md`, the home page is `home.md` and a section is `section.md`. Create **`layouts/page.md`**:

```go-html-template
# {{ .Title }}

{{ .Content | transform.HTMLToMarkdown }}
```

Then create **`layouts/home.md`** and copy it to **`layouts/section.md`**:

```go-html-template
# {{ .Site.Title }}

{{ range site.RegularPages }}- [{{ .Title }}]({{ .Permalink }})
{{ end }}
```

I used `transform.HTMLToMarkdown` instead of `.RawContent` on purpose. `.RawContent` prints the source file, so a page containing `{{< figure src="/a.png" caption="Cap" >}}` came out with the shortcode text still in it. Converting the rendered `.Content` produced `![Cap](/a.png)` and a caption line instead, and tables survived as pipe tables. The [function's documentation](https://gohugo.io/functions/transform/htmltomarkdown/) marks it as new in v0.151.0 and experimental, so pin your Hugo version and re-run the check after upgrades.

### Step 3 (Hugo): advertise the twin in the HTML head

Agents that read the page first can find the twin from a link element. Add this inside `<head>` in **`layouts/baseof.html`**, or in the partial your theme uses for the head:

```go-html-template
{{ range .AlternativeOutputFormats -}}
  <link rel="{{ .Rel }}" type="{{ .MediaType.Type }}" href="{{ .Permalink }}">
{{ end -}}
```

The built page carried `<link rel="alternate" type="text/markdown" href="https://example.com/posts/hello/index.md">`. This is a hint and does not replace Step 4.

### Step 4 (Hugo): route by Accept header in nginx

Hugo's output is a folder of files, so the rule is "serve `index.md` instead of `index.html` when the client prefers Markdown". In the `http` block of **`/etc/nginx/nginx.conf`** or a file it includes, add:

```nginx
types { text/markdown md; }

map $http_accept $index_file {
    default index.html;
    "~*^text/markdown(?!\s*;\s*q=0(\.0+)?(,|$))" index.md;
}
```

Then, in your **`server`** block, point `root` at the Hugo `public` folder and use `try_files`:

```nginx
add_header Vary Accept always;

location / {
    try_files $uri $uri$index_file $uri/$index_file =404;
}
```

Reload with **`nginx -t && nginx -s reload`**. The pattern matches only when the header begins with `text/markdown` and that type is not set to `q=0`. A browser header that starts with `text/html` never matches. `Vary: Accept` tells caches that the same URL has two answers.

If you would rather use Caddy, Apache, or a CDN rule, the logic is the same, but I tested only nginx.

### Step 5 (Hugo): send one URL several Accept values

Request the same page with different headers and read the status and content type. These are the results from the test site:

| Accept header | Content-Type returned |
|---|---|
| none | `text/html` |
| `text/markdown` | `text/markdown` |
| `text/markdown, text/html;q=0.8` | `text/markdown` |
| `text/html, text/markdown;q=0.5` | `text/html` |
| `*/*` | `text/html` |
| `text/markdown;q=0` | `text/html` |
| `text/markdown;q=0, text/html` | `text/html` |

All of them returned 200, including `/posts/hello` without the trailing slash, and every response carried `Vary: Accept`. The home page also answered Markdown once `home` was listed in `[outputs]`. Before I added it, `Accept: text/markdown` on `/` returned 404, because `index.md` did not exist at the root.

### Two traps I hit

The first is the MIME type. Without the `types { text/markdown md; }` line, nginx served `index.md` as `text/plain`, and C1 wants `text/markdown`. The stock `mime.types` file on Ubuntu has no entry for `md`.

The second is the host. A Hugo site on GitHub Pages, or any host that only serves files by path, cannot read the header, so Steps 1 to 3 still help (the `.md` files exist and are linked) but C1 will not pass. I did not test Netlify or Cloudflare Pages here; check that host's documentation for header-based rewrites, then confirm with the commands below.

## Next.js

Tested live on Next.js 16.3.7 on 30 September 2026. Next.js's own docs are the model for this pattern.

### Why a rewrite and not a `.md` URL

A page component cannot change the content type, because a page always renders HTML. A Route Handler can return any `Response`, so it is the right home for the Markdown. A rewrite then points the agent's request at that handler without changing the URL the agent asked for. The [Next.js rewrites reference](https://nextjs.org/docs/app/api-reference/config/next-config-js/rewrites) documents the `has` field: the rewrite applies only when a header, cookie, host or query value also matches.

`beforeFiles` matters. Per the same page, `beforeFiles` rewrites are checked before the filesystem and dynamic routes, so the rewrite wins over `app/blog/[slug]/page.tsx`. An `afterFiles` rewrite would never fire, because the page already matches.

### Step 1 (Next.js): add the rewrite

1. Open **next.config.ts** in the project root.
2. Add a `rewrites()` function that returns an object with a `beforeFiles` array:

```ts
import type { NextConfig } from 'next'

const config: NextConfig = {
  async rewrites() {
    return {
      beforeFiles: [
        {
          source: '/blog/:slug',
          has: [{ type: 'header', key: 'accept', value: '(.*)text/markdown(.*)' }],
          destination: '/md/blog/:slug',
        },
      ],
    }
  },
}
export default config
```

The `value` is a regex-like string, so `(.*)text/markdown(.*)` matches `text/markdown` anywhere in the header, including `text/markdown, text/html;q=0.5`. Repeat the rule for each section of the site you want covered, for example `/docs/:path*`.

### Step 2 (Next.js): add the Route Handler

1. Create the folder **app/md/blog/[slug]/** and add **route.ts** inside it.
2. Return Markdown with the right type and a `Vary` header:

```ts
export async function GET(
  _req: Request,
  { params }: { params: Promise<{ slug: string }> },
) {
  const { slug } = await params
  const body = await getPostAsMarkdown(slug) // your own loader
  if (!body) return new Response('Not found', { status: 404 })
  return new Response(body, {
    headers: {
      'Content-Type': 'text/markdown; charset=utf-8',
      Vary: 'Accept',
    },
  })
}
```

`getPostAsMarkdown` is yours. If your posts already live as MDX or Markdown files, read the file and strip the frontmatter. If they come from a CMS, convert the rich text to Markdown here. Return a real 404 for a slug that does not exist, not a 200 with an error page, or a scanner cannot tell a working route from a catch-all.

### Step 3 (Next.js): handle caching

Two caches can serve the wrong representation: a CDN in front of the site and any shared proxy. The fix is `Vary: Accept`, which tells a cache that the same URL has more than one version.

In our test on Next.js 16.3.7 with `next start`, the Markdown response carried `vary: Accept` from the Route Handler. The HTML response did not. Its `Vary` listed the RSC headers and `Accept-Encoding`, and adding a `headers()` entry in `next.config.ts` or appending the header in `proxy.ts` did not change that. So set the cache key at the layer that caches: add `Accept` to the cache key or `Vary` handling in your CDN or host settings. If you skip that, a CDN that cached the HTML first will hand HTML to the next agent.

### Which branch are you on?

| Your setup | What to do |
|---|---|
| Self-hosted `next start` behind Nginx or Cloudflare | Rewrite plus CDN `Vary: Accept` setting |
| Vercel or another managed host | Same rewrite and handler. Check the response headers before you assume the platform varies on `Accept` |
| Static export (`output: 'export'`) | Rewrites need a server, so this pattern does not apply. Generate `.md` files at build time instead |

To find your branch, run the verification below and read the `vary` line.

### When it does not work

| Symptom | Cause | Fix |
|---|---|---|
| Agent still gets HTML | The rewrite is in `afterFiles`, or `source` does not match the real path | Move it to `beforeFiles` and compare `source` with the URL you request |
| Browser shows Markdown | A cache stored the Markdown under the plain URL | Add `Accept` to the cache key, then purge that path |
| Every URL returns Markdown | `source` is a broad pattern and the handler never returns 404 | Narrow `source` and return 404 for unknown slugs |

After changing **next.config.ts**, rebuild and restart the server before you test.

## WordPress

WordPress core does no content negotiation for Markdown, so an agent that sends `Accept: text/markdown` to a post or page gets the full theme HTML. The fix is one file of PHP in `mu-plugins`, so browsers keep getting HTML and agents get clean Markdown from the same URL. Built and tested on WordPress 7.1.2 and PHP 8.3.6 on 1 October 2026. According to AIScan's own C1 guidance for WordPress (fetched from the AIScan MCP on 1 October 2026), the starter snippet serves only `/index.md` and strips all markup with `wp_strip_all_tags`, which flattens headings and links. The plugin below keeps both.

### Step 1 (WordPress): add the must-use plugin

According to the [must-use plugins documentation](https://developer.wordpress.org/advanced-administration/plugins/mu-plugins/), WordPress loads every PHP file in `wp-content/mu-plugins` on every request with no activation step. A fresh install ships without that folder.

1. Connect over SFTP or your host's file manager.
2. In the WordPress root, open **`wp-content`** and create a folder named **`mu-plugins`** if it is missing.
3. Create **`wp-content/mu-plugins/markdown-negotiation.php`** with the code below.

```php
<?php
/**
 * Plugin Name: Markdown negotiation
 * Description: Answer Accept: text/markdown on single posts and pages.
 */

function mdn_wants_markdown() {
	$accept = $_SERVER['HTTP_ACCEPT'] ?? '';
	foreach ( explode( ',', $accept ) as $part ) {
		$bits = array_map( 'trim', explode( ';', $part ) );
		if ( strtolower( $bits[0] ) !== 'text/markdown' ) {
			continue;
		}
		foreach ( array_slice( $bits, 1 ) as $param ) {
			if ( preg_match( '/^q=0(\.0*)?$/', $param ) ) {
				return false;
			}
		}
		return true;
	}
	return false;
}

function mdn_node( DOMNode $n ) {
	if ( $n->nodeType === XML_TEXT_NODE ) {
		if ( trim( $n->textContent ) === '' && in_array( $n->parentNode->nodeName, array( 'body', 'ul', 'ol', 'table', 'thead', 'tbody', 'tr' ), true ) ) {
			return '';
		}
		return preg_replace( '/\s+/', ' ', $n->textContent );
	}
	if ( $n->nodeType !== XML_ELEMENT_NODE ) {
		return '';
	}
	$inner = '';
	foreach ( $n->childNodes as $c ) {
		$inner .= mdn_node( $c );
	}
	$tag = strtolower( $n->nodeName );
	switch ( $tag ) {
		case 'h1': case 'h2': case 'h3': case 'h4': case 'h5': case 'h6':
			return "\n\n" . str_repeat( '#', (int) $tag[1] ) . ' ' . trim( $inner ) . "\n\n";
		case 'p':
			return "\n\n" . trim( $inner ) . "\n\n";
		case 'br':
			return "  \n";
		case 'strong': case 'b':
			return '**' . trim( $inner ) . '**';
		case 'em': case 'i':
			return '*' . trim( $inner ) . '*';
		case 'a':
			return '[' . trim( $inner ) . '](' . $n->getAttribute( 'href' ) . ')';
		case 'img':
			return '![' . $n->getAttribute( 'alt' ) . '](' . $n->getAttribute( 'src' ) . ')';
		case 'code':
			return $n->parentNode->nodeName === 'pre' ? $n->textContent : '`' . $n->textContent . '`';
		case 'pre':
			return "\n\n```\n" . rtrim( $n->textContent ) . "\n```\n\n";
		case 'li':
			$mark = $n->parentNode->nodeName === 'ol' ? '1. ' : '- ';
			return $mark . trim( $inner ) . "\n";
		case 'ul': case 'ol':
			return "\n\n" . $inner . "\n";
		case 'blockquote':
			return "\n\n> " . trim( str_replace( "\n", "\n> ", trim( $inner ) ) ) . "\n\n";
		case 'tr':
			$cells = array();
			foreach ( $n->childNodes as $c ) {
				if ( $c->nodeType === XML_ELEMENT_NODE ) {
					$cells[] = trim( str_replace( '|', '\|', mdn_node( $c ) ) );
				}
			}
			$row = '| ' . implode( ' | ', $cells ) . " |\n";
			if ( $n->parentNode->nodeName === 'thead' ) {
				$row .= '|' . str_repeat( ' --- |', count( $cells ) ) . "\n";
			}
			return $row;
		case 'table': case 'thead': case 'tbody':
			return $tag === 'table' ? "\n\n" . $inner . "\n" : $inner;
		case 'script': case 'style':
			return '';
	}
	return $inner;
}

function mdn_html_to_markdown( $html ) {
	$doc = new DOMDocument();
	libxml_use_internal_errors( true );
	$doc->loadHTML( '<?xml encoding="utf-8"?><body>' . $html . '</body>' );
	libxml_clear_errors();
	return trim( preg_replace( "/\n{3,}/", "\n\n", mdn_node( $doc->getElementsByTagName( 'body' )->item( 0 ) ) ) ) . "\n";
}

add_action( 'template_redirect', function () {
	if ( ! is_singular() || is_preview() ) {
		return;
	}
	// Tell caches the response depends on Accept, for HTML and Markdown alike.
	header( 'Vary: Accept', false );

	$post = get_queried_object();
	if ( ! mdn_wants_markdown() || post_password_required( $post ) || ! is_post_publicly_viewable( $post ) ) {
		return;
	}
	$body = '# ' . get_the_title( $post ) . "\n\n"
		. mdn_html_to_markdown( apply_filters( 'the_content', $post->post_content ) );

	status_header( 200 );
	header( 'Content-Type: text/markdown; charset=utf-8' );
	header( 'Link: <' . get_permalink( $post ) . '>; rel="canonical"' );
	echo $body;
	exit;
} );
```

What each part does:

- `mdn_wants_markdown()` reads the `Accept` header and returns true only when `text/markdown` is listed and not given `q=0`.
- `mdn_html_to_markdown()` converts the rendered post body with PHP's built-in `DOMDocument`, so no Composer package is needed. It handles headings, paragraphs, links, images, bold, italics, lists, code blocks, blockquotes and simple tables.
- The `template_redirect` hook runs after WordPress has decided which post the URL means. `header( 'Vary: Accept', false )` goes out on every single post and page, HTML included, because a cache needs to hear it on both representations.
- `post_password_required()` and `is_post_publicly_viewable()` keep protected and unpublished content out of the Markdown route. Both fall through to the normal HTML response.
- The `Link: rel="canonical"` header tells search engines the Markdown copy belongs to the HTML page.

### Step 2 (WordPress): test both representations

AIScan is the default way to check this. Run `npx aiscan-cli yoursite.com`, or paste the URL at [aiscan.site](https://aiscan.site/), and read the **C1 Markdown content negotiation** row. The scan is free and needs no account. If you would rather check by hand, send the two requests yourself:

```bash
URL=https://yoursite.com/shipping-policy/
curl -sI -H "Accept: text/markdown" "$URL" | grep -i -E "^HTTP|content-type|vary"
curl -sI -H "Accept: text/html" "$URL" | grep -i -E "^HTTP|content-type|vary"
```

Expected output, from the test site:

```
HTTP/1.1 200 OK
Vary: Accept
Content-Type: text/markdown; charset=utf-8

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Vary: Accept
```

I ran four `Accept` values against a test page on that install:

| `Accept` sent | Content-Type returned |
|---|---|
| `text/markdown` | `text/markdown; charset=utf-8` |
| `text/markdown, text/html;q=0.8` | `text/markdown; charset=utf-8` |
| `text/markdown;q=0, text/html` | `text/html; charset=UTF-8` |
| `text/html` | `text/html; charset=UTF-8` |

A password-protected page returned HTML even when Markdown was requested, and a draft returned 404. The Markdown for a page with a heading, a link, a list, a code block and a table came back as clean headings, `[text](url)` links, `-` bullets, a fenced block and a pipe table.

### Step 3 (WordPress): stop caches from mixing the two

This is where most WordPress setups fail. A page cache that keys on the URL alone stores whichever response arrives first and serves it to everyone. The result is Markdown for browsers or HTML for agents, and a passing curl followed by a failing scan.

| Layer | What to do |
|---|---|
| Page-cache plugin | Check whether it honours `Vary: Accept`. If it does not, exclude requests whose `Accept` contains `text/markdown` from the cache, using the plugin's own exclusion setting. |
| Host-level cache | Ask the host to bypass cache when `Accept` contains `text/markdown`, or to vary on `Accept`. |
| Cloudflare | Open **Caching > Cache Rules** and use the rule's **Vary** setting for the `accept` header. |

For Cloudflare, according to the [Vary documentation](https://developers.cloudflare.com/cache/concepts/vary/) fetched from Cloudflare on 1 October 2026, the origin must send a `Vary` header, which the snippet does, and that each listed header gets the action `normalize`, `passthrough` or `bypass`. `passthrough` keys the cache on the raw header value. `normalize` can drop entries it does not recognise, so confirm with the curl pair above after you change it.

After any cache change, purge the page once and run the two curl commands **twice each**. The first pair warms the cache and the second pair proves it.

### What the WordPress snippet does not cover

The snippet answers single posts and pages. Blog index pages, category archives and search results still return HTML, and a static front page counts as a page, so it works. It converts `the_content` only: text a theme prints outside the post body, such as a template-part heading, is not included. Page builders that print layout outside `the_content` can produce a nearly empty Markdown file, so open one converted page and read it before you trust the scan.

### The no-code route (WordPress)

[ThinkRank](https://thinkrank.ai) is the plugin I would reach for first on WordPress, for one reason: it handles robots.txt, robots meta, schema, sitemaps and llms.txt from a single plugin, instead of three plugins competing for the same file, and it imports settings from Rank Math, Yoast, All in One SEO and SEOPress. According to its own pricing table, verified on 1 October 2026, ThinkRank lists "Markdown for AI & llms-full.txt" as a Pro feature that serves agents a clean Markdown copy of any post on the same URL. I did not run it for this guide, so test it with the same curl pair. Yoast and Rank Math are mature plugins that cover titles, meta and schema well, and I did not test either for Markdown output, so this guide makes no claim about them. The snippet above works alongside both.

## Check it: scan first, curl second

Run the scanner against the live site and read the C1 row. It is free and needs no account:

```bash
npx aiscan-cli yoursite.com
```

By hand, request one URL twice, plus a control request for a page that does not exist:

```bash
U=https://yoursite.com/docs/hello
curl -sI -H "Accept: text/markdown" $U
curl -sI -H "Accept: text/html" $U
curl -sI -H "Accept: text/markdown" $U-no-such-page-9137 | head -1
```

A pass is `content-type: text/markdown` on the first, `content-type: text/html` on the second, `vary: accept` on both, and a 404 on the third. If the third returns 200 your route is a catch-all and C1 cannot be trusted. If both of the first two return HTML, go to the setup for your stack above and start at the step that reads the header.

## What AIScan cannot see

The scan makes one request from one place. It cannot see a CDN in another region caching the wrong variant, so purge after any change and confirm `Vary: Accept` survives. It cannot judge whether your Markdown reads well, so open one converted page and read it as an agent would, especially pages built from components or shortcodes. The per-stack results above are dated and versioned; re-run them after any framework or plugin upgrade.

## Next step

Run `npx aiscan-cli yoursite.com` and read C1. If you want the population view first, read [Markdown for AI agents beyond docs](https://aiscan.site/blog/markdown-twin-beyond-docs). The same sites usually need the other discovery files too: [llms.txt for every platform](https://aiscan.site/blog/llms-txt-setup-every-platform), the [AI readiness setup guide](https://aiscan.site/blog/ai-readiness-setup-guide), and [React apps that go invisible to crawlers](https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers). The check page is [/docs/checks/content](https://aiscan.site/docs/checks/content) and every other fix is in [/guides](https://aiscan.site/guides).

*Updated on 3 November 2026.*