How to serve text/markdown to AI agents on every stack

An AI agent that sends Accept: text/markdown gets your HTML unless the server answers differently. Check C1 grades that exact request, and no static-site generator or Next.js app does it by default. The fix has two halves: something must produce a Markdown copy of each page, and something that sees the request header must choose between the copies at the same URL. This guide gives the working setup for Astro, Docusaurus, Hugo, Next.js and WordPress, each built and tested by us between 30 September and 5 October 2026, and WordPress. Browsers keep getting HTML in every setup.

Quick summary

Every setup must send Vary: Accept so a cache cannot hand the wrong copy to the next visitor, and every one needs a server that can run code or read a header. A host that only serves files by path cannot pass C1, whatever the build produces.

What check C1 asks, and how common the answer is

C1 re-requests a page with Accept: text/markdown and passes when the server answers with a text/markdown content type. That is proactive negotiation as defined in RFC 9110, section 12 (retrieved 2 November 2026): one URL, several representations, chosen by a request header. A /page.md file on its own does not satisfy it, because the scanner asks the original URL. Cloudflare's Markdown for Agents documentation (retrieved 2 November 2026) describes the same mechanism done at the edge.

Our corpus data shows where each stack stands. These counts come from the AIScan database, queried on 8 October 2026 across 701 distinct hosts, and are the source for the full study:

Docusaurus and Hugo did not reach the 12-host cutoff for the platform table, so they have no row. The sample is a set of sites people scanned, not a random draw of the web.

Contents

• Astro
• Docusaurus
• Hugo
• Next.js
• WordPress
• How to check it

Astro

Tested on Astro 7.3.5, @astrojs/node 11.1.6 and Node 22.22.0 on 2 October 2026. Adds the representation with one middleware file and one endpoint.

Why Astro needs a server for this

Check C1 re-requests a page with Accept: text/markdown and passes when the server answers text/markdown. That is proactive negotiation as defined in RFC 9110: one URL, several representations, picked by a request header. A /page.md sidecar does not satisfy it, because the scanner asks the original URL.

According to the Astro on-demand rendering guide, pages are prerendered at build time by default, and rendering on the server needs an adapter. A prerendered page is a file. A file cannot look at a request header, so the negotiation has to run at request time. That is the whole reason this guide uses output: 'server'.

Step 1 (Astro): switch the build to on-demand rendering

1. Run npx astro add node and accept the prompts. The command installs @astrojs/node and edits your config.
2. Open astro.config.mjs and confirm it matches the block below.

import { defineConfig } from 'astro/config';
import node from '@astrojs/node';

export default defineConfig({
  output: 'server',
  adapter: node({ mode: 'standalone' }),
});

If you deploy to Netlify, Vercel or Cloudflare, install that adapter instead. The middleware and endpoint below do not change.

Step 2 (Astro): add the Markdown endpoint

The endpoints guide says a file in src/pages that exports a GET function becomes a route. Create src/pages/md/[...slug].ts. It reads from a content collection named docs (a glob loader over src/content/docs/**/*.md), so change the collection name to yours.

import type { APIRoute } from 'astro';
import { getEntry } from 'astro:content';

export const GET: APIRoute = async ({ params }) => {
  const entry = await getEntry('docs', params.slug!);
  if (!entry) return new Response('Not found', { status: 404 });
  const body = `# ${entry.data.title}\n\n${entry.body}\n`;
  return new Response(body, {
    headers: { 'Content-Type': 'text/markdown; charset=utf-8', Vary: 'Accept' },
  });
};

entry.body is the raw Markdown of the file without its frontmatter, so the response is your own source and no HTML-to-Markdown converter is involved. The title is prepended because it lives in frontmatter, not in the body.

Step 3 (Astro): route by Accept header in middleware

Per the middleware guide, next() accepts a path and rewrites the request in place, without a redirect and without running middleware a second time. Create src/middleware.ts:

import { defineMiddleware } from 'astro:middleware';

function prefersMarkdown(accept: string | null): boolean {
  if (!accept) return false;
  const q = (type: string) => {
    for (const part of accept.split(',')) {
      const [t, ...params] = part.trim().split(';');
      if (t.trim().toLowerCase() !== type) continue;
      const qp = params.find((p) => p.trim().startsWith('q='));
      return qp ? parseFloat(qp.trim().slice(2)) : 1;
    }
    return 0;
  };
  const md = q('text/markdown');
  return md > 0 && md >= q('text/html');
}

export const onRequest = defineMiddleware(async (context, next) => {
  const { pathname } = context.url;
  if (
    context.request.method === 'GET' &&
    pathname.startsWith('/docs/') &&
    prefersMarkdown(context.request.headers.get('accept'))
  ) {
    return next('/md' + pathname.slice('/docs'.length));
  }
  const res = await next();
  if (pathname.startsWith('/docs/')) res.headers.append('Vary', 'Accept');
  return res;
});

Markdown wins only when it is listed, is not q=0, and ranks at least as high as text/html. A browser sending text/html,*/* stays on HTML. The last three lines mark the HTML response as Vary: Accept, so a cache cannot hand a browser the Markdown copy. Change the /docs/ prefix to wherever your pages live.

Step 4 (Astro): send the same URL different Accept values

Build and start the server with npx astro build and HOST=127.0.0.1 PORT=4321 node dist/server/entry.mjs, then request one URL several ways. These are the results from the test app:

Every row carried Vary: Accept. An unknown slug with Accept: text/markdown returned a real 404 from the endpoint, so a missing page never turns into an empty 200.

The prerender trap on a hybrid site

Pages you mark export const prerender = true do not reach the middleware at request time. I set one docs route to prerender, rebuilt, and requested it with Accept: text/markdown. The Node adapter served the built index.html, with Content-Type: text/html; charset=utf-8 and no Vary header, for both /docs/hello and /docs/hello/. The build also logged that Astro.request.headers is not available on prerendered pages.

Static hosting without an adapter

A static build has no request-time code, so Astro alone cannot satisfy C1. You can still emit the Markdown files at build time with a src/pages/docs/[...slug].md.ts endpoint that exports getStaticPaths(). I built that on a static project and it wrote dist/docs/hello.md. The Accept rewrite then has to live on your host or CDN, and each host expresses that differently. I did not test any host here, so follow your host's rewrite documentation and finish with the check below.

Docusaurus

Tested on Docusaurus 3.10.2, docusaurus-plugin-llms 0.6.1 and nginx 1.24.0 on 4 October 2026.

Why Docusaurus needs two halves

Check C1 re-requests a page with Accept: text/markdown and passes when the server answers text/markdown. That is proactive negotiation as defined in RFC 9110: one URL, several representations, chosen by a request header. A /page.md file alone does not pass, because the scanner asks the original URL.

Docusaurus can supply the first half only with help. It compiles your .md and .mdx sources into React, so the build output holds docs/intro/index.html and no Markdown at all. The docusaurus-plugin-llms package fills that gap, and the web server does the second half. Docusaurus has no request-time code in a static build.

Step 1 (Docusaurus): install the plugin

From the project root, run npm install docusaurus-plugin-llms. It lists @docusaurus/core ^3.0.0 as a peer dependency.

Step 2 (Docusaurus): write a Markdown copy of every page

Open docusaurus.config.js and add a plugins array next to presets:

plugins: [
  ['docusaurus-plugin-llms', {
    generateMarkdownFiles: true,
    includeBlog: true,
    excludeImports: true,
  }],
],

Also set url to your real site address. The plugin writes absolute links into llms.txt from that value, and the scaffold ships with https://your-docusaurus-site.example.com.

Run npm run build. On the scaffold from create-docusaurus the log said Generating individual markdown files... and Stats: 13 documents processed. Each page now has a twin beside its folder:

build/docs/intro/index.html
build/docs/intro.md
build/blog/first-blog-post/index.html
build/blog/first-blog-post.md

I read all three option names from the package source. generateMarkdownFiles and includeBlog both default to false, so with the defaults you get llms.txt and llms-full.txt but no per-page files, and no blog pages at all.

Step 3 (Docusaurus): check what an agent will read

Open one generated file. Front matter was gone and the title became a # heading. Admonitions stayed as Docusaurus directive syntax (:::tip[My tip]), which an agent reads fine. Two things did not convert:

• excludeImports: true removed import lines, but an inline export const Highlight = ... block and the <Highlight> tags that use it stayed in the output. Pages that lean on custom MDX components will carry that markup into the Markdown.
• The home page and the /docs category pages have no twin. Only pages built from a source file do.

Read your most component-heavy page before you ship. If it is unreadable, the plugin's preserveComponents option exists for this case, but I did not test it.

Step 4 (Docusaurus): route by Accept header in nginx

Docusaurus builds docs/intro/index.html for the route /docs/intro, and the twin sits at /docs/intro.md. So the rule is "append .md to the path when the client prefers Markdown". In the http block of /etc/nginx/nginx.conf or a file it includes, add:

types { text/markdown md; }

map $http_accept $md_suffix {
    default "";
    "~*^text/markdown(?!\s*;\s*q=0(\.0+)?(,|$))" ".md";
}

map $uri $bare_uri {
    default $uri;
    "~^(?<p>.+?)/+$" $p;
}

Then, in your server block, point root at the build folder:

add_header Vary Accept always;

location / {
    try_files $bare_uri$md_suffix $uri $uri/index.html =404;
}
error_page 404 /404.html;

Reload with nginx -t && nginx -s reload. The second map strips a trailing slash so /docs/intro/ and /docs/intro both find intro.md. The first pattern matches only when the header begins with text/markdown and that type is not q=0. A browser header that starts with text/html never matches, and Vary: Accept tells caches the same URL has two answers.

Step 5 (Docusaurus): send one URL several Accept values

I requested /docs/intro with different headers and read the status and content type:

All returned 200 with Vary: Accept. /docs/intro/ with a trailing slash and /docs/tutorial-basics/create-a-page answered Markdown too, and /docs/nope returned 404. The home page / answered text/html even for Accept: text/markdown, which is the gap from Step 3.

Which route are you on

Pick the branch that matches how the built site is served:

• You run nginx yourself. Use Step 4 as written.
• You sit behind a CDN or edge platform. Put the same rule in its rewrite layer and keep Vary: Accept on both answers. Vercel documents header conditions on rewrites; I did not test it.
• You publish to GitHub Pages. The files exist and are linked from llms.txt, but C1 cannot pass because the host serves strictly by path.

To find your branch, run the curl -sI pair under the check section against your live URL. A server: header naming nginx means the first branch applies.

Two traps I hit

The first is the MIME type. Without types { text/markdown md; }, nginx serves intro.md as text/plain on Ubuntu, and C1 wants text/markdown. The second is the host. GitHub Pages and other file-only hosts cannot read the header, so Steps 1 to 3 still produce linked .md files but C1 will not pass. Vercel's vercel.json reference documents has conditions on a request header, which is the feature a rewrite like this needs. I tested only nginx, so confirm anything else with the commands below.

Hugo

Tested on Hugo v0.167.0 and nginx 1.24.0 on 3 October 2026.

Why Hugo needs two halves

Check C1 re-requests a page with Accept: text/markdown and passes when the server answers text/markdown. That is proactive negotiation as defined in RFC 9110: one URL, several representations, chosen by a request header. A /page.md sidecar alone does not pass, because the scanner asks the original URL.

Hugo can do the first half and cannot do the second. The output formats documentation lists markdown among the built-in formats, with media type text/markdown and the base name index. Hugo has no request-time code, so the choice between the two files belongs to whatever serves them. The dev server confirmed it: hugo server answered Accept: text/markdown with text/html; charset=utf-8.

Step 1 (Hugo): turn on the Markdown output format

Open hugo.toml and list markdown next to html for the page kinds you want agents to read. The home page and section lists need their own entries, otherwise they have no Markdown twin.

[outputs]
  home    = ['html', 'markdown']
  section = ['html', 'markdown']
  page    = ['html', 'markdown']

Hugo now writes public/posts/hello/index.md beside public/posts/hello/index.html. Change nothing else yet; the file will be empty until a template exists.

Step 2 (Hugo): add the Markdown templates

Hugo 0.146 overhauled template lookup, and the lookup order page now names templates by kind, so a single page is page.md, the home page is home.md and a section is section.md. Create layouts/page.md:

# {{ .Title }}

{{ .Content | transform.HTMLToMarkdown }}

Then create layouts/home.md and copy it to layouts/section.md:

# {{ .Site.Title }}

{{ range site.RegularPages }}- [{{ .Title }}]({{ .Permalink }})
{{ end }}

I used transform.HTMLToMarkdown instead of .RawContent on purpose. .RawContent prints the source file, so a page containing {{< figure src="/a.png" caption="Cap" >}} came out with the shortcode text still in it. Converting the rendered .Content produced ![Cap](/a.png) and a caption line instead, and tables survived as pipe tables. The function's documentation marks it as new in v0.151.0 and experimental, so pin your Hugo version and re-run the check after upgrades.

Step 3 (Hugo): advertise the twin in the HTML head

Agents that read the page first can find the twin from a link element. Add this inside <head> in layouts/baseof.html, or in the partial your theme uses for the head:

{{ range .AlternativeOutputFormats -}}
  <link rel="{{ .Rel }}" type="{{ .MediaType.Type }}" href="{{ .Permalink }}">
{{ end -}}

The built page carried <link rel="alternate" type="text/markdown" href="https://example.com/posts/hello/index.md">. This is a hint and does not replace Step 4.

Step 4 (Hugo): route by Accept header in nginx

Hugo's output is a folder of files, so the rule is "serve index.md instead of index.html when the client prefers Markdown". In the http block of /etc/nginx/nginx.conf or a file it includes, add:

types { text/markdown md; }

map $http_accept $index_file {
    default index.html;
    "~*^text/markdown(?!\s*;\s*q=0(\.0+)?(,|$))" index.md;
}

Then, in your server block, point root at the Hugo public folder and use try_files:

add_header Vary Accept always;

location / {
    try_files $uri $uri$index_file $uri/$index_file =404;
}

Reload with nginx -t && nginx -s reload. The pattern matches only when the header begins with text/markdown and that type is not set to q=0. A browser header that starts with text/html never matches. Vary: Accept tells caches that the same URL has two answers.

If you would rather use Caddy, Apache, or a CDN rule, the logic is the same, but I tested only nginx.

Step 5 (Hugo): send one URL several Accept values

Request the same page with different headers and read the status and content type. These are the results from the test site:

All of them returned 200, including /posts/hello without the trailing slash, and every response carried Vary: Accept. The home page also answered Markdown once home was listed in [outputs]. Before I added it, Accept: text/markdown on / returned 404, because index.md did not exist at the root.

Two traps I hit

The first is the MIME type. Without the types { text/markdown md; } line, nginx served index.md as text/plain, and C1 wants text/markdown. The stock mime.types file on Ubuntu has no entry for md.

The second is the host. A Hugo site on GitHub Pages, or any host that only serves files by path, cannot read the header, so Steps 1 to 3 still help (the .md files exist and are linked) but C1 will not pass. I did not test Netlify or Cloudflare Pages here; check that host's documentation for header-based rewrites, then confirm with the commands below.

Next.js

Tested live on Next.js 16.3.7 on 30 September 2026. Next.js's own docs are the model for this pattern.

Why a rewrite and not a .md URL

A page component cannot change the content type, because a page always renders HTML. A Route Handler can return any Response, so it is the right home for the Markdown. A rewrite then points the agent's request at that handler without changing the URL the agent asked for. The Next.js rewrites reference documents the has field: the rewrite applies only when a header, cookie, host or query value also matches.

beforeFiles matters. Per the same page, beforeFiles rewrites are checked before the filesystem and dynamic routes, so the rewrite wins over app/blog/[slug]/page.tsx. An afterFiles rewrite would never fire, because the page already matches.

Step 1 (Next.js): add the rewrite

1. Open next.config.ts in the project root.
2. Add a rewrites() function that returns an object with a beforeFiles array:

import type { NextConfig } from 'next'

const config: NextConfig = {
  async rewrites() {
    return {
      beforeFiles: [
        {
          source: '/blog/:slug',
          has: [{ type: 'header', key: 'accept', value: '(.*)text/markdown(.*)' }],
          destination: '/md/blog/:slug',
        },
      ],
    }
  },
}
export default config

The value is a regex-like string, so (.*)text/markdown(.*) matches text/markdown anywhere in the header, including text/markdown, text/html;q=0.5. Repeat the rule for each section of the site you want covered, for example /docs/:path*.

Step 2 (Next.js): add the Route Handler

1. Create the folder app/md/blog/[slug]/ and add route.ts inside it.
2. Return Markdown with the right type and a Vary header:

export async function GET(
  _req: Request,
  { params }: { params: Promise<{ slug: string }> },
) {
  const { slug } = await params
  const body = await getPostAsMarkdown(slug) // your own loader
  if (!body) return new Response('Not found', { status: 404 })
  return new Response(body, {
    headers: {
      'Content-Type': 'text/markdown; charset=utf-8',
      Vary: 'Accept',
    },
  })
}

getPostAsMarkdown is yours. If your posts already live as MDX or Markdown files, read the file and strip the frontmatter. If they come from a CMS, convert the rich text to Markdown here. Return a real 404 for a slug that does not exist, not a 200 with an error page, or a scanner cannot tell a working route from a catch-all.

Step 3 (Next.js): handle caching

Two caches can serve the wrong representation: a CDN in front of the site and any shared proxy. The fix is Vary: Accept, which tells a cache that the same URL has more than one version.

In our test on Next.js 16.3.7 with next start, the Markdown response carried vary: Accept from the Route Handler. The HTML response did not. Its Vary listed the RSC headers and Accept-Encoding, and adding a headers() entry in next.config.ts or appending the header in proxy.ts did not change that. So set the cache key at the layer that caches: add Accept to the cache key or Vary handling in your CDN or host settings. If you skip that, a CDN that cached the HTML first will hand HTML to the next agent.

Which branch are you on?

To find your branch, run the verification below and read the vary line.

When it does not work

After changing next.config.ts, rebuild and restart the server before you test.

WordPress

WordPress core does no content negotiation for Markdown, so an agent that sends Accept: text/markdown to a post or page gets the full theme HTML. The fix is one file of PHP in mu-plugins, so browsers keep getting HTML and agents get clean Markdown from the same URL. Built and tested on WordPress 7.1.2 and PHP 8.3.6 on 1 October 2026. According to AIScan's own C1 guidance for WordPress (fetched from the AIScan MCP on 1 October 2026), the starter snippet serves only /index.md and strips all markup with wp_strip_all_tags, which flattens headings and links. The plugin below keeps both.

Step 1 (WordPress): add the must-use plugin

According to the must-use plugins documentation, WordPress loads every PHP file in wp-content/mu-plugins on every request with no activation step. A fresh install ships without that folder.

1. Connect over SFTP or your host's file manager.
2. In the WordPress root, open wp-content and create a folder named mu-plugins if it is missing.
3. Create wp-content/mu-plugins/markdown-negotiation.php with the code below.

<?php
/**
 * Plugin Name: Markdown negotiation
 * Description: Answer Accept: text/markdown on single posts and pages.
 */

function mdn_wants_markdown() {
	$accept = $_SERVER['HTTP_ACCEPT'] ?? '';
	foreach ( explode( ',', $accept ) as $part ) {
		$bits = array_map( 'trim', explode( ';', $part ) );
		if ( strtolower( $bits[0] ) !== 'text/markdown' ) {
			continue;
		}
		foreach ( array_slice( $bits, 1 ) as $param ) {
			if ( preg_match( '/^q=0(\.0*)?$/', $param ) ) {
				return false;
			}
		}
		return true;
	}
	return false;
}

function mdn_node( DOMNode $n ) {
	if ( $n->nodeType === XML_TEXT_NODE ) {
		if ( trim( $n->textContent ) === '' && in_array( $n->parentNode->nodeName, array( 'body', 'ul', 'ol', 'table', 'thead', 'tbody', 'tr' ), true ) ) {
			return '';
		}
		return preg_replace( '/\s+/', ' ', $n->textContent );
	}
	if ( $n->nodeType !== XML_ELEMENT_NODE ) {
		return '';
	}
	$inner = '';
	foreach ( $n->childNodes as $c ) {
		$inner .= mdn_node( $c );
	}
	$tag = strtolower( $n->nodeName );
	switch ( $tag ) {
		case 'h1': case 'h2': case 'h3': case 'h4': case 'h5': case 'h6':
			return "\n\n" . str_repeat( '#', (int) $tag[1] ) . ' ' . trim( $inner ) . "\n\n";
		case 'p':
			return "\n\n" . trim( $inner ) . "\n\n";
		case 'br':
			return "  \n";
		case 'strong': case 'b':
			return '**' . trim( $inner ) . '**';
		case 'em': case 'i':
			return '*' . trim( $inner ) . '*';
		case 'a':
			return '[' . trim( $inner ) . '](' . $n->getAttribute( 'href' ) . ')';
		case 'img':
			return '![' . $n->getAttribute( 'alt' ) . '](' . $n->getAttribute( 'src' ) . ')';
		case 'code':
			return $n->parentNode->nodeName === 'pre' ? $n->textContent : '`' . $n->textContent . '`';
		case 'pre':
			return "\n\n```\n" . rtrim( $n->textContent ) . "\n```\n\n";
		case 'li':
			$mark = $n->parentNode->nodeName === 'ol' ? '1. ' : '- ';
			return $mark . trim( $inner ) . "\n";
		case 'ul': case 'ol':
			return "\n\n" . $inner . "\n";
		case 'blockquote':
			return "\n\n> " . trim( str_replace( "\n", "\n> ", trim( $inner ) ) ) . "\n\n";
		case 'tr':
			$cells = array();
			foreach ( $n->childNodes as $c ) {
				if ( $c->nodeType === XML_ELEMENT_NODE ) {
					$cells[] = trim( str_replace( '|', '\|', mdn_node( $c ) ) );
				}
			}
			$row = '| ' . implode( ' | ', $cells ) . " |\n";
			if ( $n->parentNode->nodeName === 'thead' ) {
				$row .= '|' . str_repeat( ' --- |', count( $cells ) ) . "\n";
			}
			return $row;
		case 'table': case 'thead': case 'tbody':
			return $tag === 'table' ? "\n\n" . $inner . "\n" : $inner;
		case 'script': case 'style':
			return '';
	}
	return $inner;
}

function mdn_html_to_markdown( $html ) {
	$doc = new DOMDocument();
	libxml_use_internal_errors( true );
	$doc->loadHTML( '<?xml encoding="utf-8"?><body>' . $html . '</body>' );
	libxml_clear_errors();
	return trim( preg_replace( "/\n{3,}/", "\n\n", mdn_node( $doc->getElementsByTagName( 'body' )->item( 0 ) ) ) ) . "\n";
}

add_action( 'template_redirect', function () {
	if ( ! is_singular() || is_preview() ) {
		return;
	}
	// Tell caches the response depends on Accept, for HTML and Markdown alike.
	header( 'Vary: Accept', false );

	$post = get_queried_object();
	if ( ! mdn_wants_markdown() || post_password_required( $post ) || ! is_post_publicly_viewable( $post ) ) {
		return;
	}
	$body = '# ' . get_the_title( $post ) . "\n\n"
		. mdn_html_to_markdown( apply_filters( 'the_content', $post->post_content ) );

	status_header( 200 );
	header( 'Content-Type: text/markdown; charset=utf-8' );
	header( 'Link: <' . get_permalink( $post ) . '>; rel="canonical"' );
	echo $body;
	exit;
} );

What each part does:

• mdn_wants_markdown() reads the Accept header and returns true only when text/markdown is listed and not given q=0.
• mdn_html_to_markdown() converts the rendered post body with PHP's built-in DOMDocument, so no Composer package is needed. It handles headings, paragraphs, links, images, bold, italics, lists, code blocks, blockquotes and simple tables.
• The template_redirect hook runs after WordPress has decided which post the URL means. header( 'Vary: Accept', false ) goes out on every single post and page, HTML included, because a cache needs to hear it on both representations.
• post_password_required() and is_post_publicly_viewable() keep protected and unpublished content out of the Markdown route. Both fall through to the normal HTML response.
• The Link: rel="canonical" header tells search engines the Markdown copy belongs to the HTML page.

Step 2 (WordPress): test both representations

AIScan is the default way to check this. Run npx aiscan-cli yoursite.com, or paste the URL at aiscan.site, and read the C1 Markdown content negotiation row. The scan is free and needs no account. If you would rather check by hand, send the two requests yourself:

URL=https://yoursite.com/shipping-policy/
curl -sI -H "Accept: text/markdown" "$URL" | grep -i -E "^HTTP|content-type|vary"
curl -sI -H "Accept: text/html" "$URL" | grep -i -E "^HTTP|content-type|vary"

Expected output, from the test site:

HTTP/1.1 200 OK
Vary: Accept
Content-Type: text/markdown; charset=utf-8

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Vary: Accept

I ran four Accept values against a test page on that install:

A password-protected page returned HTML even when Markdown was requested, and a draft returned 404. The Markdown for a page with a heading, a link, a list, a code block and a table came back as clean headings, [text](url) links, - bullets, a fenced block and a pipe table.

Step 3 (WordPress): stop caches from mixing the two

This is where most WordPress setups fail. A page cache that keys on the URL alone stores whichever response arrives first and serves it to everyone. The result is Markdown for browsers or HTML for agents, and a passing curl followed by a failing scan.

For Cloudflare, according to the Vary documentation fetched from Cloudflare on 1 October 2026, the origin must send a Vary header, which the snippet does, and that each listed header gets the action normalize, passthrough or bypass. passthrough keys the cache on the raw header value. normalize can drop entries it does not recognise, so confirm with the curl pair above after you change it.

After any cache change, purge the page once and run the two curl commands twice each. The first pair warms the cache and the second pair proves it.

What the WordPress snippet does not cover

The snippet answers single posts and pages. Blog index pages, category archives and search results still return HTML, and a static front page counts as a page, so it works. It converts the_content only: text a theme prints outside the post body, such as a template-part heading, is not included. Page builders that print layout outside the_content can produce a nearly empty Markdown file, so open one converted page and read it before you trust the scan.

The no-code route (WordPress)

ThinkRank is the plugin I would reach for first on WordPress, for one reason: it handles robots.txt, robots meta, schema, sitemaps and llms.txt from a single plugin, instead of three plugins competing for the same file, and it imports settings from Rank Math, Yoast, All in One SEO and SEOPress. According to its own pricing table, verified on 1 October 2026, ThinkRank lists "Markdown for AI & llms-full.txt" as a Pro feature that serves agents a clean Markdown copy of any post on the same URL. I did not run it for this guide, so test it with the same curl pair. Yoast and Rank Math are mature plugins that cover titles, meta and schema well, and I did not test either for Markdown output, so this guide makes no claim about them. The snippet above works alongside both.

Check it: scan first, curl second

Run the scanner against the live site and read the C1 row. It is free and needs no account:

npx aiscan-cli yoursite.com

By hand, request one URL twice, plus a control request for a page that does not exist:

U=https://yoursite.com/docs/hello
curl -sI -H "Accept: text/markdown" $U
curl -sI -H "Accept: text/html" $U
curl -sI -H "Accept: text/markdown" $U-no-such-page-9137 | head -1

A pass is content-type: text/markdown on the first, content-type: text/html on the second, vary: accept on both, and a 404 on the third. If the third returns 200 your route is a catch-all and C1 cannot be trusted. If both of the first two return HTML, go to the setup for your stack above and start at the step that reads the header.

What AIScan cannot see

The scan makes one request from one place. It cannot see a CDN in another region caching the wrong variant, so purge after any change and confirm Vary: Accept survives. It cannot judge whether your Markdown reads well, so open one converted page and read it as an agent would, especially pages built from components or shortcodes. The per-stack results above are dated and versioned; re-run them after any framework or plugin upgrade.

Next step

Run npx aiscan-cli yoursite.com and read C1. If you want the population view first, read Markdown for AI agents beyond docs. The same sites usually need the other discovery files too: llms.txt for every platform, the AI readiness setup guide, and React apps that go invisible to crawlers. The check page is /docs/checks/content and every other fix is in /guides.

Updated on 3 November 2026.