Set X-Robots-Tag or a Canonical Link Header on .md Copies and llms.txt: Nginx, Apache, Cloudflare, Vercel, Netlify, Next.js, Astro

Send one header on each .md copy and nothing new on the HTML: Link: <https://example.com/page>; rel="canonical". On /llms.txt, which has no HTML twin, the only header that applies is X-Robots-Tag: noindex, and you only need it if that file is showing up in search. We set both on Nginx 1.24.0, Apache 2.4.58, Next.js 16.4.6 and Astro 7.4.11 on 13 November 2026 and pasted every response below, with the config for Cloudflare, Vercel and Netlify from their own docs.

Quick summary

If you are still deciding whether to do this at all, read Should Google index your llms.txt and .md copies? first. It has our Search Console data: on 13 November 2026, 15 of 15 machine-readable copies on aiscan.site were "URL is unknown to Google". This post is the where-to-type-it half.

Why canonical beats noindex for a copy

According to Google's guide to consolidating duplicate URLs (updated 10 July 2026, verified on 13 November 2026), a rel="canonical" HTTP header is the documented way to point a non-HTML file at its preferred URL, and it uses an absolute URL. The same page says, in its own words: "We don't recommend using noindex to prevent selection of a canonical page within a single site, because it will completely block the page from Search."

A canonical header says "this is a copy, rank the HTML". A noindex says "drop this file". The copy stays reachable either way, so agents that fetch /page.md or /llms.txt directly still get it. One caution: noindex also tells any AI crawler that honours robots directives to drop the file from its index. Use it only on a file that is causing a real problem in search, and never send both headers on the same URL, because they contradict each other.

According to Google's robots meta tag and X-Robots-Tag page (updated 24 March 2026, verified on 13 November 2026), X-Robots-Tag is the way to set noindex on non-HTML files, and the page lists Apache and Nginx examples.

What we measured

We served a pricing.md, a nested blog/first-post.md and an llms.txt from each stack, then read the response headers with a short script. Each row is one HEAD request on 13 November 2026.

Three things went wrong on the way, and each is a trap you will hit too:

1. Nginx dropped the charset. charset utf-8; did nothing for .md until we added text/markdown to charset_types, because the charset module only applies to the types in that list.
2. Nginx add_header wiped a server-level header. A nosniff header set on the server block disappeared from .md and llms.txt once their location blocks had their own add_header. The headers module inherits add_header from the parent level only when the current level defines none.
3. Astro middleware never touched public/. Our middleware set both headers on rendered routes, but files copied from public/ came back with neither. Serve those files from an endpoint, or set the header at the host.

The script we used to read the results:

import sys, urllib.request
for url in sys.argv[1:]:
    with urllib.request.urlopen(urllib.request.Request(url, method='HEAD')) as r:
        h = r.headers
        print(url, r.status, h.get('Content-Type'),
              'x-robots-tag=' + (h.get('X-Robots-Tag') or '-'),
              'link=' + (h.get('Link') or '-'))

Which section is yours

Action: run curl -sI https://yoursite.com/llms.txt and look at the server: line. cloudflare, Netlify and Vercel name the layer that will set the header.

Nginx

1. Open the site's server block (often /etc/nginx/sites-available/<site>).
2. Add the type, the charset list and the map in the http context:

types { text/markdown md; }
charset_types text/markdown text/plain;
map $uri $md_canonical {
  ~^(?<base>.+)\.md$ "<https://example.com$base>; rel=\"canonical\"";
  default "";
}

1. Add the locations inside server, and repeat any server-level add_header lines inside each one:

location ~ \.md$ {
  add_header Link $md_canonical always;
  charset utf-8;
}
location = /llms.txt {
  add_header X-Robots-Tag "noindex" always;
  charset utf-8;
}

1. Run nginx -t, then systemctl reload nginx.

Apache

1. Enable the modules: a2enmod headers setenvif.
2. Add this to the virtual host or .htaccess (the mod_headers docs explain the env= condition):

AddType "text/markdown; charset=utf-8" .md
SetEnvIf Request_URI "^(.+)\.md$" MD_BASE=$1
Header set Link "<https://example.com%{MD_BASE}e>; rel=\"canonical\"" env=MD_BASE
<Files "llms.txt">
  Header set X-Robots-Tag "noindex"
</Files>

1. Run apachectl configtest, then reload.

Cloudflare

Not deployed on a live zone for this post. The steps are fetched from Cloudflare's response header Transform Rule docs and its functions reference, on 13 November 2026.

1. Go to Rules > Overview, select Create rule > Response Header Transform Rule.
2. Name it, choose Custom filter expression and enter ends_with(http.request.uri.path, ".md").
3. Under Modify response header, choose Set dynamic, header name Link, value: concat("<https://", http.host, substring(http.request.uri.path, 0, -3), ">; rel=\"canonical\"")
4. Select Deploy. Create a second rule for http.request.uri.path eq "/llms.txt" with Set static X-Robots-Tag = noindex.

substring() accepts negative indexes, so 0, -3 strips .md. Do not reach for regex_replace(): the docs limit it to rewrite and redirect expressions. If you already turned on Cloudflare's Markdown for Agents, see our rule guide first, because negotiated Markdown has no .md path to match.

Netlify and Cloudflare Pages

Both read a _headers file in the publish folder. According to Cloudflare Pages' headers docs, a value can reuse the matched path as :splat, so one rule covers every copy:

/*.md
  Link: <https://example.com/:splat>; rel="canonical"
/llms.txt
  X-Robots-Tag: noindex

Netlify's headers docs describe wildcards and placeholders in paths only, not in values (checked 13 November 2026). So on Netlify, generate one block per file at build time. We ran this against a test dist/ folder and it wrote correct entries for /pricing.md and /blog/first-post.md:

// node make-headers.mjs dist https://example.com
import { readdirSync, appendFileSync } from 'node:fs';
import { join, relative } from 'node:path';
const [dir, site] = process.argv.slice(2);
const walk = (d) => readdirSync(d, { withFileTypes: true })
  .flatMap((e) => e.isDirectory() ? walk(join(d, e.name)) : [join(d, e.name)]);
let out = '\n/llms.txt\n  X-Robots-Tag: noindex\n';
for (const f of walk(dir).filter((f) => f.endsWith('.md'))) {
  const path = '/' + relative(dir, f).split('\\').join('/');
  out += `\n${path}\n  Link: <${site}${path.slice(0, -3)}>; rel="canonical"\n`;
}
appendFileSync(join(dir, '_headers'), out);

Run it after the build: "build": "astro build && node make-headers.mjs dist https://example.com". Cloudflare Pages caps _headers at 100 rules, which is one more reason to use :splat there. Neither host applies _headers to function or SSR responses.

Next.js

Tested with next build && next start. In next.config.mjs:

const SITE = 'https://example.com';
export default {
  async headers() {
    return [
      { source: '/:path*.md',
        headers: [{ key: 'Link', value: `<${SITE}/:path*>; rel="canonical"` }] },
      { source: '/llms.txt',
        headers: [{ key: 'X-Robots-Tag', value: 'noindex' }] },
    ];
  },
};

According to the Next.js headers docs, matched parameters can be used in the value, and the nested copy came back as <https://example.com/blog/first-post>. This also works for files in public/.

Vercel without Next.js: put the same source and headers objects in vercel.json. Vercel's vercel.json reference does not say whether :path* is substituted in a value, so after the deploy check the Link line. If it comes back with a literal :path*, use one entry per file.

Astro

On a static build, Astro writes your files and the host serves them, so use the _headers or vercel.json section for your host. On a server build, move the copies out of public/ into endpoints and set the headers in middleware:

// src/middleware.js
export async function onRequest(context, next) {
  const res = await next();
  const { pathname } = context.url;
  if (pathname.endsWith('.md')) {
    const html = new URL(pathname.slice(0, -3), context.site);
    res.headers.set('Link', `<${html.href}>; rel="canonical"`);
  }
  if (pathname === '/llms.txt') res.headers.set('X-Robots-Tag', 'noindex');
  return res;
}

Set site in astro.config.mjs. Our src/pages/llms.txt.js endpoint (with export const prerender = false) returned noindex; the same file in public/ did not. Astro's middleware guide covers the hook. Our Markdown for AI agents hub shows how to generate the copies.

WordPress

The copies and llms.txt usually come from a plugin. ThinkRank manages robots meta, robots.txt and llms.txt from one plugin, so the rules for HTML and machine-readable files are not split across tools fighting over the same output, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. The response headers on the files themselves are still set by the web server, so add the Nginx or Apache block above, or ask your host.

Check it worked

The quickest check is a scan. Run npx aiscan-cli yoursite.com or paste your URL at aiscan.site, free and with no account. Checks C1 (Markdown content negotiation) and C2 (/llms.txt) should both stay passing after the change, because a header must never make the files unreachable. The content checks page explains both. AIScan does not score the Link or X-Robots-Tag values themselves, so confirm those by hand:

curl -sI https://yoursite.com/blog/post.md | grep -iE '^(link|x-robots-tag|content-type)'
curl -sI https://yoursite.com/llms.txt | grep -iE '^(x-robots-tag|content-type)'

Pass means: one Link line on every .md, pointing at the absolute HTML URL that returns 200; no noindex on any .md you canonicalised; status 200 on both files. If Content-Type lost its charset, see llms.txt Content-Type and charset. If you have no llms.txt yet, the llms.txt generator writes one from your sitemap.

Our own site, honestly: on 13 November 2026, aiscan.site's .md copies sent x-robots-tag: all and no Link, and /llms.txt sent neither. Since Google had found none of them, we have left it that way.

Maintenance

Re-run the two curl lines after any server, CDN or framework upgrade. If you change your domain, the canonical URL inside the header changes with it, so the Nginx map, the Apache line and the Cloudflare rule all need the new host.

Your next step

Scan your site at aiscan.site and check that C1 and C2 still pass, then run the curl lines above. For the setup guides for every other check, see /guides.