---
title: "Set X-Robots-Tag or a Canonical Link Header on .md Copies and llms.txt: Nginx, Apache, Cloudflare, Vercel, Netlify, Next.js, Astro"
slug: x-robots-tag-markdown-llms-txt-every-platform
published: 2026-11-13T08:16:25.480242+00:00
updated: 2026-11-13T08:16:25.480242+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: llms.txt, markdown, X-Robots-Tag, canonical, check:C1, check:C2, platform:nextjs, platform:astro, platform:cloudflare
description: "Point .md copies at their HTML with a canonical Link header and noindex llms.txt only if needed. Tested configs for Nginx, Apache, Next.js, Astro and more."
url: https://aiscan.site/blog/x-robots-tag-markdown-llms-txt-every-platform
---

Send one header on each `.md` copy and nothing new on the HTML: `Link: <https://example.com/page>; rel="canonical"`. On `/llms.txt`, which has no HTML twin, the only header that applies is `X-Robots-Tag: noindex`, and you only need it if that file is showing up in search. We set both on Nginx 1.24.0, Apache 2.4.58, Next.js 16.4.6 and Astro 7.4.11 on 13 November 2026 and pasted every response below, with the config for Cloudflare, Vercel and Netlify from their own docs.

## Quick summary

| File | Header to send | When | Tested here |
|---|---|---|---|
| `/page.md` (copy of `/page`) | `Link: <https://example.com/page>; rel="canonical"` | Default for every copy | Nginx, Apache, Next.js, Astro |
| `/page.md` | `X-Robots-Tag: noindex` | Only if the copy ranks in place of the HTML | Nginx, Apache |
| `/llms.txt`, `/llms-full.txt` | `X-Robots-Tag: noindex` | Only if the file shows up in search | All four |
| Either | Nothing | Most sites, because Google has not found the copies at all | Our own Search Console |

If you are still deciding whether to do this at all, read [Should Google index your llms.txt and .md copies?](https://aiscan.site/blog/index-llms-txt-markdown-copies-google) first. It has our Search Console data: on 13 November 2026, 15 of 15 machine-readable copies on aiscan.site were "URL is unknown to Google". This post is the where-to-type-it half.

## Why canonical beats noindex for a copy

According to Google's [guide to consolidating duplicate URLs](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) (updated 10 July 2026, verified on 13 November 2026), a `rel="canonical"` HTTP header is the documented way to point a non-HTML file at its preferred URL, and it uses an absolute URL. The same page says, in its own words: "We don't recommend using noindex to prevent selection of a canonical page within a single site, because it will completely block the page from Search."

A canonical header says "this is a copy, rank the HTML". A `noindex` says "drop this file". The copy stays reachable either way, so agents that fetch `/page.md` or `/llms.txt` directly still get it. One caution: `noindex` also tells any AI crawler that honours robots directives to drop the file from its index. Use it only on a file that is causing a real problem in search, and never send both headers on the same URL, because they contradict each other.

According to Google's [robots meta tag and X-Robots-Tag page](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) (updated 24 March 2026, verified on 13 November 2026), `X-Robots-Tag` is the way to set `noindex` on non-HTML files, and the page lists Apache and Nginx examples.

## What we measured

We served a `pricing.md`, a nested `blog/first-post.md` and an `llms.txt` from each stack, then read the response headers with a short script. Each row is one HEAD request on 13 November 2026.

| Stack | `/blog/first-post.md` Link | `/llms.txt` X-Robots-Tag | Content-Type on `.md` |
|---|---|---|---|
| Nginx 1.24.0 | `<https://example.com/blog/first-post>; rel="canonical"` | `noindex` | `text/markdown; charset=utf-8` |
| Apache 2.4.58 | same | `noindex` | `text/markdown; charset=utf-8` |
| Next.js 16.4.6 (`next start`) | same | `noindex` | `text/markdown; charset=UTF-8` |
| Astro 7.4.11 + @astrojs/node 11.1.12 | same (endpoint) | `noindex` (endpoint) | `text/markdown; charset=utf-8` |
| Astro, file in `public/` | none | none | `text/markdown; charset=utf-8` |

Three things went wrong on the way, and each is a trap you will hit too:

1. **Nginx dropped the charset.** `charset utf-8;` did nothing for `.md` until we added `text/markdown` to `charset_types`, because the [charset module](https://nginx.org/en/docs/http/ngx_http_charset_module.html) only applies to the types in that list.
2. **Nginx `add_header` wiped a server-level header.** A `nosniff` header set on the `server` block disappeared from `.md` and `llms.txt` once their `location` blocks had their own `add_header`. The [headers module](https://nginx.org/en/docs/http/ngx_http_headers_module.html) inherits `add_header` from the parent level only when the current level defines none.
3. **Astro middleware never touched `public/`.** Our middleware set both headers on rendered routes, but files copied from `public/` came back with neither. Serve those files from an endpoint, or set the header at the host.

The script we used to read the results:

```python
import sys, urllib.request
for url in sys.argv[1:]:
    with urllib.request.urlopen(urllib.request.Request(url, method='HEAD')) as r:
        h = r.headers
        print(url, r.status, h.get('Content-Type'),
              'x-robots-tag=' + (h.get('X-Robots-Tag') or '-'),
              'link=' + (h.get('Link') or '-'))
```

## Which section is yours

| Your setup | Where the header goes | Can one rule cover every `.md`? | Source |
|---|---|---|---|
| Own server, Nginx | `map` + `location` | Yes | tested here |
| Own server, Apache | `SetEnvIf` + `Header set` | Yes | tested here |
| DNS proxied by Cloudflare | Response Header Transform Rule | Yes, with `concat()` | Cloudflare docs |
| Cloudflare Pages | `_headers` with `:splat` | Yes | Cloudflare Pages docs |
| Netlify | `_headers`, one block per file | No, generate it | Netlify docs |
| Next.js | `headers()` in `next.config` | Yes | tested here |
| Vercel without Next.js | `vercel.json` `headers` | Check after deploy | Vercel docs |
| Astro | endpoint + middleware, or the host | Not for `public/` | tested here |

**Action:** run `curl -sI https://yoursite.com/llms.txt` and look at the `server:` line. `cloudflare`, `Netlify` and `Vercel` name the layer that will set the header.

## Nginx

1. Open the site's server block (often `/etc/nginx/sites-available/<site>`).
2. Add the type, the charset list and the map in the `http` context:

```nginx
types { text/markdown md; }
charset_types text/markdown text/plain;
map $uri $md_canonical {
  ~^(?<base>.+)\.md$ "<https://example.com$base>; rel=\"canonical\"";
  default "";
}
```

3. Add the locations inside `server`, and repeat any server-level `add_header` lines inside each one:

```nginx
location ~ \.md$ {
  add_header Link $md_canonical always;
  charset utf-8;
}
location = /llms.txt {
  add_header X-Robots-Tag "noindex" always;
  charset utf-8;
}
```

4. Run `nginx -t`, then `systemctl reload nginx`.

## Apache

1. Enable the modules: `a2enmod headers setenvif`.
2. Add this to the virtual host or `.htaccess` (the [mod_headers docs](https://httpd.apache.org/docs/2.4/mod/mod_headers.html) explain the `env=` condition):

```apache
AddType "text/markdown; charset=utf-8" .md
SetEnvIf Request_URI "^(.+)\.md$" MD_BASE=$1
Header set Link "<https://example.com%{MD_BASE}e>; rel=\"canonical\"" env=MD_BASE
<Files "llms.txt">
  Header set X-Robots-Tag "noindex"
</Files>
```

3. Run `apachectl configtest`, then reload.

## Cloudflare

Not deployed on a live zone for this post. The steps are fetched from Cloudflare's [response header Transform Rule docs](https://developers.cloudflare.com/rules/transform/response-header-modification/) and its [functions reference](https://developers.cloudflare.com/ruleset-engine/rules-language/functions/), on 13 November 2026.

1. Go to **Rules > Overview**, select **Create rule > Response Header Transform Rule**.
2. Name it, choose **Custom filter expression** and enter `ends_with(http.request.uri.path, ".md")`.
3. Under **Modify response header**, choose **Set dynamic**, header name `Link`, value:
   `concat("<https://", http.host, substring(http.request.uri.path, 0, -3), ">; rel=\"canonical\"")`
4. Select **Deploy**. Create a second rule for `http.request.uri.path eq "/llms.txt"` with **Set static** `X-Robots-Tag` = `noindex`.

`substring()` accepts negative indexes, so `0, -3` strips `.md`. Do not reach for `regex_replace()`: the docs limit it to rewrite and redirect expressions. If you already turned on Cloudflare's Markdown for Agents, see [our rule guide](https://aiscan.site/blog/cloudflare-markdown-for-agents-rule) first, because negotiated Markdown has no `.md` path to match.

## Netlify and Cloudflare Pages

Both read a `_headers` file in the publish folder. According to Cloudflare Pages' [headers docs](https://developers.cloudflare.com/pages/configuration/headers/), a value can reuse the matched path as `:splat`, so one rule covers every copy:

```text
/*.md
  Link: <https://example.com/:splat>; rel="canonical"
/llms.txt
  X-Robots-Tag: noindex
```

Netlify's [headers docs](https://docs.netlify.com/manage/routing/headers/) describe wildcards and placeholders in paths only, not in values (checked 13 November 2026). So on Netlify, generate one block per file at build time. We ran this against a test `dist/` folder and it wrote correct entries for `/pricing.md` and `/blog/first-post.md`:

```js
// node make-headers.mjs dist https://example.com
import { readdirSync, appendFileSync } from 'node:fs';
import { join, relative } from 'node:path';
const [dir, site] = process.argv.slice(2);
const walk = (d) => readdirSync(d, { withFileTypes: true })
  .flatMap((e) => e.isDirectory() ? walk(join(d, e.name)) : [join(d, e.name)]);
let out = '\n/llms.txt\n  X-Robots-Tag: noindex\n';
for (const f of walk(dir).filter((f) => f.endsWith('.md'))) {
  const path = '/' + relative(dir, f).split('\\').join('/');
  out += `\n${path}\n  Link: <${site}${path.slice(0, -3)}>; rel="canonical"\n`;
}
appendFileSync(join(dir, '_headers'), out);
```

Run it after the build: `"build": "astro build && node make-headers.mjs dist https://example.com"`. Cloudflare Pages caps `_headers` at 100 rules, which is one more reason to use `:splat` there. Neither host applies `_headers` to function or SSR responses.

## Next.js

Tested with `next build && next start`. In `next.config.mjs`:

```js
const SITE = 'https://example.com';
export default {
  async headers() {
    return [
      { source: '/:path*.md',
        headers: [{ key: 'Link', value: `<${SITE}/:path*>; rel="canonical"` }] },
      { source: '/llms.txt',
        headers: [{ key: 'X-Robots-Tag', value: 'noindex' }] },
    ];
  },
};
```

According to the [Next.js headers docs](https://nextjs.org/docs/app/api-reference/config/next-config-js/headers), matched parameters can be used in the value, and the nested copy came back as `<https://example.com/blog/first-post>`. This also works for files in `public/`.

**Vercel without Next.js:** put the same `source` and `headers` objects in `vercel.json`. Vercel's [vercel.json reference](https://vercel.com/docs/project-configuration/vercel-json) does not say whether `:path*` is substituted in a value, so after the deploy check the `Link` line. If it comes back with a literal `:path*`, use one entry per file.

## Astro

On a static build, Astro writes your files and the host serves them, so use the `_headers` or `vercel.json` section for your host. On a server build, move the copies out of `public/` into endpoints and set the headers in middleware:

```js
// src/middleware.js
export async function onRequest(context, next) {
  const res = await next();
  const { pathname } = context.url;
  if (pathname.endsWith('.md')) {
    const html = new URL(pathname.slice(0, -3), context.site);
    res.headers.set('Link', `<${html.href}>; rel="canonical"`);
  }
  if (pathname === '/llms.txt') res.headers.set('X-Robots-Tag', 'noindex');
  return res;
}
```

Set `site` in `astro.config.mjs`. Our `src/pages/llms.txt.js` endpoint (with `export const prerender = false`) returned `noindex`; the same file in `public/` did not. Astro's [middleware guide](https://docs.astro.build/en/guides/middleware/) covers the hook. Our [Markdown for AI agents hub](https://aiscan.site/blog/markdown-for-ai-agents#astro) shows how to generate the copies.

## WordPress

The copies and `llms.txt` usually come from a plugin. [ThinkRank](https://thinkrank.ai) manages robots meta, robots.txt and llms.txt from one plugin, so the rules for HTML and machine-readable files are not split across tools fighting over the same output, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. The response headers on the files themselves are still set by the web server, so add the Nginx or Apache block above, or ask your host.

## Check it worked

The quickest check is a scan. Run `npx aiscan-cli yoursite.com` or paste your URL at [aiscan.site](https://aiscan.site/), free and with no account. Checks **C1** (Markdown content negotiation) and **C2** (`/llms.txt`) should both stay passing after the change, because a header must never make the files unreachable. The [content checks page](https://aiscan.site/docs/checks/content) explains both. AIScan does not score the `Link` or `X-Robots-Tag` values themselves, so confirm those by hand:

```bash
curl -sI https://yoursite.com/blog/post.md | grep -iE '^(link|x-robots-tag|content-type)'
curl -sI https://yoursite.com/llms.txt | grep -iE '^(x-robots-tag|content-type)'
```

Pass means: one `Link` line on every `.md`, pointing at the absolute HTML URL that returns 200; no `noindex` on any `.md` you canonicalised; status 200 on both files. If `Content-Type` lost its charset, see [llms.txt Content-Type and charset](https://aiscan.site/blog/llms-txt-content-type-charset). If you have no `llms.txt` yet, the [llms.txt generator](https://aiscan.site/llms-txt-generator) writes one from your sitemap.

**Our own site, honestly:** on 13 November 2026, aiscan.site's `.md` copies sent `x-robots-tag: all` and no `Link`, and `/llms.txt` sent neither. Since Google had found none of them, we have left it that way.

## Maintenance

Re-run the two curl lines after any server, CDN or framework upgrade. If you change your domain, the canonical URL inside the header changes with it, so the Nginx map, the Apache line and the Cloudflare rule all need the new host.

## Your next step

Scan your site at [aiscan.site](https://aiscan.site/) and check that C1 and C2 still pass, then run the curl lines above. For the setup guides for every other check, see [/guides](https://aiscan.site/guides).