Should Google Index Your llms.txt and .md Copies? Our Search Console Data (2026)

No, and on our own site Google is not even trying. On 13 November 2026 we asked Search Console about 15 machine-readable copies on aiscan.site: /llms.txt, /llms-full.txt, /index.md and twelve .md copies of pages and blog posts. All 15 came back "URL is unknown to Google", while five of the six HTML pages behind them were "Submitted and indexed". So for most sites the duplicate-content fear is the wrong problem, and blanket noindex on those files fixes nothing and can cost you.

If you want a firm signal anyway, according to Google's own canonicalisation guide there is one documented tool for this job: a Link: <html-url>; rel="canonical" HTTP header on the copy. Below are the inspection results, our response headers, a split of 843 scanned sites by how they serve Markdown, and a decision table for the four options people argue about.

Quick summary

Short answer for a site owner: keep serving the copies, add a canonical Link header to each .md file, leave llms.txt alone, and do not noindex either unless Search Console shows a copy ranking. To see which route your site uses, run npx aiscan-cli yoursite.com (or paste your address at aiscan.site, free, no account) and read check C1 (Markdown for agents) and C2 (llms.txt) on the content checks page.

Why site owners are asking

The worry is simple. You publish a page at /pricing. A plugin or a build step also publishes /pricing.md, and an llms.txt file that lists every page. Now the same words live at two or three URLs, and years of SEO advice say duplicates dilute rankings.

People are acting on that worry. According to a thread in r/Docusaurus about a Markdown export plugin, the fix is per-bot X-Robots-Tag: noindex for the .md URLs. A founder in r/founder who generated 446 SEO pages says they noindexed "the machine-readable files (llms.txt, RSS, markdown versions)". On the other side, a log study posted to r/TechSEO found a Markdown copy of each page being fetched by seven verified crawlers, which is the reason the copies exist in the first place. All three threads were read on 12 November 2026.

So the real question has two halves. Does Google treat these files as competing pages? And if you block them from indexing, what else do you block?

What we measured, and how

We run both kinds of copy on aiscan.site, so we inspected our own.

• Search Console URL Inspection, 13 November 2026, 03:08 UTC, through the Search Console API for the sc-domain:aiscan.site property. We inspected 20 URLs: 15 machine-readable copies and the HTML twins of 5 of them, plus one more HTML post that has a .md copy. URL Inspection reports Google's last crawl, not a live test.
• Search Console performance, page dimension, 14 August to 11 November 2026 (90 days, fresh data included).
• Response headers, fetched with curl the same morning.
• A second site: xspeedcache.com, a sister product's site, has its own Search Console property and serves an llms.txt, an llms-full.txt and per-post index.md files. We inspected four URLs there.
• The AIScan scan corpus: the latest site-scope scan per host on rubric 2026.08.2, aiscan.site excluded, 843 hosts, queried 13 November 2026. Check C1 stores which route answered (the Accept header or the /index.md path) and the content type. No scanned host is named here.

We also tried to answer "which bots fetch our .md copies?" from our crawler log table. It recorded no .md or llms.txt requests at all, because those files are served before the logging code runs. That question stays open for us, and the r/TechSEO study above is the better source for it.

Result 1: Google has not found the copies

Here is every row from the aiscan.site inspection. "Last crawl" is for the HTML page; the copies have none.

Sample: 20 URLs on one site, one day. Totals: 5 indexed, 0 "not indexed", 15 unknown.

"Unknown" is a stronger result than "not indexed". It means Google has no record of the URL at all, not that it crawled the file and chose to drop it. And these copies are not hidden. Every blog post's HTML carries <link rel="alternate" type="text/markdown" href="...md"> in its head, and /llms.txt lists all 117 blog .md URLs. One thing we cannot rule out: we do not know whether those alternate links were already in the HTML on 8 October, the last time Google crawled those pages.

The performance report agrees. In 90 days, 58 URLs on aiscan.site earned at least one impression. None ends in .md or .txt. The only non-HTML file in the list is /aiscan-skill.json, with one impression, which shows Google will rank a raw file when it chooses to. It just has not chosen ours.

xspeedcache.com gave the same answer for its files: /llms.txt, /llms-full.txt and one post's index.md were all unknown to Google on 13 November. The HTML of that post was "Discovered, currently not indexed", so that site's pair tells us less: Google has not indexed the original yet either.

Result 2: what Google actually documents

Three Google pages settle most of the argument. Each was fetched from developers.google.com on 13 November 2026.

Read together, these say: Google could index your .md copy, Google gives you a header to point it at the HTML, Google tells you not to use noindex for this, and Google does not read llms.txt as a ranking input. The Link header itself is defined in RFC 8288, so any HTTP client can parse it the same way.

What Google does not document is anything specific to text/markdown or llms.txt. Everything beyond the table above is judgement, and we label it that way below.

Result 3: how 843 sites serve Markdown

The duplicate question only exists for sites that put Markdown at a second URL. Sites that answer Accept: text/markdown on the HTML URL have one URL and one page to index. Here is how the corpus splits, from the latest scan of each host on 13 November 2026.

Of the 112 sites with a real /index.md, 111 also pass or partially pass C2, so nearly every site in the duplicate case has an llms.txt as well. Across the whole corpus 468 hosts pass or partially pass C2.

By platform, for platforms with more than three C1 passes:

Two things stand out. Static builders like Astro mostly write a file, which creates the second URL. And the "HTML at /index.md" column is a bigger problem than duplicates: 80 hosts pass C1 with a page that is not Markdown at all. We wrote about why C1 lets those through in the co-pass matrix and Markdown beyond docs.

Result 4: what our own copies send

The .md copies on aiscan.site send text/markdown; charset=utf-8 and x-robots-tag: all, and their front matter carries a url: line naming the HTML page. They do not send a Link canonical header. /llms.txt sends text/plain; charset=utf-8 and no robots header. The HTML posts send a self-referencing <link rel="canonical"> and answer Accept: text/markdown with Markdown plus Vary: Accept.

So aiscan.site runs both routes at once, and the front-matter url: line is a hint only a Markdown-aware reader will see. Google reads HTTP headers, not front matter. If Google ever does crawl our copies, the canonical header is the piece we are missing, and we have filed it as a fix for the site.

xspeedcache.com serves its index.md as application/octet-stream. According to Google's indexable file types page, file type comes from Content-Type, so a copy sent that way may not be read as text at all. That is a C1 detail worth fixing before any indexing question.

The four options, compared

This is the decision people are actually making. The "Google documents it" column is fact from the pages above; the "Our verdict" column is our judgement from the data.

Why we put option 3 last. According to Google's robots meta tag page, noindex is a rule for search engine crawlers, and in practice it is an instruction to every crawler that reads robots headers, not only Google. Some AI crawlers build a search index before they answer, and a file you told them not to index is a file they may drop. That is a reasonable reading of the rule rather than something each vendor documents, so treat it as a risk, not a certainty. Per-bot noindex (sending it only to Googlebot) avoids that, but it means maintaining a user-agent list on your server, and Google's own advice is still not to use noindex for this.

Why option 2 beats option 3 when you want a signal. A canonical header keeps the copy reachable for agents, tells Google which URL to show if it ever finds the copy, and costs one header rule. It is also reversible without a re-crawl penalty, because the copy was never blocked.

What about llms.txt? It is an index file, not a copy of a page, so there is nothing to canonicalise it to. Leave it indexable. If Google ever ranks it for a query, you will see it in Search Console first. Our guide on llms.txt vs robots.txt vs sitemap.xml covers what each file is for.

How to set the canonical header

The header is the same everywhere; only the server syntax changes. For a copy at /pricing.md, send:

Link: <https://example.com/pricing>; rel="canonical"

Use an absolute URL: according to Google, the header takes absolute URLs, the same as the link element. Then check it:

curl -sI https://example.com/pricing.md | grep -i '^link'

On Nginx, one map turns every .md path into its HTML twin:

map $uri $md_canonical {
    ~^(?<base>.+)\.md$  "<https://example.com$base>; rel=\"canonical\"";
    default             "";
}
server {
    location ~ \.md$ {
        add_header Link $md_canonical always;
        default_type "text/markdown; charset=utf-8";
    }
}

On Cloudflare, a response header Transform Rule can set Link for requests whose path ends in .md, with the value built from the request path. On a static host such as Netlify or Vercel, add a header rule for the .md pattern in _headers or vercel.json. If your copies live at /page/index.md rather than /page.md, strip /index.md instead of .md. Our Markdown for AI agents guide has the per-stack setup for serving the copies in the first place, including the negotiation route.

For tested configs on every stack (Nginx, Apache, Cloudflare, Netlify, Vercel, Next.js and Astro), with the traps we hit, see how to set X-Robots-Tag or a canonical header on .md copies and llms.txt.

On WordPress, the copies usually come from a plugin, and so does robots handling. ThinkRank manages robots meta, robots.txt and llms.txt from one plugin, so the rules for the HTML and the machine-readable files are not split across two tools fighting over the same headers; it also migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Whichever plugin writes your .md copies, check the live response with the curl line above, not the settings screen.

Common mistakes we see

If you want a full-text file for agents, see when llms-full.txt beats a link list. Everything we found about whether llms.txt changes traffic at all is in does llms.txt actually work in 2026. To build a valid file, start from the llms.txt generator, and the per-platform steps are in the llms.txt setup hub.

What this post does not show

"Unknown to Google" on aiscan.site is one site's state on one day. It is not proof that Google never indexes .md or llms.txt files, and other sites may see their copies crawled, especially with stronger internal links or more crawl budget. Our blog is young and only a small share of its posts are indexed, so Google may simply not have reached the alternate links yet. We did not inspect any site we do not own. The corpus figures count how C1 passed, not whether those sites' copies are indexed, because URL Inspection only works on properties you control. And our crawler log does not see the static copy routes, so we cannot tell you which AI bots fetch them.

Your next step

Run npx aiscan-cli yoursite.com, or paste your address at aiscan.site, free and without an account. Read C1 to see whether your Markdown comes from a separate /index.md URL or from Accept negotiation, and C2 for your llms.txt. If C1 shows a separate URL, add the canonical Link header and check it with curl -sI. Then open Search Console's URL Inspection on one .md copy and one HTML page, and only consider noindex if the copy is the one getting impressions. Every other content fix is linked from /guides.