--- title: "Should Google Index Your llms.txt and .md Copies? Our Search Console Data (2026)" slug: index-llms-txt-markdown-copies-google published: 2026-11-13T03:14:03.832331+00:00 updated: 2026-11-13T08:18:15.763164+00:00 author: "M Asif Rahman" author_url: https://ar.bd category: "AI Readiness" tags: check:C1, check:C2, markdown, llms.txt, indexing description: "Should Google index your llms.txt and .md copies? Our Search Console found 0 of 15 known to Google. When to use a canonical header, noindex or one URL." url: https://aiscan.site/blog/index-llms-txt-markdown-copies-google --- No, and on our own site Google is not even trying. On 13 November 2026 we asked Search Console about 15 machine-readable copies on aiscan.site: `/llms.txt`, `/llms-full.txt`, `/index.md` and twelve `.md` copies of pages and blog posts. All 15 came back "URL is unknown to Google", while five of the six HTML pages behind them were "Submitted and indexed". So for most sites the duplicate-content fear is the wrong problem, and blanket `noindex` on those files fixes nothing and can cost you. If you want a firm signal anyway, according to Google's own canonicalisation guide there is one documented tool for this job: a `Link: ; rel="canonical"` HTTP header on the copy. Below are the inspection results, our response headers, a split of 843 scanned sites by how they serve Markdown, and a decision table for the four options people argue about. ## Quick summary | Question | Answer | Source | |---|---|---| | Did Google index our `.md` copies and llms.txt? | No. 15 of 15 "URL is unknown to Google", so 0 of 15 were known to Google at all | Search Console URL Inspection, aiscan.site, 13 Nov 2026 | | Did it index the HTML pages they copy? | Yes, 5 of the 6 we inspected | Same inspection | | Did any `.md` or `.txt` URL get a search impression? | None in 90 days; 58 URLs had impressions, all HTML except one JSON file | Search Console performance, 14 Aug to 11 Nov 2026 | | Does Google say you need these files for AI Overviews? | No. "You don't need to create new machine readable files, AI text files, or markup" | [Google, AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), updated 10 Dec 2025 | | Can Google index a text file at all? | Yes. Text files are on its indexable list, typed by `Content-Type` | [Google, indexable file types](https://developers.google.com/search/docs/crawling-indexing/indexable-file-types) | | Documented way to point a copy at the HTML | `Link: ; rel="canonical"` response header | [Google, consolidate duplicate URLs](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) | | Should you `noindex` the copies? | Only if a copy is already ranking in place of the HTML | Google advises against `noindex` for canonical selection on one site | | How sites in our corpus serve Markdown | 112 by a separate `/index.md` URL, 73 by `Accept` negotiation on the same URL | AIScan scan DB, 843 hosts, 13 Nov 2026 | **Short answer for a site owner:** keep serving the copies, add a canonical `Link` header to each `.md` file, leave `llms.txt` alone, and do not `noindex` either unless Search Console shows a copy ranking. To see which route your site uses, run `npx aiscan-cli yoursite.com` (or paste your address at [aiscan.site](https://aiscan.site/), free, no account) and read check **C1** (Markdown for agents) and **C2** (llms.txt) on the [content checks page](https://aiscan.site/docs/checks/content). ## Why site owners are asking The worry is simple. You publish a page at `/pricing`. A plugin or a build step also publishes `/pricing.md`, and an `llms.txt` file that lists every page. Now the same words live at two or three URLs, and years of SEO advice say duplicates dilute rankings. People are acting on that worry. According to a thread in [r/Docusaurus](https://www.reddit.com/r/Docusaurus/comments/1p5f8ck/) about a Markdown export plugin, the fix is per-bot `X-Robots-Tag: noindex` for the `.md` URLs. A founder in [r/founder](https://www.reddit.com/r/founder/comments/1x9a1o6/) who generated 446 SEO pages says they noindexed "the machine-readable files (llms.txt, RSS, markdown versions)". On the other side, a log study posted to [r/TechSEO](https://www.reddit.com/r/TechSEO/comments/1xn8g06/) found a Markdown copy of each page being fetched by seven verified crawlers, which is the reason the copies exist in the first place. All three threads were read on 12 November 2026. So the real question has two halves. Does Google treat these files as competing pages? And if you block them from indexing, what else do you block? ## What we measured, and how We run both kinds of copy on aiscan.site, so we inspected our own. - **Search Console URL Inspection**, 13 November 2026, 03:08 UTC, through the Search Console API for the `sc-domain:aiscan.site` property. We inspected 20 URLs: 15 machine-readable copies and the HTML twins of 5 of them, plus one more HTML post that has a `.md` copy. URL Inspection reports Google's last crawl, not a live test. - **Search Console performance**, page dimension, 14 August to 11 November 2026 (90 days, fresh data included). - **Response headers**, fetched with `curl` the same morning. - **A second site**: xspeedcache.com, a sister product's site, has its own Search Console property and serves an `llms.txt`, an `llms-full.txt` and per-post `index.md` files. We inspected four URLs there. - **The AIScan scan corpus**: the latest site-scope scan per host on rubric 2026.08.2, aiscan.site excluded, 843 hosts, queried 13 November 2026. Check C1 stores which route answered (the `Accept` header or the `/index.md` path) and the content type. No scanned host is named here. We also tried to answer "which bots fetch our `.md` copies?" from our crawler log table. It recorded no `.md` or `llms.txt` requests at all, because those files are served before the logging code runs. That question stays open for us, and the r/TechSEO study above is the better source for it. ## Result 1: Google has not found the copies Here is every row from the aiscan.site inspection. "Last crawl" is for the HTML page; the copies have none. | URL | Type served | Google's coverage state | Last crawl | |---|---|---|---| | `/llms.txt` | `text/plain; charset=utf-8` | URL is unknown to Google | none | | `/llms-full.txt` | `text/plain; charset=utf-8` | URL is unknown to Google | none | | `/index.md` | `text/markdown; charset=utf-8` | URL is unknown to Google | none | | `/blog.md`, `/guides.md` | Markdown | URL is unknown to Google (both) | none | | 10 blog post `.md` copies | `text/markdown`, `x-robots-tag: all` | URL is unknown to Google (all 10) | none | | `/blog/state-of-ai-agent-readiness-2026` (HTML) | `text/html` | Submitted and indexed | 8 Oct 2026 | | `/blog/does-llms-txt-actually-work-2026` (HTML) | `text/html` | Submitted and indexed | 8 Oct 2026 | | `/blog/agent-readiness-highest-impact-fix` (HTML) | `text/html` | Submitted and indexed | 8 Oct 2026 | | `/blog/ai-crawler-user-agent-list-2026` (HTML) | `text/html` | Submitted and indexed | 8 Oct 2026 | | `/blog/best-ai-readiness-checkers-2026` (HTML) | `text/html` | Submitted and indexed | 17 Oct 2026 | | `/blog/markdown-for-ai-agents` (HTML) | `text/html` | URL is unknown to Google at 03:08; "Discovered, currently not indexed" on a second inspection at 03:16 | none | Sample: 20 URLs on one site, one day. Totals: 5 indexed, 0 "not indexed", 15 unknown. "Unknown" is a stronger result than "not indexed". It means Google has no record of the URL at all, not that it crawled the file and chose to drop it. And these copies are not hidden. Every blog post's HTML carries `` in its head, and `/llms.txt` lists all 117 blog `.md` URLs. One thing we cannot rule out: we do not know whether those alternate links were already in the HTML on 8 October, the last time Google crawled those pages. The performance report agrees. In 90 days, 58 URLs on aiscan.site earned at least one impression. None ends in `.md` or `.txt`. The only non-HTML file in the list is `/aiscan-skill.json`, with one impression, which shows Google will rank a raw file when it chooses to. It just has not chosen ours. xspeedcache.com gave the same answer for its files: `/llms.txt`, `/llms-full.txt` and one post's `index.md` were all unknown to Google on 13 November. The HTML of that post was "Discovered, currently not indexed", so that site's pair tells us less: Google has not indexed the original yet either. ## Result 2: what Google actually documents Three Google pages settle most of the argument. Each was fetched from developers.google.com on 13 November 2026. | Claim | What Google says | Page and its "last updated" date | |---|---|---| | Google can index text files | Text files (".txt, .text, other file extensions") are indexable, and "the file type is determined by the Content-Type HTTP header" | [Indexable file types](https://developers.google.com/search/docs/crawling-indexing/indexable-file-types), 3 Feb 2026 | | A non-HTML copy can name its canonical | You "can return a rel="canonical" HTTP header to tell Googlebot what is the canonical URL for the non-HTML files" | [Consolidate duplicate URLs](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls), 10 Jul 2026 | | `noindex` is the wrong tool for duplicates on one site | "We don't recommend using noindex to prevent selection of a canonical page within a single site, because it will completely block the page from Search" | Same page | | `noindex` for non-HTML files goes in a header | "To block indexing of non-HTML resources ... use the X-Robots-Tag response header" | [Robots meta tag and X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag), 24 Mar 2026 | | AI features need no special files | "You don't need to create new machine readable files, AI text files, or markup to appear in these features" | [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), 10 Dec 2025 | Read together, these say: Google could index your `.md` copy, Google gives you a header to point it at the HTML, Google tells you not to use `noindex` for this, and Google does not read `llms.txt` as a ranking input. The `Link` header itself is defined in [RFC 8288](https://www.rfc-editor.org/rfc/rfc8288.html), so any HTTP client can parse it the same way. What Google does not document is anything specific to `text/markdown` or `llms.txt`. Everything beyond the table above is judgement, and we label it that way below. ## Result 3: how 843 sites serve Markdown The duplicate question only exists for sites that put Markdown at a second URL. Sites that answer `Accept: text/markdown` on the HTML URL have one URL and one page to index. Here is how the corpus splits, from the latest scan of each host on 13 November 2026. | How C1 passed (pass or partial) | Hosts | What it means for indexing | |---|---|---| | Separate `/index.md` URL, served as Markdown | 112 | A second URL exists. This is the case the duplicate worry is about | | `Accept: text/markdown` on the same URL | 73 | One URL. Nothing to canonicalise; send `Vary: Accept` | | `/index.md` answered 200 with `text/html` | 80 | Usually a soft 404: the site's HTML shell at a path that does not exist | | `/index.md` as `application/octet-stream` or `text/plain` | 13 | A second URL that agents may not read as Markdown | | `Accept` route returned 404 (partial) | 1 | Counted for completeness | | **All C1 passes and partials** | **279 of 843** | 33.1% of the corpus | Of the 112 sites with a real `/index.md`, 111 also pass or partially pass C2, so nearly every site in the duplicate case has an `llms.txt` as well. Across the whole corpus 468 hosts pass or partially pass C2. By platform, for platforms with more than three C1 passes: | Platform | `/index.md` as Markdown | `Accept` negotiation | `/index.md` as HTML | |---|---|---|---| | Next.js | 33 | 31 | 12 | | Astro | 23 | 4 | 1 | | WordPress | 22 | 11 | 15 | | Not detected | 23 | 20 | 34 | | SvelteKit | 4 | 0 | 0 | | Framer | 0 | 4 | 0 | | Lovable | 2 | 2 | 2 | | Shopify | 1 | 1 | 12 | Two things stand out. Static builders like Astro mostly write a file, which creates the second URL. And the "HTML at `/index.md`" column is a bigger problem than duplicates: 80 hosts pass C1 with a page that is not Markdown at all. We wrote about why C1 lets those through in [the co-pass matrix](https://aiscan.site/blog/check-co-pass-matrix) and [Markdown beyond docs](https://aiscan.site/blog/markdown-twin-beyond-docs). ## Result 4: what our own copies send The `.md` copies on aiscan.site send `text/markdown; charset=utf-8` and `x-robots-tag: all`, and their front matter carries a `url:` line naming the HTML page. They do not send a `Link` canonical header. `/llms.txt` sends `text/plain; charset=utf-8` and no robots header. The HTML posts send a self-referencing `` and answer `Accept: text/markdown` with Markdown plus `Vary: Accept`. So aiscan.site runs both routes at once, and the front-matter `url:` line is a hint only a Markdown-aware reader will see. Google reads HTTP headers, not front matter. If Google ever does crawl our copies, the canonical header is the piece we are missing, and we have filed it as a fix for the site. xspeedcache.com serves its `index.md` as `application/octet-stream`. According to Google's indexable file types page, file type comes from `Content-Type`, so a copy sent that way may not be read as text at all. That is a C1 detail worth fixing before any indexing question. ## The four options, compared This is the decision people are actually making. The "Google documents it" column is fact from the pages above; the "Our verdict" column is our judgement from the data. | Option | What you send | Google documents it? | Effect on AI agents | Our verdict | |---|---|---|---|---| | 1. Do nothing | Copies with no robots or canonical header | n/a | Full access | Fine for most sites today: Google did not even find ours | | 2. Canonical header on the copy | `Link: ; rel="canonical"` on each `.md` | Yes, for non-HTML files | Full access; agents can also follow the header back to the HTML | Best default for any site with separate `.md` URLs | | 3. `noindex` on the copy | `X-Robots-Tag: noindex` on `.md` and `.txt` | Yes, but Google advises against it for duplicates on one site | Any crawler that honours `X-Robots-Tag` is told not to index the file | Only if a copy already ranks in place of the HTML | | 4. One URL, negotiated | Markdown on the HTML URL for `Accept: text/markdown`, plus `Vary: Accept` | No Markdown-specific guidance; standard HTTP | Only agents that send the header get Markdown | Cleanest for new builds; no second URL exists | Why we put option 3 last. According to Google's robots meta tag page, `noindex` is a rule for search engine crawlers, and in practice it is an instruction to every crawler that reads robots headers, not only Google. Some AI crawlers build a search index before they answer, and a file you told them not to index is a file they may drop. That is a reasonable reading of the rule rather than something each vendor documents, so treat it as a risk, not a certainty. Per-bot `noindex` (sending it only to Googlebot) avoids that, but it means maintaining a user-agent list on your server, and Google's own advice is still not to use `noindex` for this. Why option 2 beats option 3 when you want a signal. A canonical header keeps the copy reachable for agents, tells Google which URL to show if it ever finds the copy, and costs one header rule. It is also reversible without a re-crawl penalty, because the copy was never blocked. What about `llms.txt`? It is an index file, not a copy of a page, so there is nothing to canonicalise it to. Leave it indexable. If Google ever ranks it for a query, you will see it in Search Console first. Our guide on [llms.txt vs robots.txt vs sitemap.xml](https://aiscan.site/blog/llms-txt-vs-robots-txt-vs-sitemap) covers what each file is for. ## How to set the canonical header The header is the same everywhere; only the server syntax changes. For a copy at `/pricing.md`, send: ```http Link: ; rel="canonical" ``` Use an absolute URL: according to Google, the header takes absolute URLs, the same as the link element. Then check it: ```bash curl -sI https://example.com/pricing.md | grep -i '^link' ``` On Nginx, one `map` turns every `.md` path into its HTML twin: ```nginx map $uri $md_canonical { ~^(?.+)\.md$ "; rel=\"canonical\""; default ""; } server { location ~ \.md$ { add_header Link $md_canonical always; default_type "text/markdown; charset=utf-8"; } } ``` On Cloudflare, a response header Transform Rule can set `Link` for requests whose path ends in `.md`, with the value built from the request path. On a static host such as Netlify or Vercel, add a header rule for the `.md` pattern in `_headers` or `vercel.json`. If your copies live at `/page/index.md` rather than `/page.md`, strip `/index.md` instead of `.md`. Our [Markdown for AI agents guide](https://aiscan.site/blog/markdown-for-ai-agents) has the per-stack setup for serving the copies in the first place, including the negotiation route. For tested configs on every stack (Nginx, Apache, Cloudflare, Netlify, Vercel, Next.js and Astro), with the traps we hit, see [how to set X-Robots-Tag or a canonical header on .md copies and llms.txt](https://aiscan.site/blog/x-robots-tag-markdown-llms-txt-every-platform). On WordPress, the copies usually come from a plugin, and so does robots handling. [ThinkRank](https://thinkrank.ai) manages robots meta, robots.txt and llms.txt from one plugin, so the rules for the HTML and the machine-readable files are not split across two tools fighting over the same headers; it also migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Whichever plugin writes your `.md` copies, check the live response with the `curl` line above, not the settings screen. ## Common mistakes we see | Mistake | What happens | Fix | |---|---|---| | `noindex` on every `.md` and `.txt` "to be safe" | Crawlers that honour robots headers are told to drop the files you built for them | Remove it; use the canonical header if you want a signal | | Disallowing `*.md` in robots.txt | Agents that obey robots.txt cannot fetch the copies at all, and Google cannot read a canonical header it is not allowed to fetch | Allow the copies; robots.txt is for crawling, not indexing | | Relative URL in the `Link` header | Some clients resolve it differently | Use the absolute HTML URL | | `.md` served as `application/octet-stream` | Agents and Google may not treat it as text | Send `text/markdown; charset=utf-8` | | `/index.md` returning the HTML shell with 200 | A soft 404 that still passes a status-only check | Return 404 for paths that do not exist | | Listing `.md` copies in the XML sitemap | You ask Google to index the duplicate | List the HTML only; let `llms.txt` list the copies | | Negotiation without `Vary: Accept` | A cache can serve Markdown to a browser or HTML to an agent | Send `Vary: Accept` on every negotiated response | If you want a full-text file for agents, see [when llms-full.txt beats a link list](https://aiscan.site/blog/llms-full-txt-when-to-use). Everything we found about whether llms.txt changes traffic at all is in [does llms.txt actually work in 2026](https://aiscan.site/blog/does-llms-txt-actually-work-2026). To build a valid file, start from the [llms.txt generator](https://aiscan.site/llms-txt-generator), and the per-platform steps are in the [llms.txt setup hub](https://aiscan.site/blog/llms-txt-setup-every-platform). ## What this post does not show "Unknown to Google" on aiscan.site is one site's state on one day. It is not proof that Google never indexes `.md` or `llms.txt` files, and other sites may see their copies crawled, especially with stronger internal links or more crawl budget. Our blog is young and only a small share of its posts are indexed, so Google may simply not have reached the alternate links yet. We did not inspect any site we do not own. The corpus figures count how C1 passed, not whether those sites' copies are indexed, because URL Inspection only works on properties you control. And our crawler log does not see the static copy routes, so we cannot tell you which AI bots fetch them. ## Your next step Run `npx aiscan-cli yoursite.com`, or paste your address at [aiscan.site](https://aiscan.site/), free and without an account. Read **C1** to see whether your Markdown comes from a separate `/index.md` URL or from `Accept` negotiation, and **C2** for your llms.txt. If C1 shows a separate URL, add the canonical `Link` header and check it with `curl -sI`. Then open Search Console's URL Inspection on one `.md` copy and one HTML page, and only consider `noindex` if the copy is the one getting impressions. Every other content fix is linked from [/guides](https://aiscan.site/guides).