---
title: "llms.txt vs robots.txt vs sitemap.xml in 2026: Six Files, Six Jobs"
slug: llms-txt-vs-robots-txt-vs-sitemap
published: 2026-09-04T03:21:57.432092+00:00
updated: 2026-09-04T03:21:57.432092+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: check:D1, check:D2, check:D3, check:C1, check:C2, check:P2, check:B1, check:B2, check:E1, platform:wordpress, platform:shopify, platform:nextjs, llms.txt, robots.txt, sitemap.xml, agent readiness, AI crawlers, discovery
description: "robots.txt, sitemap.xml and llms.txt answer three different questions, and by 2026 there are six discovery surfaces. We measured all six on 477 real sites."
url: https://aiscan.site/blog/llms-txt-vs-robots-txt-vs-sitemap
---

**Verified 4 September 2026.** Every figure below was measured or fetched on that date.

Three files keep getting compared as if they were competing for the same job: `robots.txt`, `sitemap.xml` and `llms.txt`. They are not alternatives. They answer three different questions, and by 2026 there are three more surfaces answering three more questions that most comparisons never mention.

We measured how many of the six a real site actually publishes. Across **477 sites** scanned on our own rubric between 25 August and 4 September 2026, the median site carries **three** of them, **75 sites carry none**, and only **13 carry all six**.

## Quick summary

| The question an agent is asking | The file that answers it | AIScan check | Sites passing (n=477) |
|---|---|---|---|
| May I fetch this? | `robots.txt` | D1, B1, B2 | 72.1% |
| What pages exist here? | `sitemap.xml` | D2 | 75.9% |
| Which pages actually matter? | `/llms.txt` | C2 | 44.4% |
| Can I have that in a form I can parse? | Markdown negotiation | C1 | 30.8% |
| Where are all of the above? | HTTP `Link:` header | D3 | 52.2% |
| Can I do something, not just read? | MCP endpoint or server card | P2 | 3.4% |

**The short version.** `robots.txt` is a permission statement, not a lock. `sitemap.xml` is an inventory, and the line that declares it is not part of the robots.txt standard at all. `llms.txt` is an editorial shortlist that Google says it ignores. Markdown negotiation is the same page in a cheaper form. The `Link:` header is the only surface that tells an agent the other five exist. The MCP endpoint is the only one that lets an agent act rather than read.

**If you publish only two, publish `robots.txt` and `sitemap.xml`.** 328 of 477 sites already do. **If you publish two and want a third, the evidence points at `llms.txt`**, because 199 of those 328 sites have already added it and the rest are now the minority.

## Six files, six different jobs

All six live at predictable paths and all six are fetched by machines. The difference is what a machine learns from each one.

| Surface | Location | Format | Governed by | Enforced? |
|---|---|---|---|---|
| `robots.txt` | origin root only | plain text, grouped records | RFC 9309 (IETF Standards Track) | No, voluntary |
| XML sitemap | any path, declared in `robots.txt` | XML | sitemaps.org 0.90 | No, a hint |
| `llms.txt` | root or any subpath | Markdown | llmstxt.org, no standards body | No |
| Markdown twin | same URL, different `Accept` | `text/markdown` | HTTP content negotiation | Server-side, yes |
| `Link:` header | HTTP response header | RFC 8288 link relations | llms.txt v2 for two relations | N/A |
| MCP endpoint | platform-specific, often `/api/mcp` | JSON-RPC | Model Context Protocol | Yes, it executes |

Each arrived to solve a problem the previous one left open.

| Date | What shipped | Why it mattered |
|---|---|---|
| 1994 | Robots Exclusion Protocol, Martijn Koster | Crawlers were overwhelming servers; publishers needed a way to say no |
| April 2020 | Sitemaps protocol 0.90 documented at sitemaps.org | Link-following missed pages nobody linked to |
| September 2022 | RFC 9309 makes robots.txt an IETF Standards Track document | Thirty years of convention finally written down |
| 3 September 2024 | llms.txt proposed by Jeremy Howard | Context windows are small; a sitemap of 50,000 URLs is useless to a model |
| 24 September 2025 | Cloudflare Content Signals adds `ai-train` and `ai-input` to robots.txt | "Do not crawl" and "do not train" turned out to be different requests |
| 10 August 2026 | llms.txt v2 adds link relations and subpath scoping | Discovery needed to work without guessing a path |

## robots.txt says who may, and nothing more

[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) is the only one of these six that is an actual internet standard, published September 2022 by Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman, the last three at Google.

According to its own Security Considerations section, the file is not a lock:

> "The Robots Exclusion Protocol is not a substitute for valid content security measures. Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable."

So a `Disallow` line is a request, and it advertises the path you are asking crawlers to skip. Anything genuinely private belongs behind authentication, and a `robots.txt` full of admin paths is a map for anyone who reads it.

The AI-specific layer sits on top of the same file. **B2** grades whether any known AI crawler is named at all, and it passes on just **36.9%** of the 477 sites. **B1** grades Cloudflare's Content Signals vocabulary (`search`, `ai-input`, `ai-train`) and passes on **21.2%**. Both are documented on our [bot access reference page](https://aiscan.site/docs/checks/bot-access). That vocabulary is contested: according to Google's John Mueller, writing on Reddit on 6 July 2026, "none of the crawlers / llms use the 'content-signal' robots.txt directives. It was made up by a CDN."

Naming a specific bot is not optional detail. Google's own robots.txt specification says only one group applies to any given crawler, the most specific one that matches its token. A bot with its own `User-agent` block never reads your `*` block. Our [AI crawler user-agent list](https://aiscan.site/blog/ai-crawler-user-agent-list-2026) covers which operators publish which tokens, and which document that they ignore the file.

## sitemap.xml is an inventory, and the line that declares it is not a robots.txt record

The detail almost every comparison gets wrong: the `Sitemap:` line lives in `robots.txt`, but RFC 9309 does not define it. Section 2.2.4, "Other Records", puts it outside the standard:

> "Crawlers MAY interpret other records that are not part of the robots.txt protocol -- for example, 'Sitemaps'."

Two independent conventions in one file, one of them optional for a compliant parser. That is why scanners disagree about sitemaps more than anything else.

A worked example, verified on 4 September 2026. `stripe.com/robots.txt` opens with `Sitemap: https://stripe.com/sitemap/sitemap.xml`. That URL returns **200, `application/xml`, 805 bytes**. Meanwhile `stripe.com/sitemap.xml` returns a hard **404**. A scanner that probes the conventional path calls Stripe's sitemap missing. One that reads the declaration finds it immediately. Same correctly configured site.

The sitemap's job is coverage, not priority. [Sitemaps.org](https://www.sitemaps.org/protocol.html) states it in its own words: "Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site." Its hard limits are 50,000 URLs and 50MB uncompressed per file, which is why it fails as an agent-facing document. A model with a finite context window cannot read a 50,000-row inventory to find your pricing page.

**D2** passes on **75.9%** of our 477 sites, the highest pass rate of any check in the whole rubric. This is the surface publishers have already finished.

## llms.txt is the shortlist, with a subpath rule most tools ignore

`llms.txt` exists because a sitemap answers "what exists" and nobody was answering "what matters". It is Markdown, it lives at the root or any subpath, and per [llmstxt.org](https://llmstxt.org/) an H1 with the site name "is the only required section". The spec page records its own dates: published 3 September 2024, modified 10 August 2026.

That August revision added the rule that breaks most validators, verbatim from the spec:

> "A file covers the URLs under its path, and where more than one file applies, agents should use the most specific one."

Subpath scoping means `example.com/docs/llms.txt` is legitimate even when the root file does not exist. Our own **C2** check probes the origin root and stops, a defect we published rather than hid. Our [teardown of 39 real llms.txt files](https://aiscan.site/blog/llms-txt-validator-common-mistakes) documents that case against the spec author's own reference implementation.

Adoption depends entirely on how the sample was drawn, and the spread is enormous:

| Source | Sample | Adoption | Dated |
|---|---|---|---|
| AIScan (this study) | 477 scanned sites | 44.4% pass, 9.4% partial | 4 Sep 2026 |
| [Ahrefs](https://ahrefs.com/blog/llmstxt-study/) | 137,210 domains with traffic | 28% | 15 Jun 2026 |
| [Rankability](https://www.rankability.com/blog/llms-txt-adoption/) | Tranco top 1,000 | 8.7% overall, 15.8% of the 549 reachable | updated 23 Aug 2026 |
| SE Ranking | ~300,000 domains | 10.13% | reported 20 Nov 2025 |

Ours is the highest because our sample is the most self-selected: owners who chose to run an agent-readiness scanner. Read it as an upper bound for motivated publishers, not a figure for the web. According to Ahrefs, its own sample skews "more technical and SEO-aware than the web at large".

Publishing is also not consumption. According to Ahrefs, **97% of the valid llms.txt files it found received zero requests** in May 2026. And [Google's AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), last updated 2026-07-10, is unambiguous about its own behaviour:

> "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

Our [evidence review on whether llms.txt does anything](https://aiscan.site/blog/does-llms-txt-actually-work-2026) sets out the three candidate consumers and which of them there is evidence for.

## Markdown negotiation delivers the same page in a form a model can afford

The fourth surface is not a file. It is the same URL returning a different representation when the client asks, using the `Accept` header HTTP has always had.

The best live implementation belongs to the Next.js team, verified on 4 September 2026:

```bash
curl -sI -H 'Accept: text/markdown' https://nextjs.org/ | grep -iE 'content-type|vary|x-matched'
# content-type: text/markdown; charset=utf-8
# vary: Accept
# x-matched-path: /llms.md
```

That response is **3,939 bytes**. The same URL fetched with a browser user-agent returns **347,764 bytes** of HTML. An agent gets 1.1% of the payload and loses nothing it wanted.

**C1** passes on **30.8%** of our 477 sites, and that number needs a warning we would rather publish than bury. C1 passes on 147 sites, and **55 of those 147 sit on hosts that do not cleanly pass E1**, our correct-404 check. On a host that answers 200 for every path, a Markdown probe "succeeds" because nothing can fail. The honest floor is closer to 20%, which lands much nearer Cloudflare Radar's independent finding of **3.9%** of the top 200,000 domains passing Markdown negotiation, reported in its [Agent Readiness announcement](https://blog.cloudflare.com/agent-readiness/) of 17 April 2026.

## Link headers announce where the other five are

Every convention above requires an agent to guess a path. llms.txt v2 fixed that by borrowing two RFC 8288 link relations, deliverable as HTML `<link>` tags or as HTTP `Link:` headers: `rel="describedby"` points at the llms.txt covering a page, and `rel="alternate" type="text/markdown"` points at its Markdown twin.

The header form matters more than it sounds. It works for non-HTML resources, and it can be set at the CDN without touching a single page. Our own homepage, verified on 4 September 2026, answers:

```
link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json",
      </llms.txt>; rel="describedby"; type="text/markdown",
      </sitemap.xml>; rel="sitemap"; type="application/xml",
      </api/scan>; rel="service-desc"; type="text/event-stream"
```

Four relations, one header, no path guessing required.

**D3** passes on **52.2%** of the 477 sites, which is much higher than it looks, because our D3 accepts any agent-useful relation rather than only the two that llms.txt v2 names. Measured strictly against those two relations across 39 llms.txt-publishing sites on 1 September 2026, `rel="describedby"` appeared on **4**. Say which definition your number uses, because the two differ by an order of magnitude.

## The MCP endpoint is the only surface that lets an agent act

The first five surfaces hand an agent something to read. The sixth hands it something to do, and it is missing from every comparison article we could find.

An MCP server card at `/.well-known/mcp/server-card.json` is the file version. Our **P2** check passes on **3.4%** of 477 sites, 16 in total, the lowest pass rate in the rubric after x402 payments. That number understates reality badly, and here is the proof, verified on 4 September 2026:

```bash
# The well-known path
curl -s -o /dev/null -w '%{http_code}\n' https://www.allbirds.com/.well-known/mcp/server-card.json
# 404

# The live endpoint the platform actually runs
curl -s -X POST https://www.allbirds.com/api/mcp -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
# 200 application/json, with a callable tool listed
```

The file is absent and the capability is live. On 2 September 2026 we sent that same request to 25 Shopify storefronts and **15 answered with a valid tool list**, nine of them exposing a `update_cart` tool. Our [WebMCP measurement](https://aiscan.site/blog/webmcp-do-you-need-it-yet) covers what that surface exposes and what it does not.

A standards footnote worth carrying: IANA's Well-Known URIs registry held **213** registered suffixes on 2 September 2026. `agent-card.json` is registered. Neither `mcp` nor `webmcp` is. So the file version of this surface is a convention, and the endpoint version is not declared anywhere at all.

## How many of the six do real sites actually publish?

Population: every scan in our production database on rubric version `2026.08.2` at site scope, deduplicated to the single latest scan per host, with our own domain removed. That leaves **477 sites scanned between 25 August and 4 September 2026**, mean score 56.3. It is the same population definition as our 473-site state-of-the-web study, refreshed a day later. No scanned URL is named here; the corpus is other people's sites.

| Surfaces published (of 6) | Sites | Share | Mean readiness score |
|---|---|---|---|
| 0 | 75 | 15.7% | 19.2 |
| 1 | 38 | 8.0% | 41.9 |
| 2 | 78 | 16.4% | 46.3 |
| 3 | 111 | 23.3% | 55.3 |
| 4 | 85 | 17.8% | 70.8 |
| 5 | 77 | 16.1% | 88.2 |
| 6 | 13 | 2.7% | 95.9 |

![Mean AIScan score rising from 19.2 to 95.9 as sites publish more of the six discovery surfaces, across 477 sites measured on 4 September 2026](https://gqdxuwsuuasfjeeqafyq.supabase.co/storage/v1/object/public/blog-covers/llms-txt-vs-robots-txt-vs-sitemap-surfaces.jpg)

The gradient is monotonic and steep, roughly 13 points per additional surface. That is partly circular, because five of the six are graded checks, so it is not claimed as an independent effect. What is not circular is the shape of the distribution. **A quarter of the web we scanned sits at exactly three surfaces**, and those three are almost always robots.txt, a sitemap, and one more.

## What having the classic two predicts about the newer four

Do publishers who did the old work do the new work? They do, and the effect is large.

| Group | Sites | Also publish llms.txt |
|---|---|---|
| Pass both robots.txt and sitemap | 328 | 199 (60.7%) |
| Do not pass both | 149 | 13 (8.7%) |

A site with both classic files is **seven times more likely** to publish llms.txt than a site without them. The plausible reading is not that a sitemap causes an llms.txt, but that both come from the same habit: a team that treats machine readers as an audience.

The counter-population is worth naming. **52 of 477 sites pass both classic files and none of the four newer ones**, technically competent sites that finished the 2005 checklist and never started the 2026 one. **75 sites pass none of the six at all**, with a mean readiness score of 19.2.

## Which surface to add next, based on what you already have

Run the scan first, then branch on what it returns.

| What your scan shows | Add this next | Why |
|---|---|---|
| D1 or D2 failing | The failing one of robots.txt or sitemap | Everything else is discovered through them. 328 of 477 sites already have both |
| D1 and D2 pass, C2 fails | `/llms.txt` | Cheapest remaining surface, and the majority of your peer group already has it |
| C2 passes, C1 fails | Markdown twins for your top 20 pages | Cuts payload by roughly 99% on a docs-style page |
| C1 and C2 pass, D3 fails | Two `Link:` relations at the CDN | One header change, no page edits, and it is the surface that makes the others findable |
| B2 fails | Name specific AI crawler tokens in robots.txt | 292 of 477 sites name none, so wildcard rules are doing all the work |
| Everything passes, P2 is `info` | Nothing yet, unless you sell something | 3.4% adoption. Worth watching, not worth a sprint |
| Correct-404 check E1 failing | Fix the soft 404 before anything else | On a host that answers 200 for every path, four of these six checks return meaningless passes |

That last row is the one people skip: a soft 404 makes every well-known-path probe look successful, including ours.

## How to check all six on your own site

**The fastest route is one command, free, no account:**

```bash
npx aiscan-cli yoursite.com
```

Or paste the URL at [aiscan.site](https://aiscan.site/). The scan grades all six surfaces in a single pass and names the check IDs used throughout this article: **D1** robots.txt present and sane, **D2** XML sitemap, **D3** Link header for discovery, **C1** Markdown content negotiation, **C2** `/llms.txt`, **P2** MCP Server Card, plus **B1** and **B2** for the AI-specific robots.txt layer and **E1** for correct 404 handling. The [discoverability](https://aiscan.site/docs/checks/discoverability), [content](https://aiscan.site/docs/checks/content) and [capabilities](https://aiscan.site/docs/checks/capabilities) reference pages document exactly what each one looks for.

**If you would rather check by hand,** six commands cover it:

```bash
S=https://yoursite.com
curl -s $S/robots.txt | head -30                              # D1, B1, B2
curl -s $S/robots.txt | grep -i '^sitemap:'                   # D2, read the declaration
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' $S/llms.txt          # C2
curl -sI -H 'Accept: text/markdown' $S/ | grep -i content-type               # C1
curl -sI $S/ | grep -i '^link:'                                              # D3
curl -s -o /dev/null -w '%{http_code}\n' $S/.well-known/mcp/server-card.json # P2

# And the one that validates the other five
curl -s -o /dev/null -w '%{http_code}\n' $S/definitely-not-a-real-page-xyz    # E1, must be 404
```

Run the last command first. If it returns 200, every result above it is unreliable.

## Fixing them on WordPress and on Shopify

On WordPress the recurring problem is not knowing what to publish, it is three plugins each believing they own `robots.txt`. [ThinkRank](https://thinkrank.ai) puts all five of the surfaces a WordPress site can control (the robots file, robots meta, sitemaps, schema and llms.txt) behind one plugin, which ends the argument rather than adding a fourth voice to it. Switching costs nothing in re-entered configuration either: it imports what Yoast, Rank Math, SEOPress or All in One SEO already hold. Named honestly: Rank Math and Yoast both have deeper traditional-SEO content analysis and far larger install bases, and if that analysis is your daily workflow, they remain reasonable choices. Neither ships llms.txt generation today. Our [WordPress platform page](https://aiscan.site/docs/platforms/wordpress) lists the check-by-check guides.

On Shopify the constraint is different, because the theme does not give you filesystem access to drop a file at the root. [StoreSEO](https://storeseo.com/) generates llms.txt from products, collections, pages and articles, edits `agents.md`, and handles schema and AEO markup, which is precisely the half of the stack Shopify's own agent rollout leaves untouched. It is rated 5.0 from 733 reviews on [the Shopify App Store](https://apps.shopify.com/storeseo) as of 4 September 2026 and carries the Built for Shopify badge. Honest alternatives: Yoast SEO for Shopify and Avada SEO both cover schema and metadata well, and neither generates llms.txt or agents.md today. The [Shopify platform page](https://aiscan.site/docs/platforms/shopify) has the per-check steps.

For generating a first llms.txt on any stack, our [llms.txt generator](https://aiscan.site/llms-txt-generator) writes a spec-shaped file from a URL. Be aware of its limit, stated plainly: it generates, it does not validate a file you already have.

## Where AIScan fits, and where it doesn't

| What AIScan sees | What it cannot see |
|---|---|
| Whether each of the six surfaces responds, and with what content type | Whether any agent ever fetched them. Server logs answer that, we do not |
| The declared `Sitemap:` line in robots.txt, not only the conventional path | Whether the URLs inside your sitemap are the ones worth crawling |
| `/llms.txt` at the origin root | An llms.txt at a subpath, which llms.txt v2 explicitly permits. This is a defect in our C2, published on 1 September 2026 |
| A Markdown content type on a probe path | Whether the Markdown is a faithful rendering of the page or a stub |
| Any agent-useful `Link:` relation | Whether the targets it names resolve to what they claim |
| An MCP server card at the well-known path | A live MCP endpoint at a platform-specific path such as `/api/mcp`, which is where the real adoption is |

Two of those rows are our own open bugs. We publish them because a scanner that hides its false negatives is worth less than one that names them.

## The six-surface checklist, ten minutes

- [ ] Probe a nonsense path. Confirm it returns 404, not 200
- [ ] `robots.txt` exists, returns `text/plain`, and names at least one AI crawler token explicitly
- [ ] `robots.txt` carries a `Sitemap:` line pointing at a URL that returns 200 and XML
- [ ] Each sitemap file is under 50,000 URLs and 50MB uncompressed
- [ ] `/llms.txt` returns 200 with `text/plain` and starts with a single `# ` H1 line
- [ ] If your docs live under a subpath, publish an llms.txt there too
- [ ] `Accept: text/markdown` on your busiest page returns `text/markdown`, not `text/html`
- [ ] Response headers carry the two v2 relations, `describedby` and `alternate` with `type="text/markdown"`
- [ ] If you run a store, send one JSON-RPC `tools/list` to your platform's MCP endpoint and read what it exposes

## What to do with this

Six surfaces, six questions, and no site needs all six. The evidence from 477 real sites is that most publishers stop at three, that the classic two are finished work, and that the gap between a site scoring 46 and one scoring 88 is two files and one HTTP header.

Start by finding out which of the six you already have. One command, `npx aiscan-cli yoursite.com`, or your URL pasted into [aiscan.site](https://aiscan.site/), returns all of them. Read **E1** first, because it decides whether the rest mean anything, then **D1**, **D2**, **D3**, **C1**, **C2** and **P2**. Then work down the decision table above, one surface at a time.

The [guides index](https://aiscan.site/guides) has the per-platform walkthroughs, and the [state-of-the-web study](https://aiscan.site/blog/state-of-ai-agent-readiness-2026) has the full distribution this article's population came from.