llms.txt vs robots.txt vs sitemap.xml in 2026: Six Files, Six Jobs

Verified 4 September 2026. Every figure below was measured or fetched on that date.

Three files keep getting compared as if they were competing for the same job: robots.txt, sitemap.xml and llms.txt. They are not alternatives. They answer three different questions, and by 2026 there are three more surfaces answering three more questions that most comparisons never mention.

We measured how many of the six a real site actually publishes. Across 477 sites scanned on our own rubric between 25 August and 4 September 2026, the median site carries three of them, 75 sites carry none, and only 13 carry all six.

Quick summary

The short version. robots.txt is a permission statement, not a lock. sitemap.xml is an inventory, and the line that declares it is not part of the robots.txt standard at all. llms.txt is an editorial shortlist that Google says it ignores. Markdown negotiation is the same page in a cheaper form. The Link: header is the only surface that tells an agent the other five exist. The MCP endpoint is the only one that lets an agent act rather than read.

If you publish only two, publish robots.txt and sitemap.xml. 328 of 477 sites already do. If you publish two and want a third, the evidence points at llms.txt, because 199 of those 328 sites have already added it and the rest are now the minority.

Six files, six different jobs

All six live at predictable paths and all six are fetched by machines. The difference is what a machine learns from each one.

Each arrived to solve a problem the previous one left open.

robots.txt says who may, and nothing more

RFC 9309 is the only one of these six that is an actual internet standard, published September 2022 by Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman, the last three at Google.

According to its own Security Considerations section, the file is not a lock:

"The Robots Exclusion Protocol is not a substitute for valid content security measures. Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable."

So a Disallow line is a request, and it advertises the path you are asking crawlers to skip. Anything genuinely private belongs behind authentication, and a robots.txt full of admin paths is a map for anyone who reads it.

The AI-specific layer sits on top of the same file. B2 grades whether any known AI crawler is named at all, and it passes on just 36.9% of the 477 sites. B1 grades Cloudflare's Content Signals vocabulary (search, ai-input, ai-train) and passes on 21.2%. Both are documented on our bot access reference page. That vocabulary is contested: according to Google's John Mueller, writing on Reddit on 6 July 2026, "none of the crawlers / llms use the 'content-signal' robots.txt directives. It was made up by a CDN."

Naming a specific bot is not optional detail. Google's own robots.txt specification says only one group applies to any given crawler, the most specific one that matches its token. A bot with its own User-agent block never reads your * block. Our AI crawler user-agent list covers which operators publish which tokens, and which document that they ignore the file.

sitemap.xml is an inventory, and the line that declares it is not a robots.txt record

The detail almost every comparison gets wrong: the Sitemap: line lives in robots.txt, but RFC 9309 does not define it. Section 2.2.4, "Other Records", puts it outside the standard:

"Crawlers MAY interpret other records that are not part of the robots.txt protocol -- for example, 'Sitemaps'."

Two independent conventions in one file, one of them optional for a compliant parser. That is why scanners disagree about sitemaps more than anything else.

A worked example, verified on 4 September 2026. stripe.com/robots.txt opens with Sitemap: https://stripe.com/sitemap/sitemap.xml. That URL returns 200, application/xml, 805 bytes. Meanwhile stripe.com/sitemap.xml returns a hard 404. A scanner that probes the conventional path calls Stripe's sitemap missing. One that reads the declaration finds it immediately. Same correctly configured site.

The sitemap's job is coverage, not priority. Sitemaps.org states it in its own words: "Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site." Its hard limits are 50,000 URLs and 50MB uncompressed per file, which is why it fails as an agent-facing document. A model with a finite context window cannot read a 50,000-row inventory to find your pricing page.

D2 passes on 75.9% of our 477 sites, the highest pass rate of any check in the whole rubric. This is the surface publishers have already finished.

llms.txt is the shortlist, with a subpath rule most tools ignore

llms.txt exists because a sitemap answers "what exists" and nobody was answering "what matters". It is Markdown, it lives at the root or any subpath, and per llmstxt.org an H1 with the site name "is the only required section". The spec page records its own dates: published 3 September 2024, modified 10 August 2026.

That August revision added the rule that breaks most validators, verbatim from the spec:

"A file covers the URLs under its path, and where more than one file applies, agents should use the most specific one."

Subpath scoping means example.com/docs/llms.txt is legitimate even when the root file does not exist. Our own C2 check probes the origin root and stops, a defect we published rather than hid. Our teardown of 39 real llms.txt files documents that case against the spec author's own reference implementation.

Adoption depends entirely on how the sample was drawn, and the spread is enormous:

Ours is the highest because our sample is the most self-selected: owners who chose to run an agent-readiness scanner. Read it as an upper bound for motivated publishers, not a figure for the web. According to Ahrefs, its own sample skews "more technical and SEO-aware than the web at large".

Publishing is also not consumption. According to Ahrefs, 97% of the valid llms.txt files it found received zero requests in May 2026. And Google's AI optimization guide, last updated 2026-07-10, is unambiguous about its own behaviour:

"Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

Our evidence review on whether llms.txt does anything sets out the three candidate consumers and which of them there is evidence for.

Markdown negotiation delivers the same page in a form a model can afford

The fourth surface is not a file. It is the same URL returning a different representation when the client asks, using the Accept header HTTP has always had.

The best live implementation belongs to the Next.js team, verified on 4 September 2026:

curl -sI -H 'Accept: text/markdown' https://nextjs.org/ | grep -iE 'content-type|vary|x-matched'
# content-type: text/markdown; charset=utf-8
# vary: Accept
# x-matched-path: /llms.md

That response is 3,939 bytes. The same URL fetched with a browser user-agent returns 347,764 bytes of HTML. An agent gets 1.1% of the payload and loses nothing it wanted.

C1 passes on 30.8% of our 477 sites, and that number needs a warning we would rather publish than bury. C1 passes on 147 sites, and 55 of those 147 sit on hosts that do not cleanly pass E1, our correct-404 check. On a host that answers 200 for every path, a Markdown probe "succeeds" because nothing can fail. The honest floor is closer to 20%, which lands much nearer Cloudflare Radar's independent finding of 3.9% of the top 200,000 domains passing Markdown negotiation, reported in its Agent Readiness announcement of 17 April 2026.

Link headers announce where the other five are

Every convention above requires an agent to guess a path. llms.txt v2 fixed that by borrowing two RFC 8288 link relations, deliverable as HTML <link> tags or as HTTP Link: headers: rel="describedby" points at the llms.txt covering a page, and rel="alternate" type="text/markdown" points at its Markdown twin.

The header form matters more than it sounds. It works for non-HTML resources, and it can be set at the CDN without touching a single page. Our own homepage, verified on 4 September 2026, answers:

link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json",
      </llms.txt>; rel="describedby"; type="text/markdown",
      </sitemap.xml>; rel="sitemap"; type="application/xml",
      </api/scan>; rel="service-desc"; type="text/event-stream"

Four relations, one header, no path guessing required.

D3 passes on 52.2% of the 477 sites, which is much higher than it looks, because our D3 accepts any agent-useful relation rather than only the two that llms.txt v2 names. Measured strictly against those two relations across 39 llms.txt-publishing sites on 1 September 2026, rel="describedby" appeared on 4. Say which definition your number uses, because the two differ by an order of magnitude.

The MCP endpoint is the only surface that lets an agent act

The first five surfaces hand an agent something to read. The sixth hands it something to do, and it is missing from every comparison article we could find.

An MCP server card at /.well-known/mcp/server-card.json is the file version. Our P2 check passes on 3.4% of 477 sites, 16 in total, the lowest pass rate in the rubric after x402 payments. That number understates reality badly, and here is the proof, verified on 4 September 2026:

# The well-known path
curl -s -o /dev/null -w '%{http_code}\n' https://www.allbirds.com/.well-known/mcp/server-card.json
# 404

# The live endpoint the platform actually runs
curl -s -X POST https://www.allbirds.com/api/mcp -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
# 200 application/json, with a callable tool listed

The file is absent and the capability is live. On 2 September 2026 we sent that same request to 25 Shopify storefronts and 15 answered with a valid tool list, nine of them exposing a update_cart tool. Our WebMCP measurement covers what that surface exposes and what it does not.

A standards footnote worth carrying: IANA's Well-Known URIs registry held 213 registered suffixes on 2 September 2026. agent-card.json is registered. Neither mcp nor webmcp is. So the file version of this surface is a convention, and the endpoint version is not declared anywhere at all.

How many of the six do real sites actually publish?

Population: every scan in our production database on rubric version 2026.08.2 at site scope, deduplicated to the single latest scan per host, with our own domain removed. That leaves 477 sites scanned between 25 August and 4 September 2026, mean score 56.3. It is the same population definition as our 473-site state-of-the-web study, refreshed a day later. No scanned URL is named here; the corpus is other people's sites.

The gradient is monotonic and steep, roughly 13 points per additional surface. That is partly circular, because five of the six are graded checks, so it is not claimed as an independent effect. What is not circular is the shape of the distribution. A quarter of the web we scanned sits at exactly three surfaces, and those three are almost always robots.txt, a sitemap, and one more.

What having the classic two predicts about the newer four

Do publishers who did the old work do the new work? They do, and the effect is large.

A site with both classic files is seven times more likely to publish llms.txt than a site without them. The plausible reading is not that a sitemap causes an llms.txt, but that both come from the same habit: a team that treats machine readers as an audience.

The counter-population is worth naming. 52 of 477 sites pass both classic files and none of the four newer ones, technically competent sites that finished the 2005 checklist and never started the 2026 one. 75 sites pass none of the six at all, with a mean readiness score of 19.2.

Which surface to add next, based on what you already have

Run the scan first, then branch on what it returns.

That last row is the one people skip: a soft 404 makes every well-known-path probe look successful, including ours.

How to check all six on your own site

The fastest route is one command, free, no account:

npx aiscan-cli yoursite.com

Or paste the URL at aiscan.site. The scan grades all six surfaces in a single pass and names the check IDs used throughout this article: D1 robots.txt present and sane, D2 XML sitemap, D3 Link header for discovery, C1 Markdown content negotiation, C2 /llms.txt, P2 MCP Server Card, plus B1 and B2 for the AI-specific robots.txt layer and E1 for correct 404 handling. The discoverability, content and capabilities reference pages document exactly what each one looks for.

If you would rather check by hand, six commands cover it:

S=https://yoursite.com
curl -s $S/robots.txt | head -30                              # D1, B1, B2
curl -s $S/robots.txt | grep -i '^sitemap:'                   # D2, read the declaration
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' $S/llms.txt          # C2
curl -sI -H 'Accept: text/markdown' $S/ | grep -i content-type               # C1
curl -sI $S/ | grep -i '^link:'                                              # D3
curl -s -o /dev/null -w '%{http_code}\n' $S/.well-known/mcp/server-card.json # P2

# And the one that validates the other five
curl -s -o /dev/null -w '%{http_code}\n' $S/definitely-not-a-real-page-xyz    # E1, must be 404

Run the last command first. If it returns 200, every result above it is unreliable.

Fixing them on WordPress and on Shopify

On WordPress the recurring problem is not knowing what to publish, it is three plugins each believing they own robots.txt. ThinkRank puts all five of the surfaces a WordPress site can control (the robots file, robots meta, sitemaps, schema and llms.txt) behind one plugin, which ends the argument rather than adding a fourth voice to it. Switching costs nothing in re-entered configuration either: it imports what Yoast, Rank Math, SEOPress or All in One SEO already hold. Named honestly: Rank Math and Yoast both have deeper traditional-SEO content analysis and far larger install bases, and if that analysis is your daily workflow, they remain reasonable choices. Neither ships llms.txt generation today. Our WordPress platform page lists the check-by-check guides.

On Shopify the constraint is different, because the theme does not give you filesystem access to drop a file at the root. StoreSEO generates llms.txt from products, collections, pages and articles, edits agents.md, and handles schema and AEO markup, which is precisely the half of the stack Shopify's own agent rollout leaves untouched. It is rated 5.0 from 733 reviews on the Shopify App Store as of 4 September 2026 and carries the Built for Shopify badge. Honest alternatives: Yoast SEO for Shopify and Avada SEO both cover schema and metadata well, and neither generates llms.txt or agents.md today. The Shopify platform page has the per-check steps.

For generating a first llms.txt on any stack, our llms.txt generator writes a spec-shaped file from a URL. Be aware of its limit, stated plainly: it generates, it does not validate a file you already have.

Where AIScan fits, and where it doesn't

Two of those rows are our own open bugs. We publish them because a scanner that hides its false negatives is worth less than one that names them.

The six-surface checklist, ten minutes

• Probe a nonsense path. Confirm it returns 404, not 200
• robots.txt exists, returns text/plain, and names at least one AI crawler token explicitly
• robots.txt carries a Sitemap: line pointing at a URL that returns 200 and XML
• Each sitemap file is under 50,000 URLs and 50MB uncompressed
• /llms.txt returns 200 with text/plain and starts with a single # H1 line
• If your docs live under a subpath, publish an llms.txt there too
• Accept: text/markdown on your busiest page returns text/markdown, not text/html
• Response headers carry the two v2 relations, describedby and alternate with type="text/markdown"
• If you run a store, send one JSON-RPC tools/list to your platform's MCP endpoint and read what it exposes

What to do with this

Six surfaces, six questions, and no site needs all six. The evidence from 477 real sites is that most publishers stop at three, that the classic two are finished work, and that the gap between a site scoring 46 and one scoring 88 is two files and one HTTP header.

Start by finding out which of the six you already have. One command, npx aiscan-cli yoursite.com, or your URL pasted into aiscan.site, returns all of them. Read E1 first, because it decides whether the rest mean anything, then D1, D2, D3, C1, C2 and P2. Then work down the decision table above, one surface at a time.

The guides index has the per-platform walkthroughs, and the state-of-the-web study has the full distribution this article's population came from.