---
title: "Does llms.txt Actually Work in 2026? Google Ignores It, Chrome Audits It, 97% Get Zero Requests"
slug: does-llms-txt-actually-work-2026
published: 2026-08-26T03:32:43.57141+00:00
updated: 2026-08-31T02:45:12.822731+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: llms.txt, AI crawlers, AI search, Lighthouse, Google Search, technical SEO, check:C2, check:C3, check:E3, check:B1, check:B2
description: "Google Search ignores llms.txt. Chrome's Lighthouse audits it. Ahrefs found 97% of files get zero requests. What the evidence actually supports in 2026."
url: https://aiscan.site/blog/does-llms-txt-actually-work-2026
---

**Updated 26 August 2026.** In June, Google added a line to its own AI search documentation saying that Google Search ignores llms.txt files. Two months earlier, a different Google team shipped a Lighthouse audit in Chrome that checks whether your site has one. Ahrefs then looked at server logs across 137,210 domains and found that 97% of published llms.txt files received zero requests in a month.

Three facts, all verifiable, all pointing in different directions. Most of what has been written about llms.txt picks one of them and stops. This piece takes all three seriously, because the reason they disagree turns out to be the most useful thing about the topic: they are answers to different questions about different software. Once you separate the consumers, the contradiction dissolves and you get a decision you can actually make.

We should say upfront that AIScan sells a scanner that grades sites on llms.txt as check C2. Writing an honest evidence review about a file we score is awkward, and we would rather be awkward than sell you optimism. Where the evidence says the file does nothing for you, this article says so.

## Quick summary: does llms.txt work in 2026?

**Too long; didn't read?** Here is what the evidence supports as of 26 August 2026.

| If you want to know… | The verified answer | Source and date |
|---|---|---|
| Does Google Search use llms.txt? | No. Google's docs state Search "ignores them" and that publishing one "won't harm (nor help)" rankings | Google Search Central, reported 15 Jun 2026 |
| Do AI retrieval crawlers fetch it? | Almost never. 97% of files got zero requests in May 2026 | Ahrefs, 137,210 domains, 15 Jun 2026 |
| Does anything official audit it? | Yes. Chrome's Lighthouse "Agentic Browsing" category, Chrome 150+ | developer.chrome.com, verified 26 Aug 2026 |
| Is it a real standard? | No formal standards body. A community spec by Jeremy Howard | llmstxt.org, first published 3 Sep 2024 |
| Has the spec changed? | Yes. Version 2 shipped 10 Aug 2026, adding link relations for discovery | llmstxt.org, 10 Aug 2026 |
| How many sites publish one? | Between 8.7% and 28%, depending heavily on the sample | Rankability (Jun 2026), Ahrefs (May 2026) |
| Should you publish one? | Depends on whether you run documentation or a marketing site | See the decision table below |

**If you run documentation, an API reference or a developer product:** publish one. The people pointing coding agents and RAG pipelines at your URLs benefit immediately, and discovery is not the mechanism that makes it work.

**If you run a marketing site, a blog or an e-commerce store:** the file will most likely sit there unread. Fix your server-rendered HTML first. That is what AI crawlers actually fetch.

**If you already have one:** the failure mode worth checking is not absence. It is a file that returns HTML, 404s, or points at pages that moved six months ago.

## What llms.txt is, and what it was actually for

Jeremy Howard of Answer.AI proposed llms.txt on 3 September 2024. The idea was narrow and reasonable. Language models work inside a limited context window, HTML pages are full of navigation, cookie banners and markup that waste that budget, so give the model a curated Markdown map instead. Put it at the root of your domain, list your important pages as Markdown links, and a model can find the good parts without wading through your site chrome.

The spec is short. [**According to llmstxt.org**](https://llmstxt.org/), an H1 with the name of the project is "the only required section." Everything else, including a blockquote summary, some prose, and H2-delimited lists of links, is optional. There is no formal standards body behind it. It is not an IETF draft or a W3C recommendation. It lives on GitHub and is "open for community input," which means adoption happens because platforms and publishers decide to do it, not because anyone is obliged to.

That origin matters for a reason the SEO framing tends to lose. Howard's argument was about context efficiency for a model that has already been pointed at your site. It was never a claim that crawlers would go looking for the file and reward you for having it. The SEO industry supplied that second idea on its own, and it is the part the data has been steadily demolishing.

On 10 August 2026 the spec got its first substantive revision. Version 2 adds two standard link relations: `rel="alternate" type="text/markdown"` to point from an HTML page to its Markdown equivalent, and `rel="describedby"` to point from a page to the llms.txt file that covers it. Both can be delivered as HTML `<link>` tags or HTTP `Link` headers. It also allows flexible Markdown filenames (`page.html.md` or `page.md`) and clarifies that a subpath file like `/docs/llms.txt` scopes to everything beneath it, with the most specific file winning.

Read that update next to the traffic data and it looks like exactly what it is: the spec authors responding to a discovery problem. If nothing finds your file, giving pages a formal way to announce it is the obvious fix to try.

## The case against llms.txt: three independent measurements

The evidence that llms.txt does nothing for AI visibility is not one study. It is three, from different people, using different methods, arriving at the same place.

![Bar chart of the Ahrefs llms.txt study: 137,210 domains analysed, 28% publish a valid file, only 3% of those files received any request in May 2026, and just 19.5% of those requests came from identified AI bots.](https://gqdxuwsuuasfjeeqafyq.supabase.co/storage/v1/object/public/blog-covers/llmstxt-2026-fig1.jpg)

[**According to Ahrefs**](https://ahrefs.com/blog/llmstxt-study/), in research published on 15 June 2026, Louise Linehan and Xibeijia Guan analysed 137,210 domains in Ahrefs Web Analytics that received traffic in May 2026. Of those, 28% published a valid llms.txt, meaning a real HTTP 200 with actual Markdown, with soft-404s screened out. Then they looked at what reached those files. **97% received zero requests.** Roughly 1,100 domains got any request at all, totalling about 22,000 requests. Of the requests that did arrive, 96% came from bots and 4% from humans. When all AI bot categories were combined, AI bots were the largest single bucket at 19.5%, meaning four-fifths of the bot traffic to these files was something other than an AI system. They also noted that AI bots did not probe for llms.txt on sites that lacked one.

The authors are careful about their own limits, and quoting them properly matters more than the headline number. Ahrefs Web Analytics customers "skew more technical and SEO-aware than the web at large, so treat the 28% adoption figure as an upper bound." On the traffic side they note that "requests are the generous measure" and that "whether bots act on what they fetch is another question."

That second caveat cuts both ways, and it is the honest way to read the study. A zero-request file is strong evidence of no retrieval. A file that got fetched once is weak evidence of anything.

The second measurement came earlier. [**According to Search Engine Journal**](https://www.searchenginejournal.com/llms-txt-shows-no-clear-effect-on-ai-citations-based-on-300k-domains/561542/), reporting on 20 November 2025, an SE Ranking study of roughly 300,000 domains found 10.13% adoption and no clear effect on AI citations. The modelling detail is the interesting part: llms.txt added no predictive value to whether a site got cited, and dropping it as a feature improved model accuracy.

The third is Google, repeatedly. John Mueller's position, [**as reported by Search Engine Journal**](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/) in April 2025: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag." Gary Illyes made the same comparison on Bluesky on 1 August 2025, [**quoted by Search Engine Roundtable**](https://www.seroundtable.com/google-parallel-meta-keywords-llmstxt-39862.html): "it's very easy to draw a parallel between 1990's keywords meta tag and this, and we all know how... useful the keywords meta tag became, very fast."

Then in June 2026 Google put it in the documentation, which is where informal opinion becomes something you can plan against. [**According to Search Engine Land's Barry Schwartz**](https://searchengineland.com/google-says-llms-txt-files-wont-harm-or-help-your-search-rankings-480264), reporting the change on 15 June 2026, Google's [AI search optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) now says: "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so won't harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them."

A second line pre-empts a common misreading: "Google Search itself doesn't use them. Note that Google may discover, crawl, and index many kinds of files in addition to HTML on a website: this doesn't mean that the file is treated in a special way." If you see Googlebot fetch your llms.txt in your logs, that is Google crawling a file, not Google using a signal.

Notice how the tone moved. In 2025 the comparison was to the keywords meta tag, close to an insult in SEO. By mid-2026 the official wording is neutral: fine to have, does nothing here. The conclusion never changed.

## The case for llms.txt: Chrome, coding agents, and directed tools

If that were the whole story, this would be a short article ending in "don't bother." It is not, and the reason is that a different part of Google spent 2026 building llms.txt into a browser audit.

![Three-column comparison: retrieval crawlers show no evidence of using llms.txt, agentic browsers audit it but are unproven, and directed tools like coding agents work by construction because a human supplies the URL.](https://gqdxuwsuuasfjeeqafyq.supabase.co/storage/v1/object/public/blog-covers/llmstxt-2026-fig2.jpg)

Chrome's Lighthouse now has an **Agentic Browsing** category. [**According to Chrome's scoring documentation**](https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring), verified 26 August 2026, it covers registered WebMCP tools, forms missing declarative WebMCP, WebMCP schema validity, llms.txt, accessibility for agents, and layout stability. The llms.txt audit sits under "Stability and Discoverability" and checks for a machine-readable summary at the domain root.

Two details about that category deserve more attention than they get. First, it is deliberately not scored the way the rest of Lighthouse is. Chrome states: "Unlike other Lighthouse categories, the Agentic Browsing category does not have a weighted average score from 0 to 100." You get a fractional score and per-audit pass or fail instead. Second, "testing this category requires Chrome 150 or later, and WebMCP audits require registering for the WebMCP origin trial."

A team that believed llms.txt was an established ranking factor would have given it a weighted score. Shipping it unscored is Chrome saying, roughly: we think this class of thing will matter to agents, we are not prepared to tell you how much.

The audit's own reliability is worth knowing about before you treat it as a verdict. An [**issue on the Lighthouse repository**](https://github.com/GoogleChrome/lighthouse/issues/17082), filed on 18 June 2026 and titled "Agentic Browsing audit fails llms.txt for spec-compliant files on PSI Chrome 146," reports the audit returning a fetch failure for a valid file. The reporter's summary: "The error message 'Fetch of llms.txt failed' is misleading, the fetch demonstrably succeeded twice." PageSpeed Insights was running Chrome 146 while the docs require 150 or later. If your Lighthouse run fails this audit, check that before you rewrite your file.

The stronger case for llms.txt has nothing to do with either crawlers or audits. It is the case Howard originally made, and it survives the traffic data intact, because it does not depend on discovery at all. When a developer points a coding agent at your documentation, or an engineer builds a RAG pipeline over your API reference, a human supplies the URL. Nobody has to find the file. A curated Markdown index genuinely beats making a model parse rendered HTML, and documentation platforms including Mintlify generate llms.txt automatically for exactly this reason.

This is the distinction most coverage collapses, so it is worth stating flatly: a platform generating the file is not evidence that an AI system reads it. Anthropic publishes an llms.txt for its own developer documentation. That tells you Anthropic thinks it is useful for people working with its docs. It tells you nothing about whether Claude fetches yours. We looked for documentation from any major AI vendor (OpenAI, Anthropic, Perplexity, Google, Mistral, Cohere) stating that their crawlers read third-party llms.txt files, and found none.

## Why Google's two answers are not a contradiction

Here is the reconciliation, and it is simpler than the discourse suggests.

Google Search ignores llms.txt because Google Search does not need it. Googlebot already crawls your HTML and renders your JavaScript, and it would have to decide whether to trust your summary of yourself over its own crawl. The keywords meta tag comparison is apt for precisely this reason. It is a publisher-controlled assertion, and publisher-controlled assertions get discounted.

Chrome audits llms.txt because an agent has a different budget. An agent working through a task is not indexing the web. It is trying to accomplish one thing, in one session, with a finite context window and a user waiting. For that consumer, a short authoritative map of a site is worth something in a way it is not to a crawler with unlimited passes.

Different consumer, different answer, both correct. The mistake is assuming "AI" is one audience. Retrieval crawlers, agentic browsers and directed tools have almost nothing in common except the acronym.

What follows is a forecast rather than a fact, so treat it as ours and not as evidence: the agentic-browsing consumer Chrome is building for mostly does not exist at scale yet. That is why the audit is unscored and why the traffic data is flat. If it arrives, llms.txt is cheap insurance. If it does not, you have spent an hour on a file.

## Should you publish an llms.txt file? A decision table

Adoption numbers vary enormously by sample, which is itself informative. [**According to Rankability**](https://rankability.com/data/llms-txt-adoption/), which tracks the Tranco top 1,000 monthly, 87 sites had a valid file as of the June 2026 list. That is 8.7% of the full sample and 15.8% of the 549 domains that were reachable. Ahrefs found 28% in a technical, SEO-aware customer base and called it an upper bound. SE Ranking found 10.13% across 300,000 domains. Nobody is close to majority adoption, and the sites that do publish are the ones most likely to be reading articles like this one.

| Your situation | Publish llms.txt? | Do this first instead |
|---|---|---|
| Developer docs, API reference, SDK | **Yes** | Make sure the linked Markdown is current |
| Open-source project or technical product | **Yes** | Keep it short and point at canonical pages |
| SaaS with a docs subdirectory | **Yes, scoped** to `/docs/llms.txt` | Use v2 subpath scoping, not one giant root file |
| Marketing site or brand site | **Optional, low priority** | Server-rendered HTML, then structured data |
| Blog or publisher | **Optional, low priority** | Check your posts render without JavaScript |
| E-commerce store | **No, not yet** | Product schema and crawlable category pages |
| Site that already 404s the file | **Fix or remove it** | A broken file is worse than no file |

The honest summary of that table: llms.txt is a documentation tool that got marketed as an SEO tool. Judged as documentation it works fine. Judged as SEO it has failed three separate measurements.

If you run WordPress and want the file maintained rather than hand-written, [**ThinkRank**](https://thinkrank.ai) — our own WordPress SEO plugin — generates llms.txt automatically alongside robots.txt, robots meta and schema, which removes the staleness problem described next. It migrates existing directives from Rank Math, Yoast, All in One SEO and SEOPress, so adopting it does not mean re-entering settings you have already made. To see what a correct file looks like for your site before committing to anything, our [llms.txt generator](https://aiscan.site/llms-txt-generator) is free and needs no account.

## Seven mistakes that break an llms.txt file

Every one of these is something a validator or a scanner will flag, and most are more damaging than not having the file at all.

**1. It returns HTML, not text.** The single most common failure. Your framework's catch-all route serves the SPA shell for any unmatched path, so `/llms.txt` returns a 200 with a full HTML document. Ahrefs had to screen for exactly this. Test with `curl -sI https://yoursite.com/llms.txt` and confirm `content-type: text/plain`.

**2. It 404s in production but works locally.** Static files in the wrong build directory. Check the deployed URL, never the dev server.

**3. The links point at pages that moved.** A stale index is worse than none, because the tool that trusts it gets 404s and gives up. If you cannot commit to updating it, generate it.

**4. It is a full content dump.** Version 2 clarifies that agents should read and search the file and follow links, rather than treat it as the entire corpus. A 400KB file defeats the context-efficiency argument that justifies the file existing.

**5. It has no H1.** The one genuinely required element in the spec. Files without it are not spec-compliant, even if they look fine.

**6. Nothing points at it.** Before v2, discovery was "hope something guesses the root path." If you are publishing now, add the v2 link relations: `rel="describedby"` from your pages to the file, and `rel="alternate" type="text/markdown"` from a page to its Markdown version.

**7. You built it instead of fixing your rendering.** The one that actually costs you traffic. A file listing pages that return an empty DOM to non-JavaScript crawlers is an index of content nothing can read.

We know that last one from the inside. This blog shipped 73 words and zero headings to any crawler that did not execute JavaScript until 25 August 2026, because a sanitiser library pulled a Node-only dependency into a Cloudflare Worker render path and React silently discarded the server render. It looked perfect in a browser. Publishing an llms.txt during that window would have been an index pointing at nothing.

## What AIScan checks, and what no scanner can see

AIScan grades llms.txt as **check C2** in the content dimension, described on our [content checks page](https://aiscan.site/docs/checks/content). We fetch the file, confirm it returns a real `text/plain` response rather than an HTML shell, and check it against the spec's structural requirements. C2 is one check among the content dimension's five, weighted well below **C3 and E3**, which test whether your page body is present in server-rendered HTML at all. That ordering is deliberate and reflects the evidence in this article. llms.txt is a nice-to-have. Server-rendered content decides whether an AI system can read you.

Alongside it, **B1** checks your Content Signals policy and **B2** checks whether your robots.txt has explicit rules for named AI crawlers. Those live on our [bot access checks page](https://aiscan.site/docs/checks/bot-access), and they matter more than llms.txt because they govern access rather than convenience. Our post on [Cloudflare's Search, Agent and Training categories](https://aiscan.site/blog/cloudflare-search-agent-training-crawler-categories) covers how those three settings interact, and the [15 September 2026 default change](https://aiscan.site/blog/cloudflare-ai-bot-block-googlebot-2026) is worth understanding before you touch any of it.

Here is what AIScan cannot tell you, and no scanner can:

- **Whether anything ever read your llms.txt.** We confirm the file exists and parses. Only your server logs show whether a request arrived. If this question matters to you, grep your access logs for `llms.txt`. That is the measurement Ahrefs made, and you can make it for your own domain in one command.
- **Whether the pages your file links to are accurate or current.** We check the file, not your editorial hygiene.
- **Whether your content is worth citing.** A perfectly formed llms.txt on thin content is a well-organised index of nothing.
- **Whether an AI system actually cited you.** That is visibility monitoring, a different product category, and any scanner claiming to measure it from a crawl is guessing.

You do not need AIScan for any of this. Every check here is one you can run yourself with `curl`, your server logs and a text editor. What a scanner buys you is doing it on a schedule, across a whole site, without having to remember. If you would rather do it by hand, the [guides](https://aiscan.site/guides) explain how.

## The honest bottom line

llms.txt has failed as an SEO tactic. Three independent measurements say the same thing, Google has now said it in documentation, and anyone selling you the file as a route to ChatGPT citations is selling something the evidence does not support.

It has not failed as a documentation practice, and the difference is who supplies the URL. When a human points a tool at your docs, a curated Markdown index earns its keep immediately and always did. That was the original argument. It was never really about crawlers.

Chrome's audit and the version 2 spec are two institutions betting that a third consumer, the agentic browser, will make discovery matter eventually. That bet might pay off. It has not yet, the audit is unscored for that reason, and the file costs about an hour, which is why "publish one if you run docs, deprioritise it otherwise" is the position the evidence actually supports.

Fix your server-rendered HTML first. That is not a plug for anything. It is the thing currently deciding whether AI systems can read your site, while everyone argues about a text file.

If you want to know where your own site stands on that, [AIScan](https://aiscan.site) grades it as **C3** and **E3** — structured HTML, and how much of your copy exists before JavaScript runs — alongside llms.txt as C2, so you can see the thing that matters and the thing everyone argues about in the same report. It is free for 20 scans a month and needs no account:

```bash
npx aiscan-cli yoursite.com
```

More platform walkthroughs are in [our guides index](https://aiscan.site/guides).