How Big Is an llms.txt? Size and Structure Across 779 Real Sites

Of the 779 sites in our scan database on 28 October 2026, 427 served an /llms.txt file that returned a 200, and the typical file that passes our check is small: a median of 4,403 bytes, with 80% of passing files between 1,605 and 39,550 bytes. The files that fail to pass despite answering 200 are a different animal. Their median is 33,308 bytes, 52 of the 57 lack a top-level heading, and 37 files are over 100 KB, which looks like a full-text dump or an app shell served at the index path rather than an index.

This post is a census of what AIScan's check C2 stored for those files: size, and whether the file has a heading, sections and links. It is written by M Asif Rahman for site owners who want to know how big an llms.txt should be and what structure the passing ones share. The evidence has no file bodies, so what we can say is limited to size and three structure flags, and the limits section says where that stops.

Quick summary

For site owners: aim for a file that fits in one screen of an index: a heading, a few sections, links with one-line descriptions, a few KB. For anyone building a generator: the failure to avoid is the 100 KB file with no heading.

How we measured it, and what it cannot tell you

The corpus. Every AIScan report stores the result of each check. We took the latest site-scope scan per host under rubric 2026.08.2, excluded aiscan.site, and read the C2 result for each. On 28 October 2026 that was 779 hosts. We name no scanned site. The sites are ones people chose to scan, not a random sample of the web, and the platform mix (a lot of Next.js and WordPress) reflects who uses a free agent-readiness scanner.

What C2 stores. The check requests /llms.txt and records the status, the byte count and three flags: has H1, has sections, has links. According to the check's documentation, it looks for a file that returns 200 with an H1, at least one ## section and markdown links, following the llms.txt proposal. A passing file in the data does not always have all three: 338 passes have all three flags, 30 have an H1 and sections but no links, and 2 have an H1 and links but no sections. Every pass has an H1.

What it cannot tell you. The evidence holds no body. We do not know whether the links resolve, whether the Markdown is valid, whether a 100 KB file is a real llms-full.txt, HTML, or JSON. Our validator study read 39 files in full, and this post does not repeat it. We also do not re-argue whether the file changes what AI systems do; that question has its own post. And a fail row here has no byte count at all: all 352 failures are cases where no file answered 200, so size says nothing about them.

Scores. The "average score" figures below are the overall AIScan score of the host, not a C2 score, so they show what else tends to be true of the site, not an effect of the file.

The size ladder

We cut the 427 files that answered 200 into five rungs by byte count. "Pass" and "partial" are C2's own statuses.

Source: AIScan scan database, rubric 2026.08.2, latest site-scope scan per host, aiscan.site excluded, n = 427 files with a 200 response out of 779 hosts, 28 October 2026.

Three readings follow from the table.

The middle is crowded. 280 of 427 files (65.6%) are between 500 B and 10 KB. That is the size of a short index: a name, a paragraph, three to ten sections and a link list. If you are writing one by hand, you will land here without trying.

Tiny files are mostly broken, not minimal. Nine files are under 500 bytes and five of them are partials. The smallest passing file in the next band is 518 bytes, so a file under 500 B is short enough that it is probably a placeholder or a single line. We cannot see the bodies to say which. Four still pass, which says the check accepts a very small file that has the right shape.

Size alone does not fail a file, but it correlates with the wrong shape. In the 10 KB to 99 KB band 85% pass. Above 100 KB the pass share halves to 49%. The next section shows why: the big partials are overwhelmingly missing a heading.

The overall distribution for passes:

A median partial that is 7.6 times the size of a median pass is the signal in this census.

Structure: what the passes share

C2's evidence strings come in a handful of shapes. We grouped on the shape with the number replaced, so each row is one combination of flags.

Same corpus and date, n = 427.

An H1 is the dividing line. All 370 passes have one. Of the 57 partials, 52 do not. The five that do have an H1 are the tiny ones: a heading and nothing else, median 268 bytes. In other words, the partial population is two groups. One is a heading with no content. The other, much larger, is a file with content and no heading.

Sections or links are enough with the H1, though most files have both. 338 of 370 passes (91.4%) have all three. The 32 that miss one are rare enough that we would not design around them. Write all three.

The 52 headless partials are large. The 44 with no flags have a median of 39,835 bytes, the six with links only a median of 107,834. A 40 KB file with no H1, no ## and no markdown links is not an index of anything in the llms.txt sense. The stored evidence cannot say what it is. Our reading is that it is prose, HTML or an app shell, because a serious llms-full file would normally open with a title. That is a reading, not a measurement, and we did not fetch third-party files to confirm it.

Over 100 KB, a heading is the exception. Of the 37 files at 100 KB and over, 19 have no H1. The two largest shapes above (265,974 B median with sections and links but no H1, 107,834 B with links only) are the extreme end. If you generate a file from your whole site and it comes out above 100 KB, check the first line before anything else.

By platform

For platforms with at least 15 hosts in the corpus. "Serves a file" is the share of that platform's hosts whose /llms.txt answered 200.

Same corpus and date. Shopify has 16 hosts, so treat its row as indicative only.

Existence differs far more than size. Serving a file ranges from 40% to 94% across platforms, while median pass size sits between 3.5 KB and 9.6 KB. Astro hosts are the likeliest to serve a file among the larger groups and every one that served a file passed (34 of 34). We cannot say whether that is the framework's docs tooling, a plugin, or the sort of site that picks Astro.

WordPress has the long tail. 17 of the 37 oversize files are on WordPress sites, 15.2% of the WordPress files, against 5.8% on Next.js. A file generated from a post list grows with the site, which is one possible reason. We did not check how each file was made, so this is a pattern in the data and not a finding about any plugin.

What the data says about how big yours should be

The census answers "what exists", not "what works best for agents", and the llmstxt.org proposal sets no size limit. Within that limit, here is what the passing files do.

For the full-text case, our post on llms-full.txt covers when a second file is worth having. The idea is the same as in the data: keep /llms.txt as the index and let the big file live elsewhere.

What a good file looks like

Here is the shape the 338 full passes share, with about 1 KB of content:

# Acme Analytics

> Privacy-first web analytics for small teams. Open source, self-hostable.

## Docs
- [Quick start](https://example.com/docs/start): Install the script and see your first pageview.
- [API reference](https://example.com/docs/api): Endpoints, parameters and examples.

## Product
- [Pricing](https://example.com/pricing): Plans and limits.
- [Changelog](https://example.com/changelog): What shipped, by date.

## Optional
- [Blog](https://example.com/blog): Guides and release notes.

That has an H1, a one-line summary, three ## sections and links with short descriptions, in under 800 bytes. If you build your file with a generator, the AIScan llms.txt generator produces this layout, and the per-platform steps are in the llms.txt setup guide for every platform.

Fixing a file that is flagged partial

The data says the cause is almost always a missing top-level heading, so check in this order:

1. Fetch the file and read the first line. If it does not start with # Your Site Name, add it. In our data 52 of the 57 partials have no H1, so this is the likeliest single fix.
2. Check what is being served at the path. An app shell or an HTML page served with a 200 at /llms.txt will read as a headless file. Your framework's catch-all route is a common cause.
3. Add at least one ## section and some links in Markdown form, [title](url).
4. If the file is over 100 KB, split it. Keep the index small and move the full text to a second file.
5. Re-scan. Run npx aiscan-cli yoursite.com or paste the address at aiscan.site, free and with no account, and read the C2 line.

If you run WordPress, ThinkRank writes /llms.txt from the same settings that control robots.txt and sitemaps, so one plugin handles the files that would otherwise fight over the same paths, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Other SEO plugins also generate the file, and the size pattern above applies to any of them: open the output and look at the first line. On Shopify, StoreSEO is our default recommendation for store AEO work; whichever tool writes the file, check its first line against the shape above.

Notes for the AIScan team

Two things in the stored evidence limit what a census can say, and we have written them up for the product backlog. The C2 evidence holds no body or content type, so an HTML page served with a 200 cannot be told apart from Markdown except by the missing flags. And the pass rule in the data (an H1 plus sections or links) is looser than the rule the docs page states (all three). Both are small changes: store the content type and a short prefix of the body, and either tighten the rule or update the page.

Next step

Scan your site free at aiscan.site or with npx aiscan-cli yoursite.com and read check C2 on the content checks page. If C2 says partial, check the first line of the file against the layout above. The rest of the checks are in the AIScan guides.