llms-full.txt: When a Full-Text File Beats a Link List

Table of contents

Quick summary
What the two files do
How common is it
When the full-text file wins, and when it does not
Path 1: generate it at build time from Markdown
Path 2: serve it from a framework route
What goes in the file, and what stays out
Common failures
If you are on WordPress or Shopify
Verify it
Keep it in sync
Next action

An agent that reads your llms.txt gets a map. To answer one question it then has to fetch three or four of the linked pages, one request each. llms-full.txt removes that loop: it is one file with the actual text of those pages inline. On 30 September 2026 we fetched the file from three large documentation sites and got 5.0 MB from OpenAI's API docs, 38.9 MB from Anthropic's and 62.4 MB from Cloudflare's. Whether you should publish one depends on how much your site is worth reading in bulk, and the answer is often "not yet".

Quick summary

If you want to…	Do this	Time	Catch
Decide whether you need it at all	Publish it only if a reader benefits from your whole text in one request (docs, API reference, policies)	5 min	A marketing site with ten pages rarely does
Ship one from static content	Run a build-time script that concatenates your Markdown into `/llms-full.txt`	20 min	You must rebuild it on every content change
Ship one from a framework	Add a route that serves the same concatenation	20 min	Cache it, or every hit re-reads every file
Confirm agents can fetch it	`npx aiscan-cli yoursite.com`, then request the file directly	2 min	AIScan check C2 grades `llms.txt`, not the full-text file

Short answer. Keep llms.txt as the index and add llms-full.txt only when the content behind it is stable, Markdown-first and small enough to fetch in one go. Generate it from the same source as your pages so the two cannot drift, and keep it under a size an agent can actually load.

What the two files do

The llmstxt.org proposal defines /llms.txt as a Markdown file with an H1, a short summary and sections of annotated links. It is built for an agent to read first and then choose what to open. Its own example puts secondary material under an ## Optional section that can be skipped when context is short.

llms-full.txt is a convention that grew around that proposal rather than a section of it. There is no formal spec for it, so treat any tool that claims to "validate" it with suspicion. In practice it is the same idea inverted: instead of links, the file carries the content.

Think of a restaurant. llms.txt is the menu, and the agent orders what it wants. llms-full.txt is the whole kitchen's recipe binder handed over at the door. The binder saves a round trip and is useless if it weighs 40 MB.

	`llms.txt`	`llms-full.txt`
Contains	Titles, links, one-line descriptions	The page text itself, in Markdown
Typical size	Kilobytes	Hundreds of kilobytes to tens of megabytes
Agent cost	Index read plus one fetch per page	One fetch, then search inside it
Fails when	Links point at HTML the agent must clean	The file is too large to load or search
Graded by AIScan	Yes, check C2	No

Real sites show both shapes. OpenAI's file opens with "Single-file Markdown export of API guides and docs" and points back to its curated index. Cloudflare's root llms.txt goes the other way: developers.cloudflare.com/llms.txt links each product to its own llms.txt, and the root llms-full.txt covers the lot. Sizes are measured 30 September 2026 and will drift.

How common is it

Adoption is thin. Rankability's llms.txt adoption study (data as of June 2026, Tranco top 10,000) counted 558 confirmed sites with either file, 556 with llms.txt and 112 with llms-full.txt. Of the 87 confirmed sites in the top 1,000, 15 had the full-text file. So the file exists, mostly on documentation-heavy technology sites. Our earlier piece on whether llms.txt actually works covers the usage evidence, and it applies here too: a file nobody requests still costs you a build step.

When the full-text file wins, and when it does not

Publish it when most of these are true:

Your content is documentation, an API reference, a knowledge base or a policy set that people query in pieces.
The pages already exist as Markdown, or convert cleanly to it.
The total text is under roughly 5 MB. Ghost's own implementation caps its file at 5 MiB and 500 posts, according to the source of Ghost 6.61.1-rc.0 as of 20 September 2026, which is a reasonable ceiling for a general-purpose limit.
You can regenerate it automatically on every deploy.

Skip it when your site is a handful of marketing pages (the pages are already short), when content changes hourly (stale text is worse than none), or when pricing and availability live in the file (put those in a machine-readable pricing page, as Squarespace's llms.txt does with /pricing.txt).

Path 1: generate it at build time from Markdown

This works for Astro, Hugo, Docusaurus, Eleventy or a plain folder of .md files. The example is Node, but the idea is a concatenation with a stable order.

Put your source pages in one folder, for example content/. Use the same files that produce your HTML so the two cannot disagree.
Create scripts/build-llms-full.mjs:

import { readdir, readFile, writeFile } from "node:fs/promises";
import path from "node:path";

const root = "content";
const site = "https://example.com";
const files = (await readdir(root, { recursive: true }))
  .filter((f) => f.endsWith(".md"))
  .sort();

let out = "# Example Inc. full documentation\n\n";
out += "> Full text of the documentation. Index: " + site + "/llms.txt\n\n";

for (const f of files) {
  const text = await readFile(path.join(root, f), "utf8");
  const url = site + "/" + f.replace(/\.md$/, "/");
  out += "---\n\nSource: " + url + "\n\n" + text.trim() + "\n\n";
}

await writeFile("public/llms-full.txt", out);
console.log(files.length + " pages, " + Buffer.byteLength(out) + " bytes");

Run it before your site build, for example "build": "node scripts/build-llms-full.mjs && <your usual build command>".
Point the top of llms.txt at it, with a line such as Full text of everything below: https://example.com/llms-full.txt.
Deploy and confirm the file is served (see the verification section).

The Source: line matters more than it looks. Without it an agent that quotes a paragraph cannot tell you where it came from.

Path 2: serve it from a framework route

Use this when content lives in a CMS or database and there is no build step to hook. In a Next.js App Router project, a route handler at app/llms-full.txt/route.ts can assemble the file and return it as plain text:

Write a function that returns your pages as Markdown, in a fixed order.
Return new Response(body, { headers: { "Content-Type": "text/plain; charset=utf-8", "Cache-Control": "public, max-age=3600" } }).
Keep the one-hour Cache-Control or longer. Without it, every agent request rebuilds the whole file.
Redeploy, then request /llms-full.txt and check the size.

If your framework is not Next.js, the same three parts apply: a fixed page order, a text content type and a cache header.

What goes in the file, and what stays out

A full-text file is only as good as what you feed it. Three rules keep it useful:

Strip page furniture. Navigation, cookie banners, footers and "related posts" lists repeat on every page. Multiplied by a few hundred pages they bury the content and inflate the size for nothing. Build from the Markdown source, not from scraped HTML.
Drop front matter and drafts. YAML blocks and unpublished pages confuse readers and can leak content you never meant to ship. Filter on your published flag before you concatenate.
Leave out anything private or personalised. Account pages, checkout flows and gated content do not belong here. The file is public, so treat it like a sitemap: if a URL would not appear in one, its text should not appear here.

Order matters too. Put the pages most people ask about first (getting started, pricing, the core reference), because an agent that truncates the file keeps the top. A sort by path, as in the script above, is fine for a first version. Move to an explicit order in a manifest once you see what agents actually pull.

Common failures

Symptom	Likely cause	Fix
File returns 200 but `Content-Type` is `text/html`	A catch-all route or CDN fallback is answering	Serve a real file or route and confirm with `curl -sI`
File contains `<div>` and `<nav>` tags	Built from rendered HTML	Build from the Markdown source
File is 60 MB	Whole site concatenated	Split per product and link each from `llms.txt`
File shows last month's prices	Regeneration is not in the deploy	Add the script to the build command
Returns 404 after a deploy	The file was written outside the published folder	Write to `public/` or your framework's static directory

If you are on WordPress or Shopify

On WordPress, ThinkRank is the plugin to start with. It manages llms.txt from the same plugin that handles robots.txt, robots meta, schema and sitemaps, so you are not running three plugins that fight over one file, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress without re-entering settings. Check its llms.txt settings in your installed version for whether it also writes a full-text file, since that varies by release. If it does not, Path 2 as a small custom plugin or a scheduled export covers it.

On Shopify, StoreSEO generates llms.txt from live products, collections, pages and articles and includes an agents.md editor. A store's product catalogue changes too often for a static full-text dump to stay honest, which is one more reason to keep the index and skip the binder. Yoast and Rank Math users can keep their plugin and add a generated file alongside it.

Verify it

Start with the scan. Run npx aiscan-cli yoursite.com or paste the URL at aiscan.site. It is free and needs no account. Check C2 confirms your llms.txt returns 200 and has an H1, a ## section and Markdown links, and C1 tests whether your pages answer Accept: text/markdown. AIScan does not grade llms-full.txt itself and cannot tell you whether any agent reads it, so the file checks below are yours to run.

If you would rather check by hand:

Status and type: curl -sI https://yoursite.com/llms-full.txt. You want 200 and a text/plain or text/markdown content type. A text/html type means a CMS or CDN is serving its 404 page with a 200 status, and an agent will ingest that as your content. Vercel's /llms-full.txt returned an HTML page when we checked, which is exactly this failure.
Size: curl -s https://yoursite.com/llms-full.txt | wc -c. Under about 5 MB is comfortable. Past 10 MB, split it by product or section and link the pieces from llms.txt, the way Cloudflare does.
Content: curl -s https://yoursite.com/llms-full.txt | head -40. It should start with your H1 and contain readable Markdown, not HTML tags.
Freshness: compare one recently edited page against its copy in the file. If they differ, your regeneration step is not running.
Sources: grep -c '^Source:' llms-full.txt should equal your page count.

Keep it in sync

Regenerate on every deploy, from the same content the pages are built from. Do not hand-edit the output. Add a check to CI that fails the build if the file is empty or larger than your ceiling. Once a quarter, diff the file against your live sitemap to catch pages that fell out.

Our llms.txt validator mistakes guide lists the index-file errors to fix first, and llms.txt vs robots.txt vs sitemap shows how the files divide the work.

Next action

Run the scan on your site and read C2 and C1 in the report. Fix llms.txt first, because that is the file AIScan grades and the one agents look for. Add llms-full.txt only if your content passes the checklist above. The check reference for content explains C1 to C3, the llms.txt generator builds the index file, and /guides has the platform walkthroughs.

Frequently asked questions

What is the difference between llms.txt and llms-full.txt?

llms.txt is a curated index of links with short descriptions, and llms-full.txt carries the page text itself. The index has an agreed format at llmstxt.org. The full-text file is a convention with no formal spec.

Do I need llms-full.txt if I already have llms.txt?

Not usually. It pays off for documentation, API references and policy content that agents query in pieces. A marketing site with a handful of short pages gains little, and AIScan check C2 grades only llms.txt.

My llms-full.txt returns 200 but agents read HTML. What is wrong?

A catch-all route or CDN fallback is answering with your 404 or app shell page and a 200 status. Run curl -sI on the URL and check the content type is text/plain or text/markdown, then serve a real file.

My llms-full.txt is 404 after deploy. How do I fix it?

The file was almost certainly written outside the folder your host publishes. Write it to public/ or your framework's static directory, confirm it appears in the build output and redeploy.

The file shows old content after I edited a page. Why?

Regeneration is not part of your deploy, or a cache is holding the old copy. Add the generator to the build command and lower the Cache-Control max-age, then compare one edited page with its copy in the file.

How large can llms-full.txt be?

There is no official limit. Ghost caps its generated file at 5 MiB and 500 posts, while Cloudflare's root file measured 62.4 MB on 30 September 2026. Under about 5 MB is comfortable, and past 10 MB you should split by product.

Does AIScan check llms-full.txt?

No. Check C2 tests that llms.txt returns 200 with an H1, at least one ## section and Markdown links. Check C1 tests Markdown content negotiation. Fetching llms-full.txt by hand is the way to verify it.

Can I generate llms-full.txt on WordPress or Shopify?

Yes. ThinkRank manages llms.txt on WordPress, so check its settings for a full-text option in your version. StoreSEO generates llms.txt on Shopify. Otherwise a scheduled export or a small route that concatenates Markdown produces the file.