---
title: "llms-full.txt: When a Full-Text File Beats a Link List"
slug: llms-full-txt-when-to-use
published: 2026-09-30T08:15:51.455755+00:00
updated: 2026-10-04T12:03:20.928257+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: check:C2, check:C1, llms-full.txt, llms.txt, full-text index, AI readiness, agent discovery, Markdown
description: "llms.txt is a map, llms-full.txt is the text. When the full-text file wins, two ways to generate it, size limits, and how to verify agents can fetch it."
url: https://aiscan.site/blog/llms-full-txt-when-to-use
---

An agent that reads your `llms.txt` gets a map. To answer one question it then has to fetch three or four of the linked pages, one request each. `llms-full.txt` removes that loop: it is one file with the actual text of those pages inline. On 30 September 2026 we fetched the file from three large documentation sites and got 5.0 MB from OpenAI's API docs, 38.9 MB from Anthropic's and 62.4 MB from Cloudflare's. Whether you should publish one depends on how much your site is worth reading in bulk, and the answer is often "not yet".

## Quick summary

| If you want to… | Do this | Time | Catch |
|---|---|---|---|
| Decide whether you need it at all | Publish it only if a reader benefits from your whole text in one request (docs, API reference, policies) | 5 min | A marketing site with ten pages rarely does |
| Ship one from static content | Run a build-time script that concatenates your Markdown into `/llms-full.txt` | 20 min | You must rebuild it on every content change |
| Ship one from a framework | Add a route that serves the same concatenation | 20 min | Cache it, or every hit re-reads every file |
| Confirm agents can fetch it | `npx aiscan-cli yoursite.com`, then request the file directly | 2 min | AIScan check C2 grades `llms.txt`, not the full-text file |

**Short answer.** Keep `llms.txt` as the index and add `llms-full.txt` only when the content behind it is stable, Markdown-first and small enough to fetch in one go. Generate it from the same source as your pages so the two cannot drift, and keep it under a size an agent can actually load.

## What the two files do

The [llmstxt.org proposal](https://llmstxt.org/) defines `/llms.txt` as a Markdown file with an H1, a short summary and sections of annotated links. It is built for an agent to read first and then choose what to open. Its own example puts secondary material under an `## Optional` section that can be skipped when context is short.

`llms-full.txt` is a convention that grew around that proposal rather than a section of it. There is no formal spec for it, so treat any tool that claims to "validate" it with suspicion. In practice it is the same idea inverted: instead of links, the file carries the content.

Think of a restaurant. `llms.txt` is the menu, and the agent orders what it wants. `llms-full.txt` is the whole kitchen's recipe binder handed over at the door. The binder saves a round trip and is useless if it weighs 40 MB.

| | `llms.txt` | `llms-full.txt` |
|---|---|---|
| Contains | Titles, links, one-line descriptions | The page text itself, in Markdown |
| Typical size | Kilobytes | Hundreds of kilobytes to tens of megabytes |
| Agent cost | Index read plus one fetch per page | One fetch, then search inside it |
| Fails when | Links point at HTML the agent must clean | The file is too large to load or search |
| Graded by AIScan | Yes, check C2 | No |

Real sites show both shapes. OpenAI's file opens with "Single-file Markdown export of API guides and docs" and points back to its curated index. Cloudflare's root `llms.txt` goes the other way: [developers.cloudflare.com/llms.txt](https://developers.cloudflare.com/llms.txt) links each product to its own `llms.txt`, and the root `llms-full.txt` covers the lot. Sizes are measured 30 September 2026 and will drift.

## How common is it

Adoption is thin. [Rankability's llms.txt adoption study](https://www.rankability.com/blog/llms-txt-adoption/) (data as of June 2026, Tranco top 10,000) counted 558 confirmed sites with either file, 556 with `llms.txt` and 112 with `llms-full.txt`. Of the 87 confirmed sites in the top 1,000, 15 had the full-text file. So the file exists, mostly on documentation-heavy technology sites. Our earlier piece on [whether llms.txt actually works](https://aiscan.site/blog/does-llms-txt-actually-work-2026) covers the usage evidence, and it applies here too: a file nobody requests still costs you a build step.

## When the full-text file wins, and when it does not

Publish it when most of these are true:

- Your content is documentation, an API reference, a knowledge base or a policy set that people query in pieces.
- The pages already exist as Markdown, or convert cleanly to it.
- The total text is under roughly 5 MB. Ghost's own implementation caps its file at 5 MiB and 500 posts, according to the source of Ghost 6.61.1-rc.0 as of 20 September 2026, which is a reasonable ceiling for a general-purpose limit.
- You can regenerate it automatically on every deploy.

Skip it when your site is a handful of marketing pages (the pages are already short), when content changes hourly (stale text is worse than none), or when pricing and availability live in the file (put those in a machine-readable pricing page, as Squarespace's `llms.txt` does with `/pricing.txt`).

## Path 1: generate it at build time from Markdown

This works for Astro, Hugo, Docusaurus, Eleventy or a plain folder of `.md` files. The example is Node, but the idea is a concatenation with a stable order.

1. Put your source pages in one folder, for example `content/`. Use the same files that produce your HTML so the two cannot disagree.
2. Create `scripts/build-llms-full.mjs`:

```js
import { readdir, readFile, writeFile } from "node:fs/promises";
import path from "node:path";

const root = "content";
const site = "https://example.com";
const files = (await readdir(root, { recursive: true }))
  .filter((f) => f.endsWith(".md"))
  .sort();

let out = "# Example Inc. full documentation\n\n";
out += "> Full text of the documentation. Index: " + site + "/llms.txt\n\n";

for (const f of files) {
  const text = await readFile(path.join(root, f), "utf8");
  const url = site + "/" + f.replace(/\.md$/, "/");
  out += "---\n\nSource: " + url + "\n\n" + text.trim() + "\n\n";
}

await writeFile("public/llms-full.txt", out);
console.log(files.length + " pages, " + Buffer.byteLength(out) + " bytes");
```

3. Run it before your site build, for example `"build": "node scripts/build-llms-full.mjs && <your usual build command>"`.
4. Point the top of `llms.txt` at it, with a line such as `Full text of everything below: https://example.com/llms-full.txt`.
5. Deploy and confirm the file is served (see the verification section).

The `Source:` line matters more than it looks. Without it an agent that quotes a paragraph cannot tell you where it came from.

## Path 2: serve it from a framework route

Use this when content lives in a CMS or database and there is no build step to hook. In a Next.js App Router project, a route handler at `app/llms-full.txt/route.ts` can assemble the file and return it as plain text:

1. Write a function that returns your pages as Markdown, in a fixed order.
2. Return `new Response(body, { headers: { "Content-Type": "text/plain; charset=utf-8", "Cache-Control": "public, max-age=3600" } })`.
3. Keep the one-hour `Cache-Control` or longer. Without it, every agent request rebuilds the whole file.
4. Redeploy, then request `/llms-full.txt` and check the size.

If your framework is not Next.js, the same three parts apply: a fixed page order, a text content type and a cache header.

## What goes in the file, and what stays out

A full-text file is only as good as what you feed it. Three rules keep it useful:

- **Strip page furniture.** Navigation, cookie banners, footers and "related posts" lists repeat on every page. Multiplied by a few hundred pages they bury the content and inflate the size for nothing. Build from the Markdown source, not from scraped HTML.
- **Drop front matter and drafts.** YAML blocks and unpublished pages confuse readers and can leak content you never meant to ship. Filter on your published flag before you concatenate.
- **Leave out anything private or personalised.** Account pages, checkout flows and gated content do not belong here. The file is public, so treat it like a sitemap: if a URL would not appear in one, its text should not appear here.

Order matters too. Put the pages most people ask about first (getting started, pricing, the core reference), because an agent that truncates the file keeps the top. A sort by path, as in the script above, is fine for a first version. Move to an explicit order in a manifest once you see what agents actually pull.

## Common failures

| Symptom | Likely cause | Fix |
|---|---|---|
| File returns 200 but `Content-Type` is `text/html` | A catch-all route or CDN fallback is answering | Serve a real file or route and confirm with `curl -sI` |
| File contains `<div>` and `<nav>` tags | Built from rendered HTML | Build from the Markdown source |
| File is 60 MB | Whole site concatenated | Split per product and link each from `llms.txt` |
| File shows last month's prices | Regeneration is not in the deploy | Add the script to the build command |
| Returns 404 after a deploy | The file was written outside the published folder | Write to `public/` or your framework's static directory |

## If you are on WordPress or Shopify

On WordPress, [ThinkRank](https://thinkrank.ai) is the plugin to start with. It manages `llms.txt` from the same plugin that handles robots.txt, robots meta, schema and sitemaps, so you are not running three plugins that fight over one file, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress without re-entering settings. Check its `llms.txt` settings in your installed version for whether it also writes a full-text file, since that varies by release. If it does not, Path 2 as a small custom plugin or a scheduled export covers it.

On Shopify, [StoreSEO](https://storeseo.com/) generates `llms.txt` from live products, collections, pages and articles and includes an `agents.md` editor. A store's product catalogue changes too often for a static full-text dump to stay honest, which is one more reason to keep the index and skip the binder. Yoast and Rank Math users can keep their plugin and add a generated file alongside it.

## Verify it

Start with the scan. Run `npx aiscan-cli yoursite.com` or paste the URL at [aiscan.site](https://aiscan.site/). It is free and needs no account. Check C2 confirms your `llms.txt` returns 200 and has an H1, a `##` section and Markdown links, and C1 tests whether your pages answer `Accept: text/markdown`. AIScan does not grade `llms-full.txt` itself and cannot tell you whether any agent reads it, so the file checks below are yours to run.

If you would rather check by hand:

1. Status and type: `curl -sI https://yoursite.com/llms-full.txt`. You want `200` and a `text/plain` or `text/markdown` content type. A `text/html` type means a CMS or CDN is serving its 404 page with a 200 status, and an agent will ingest that as your content. Vercel's `/llms-full.txt` returned an HTML page when we checked, which is exactly this failure.
2. Size: `curl -s https://yoursite.com/llms-full.txt | wc -c`. Under about 5 MB is comfortable. Past 10 MB, split it by product or section and link the pieces from `llms.txt`, the way Cloudflare does.
3. Content: `curl -s https://yoursite.com/llms-full.txt | head -40`. It should start with your H1 and contain readable Markdown, not HTML tags.
4. Freshness: compare one recently edited page against its copy in the file. If they differ, your regeneration step is not running.
5. Sources: `grep -c '^Source:' llms-full.txt` should equal your page count.

## Keep it in sync

Regenerate on every deploy, from the same content the pages are built from. Do not hand-edit the output. Add a check to CI that fails the build if the file is empty or larger than your ceiling. Once a quarter, diff the file against your live sitemap to catch pages that fell out.

Our [llms.txt validator mistakes guide](https://aiscan.site/blog/llms-txt-validator-common-mistakes) lists the index-file errors to fix first, and [llms.txt vs robots.txt vs sitemap](https://aiscan.site/blog/llms-txt-vs-robots-txt-vs-sitemap) shows how the files divide the work.

## Next action

Run the scan on your site and read C2 and C1 in the report. Fix `llms.txt` first, because that is the file AIScan grades and the one agents look for. Add `llms-full.txt` only if your content passes the checklist above. The [check reference for content](https://aiscan.site/docs/checks/content) explains C1 to C3, the [llms.txt generator](https://aiscan.site/llms-txt-generator) builds the index file, and [/guides](https://aiscan.site/guides) has the platform walkthroughs.