---
title: "The complete AI readiness setup for Docusaurus in 2026"
slug: ai-readiness-setup-docusaurus
published: 2026-09-07T08:15:43.214016+00:00
updated: 2026-09-07T08:15:43.214016+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: platform:docusaurus, check:D1, check:D2, check:B2, check:C1, check:C2, check:C3, check:E1, check:E3, check:E5, docusaurus, llms.txt, robots.txt, ai crawlers, documentation
description: "Docusaurus writes your sitemap and feed but no robots.txt or llms.txt. What 15 live sites publish, the static and postBuild paths, and the empty-shell trap."
url: https://aiscan.site/blog/ai-readiness-setup-docusaurus
---
Docusaurus is the framework most likely to be sitting between an AI agent and the answer it is looking for. It runs a very large share of the developer documentation on the web, and developer documentation is exactly the material that retrieval systems reach for. So it is worth knowing what a Docusaurus site actually hands a crawler, rather than what the build log says it produced.
The short version: the discovery files are the easy half. The hard half is whether a given page contains any words at all, and Docusaurus tells you so in its own documentation.
## Quick summary
| Check | What Docusaurus gives you | What you have to do |
|---|---|---|
| D1 robots.txt | Nothing. No file is generated | Drop one in `static/`, or write it in `postBuild` |
| D2 sitemap | `sitemap.xml`, automatically, via `preset-classic` | Nothing, unless `noIndex` is on |
| B2 AI bot rules | Nothing | Named groups in your own robots.txt |
| C2 llms.txt | Nothing in core | `static/llms.txt`, or generate it from real routes |
| C1 Markdown negotiation | Nothing. A true negative | Out of scope without a plugin |
| C3 / E3 page content | Static HTML per route, if the route renders on the server | Audit the routes that do not |
| E5 feed | RSS and Atom plus the `` tags, if you run the blog plugin | Nothing |
| E1 real 404 | A genuine 404 on every host measured | Nothing |
## Why a Docusaurus page can be empty and still be "static"
Every other platform in this series either renders your HTML on a server or compiles it at publish time, and the content question is settled before you arrive. Docusaurus is a React single-page application that emits static HTML as a performance optimisation. That is not an outside characterisation. The framework's own Static Site Generation page, fetched from `docusaurus.io/docs/advanced/ssg` and verified on 7 September 2026, says it in its own words:
> Docusaurus is ultimately a single-page application, so static site generation is only an optimization (progressive enhancement, as it's called), but our functionality does not fully depend on those HTML files.
The same page, in its own words, explains that "the theme is built twice", once through React DOM Server and once for the browser, and describes what the failure looks like: "in CSR-only apps, all DOM elements are generated on client side with React, and the HTML file only ever contains one root element for React to mount DOM to."
That root element is ``. When a route escapes server rendering, that div ships empty and the page is a shell.
This is not theoretical. Fetching the Cypress documentation home page on 7 September 2026 with an ordinary desktop browser user agent returned 6,826 bytes containing exactly that: an empty `__docusaurus` div, `data-has-hydrated="false"`, and two words of visible text. Our own scanner grades that site **72 / Level 4** with C3 and E3 both failing on the evidence string "2 words of server-rendered text, 0 ``, 0 ``". The same site's robots.txt names eight AI crawlers and its llms.txt is roughly 70 KB. It did all the discovery work and its front door says nothing.
To be fair to it: the failure is one route, not the site. Two documentation pages on the same host returned 2,196 and 3,897 words with a single `` each. That is the shape to remember. On Docusaurus, C3 and E3 are a per-route build outcome rather than a platform guarantee.
## What fifteen live Docusaurus sites actually publish
We fingerprinted thirty candidate documentation domains and confirmed fifteen as Docusaurus, then probed six paths on each: ninety requests, 7 September 2026. The confirmed hosts were:
```
docusaurus.io jestjs.io reactnative.dev
redux.js.org prettier.io babeljs.io
create-react-app.dev electronjs.org sequelize.org
typeorm.io docs.dagger.io docusaurus.community
docs.cypress.io recoiljs.org docs.getdbt.com
```
| Surface | Result across 15 sites |
|---|---|
| `` | **15 of 15**, versions 2.0.0-beta.14 to 3.10.2 |
| `/sitemap.xml` returns 200 | **15 of 15** |
| Real 404 for an unknown `.txt` path | **15 of 15**, no soft 200 anywhere |
| `/robots.txt` returns 200 | 9 of 15 |
| robots.txt naming any AI crawler | **1 of 15** |
| `/llms.txt` returns 200 | 6 of 15 |
| `/blog/rss.xml` returns 200 | 9 of 15 |
| Feed declared in `` | **9 of 9**, automatically |
Three of those rows deserve a sentence each.
**The generator tag exists, and seven of the fifteen write it unquoted** as ``. A pattern that expects quotes around the attribute name silently reports half the population as unidentified. We know because a first pass here did exactly that.
**Four of the nine robots.txt files are byte identical**, 1,248 bytes, same MD5, and contain zero non-comment lines. They are Cloudflare's Content Signals licence preamble and nothing else: no `User-agent`, no `Allow`, no `Disallow`, no `Sitemap`. RFC 9309 section 2.2.1 says, verbatim, what that means: "If no group matches the product token and there is no group with a user-agent line with the `*` value, or no groups are present at all, no rules apply." Four sites have a robots.txt that returns 200 and says nothing. One of them is the Docusaurus documentation site, which scores **53 / Level 3** on our own scanner, with D1 marked partial on the evidence "200 OK, 1248 bytes".
**The one site that names AI crawlers stacks all eight names into a single group with `*`**, and its own inline comment concedes the point: "Explicitly allow common AI crawlers (duplicative with `*` above, but including for clarity)". Consecutive `User-agent` lines form one group, so no named agent receives a rule of its own. In that case the intent was to allow everything anyway, so nothing is lost. If you are writing rules that differ per crawler, a blank line between groups is what makes them separate.
## Path 1: the static folder, no build code
`static/` is copied verbatim into the build output. The Static Assets page states it as: "Every file you put into that directory will be copied into the root of the generated build folder with the directory hierarchy preserved". Use this path when the file content is stable.
1. Create `static/robots.txt` in your site directory.
2. Write real groups, separated by blank lines, and a `Sitemap:` line:
```
User-agent: *
Allow: /
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /
Sitemap: https://example.org/sitemap.xml
```
3. Add `static/llms.txt` with an H1, a blockquote summary, and H2 sections of `[title](url)` links. Our [llms.txt generator](https://aiscan.site/llms-txt-generator) writes the skeleton if you would rather not start from a blank file.
4. If you need `.well-known` files, `static/.well-known/` works. There is no file type filter and no plan gate, because `static/` has neither.
5. On GitHub Pages, add an empty `static/.nojekyll`.
6. Run `npm run build` and confirm the files landed in `build/`.
Take the agent tokens above from our maintained [crawler inventory](https://aiscan.site/blog/ai-crawler-user-agent-list-2026) rather than from memory. Operators do rename them, and a misspelt token is a group that matches nobody.
## Path 2: generate the files from your real routes
Docusaurus has no output format system the way a template driven generator does. What it has instead is a plugin lifecycle, and the hook that matters is `postBuild(props)`, which the lifecycle API documentation describes as "Called when a (production) build finishes". Its props carry `siteConfig`, `outDir`, `baseUrl` and `routesPaths`, which is every input a discovery file needs.
1. Create `plugins/discovery-files.js`:
```js
import fs from 'node:fs/promises';
import path from 'node:path';
export default function discoveryFiles() {
return {
name: 'discovery-files',
async postBuild({siteConfig, outDir, baseUrl, routesPaths}) {
if (baseUrl !== '/') {
throw new Error(
`robots.txt is void under baseUrl "${baseUrl}". ` +
'Deploy at a domain root, or remove the robots.txt half.'
);
}
const origin = siteConfig.url.replace(/\/$/, '');
await fs.writeFile(
path.join(outDir, 'robots.txt'),
`User-agent: *\nAllow: /\n\nSitemap: ${origin}/sitemap.xml\n`,
);
const docs = routesPaths.filter((r) => r.startsWith('/docs/'));
await fs.writeFile(
path.join(outDir, 'llms.txt'),
[`# ${siteConfig.title}`, '', `> ${siteConfig.tagline}`, '',
'## Docs', '',
...docs.map((r) => `- [${r}](${origin}${r})`), ''].join('\n'),
);
},
};
}
```
2. Register it in `docusaurus.config.js` with `plugins: ['./plugins/discovery-files.js']`.
3. Run `npm run build`, not `npm start`. `postBuild` fires only on a production build.
4. Check `build/llms.txt` lists the routes you expect.
Two things this buys you. The link list can never drift from the site, because it is derived from the route table rather than typed. And the `baseUrl` guard converts a silent deployment failure into a failed build, which is the whole reason to prefer generated files over copied ones.
If you would rather not maintain the llms.txt half, `docusaurus-plugin-llms` (version 0.6.0, published to npm on 1 September 2026) does that job. It is a community package rather than part of `@docusaurus/core`, so pin the version.
## The baseUrl trap, and why it hits two files differently
`static/` is copied to the build root, and the build root is served under `baseUrl`. The deployment guide gives the shape: for a site at `https://my-org.com/my-project/`, `baseUrl` is `/my-project/`. So a perfectly valid `static/robots.txt` answers at `/my-project/robots.txt` while the origin root returns 404.
For llms.txt that is survivable. The specification permits a file at a subpath, so your file is valid and only a scanner that probes the origin root alone will miss it. Our C2 check is one of those, which we have written up as an open bug against our own product.
For robots.txt it is fatal. RFC 9309 section 2.3 says, verbatim: "The rules MUST be accessible in a file named `/robots.txt` (all lowercase) in the top-level path of the service". A subpath file is not hidden, it is void. Re-measured on 7 September 2026: neither `facebook.github.io` nor `microsoft.github.io` serves a robots.txt at all, both answering 404, since that origin root is owned by the GitHub account and no individual repository can write to it. Three ways out of that are worked through in our [Docusaurus robots.txt guide](https://aiscan.site/blog/robots-txt-ai-bots-docusaurus). The C2 half gets its own treatment in [publishing llms.txt on Docusaurus](https://aiscan.site/blog/llms-txt-docusaurus).
## The sitemap and the feed are already handled
`@docusaurus/plugin-sitemap` ships inside `preset-classic`. According to its own reference page, it "is always inactive in development and only active in production because it works on the build output", which is why a local check finds nothing. Defaults are `lastmod: null`, `changefreq: 'weekly'`, `priority: 0.5`, `filename: 'sitemap.xml'`. It also respects two site settings, and the documentation lists the consequence plainly: `noIndex` "results in no sitemap generated", and `trailingSlash` decides whether sitemap URLs carry one. If `sitemap.xml` is missing on a live build, check `noIndex` before you check anything else.
The blog plugin is the quiet success story of this platform. `feedOptions` defaults to `{type: ['rss', 'atom']}` with a limit of 20 posts, and it injects the `rel="alternate"` autodiscovery tags for you. Our sweep found nine of fifteen sites serving `/blog/rss.xml`, and all nine declaring both feeds in ``. Compare that with [the same measurement on Hugo](https://aiscan.site/blog/ai-readiness-setup-hugo), where two of eight feed-serving sites declared nothing and a third pointed its tag at the homepage. Setting `feedOptions.type` to `null` is the only way to end up with no feed here.
## Auditing the half that can actually break
Everything above is a file. This part is a rendering question, and it is the one worth your time.
| What you see | What it means | Where to look |
|---|---|---|
| E3 reports a handful of words | That route escaped server rendering | The page component, for `` or a `window` reference |
| `sitemap.xml` returns 404 on a live build | `noIndex` is on, or the preset was replaced | `docusaurus.config.js` |
| `robots.txt` 404s at the origin root | The site is deployed under a `baseUrl` subpath | The deployment target, not the file |
| `llms.txt` 200s locally, 404s in CI | `postBuild` did not run | Use `docusaurus build`, not `docusaurus start` |
1. Search your source for browser-only rendering: `grep -rn "BrowserOnly\|useIsBrowser\|typeof window" src/`. The framework provides `` deliberately, and its warning is that "it is important for the first client-side render to produce the exact same DOM structure as server-side rendering".
2. Build, then check the built HTML rather than the dev server: `grep -c 'id="__docusaurus">' build/index.html`. A match means that page shipped an empty root.
3. Do the same on the routes that matter, not just the homepage. Custom landing pages written as React components are where this goes wrong; generated documentation routes almost never do.
4. Strip scripts and count words on the built file. Under a few dozen words on a page that should have hundreds is the signal.
We wrote up the identical failure on a different stack in [React apps that go invisible to crawlers](https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers); the diagnostic steps transfer without change.
## Check it: scan first, curl second
Run the scan. It is free, needs no account, and covers every check named in this guide in one pass:
```
npx aiscan-cli yoursite.com
```
A URL pasted into [the scanner](https://aiscan.site/) does the same thing without a terminal. Results land under [discoverability](https://aiscan.site/docs/checks/discoverability) for D1 and D2, [bot access](https://aiscan.site/docs/checks/bot-access) for B2, and [content](https://aiscan.site/docs/checks/content) for C2, C3, E3 and E5. Each row prints its evidence string, so you see the actual fetch and not only a verdict. E3 is the row to read first on this platform, because it prints the server-rendered word count.
If you would rather check by hand, this is the complete manual pass:
```bash
S=https://yoursite.com
curl -sI -o /dev/null -w 'robots %{http_code}\n' $S/robots.txt
curl -sI -o /dev/null -w 'sitemap %{http_code}\n' $S/sitemap.xml
curl -sI -o /dev/null -w 'llms %{http_code}\n' $S/llms.txt
curl -sI -o /dev/null -w 'probe %{http_code}\n' $S/no-such-file-9137.txt
curl -s $S/ | grep -c 'id="__docusaurus">'
```
The fourth line is the control. If a path that cannot exist returns 200, every other result on the list is meaningless, and you are reading the shell. The fifth line returns 1 when the homepage shipped an empty root.
If your documentation sits beside a WordPress marketing site, that half has the three-plugins-fighting-over-one-robots.txt problem instead. One plugin covering all four surfaces is the fix there, and [ThinkRank](https://thinkrank.ai) is what we point people at: it owns the robots file, the page-level robots meta, schema and the llms.txt together, which is what stops two SEO plugins overwriting each other. It also imports existing configuration from Rank Math, Yoast, All in One SEO and SEOPress, so a switch costs no re-entered settings. Where the neighbouring property is a Shopify store instead, [StoreSEO](https://storeseo.com/) builds the llms.txt out of your live catalogue and carries an agents.md editor.
## Where AIScan fits, and where it doesn't
One URL per scan is the constraint that bites hardest here. A documentation site's homepage is frequently the single route built differently from every other, so grading it alone tells you least. Give the scanner a deep documentation page too, then read the two E3 word counts side by side.
It also fetches with our own identity, so it cannot see per-crawler policy: a host that answers a browser with 200 and an AI crawler with 402 or 403 scores identically either way. And our C2 probes the origin root only, so a valid llms.txt under a `baseUrl` subpath is reported absent. Both are open bugs on our side rather than problems with your site, and we would rather say so than let you act on a wrong reading.
## Scan two URLs, not one
Scan your documentation site and read four rows: D1 for whether robots.txt exists and contains rules, B2 for whether any AI crawler is addressed, C2 for llms.txt, and E3 for the server-rendered word count on the page you scanned. Then scan a second URL from deeper in the docs and compare E3 across the two. If the numbers diverge, you have found the route worth fixing.
```
npx aiscan-cli yoursite.com
```
More platform walkthroughs are in the [guides index](https://aiscan.site/guides).