The complete AI readiness setup for Docusaurus in 2026

Docusaurus is the framework most likely to be sitting between an AI agent and the answer it is looking for. It runs a very large share of the developer documentation on the web, and developer documentation is exactly the material that retrieval systems reach for. So it is worth knowing what a Docusaurus site actually hands a crawler, rather than what the build log says it produced.

The short version: the discovery files are the easy half. The hard half is whether a given page contains any words at all, and Docusaurus tells you so in its own documentation.

Quick summary

Why a Docusaurus page can be empty and still be "static"

Every other platform in this series either renders your HTML on a server or compiles it at publish time, and the content question is settled before you arrive. Docusaurus is a React single-page application that emits static HTML as a performance optimisation. That is not an outside characterisation. The framework's own Static Site Generation page, fetched from docusaurus.io/docs/advanced/ssg and verified on 7 September 2026, says it in its own words:

Docusaurus is ultimately a single-page application, so static site generation is only an optimization (progressive enhancement, as it's called), but our functionality does not fully depend on those HTML files.

The same page, in its own words, explains that "the theme is built twice", once through React DOM Server and once for the browser, and describes what the failure looks like: "in CSR-only apps, all DOM elements are generated on client side with React, and the HTML file only ever contains one root element for React to mount DOM to."

That root element is <div id="__docusaurus"></div>. When a route escapes server rendering, that div ships empty and the page is a shell.

This is not theoretical. Fetching the Cypress documentation home page on 7 September 2026 with an ordinary desktop browser user agent returned 6,826 bytes containing exactly that: an empty __docusaurus div, data-has-hydrated="false", and two words of visible text. Our own scanner grades that site 72 / Level 4 with C3 and E3 both failing on the evidence string "2 words of server-rendered text, 0 <h2>, 0 <h3>". The same site's robots.txt names eight AI crawlers and its llms.txt is roughly 70 KB. It did all the discovery work and its front door says nothing.

To be fair to it: the failure is one route, not the site. Two documentation pages on the same host returned 2,196 and 3,897 words with a single <h1> each. That is the shape to remember. On Docusaurus, C3 and E3 are a per-route build outcome rather than a platform guarantee.

What fifteen live Docusaurus sites actually publish

We fingerprinted thirty candidate documentation domains and confirmed fifteen as Docusaurus, then probed six paths on each: ninety requests, 7 September 2026. The confirmed hosts were:

docusaurus.io          jestjs.io            reactnative.dev
redux.js.org           prettier.io          babeljs.io
create-react-app.dev   electronjs.org       sequelize.org
typeorm.io             docs.dagger.io       docusaurus.community
docs.cypress.io        recoiljs.org         docs.getdbt.com

Three of those rows deserve a sentence each.

The generator tag exists, and seven of the fifteen write it unquoted as <meta name=generator content="Docusaurus v3.10.2">. A pattern that expects quotes around the attribute name silently reports half the population as unidentified. We know because a first pass here did exactly that.

Four of the nine robots.txt files are byte identical, 1,248 bytes, same MD5, and contain zero non-comment lines. They are Cloudflare's Content Signals licence preamble and nothing else: no User-agent, no Allow, no Disallow, no Sitemap. RFC 9309 section 2.2.1 says, verbatim, what that means: "If no group matches the product token and there is no group with a user-agent line with the * value, or no groups are present at all, no rules apply." Four sites have a robots.txt that returns 200 and says nothing. One of them is the Docusaurus documentation site, which scores 53 / Level 3 on our own scanner, with D1 marked partial on the evidence "200 OK, 1248 bytes".

The one site that names AI crawlers stacks all eight names into a single group with *, and its own inline comment concedes the point: "Explicitly allow common AI crawlers (duplicative with * above, but including for clarity)". Consecutive User-agent lines form one group, so no named agent receives a rule of its own. In that case the intent was to allow everything anyway, so nothing is lost. If you are writing rules that differ per crawler, a blank line between groups is what makes them separate.

Path 1: the static folder, no build code

static/ is copied verbatim into the build output. The Static Assets page states it as: "Every file you put into that directory will be copied into the root of the generated build folder with the directory hierarchy preserved". Use this path when the file content is stable.

1. Create static/robots.txt in your site directory.
2. Write real groups, separated by blank lines, and a Sitemap: line:

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /

Sitemap: https://example.org/sitemap.xml

1. Add static/llms.txt with an H1, a blockquote summary, and H2 sections of [title](url) links. Our llms.txt generator writes the skeleton if you would rather not start from a blank file.
2. If you need .well-known files, static/.well-known/ works. There is no file type filter and no plan gate, because static/ has neither.
3. On GitHub Pages, add an empty static/.nojekyll.
4. Run npm run build and confirm the files landed in build/.

Take the agent tokens above from our maintained crawler inventory rather than from memory. Operators do rename them, and a misspelt token is a group that matches nobody.

Path 2: generate the files from your real routes

Docusaurus has no output format system the way a template driven generator does. What it has instead is a plugin lifecycle, and the hook that matters is postBuild(props), which the lifecycle API documentation describes as "Called when a (production) build finishes". Its props carry siteConfig, outDir, baseUrl and routesPaths, which is every input a discovery file needs.

1. Create plugins/discovery-files.js:

import fs from 'node:fs/promises';
import path from 'node:path';

export default function discoveryFiles() {
  return {
    name: 'discovery-files',
    async postBuild({siteConfig, outDir, baseUrl, routesPaths}) {
      if (baseUrl !== '/') {
        throw new Error(
          `robots.txt is void under baseUrl "${baseUrl}". ` +
          'Deploy at a domain root, or remove the robots.txt half.'
        );
      }
      const origin = siteConfig.url.replace(/\/$/, '');
      await fs.writeFile(
        path.join(outDir, 'robots.txt'),
        `User-agent: *\nAllow: /\n\nSitemap: ${origin}/sitemap.xml\n`,
      );
      const docs = routesPaths.filter((r) => r.startsWith('/docs/'));
      await fs.writeFile(
        path.join(outDir, 'llms.txt'),
        [`# ${siteConfig.title}`, '', `> ${siteConfig.tagline}`, '',
         '## Docs', '',
         ...docs.map((r) => `- [${r}](${origin}${r})`), ''].join('\n'),
      );
    },
  };
}

1. Register it in docusaurus.config.js with plugins: ['./plugins/discovery-files.js'].
2. Run npm run build, not npm start. postBuild fires only on a production build.
3. Check build/llms.txt lists the routes you expect.

Two things this buys you. The link list can never drift from the site, because it is derived from the route table rather than typed. And the baseUrl guard converts a silent deployment failure into a failed build, which is the whole reason to prefer generated files over copied ones.

If you would rather not maintain the llms.txt half, docusaurus-plugin-llms (version 0.6.0, published to npm on 1 September 2026) does that job. It is a community package rather than part of @docusaurus/core, so pin the version.

The baseUrl trap, and why it hits two files differently

static/ is copied to the build root, and the build root is served under baseUrl. The deployment guide gives the shape: for a site at https://my-org.com/my-project/, baseUrl is /my-project/. So a perfectly valid static/robots.txt answers at /my-project/robots.txt while the origin root returns 404.

For llms.txt that is survivable. The specification permits a file at a subpath, so your file is valid and only a scanner that probes the origin root alone will miss it. Our C2 check is one of those, which we have written up as an open bug against our own product.

For robots.txt it is fatal. RFC 9309 section 2.3 says, verbatim: "The rules MUST be accessible in a file named /robots.txt (all lowercase) in the top-level path of the service". A subpath file is not hidden, it is void. Re-measured on 7 September 2026: neither facebook.github.io nor microsoft.github.io serves a robots.txt at all, both answering 404, since that origin root is owned by the GitHub account and no individual repository can write to it. Three ways out of that are worked through in our Docusaurus robots.txt guide. The C2 half gets its own treatment in publishing llms.txt on Docusaurus.

The sitemap and the feed are already handled

@docusaurus/plugin-sitemap ships inside preset-classic. According to its own reference page, it "is always inactive in development and only active in production because it works on the build output", which is why a local check finds nothing. Defaults are lastmod: null, changefreq: 'weekly', priority: 0.5, filename: 'sitemap.xml'. It also respects two site settings, and the documentation lists the consequence plainly: noIndex "results in no sitemap generated", and trailingSlash decides whether sitemap URLs carry one. If sitemap.xml is missing on a live build, check noIndex before you check anything else.

The blog plugin is the quiet success story of this platform. feedOptions defaults to {type: ['rss', 'atom']} with a limit of 20 posts, and it injects the rel="alternate" autodiscovery tags for you. Our sweep found nine of fifteen sites serving /blog/rss.xml, and all nine declaring both feeds in <head>. Compare that with the same measurement on Hugo, where two of eight feed-serving sites declared nothing and a third pointed its tag at the homepage. Setting feedOptions.type to null is the only way to end up with no feed here.

Auditing the half that can actually break

Everything above is a file. This part is a rendering question, and it is the one worth your time.

1. Search your source for browser-only rendering: grep -rn "BrowserOnly\|useIsBrowser\|typeof window" src/. The framework provides <BrowserOnly> deliberately, and its warning is that "it is important for the first client-side render to produce the exact same DOM structure as server-side rendering".
2. Build, then check the built HTML rather than the dev server: grep -c 'id="__docusaurus"></div>' build/index.html. A match means that page shipped an empty root.
3. Do the same on the routes that matter, not just the homepage. Custom landing pages written as React components are where this goes wrong; generated documentation routes almost never do.
4. Strip scripts and count words on the built file. Under a few dozen words on a page that should have hundreds is the signal.

We wrote up the identical failure on a different stack in React apps that go invisible to crawlers; the diagnostic steps transfer without change.

Check it: scan first, curl second

Run the scan. It is free, needs no account, and covers every check named in this guide in one pass:

npx aiscan-cli yoursite.com

A URL pasted into the scanner does the same thing without a terminal. Results land under discoverability for D1 and D2, bot access for B2, and content for C2, C3, E3 and E5. Each row prints its evidence string, so you see the actual fetch and not only a verdict. E3 is the row to read first on this platform, because it prints the server-rendered word count.

If you would rather check by hand, this is the complete manual pass:

S=https://yoursite.com
curl -sI -o /dev/null -w 'robots %{http_code}\n' $S/robots.txt
curl -sI -o /dev/null -w 'sitemap %{http_code}\n' $S/sitemap.xml
curl -sI -o /dev/null -w 'llms %{http_code}\n'    $S/llms.txt
curl -sI -o /dev/null -w 'probe %{http_code}\n'   $S/no-such-file-9137.txt
curl -s $S/ | grep -c 'id="__docusaurus"></div>'

The fourth line is the control. If a path that cannot exist returns 200, every other result on the list is meaningless, and you are reading the shell. The fifth line returns 1 when the homepage shipped an empty root.

If your documentation sits beside a WordPress marketing site, that half has the three-plugins-fighting-over-one-robots.txt problem instead. One plugin covering all four surfaces is the fix there, and ThinkRank is what we point people at: it owns the robots file, the page-level robots meta, schema and the llms.txt together, which is what stops two SEO plugins overwriting each other. It also imports existing configuration from Rank Math, Yoast, All in One SEO and SEOPress, so a switch costs no re-entered settings. Where the neighbouring property is a Shopify store instead, StoreSEO builds the llms.txt out of your live catalogue and carries an agents.md editor.

Where AIScan fits, and where it doesn't

One URL per scan is the constraint that bites hardest here. A documentation site's homepage is frequently the single route built differently from every other, so grading it alone tells you least. Give the scanner a deep documentation page too, then read the two E3 word counts side by side.

It also fetches with our own identity, so it cannot see per-crawler policy: a host that answers a browser with 200 and an AI crawler with 402 or 403 scores identically either way. And our C2 probes the origin root only, so a valid llms.txt under a baseUrl subpath is reported absent. Both are open bugs on our side rather than problems with your site, and we would rather say so than let you act on a wrong reading.

Scan two URLs, not one

Scan your documentation site and read four rows: D1 for whether robots.txt exists and contains rules, B2 for whether any AI crawler is addressed, C2 for llms.txt, and E3 for the server-rendered word count on the page you scanned. Then scan a second URL from deeper in the docs and compare E3 across the two. If the numbers diverge, you have found the route worth fixing.

npx aiscan-cli yoursite.com

More platform walkthroughs are in the guides index.