How to declare allow/deny rules for named AI crawlers in robots.txt on Docusaurus

Docusaurus gives you somewhere to put a robots.txt in about ten seconds. Making sure a crawler ever reads it is the harder half, because the most common Docusaurus deployment shape files the rules at a URL where they carry no force at all.

That gap bites harder here than anywhere else. When we covered llms.txt on Docusaurus, a file under a repository path was still valid, because the llms.txt specification allows one at any subpath. The Robots Exclusion Protocol allows nothing of the kind.

Quick summary

The one thing to get right: the rules have to answer at https://yourdomain.com/robots.txt, not at https://yourdomain.com/your-project/robots.txt.

A subpath robots.txt is not a robots.txt

RFC 9309, the standard that defines the protocol, says verbatim:

The rules MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service.

The document then gives the only URI form a crawler will construct: scheme:[//authority]/robots.txt. There is no second location and no fallback. A crawler asks the authority for /robots.txt and stops.

The consequence is unforgiving. If your rules answer one directory down, a compliant crawler never sees them, and RFC 9309 is equally clear about what happens then: if no group matches a crawler and no * group exists, "or no groups are present at all, no rules apply." Your carefully written allow and deny lines do not fail loudly. They simply do not exist, and every bot proceeds as though you had said nothing.

Where baseUrl puts your file

Docusaurus copies the static folder wholesale. In its own words, from the static assets documentation:

Every file you put into that directory will be copied into the root of the generated build folder with the directory hierarchy preserved.

The build root is then served under baseUrl, and the same page spells out both cases:

According to the deployment guide, a site published at https://my-org.com/my-project/ needs url set to https://my-org.com/ and baseUrl set to /my-project/. That configuration is correct, documented, and the reason your rules are invisible.

Write the rules into static/robots.txt

Create the file at static/robots.txt in your project root. Other source folders added through staticDirectories behave identically.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://yoursite.com/sitemap.xml

Two decisions are packed into that block. Each crawler gets a named group of its own, because a scanner cannot distinguish a deliberate allow from an absent rule when all it finds is User-agent: *. And the retrieval bots sit apart from the training bots, so you can change your mind about model training without also removing yourself from the answers those tools cite. Our verified AI crawler user-agent list has every current token, and the crawler category breakdown explains which bot belongs in which group before you commit.

Deploying under a repository path

If your docs live at org.github.io/project/, no edit to static/robots.txt can fix this, because the origin root belongs to the account rather than to your repository. Verified on 6 September 2026, facebook.github.io/robots.txt and microsoft.github.io/robots.txt both return 404: even those accounts publish no rules at the only address that counts.

You have three honest options.

A user or organisation Pages site is served at the root, so a robots.txt there is read. It also governs everything else that account publishes, which is a real trade rather than a workaround.

Build the file from siteConfig instead of typing your domain

Hard-coding the Sitemap: line means it drifts the first time the domain changes. Docusaurus exposes a plugin lifecycle hook that runs once the build is finished, documented as postBuild(props), with siteConfig, baseUrl and outDir on the props object:

import fs from 'fs';
import path from 'path';

export default function robotsPlugin() {
  return {
    name: 'robots-txt',
    async postBuild({siteConfig, baseUrl, outDir}) {
      if (baseUrl !== '/') {
        throw new Error(`robots.txt is unreachable under baseUrl "${baseUrl}"`);
      }
      const rules = `User-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nSitemap: ${siteConfig.url}/sitemap.xml\n`;
      fs.writeFileSync(path.join(outDir, 'robots.txt'), rules);
    },
  };
}

The thrown error is the useful part. It turns the silent subpath problem into a failed build.

The Docusaurus project's own site names no crawler

Fetched from docusaurus.io/robots.txt on 6 September 2026, the file returns 200 and text/plain and runs to twenty lines, every one of them a comment: the Content Signals licence preamble and nothing else. Zero User-agent lines, zero Allow or Disallow rules, no Sitemap line. Under the rule quoted earlier, no groups present means no rules apply.

The file exists, it returns 200, and a status-code check calls it healthy while it says nothing about AI crawlers.

Read it back the way a crawler would

Start with the scan, because it answers the question in one command and reads both halves of it:

npx aiscan-cli yoursite.com

Read D1 for whether a parseable robots.txt answers at all, and B2 for whether it names AI crawlers explicitly. B2 is what fails when the file exists but carries only a * group, and it is what fails when baseUrl hid the file. You can also paste the URL at aiscan.site, free and without an account.

Prefer to check by hand? Two commands finish it:

curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://yoursite.com/robots.txt
curl -s https://yoursite.com/robots.txt | grep -iE 'GPTBot|SearchBot|Perplexity|Sitemap'

The first must print 200 text/plain. The second must print your named groups. If the first prints 404 while the same file answers under your project path, baseUrl is the cause and no amount of editing the file will change it.

Where AIScan fits, and where it doesn't

AIScan grades what your site returns to an ordinary fetch: whether the file is there, whether it parses, whether it names AI crawlers. It cannot tell you whether a crawler obeyed the rules, and the protocol itself notes these rules "are not a form of access authorization." A Disallow line is a request. It is not a lock, and anything you need genuinely restricted belongs behind authentication.

Our own scanner also asks the origin root only, which is why a Docusaurus site on a project path shows a bare HTTP 404 in the evidence field rather than "your file is one directory too deep." That is a known limit on our side, not a verdict on your file.

Push it live, then re-run the scan

Commit static/robots.txt, deploy, and run npx aiscan-cli yoursite.com once more to confirm D1 and B2 both pass. The bot access checks page explains what each one grades, what belongs in robots.txt against llms.txt and sitemap.xml settles which file carries which job, and the rest of the platform walkthroughs are indexed at aiscan.site/guides. If a WordPress blog sits alongside your docs, ThinkRank manages robots.txt, robots meta, schema and llms.txt there from one plugin, which is the shorter path than reconciling three that each want to own the same virtual file.