---
title: "How to declare allow/deny rules for named AI crawlers in robots.txt on Docusaurus"
slug: robots-txt-ai-bots-docusaurus
published: 2026-09-06T13:17:09.71625+00:00
updated: 2026-09-06T13:17:09.71625+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: check:B2, check:D1, platform:docusaurus, robots.txt, Docusaurus, AI crawlers, GitHub Pages, static site generators, AI readiness
description: "Docusaurus keeps robots.txt in static/, but baseUrl decides whether crawlers ever read it. The exact file, the GitHub Pages trap, and how to verify check B2."
url: https://aiscan.site/blog/robots-txt-ai-bots-docusaurus
---

Docusaurus gives you somewhere to put a `robots.txt` in about ten seconds. Making sure a crawler ever reads it is the harder half, because the most common Docusaurus deployment shape files the rules at a URL where they carry no force at all.

That gap bites harder here than anywhere else. When we covered [llms.txt on Docusaurus](https://aiscan.site/blog/llms-txt-docusaurus), a file under a repository path was still valid, because the llms.txt specification allows one at any subpath. The Robots Exclusion Protocol allows nothing of the kind.

## Quick summary

| Step | What you do | Where | Time |
|---|---|---|---|
| 1 | See which AI rules your site already serves | `npx aiscan-cli yoursite.com`, read D1 and B2 | 2 min |
| 2 | Write named crawler groups | `static/robots.txt` | 5 min |
| 3 | Check where `baseUrl` will publish it | `docusaurus.config.js` | 2 min |
| 4 | Move to a custom domain if you are on a project path | Host settings | 15 min |
| 5 | Read the live URL back | `curl` at the origin root | 1 min |

**The one thing to get right:** the rules have to answer at `https://yourdomain.com/robots.txt`, not at `https://yourdomain.com/your-project/robots.txt`.

## A subpath robots.txt is not a robots.txt

RFC 9309, the standard that defines the protocol, says verbatim:

> The rules MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service.

The document then gives the only URI form a crawler will construct: `scheme:[//authority]/robots.txt`. There is no second location and no fallback. A crawler asks the authority for `/robots.txt` and stops.

The consequence is unforgiving. If your rules answer one directory down, a compliant crawler never sees them, and RFC 9309 is equally clear about what happens then: if no group matches a crawler and no `*` group exists, "or no groups are present at all, no rules apply." Your carefully written allow and deny lines do not fail loudly. They simply do not exist, and every bot proceeds as though you had said nothing.

## Where baseUrl puts your file

Docusaurus copies the `static` folder wholesale. In its own words, from the static assets documentation:

> Every file you put into that directory will be copied into the root of the generated build folder with the directory hierarchy preserved.

The build root is then served under `baseUrl`, and the same page spells out both cases:

| `baseUrl` in your config | `static/robots.txt` answers at | Crawler reads it? |
|---|---|---|
| `'/'` | `https://yoursite.com/robots.txt` | Yes |
| `'/my-project/'` | `https://yoursite.com/my-project/robots.txt` | No |

According to the deployment guide, a site published at `https://my-org.com/my-project/` needs `url` set to `https://my-org.com/` and `baseUrl` set to `/my-project/`. That configuration is correct, documented, and the reason your rules are invisible.

## Write the rules into static/robots.txt

Create the file at `static/robots.txt` in your project root. Other source folders added through `staticDirectories` behave identically.

```
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://yoursite.com/sitemap.xml
```

Two decisions are packed into that block. Each crawler gets a named group of its own, because a scanner cannot distinguish a deliberate allow from an absent rule when all it finds is `User-agent: *`. And the retrieval bots sit apart from the training bots, so you can change your mind about model training without also removing yourself from the answers those tools cite. Our [verified AI crawler user-agent list](https://aiscan.site/blog/ai-crawler-user-agent-list-2026) has every current token, and the [crawler category breakdown](https://aiscan.site/blog/cloudflare-search-agent-training-crawler-categories) explains which bot belongs in which group before you commit.

## Deploying under a repository path

If your docs live at `org.github.io/project/`, no edit to `static/robots.txt` can fix this, because the origin root belongs to the account rather than to your repository. Verified on 6 September 2026, `facebook.github.io/robots.txt` and `microsoft.github.io/robots.txt` both return 404: even those accounts publish no rules at the only address that counts.

You have three honest options.

| Situation | What to do |
|---|---|
| Docs on a project path you control the domain for | Attach a custom domain, set `baseUrl: '/'`, redeploy |
| Docs on Netlify, Vercel, Render or Cloudflare Pages | Already at the root; `static/robots.txt` works as written |
| Docs genuinely stuck on `org.github.io/project/` | Publish rules on the user or organisation site instead, and accept they cover every repository |

A user or organisation Pages site is served at the root, so a `robots.txt` there is read. It also governs everything else that account publishes, which is a real trade rather than a workaround.

## Build the file from siteConfig instead of typing your domain

Hard-coding the `Sitemap:` line means it drifts the first time the domain changes. Docusaurus exposes a plugin lifecycle hook that runs once the build is finished, documented as `postBuild(props)`, with `siteConfig`, `baseUrl` and `outDir` on the props object:

```js
import fs from 'fs';
import path from 'path';

export default function robotsPlugin() {
  return {
    name: 'robots-txt',
    async postBuild({siteConfig, baseUrl, outDir}) {
      if (baseUrl !== '/') {
        throw new Error(`robots.txt is unreachable under baseUrl "${baseUrl}"`);
      }
      const rules = `User-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nSitemap: ${siteConfig.url}/sitemap.xml\n`;
      fs.writeFileSync(path.join(outDir, 'robots.txt'), rules);
    },
  };
}
```

The thrown error is the useful part. It turns the silent subpath problem into a failed build.

## The Docusaurus project's own site names no crawler

Fetched from `docusaurus.io/robots.txt` on 6 September 2026, the file returns 200 and `text/plain` and runs to twenty lines, every one of them a comment: the Content Signals licence preamble and nothing else. Zero `User-agent` lines, zero `Allow` or `Disallow` rules, no `Sitemap` line. Under the rule quoted earlier, no groups present means no rules apply.

The file exists, it returns 200, and a status-code check calls it healthy while it says nothing about AI crawlers.

## Read it back the way a crawler would

Start with the scan, because it answers the question in one command and reads both halves of it:

```bash
npx aiscan-cli yoursite.com
```

Read **D1** for whether a parseable `robots.txt` answers at all, and **B2** for whether it names AI crawlers explicitly. B2 is what fails when the file exists but carries only a `*` group, and it is what fails when `baseUrl` hid the file. You can also paste the URL at [aiscan.site](https://aiscan.site/), free and without an account.

Prefer to check by hand? Two commands finish it:

```bash
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://yoursite.com/robots.txt
curl -s https://yoursite.com/robots.txt | grep -iE 'GPTBot|SearchBot|Perplexity|Sitemap'
```

The first must print `200 text/plain`. The second must print your named groups. If the first prints 404 while the same file answers under your project path, `baseUrl` is the cause and no amount of editing the file will change it.

## Where AIScan fits, and where it doesn't

AIScan grades what your site returns to an ordinary fetch: whether the file is there, whether it parses, whether it names AI crawlers. It cannot tell you whether a crawler obeyed the rules, and the protocol itself notes these rules "are not a form of access authorization." A `Disallow` line is a request. It is not a lock, and anything you need genuinely restricted belongs behind authentication.

Our own scanner also asks the origin root only, which is why a Docusaurus site on a project path shows a bare `HTTP 404` in the evidence field rather than "your file is one directory too deep." That is a known limit on our side, not a verdict on your file.

## Push it live, then re-run the scan

Commit `static/robots.txt`, deploy, and run `npx aiscan-cli yoursite.com` once more to confirm D1 and B2 both pass. The [bot access checks](https://aiscan.site/docs/checks/bot-access) page explains what each one grades, [what belongs in robots.txt against llms.txt and sitemap.xml](https://aiscan.site/blog/llms-txt-vs-robots-txt-vs-sitemap) settles which file carries which job, and the rest of the platform walkthroughs are indexed at [aiscan.site/guides](https://aiscan.site/guides). If a WordPress blog sits alongside your docs, [ThinkRank](https://thinkrank.ai) manages robots.txt, robots meta, schema and llms.txt there from one plugin, which is the shorter path than reconciling three that each want to own the same virtual file.