Control which AI crawlers reach your site: a robots.txt starter

Most sites that "block AI" with a robots.txt file end up with one of two outcomes: a file that blocks nothing because a bot name is misspelled, or a file that blocks the crawlers that would have sent them traffic. This guide gives you a starter file you can paste in, the three decisions that shape it, and a two-minute test that proves each rule does what you meant. The starter below takes about 10 minutes to adapt and ship.

Updated on 5 November 2026: added Docusaurus and Wix sections, folded in from the platform guides that used to sit at their own URLs.

Quick summary

Using Docusaurus or Wix? Jump to Docusaurus or Wix. If you are unsure whether the Sitemap line is worth keeping, see what 782 robots.txt files showed about declaring a sitemap.

Expert guidance in one line: OpenAI's crawler documentation says each of its settings is independent, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Anthropic's crawler help page draws the same split between ClaudeBot (training) and Claude-User (answering a person's question). Treat each operator as three bots, not one.

Check your current file first: npx aiscan-cli yoursite.com reports B1 (Content Signals) and B2 (explicit AI bot rules) in about 20 seconds.

What a robots.txt file can and cannot do

RFC 9309 defines the format and says plainly that the Robots Exclusion Protocol is not a substitute for valid content security measures. A rule is a request. A well-behaved crawler reads it and complies. A badly behaved one ignores it.

That is not a theory. In AIScan's measurement of 40 hosts, a full Disallow: / was backed by an actual refusal at the server edge only 45% to 67% of the time, depending on the bot (see Does blocking AI training protect your content?). Use robots.txt to state your policy. Use your CDN or firewall if you need to enforce it. A noai meta tag is not a substitute for either: none of eight AI crawler operators documents it (see do noai and noimageai meta tags stop AI crawlers?).

Three mechanics decide whether your file works:

1. A bot obeys the most specific group that names it. Per RFC 9309 section 2.2.1, if no group matches a bot's token, it falls back to the * group. If a group does match, the * group no longer applies to it. Add a named group for OAI-SearchBot and it stops inheriting your Disallow: /admin/ line from *, unless you repeat it.
2. Tokens are not user-agent strings. Google's crawler documentation says Google-Extended has no separate HTTP user agent. Crawling uses existing Google strings and the token only controls how the content is used. Applebot-Extended works the same way. A named Google-Extended group also outranks the new shared training token at Google, so an Allow: / here cancels that opt-out; see the precedence table.
3. Some user-triggered fetchers ignore robots.txt by design. Perplexity documents that Perplexity-User generally ignores it. OpenAI describes ChatGPT-User as visiting pages when a person asks, not crawling automatically. The full list of 33 tokens, with the dates each was checked, is in the AI crawler user-agent list.

Decide first: three jobs, three tokens

Blocking the first column keeps you out of future training sets. Blocking the second removes you from that operator's search answers. Blocking the third stops an assistant fetching your page when a person asks about it. Anthropic states that disabling Claude-User may reduce your site's visibility. Most publishers who want citations block the first column and leave the other two open. That is the profile in the starter file.

Training exclusion applies to future collection. Nothing in a robots.txt file removes content a crawler already holds. A shared training token, __usage_ai-training, is also on the way (Google and OpenAI announced plans on 30 Oct 2026); where to put it without blocking your crawlers is in the __usage_ai-training token, tested.

Path 1: write the file by hand

Use this on any host where you can edit the file at the site root.

1. Open or create robots.txt at your site root. It must be reachable at https://yoursite.com/robots.txt, all lowercase, and return a 200 with text/plain.
2. Paste the starter below, replace example.com and the two Disallow paths with your own private paths.
3. Keep your existing Sitemap: line. Crawlers may read it, but RFC 9309 does not define it, so do not rely on it as a rule.
4. Save and deploy. OpenAI says its systems take about 24 hours to pick up a change, so do not judge the result from your logs the same afternoon.

User-agent: *
Disallow: /admin/
Disallow: /cart/

# Training crawlers: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
Disallow: /

# Search crawlers: stay visible (repeat the private paths from the * group)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

Several User-agent lines above one set of rules is valid under RFC 9309 and keeps the file short. Bots with no group, such as ChatGPT-User, Claude-User and Googlebot, fall through to * and are only kept out of /admin/ and /cart/.

To declare your policy in one extra line, add a Content Signal. Cloudflare's proposal uses Content-Signal: search=yes, ai-train=no, ai-input=yes. It is a draft and not part of RFC 9309, so treat it as a statement of intent. AIScan reports it as check B1 and it is informational only.

Path 2: use your platform's editor

WordPress. Three plugins often compete to write robots.txt and the last one loaded wins. ThinkRank manages robots.txt, robots meta, schema and llms.txt from one plugin, so there is a single place to paste the starter, and it imports settings from Rank Math, Yoast, AIOSEO and SEOPress so nothing is re-entered. Rank Math and Yoast both have a robots.txt editor too and are fine if you already run one of them. Whichever you use, deactivate the other editors so only one file is served, then paste the starter.

Shopify. Shopify's robots.txt.liquid documentation says adding that template replaces the generated file. In the theme code editor, right-click the Templates folder, choose New File, name it robots.txt.liquid, and save a copy of your current /robots.txt output first so you can compare. Keep Shopify's default rules in the template and add the training groups from the starter. For the files that sit beside robots.txt, StoreSEO generates llms.txt from live products, collections, pages and articles and has an agents.md editor.

Docusaurus and Wix. These two have sections of their own below: Docusaurus for the baseUrl trap and Wix for a generated file that your named groups sit beside. Other platform editors rewrite defaults in different ways, so read the file back after every save.

Docusaurus

Docusaurus gives you somewhere to put a robots.txt in about ten seconds. Making sure a crawler ever reads it is the harder half, because the most common Docusaurus deployment shape files the rules at a URL where they carry no force at all. We re-checked the Docusaurus static assets documentation on 5 November 2026 and the behaviour below still holds.

The one thing to get right: the rules have to answer at https://yourdomain.com/robots.txt, not at https://yourdomain.com/your-project/robots.txt.

A subpath robots.txt is not a robots.txt

RFC 9309 says the rules MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service. A crawler asks the authority for /robots.txt and stops. There is no second location and no fallback. If your rules answer one directory down, a compliant crawler never sees them, and the RFC is equally clear about the result: if no group matches and no * group exists, "or no groups are present at all, no rules apply." Your allow and deny lines do not fail loudly. They simply do not exist.

That gap bites harder here than on most platforms. The llms.txt specification allows a file at any subpath, so a file under a repository path is still valid there (see llms.txt on Docusaurus). The Robots Exclusion Protocol allows nothing of the kind.

Where baseUrl puts your file

Docusaurus copies the static folder wholesale. Its documentation says every file in that directory is copied into the root of the generated build folder with the directory hierarchy preserved, and the build root is served under baseUrl:

A site published at https://my-org.com/my-project/ needs url set to https://my-org.com/ and baseUrl set to /my-project/. That configuration is correct, documented, and the reason your rules are invisible.

Write the rules into static/robots.txt

Create the file at static/robots.txt next to docusaurus.config.js. Other source folders added through staticDirectories behave identically, so staticDirectories: ['public', 'static'] lets you keep it in public/ instead.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://yoursite.com/sitemap.xml

Each crawler gets a named group of its own, because a scanner cannot tell a deliberate allow from an absent rule when all it finds is User-agent: *. The retrieval bots sit apart from the training bots, so you can change your mind about model training without removing yourself from the answers those tools cite. The verified AI crawler user-agent list has every current token, and the crawler category breakdown explains which bot belongs in which group.

Deploying under a repository path

If your docs live at org.github.io/project/, no edit to static/robots.txt can fix this, because the origin root belongs to the account rather than to your repository. Checked on 6 September 2026, facebook.github.io/robots.txt and microsoft.github.io/robots.txt both return 404: even those accounts publish no rules at the only address that counts.

A user or organisation Pages site is served at the root, so a robots.txt there is read. It also governs everything else that account publishes, which is a real trade rather than a workaround.

Build the file from siteConfig

Hard-coding the Sitemap: line means it drifts the first time the domain changes. Docusaurus exposes a plugin lifecycle hook, postBuild(props), that runs once the build is finished, with siteConfig, baseUrl and outDir on the props object:

import fs from 'fs';
import path from 'path';

export default function robotsPlugin() {
  return {
    name: 'robots-txt',
    async postBuild({siteConfig, baseUrl, outDir}) {
      if (baseUrl !== '/') {
        throw new Error(`robots.txt is unreachable under baseUrl "${baseUrl}"`);
      }
      const rules = `User-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nSitemap: ${siteConfig.url}/sitemap.xml\n`;
      fs.writeFileSync(path.join(outDir, 'robots.txt'), rules);
    },
  };
}

The thrown error is the useful part. It turns the silent subpath problem into a failed build.

The Docusaurus project's own site names no crawler

Fetched from docusaurus.io/robots.txt on 6 September 2026, the file returned 200 and text/plain and ran to twenty lines, every one of them a comment: the Content Signals licence preamble and nothing else. Zero User-agent lines, zero Allow or Disallow rules, no Sitemap line. The file exists and a status-code check calls it healthy while it says nothing about AI crawlers.

Read it back from the origin root

npx aiscan-cli yoursite.com
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://yoursite.com/robots.txt
curl -s https://yoursite.com/robots.txt | grep -iE 'GPTBot|SearchBot|Perplexity|Sitemap'

Read D1 for whether a parseable robots.txt answers at all, and B2 for whether it names AI crawlers explicitly. B2 fails when the file exists but carries only a * group, and when baseUrl hid the file. The first curl must print 200 text/plain. If it prints 404 while the same file answers under your project path, baseUrl is the cause and no edit to the file will change it. AIScan asks the origin root only, so a Docusaurus site on a project path shows a bare HTTP 404 in the evidence field rather than "your file is one directory too deep". That is a known limit of the scanner, not a verdict on your file.

Wix

Wix writes your robots.txt for you. Fetched on 7 September 2026, the file at www.wix.com/robots.txt ran to 90 non-empty lines: one User-agent: * group, 87 Disallow rules, and a single Sitemap: line at the bottom. Adding a rule for GPTBot or ClaudeBot takes about two minutes in the Robots.txt Editor, and it does something most guides never mention: the crawler you just named stops reading every one of those 87 lines. We re-checked Wix's Robots.txt Editor article on 5 November 2026 and the path below is unchanged.

Wix keeps rewriting the file you are about to edit

Wix says it maintains the file continuously: "If you need to update a page, Wix automatically updates your robots.txt after you publish the page. If you change your site's settings, your robots.txt file is updated immediately." That is the opposite of how the other generated file on Wix behaves. As covered in publishing a valid llms.txt on Wix, Wix says that once you edit llms.txt, "it will stop updating automatically so your changes are preserved." Robots.txt carries no such lock: your named groups sit alongside a * group Wix goes on regenerating around them. Wix also puts a warning above the procedure worth repeating: "Wix Customer Care cannot help you with any changes you make to your robots.txt file."

Open the Robots.txt Editor

1. Go to SEO & GEO in your site's dashboard.
2. Scroll down to Tools and settings.
3. Click Robots.txt Editor.
4. Click View File.
5. Type your directives under This is your current file.
6. Click Save Changes, then Save.

To undo everything, repeat the first four clicks, then Reset to Default and Reset. There is no file manager and no FTP on Wix, so this panel is the only way in.

A named group replaces the defaults, it does not extend them

RFC 9309 section 2.2.1 states: "Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group." Groups for different tokens do not stack; the spec merges only groups matching the same token.

Every rule Wix generated sits under User-agent: *. The moment you add User-agent: GPTBot, that crawler reads your group and ignores the * group entirely. A group of Disallow: / is harmless, because you blocked everything anyway. Write Allow: / under a named token and you have handed that crawler the internal paths Wix was keeping shut.

The safe shape on Wix: name a crawler only to deny it, and let the rest fall through to *. One syntax rule from section 2.2.1 catches people out: a product token "MUST contain only uppercase and lowercase letters", plus hyphen and underscore. No slashes, no version numbers, no wildcards in the name.

Wix's four example names, checked against 2026

Wix's article Blocking AI Crawlers from Your Site gives a worked example: CCBot, GPTBot, ChatGPT-User and BingAI, each with Disallow: /. Checked against our AI crawler user-agent list:

Missing from the example and crawling in 2026: ClaudeBot, PerplexityBot, Google-Extended, meta-externalagent and Applebot-Extended. Wix's own FAQ is honest about the ceiling: "some crawlers may ignore these directives, so it's not an airtight solution."

Rules that keep you in the answers and out of the training set

Paste this under This is your current file, below what is already there:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

OAI-SearchBot, PerplexityBot and Bingbot are unlisted, so they keep matching * and keep indexing you. Block OAI-SearchBot by accident and you leave the surface you were trying to win.

Read the file back from your own domain

Assert the content type, not the status code:

curl -sI https://yoursite.com/robots.txt
curl -s https://yoursite.com/robots.txt | grep -A1 -i 'gptbot'

A working file answers 200 with content-type: text/plain. If you get text/html, you are reading an error page, not a file. The second command should print the two lines you typed. An empty result means the save did not commit; reopen View File rather than typing it somewhere else. The envelope matters as much as the rules, and serving robots.txt with the right status and content type covers what crawlers do on a 403, 503 or HTML response.

When the editor is not the problem

Wix is direct that most robots.txt complaints are not robots.txt complaints. If Wix Site Inspection or Google Search Console reports Blocked by robots.txt, Wix says "you don't need to edit your robots.txt file, especially if you never edited it before."

For that last row, in the editor go to Pages & Menu > More Actions > SEO basics > Advanced SEO > Robots meta tag and tick nosnippet, which Wix describes as preventing a page's content "from being used as a direct input for AI overviews". One caution before denying broadly: Wix warns that blocking AI crawlers "may limit NLWeb functionality, as AI agents need access to view your structured content." The scan cannot see your server logs or whether a crawler obeyed, and Wix does not expose logs.

Verify it did what you meant

Start with a scan. Run npx aiscan-cli yoursite.com, or paste your URL at aiscan.site. It is free and needs no account. Three checks apply here:

• B1, Content Signals: whether robots.txt declares AI usage preferences. Informational, it cannot lower your score.
• B2, explicit AI bot rules: whether the AI crawlers have their own named groups. Four or more named bots passes, one to three is partial.
• B3, Web Bot Auth: a signing key directory for bots you operate. Most site owners publish nothing here.

The scan reads what you publish. It cannot see your server logs or whether the server refuses a bot, which is why the hand checks below still matter. Details for each check are on the bot access checks page.

If you would rather check by hand:

curl -sI https://yoursite.com/robots.txt | head -3      # expect 200 and text/plain
curl -s https://yoursite.com/robots.txt | grep -ciE 'gptbot|claudebot|oai-searchbot'
python3 -c "
from urllib.robotparser import RobotFileParser as R
r=R(); r.set_url('https://yoursite.com/robots.txt'); r.read()
for ua in ['GPTBot','ClaudeBot','OAI-SearchBot','PerplexityBot','Googlebot']:
    print(ua, r.can_fetch(ua,'https://yoursite.com/'), r.can_fetch(ua,'https://yoursite.com/admin/x'))
"

Against the starter file above, that check prints False False for GPTBot and ClaudeBot, True False for OAI-SearchBot, PerplexityBot and Googlebot. We ran it on 7 October 2026. The expected result for your site is the same pattern. Python's parser applies first-match rules rather than the longest-match rule in RFC 9309, so use it for group logic, not for complicated Allow and Disallow overlaps.

To test enforcement rather than the file, request a page with the bot's user-agent string and compare it to a browser request, as shown in Fetch your site the way an AI agent sees it. A spoofed string proves the server rule, not the real crawler, so check real traffic against the operator's published IP ranges.

Two variants of the starter

Open to everything. Write nothing. An absent or empty robots.txt means crawl freely. The trap is a robots.txt URL that returns a 500 or an HTML soft-404 page, which some crawlers treat as a signal to stay away. Make sure the URL returns a real 200 with plain text, even if the file only holds a Sitemap: line.

Closed to AI, open to search. Take the starter and move the search and live-fetch tokens into the Disallow: / group: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, MistralAI-Index, plus ChatGPT-User, Claude-User, Perplexity-User, Amzn-User and MistralAI-User. Leave Googlebot and bingbot alone. Expect to disappear from AI answers, and treat that as a decision you made on purpose. For the argument about whether this is worth it, read the training-opt-out evidence rather than this guide.

When a rule does not seem to work

• "GPTBot is still in my logs after I blocked it." Wait 24 hours for OpenAI to pick up the change, then check the source IP against the ranges OpenAI publishes. A request that only claims to be GPTBot can be anyone.
• "My named group ignores my * rules." That is the specificity rule working as designed. Repeat the Disallow lines inside the named group.
• "A bot I blocked still fetches pages when I ask an assistant about them." User-triggered fetchers such as Perplexity-User generally ignore robots.txt, and ChatGPT-User acts on a person's request. Use a firewall rule if you need those refused.
• "The scan says B2 is partial." B2 counts named groups. Fewer than four named AI bots gives partial credit, so add the missing groups from the starter.
• "The scan cannot find my robots.txt." Open the URL in a browser. A redirect to a login page or an HTML error page means crawlers see no valid file.

Maintenance

• Re-check the token list every quarter. Operators add bots: Amazon, Mistral and Meta all now split training, search and live fetch.
• After any theme change, plugin install or migration, re-run the scan. Platform editors overwrite robots.txt more often than you would expect.
• Keep the private paths in sync. Whenever you edit the * group, update every named group that repeats it.
• Do not use Disallow to hide a secret path. RFC 9309 notes that listing paths in robots.txt exposes them publicly.

If your site is behind Cloudflare, you can also enforce these preferences at the edge. Cloudflare AI Crawl Control shows how to allow or block each crawler and how its managed robots.txt joins your file.

What to do next

Run npx aiscan-cli yoursite.com and read B1 and B2. If B2 is partial, paste the starter, redeploy and scan again. For the wider picture, browse the AI readiness guides, and read the sitemap guide next, because the Sitemap: line sits in the same file.

If the file you serve blocks everything and you never meant it to, see Your robots.txt blocks everything by accident for how to find the stray Disallow: / and guard against it.

If a Python script reports your whole site as blocked while the file looks fine, the cause may be a 403 on robots.txt that Python's parser reads as disallow-all: see Python's robots.txt parser treats a 403 as disallow-all.