Control which AI crawlers reach your site: a robots.txt starter

Most sites that "block AI" with a robots.txt file end up with one of two outcomes: a file that blocks nothing because a bot name is misspelled, or a file that blocks the crawlers that would have sent them traffic. This guide gives you a starter file you can paste in, the three decisions that shape it, and a two-minute test that proves each rule does what you meant. The starter below takes about 10 minutes to adapt and ship.

Quick summary

Expert guidance in one line: OpenAI's crawler documentation says each of its settings is independent, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Anthropic's crawler help page draws the same split between ClaudeBot (training) and Claude-User (answering a person's question). Treat each operator as three bots, not one.

Check your current file first: npx aiscan-cli yoursite.com reports B1 (Content Signals) and B2 (explicit AI bot rules) in about 20 seconds.

What a robots.txt file can and cannot do

RFC 9309 defines the format and says plainly that the Robots Exclusion Protocol is not a substitute for valid content security measures. A rule is a request. A well-behaved crawler reads it and complies. A badly behaved one ignores it.

That is not a theory. In AIScan's measurement of 40 hosts, a full Disallow: / was backed by an actual refusal at the server edge only 45% to 67% of the time, depending on the bot (see Does blocking AI training protect your content?). Use robots.txt to state your policy. Use your CDN or firewall if you need to enforce it.

Three mechanics decide whether your file works:

1. A bot obeys the most specific group that names it. Per RFC 9309 section 2.2.1, if no group matches a bot's token, it falls back to the * group. If a group does match, the * group no longer applies to it. Add a named group for OAI-SearchBot and it stops inheriting your Disallow: /admin/ line from *, unless you repeat it.
2. Tokens are not user-agent strings. Google's crawler documentation says Google-Extended has no separate HTTP user agent. Crawling uses existing Google strings and the token only controls how the content is used. Applebot-Extended works the same way.
3. Some user-triggered fetchers ignore robots.txt by design. Perplexity documents that Perplexity-User generally ignores it. OpenAI describes ChatGPT-User as visiting pages when a person asks, not crawling automatically. The full list of 33 tokens, with the dates each was checked, is in the AI crawler user-agent list.

Decide first: three jobs, three tokens

Blocking the first column keeps you out of future training sets. Blocking the second removes you from that operator's search answers. Blocking the third stops an assistant fetching your page when a person asks about it. Anthropic states that disabling Claude-User may reduce your site's visibility. Most publishers who want citations block the first column and leave the other two open. That is the profile in the starter file.

Training exclusion applies to future collection. Nothing in a robots.txt file removes content a crawler already holds.

Path 1: write the file by hand

Use this on any host where you can edit the file at the site root.

1. Open or create robots.txt at your site root. It must be reachable at https://yoursite.com/robots.txt, all lowercase, and return a 200 with text/plain.
2. Paste the starter below, replace example.com and the two Disallow paths with your own private paths.
3. Keep your existing Sitemap: line. Crawlers may read it, but RFC 9309 does not define it, so do not rely on it as a rule.
4. Save and deploy. OpenAI says its systems take about 24 hours to pick up a change, so do not judge the result from your logs the same afternoon.

User-agent: *
Disallow: /admin/
Disallow: /cart/

# Training crawlers: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
Disallow: /

# Search crawlers: stay visible (repeat the private paths from the * group)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

Several User-agent lines above one set of rules is valid under RFC 9309 and keeps the file short. Bots with no group, such as ChatGPT-User, Claude-User and Googlebot, fall through to * and are only kept out of /admin/ and /cart/.

To declare your policy in one extra line, add a Content Signal. Cloudflare's proposal uses Content-Signal: search=yes, ai-train=no, ai-input=yes. It is a draft and not part of RFC 9309, so treat it as a statement of intent. AIScan reports it as check B1 and it is informational only.

Path 2: use your platform's editor

WordPress. Three plugins often compete to write robots.txt and the last one loaded wins. ThinkRank manages robots.txt, robots meta, schema and llms.txt from one plugin, so there is a single place to paste the starter, and it imports settings from Rank Math, Yoast, AIOSEO and SEOPress so nothing is re-entered. Rank Math and Yoast both have a robots.txt editor too and are fine if you already run one of them. Whichever you use, deactivate the other editors so only one file is served, then paste the starter.

Shopify. Shopify's robots.txt.liquid documentation says adding that template replaces the generated file. In the theme code editor, right-click the Templates folder, choose New File, name it robots.txt.liquid, and save a copy of your current /robots.txt output first so you can compare. Keep Shopify's default rules in the template and add the training groups from the starter. For the files that sit beside robots.txt, StoreSEO generates llms.txt from live products, collections, pages and articles and has an agents.md editor.

Wix, Docusaurus and others. Platform editors rewrite defaults in different ways. See robots.txt for AI bots on Wix for a worked example of a named group replacing the defaults.

Verify it did what you meant

Start with a scan. Run npx aiscan-cli yoursite.com, or paste your URL at aiscan.site. It is free and needs no account. Three checks apply here:

• B1, Content Signals: whether robots.txt declares AI usage preferences. Informational, it cannot lower your score.
• B2, explicit AI bot rules: whether the AI crawlers have their own named groups. Four or more named bots passes, one to three is partial.
• B3, Web Bot Auth: a signing key directory for bots you operate. Most site owners publish nothing here.

The scan reads what you publish. It cannot see your server logs or whether the server refuses a bot, which is why the hand checks below still matter. Details for each check are on the bot access checks page.

If you would rather check by hand:

curl -sI https://yoursite.com/robots.txt | head -3      # expect 200 and text/plain
curl -s https://yoursite.com/robots.txt | grep -ciE 'gptbot|claudebot|oai-searchbot'
python3 -c "
from urllib.robotparser import RobotFileParser as R
r=R(); r.set_url('https://yoursite.com/robots.txt'); r.read()
for ua in ['GPTBot','ClaudeBot','OAI-SearchBot','PerplexityBot','Googlebot']:
    print(ua, r.can_fetch(ua,'https://yoursite.com/'), r.can_fetch(ua,'https://yoursite.com/admin/x'))
"

Against the starter file above, that check prints False False for GPTBot and ClaudeBot, True False for OAI-SearchBot, PerplexityBot and Googlebot. We ran it on 7 October 2026. The expected result for your site is the same pattern. Python's parser applies first-match rules rather than the longest-match rule in RFC 9309, so use it for group logic, not for complicated Allow and Disallow overlaps.

To test enforcement rather than the file, request a page with the bot's user-agent string and compare it to a browser request, as shown in Fetch your site the way an AI agent sees it. A spoofed string proves the server rule, not the real crawler, so check real traffic against the operator's published IP ranges.

Two variants of the starter

Open to everything. Write nothing. An absent or empty robots.txt means crawl freely. The trap is a robots.txt URL that returns a 500 or an HTML soft-404 page, which some crawlers treat as a signal to stay away. Make sure the URL returns a real 200 with plain text, even if the file only holds a Sitemap: line.

Closed to AI, open to search. Take the starter and move the search and live-fetch tokens into the Disallow: / group: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, MistralAI-Index, plus ChatGPT-User, Claude-User, Perplexity-User, Amzn-User and MistralAI-User. Leave Googlebot and bingbot alone. Expect to disappear from AI answers, and treat that as a decision you made on purpose. For the argument about whether this is worth it, read the training-opt-out evidence rather than this guide.

When a rule does not seem to work

• "GPTBot is still in my logs after I blocked it." Wait 24 hours for OpenAI to pick up the change, then check the source IP against the ranges OpenAI publishes. A request that only claims to be GPTBot can be anyone.
• "My named group ignores my * rules." That is the specificity rule working as designed. Repeat the Disallow lines inside the named group.
• "A bot I blocked still fetches pages when I ask an assistant about them." User-triggered fetchers such as Perplexity-User generally ignore robots.txt, and ChatGPT-User acts on a person's request. Use a firewall rule if you need those refused.
• "The scan says B2 is partial." B2 counts named groups. Fewer than four named AI bots gives partial credit, so add the missing groups from the starter.
• "The scan cannot find my robots.txt." Open the URL in a browser. A redirect to a login page or an HTML error page means crawlers see no valid file.

Maintenance

• Re-check the token list every quarter. Operators add bots: Amazon, Mistral and Meta all now split training, search and live fetch.
• After any theme change, plugin install or migration, re-run the scan. Platform editors overwrite robots.txt more often than you would expect.
• Keep the private paths in sync. Whenever you edit the * group, update every named group that repeats it.
• Do not use Disallow to hide a secret path. RFC 9309 notes that listing paths in robots.txt exposes them publicly.

What to do next

Run npx aiscan-cli yoursite.com and read B1 and B2. If B2 is partial, paste the starter, redeploy and scan again. For the wider picture, browse the AI readiness guides, and read the sitemap guide next, because the Sitemap: line sits in the same file.