---
title: "Control which AI crawlers reach your site: a robots.txt starter"
slug: robots-txt-starter-for-ai-crawlers
published: 2026-10-07T08:16:12.2189+00:00
updated: 2026-10-07T08:16:12.2189+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: check:B1, check:B2, robots.txt, AI crawlers, GPTBot, ClaudeBot, bot access
description: "A copy-paste robots.txt for AI crawlers: block training bots, keep search and citations, and test every rule with an AIScan scan and a two-minute check."
url: https://aiscan.site/blog/robots-txt-starter-for-ai-crawlers
---

Most sites that "block AI" with a robots.txt file end up with one of two outcomes: a file that blocks nothing because a bot name is misspelled, or a file that blocks the crawlers that would have sent them traffic. This guide gives you a starter file you can paste in, the three decisions that shape it, and a two-minute test that proves each rule does what you meant. The starter below takes about 10 minutes to adapt and ship.

## Quick summary

| If you want to... | Block these tokens | Leave these open | Time |
|---|---|---|---|
| Stay in AI answers, stay out of training sets | `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, `Applebot-Extended`, `meta-externalagent`, `Amazonbot` | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `ChatGPT-User`, `Claude-User`, `Googlebot` | 10 min |
| Be open to everything | Nothing | Everything | 2 min (verify only) |
| Opt out of AI entirely | All of the above plus the search and user-fetch tokens | `Googlebot`, `bingbot` | 15 min |

**Expert guidance in one line:** [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) says each of its settings is independent, so a site can allow `OAI-SearchBot` to appear in ChatGPT search while disallowing `GPTBot`. Anthropic's [crawler help page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) draws the same split between `ClaudeBot` (training) and `Claude-User` (answering a person's question). Treat each operator as three bots, not one.

Check your current file first: `npx aiscan-cli yoursite.com` reports B1 (Content Signals) and B2 (explicit AI bot rules) in about 20 seconds.

## What a robots.txt file can and cannot do

[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.txt) defines the format and says plainly that the Robots Exclusion Protocol is not a substitute for valid content security measures. A rule is a request. A well-behaved crawler reads it and complies. A badly behaved one ignores it.

That is not a theory. In AIScan's measurement of 40 hosts, a full `Disallow: /` was backed by an actual refusal at the server edge only 45% to 67% of the time, depending on the bot (see [Does blocking AI training protect your content?](/blog/does-blocking-ai-training-protect-content)). Use robots.txt to state your policy. Use your CDN or firewall if you need to enforce it.

Three mechanics decide whether your file works:

1. **A bot obeys the most specific group that names it.** Per RFC 9309 section 2.2.1, if no group matches a bot's token, it falls back to the `*` group. If a group does match, the `*` group no longer applies to it. Add a named group for `OAI-SearchBot` and it stops inheriting your `Disallow: /admin/` line from `*`, unless you repeat it.
2. **Tokens are not user-agent strings.** Google's [crawler documentation](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) says `Google-Extended` has no separate HTTP user agent. Crawling uses existing Google strings and the token only controls how the content is used. Applebot-Extended works the same way.
3. **Some user-triggered fetchers ignore robots.txt by design.** Perplexity documents that `Perplexity-User` generally ignores it. OpenAI describes `ChatGPT-User` as visiting pages when a person asks, not crawling automatically. The full list of 33 tokens, with the dates each was checked, is in [the AI crawler user-agent list](/blog/ai-crawler-user-agent-list-2026).

## Decide first: three jobs, three tokens

| Operator | Training | Search index | Live fetch for a person |
|---|---|---|---|
| OpenAI | `GPTBot` | `OAI-SearchBot` | `ChatGPT-User` |
| Anthropic | `ClaudeBot` | `Claude-SearchBot` | `Claude-User` |
| Perplexity | none | `PerplexityBot` | `Perplexity-User` |
| Amazon | `Amazonbot` | `Amzn-SearchBot` | `Amzn-User` |
| Mistral | `MistralAI-Training` | `MistralAI-Index` | `MistralAI-User` |
| Google | `Google-Extended` (token only) | `Googlebot` | `Google-Agent` |
| Common Crawl | `CCBot` | none | none |

Blocking the first column keeps you out of future training sets. Blocking the second removes you from that operator's search answers. Blocking the third stops an assistant fetching your page when a person asks about it. Anthropic states that disabling `Claude-User` may reduce your site's visibility. Most publishers who want citations block the first column and leave the other two open. That is the profile in the starter file.

Training exclusion applies to future collection. Nothing in a robots.txt file removes content a crawler already holds.

## Path 1: write the file by hand

Use this on any host where you can edit the file at the site root.

1. Open or create `robots.txt` at your site root. It must be reachable at `https://yoursite.com/robots.txt`, all lowercase, and return a 200 with `text/plain`.
2. Paste the starter below, replace `example.com` and the two `Disallow` paths with your own private paths.
3. Keep your existing `Sitemap:` line. Crawlers may read it, but RFC 9309 does not define it, so do not rely on it as a rule.
4. Save and deploy. OpenAI says its systems take about 24 hours to pick up a change, so do not judge the result from your logs the same afternoon.

```
User-agent: *
Disallow: /admin/
Disallow: /cart/

# Training crawlers: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
Disallow: /

# Search crawlers: stay visible (repeat the private paths from the * group)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml
```

Several `User-agent` lines above one set of rules is valid under RFC 9309 and keeps the file short. Bots with no group, such as `ChatGPT-User`, `Claude-User` and `Googlebot`, fall through to `*` and are only kept out of `/admin/` and `/cart/`.

To declare your policy in one extra line, add a Content Signal. Cloudflare's proposal uses `Content-Signal: search=yes, ai-train=no, ai-input=yes`. It is a draft and not part of RFC 9309, so treat it as a statement of intent. AIScan reports it as check B1 and it is informational only.

## Path 2: use your platform's editor

**WordPress.** Three plugins often compete to write robots.txt and the last one loaded wins. [ThinkRank](https://thinkrank.ai) manages robots.txt, robots meta, schema and llms.txt from one plugin, so there is a single place to paste the starter, and it imports settings from Rank Math, Yoast, AIOSEO and SEOPress so nothing is re-entered. Rank Math and Yoast both have a robots.txt editor too and are fine if you already run one of them. Whichever you use, deactivate the other editors so only one file is served, then paste the starter.

**Shopify.** Shopify's [robots.txt.liquid documentation](https://shopify.dev/docs/storefronts/themes/seo/robots-txt) says adding that template replaces the generated file. In the theme code editor, right-click the **Templates** folder, choose **New File**, name it `robots.txt.liquid`, and save a copy of your current `/robots.txt` output first so you can compare. Keep Shopify's default rules in the template and add the training groups from the starter. For the files that sit beside robots.txt, [StoreSEO](https://storeseo.com/) generates llms.txt from live products, collections, pages and articles and has an agents.md editor.

**Wix, Docusaurus and others.** Platform editors rewrite defaults in different ways. See [robots.txt for AI bots on Wix](/blog/robots-txt-ai-bots-wix) for a worked example of a named group replacing the defaults.

## Verify it did what you meant

**Start with a scan.** Run `npx aiscan-cli yoursite.com`, or paste your URL at [aiscan.site](https://aiscan.site). It is free and needs no account. Three checks apply here:

- **B1, Content Signals:** whether robots.txt declares AI usage preferences. Informational, it cannot lower your score.
- **B2, explicit AI bot rules:** whether the AI crawlers have their own named groups. Four or more named bots passes, one to three is partial.
- **B3, Web Bot Auth:** a signing key directory for bots you operate. Most site owners publish nothing here.

The scan reads what you publish. It cannot see your server logs or whether the server refuses a bot, which is why the hand checks below still matter. Details for each check are on [the bot access checks page](/docs/checks/bot-access).

If you would rather check by hand:

```bash
curl -sI https://yoursite.com/robots.txt | head -3      # expect 200 and text/plain
curl -s https://yoursite.com/robots.txt | grep -ciE 'gptbot|claudebot|oai-searchbot'
python3 -c "
from urllib.robotparser import RobotFileParser as R
r=R(); r.set_url('https://yoursite.com/robots.txt'); r.read()
for ua in ['GPTBot','ClaudeBot','OAI-SearchBot','PerplexityBot','Googlebot']:
    print(ua, r.can_fetch(ua,'https://yoursite.com/'), r.can_fetch(ua,'https://yoursite.com/admin/x'))
"
```

Against the starter file above, that check prints `False False` for `GPTBot` and `ClaudeBot`, `True False` for `OAI-SearchBot`, `PerplexityBot` and `Googlebot`. We ran it on 7 October 2026. The expected result for your site is the same pattern. Python's parser applies first-match rules rather than the longest-match rule in RFC 9309, so use it for group logic, not for complicated Allow and Disallow overlaps.

To test enforcement rather than the file, request a page with the bot's user-agent string and compare it to a browser request, as shown in [Fetch your site the way an AI agent sees it](/blog/fetch-site-as-ai-agent-curl). A spoofed string proves the server rule, not the real crawler, so check real traffic against the operator's published IP ranges.

## Two variants of the starter

**Open to everything.** Write nothing. An absent or empty robots.txt means crawl freely. The trap is a robots.txt URL that returns a 500 or an HTML soft-404 page, which some crawlers treat as a signal to stay away. Make sure the URL returns a real 200 with plain text, even if the file only holds a `Sitemap:` line.

**Closed to AI, open to search.** Take the starter and move the search and live-fetch tokens into the `Disallow: /` group: `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Amzn-SearchBot`, `MistralAI-Index`, plus `ChatGPT-User`, `Claude-User`, `Perplexity-User`, `Amzn-User` and `MistralAI-User`. Leave `Googlebot` and `bingbot` alone. Expect to disappear from AI answers, and treat that as a decision you made on purpose. For the argument about whether this is worth it, read [the training-opt-out evidence](/blog/does-blocking-ai-training-protect-content) rather than this guide.

## When a rule does not seem to work

- **"GPTBot is still in my logs after I blocked it."** Wait 24 hours for OpenAI to pick up the change, then check the source IP against the ranges OpenAI publishes. A request that only claims to be `GPTBot` can be anyone.
- **"My named group ignores my `*` rules."** That is the specificity rule working as designed. Repeat the `Disallow` lines inside the named group.
- **"A bot I blocked still fetches pages when I ask an assistant about them."** User-triggered fetchers such as `Perplexity-User` generally ignore robots.txt, and `ChatGPT-User` acts on a person's request. Use a firewall rule if you need those refused.
- **"The scan says B2 is partial."** B2 counts named groups. Fewer than four named AI bots gives partial credit, so add the missing groups from the starter.
- **"The scan cannot find my robots.txt."** Open the URL in a browser. A redirect to a login page or an HTML error page means crawlers see no valid file.

## Maintenance

- Re-check the token list every quarter. Operators add bots: Amazon, Mistral and Meta all now split training, search and live fetch.
- After any theme change, plugin install or migration, re-run the scan. Platform editors overwrite robots.txt more often than you would expect.
- Keep the private paths in sync. Whenever you edit the `*` group, update every named group that repeats it.
- Do not use `Disallow` to hide a secret path. RFC 9309 notes that listing paths in robots.txt exposes them publicly.

## What to do next

Run `npx aiscan-cli yoursite.com` and read B1 and B2. If B2 is partial, paste the starter, redeploy and scan again. For the wider picture, browse the [AI readiness guides](/guides), and read [the sitemap guide](/blog/sitemap-ai-crawlers-can-find) next, because the `Sitemap:` line sits in the same file.