What Sites Tell GPTBot in robots.txt, and Whether They Mean It

We pulled robots.txt from 538 hosts in the AIScan scan corpus on 2 October 2026 and parsed 451 of them with a group parser that follows RFC 9309. The headline: 173 of the 451 files (38.4%) name at least one of fourteen AI crawler tokens, and only 30 files (6.7%) contain a full-site block aimed at GPTBot, ClaudeBot, CCBot, Google-Extended or PerplexityBot. Then we asked the question a declaration census usually skips: when a site says no, does the server agree? For GPTBot, 7 of 19 blocking hosts we could test refused a self-declared GPTBot request. For PerplexityBot it was 1 of 9.

Quick summary

Best for: a site owner deciding what to put in robots.txt next, and anyone quoting an "X% of sites block AI" figure who wants to know what that number does and does not measure.

If you want the same answer for your own domain, run npx aiscan-cli yoursite.com (or paste the URL at aiscan.site, free, no account). Check B2 reports whether your file contains explicit AI crawler rules, and check B1 reports whether it carries a Content Signals declaration. The manual route is further down.

What was counted, and what was not

The corpus is the set of hosts people have submitted to AIScan. It leans toward small business sites, agency and developer portfolios, WordPress and Shopify plugin vendors, and a few dozen large brands. It is not a random sample of the web, and nothing here should be quoted as "the web does X". It is a census of 538 specific hosts, 451 of which served a usable file.

Fetch. One GET https://<host>/robots.txt per host, following redirects, 15-second limit, a desktop Chrome user agent. Of 538 attempts:

The 17 soft-200 hosts matter. They return a page with status 200 where a robots file should be. A tool that treats any 200 as "file present" counts those hosts as having a policy. We dropped them. This is the same trap the sitemap study found on D2, and it applies to every well-known path.

Parse. We split each file into groups the way RFC 9309 describes: consecutive User-agent lines share a group, a crawler uses the group that names its own token and falls back to * only when no group names it, and the Allow and Disallow lines of the chosen group set the policy. We matched tokens case-insensitively. For each token we recorded one of five states:

• blocked: Disallow: / with no Allow: / in the same group
• partial: some path is disallowed but not the root
• allowed: the token has a group or falls to * and nothing is disallowed
• named: a separate flag, true when a group lists the token itself
• none: no applicable group at all

"Blocked" is the strict reading. A site that disallows /blog/ for GPTBot is counted as partial, not blocked. We did not try to evaluate wildcards inside paths, so a few Disallow: /* style files may be classed as partial when a crawler would treat them as blocked.

Edge probe. For every host that blocked a token, we sent two requests for the home page, once with a desktop browser user agent as a control and then twice with that crawler's published user agent string. We called a host enforced when the control returned 200 and both crawler requests returned 401, 403, 406, 429, 451 or 503. We dropped hosts where the control itself did not return 200. We tested GPTBot, ClaudeBot, PerplexityBot, CCBot and Bytespider.

The honest limit of the probe: we sent the user agent string from one datacentre address. A site that checks the source IP against the operator's published ranges would pass the fake and look "open" to us, because it would serve a request it will not recognise as the real crawler anyway. Our "open" bucket therefore means "a self-declared crawler header from a cloud IP is not refused", which is the only thing a header-based rule can enforce and is the same test the sibling posts in this series use. It does not prove that the real GPTBot from OpenAI's ranges gets through. See the user-agent list post for how verification by IP and reverse DNS works per operator.

The declaration table

Percentages use the 451 parsed files as the base.

Three patterns stand out.

Naming is common, blocking is not. GPTBot, ClaudeBot, PerplexityBot and Google-Extended are each named in about a third of the parsed files. Yet for GPTBot only 14 of the 164 files that name it (8.5%) block it outright. The other 150 either allow it or restrict a subset of paths. The largest single group, 94 files, names GPTBot with nothing disallowed. We did not trace where each file came from, but the shape is a list of explicit allowances for individual AI bots. It is a declaration of welcome, and it is what check B2 counts as an explicit AI crawler rule.

The blockers who do block, block the older, louder names. Among tokens where a site had to write its own group to block, Bytespider (22) and CCBot (18) lead, ahead of GPTBot (14). Both are older, narrower-purpose names than the AI search tokens. Block rates fall as tokens get newer: Claude-SearchBot is named in 57 files and blocked by name in none.

The * block is a separate population. Between 6 and 9 hosts depending on token get their block from User-agent: * rather than from a rule about AI. These are staging sites, pre-launch sites and a few private tools that disallow everything for everyone. Eleven files in total carry a full * block (the count differs by token because a token with its own Allow: / group escapes it). They are not an AI policy, and a census that counts them as AI blockers overstates the stance.

What the numbers say about the "X% of sites block AI" headline

Across the five tokens that matter most for training and general crawling (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot), 30 of 451 files (6.7%) contain a full block of at least one. Count only blocks written for an AI token by name, and the figure is lower still. Count partial restrictions, and 56 files restrict GPTBot to some paths. Any headline percentage depends entirely on which of those three questions it answers, and most articles do not say.

Our own corpus skews toward sites that care about being found by AI systems enough to run a scan, so this figure probably understates blocking among large publishers. For that population, see the training-opt-out post, which covers what a block does and does not protect.

Training token versus user-driven token

Operators now publish separate tokens for bulk collection and for a fetch triggered by one person's request. OpenAI lists GPTBot and ChatGPT-User, Anthropic lists ClaudeBot and Claude-User, Perplexity lists PerplexityBot and Perplexity-User. The debate over whether the second kind should obey robots.txt at all is covered in the stealth crawling post. Here we only count what sites wrote.

Two readings.

First, a deliberate split exists but is rare. Nine hosts block GPTBot and leave ChatGPT-User open; nine do the same for ClaudeBot and Claude-User. Eleven block GPTBot and leave OAI-SearchBot, the search crawler, open. That pattern is what an owner writes when the goal is "stay out of training, stay visible in answers". Fewer than one file in forty makes that choice, but it is the most precise policy anyone wrote.

Second, a block on the training token usually reaches the other tokens by accident. Most of the "both blocked" rows come from * blocks, or from a template that lists several AI user agents under one Disallow: /. An owner who wants the split has to write separate groups, because a group names a token only when its User-agent line does.

Does the server agree? Declaration against enforcement

For every host whose file blocked a token, we sent the token's user agent string at the home page. Control requests with a desktop browser had to return 200 for a host to count.

Roughly half of the blocks are backed by an edge rule at best, and for PerplexityBot the file and the server rarely agree. That is RFC 9309 working as written: the standard says the rules in the file are not a form of access authorization, and a crawler that chooses to ignore them can. The same spread showed up in our earlier matched-pair work, where only 45% to 67% of full blocks were backed by an edge refusal. A different sample, a different token mix and a different method gave a similar band, which is the part we would trust.

Two cautions on reading the table. The "refused" column needs an active rule at the CDN or web server, for example a WAF rule on user agent, so a high figure means someone configured the server, though we did not identify which product did it on each host. And the IP-checking caveat from the method section still holds: a host in the "served normally" column might still refuse the genuine crawler by address. The column is a measure of what a header can do, not of what happens to OpenAI's actual traffic.

The sentence to keep

A Disallow: / for a crawler tells the crawlers that read it what you want. It does nothing about the ones that do not. If the content must stay out of reach, the control is at the server, and the file is the polite request that sits alongside it.

Four patterns worth a second look

1. The template that welcomes everything. The 94 files that name GPTBot with nothing disallowed are mostly boilerplate: an explicit Allow: / for each AI bot, listed one after another. They are harmless. They also make check B2 pass, and rightly so, because the file is an explicit statement. But an Allow: / group for GPTBot overrides a * block for GPTBot only. Anyone relying on a * block to keep AI out, who then pastes in an AI-bot allow template, has opened the door for exactly those bots.

2. Legacy tokens still doing the work. anthropic-ai is named in 86 files and blocked by name in 8, against 10 for ClaudeBot. The older name still appears in templates years after the operator's documentation moved to newer tokens. A file that blocks only anthropic-ai has made a policy for a token that is not the one the operator's current crawler announces itself with. Check the operator's current page before you copy a list.

3. Newer tokens barely appear. Claude-SearchBot is named in 57 files (12.6%) and Claude-User in 62 (13.7%). A site that wrote its AI rules in 2024 or early 2025 never mentions them, so they fall to *. For most files that means they are allowed, which is the default, but for a site that wants to block "all AI" the list is already out of date.

4. Soft-200 and blocked robots files. Of 538 hosts, 17 served HTML at the robots path, 11 returned 403, 3 returned 429 and 28 never connected. RFC 9309 treats a 4xx answer for the file as "no rules", so a crawler may fetch anything, while a 5xx or an unreachable server means the crawler should assume everything is disallowed. We could not tell from outside whether any of the 11 forbidden responses was intentional, and a 403 meant for scrapers can land on a crawler that would have obeyed the file. That is 70 of 538 hosts (13.0%) where the file as a declaration was missing or unreadable.

How to check your own file

Start with the scan. npx aiscan-cli yoursite.com runs B1 (Content Signals declared) and B2 (explicit rules for AI crawlers) among its checks, and the report shows the rule it found. AIScan reads the file as served, so it also notices when the path returns a page instead of a file. What it cannot see: whether your server actually refuses a given crawler, which needs the request-level test described above, and whether a CDN rule changes the answer by source address.

If you would rather check by hand:

curl -sIL https://example.com/robots.txt | head -5
curl -sL https://example.com/robots.txt | grep -iE 'gptbot|claudebot|perplexity|ccbot|google-extended'
curl -sI -A "GPTBot/1.2" https://example.com/ | head -1
curl -sI -A "Mozilla/5.0 Chrome/124" https://example.com/ | head -1

The first line should show a 200 and a text/plain content type. The last two should differ if your server enforces the block.

On WordPress

ThinkRank is the plugin we would reach for here. It manages robots.txt, robots meta, schema and llms.txt from one place, so the AI-bot rules are written once and not by three plugins editing the same virtual file. It migrates settings from Rank Math, Yoast, AIOSEO and SEOPress, so switching does not mean re-entering what you already set. Rank Math, Yoast SEO and AIOSEO all let you edit the robots.txt text, and each is a fair choice if it is already installed. Whichever you use, confirm the served file with the scan afterwards, because a physical robots.txt in the web root overrides every plugin.

On Shopify

StoreSEO is the Shopify app to look at for store-level AI visibility work. Shopify generates robots.txt from a theme template (robots.txt.liquid), so the AI-bot group belongs there, and the file you serve is the one to verify.

A rule set that matches the census

A policy that separates training from answers, using the tokens operators publish as of this date:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /admin/

Each group names its tokens, so nothing falls to * by accident. Then back it with a server rule if you need the training block to hold, and re-test after any CDN change. Put a Content Signals line in the file as well, so B1 has something to read. See the bot access docs for what B1 and B2 each look for.

Common mistakes the census suggests

Next action

Run npx aiscan-cli yoursite.com and read checks B1 and B2, then use the four curl lines above to test whether your server agrees with your file. Browse the rest of the series in the guides index and the bot access check page.