---
title: "Every AI Crawler User-Agent in 2026: The Verified List and How to Spot a Fake"
slug: ai-crawler-user-agent-list-2026
published: 2026-08-27T03:28:10.542699+00:00
updated: 2026-08-27T08:48:00.894833+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: AI crawlers, user agent, robots.txt, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, bot verification, reverse DNS, check:B1, check:B2, check:B3
description: "Every AI crawler user-agent documented by its operator, verified 27 Aug 2026, plus the three major crawlers where reverse DNS verification quietly fails."
url: https://aiscan.site/blog/ai-crawler-user-agent-list-2026
---

**Verified 27 August 2026.** Every user-agent below was read from the operator's own documentation on that date. Every IP range file was fetched and counted. Every reverse-DNS result was tested live.

Roughly 1 in every 18 requests carrying an AI crawler user-agent is fake. That is [**HUMAN Security's Satori Threat Intelligence team**](https://www.humansecurity.com/learn/blog/ai-crawler-spoofing-chatgpt-mistral-perplexity/), reporting a 1:17 average spoof ratio across a two-week study published 9 September 2025. `ChatGPT-User` was the worst at 1:5.

Here is the awkward part. The verification method almost every guide recommends, forward-confirmed reverse DNS, does not work for OpenAI, Anthropic or Perplexity. Not "works badly". Does not work. I ran it against their own published IP ranges this morning and got back no PTR record for OpenAI, no PTR record for Anthropic's primary range, and for Perplexity a hostname reading `ec2-107-20-236-150.compute-1.amazonaws.com`, which anyone renting an EC2 instance can obtain.

This is the current list of AI crawler user-agents, what each one is actually for, and the verification method that works for each.

## Quick summary

| If you want to… | Do this | Which bots it affects | Time |
|---|---|---|---|
| Appear in ChatGPT answers | Allow `OAI-SearchBot` | OpenAI retrieval | 2 min + ~24h propagation |
| Stay out of model training | Disallow `GPTBot`, `ClaudeBot`, `CCBot`, `meta-externalagent`, `Applebot-Extended`, `Google-Extended`, `MistralAI-Training` | Training bots only | 10 min |
| Confirm a bot is genuine | Match its source IP against the operator's published JSON | All except Meta, ByteDance, DuckDuckGo, AI2 | 1 min per request |
| Find out what your site currently allows | `npx aiscan-cli yoursite.com` — checks B1, B2, B3 | Everything above | ~30 seconds |
| Catch a spoofed crawler in your logs | IP-range match, not reverse DNS, for OpenAI / Anthropic / Perplexity | The three most-impersonated brands | Ongoing |

**The single most useful thing on this page:** the verification-method column in the table further down. Reverse DNS works for Google, Apple, Bing and Common Crawl. It is unavailable for OpenAI, Anthropic and Perplexity, and it produces a false pass for Perplexity. If your bot-verification logic is built on FCrDNS alone, it is verifying four crawlers and quietly waving through the rest.

## Most published crawler lists are a year out of date

Three things changed in 2026 that older lists have not caught up with.

**Operators split into three bots, not two.** OpenAI, Anthropic, Amazon, Mistral and Meta now each run a separate crawler for training, for search indexing, and for live user-triggered fetches. Amazon's own doc is explicit that "each user agent setting is independent of the others."

**New user-agents nobody covers.** `OAI-AdsBot`, `Amzn-SearchBot`, `Amzn-User`, `MistralAI-Index`, `MistralAI-Training`, `Meta-WebIndexer`, `Google-Agent` and `Google-GeminiNotebook` are all documented and missing from most published lists.

**Some tokens are retired.** `anthropic-ai` and `Claude-Web` no longer appear in Anthropic's crawler documentation, which lists exactly three bots. `FacebookBot` is gone from Meta's crawler page. Leaving them in your robots.txt is harmless, but do not present them as current, and do not build a monitoring rule around them.

One more trap worth knowing: Google moved its crawler documentation from `/search/docs/crawling-indexing/*` to `/crawling/docs/*` this year. Any article still linking the old path is already pointing at a stub.

## Every documented AI crawler user-agent, August 2026

`Robots` column: **Y** = obeys robots.txt · **N** = documented to ignore or bypass it · **Partial** = may not apply because a human triggered the fetch.

| Operator | Token for robots.txt | Job | Robots | Verification |
|---|---|---|---|---|
| OpenAI | `GPTBot` | Training | Y | [gptbot.json](https://openai.com/gptbot.json) — 21 ranges |
| OpenAI | `OAI-SearchBot` | Search index / ChatGPT citations | Y | [searchbot.json](https://openai.com/searchbot.json) — 35 ranges |
| OpenAI | `ChatGPT-User` | Live user-triggered fetch | Partial | [chatgpt-user.json](https://openai.com/chatgpt-user.json) — 204 ranges |
| OpenAI | `OAI-AdsBot` | Ad landing-page safety check | not stated | [adsbot.json](https://openai.com/adsbot.json) — 2 ranges |
| Anthropic | `ClaudeBot` | Training | Y | [bots.json](https://claude.com/crawling/bots.json) — 26 ranges, shared |
| Anthropic | `Claude-SearchBot` | Search index | Y | same file |
| Anthropic | `Claude-User` | Live user-triggered fetch | Y | same file |
| Google | `Googlebot` | Search crawl | Y | rDNS + [googlebot.json](https://developers.google.com/search/apis/ipranges/googlebot.json) — 315 |
| Google | `Google-Extended` | Gemini training opt-out (token only, no UA string) | Y | n/a |
| Google | `GoogleOther` | Internal R&D crawl | Y | rDNS + special-crawlers.json |
| Google | `Google-CloudVertexBot` | Vertex AI customer-directed crawl | Y | rDNS + IP JSON |
| Google | `Google-Agent` | Agentic browsing | fetcher — generally ignores | rDNS + IP JSON + **Web Bot Auth** |
| Google | `Google-NotebookLM` (UA reads `Google-GeminiNotebook`) | NotebookLM source fetch | fetcher — generally ignores | rDNS + IP JSON |
| Google | `Google-Safety` | Abuse and malware scanning | **N — explicitly** | rDNS |
| Perplexity | `PerplexityBot` | Search index / linking | Y | [perplexitybot.json](https://www.perplexity.com/perplexitybot.json) — 8 ranges |
| Perplexity | `Perplexity-User` | Live user-triggered fetch | **N — "generally ignores robots.txt"** | [perplexity-user.json](https://www.perplexity.com/perplexity-user.json) — 4 ranges |
| Meta | `meta-externalagent` | Training + indexing | Y | none published |
| Meta | `meta-webindexer` | Meta AI search index / citations | Y | none published |
| Meta | `meta-externalfetcher` | Live / agentic fetch | **N — may bypass** | none published |
| Meta | `meta-externalads` | Ads product crawl | Y | none published |
| Meta | `facebookexternalhit` | Link unfurl | N for integrity checks | none published |
| Apple | `Applebot` | Siri / Spotlight / Safari search | Y | rDNS + [applebot.json](https://search.developer.apple.com/applebot.json) — 33 |
| Apple | `Applebot-Extended` | Foundation-model training opt-out (token only) | Y | n/a |
| Amazon | `Amazonbot` | Training + product improvement | Y | IP list (HTML page) |
| Amazon | `Amzn-SearchBot` | Amazon / Alexa search index | Y | IP list (HTML page) |
| Amazon | `Amzn-User` | Live Alexa user fetch | Partial | IP list (HTML page) |
| Common Crawl | `CCBot` | Open training corpus | Y | rDNS + [ccbot.json](https://index.commoncrawl.org/ccbot.json) — 5 |
| Mistral | `MistralAI-Index` | Search index for Vibe | Y | [index IPs](https://mistral.ai/mistralai-index-ips.json) — 2 |
| Mistral | `MistralAI-User` | Live user fetch | Y | [user IPs](https://mistral.ai/mistralai-user-ips.json) — 4 |
| Mistral | `MistralAI-Training` | Training datasets | Y | **none — the IP file 404s** |
| DuckDuckGo | `DuckAssistBot` | DuckAssist answers | Y (72h to take effect) | none published |
| Allen Institute | `AI2Bot` | Open research corpus | Y | none published |

Sources, all fetched 27 August 2026: [OpenAI](https://platform.openai.com/docs/bots), [Anthropic](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent), [Perplexity](https://docs.perplexity.ai/guides/bots), [Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/), [Apple](https://support.apple.com/en-us/119829), [Amazon](https://developer.amazon.com/amazonbot), [Mistral](https://docs.mistral.ai/robots/), [Common Crawl](https://index.commoncrawl.org/ccbot.json).

### The exact strings

Match on the token, not the version number. OpenAI's doc says outright that the version may change.

```
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
CCBot/2.0 (https://commoncrawl.org/faq/)
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
```

`W.X.Y.Z` is literal in Amazon's documentation, not a placeholder they forgot to fill in.

## Three jobs, three opposite consequences

Blocking a training bot and blocking a retrieval bot are not the same decision, and getting them backwards is the most expensive mistake on this page. The operators say so themselves.

OpenAI, [verbatim](https://platform.openai.com/docs/bots):

> "Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models."

And on the cost of getting it wrong:

> "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

Anthropic on `Claude-User`: disabling it "prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search." Google on `Google-Extended`: it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Meta on `Meta-WebIndexer`: allowing it "helps us cite and link to your content in Meta AI's responses." Apple: disallowing `Applebot-Extended` blocks training, but "your content will remain discoverable through Spotlight, Siri, and Safari."

That is five operators saying the same thing in five different words. Training and retrieval are separate switches. A blanket `Disallow: /` for everything with "AI" in the name removes you from AI answers while doing nothing about the training corpora that already contain your pages.

One more OpenAI detail that undermines naive blocklists: "If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling."

Propagation is not instant either. OpenAI says about 24 hours from a robots.txt change. Perplexity says up to 24 hours. Amazon says roughly 24 hours and caches robots.txt for up to 30 days. DuckDuckGo says 72 hours.

## Verification, method one: the published IP ranges

Nine operators publish machine-readable IP ranges. All of these returned 200 on 27 August 2026, with the prefix count and file timestamp taken from the file itself:

| File | Ranges | Last regenerated |
|---|---|---|
| `openai.com/gptbot.json` | 21 | **30 Oct 2025** |
| `openai.com/searchbot.json` | 35 | 2 Jan 2026 |
| `openai.com/chatgpt-user.json` | 204 | 14 Aug 2026 |
| `openai.com/adsbot.json` | 2 | 12 May 2026 |
| `claude.com/crawling/bots.json` | 26 | 18 Aug 2026 |
| `perplexity.com/perplexitybot.json` | 8 | **7 Feb 2025** |
| `perplexity.com/perplexity-user.json` | 4 | 17 Oct 2025 |
| `developers.google.com/…/googlebot.json` | 315 | 26 Aug 2026 |
| `…/special-crawlers.json` | 270 | 26 Aug 2026 |
| `…/user-triggered-fetchers.json` | 1,056 | 26 Aug 2026 |
| `search.developer.apple.com/applebot.json` | 33 | 31 Jul 2026 |
| `index.commoncrawl.org/ccbot.json` | 5 | 11 Aug 2026 |
| `mistral.ai/mistralai-index-ips.json` | 2 | 19 Apr 2026 |
| `mistral.ai/mistralai-user-ips.json` | 4 | 19 Feb 2025 |

Two things stand out. Google regenerates daily. OpenAI's `gptbot.json` has not been touched since 30 October 2025, close to ten months, and Perplexity's `perplexitybot.json` since 7 February 2025, over eighteen. If you build an allowlist from either, you are trusting a very old snapshot, and a genuine crawler operating from a newer address will fail your check.

Two files that people cite do not exist. `https://www.anthropic.com/claudebot.json` returns 404; the live location is on `claude.com`. `https://mistral.ai/mistralai-training-ips.json` also 404s, so Mistral's training bot is the one crawler here you cannot verify by IP at all. Any crawler list still pointing at the anthropic.com URL is a quick test of whether that list is maintained.

Anthropic adds a warning worth repeating, because it is the opposite of what a lot of people do:

> "Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."

Block the bot in robots.txt. Use the IP list to confirm identity, not to enforce policy.

## Verification, method two: reverse DNS, and where it quietly fails

Forward-confirmed reverse DNS is the standard advice: take the source IP, look up its PTR record, resolve that hostname forward, and confirm it comes back to the same IP. It is a good method. It is also unavailable for the three brands most likely to be impersonated.

I ran it on 27 August 2026 against the first usable address in each operator's own published range:

| Crawler | Test IP | PTR result | Verdict |
|---|---|---|---|
| Googlebot | 66.249.66.1 | `crawl-66-249-66-1.googlebot.com` | **PASS** — operator-controlled suffix |
| Applebot | 17.241.208.161 | `17-241-208-161.applebot.apple.com` | **PASS** — operator-controlled suffix |
| Bingbot | 157.55.39.1 | `msnbot-157-55-39-1.search.msn.com` | **PASS** — operator-controlled suffix |
| CCBot | 18.97.14.84 | `18-97-14-84.crawl.commoncrawl.org` | **PASS** — operator-controlled suffix |
| GPTBot | 132.196.86.1 | no PTR record | **UNAVAILABLE** |
| ChatGPT-User | 104.208.184.193 | no PTR record | **UNAVAILABLE** |
| ClaudeBot | 216.73.216.1 | no PTR record | **UNAVAILABLE** |
| ClaudeBot (GCP range) | 34.162.230.222 | `222.230.162.34.bc.googleusercontent.com` | **FALSE PASS** — generic Google Cloud |
| PerplexityBot | 107.20.236.150 | `ec2-107-20-236-150.compute-1.amazonaws.com` | **FALSE PASS** — generic AWS |

![Forward-confirmed reverse DNS tested against nine AI crawler IP ranges on 27 August 2026: Googlebot, Applebot, Bingbot and CCBot verify; GPTBot, ChatGPT-User and ClaudeBot return no PTR record; ClaudeBot's Google Cloud range and PerplexityBot return generic cloud hostnames that produce a false pass.](https://gqdxuwsuuasfjeeqafyq.supabase.co/storage/v1/object/public/blog-covers/ua-list-2026-fcrdns.jpg)

The bottom two rows are the reason this section exists. A naive FCrDNS check on that Perplexity address passes: the PTR resolves, and it resolves forward to the same IP. What it proves is that the host rents an EC2 instance. Anyone can rent an EC2 instance. The same is true of the Anthropic address sitting in a Google Cloud range whose PTR ends in `bc.googleusercontent.com`.

So the rule is: **FCrDNS is only meaningful when you also assert the hostname suffix**, and only Google, Apple, Bing and Common Crawl publish a suffix worth asserting. Google's masks, from its own verification page:

```
crawl-***-***-***-***.googlebot.com
geo-crawl-***-***-***-***.geo.googlebot.com
rate-limited-proxy-***-***-***-***.google.com
google-proxy-***-***-***-***.google.com
***-***-***-***.gae.googleusercontent.com
```

For OpenAI, Anthropic and Perplexity, the IP list is the only method. There is no fallback.

## The two commands

Reverse DNS with a suffix assertion, which is the only form worth running:

```bash
python3 -c "
import socket,sys
ip,suf=sys.argv[1],sys.argv[2]
try: h=socket.gethostbyaddr(ip)[0]
except Exception: print(ip,'NO PTR -> use the IP list instead'); raise SystemExit
ok = h.endswith(suf) and ip in socket.gethostbyname_ex(h)[2]
print(ip, h, 'VERIFIED' if ok else 'REJECT')" 66.249.66.1 googlebot.com
```

Output on the test above: `66.249.66.1 crawl-66-249-66-1.googlebot.com VERIFIED`.

IP against a published CIDR list, which is what you need for the OpenAI/Anthropic/Perplexity group:

```bash
curl -sL --compressed -o gptbot.json https://openai.com/gptbot.json
python3 -c "
import json,ipaddress,sys
ip=ipaddress.ip_address(sys.argv[1])
nets=[ipaddress.ip_network(p.get('ipv4Prefix') or p.get('ipv6Prefix'))
      for p in json.load(open('gptbot.json'))['prefixes']]
print(sys.argv[1],'VERIFIED' if any(ip in n for n in nets) else 'SPOOFED')" 8.8.8.8
```

Refetch the JSON on a schedule. Google's changes daily; treating any of these as a static file is how you start rejecting real crawlers.

## Web Bot Auth is where this is going

Both problems above, no PTR records and stale IP files, go away if the crawler signs its requests cryptographically. That is what Web Bot Auth does, and two operators have shipped it.

Cloudflare, in the Perplexity investigation: "ChatGPT Agent is signing http requests using the newly proposed open standard Web Bot Auth." Google's `Google-Agent` documentation adds: "Google is also experimenting with the Web Bot Auth protocol, using the `https://agent.bot.goog` identity."

HUMAN Security makes the practical difference explicit: signed requests "cannot be spoofed in the same way as crawler traffic," because they are validated against the operator's keys rather than against a list of addresses. Two operators is not an ecosystem yet, but it is the direction, and worth watching before you invest heavily in IP-list tooling.

## How much of this traffic is actually fake

Two independent pieces of evidence, and both are worth reading in full.

**Cloudflare vs Perplexity, 4 August 2025.** Cloudflare created brand-new domains that had never been indexed or published anywhere, put a `Disallow: /` in robots.txt, and watched. From [their post](https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/):

> "We observed that Perplexity uses not only their declared user-agent, but also a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked."

Declared traffic ran 20–25 million requests a day; the undeclared browser-lookalike ran 3–6 million. Cloudflare adds that it "utilized multiple IPs not listed in Perplexity's official IP range" and rotated through ASNs, "across tens of thousands of domains and millions of requests per day."

The control condition in the same experiment is the part most coverage skipped: "we found that ChatGPT-User fetched the robots file and stopped crawling when it was disallowed. We did not observe follow-up crawls from any other user agents or third party bots."

**HUMAN Security, 9 September 2025.** A two-week study across HUMAN's customer base, spanning multiple industries and geographies, measuring how much traffic claims an AI user-agent but fails IP verification:

| User-agent | Spoof ratio |
|---|---|
| `ChatGPT-User` | 1:5 |
| `MistralAI-User` | 1:37 |
| `Perplexity-User` | 1:88 |

By brand: OpenAI 1:9, Mistral 1:37, Perplexity 1:138, DuckDuckGo 1:772. Overall, "the average spoof ratio is 1:17" and "spoofed requests make up 5.7% of all traffic labeled as coming from AI crawlers."

Note the direction of that finding. The most-impersonated user-agent belongs to the best-behaved operator, because impersonating a bot site owners deliberately allow is more useful than impersonating one they block. Both figures are September 2025; no vendor has published a verifiable 2026 equivalent, so treat them as a floor.

## robots.txt rules that trip people up

From [Google's robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec), which the other operators broadly follow:

- **The user-agent value is case-insensitive.** `User-agent: GPTBot` and `user-agent: gptbot` are the same rule.
- **Path values are case-sensitive.** So is the robots.txt URL itself.
- **Only one group applies.** Google picks "the most specific group that matches the crawler's user agent. Other groups are ignored." A bot named in its own group never reads your `User-agent: *` group. Writing `Disallow: /private/` under `*` and forgetting to repeat it under `GPTBot` leaves that path open to GPTBot.
- **You match the token, not the full string.** Never paste a whole Mozilla string into robots.txt.
- **Longest matching path wins**, and where rules conflict, Google applies the least restrictive one.

Two operator quirks: Anthropic supports the non-standard `Crawl-delay` directive, with `User-agent: ClaudeBot` / `Crawl-delay: 1` as their own example. And `Amzn-SearchBot`, per Amazon's doc, "will crawl in accordance with the robots.txt directives given to other search bots" if you never mention it, so a rule aimed at Googlebot can silently govern Amazon.

## The bots we could not verify

Every entry in the main table came from an operator's own documentation. These did not, and are listed separately rather than quietly folded in:

| Bot | Why it is not in the table |
|---|---|
| `Bytespider` (ByteDance) | No operator crawler documentation found anywhere |
| xAI / Grok crawler | `x.ai/crawlers` returns 403, `docs.x.ai/crawlers` returns 404 |
| `cohere-ai` | Every documentation path tried returns 404 or empty |
| `Diffbot` | Its crawler support page returns 404 |
| `Ai2Bot-Dolma` | Allen Institute's page documents `AI2Bot` only |
| Brave's crawler | Its crawler help page refuses automated requests |
| `YouBot`, `Andibot`, `PanguBot`, `ImagesiftBot`, `Timpi`, `Omgili` | Not verified against an operator source |

These bots may well be real and active in your logs. The claim here is narrower: we could not read a user-agent string from the operator, so we will not print one. A list that hands you exact UA strings for all of them without saying where they came from is worth a second look.

## Copy-paste robots.txt recipes

**Maximum AI visibility, no training.** The common case for a publisher who wants citations but not corpus inclusion.

```
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: meta-webindexer
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: MistralAI-Index
Allow: /

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: MistralAI-Training
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
```

**Everything open.** Nothing to write. An absent robots.txt and an empty one both mean "crawl freely", but a robots.txt that returns a 500 or a soft-404 HTML page does not, and some crawlers treat that as "disallow everything". Check that yours returns a real 200 with `text/plain`.

**Closed to AI, open to search.** Add `Disallow: /` groups for every training and retrieval bot above, and leave `Googlebot` and `bingbot` alone. Expect to lose AI citations entirely. That is the trade, and it should be a decision rather than an accident.

## Check what your site actually does

Writing the rules is the easy half. Confirming they took effect, that your robots.txt is reachable and well-formed, and that AI crawlers can read a rendered page once they get in, is where sites quietly fail.

**Start with a scan.** [`npx aiscan-cli yoursite.com`](https://aiscan.site), or paste your URL at [aiscan.site](https://aiscan.site). Free, no account. It covers the three bot-access checks directly relevant here:

- **B1, Content Signals:** whether your robots.txt declares AI usage preferences at all
- **B2, explicit AI bot rules:** whether the named AI crawlers above have their own groups, or are silently inheriting your `*` group
- **B3, robots.txt reachability and syntax:** whether the file returns a clean 200 as `text/plain`

Full detail on each is on [the bot-access checks page](https://aiscan.site/docs/checks/bot-access).

If you would rather check by hand, this gets you most of the way:

```bash
curl -sI https://yoursite.com/robots.txt | head -3        # expect 200 + text/plain
curl -s https://yoursite.com/robots.txt | grep -iE 'gptbot|oai-searchbot|claudebot|perplexity'
```

**What a scan cannot tell you.** AIScan reads what your site publishes. It cannot read your server logs, so it cannot tell you whether a request last Tuesday claiming to be `GPTBot` was genuine. That part is yours: pull the IP from your access log and run the CIDR command above. And no scanner can tell you whether a crawler that fetched your page actually used the content. Nobody can measure that from the outside.

**On WordPress**, the awkward bit is usually that three plugins each want to own robots.txt and the winner is whichever loaded last. [**ThinkRank**](https://thinkrank.ai) handles robots.txt, robots meta, schema and llms.txt from one plugin, which removes the conflict rather than adding a fourth contender, and it imports existing settings from Rank Math, Yoast, AIOSEO and SEOPress so switching costs nothing in re-entered configuration. Rank Math and Yoast both edit robots.txt perfectly well and have far deeper traditional-SEO feature sets; if you are already happy with one and only need to add a few crawler groups, there is no reason to move. AIOSEO ships an llms.txt generator too. The case for ThinkRank is specifically the one-file-one-owner problem, not that the others are bad at their jobs.

## Common mistakes

| Mistake | What actually happens |
|---|---|
| Blocking the retrieval bot when you meant the training bot | `GPTBot` is training, `OAI-SearchBot` is citations. Disallowing the second removes you from ChatGPT search answers |
| Trusting reverse DNS for OpenAI, Anthropic or Perplexity | There is no PTR record to trust, and where one exists it belongs to AWS or Google Cloud |
| Treating a downloaded IP list as permanent | Google's regenerates daily; a real crawler on a new address fails your check |
| Naming a bot in its own group, then forgetting your global rules | Only one group applies per crawler, so your `*` rules are ignored for that bot |
| Blocking by IP as your opt-out | Anthropic's documentation says this can stop them reading your robots.txt at all, which defeats the purpose |
| Copying a list that cites `anthropic.com/claudebot.json` | It has been a 404 for a while; the live file is on `claude.com` |

## What to do next

Run [`npx aiscan-cli yoursite.com`](https://aiscan.site) and read the B1, B2 and B3 results. Those three tell you whether the rules on this page are actually live on your site or only in your notes. If B2 comes back partial, you have a `*` group doing work you think a named group is doing.

Then pick one address out of your access log that claims to be an AI crawler and run the CIDR check on it. Most people find out something they did not expect on the first try.

More on how the checks work: [Cloudflare's Search, Agent and Training categories](https://aiscan.site/blog/cloudflare-search-agent-training-crawler-categories) covers the policy layer sitting above these user-agents, and [why Cloudflare's AI bot blocking can stop Googlebot](https://aiscan.site/blog/cloudflare-ai-bot-block-googlebot-2026) covers the mixed-purpose crawler problem the 15 September 2026 default change makes real. The rest of the walkthroughs are on [aiscan.site/guides](https://aiscan.site/guides).