Every AI Crawler User-Agent in 2026: The Verified List and How to Spot a Fake

Verified 27 August 2026. Every user-agent below was read from the operator's own documentation on that date. Every IP range file was fetched and counted. Every reverse-DNS result was tested live.

Roughly 1 in every 18 requests carrying an AI crawler user-agent is fake. That is HUMAN Security's Satori Threat Intelligence team, reporting a 1:17 average spoof ratio across a two-week study published 9 September 2025. ChatGPT-User was the worst at 1:5.

Here is the awkward part. The verification method almost every guide recommends, forward-confirmed reverse DNS, does not work for OpenAI, Anthropic or Perplexity. Not "works badly". Does not work. I ran it against their own published IP ranges this morning and got back no PTR record for OpenAI, no PTR record for Anthropic's primary range, and for Perplexity a hostname reading ec2-107-20-236-150.compute-1.amazonaws.com, which anyone renting an EC2 instance can obtain.

This is the current list of AI crawler user-agents, what each one is actually for, and the verification method that works for each.

Quick summary

The single most useful thing on this page: the verification-method column in the table further down. Reverse DNS works for Google, Apple, Bing and Common Crawl. It is unavailable for OpenAI, Anthropic and Perplexity, and it produces a false pass for Perplexity. If your bot-verification logic is built on FCrDNS alone, it is verifying four crawlers and quietly waving through the rest.

Most published crawler lists are a year out of date

Three things changed in 2026 that older lists have not caught up with.

Operators split into three bots, not two. OpenAI, Anthropic, Amazon, Mistral and Meta now each run a separate crawler for training, for search indexing, and for live user-triggered fetches. Amazon's own doc is explicit that "each user agent setting is independent of the others."

New user-agents nobody covers. OAI-AdsBot, Amzn-SearchBot, Amzn-User, MistralAI-Index, MistralAI-Training, Meta-WebIndexer, Google-Agent and Google-GeminiNotebook are all documented and missing from most published lists.

Some tokens are retired. anthropic-ai and Claude-Web no longer appear in Anthropic's crawler documentation, which lists exactly three bots. FacebookBot is gone from Meta's crawler page. Leaving them in your robots.txt is harmless, but do not present them as current, and do not build a monitoring rule around them.

One more trap worth knowing: Google moved its crawler documentation from /search/docs/crawling-indexing/* to /crawling/docs/* this year. Any article still linking the old path is already pointing at a stub.

Every documented AI crawler user-agent, August 2026

Robots column: Y = obeys robots.txt · N = documented to ignore or bypass it · Partial = may not apply because a human triggered the fetch.

Sources, all fetched 27 August 2026: OpenAI, Anthropic, Google, Perplexity, Meta, Apple, Amazon, Mistral, Common Crawl.

The exact strings

Match on the token, not the version number. OpenAI's doc says outright that the version may change.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
CCBot/2.0 (https://commoncrawl.org/faq/)
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)

W.X.Y.Z is literal in Amazon's documentation, not a placeholder they forgot to fill in.

Three jobs, three opposite consequences

Blocking a training bot and blocking a retrieval bot are not the same decision, and getting them backwards is the most expensive mistake on this page. The operators say so themselves.

OpenAI, verbatim:

"Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models."

And on the cost of getting it wrong:

"Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

Anthropic on Claude-User: disabling it "prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search." Google on Google-Extended: it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Meta on Meta-WebIndexer: allowing it "helps us cite and link to your content in Meta AI's responses." Apple: disallowing Applebot-Extended blocks training, but "your content will remain discoverable through Spotlight, Siri, and Safari."

That is five operators saying the same thing in five different words. Training and retrieval are separate switches. A blanket Disallow: / for everything with "AI" in the name removes you from AI answers while doing nothing about the training corpora that already contain your pages.

One more OpenAI detail that undermines naive blocklists: "If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling."

Propagation is not instant either. OpenAI says about 24 hours from a robots.txt change. Perplexity says up to 24 hours. Amazon says roughly 24 hours and caches robots.txt for up to 30 days. DuckDuckGo says 72 hours.

Verification, method one: the published IP ranges

Nine operators publish machine-readable IP ranges. All of these returned 200 on 27 August 2026, with the prefix count and file timestamp taken from the file itself:

Two things stand out. Google regenerates daily. OpenAI's gptbot.json has not been touched since 30 October 2025, close to ten months, and Perplexity's perplexitybot.json since 7 February 2025, over eighteen. If you build an allowlist from either, you are trusting a very old snapshot, and a genuine crawler operating from a newer address will fail your check.

Two files that people cite do not exist. https://www.anthropic.com/claudebot.json returns 404; the live location is on claude.com. https://mistral.ai/mistralai-training-ips.json also 404s, so Mistral's training bot is the one crawler here you cannot verify by IP at all. Any crawler list still pointing at the anthropic.com URL is a quick test of whether that list is maintained.

Anthropic adds a warning worth repeating, because it is the opposite of what a lot of people do:

"Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."

Block the bot in robots.txt. Use the IP list to confirm identity, not to enforce policy.

Verification, method two: reverse DNS, and where it quietly fails

Forward-confirmed reverse DNS is the standard advice: take the source IP, look up its PTR record, resolve that hostname forward, and confirm it comes back to the same IP. It is a good method. It is also unavailable for the three brands most likely to be impersonated.

I ran it on 27 August 2026 against the first usable address in each operator's own published range:

The bottom two rows are the reason this section exists. A naive FCrDNS check on that Perplexity address passes: the PTR resolves, and it resolves forward to the same IP. What it proves is that the host rents an EC2 instance. Anyone can rent an EC2 instance. The same is true of the Anthropic address sitting in a Google Cloud range whose PTR ends in bc.googleusercontent.com.

So the rule is: FCrDNS is only meaningful when you also assert the hostname suffix, and only Google, Apple, Bing and Common Crawl publish a suffix worth asserting. Google's masks, from its own verification page:

crawl-***-***-***-***.googlebot.com
geo-crawl-***-***-***-***.geo.googlebot.com
rate-limited-proxy-***-***-***-***.google.com
google-proxy-***-***-***-***.google.com
***-***-***-***.gae.googleusercontent.com

For OpenAI, Anthropic and Perplexity, the IP list is the only method. There is no fallback.

The two commands

Reverse DNS with a suffix assertion, which is the only form worth running:

python3 -c "
import socket,sys
ip,suf=sys.argv[1],sys.argv[2]
try: h=socket.gethostbyaddr(ip)[0]
except Exception: print(ip,'NO PTR -> use the IP list instead'); raise SystemExit
ok = h.endswith(suf) and ip in socket.gethostbyname_ex(h)[2]
print(ip, h, 'VERIFIED' if ok else 'REJECT')" 66.249.66.1 googlebot.com

Output on the test above: 66.249.66.1 crawl-66-249-66-1.googlebot.com VERIFIED.

IP against a published CIDR list, which is what you need for the OpenAI/Anthropic/Perplexity group:

curl -sL --compressed -o gptbot.json https://openai.com/gptbot.json
python3 -c "
import json,ipaddress,sys
ip=ipaddress.ip_address(sys.argv[1])
nets=[ipaddress.ip_network(p.get('ipv4Prefix') or p.get('ipv6Prefix'))
      for p in json.load(open('gptbot.json'))['prefixes']]
print(sys.argv[1],'VERIFIED' if any(ip in n for n in nets) else 'SPOOFED')" 8.8.8.8

Refetch the JSON on a schedule. Google's changes daily; treating any of these as a static file is how you start rejecting real crawlers.

Web Bot Auth is where this is going

Both problems above, no PTR records and stale IP files, go away if the crawler signs its requests cryptographically. That is what Web Bot Auth does, and two operators have shipped it.

Cloudflare, in the Perplexity investigation: "ChatGPT Agent is signing http requests using the newly proposed open standard Web Bot Auth." Google's Google-Agent documentation adds: "Google is also experimenting with the Web Bot Auth protocol, using the https://agent.bot.goog identity."

HUMAN Security makes the practical difference explicit: signed requests "cannot be spoofed in the same way as crawler traffic," because they are validated against the operator's keys rather than against a list of addresses. Two operators is not an ecosystem yet, but it is the direction, and worth watching before you invest heavily in IP-list tooling.

How much of this traffic is actually fake

Two independent pieces of evidence, and both are worth reading in full.

Cloudflare vs Perplexity, 4 August 2025. Cloudflare created brand-new domains that had never been indexed or published anywhere, put a Disallow: / in robots.txt, and watched. From their post:

"We observed that Perplexity uses not only their declared user-agent, but also a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked."

Declared traffic ran 20–25 million requests a day; the undeclared browser-lookalike ran 3–6 million. Cloudflare adds that it "utilized multiple IPs not listed in Perplexity's official IP range" and rotated through ASNs, "across tens of thousands of domains and millions of requests per day."

The control condition in the same experiment is the part most coverage skipped: "we found that ChatGPT-User fetched the robots file and stopped crawling when it was disallowed. We did not observe follow-up crawls from any other user agents or third party bots."

HUMAN Security, 9 September 2025. A two-week study across HUMAN's customer base, spanning multiple industries and geographies, measuring how much traffic claims an AI user-agent but fails IP verification:

By brand: OpenAI 1:9, Mistral 1:37, Perplexity 1:138, DuckDuckGo 1:772. Overall, "the average spoof ratio is 1:17" and "spoofed requests make up 5.7% of all traffic labeled as coming from AI crawlers."

Note the direction of that finding. The most-impersonated user-agent belongs to the best-behaved operator, because impersonating a bot site owners deliberately allow is more useful than impersonating one they block. Both figures are September 2025; no vendor has published a verifiable 2026 equivalent, so treat them as a floor.

robots.txt rules that trip people up

From Google's robots.txt specification, which the other operators broadly follow:

• The user-agent value is case-insensitive. User-agent: GPTBot and user-agent: gptbot are the same rule.
• Path values are case-sensitive. So is the robots.txt URL itself.
• Only one group applies. Google picks "the most specific group that matches the crawler's user agent. Other groups are ignored." A bot named in its own group never reads your User-agent: * group. Writing Disallow: /private/ under * and forgetting to repeat it under GPTBot leaves that path open to GPTBot.
• You match the token, not the full string. Never paste a whole Mozilla string into robots.txt.
• Longest matching path wins, and where rules conflict, Google applies the least restrictive one.

Two operator quirks: Anthropic supports the non-standard Crawl-delay directive, with User-agent: ClaudeBot / Crawl-delay: 1 as their own example. And Amzn-SearchBot, per Amazon's doc, "will crawl in accordance with the robots.txt directives given to other search bots" if you never mention it, so a rule aimed at Googlebot can silently govern Amazon.

The bots we could not verify

Every entry in the main table came from an operator's own documentation. These did not, and are listed separately rather than quietly folded in:

These bots may well be real and active in your logs. The claim here is narrower: we could not read a user-agent string from the operator, so we will not print one. A list that hands you exact UA strings for all of them without saying where they came from is worth a second look.

Copy-paste robots.txt recipes

Maximum AI visibility, no training. The common case for a publisher who wants citations but not corpus inclusion.

User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: meta-webindexer
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: MistralAI-Index
Allow: /

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: MistralAI-Training
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /

Everything open. Nothing to write. An absent robots.txt and an empty one both mean "crawl freely", but a robots.txt that returns a 500 or a soft-404 HTML page does not, and some crawlers treat that as "disallow everything". Check that yours returns a real 200 with text/plain.

Closed to AI, open to search. Add Disallow: / groups for every training and retrieval bot above, and leave Googlebot and bingbot alone. Expect to lose AI citations entirely. That is the trade, and it should be a decision rather than an accident.

Check what your site actually does

Writing the rules is the easy half. Confirming they took effect, that your robots.txt is reachable and well-formed, and that AI crawlers can read a rendered page once they get in, is where sites quietly fail.

Start with a scan. npx aiscan-cli yoursite.com, or paste your URL at aiscan.site. Free, no account. It covers the three bot-access checks directly relevant here:

• B1, Content Signals: whether your robots.txt declares AI usage preferences at all
• B2, explicit AI bot rules: whether the named AI crawlers above have their own groups, or are silently inheriting your * group
• B3, robots.txt reachability and syntax: whether the file returns a clean 200 as text/plain

Full detail on each is on the bot-access checks page.

If you would rather check by hand, this gets you most of the way:

curl -sI https://yoursite.com/robots.txt | head -3        # expect 200 + text/plain
curl -s https://yoursite.com/robots.txt | grep -iE 'gptbot|oai-searchbot|claudebot|perplexity'

What a scan cannot tell you. AIScan reads what your site publishes. It cannot read your server logs, so it cannot tell you whether a request last Tuesday claiming to be GPTBot was genuine. That part is yours: pull the IP from your access log and run the CIDR command above. And no scanner can tell you whether a crawler that fetched your page actually used the content. Nobody can measure that from the outside.

On WordPress, the awkward bit is usually that three plugins each want to own robots.txt and the winner is whichever loaded last. ThinkRank handles robots.txt, robots meta, schema and llms.txt from one plugin, which removes the conflict rather than adding a fourth contender, and it imports existing settings from Rank Math, Yoast, AIOSEO and SEOPress so switching costs nothing in re-entered configuration. Rank Math and Yoast both edit robots.txt perfectly well and have far deeper traditional-SEO feature sets; if you are already happy with one and only need to add a few crawler groups, there is no reason to move. AIOSEO ships an llms.txt generator too. The case for ThinkRank is specifically the one-file-one-owner problem, not that the others are bad at their jobs.

Common mistakes

What to do next

Run npx aiscan-cli yoursite.com and read the B1, B2 and B3 results. Those three tell you whether the rules on this page are actually live on your site or only in your notes. If B2 comes back partial, you have a * group doing work you think a named group is doing.

Then pick one address out of your access log that claims to be an AI crawler and run the CIDR check on it. Most people find out something they did not expect on the first try.

More on how the checks work: Cloudflare's Search, Agent and Training categories covers the policy layer sitting above these user-agents, and why Cloudflare's AI bot blocking can stop Googlebot covers the mixed-purpose crawler problem the 15 September 2026 default change makes real. The rest of the walkthroughs are on aiscan.site/guides.