Fake AI Crawler Traffic: 63 Days of Our Own Server Logs

Of the 2,107 requests our own logger recorded under an AI crawler's name between 25 August and 27 October 2026, 61.6% came from 22 sources that each used 10 to 15 different crawler names, and not one of those 1,297 requests received a page. The traffic that did get pages came from 111 sources that each used a single name and kept it. If your logs show GPTBot, Bytespider or Google-Extended at 3 a.m., the name on the request is the least reliable thing about it.

AIScan keeps a table of every request that arrives with a known AI crawler or AI assistant user agent. We queried it on 27 October 2026 and split the traffic by who was really behind the name. This post is the result: a per-bot table, a test you can run on your own logs in one query, and what a defender should and should not do with the answer. It is a study of one small site, and the limits come before the table.

Quick summary

For site owners: a request is only as trustworthy as its source address, and your log can tell you which names share one. For defenders: a 404 is already the right answer for a path your site does not have, so most of this needs no blocking rule.

How we measured it, and what it cannot tell you

The limits decide how much of this you should repeat.

The source. The ai_crawler_hits table on aiscan.site holds one row per logged request with the claimed bot name, the user agent, the path, the status code, a hashed source address and a timestamp. On 27 October 2026 it held 2,107 rows from 25 August to 27 October 2026 (63 days), 18 distinct claimed names and 147 distinct source hashes. The hash is one-way. We cannot look up who owns an address, run a reverse DNS check or compare to a published address range, and we never print a hash here.

What "fake" means in this post. We do not claim to have proven that a request was not from OpenAI, Anthropic or anyone else. We say a source is shared-identity when the same hashed address presented 10 or more different crawler names. A real crawler operator sends one product's user agent from its own address space. A source that presents GPTBot, Googlebot, Bytespider and ten other names within the same hour is behaving like a scanner that rotates through a list of user agents, which is a different thing from a crawler.

What the logger misses. It records some routes and not others. Blog pages are served from a prerender path and are under-counted, so this is not a crawl-volume study and you should not read 718 or 690 served requests as "how often AI crawlers read our posts". It is a study of identity: who says they are what, and what they got back.

What a 404 means here. Our site returns 404 for a path it does not have. A 404 in this table is the site doing its job, not an error, and a 200 is a page or endpoint that exists. 32 requests returned 401, all of them to our MCP endpoint, which asks for authentication on some calls. We count those as neither.

Path classes. We sorted paths into classes by pattern: secret-bearing files, configuration files, CMS admin paths and API spec paths. A path can match more than one pattern and was counted once, in that order. We describe the classes and do not list the paths, because a list of probed paths reads as a recipe.

Per bot: what each name asked for and got

Here are all 18 names, ordered by volume. "Served" is the count of 200 responses. "From shared-identity sources" is the share of that name's requests that came from the 22 sources presenting 10 or more names.

Source: aiscan.site ai_crawler_hits, 25 August to 27 October 2026, n = 2,107 requests from 147 sources. Table totals: 690 served, 1,385 not found, 32 unauthorised.

Three things stand out.

First, there are two kinds of name. Claude-User, ClaudeBot and meta-externalagent get real pages, and most of their requests come from sources that use one name. Everything else in the table reads like a bad template: more than 85% 404s, and 88% to 97% of requests from the shared-identity sources.

Second, six names never got a real page: Bytespider, YouBot, Google-Extended, DuckAssistBot, cohere-ai and MistralAI-User. That is 449 requests and zero served. For one of them the reason is documented. According to Google's crawler documentation, Google-Extended "doesn't have a separate HTTP request user agent string": crawling is done with Google's existing user agents, and Google-Extended is only a token you write in robots.txt. A request that arrives with Google-Extended as its user agent is therefore not Google. We logged 75 of them.

Third, even the busiest legitimate name is mixed. 230 of ClaudeBot's 421 requests (54.6%) came from shared-identity sources. The other 176 came from four sources that only ever called themselves ClaudeBot, and 173 of those got a 200. A rule that blocks "ClaudeBot" by name would block real traffic and fake traffic alike, and a rule that trusts the name would trust both.

The test that separates them

The query is short. Count how many distinct claimed names each hashed source address presents, then group:

with i as (
  select ip_hash, count(distinct bot_name) names
  from ai_crawler_hits group by 1)
select case when names = 1 then '1 name'
            when names <= 3 then '2-3 names'
            when names <= 9 then '4-9 names'
            else '10+ names' end as sources_presenting,
       count(*) as sources,
       sum(h.hits) as requests
from i join (select ip_hash, count(*) hits from ai_crawler_hits group by 1) h using (ip_hash)
group by 1 order by 1;

Run on our table it gave this:

Same table and dates, n = 147 sources. The 1-name row also holds the 32 requests that returned 401.

The 10-name row is the finding. 22 sources, 15.0% of all sources, produced 61.6% of all requests, and none of those requests succeeded. Each of the 22 presented between 10 and 15 names. The median source sent 40.5 requests and the median span between its first and last request was 57 minutes. The longest-lived of them was active on 5 separate days.

The 1-name row is the contrast. 111 sources, 685 requests, and 648 of them served. That is the shape of a crawler or an assistant doing a job: one identity, the pages it wanted, and a 200.

The 4-to-9-name row is honest uncertainty. Seven sources, 108 requests, 42 served. Some of those 42 are OAI-SearchBot, PerplexityBot and Bingbot requests that did get pages. A shared egress network that several products use is a plausible explanation, and we cannot test it without the address. We left this row out of the headline figure, which makes 61.6% a floor rather than an estimate.

One bias to state: a source that sent a single request can only have one name. That is why we read the 10-plus row, where the effect cannot come from low volume, and treat the 1-name row as a comparison and not as proof that every member is genuine.

What they asked for

The shared-identity sources did not ask for pages. Of their 1,297 requests, this is what the paths looked like by class:

Same table and dates, n = 1,297 requests. Classes were applied in the order secrets, API spec, CMS admin, configuration, so no request counts twice.

Across the whole table, 856 of the 1,385 not-found responses (61.8%) went to one of the first four classes, spread over 180 distinct missing paths. A reader checking their own logs will recognise the pattern: someone is looking for a file a careless deploy left behind.

This matches what the security industry has reported. A GreyNoise write-up, Threat Actors Are Posing as OpenAI, Anthropic and DeepSeek to Target Credentials and Secrets, describes scanners using AI crawler names as cover. We cannot confirm that our 22 sources are the same actors, and the title is the claim we are citing, not a finding of ours. What our logs show is the behaviour: an AI crawler name attached to a request for something no crawler operator documents asking for.

The reason this works for the scanner is simple. Teams write allow rules for AI bots, because blocking them costs visibility, and some firewall rules exempt known AI user agents from rate limits and challenges. A scanner that looks like one of them can slip through a rule written to be generous.

The traffic that was real

Here is the other side of the table: the 690 requests that got a 200, grouped by what they asked for.

Same table and dates, n = 690 served requests.

Of 364 Claude-User requests, 333 were served and 31 returned 401. Claude-User is the identity Anthropic describes for fetches triggered by a person using Claude, and here those fetches were people connecting Claude to AIScan's MCP endpoint. That is user-driven fetching, not crawling, which the stealth-crawling debate explains. In a log that counts every AI-named request as "AI crawler traffic", the biggest real contributor is a person with an assistant, not a crawler at all.

ClaudeBot's 173 served requests came from four sources that only ever called themselves ClaudeBot, and 139 of them were scan result pages. meta-externalagent's 140 served requests came from 59 sources that each used only that name, and 139 of them were calls to our scan API. Those are the two names where a crawler operator's traffic is visible in our data. No OpenAI, Google, ByteDance, Perplexity or Common Crawl name appears in the 1-name group with meaningful served volume, other than a single Googlebot and a single Bingbot request. Our site is small and new, and the real crawlers of those operators have not fetched much of it yet, which is a fact about our site and not about those operators.

When it happened

The shared-identity traffic did not run steadily for 63 days. It started in one week and has faded since.

Same table, weeks by timestamp in UTC, n = 2,107.

For the first 246 requests, in the three weeks to 13 September, no source presented 10 or more names and nearly every request was served. In the week of 14 September, 443 of 477 requests came from shared-identity sources. We do not know what started it. A new site in AI-visibility tools and public scanner lists is one possibility, and a single probe round that rotated names is another. The table cannot separate them, so we do not choose.

The point for a site owner is that a log with three weeks of history looks like a clean crawl and a log with the week of 14 September in it looks like a siege. Both are true of the same site.

What to do with this

A few conclusions follow from the table, and none of them is "block every AI user agent".

A 404 is the correct answer, so check that you give one. If your site does have files at those paths, this guide shows the nginx, Apache and Cloudflare rules that refuse them without breaking /.well-known/. Everything the shared-identity sources asked for got a 404 from us, and that is why this study found nothing worse. A site that answers 200 to every unknown path (a soft 404) hands a scanner a page, and AIScan's E1 check tests exactly that. Our correct-404 guide walks through the fix.

Do not trust a user agent. Verify the source. OpenAI's crawler documentation lists the user agent strings for OAI-SearchBot, GPTBot and ChatGPT-User, and says each has a published list of IP addresses. That list, not the string, is what separates OpenAI from someone who copied the string. Our user-agent list covers the IP-range method and the reverse-DNS method step by step, and the IP list staleness post covers why you should refetch the lists. Web Bot Auth is the signed-request approach that makes the check unnecessary for operators who adopt it.

Do not write allow rules by name alone. If a firewall rule says "allow user agent contains GPTBot", the 78 requests in our table that said GPTBot are all covered, including the 70 from shared-identity sources. Pair the name with the operator's published address range, or with a verified signature, or do not exempt it at all. Cloudflare's crawl control setup shows how to do that with verified-bot signals instead of strings.

robots.txt is not a lock. RFC 9309, the Robots Exclusion Protocol, says in so many words that its rules "are not a form of access authorization". A compliant crawler reads it and follows it. A scanner that imitates a crawler has no reason to. Disallow lines are how you tell honest crawlers what you want, and our robots.txt starter shows rules for the named AI agents. They do nothing to the 22 sources above.

Rate limit by source, and do not rate limit by name. A source presenting 12 identities in an hour is one source. Rate limiting AI crawlers without blocking them covers rules keyed on address and behaviour so that real crawlers keep working.

WordPress sites have a cheap first step. Most of the paths the shared-identity sources asked for are the ones a WordPress site is most likely to have left behind (backup files, configuration copies, debug logs). For the AI-bot rules in robots.txt, ThinkRank manages named AI agents next to robots meta, schema and llms.txt in one plugin, so that three plugins are not each editing the same file. For hidden-bot detection on WordPress, BetterShield is a WPDeveloper product with a bot check built for exactly this kind of traffic.

Check your own site first

Start with the scan, because it tells you what your site says about itself to a crawler before any log does. Run npx aiscan-cli yoursite.com, or paste your address at aiscan.site, free and without an account. The relevant checks are B2 (explicit rules for named AI agents in robots.txt), D1 (robots.txt itself) and E1 (a correct 404 for a missing page). What AIScan cannot see is your server log, so it cannot tell you which of your visitors share an identity. For that you need the query above.

If you would rather check by hand, this counts distinct claimed names per source in an nginx combined log. It assumes the user agent is the last quoted field and uses the address as the source. Replace the pattern with the names you care about:

awk -F'"' '{split($1,a," "); ua=$6; ip=a[1];
  if (ua ~ /GPTBot|ClaudeBot|Bytespider|CCBot|PerplexityBot|Amazonbot/) { match(ua,/(GPTBot|ClaudeBot|Bytespider|CCBot|PerplexityBot|Amazonbot)/); n=substr(ua,RSTART,RLENGTH); seen[ip" "n]=1 } }
  END { for (k in seen) { split(k,p," "); c[p[1]]++ } for (i in c) if (c[i] >= 5) print c[i], i }' access.log | sort -rn | head

Any address that prints a count of 5 or more is presenting five or more different crawler identities. Compare it with the operator's published ranges before you do anything else.

Next, read the bot access check guide for what each B-check expects, and the full list at /guides.