---
title: "Find AI Crawlers in Your Server Logs and Verify Them"
slug: find-ai-crawlers-in-server-logs
published: 2026-11-01T08:11:42.436857+00:00
updated: 2026-11-08T08:16:05.701374+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:B2, check:D1, platform:nginx, platform:apache, platform:cloudflare, AI crawlers, server logs, GPTBot, AI readiness
description: "Find GPTBot, ClaudeBot and the rest in your nginx, Apache or Cloudflare logs, count what they got, and check each address against its published range."
url: https://aiscan.site/blog/find-ai-crawlers-in-server-logs
---

Your analytics tool will not tell you which AI crawlers visit your site, because a crawler that fetches the HTML and leaves does not run its script. Only the server or the edge in front of it sees every request, and its log holds a user agent, an address, a path and a status for each one. This guide shows how to pull the AI crawlers out of that log, count what they got, and check each address against the range file its operator publishes.

The last step is the one that matters. A user-agent string is text the client chose, so a count by name tells you who claimed to visit. On our own site, 1,297 of 2,107 requests under an AI crawler's name came from 22 sources that each used 10 to 15 different names, and none of them received a page. Checking the address is what separates those from the real thing.

## Quick summary

| Where your logs live | Path to the numbers | Tested here | Time |
|---|---|---|---|
| nginx or Apache access log | One `awk` line, then the range-check script below | Yes, on a 74-line sample log | 10 min |
| Cloudflare | AI Crawl Control, Metrics tab, or its GraphQL Analytics API | No, read from Cloudflare's docs | 5 min |
| A host that keeps request logs | Export them in a combined-style format and use the script | Format only | 15 min |
| An app you run yourself | A short middleware that writes one JSON line per AI hit | Yes, Node 24, 4 requests | 20 min |

**Recommended order:** run `npx aiscan-cli yoursite.com` for what your robots.txt allows (checks B2 and D1), then use your log to see what actually arrived.

## What a log can tell you that a scan cannot

AIScan reads the rules you publish. Check [B2](/docs/checks/bot-access) looks for explicit `User-agent:` blocks naming the known AI crawlers, and [D1](/docs/checks/discoverability) checks that `robots.txt` returns 200, does not block everything and names your sitemap. Both describe your policy. Neither can see who knocked, how often, or what they received, because that happens in your logs, not on the page a scan fetches.

Reading the two together answers questions a scan alone cannot. Does a crawler you disallowed still fetch pages? Is the bot you wanted in ChatGPT's search answers visiting at all? Are the requests under a famous name getting pages or a wall of 404s? Our [user-agent list](/blog/ai-crawler-user-agent-list-2026) says what each token is for. This guide is the procedure for finding them.

## Step 1: find the log and its format

Both common web servers can write a usable log.

- **nginx** logs to `logs/access.log` in the predefined `combined` format when you set nothing else, per the [nginx log module documentation](https://nginx.org/en/docs/http/ngx_http_log_module.html). Packaged installs often move the file, so check the `access_log` directive in your config.
- **Apache** writes whatever `CustomLog` points at. The [Apache logging guide](https://httpd.apache.org/docs/2.4/logs.html) defines the Combined Log Format, which adds the referrer and the user agent to the common one.

Both formats put the client address first and the user agent last, in quotes. That is all the commands below need. If your log lacks the user agent, you cannot do this from it. Change the format first.

**Behind a CDN or load balancer, the first field is the proxy's address, not the visitor's.** Every request then looks like it came from a handful of sources and the range check fails for everyone. Use the header your CDN documents for the client address; Cloudflare lists `CF-Connecting-IP` among its [request headers](https://developers.cloudflare.com/fundamentals/reference/http-headers/). Log that header, or have your server restore it, before you trust any address.

## Path 1: your own server log

### Count the AI crawlers by name

This `awk` line counts hits and 404s for each token in a list. It splits on double quotes, so field 6 is the user agent and field 3 starts with the status.

```bash
awk -F'"' 'BEGIN{n=split("GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User PerplexityBot Perplexity-User Googlebot Bytespider",T," ")}
{u=tolower($6); for(i=1;i<=n;i++) if(index(u,tolower(T[i]))){c[T[i]]++; split($3,s," "); if(s[1]==404) nf[T[i]]++; break}}
END{for(t in c) printf "%-16s %4d %4d\n",t,c[t],nf[t]+0}' access.log | sort -k2 -nr
```

On our sample log it printed one line per token, hits then 404s. Two details matter. The `break` stops one request being counted under two names. And a plain `grep -o` is wrong here, because several user agents repeat their own name (ClaudeBot's string contains `ClaudeBot` and `claudebot@anthropic.com`), so it double counts.

OpenAI also documents a marker. When it fetches `robots.txt`, `OAI-SearchBot` may add `robots.txt;` to its user agent, which [OpenAI's crawler page](https://developers.openai.com/api/docs/bots) says helps when your logs do not include paths. `grep 'robots.txt;' access.log` finds those.

### Check each address against the operator's file

A count by name is a claim. This script turns it into a check. It reads the combined log, finds the token in each user agent, and tests the address against the operator's published range file. It prints counts only and never a visitor address.

```python
#!/usr/bin/env python3
import re, sys, json, ipaddress, argparse, collections, pathlib

# token -> range file, or None when the operator publishes none
TOKENS = {
    "GPTBot": "gptbot.json", "OAI-SearchBot": "searchbot.json", "ChatGPT-User": "chatgpt-user.json",
    "ClaudeBot": "claude.json", "Claude-User": "claude.json", "Claude-SearchBot": "claude.json",
    "PerplexityBot": "perplexity.json", "Perplexity-User": "perplexity.json",
    "Googlebot": "google.json", "Bytespider": None, "CCBot": None, "meta-externalagent": None,
}
ORDER = sorted(TOKENS, key=len, reverse=True)  # longest first, so no token shadows another
LINE = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "(\S+) (\S+)[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')

ap = argparse.ArgumentParser()
ap.add_argument("log"); ap.add_argument("--ranges", default="."); ap.add_argument("--min-names", type=int, default=3)
a = ap.parse_args()

nets = {}
for tok, f in TOKENS.items():
    p = pathlib.Path(a.ranges) / f if f else None
    if p and p.exists():
        d = json.load(open(p))
        nets[tok] = [ipaddress.ip_network(list(x.values())[0]) for x in d["prefixes"]]

rows = collections.defaultdict(collections.Counter)
names_by_ip = collections.defaultdict(set)
for line in open(a.log):
    m = LINE.match(line)
    if not m: continue
    ip, _, path, status, ua = m.groups()
    tok = next((t for t in ORDER if t.lower() in ua.lower()), None)
    if not tok: continue
    r = rows[tok]; r["hits"] += 1
    r["ok"] += status.startswith("2"); r["nf"] += status == "404"
    r["other"] += not (status.startswith("2") or status == "404")
    names_by_ip[ip].add(tok)
    if tok in nets:
        hit = any(ipaddress.ip_address(ip) in n for n in nets[tok])
        r["in"] += hit; r["out"] += not hit
    else:
        r["norange"] += 1

print(f"{'token':<18}{'hits':>5}{'2xx':>5}{'404':>5}{'oth':>5}  {'in range':>8}{'outside':>8}{'no file':>8}")
for tok, r in sorted(rows.items(), key=lambda kv: -kv[1]["hits"]):
    print(f"{tok:<18}{r['hits']:>5}{r['ok']:>5}{r['nf']:>5}{r['other']:>5}  {r['in']:>8}{r['out']:>8}{r['norange']:>8}")
shared = [i for i, s in names_by_ip.items() if len(s) >= a.min_names]
print(f"\nsources presenting {a.min_names}+ different crawler names: {len(shared)} of {len(names_by_ip)}")
```

Download the range files it reads from the operators, and keep them in one folder:

```bash
curl -sO https://openai.com/gptbot.json -O https://openai.com/searchbot.json -O https://openai.com/chatgpt-user.json
curl -so claude.json https://claude.com/crawling/bots.json
curl -so perplexity.json https://www.perplexity.com/perplexitybot.json
curl -so google.json https://developers.google.com/static/crawling/ipranges/common-crawlers.json
python3 aicrawl-log.py /var/log/nginx/access.log --ranges .
```

The files are the operators' own. OpenAI links its range files from [its crawler page](https://developers.openai.com/api/docs/bots), and Anthropic's [support article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) says a crawler with a source address on its list "is coming from Anthropic". Google's `common-crawlers.json` holds both IPv4 and IPv6 prefixes (170 of its 317 on 1 November 2026 were IPv4), which is why the script reads whichever key a prefix has.

## What the test showed

We built a 74-line synthetic combined-format log: 18 browser requests and 56 from nine crawler tokens. Real crawler requests were given an address taken from the operator's file, fetched on 1 November 2026, and impostor requests came from `203.0.113.x`, a range reserved for documentation. It proves the script works. It says nothing about how common impostors are, which is what our real data is for.

| Token | Hits | 2xx | 404 | In operator range | Outside range |
|---|---|---|---|---|---|
| ClaudeBot | 13 | 8 | 5 | 8 | 5 |
| GPTBot | 9 | 6 | 3 | 6 | 3 |
| Googlebot | 8 | 5 | 3 | 5 | 3 |
| OAI-SearchBot | 6 | 4 | 2 | 4 | 2 |
| PerplexityBot | 6 | 3 | 3 | 3 | 3 |
| ChatGPT-User | 5 | 3 | 2 | 3 | 2 |
| Bytespider | 4 | 1 | 3 | no file | no file |
| Claude-User | 3 | 3 | 0 | 3 | 0 |
| Perplexity-User | 2 | 2 | 0 | 2 | 0 |

*Source: synthetic sample log, 74 lines, run 1 November 2026. Range files fetched the same day.*

Three things to read from it. Every request outside its range was a 404 and came from one of two made-up sources. Those two sources presented four and three different crawler names, which the last line counted: 2 of 11 sources with three or more names. And Bytespider shows `no file`: its operator publishes no range file we could use, so the script can say how many requests arrived and nothing about whose they were. Treat `no file` as unknown, not as good or bad. Our [user-agent list](/blog/ai-crawler-user-agent-list-2026) says which operators publish ranges, as of 27 August 2026.

### What real traffic looks like

Our own table, `ai_crawler_hits` on aiscan.site, covered 25 August to 27 October 2026 and held 2,107 requests from 147 sources. Part of it is below, from the [full study](/blog/fake-ai-crawler-traffic-own-logs). We store a hashed address, so we could not run the range check on it. That is a real cost of hashing at write time, and the reason to keep the raw address in your log for as long as you plan to verify.

| Claimed name | Requests | Served (200) | 404 share |
|---|---|---|---|
| ClaudeBot | 421 | 173 | 58.9% |
| Claude-User | 364 | 333 | 0.0% |
| Bytespider | 90 | 0 | 100% |
| GPTBot | 78 | 4 | 94.9% |

*Source: aiscan.site `ai_crawler_hits`, n = 2,107 requests, 25 August to 27 October 2026. Blog pages are under-counted in this table, so read it as identity, not volume.*

A name with a 95% to 100% 404 share and almost nothing served is the pattern to look for in your own output. The cause is usually a scanner that borrowed the name and asked for paths your site does not have. A 404 is the right answer to that. [Blocking the secret paths](/blog/block-fake-ai-crawlers-secret-probes) is a separate job, and a user-agent rule is the wrong tool for it.

## Path 2: Cloudflare, without touching a log file

If your site sits behind Cloudflare, you can see this without a log. [AI Crawl Control](/blog/cloudflare-ai-crawl-control-setup) has Overview, Crawlers and Metrics tabs. Per [Cloudflare's analytics page](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/) (last updated 23 April 2026), Metrics shows requests over time, a status code distribution, top referrers and most popular paths, and you can filter by crawler, operator, hostname or path.

The status code view is the same question as the 404 column above. The most popular paths table shows which pages the crawlers requested. Referral figures are marked paid plans only. For anything the dashboard does not chart, the same page points to the GraphQL Analytics API, which exposes the data the dashboard shows. We did not run the dashboard on a zone of ours for this guide. Check the tab names against your own account.

## Path 3: an app or host with no access log

Some hosts keep short-lived request logs, drop the user agent, or hide the client address. Check what yours keeps before you rely on it. If it does not keep enough, log the crawlers yourself with a small piece of middleware that writes one line per AI hit. This one records the token, the path, the status and a salted hash of the address. We ran it with Node 24 on four requests: two GPTBot requests (a 200 and a 404), one browser request, and one Claude-User request. Three lines were written, and the browser request was not logged.

```javascript
import { createServer } from 'node:http';
import { appendFileSync } from 'node:fs';
import { createHash } from 'node:crypto';

const TOKENS = ['GPTBot','OAI-SearchBot','ChatGPT-User','ClaudeBot','Claude-SearchBot','Claude-User','PerplexityBot','Perplexity-User'];
const SALT = process.env.LOG_SALT ?? 'change-me';

export function logAiHit(req, res) {
  const ua = req.headers['user-agent'] ?? '';
  const token = TOKENS.find(t => ua.toLowerCase().includes(t.toLowerCase()));
  if (!token) return;
  const ip = req.headers['x-forwarded-for']?.split(',')[0].trim() ?? req.socket.remoteAddress;
  res.on('finish', () => appendFileSync('ai-hits.jsonl', JSON.stringify({
    t: new Date().toISOString(), token, path: req.url.split('?')[0], status: res.statusCode,
    src: createHash('sha256').update(SALT + ip).digest('hex').slice(0, 12),
  }) + '\n'));
}
```

Call `logAiHit(req, res)` at the top of your request handler. Trust `x-forwarded-for` only when your own proxy sets it, since a client can send that header too. If you want the range check later, store the address itself (or run the check inside the middleware and store the result), because a salted hash cannot be matched to a range afterwards.

Before you dig through logs, a quick [per-template probe](https://aiscan.site/blog/probe-ai-crawler-access-per-page-template) shows which sections of the site answer an AI crawler's user-agent with an error, so you know where to look.

## Verify it worked

Pass: the script runs, prints a row for each AI token present in your log, and the `2xx`, `404` and `other` columns add up to `hits`. Fail: no rows, which means your log has no user agent field or no AI crawler reached it.

Then spot-check one address by hand. For Google, [Google's verification page](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) describes a reverse DNS lookup that must end in `googlebot.com`, `google.com` or `googleusercontent.com`, followed by a forward lookup that returns the same address. For OpenAI, Anthropic and Perplexity, use the range files. Our August test found reverse DNS unavailable for those three, so the range check is the working method. [OpenAI's, Anthropic's and others' files change](/blog/ai-crawler-ip-list-staleness), so re-download them on a schedule and read the `creationTime` field first.

Last, compare the log to your policy. If `robots.txt` disallows a token and that token still fetches pages from an address inside its range, that is a finding. [Rate limiting](/blog/rate-limit-ai-crawlers-without-blocking) is the gentler control than a block. ChatGPT-User is the exception to expect: OpenAI states that because its requests are user-initiated, robots.txt rules may not apply to them.

## What this cannot tell you

- **Whether a real crawler's visit led anywhere.** A hit is a fetch, not a citation or a click.
- **Anything about a bot with no range file.** Count it, and mark the identity unknown.
- **Traffic your CDN answered from cache.** If the edge served it, your origin log never saw it. Use the CDN's own analytics for those.
- **JavaScript analytics.** A tag that runs in the browser records visitors whose browser runs it. A crawler that fetches HTML and leaves does not appear there.

Never publish a visitor address. Counts per token are enough for a report.

## Next step

Scan your site with `npx aiscan-cli yoursite.com` and read B2 and D1, then run the script on a week of logs and compare the two. The [AIScan guides](/guides) cover the fixes for whatever the scan flags, and the [robots.txt starter](/blog/robots-txt-starter-for-ai-crawlers) is the place to begin if you have no AI rules yet.

If robots.txt allows the crawler but a WordPress firewall or security plugin still returns `403`, see [a WordPress security plugin is blocking GPTBot, ClaudeBot and PerplexityBot](https://aiscan.site/blog/wordpress-security-plugin-blocks-ai-crawlers).