How Stale Are AI Crawler IP Lists? We Checked All 15 Files

Every guide on verifying an AI crawler tells you to check its request against the operator's published IP range file. Almost none of them tell you when that file was last regenerated. The answer is sitting in the file itself, one field, and it took one curl per operator to read it across all fifteen files still being published in September 2026.

Quick summary

The honest picture is not "these files are stale." It is that staleness is unevenly distributed and invisible from outside. Google regenerates four separate range files daily. OpenAI's chatgpt-user.json grew from 207 to 230 entries between 4 and 25 September 2026. And the file that would have been this article's headline example three weeks ago, OpenAI's gptbot.json, sat frozen for roughly eleven months and then regenerated on 22 September 2026, four days before this was fetched. A snapshot from three weeks ago is already wrong about which files are the stale ones.

Every AI crawler IP file publishes a creationTime (or equivalent) field alongside the prefix list. It is the regeneration date of the file, not a promise about the ranges inside it, and reading it costs one request.

One field, fifteen files, verified today

The method is deliberately boring. Fetch each operator's published range file with curl -L (several of these URLs redirect, and a plain curl without -L silently reads a small stub instead of the file), parse the JSON, and print two numbers: how many prefixes it holds and the creationTime timestamp inside it. No login, no API key, about fifteen requests total.

Fetched from each operator's own domain and verified on 26 September 2026:

*Common Crawl's endpoint returned a connection failure (curl exit 35, HTTP 000) on three attempts today rather than a fresh read. That is a known intermittent fault on this specific host, not evidence of a stale file, so the last confirmed figures are carried forward and flagged rather than treated as current. www.anthropic.com/claudebot.json and mistral.ai/mistralai-training-ips.json were checked and remain hard 404s: neither exists to go stale.

Why one operator publishes four files, not one

Google alone accounts for four of the fifteen files, and the split is not arbitrary padding, it maps to a distinction Google's own crawler-verification documentation draws explicitly. According to that page, fetched on 26 September 2026, Google's crawlers and fetchers fall into three named categories, each with its own file: "common crawlers" (Googlebot and the other automated crawlers that "always respect robots.txt rules"), "special-case crawlers" (product-specific fetchers such as AdsBot that "may or may not respect robots.txt rules"), and "user-triggered fetchers" (tools like Site Verifier that act because a person asked them to, not on a schedule). The fourth file, user-triggered-agents.json, covers the newer Google-Agent identity for agentic browsing and is not part of that original three-way split, which is consistent with it being the file that grew fastest, from 4 prefixes to 20 in the same window the other three held their counts steady.

The practical upshot: "does Google publish a current IP file" is not one yes-or-no answer. A verifier built against common-crawlers.json alone will correctly validate Googlebot and miss every AdsBot or Site Verifier request, because those live in different files with different content and, this article's whole point, potentially different regeneration schedules. Checking one file and generalizing to "Google's ranges are current" checks a quarter of the actual surface.

The field nobody documents

creationTime (or, on a few files, a differently-named but equivalent timestamp) sits inside every one of these fifteen JSON files, and none of the operators' own crawler-verification pages mention it. Google's verifying-Googlebot documentation, fetched in full on 26 September 2026, explains what each file contains and how to match a prefix against it; it says nothing about reading the file's own regeneration date. OpenAI's public GPTBot documentation, checked the same day, is the same: it tells a publisher how to block or allow the crawler and does not mention that the range file it links to carries a timestamp at all.

That gap is why this measurement had to be done by hand rather than quoted from a guide: there is no operator-published guidance on how to interpret the field this whole article turns on. The convention exists because whoever built each file's generator happened to timestamp it, not because any operator considers it part of the public verification contract. Reading it is a courtesy the file's format allows, not a feature its documentation promises will always be there or always mean the same thing from one operator to the next.

The file that flipped

This table would have looked different on 7 September 2026, and the difference is the whole point. According to a research pass fetched from the same file on 7 September 2026, gptbot.json had not been regenerated since 30 October 2025, ten months earlier, and it was the single clearest example of an abandoned file: same 21 prefixes, same timestamp, checked three separate times over three weeks. As of today it carries 18 prefixes and a 22 September timestamp. Someone at OpenAI regenerated it four days before this article was fetched, after roughly eleven months of silence.

Nothing about that is visible from the file's structure. There is no changelog, no version number, no way to tell from the JSON itself whether a creationTime of nine months ago means "this hasn't needed to change" or "nobody is maintaining it" until it moves. A file that looked abandoned in September looks current in September of the same month, and the only way to know which state you are reading is to check it the moment before you rely on it.

The files that never sleep, and the ones that are

Set the flip aside and the fifteen files split into two populations that do not overlap. Google's four range files and OpenAI's chatgpt-user.json regenerate on something close to a daily cadence: the user-triggered-agents.json file (the Google-Agent identity) went from 4 prefixes in a March reading to 20 today, and chatgpt-user.json added 23 prefixes in three weeks. These are operators actively expanding infrastructure and republishing the evidence of it.

Then there is Perplexity's perplexitybot.json, unchanged since 7 February 2025, and Mistral's mistralai-user-ips.json, unchanged since 19 February 2025. Both are past nineteen months. Whatever changed about either company's crawling infrastructure since early 2025, none of it reached the file a publisher is told to check. Mistral's training IP file does not exist at all; it returns a 404, confirmed again today. A missing file and a nineteen-month-old file produce the identical outcome for anyone trying to verify a request: neither tells you anything current.

The lesson is not "trust Google, distrust Perplexity." It is that a single publishing convention, a JSON file of IP prefixes with a timestamp, is being used by nine or ten different organizations on completely different internal schedules, and nothing in the convention itself signals which schedule you are looking at.

Bug class two: the files also move, and most checkers do not follow

Staleness inside the file is one failure mode. The other is the file moving entirely. Google's historical URL, developers.google.com/search/apis/ipranges/googlebot.json, now 301s to developers.google.com/static/crawling/ipranges/common-crawlers.json, a different path and a different filename. All three of Google's crawler files relocated the same way. Perplexity's files moved from www.perplexity.com to www.perplexity.ai behind a 302.

Both old URLs still resolve, through the redirect, so this only breaks a checker that does not follow one. That describes most naive implementations: a script that reads the response body directly from a requests.get() or fetch() call without allow_redirects=True (or the language equivalent) gets whatever the old host serves at that path today, which in Google's case is a redirect stub of a few hundred bytes that parses as nothing useful, and in the worst case as an empty allowlist. A verifier that silently allowlists nobody fails closed and looks, from the outside, exactly like a correctly working check that happens to have found no valid ranges. Nobody gets an error. The IP just never matches, forever, and the mismatch is indistinguishable from a real spoofing attempt until someone thinks to check the raw response.

This is not a hypothetical. The research pass behind this article made the same mistake once: the first sweep of these fifteen files, run without -L, mis-recorded two operators as having moved to an empty or broken endpoint, when both were serving a normal file one redirect away. The fix was one flag. Finding it required noticing that two "broken" files belonged to operators with no history of publishing broken files, which is a slower way to catch a bug than just always following redirects in the first place.

What this costs in practice

Most sites do not hand-check an IP file before every request; they bake a snapshot into a WAF rule, an nginx allow block, or a scheduled job that regenerates an allowlist weekly or monthly. That gap between "when the file changes" and "when the allowlist regenerates" is where the two failure modes in this article turn into real outcomes rather than curiosities.

Against Google's four files, which regenerate close to daily, a monthly cron job is stale for most of the month by construction, and the fix is straightforward: shorten the interval or fetch on demand. Against Perplexity's perplexitybot.json, frozen for over nineteen months, a monthly cron job wastes a request checking a file that has not moved since early 2025, but it does no harm. The failure that actually costs something is the migration: a site that pinned developers.google.com/search/apis/ipranges/googlebot.json into a config file, rather than following the redirect at request time, is now reading a URL that still returns 200 but, depending on how the fetch was written, may be reading a stub rather than the live file it thinks it has. The allowlist looks unchanged in the config; the thing it actually resolves to has moved underneath it.

The asymmetry matters for prioritization: staleness inside a rarely-changing file is low-risk, and staleness caused by an unfollowed redirect is high-risk, because it can silently zero out an allowlist that a dashboard still reports as "loaded successfully." Any automated verification pipeline built against these files should log the resolved URL and the prefix count on every fetch, not just a success flag, specifically so a migration like Google's shows up as a visible change instead of a silent one.

Two distinct failure modes are worth telling apart, because the fix for each is different:

Verification does not stop at the IP list

An IP match is the first of two checks most guides recommend; the second is reverse DNS, and it does not cover every crawler either. Every documented AI crawler user-agent, and how to verify it carries the full thirty-three-token inventory and the reverse-DNS results for each one, and it is worth reading in full before building a verifier around either method alone, because the coverage gaps do not line up: some crawlers that publish a good IP file have no reverse-DNS hostname to check, and at least one range (a Google Cloud block also used by Anthropic-run infrastructure) passes a reverse-DNS check for the wrong operator entirely.

Six of ninety-one publisher hosts checked in an earlier pass, dated 7 September 2026, refused a self-declared Googlebot/2.1 request outright, five with a 403 and one with a 429, while serving an ordinary browser normally, meaning they verify by something other than trusting the header. That is the correct instinct. It is also evidence that this stopped being a theoretical exercise: real infrastructure is already checking, and a site that skips verification is depending on nobody testing it, which is a worse position than the alternative of checking and finding gaps.

None of this article's measurement extends to whether the prefixes inside a fresh file are themselves accurate, only to whether the file's own metadata says it was recently regenerated. A file can be freshly timestamped and still miss a range an operator brought online an hour earlier; regeneration cadence is a floor on trustworthiness, not a ceiling. Confirming prefix-level accuracy would require comparing a file's ranges against live traffic from that operator, which is a different, harder measurement this piece does not attempt.

Why this stopped being academic: identity is now a billing decision

According to Cloudflare's own Pay Per Crawl documentation, it bills per successful retrieval and settles the payment on a Web Bot Auth signature. An unsigned request offering the exact asking price gets 403 PaymentFailed, and the Discovery API refuses an unverified caller with 403 {"error":"Only verified bots can use this endpoint"}. Cloudflare Pay Per Crawl: should you charge AI crawlers? covers the mechanics in full; the relevant point here is narrower: the same identity question this article is measuring, which IP belongs to which crawler, and how current that mapping is, is no longer only a bot-access decision. It is the input to whether a request gets charged, refused, or served for free. A stale allowlist used to mean a missed crawler. It can now mean an operator paying for traffic your published ranges say does not exist, or your own site refusing a legitimate paying crawler because the file you checked it against was nineteen months old.

Stealth crawling vs. user-driven fetching measured a related edge on 75 real sites: a Disallow: / aimed at a training crawler is backed by an actual server-side refusal only 45 to 67 percent of the time, depending on the operator. Declaring a rule and enforcing it are two different projects, and the IP files this article measures are the enforcement half for the small number of publishers who do more than write a robots.txt line and hope. Whether a declared block actually stops training on already-published content is a separate question again, covered in Does blocking AI training protect your content?, but it rests on the same gap: a declaration is not a verified outcome.

What AIScan checks, and what it still gets wrong

AIScan's bot-access dimension covers both checks named in this article's tags. B2 reads whether a site's robots.txt names any known AI agent at all, which is a one-time, static read of a file the site controls. B3 checks for a published Web Bot Auth key directory, the mechanism the previous section just described as a live payment gate. According to AIScan's own production database, queried live across 739 real sites on 26 September 2026, B2 passes on 40.1 percent and B3 on 7.7 percent, both dimensions AIScan can grade purely from what a site publishes.

What AIScan cannot see is any of the fifteen files this article fetched by hand. Verifying that a specific inbound request actually originated from the IP range an operator currently publishes requires access to that request's source address at the moment it arrives, which is server-log or edge-log data no external scanner is ever handed. A 100/100 AIScan score is a statement about what a site declares and how it responds to a probe, not a guarantee that its edge is checking incoming crawler traffic against a current file. Those are different questions, and conflating them is exactly the mistake this whole article is about: trusting a static signal to answer a question that only a fresh check can answer.

B3 has a bug worth naming rather than hiding, because it is one more instance of the same problem: a piece of published metadata giving no way to tell current from stale. Grouped by evidence string across the same 739-site read, B3 reports the literal string HTTP 200 for 57 sites marked pass and the identical literal string HTTP 200 for 51 sites marked partial. The verdict differs; the evidence given for the verdict does not. First published against a 499-site read on 9 September, still reproducing today. Anyone who reads a check's status without reading what actually backs it, ours included, is trusting a label instead of checking the file, which is precisely the habit this article is arguing against.

How to check today, in under a minute

The fastest way to see where a specific site stands on this is to run it through AIScan: paste the URL at aiscan.site or run npx aiscan-cli yoursite.com, no account required, and read the B2 and B3 rows directly. That answers what the site declares and publishes.

To check an operator's IP file yourself, by hand, the command is one line per file:

curl -sL https://openai.com/gptbot.json | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('creationTime'), len(d.get('prefixes', d.get('ips', []))))"

Swap in any of the fifteen URLs above. -L is not optional; three of them redirect. If a file's creationTime is months old, that is not automatically a problem, some operators genuinely do not need to change often, but it means the file is telling you what it looked like months ago, and you are the one deciding whether that is still good enough for what you are about to trust it for.

Because this article's own headline finding is that a three-week-old snapshot already misrepresented one of fifteen files, treat every number in the table above the same way: current as of 26 September 2026, not current forever. The fix is not to memorize which operators are fast and which are slow, it is to make the one-line check above part of whatever process relies on these files, on a schedule that matches how often the file in question has actually moved historically, monthly for the daily-regenerating ones is overkill, and a single annual check for a file frozen nineteen months is still a check worth doing, because nineteen months is exactly long enough for an operator to finally move it without anyone downstream noticing.

For a WordPress site making any of these declarations, ThinkRank manages robots.txt, robots meta tags, schema and llms.txt from one plugin rather than three separate ones fighting over the same file, and it migrates existing settings from Rank Math, Yoast, All in One SEO or SEOPress, so adopting it costs nothing already configured. For a Shopify store weighing the same AI-bot-access questions, StoreSEO covers the llms.txt and agents.md half of the same problem for commerce sites specifically.

Your next step

Run your own site through AIScan and read the B2 and B3 rows: aiscan.site or npx aiscan-cli yoursite.com. Then pick the one operator whose crawler matters most to your traffic and check its creationTime field directly, today, with the one-line command above. A file that was fresh in a previous article, including this one, is not a fact about the file, only about the day it was read. For everything else this dimension covers, the full checklist lives at aiscan.site/guides.