AIScan fix guide cover: Does Crawl-delay slow AI crawlers? Anthropic and Common Crawl say yes, OpenAI and Perplexity say nothing.
AIScan fix guide cover: Does Crawl-delay slow AI crawlers? Anthropic and Common Crawl say yes, OpenAI and Perplexity say nothing.
AI Readiness

Does Crawl-delay Slow GPTBot, ClaudeBot or PerplexityBot? What Each Vendor Documents (2026)

Which AI crawlers honour robots.txt Crawl-delay: each vendor's own docs quoted, our crawler pace logs, a parser test, and the rate-limit fix for the rest.

20 Nov 2026 9 min read
#Crawl-delay#robots.txt#AI crawlers#rate limiting#GPTBot#ClaudeBot

This guide covers B2 · Bot Access, D1 · Discoverability.

Table of contents

Only some do. Of the AI crawler operators whose documentation we read on 20 November 2026, Anthropic (ClaudeBot) and Common Crawl (CCBot) say they honour Crawl-delay, and Microsoft's Bing documents its own reading of the line. Google, Amazon and Apple say they ignore it, and OpenAI (GPTBot, OAI-SearchBot) and Perplexity (PerplexityBot) say nothing about it, so a Crawl-delay line is a request to two AI crawlers and a comment to the rest.

This guide gives the per-vendor answer with each source quoted, shows how fast those crawlers hit aiscan.site when nobody asked them to slow down (under three months of our own logs), and gives the fix that works for every crawler: a whole-number Crawl-delay for the ones that read it, and a rate limit that never touches your discovery files for the ones that do not.

TL;DR

If you want to...Do thisWorks on
Slow ClaudeBot or CCBotCrawl-delay: 2 in that bot's own robots.txt groupClaudeBot, CCBot (documented)
Slow GPTBot, OAI-SearchBot, PerplexityBotA per-bot rate limit at the server or CDN that answers 429, never 403Any client, documented or not
Slow Googlebot, Amazonbot, ApplebotNothing in robots.txt: their docs say the line is ignoredGoogle, Amazon and Apple each throttle automatically on errors
Keep agents finding you while throttledExempt /robots.txt, /llms.txt, /sitemap.xml and /.well-known/ from the limitEveryone

Write the value as a whole number of seconds. Python's standard-library parser returns None for Crawl-delay: 0.5 (our test below), so a fractional value can vanish for any crawler built on it.

Why Crawl-delay is a request, not a rule

Crawl-delay was never part of the robots.txt standard. RFC 9309, published in 2022, defines user-agent, allow and disallow and does not contain the string "crawl-delay" at all. Every operator that supports it chose to, and each one decides what the number means. Bing reads it as a window rather than a rate: according to Bing's webmaster blog (May 2012), a value of 5 means "BingBot will slice the day in smaller five-second windows, crawling only one page (or none) in each of these, for a maximum of around 17,280 pages during the day."

So the line is a polite request. Think of it as a sign on the door asking visitors to wait between knocks: a visitor who reads signs waits, and one who does not walks in. Which visitors read it is a matter of each vendor's own documentation.

What each AI crawler vendor documents (read 20 November 2026)

Each row below quotes the operator in its own words, fetched from its own crawler page on 20 November 2026.

CrawlerOperatorCrawl-delay?What the operator's page says
ClaudeBotAnthropicYes"we support the non-standard Crawl-delay extension to robots.txt" (support article 8896518)
CCBotCommon CrawlYes"We obey the Crawl-delay parameter for robots.txt" (Common Crawl FAQ)
Bingbot (feeds Copilot)MicrosoftYes, as a time window"BingBot honors the Crawl-delay directive" (Bing webmaster blog, 2012)
GPTBot, OAI-SearchBotOpenAINot documentedOpenAI's crawler page covers robots.txt tags and IP ranges; the word does not appear
PerplexityBotPerplexityNot documentedPerplexity's crawler page covers robots.txt tags, IP ranges and WAF rules; the word does not appear
Googlebot, Google-ExtendedGoogleNo"other fields such as crawl-delay aren't supported" (Google's robots.txt specification)
AmazonbotAmazonNo"They do not support the crawl-delay directive" (Amazonbot page)
ApplebotAppleNo"Applebot does not follow crawl-delay" (About Applebot)

"Not documented" means the page was silent, not that the crawler ignores the line. OpenAI or Perplexity may read it; neither says so, and a guide that tells you GPTBot honours it is guessing. We could not read Meta's crawler page on the day (it answered 400 to our request), so Meta is left out rather than guessed.

Two operators document a replacement. According to its own FAQ, Common Crawl's CCBot also slows down "if your web server is responding with a HTTP 429 or 5xx status", and according to Apple, Applebot's "crawl rate adjusts automatically when a site slows down or returns errors". A 429 is a slow-down signal that works even where Crawl-delay is never read.

How fast AI crawlers hit us when nobody asked them to slow down

Our robots.txt carries no Crawl-delay (0 matches on 20 November), so these numbers are each crawler's own pace, not a test of obedience. We logged every request from a known AI user agent to aiscan.site from 25 August to 20 November 2026 and measured the gap between consecutive hits from the same hashed IP address.

Claimed user agentGaps measuredDistinct IPsMedian gapGaps under 1 second
ClaudeBot410381.0 s204 (50%)
PerplexityBot136343.9 s50 (37%)
OAI-SearchBot79347.0 s21 (27%)
GPTBot58279.5 s19 (33%)
Amazonbot562910.5 s14 (25%)
Bytespider722910.9 s20 (28%)
CCBot662914.3 s14 (21%)

The crawler that documents Crawl-delay support is also the fastest one we see, with half its gaps under a second. That is not a contradiction: it is the crawler most worth giving a number to. These are claimed user agents, not verified ones (we explain why that matters in what fake AI crawler traffic looks like in our own logs), and the volume is small, under 450 hits per crawler in under three months.

Do robots.txt parsers even read the value?

We wrote one robots.txt with a ClaudeBot group (Crawl-delay: 5) and a wildcard group (Crawl-delay: 10) and asked three common open-source parsers what delay each bot gets: Python 3.11's urllib.robotparser, Protego 0.7.0 (the parser in Scrapy) and the npm package robots-parser 3.0.1.

ParserClaudeBot (own group, 5)GPTBot (wildcard, 10)ClaudeBot with Crawl-delay: 0.5
urllib.robotparser510None
Protego 0.7.05.010.00.5
robots-parser 3.0.15100.5

All three pick the right group, and a bot with no group of its own falls back to the wildcard delay. The one disagreement is the fraction: Python's crawl_delay(), verified on 3.11.18, returned None for 0.5 because the standard library only accepts digits. Use whole seconds. For the wider set of ways parsers split on the same file, see where robots.txt parsers disagree.

The fix: one robots.txt group per crawler that reads it, a rate limit for the rest

Step 1. Add the delay to each documented crawler's own group. A bot obeys only the most specific group that names it, so put the line inside its group, not only under *:

User-agent: ClaudeBot
Crawl-delay: 2
Allow: /

User-agent: CCBot
Crawl-delay: 5
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

A value of 1 to 5 seconds caps a crawler at 17,280 to 86,400 requests a day, which no small site needs to go below. On WordPress, ThinkRank edits this file from one robots.txt screen, so you do not end up with an SEO plugin and a security plugin each writing their own copy, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Yoast and Rank Math also have robots.txt editors if you already use them.

Step 2. Rate-limit everything else at the server or CDN, and answer 429. For GPTBot, OAI-SearchBot and PerplexityBot, the line may do nothing, so the limit has to be enforced. Our guide to rate-limiting AI crawlers without blocking them has the nginx, Cloudflare and Apache rules. Answer 429 with a Retry-After header, not 403: a 403 reads as "go away", and a 429 is the slow-down signal Common Crawl and Apple both document.

Step 3. Exempt discovery files from the limit. /robots.txt, /llms.txt, /sitemap.xml and everything under /.well-known/ should never be throttled. A crawler that gets a 429 on robots.txt cannot read your delay at all. Worse, RFC 9309 section 2.3.1.3 lets a crawler treat any 4xx robots.txt as "unavailable" and crawl everything, so a throttled robots.txt can mean no rules rather than slower crawling. Google's specification says it treats every 4xx except 429 as if no file existed, and adds: "Don't use 401 and 403 status codes for limiting the crawl rate."

Step 4. Make repeat visits cheap. Fewer bytes per visit often matters more than fewer visits. Conditional requests with ETag and Last-Modified let a crawler get a 304 for a page that has not changed.

Verify it

Scan the site first. AIScan's B2 check reads robots.txt the way AI crawlers do, and D1 confirms the file answers 200 as text/plain:

npx aiscan-cli yoursite.com

Expected: B2 and D1 pass. Neither check grades Crawl-delay itself, and adding the line does not change your score; a pass means the file is reachable and your bot groups parse.

By hand, confirm the line is in the group you meant and that robots.txt is not caught by your rate limit:

curl -s https://yoursite.com/robots.txt | grep -i -B2 'crawl-delay'
for i in $(seq 1 30); do curl -s -o /dev/null -w '%{http_code} ' -A 'ClaudeBot' https://yoursite.com/robots.txt; done; echo

Expected: the User-agent: ClaudeBot line sits directly above its Crawl-delay, and the loop prints thirty 200s. A 429 in the loop means the exemption in step 3 is missing. Then watch your access log for a week: ClaudeBot's gap between hits from one IP should not fall below the value you set. For the full list of AI user agents to group, see the AI crawler user-agent list, and for a complete starting file, the robots.txt starter for AI crawlers.

Next step

Run a free AIScan scan of your site and open the bot access checks to see how B2 reads your robots.txt groups. More fixes like this one are in the AIScan guides.

Frequently asked questions

Does GPTBot honour Crawl-delay?

OpenAI does not say. Its crawler page, read on 20 November 2026, covers robots.txt tags and published IP ranges for GPTBot and OAI-SearchBot and never mentions Crawl-delay. Treat the line as unconfirmed for OpenAI and enforce a rate limit at the server if you need GPTBot slower.

Does ClaudeBot respect Crawl-delay?

Yes. Anthropic's support article 8896518 says "we support the non-standard Crawl-delay extension to robots.txt" and gives User-agent: ClaudeBot with Crawl-delay: 1 as its own example. Put the line in a ClaudeBot group, not only under User-agent: *.

I added Crawl-delay: 10 and GPTBot still hits me every few seconds. What is wrong?

Probably nothing in your file. OpenAI does not document Crawl-delay support, so the line may simply be ignored. Add a per-bot rate limit at nginx, Apache or your CDN that answers 429 with a Retry-After header, and keep robots.txt itself outside that limit.

Googlebot ignores my crawl-delay line. Is my robots.txt broken?

No. Google's robots.txt specification says other fields such as crawl-delay aren't supported, so Googlebot skips that one line and reads the rest of the file normally. An unsupported line is not an error.

My rate limit returns 429 on robots.txt. Why did crawlers start ignoring my rules?

RFC 9309 lets a crawler treat any 4xx answer on robots.txt as unavailable and then crawl everything, so a throttled robots.txt can mean no rules at all. Exempt /robots.txt, /llms.txt, /sitemap.xml and /.well-known/ from every rate limit.

Crawl-delay: 0.5 seems to do nothing. Why?

Some parsers drop fractional values. Python's standard-library urllib.robotparser returned None for 0.5 in our test on Python 3.11.18, while Protego and the npm robots-parser read 0.5. Use whole seconds, such as 1 or 2.

Does Crawl-delay affect my AIScan score?

No. AIScan's B2 check reads your robots.txt groups for AI crawlers and D1 checks that the file answers 200 as text/plain. Neither grades Crawl-delay, so adding or removing it leaves the score unchanged.

What Crawl-delay value should I use?

Between 1 and 5 seconds for a crawler that reads it. By Bing's own reading of the line, 1 allows about 86,400 requests a day and 5 about 17,280, which is already gentle for a small site. Higher values mostly mean fewer of your pages get crawled.

M Asif Rahman

About the author

Entrepreneur, curious about AI

I have been building companies for a long time, and lately most of my curiosity goes to AI. AIScan is a small personal project I built on a flight from Dubai to Narita. I use it to understand how ready the web is for AI agents, and I share what the scans teach me.

Related guides

Browse every guide by check and platform