---
title: "Does Crawl-delay Slow GPTBot, ClaudeBot or PerplexityBot? What Each Vendor Documents (2026)"
slug: crawl-delay-ai-crawlers
published: 2026-11-20T13:14:00.243576+00:00
updated: 2026-11-20T13:14:14.233785+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:B2, check:D1, Crawl-delay, robots.txt, AI crawlers, rate limiting, GPTBot, ClaudeBot
description: "Which AI crawlers honour robots.txt Crawl-delay: each vendor's own docs quoted, our crawler pace logs, a parser test, and the rate-limit fix for the rest."
url: https://aiscan.site/blog/crawl-delay-ai-crawlers
---

Only some do. Of the AI crawler operators whose documentation we read on 20 November 2026, Anthropic (ClaudeBot) and Common Crawl (CCBot) say they honour `Crawl-delay`, and Microsoft's Bing documents its own reading of the line. Google, Amazon and Apple say they ignore it, and OpenAI (GPTBot, OAI-SearchBot) and Perplexity (PerplexityBot) say nothing about it, so a `Crawl-delay` line is a request to two AI crawlers and a comment to the rest.

This guide gives the per-vendor answer with each source quoted, shows how fast those crawlers hit aiscan.site when nobody asked them to slow down (under three months of our own logs), and gives the fix that works for every crawler: a whole-number `Crawl-delay` for the ones that read it, and a rate limit that never touches your discovery files for the ones that do not.

## TL;DR

| If you want to... | Do this | Works on |
|---|---|---|
| Slow ClaudeBot or CCBot | `Crawl-delay: 2` in that bot's own robots.txt group | ClaudeBot, CCBot (documented) |
| Slow GPTBot, OAI-SearchBot, PerplexityBot | A per-bot rate limit at the server or CDN that answers 429, never 403 | Any client, documented or not |
| Slow Googlebot, Amazonbot, Applebot | Nothing in robots.txt: their docs say the line is ignored | Google, Amazon and Apple each throttle automatically on errors |
| Keep agents finding you while throttled | Exempt `/robots.txt`, `/llms.txt`, `/sitemap.xml` and `/.well-known/` from the limit | Everyone |

Write the value as a whole number of seconds. Python's standard-library parser returns `None` for `Crawl-delay: 0.5` (our test below), so a fractional value can vanish for any crawler built on it.

## Why Crawl-delay is a request, not a rule

`Crawl-delay` was never part of the robots.txt standard. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), published in 2022, defines `user-agent`, `allow` and `disallow` and does not contain the string "crawl-delay" at all. Every operator that supports it chose to, and each one decides what the number means. Bing reads it as a window rather than a rate: [**according to Bing's webmaster blog**](https://blogs.bing.com/webmaster/2012/5/To-crawl-or-not-to-crawl,-that-is-BingBot-s-questi/) (May 2012), a value of 5 means "BingBot will slice the day in smaller five-second windows, crawling only one page (or none) in each of these, for a maximum of around 17,280 pages during the day."

So the line is a polite request. Think of it as a sign on the door asking visitors to wait between knocks: a visitor who reads signs waits, and one who does not walks in. Which visitors read it is a matter of each vendor's own documentation.

## What each AI crawler vendor documents (read 20 November 2026)

Each row below quotes the operator in its own words, fetched from its own crawler page on 20 November 2026.

| Crawler | Operator | Crawl-delay? | What the operator's page says |
|---|---|---|---|
| ClaudeBot | Anthropic | Yes | "we support the non-standard Crawl-delay extension to robots.txt" ([support article 8896518](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) |
| CCBot | Common Crawl | Yes | "We obey the Crawl-delay parameter for robots.txt" ([Common Crawl FAQ](https://commoncrawl.org/faq)) |
| Bingbot (feeds Copilot) | Microsoft | Yes, as a time window | "BingBot honors the Crawl-delay directive" (Bing webmaster blog, 2012) |
| GPTBot, OAI-SearchBot | OpenAI | Not documented | [OpenAI's crawler page](https://platform.openai.com/docs/bots) covers robots.txt tags and IP ranges; the word does not appear |
| PerplexityBot | Perplexity | Not documented | [Perplexity's crawler page](https://docs.perplexity.ai/guides/bots) covers robots.txt tags, IP ranges and WAF rules; the word does not appear |
| Googlebot, Google-Extended | Google | No | "other fields such as crawl-delay aren't supported" ([Google's robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)) |
| Amazonbot | Amazon | No | "They do not support the crawl-delay directive" ([Amazonbot page](https://developer.amazon.com/amazonbot)) |
| Applebot | Apple | No | "Applebot does not follow crawl-delay" ([About Applebot](https://support.apple.com/en-us/119829)) |

"Not documented" means the page was silent, not that the crawler ignores the line. OpenAI or Perplexity may read it; neither says so, and a guide that tells you GPTBot honours it is guessing. We could not read Meta's crawler page on the day (it answered 400 to our request), so Meta is left out rather than guessed.

Two operators document a replacement. According to its own FAQ, Common Crawl's CCBot also slows down "if your web server is responding with a HTTP 429 or 5xx status", and according to Apple, Applebot's "crawl rate adjusts automatically when a site slows down or returns errors". A 429 is a slow-down signal that works even where `Crawl-delay` is never read.

## How fast AI crawlers hit us when nobody asked them to slow down

Our robots.txt carries no `Crawl-delay` (0 matches on 20 November), so these numbers are each crawler's own pace, not a test of obedience. We logged every request from a known AI user agent to aiscan.site from 25 August to 20 November 2026 and measured the gap between consecutive hits from the same hashed IP address.

| Claimed user agent | Gaps measured | Distinct IPs | Median gap | Gaps under 1 second |
|---|---|---|---|---|
| ClaudeBot | 410 | 38 | 1.0 s | 204 (50%) |
| PerplexityBot | 136 | 34 | 3.9 s | 50 (37%) |
| OAI-SearchBot | 79 | 34 | 7.0 s | 21 (27%) |
| GPTBot | 58 | 27 | 9.5 s | 19 (33%) |
| Amazonbot | 56 | 29 | 10.5 s | 14 (25%) |
| Bytespider | 72 | 29 | 10.9 s | 20 (28%) |
| CCBot | 66 | 29 | 14.3 s | 14 (21%) |

The crawler that documents `Crawl-delay` support is also the fastest one we see, with half its gaps under a second. That is not a contradiction: it is the crawler most worth giving a number to. These are claimed user agents, not verified ones (we explain why that matters in [what fake AI crawler traffic looks like in our own logs](https://aiscan.site/blog/fake-ai-crawler-traffic-own-logs)), and the volume is small, under 450 hits per crawler in under three months.

## Do robots.txt parsers even read the value?

We wrote one robots.txt with a ClaudeBot group (`Crawl-delay: 5`) and a wildcard group (`Crawl-delay: 10`) and asked three common open-source parsers what delay each bot gets: Python 3.11's `urllib.robotparser`, [Protego](https://pypi.org/project/Protego/) 0.7.0 (the parser in Scrapy) and the npm package `robots-parser` 3.0.1.

| Parser | ClaudeBot (own group, 5) | GPTBot (wildcard, 10) | ClaudeBot with `Crawl-delay: 0.5` |
|---|---|---|---|
| `urllib.robotparser` | 5 | 10 | `None` |
| Protego 0.7.0 | 5.0 | 10.0 | 0.5 |
| `robots-parser` 3.0.1 | 5 | 10 | 0.5 |

All three pick the right group, and a bot with no group of its own falls back to the wildcard delay. The one disagreement is the fraction: [Python's `crawl_delay()`, verified on 3.11.18,](https://docs.python.org/3/library/urllib.robotparser.html) returned `None` for 0.5 because the standard library only accepts digits. Use whole seconds. For the wider set of ways parsers split on the same file, see [where robots.txt parsers disagree](https://aiscan.site/blog/robots-txt-parsers-disagree-ai-crawlers).

## The fix: one robots.txt group per crawler that reads it, a rate limit for the rest

**Step 1. Add the delay to each documented crawler's own group.** A bot obeys only the most specific group that names it, so put the line inside its group, not only under `*`:

```txt
User-agent: ClaudeBot
Crawl-delay: 2
Allow: /

User-agent: CCBot
Crawl-delay: 5
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
```

A value of 1 to 5 seconds caps a crawler at 17,280 to 86,400 requests a day, which no small site needs to go below. On WordPress, [ThinkRank](https://thinkrank.ai) edits this file from one robots.txt screen, so you do not end up with an SEO plugin and a security plugin each writing their own copy, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Yoast and Rank Math also have robots.txt editors if you already use them.

**Step 2. Rate-limit everything else at the server or CDN, and answer 429.** For GPTBot, OAI-SearchBot and PerplexityBot, the line may do nothing, so the limit has to be enforced. Our [guide to rate-limiting AI crawlers without blocking them](https://aiscan.site/blog/rate-limit-ai-crawlers-without-blocking) has the nginx, Cloudflare and Apache rules. Answer 429 with a `Retry-After` header, not 403: a 403 reads as "go away", and a 429 is the slow-down signal Common Crawl and Apple both document.

**Step 3. Exempt discovery files from the limit.** `/robots.txt`, `/llms.txt`, `/sitemap.xml` and everything under `/.well-known/` should never be throttled. A crawler that gets a 429 on robots.txt cannot read your delay at all. Worse, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) section 2.3.1.3 lets a crawler treat any 4xx robots.txt as "unavailable" and crawl everything, so a throttled robots.txt can mean no rules rather than slower crawling. Google's specification says it treats every 4xx except 429 as if no file existed, and adds: "Don't use 401 and 403 status codes for limiting the crawl rate."

**Step 4. Make repeat visits cheap.** Fewer bytes per visit often matters more than fewer visits. [Conditional requests with ETag and Last-Modified](https://aiscan.site/blog/conditional-requests-etag-last-modified) let a crawler get a 304 for a page that has not changed.

## Verify it

Scan the site first. AIScan's B2 check reads robots.txt the way AI crawlers do, and D1 confirms the file answers 200 as `text/plain`:

```bash
npx aiscan-cli yoursite.com
```

Expected: B2 and D1 pass. Neither check grades `Crawl-delay` itself, and adding the line does not change your score; a pass means the file is reachable and your bot groups parse.

By hand, confirm the line is in the group you meant and that robots.txt is not caught by your rate limit:

```bash
curl -s https://yoursite.com/robots.txt | grep -i -B2 'crawl-delay'
for i in $(seq 1 30); do curl -s -o /dev/null -w '%{http_code} ' -A 'ClaudeBot' https://yoursite.com/robots.txt; done; echo
```

Expected: the `User-agent: ClaudeBot` line sits directly above its `Crawl-delay`, and the loop prints thirty 200s. A 429 in the loop means the exemption in step 3 is missing. Then watch your access log for a week: ClaudeBot's gap between hits from one IP should not fall below the value you set. For the full list of AI user agents to group, see the [AI crawler user-agent list](https://aiscan.site/blog/ai-crawler-user-agent-list-2026), and for a complete starting file, the [robots.txt starter for AI crawlers](https://aiscan.site/blog/robots-txt-starter-for-ai-crawlers).

## Next step

Run a [free AIScan scan](https://aiscan.site/) of your site and open the [bot access checks](https://aiscan.site/docs/checks/bot-access) to see how B2 reads your robots.txt groups. More fixes like this one are in the [AIScan guides](https://aiscan.site/guides).

