---
title: "Why robots.txt Checkers Disagree About GPTBot: 5 Parsers, 15 Files, Tested (2026)"
slug: robots-txt-parsers-disagree-ai-crawlers
published: 2026-11-12T03:13:41.760453+00:00
updated: 2026-11-12T03:13:41.760453+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:B2, check:D1, robots.txt, AI crawlers
description: "Robots.txt checkers disagree about GPTBot: 5 parsers, 15 files tested. Version tokens, a BOM and files over 500 KiB split them. Raw verdicts and a spec checker."
url: https://aiscan.site/blog/robots-txt-parsers-disagree-ai-crawlers
---

Robots.txt checkers disagree about GPTBot because each one is built on a parser library, and the libraries do not agree on the rules. We ran 15 small robots.txt files through five parsers that checker tools are commonly built on. On 21 test cases all five matched RFC 9309 and Google's documented behaviour on 18. The three they missed are the ones that decide real verdicts: a version number in the `User-agent` line, a byte order mark at the start of the file, and a file over 500 KiB. Python's built-in parser also changed its answer on five cases between Python 3.12 and 3.13.

Below are the files, the raw verdicts, the RFC 9309 rule behind each row, and a short spec-aware checker you can copy. It matches the reference column on all 21 cases.

## Quick summary

| Question | Answer | Source |
|---|---|---|
| Do robots.txt parsers agree? | On 18 of 21 cases all five matched the spec. They split on 2 cases and all five got a third wrong | AIScan parser lab, 15 files, 21 cases, 12 Nov 2026 |
| Most common miss | A file over 500 KiB: all five read past the limit Google stops at | [Google robots.txt spec](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt), read 12 Nov 2026 |
| `User-agent: GPTBot/1.1` | Google reads it as `GPTBot`. Three of five parsers ignored the group | Google spec ("googlebot/1.2 ... equivalent to googlebot") |
| UTF-8 byte order mark before line 1 | Google skips it. Three of five parsers lost the first group | Same lab, row 12 |
| Allow and Disallow of equal length | Allow wins, and all five agreed | [RFC 9309 §2.2.2](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.2) |
| Python `urllib.robotparser` | 3.11.18 and 3.12.3 missed 8 of 21 cases; 3.13.18 missed 3 | Same lab, three Python builds |
| Does a parser verdict tell you what a bot did? | No. Only your server logs show that | [Find AI crawlers in your server logs](https://aiscan.site/blog/find-ai-crawlers-in-server-logs) |

**Short answer for a site owner:** if two checkers disagree about your file, the file almost certainly has one of the three problems above, and each has a one-line fix. Run `npx aiscan-cli yoursite.com` (or paste your address at [aiscan.site](https://aiscan.site/), free, no account) and read check **B2** (explicit AI bot rules) and **D1** (robots.txt served and readable) on the [bot access](https://aiscan.site/docs/checks/bot-access) and [discoverability](https://aiscan.site/docs/checks/discoverability) pages. Then fix the file with the table at the end of this post.

**Short answer for someone building a checker:** behave like Google's open-source matcher. Read the product token up to the first character that is not a letter, `-` or `_`. Strip a leading BOM. Stop parsing at 500 KiB. Merge every group that names the agent. Pick the longest matching rule, and let Allow win a tie.

## Why this matters to anyone who checks AI crawler access

People ask one question about their robots.txt more than any other: is GPTBot (or ClaudeBot, or PerplexityBot) allowed? In the last month three public threads show people building tools to answer it: [a resolver run over 122 well-known sites for 19 AI crawlers](https://www.reddit.com/r/TechSEO/comments/1xa8nld/), [a checker for 14 AI crawlers](https://www.reddit.com/r/webdev/comments/1xeb64m/), and [a robots.txt tester looking for feedback](https://www.reddit.com/r/alphaandbetausers/comments/1wib9bq/). Each sits on a parser. When the parser reads a file differently from the crawler, the tool gives a confident wrong answer.

The rules themselves are settled. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) (September 2022) standardised the Robots Exclusion Protocol. [Google's robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) documents how Google applies it, including choices the RFC leaves to the crawler, such as the 500 KiB limit. The gaps are in the libraries.

This post owns one question: how parsers match agents and paths. Three neighbouring questions have their own posts. What Python does with a 403 or 5xx response is covered in [Python's robots.txt parser treats a 403 as disallow-all](https://aiscan.site/blog/robots-txt-403-python-robotparser-disallow-all). The new AI-training policy token is covered in [the `__usage_ai-training` token, tested](https://aiscan.site/blog/usage-ai-training-robots-txt-token). A leftover `Disallow: /` is covered in [your robots.txt blocks everything by accident](https://aiscan.site/blog/robots-txt-disallow-all-by-accident).

## How we tested

Everything ran locally on 12 Nov 2026. We fetched nothing from anyone else's site, so every file is one we wrote, each isolating one edge case.

| Parser | Version | Language | Where it shows up |
|---|---|---|---|
| `urllib.robotparser` | Python 3.13.18 (also 3.12.3 and 3.11.18) | Python standard library | Scripts, notebooks, quick checkers |
| [Protego](https://pypi.org/project/Protego/) | 0.7.0 | Python | Scrapy's default robots.txt parser |
| [robotspy](https://pypi.org/project/robotspy/) | 0.13.0 | Python | Standalone checkers and CLIs |
| [robotstxt](https://pypi.org/project/robotstxt/) | 1.1.2 | Python | Standalone checkers. Not a wrapper around Google's C++ code: it is a separate pure-Python package |
| robots-parser (npm) | 3.0.1 | JavaScript (Node 24.22.1) | Node crawlers and web-based testers |

**Method.** Each file was saved as UTF-8 bytes and decoded the way a plain fetcher does (`decode("utf-8")`, no BOM stripping). Each parser then got the same text, the same agent product token and the same URL on `https://example.com`. That gave 21 cases (file × agent × path). The reference column applies RFC 9309 and, where the RFC leaves a choice, Google's specification page.

**What we could not run.** We meant to build [Google's open-source matcher](https://developers.google.com/search/blog/2019/07/repp-oss) (the C++ `google/robotstxt` library) as a sixth column. The sandbox for this run does not allow compiling and running outside code, so we dropped it. We read its source for the three cases where parsers split, and we quote what it does below, but we did not run it.

## The 15 files

| File | What it isolates | The rule that decides it |
|---|---|---|
| 01 | `User-agent: GPTBot/1.1` then `Disallow: /`, plus `User-agent: *` / `Allow: /` | Google: "googlebot/1.2 and googlebot* are equivalent to googlebot" |
| 02 | `User-agent: gptbot` (lower case) | RFC 9309 §2.2.1: case-insensitive matching |
| 03 | `Disallow: /page` and `Allow: /page` | RFC 9309 §2.2.2: on a tie, allow SHOULD be used |
| 04 | `Disallow: /docs/` above `Allow: /docs/public/` | RFC 9309 §2.2.2: the longest match wins |
| 05 | `Disallow: /*.pdf$` and `Disallow: /tmp*` | RFC 9309 §2.2.3: `*` and `$` |
| 06 | `Disallow:` (empty) for GPTBot, `Disallow: /` for `*` | an empty rule matches nothing |
| 07 | Two GPTBot groups with a `*` group between them | RFC 9309 §2.2.1: matching groups MUST be combined |
| 08 | `User-agent: GPTBot`, blank line, `User-agent: *`, `Disallow: /` | RFC 9309 §2.2 grammar allows empty lines between user-agent lines |
| 09 | A UTF-8 BOM before `User-agent: GPTBot` | Google ignores a leading BOM |
| 10 | CRLF line endings | RFC 9309 §2.2: CR, LF or CRLF |
| 11 | A `Sitemap:` line inside the GPTBot group | RFC 9309 §2.2.4: other records must not interfere |
| 12 | `Disallow: /café/` | RFC 9309 §2.2.2: compare percent-encoded UTF-8 |
| 13 | 630,054 bytes: an `Allow: /` group, padding comments, then `User-agent: GPTBot` / `Disallow: /` at the end | RFC 9309 §2.5 and Google's 500 KiB limit |
| 14 | `User-agent: *` / `Disallow: /`, then `User-agent: GPTBot` / `Allow: /` | a named group beats `*` |
| 15 | `User-agent: GPTBot # OpenAI` stacked on `User-agent: ClaudeBot`, `Disallow: /` | comments are ignored; stacked lines share a group |

## The results, case by case

A ✗ marks a verdict that differs from the reference column.

| # | File (agent, path) | RFC 9309 / Google | urllib 3.13.18 | Protego | robotspy | robotstxt | robots-parser |
|---|---|---|---|---|---|---|---|
| 1 | `01-version-token` (GPTBot, `/`) | block | allow ✗ | allow ✗ | block | allow ✗ | block |
| 2 | `02-lowercase-agent` (GPTBot, `/private/x`) | block | block | block | block | block | block |
| 3 | `03-equal-length-tie` (GPTBot, `/page`) | allow | allow | allow | allow | allow | allow |
| 4 | `04-longest-vs-first` (GPTBot, `/docs/public/a`) | allow | allow | allow | allow | allow | allow |
| 5 | `05-wildcards` (GPTBot, `/files/report.pdf`) | block | block | block | block | block | block |
| 6 | `05-wildcards` (GPTBot, `/files/report.pdf?v=2`) | allow | allow | allow | allow | allow | allow |
| 7 | `05-wildcards` (GPTBot, `/tmp-cache/x`) | block | block | block | block | block | block |
| 8 | `06-empty-disallow` (GPTBot, `/`) | allow | allow | allow | allow | allow | allow |
| 9 | `07-split-groups` (GPTBot, `/b/x`) | block | block | block | block | block | block |
| 10 | `07-split-groups` (GPTBot, `/c`) | allow | allow | allow | allow | allow | allow |
| 11 | `08-agent-no-rules` (GPTBot, `/`) | block | block | block | block | block | block |
| 12 | `09-utf8-bom` (GPTBot, `/`) | block | allow ✗ | allow ✗ | allow ✗ | block | block |
| 13 | `10-crlf` (GPTBot, `/private/x`) | block | block | block | block | block | block |
| 14 | `11-sitemap-in-group` (GPTBot, `/private/x`) | block | block | block | block | block | block |
| 15 | `12-non-ascii-path` (GPTBot, `/caf%C3%A9/menu`) | block | block | block | block | block | block |
| 16 | `12-non-ascii-path` (GPTBot, `/café/menu`) | block | block | block | block | block | block |
| 17 | `13-over-500kib` (GPTBot, `/`) | allow | block ✗ | block ✗ | block ✗ | block ✗ | block ✗ |
| 18 | `14-specific-beats-star` (GPTBot, `/`) | allow | allow | allow | allow | allow | allow |
| 19 | `14-specific-beats-star` (Googlebot, `/`) | block | block | block | block | block | block |
| 20 | `15-stacked-with-comment` (GPTBot, `/`) | block | block | block | block | block | block |
| 21 | `15-stacked-with-comment` (ClaudeBot, `/`) | block | block | block | block | block | block |

Tally of departures from the reference, out of 21:

| Parser | Wrong | Which rows |
|---|---|---|
| robots-parser 3.0.1 | 1 | 17 (over 500 KiB) |
| robotspy 0.13.0 | 2 | 12 (BOM), 17 |
| robotstxt 1.1.2 | 2 | 1 (version token), 17 |
| Protego 0.7.0 | 3 | 1, 12, 17 |
| urllib.robotparser, Python 3.13.18 | 3 | 1, 12, 17 |
| urllib.robotparser, Python 3.12.3 and 3.11.18 | 8 | 1, 3, 4, 5, 7, 9, 12, 17 |

The good news comes first. Every parser handled case, CRLF, comments, stacked agents, empty `Disallow:`, the `Sitemap:` line inside a group, the non-ASCII path in both encodings, and the named-group-beats-`*` rule. If your checker is wrong, it is wrong on one of the three cases below.

## Disagreement 1: a version number in the User-agent line

File 01 says `User-agent: GPTBot/1.1` with `Disallow: /`, and a `*` group allows everything. Google's specification says "All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot)". Google's matcher implements this in `ExtractUserAgent`, which keeps only letters, `-` and `_` from the start of the value. So for Google-style matching, GPTBot is blocked.

robotspy and robots-parser agreed and blocked GPTBot. `urllib.robotparser`, Protego and robotstxt treated `GPTBot/1.1` as a different name, fell through to `*`, and said GPTBot was allowed.

RFC 9309 §2.2.1 says the product token in a user-agent line may contain only letters, underscores and hyphens, so `GPTBot/1.1` is not a valid line. The fix belongs in the file: write `User-agent: GPTBot`. Bot documentation pages give full user-agent strings such as `GPTBot/1.1`, and people paste them in. The [AI crawler user-agent list](https://aiscan.site/blog/ai-crawler-user-agent-list-2026) gives the bare token for each crawler, which is what goes in robots.txt.

## Disagreement 2: a byte order mark before the first line

File 09 is two lines, `User-agent: GPTBot` and `Disallow: /`, saved with a UTF-8 byte order mark (the bytes `EF BB BF`). Many Windows editors add one silently. Google's specification says Google "ignores invalid lines in robots.txt files, including the Unicode Byte Order Mark (BOM) at the beginning", and its matcher skips the BOM byte by byte. For Google, GPTBot is blocked.

robotstxt and robots-parser coped. `urllib.robotparser`, Protego and robotspy read the first line as a field named `user-agent`, did not recognise it, dropped the whole group, and reported GPTBot as allowed.

So a site that blocks GPTBot in its first group can see a checker built on Scrapy's parser say the opposite. The fix is to save robots.txt as UTF-8 without a BOM. If the first group must stay first, a comment line at the top also works: a stray BOM then spoils only the comment.

## Disagreement 3: everything after 500 KiB

File 13 is 630,054 bytes. It opens with `User-agent: *` / `Allow: /`, runs 9,000 comment lines, and ends with `User-agent: GPTBot` / `Disallow: /`. RFC 9309 §2.5 says "The parsing limit MUST be at least 500 kibibytes", so a crawler may stop there. Google says "Content which is after the maximum file size is ignored." For Google, the GPTBot group past byte 512,000 does not exist, and GPTBot falls back to `Allow: /`.

All five parsers read the whole file and blocked GPTBot. That is the only row where every library agreed with each other and not with Google. Other crawlers may set a higher limit (the RFC sets only a minimum), so this row has no single right answer for every bot. Within Google's documented behaviour, the checkers are wrong.

Files this large are rare and usually generated: one rule per URL, or a plugin appending to the file on every save. The fix is to put named AI crawler groups near the top and merge per-URL rules into directory rules. Our [D1 check](https://aiscan.site/docs/checks/discoverability) confirms the file is served and readable. It does not judge its size.

## The surprises in the rows every parser got right

Eighteen rows agreed, but several of them surprise people who write robots.txt by hand. They are worth knowing before you edit a live file.

**A blank line does not end a group (file 08).** Many people separate `User-agent: GPTBot` from `User-agent: *` with a blank line and expect GPTBot to get a group with no rules, which would mean "allowed everywhere". The RFC 9309 grammar allows empty lines between user-agent lines, so the two lines form one group and GPTBot inherits `Disallow: /`. All five parsers blocked GPTBot, and so does Google's matcher, which ignores blank lines. If you want GPTBot allowed, give it a rule: `Allow: /`.

**Two groups for the same bot are one group (file 07).** RFC 9309 §2.2.1 says matching groups "MUST be combined into one group". A GPTBot group at the top and another at the bottom both apply, so `/b/` stays blocked. Files grow this way when a plugin appends its own group and nobody merges them. Every current parser merged them; only Python 3.12 and older did not.

**A named group replaces `*` and does not add to it (file 14).** `User-agent: GPTBot` with `Allow: /` frees GPTBot from a site-wide `Disallow: /`, and Googlebot stays blocked. The reverse applies too: if you give GPTBot a group to block one folder, it no longer inherits anything from `*`, including your other `Disallow` lines. Google's spec says it directly: "User agent specific groups and global groups (*) are not combined."

**An empty `Disallow:` allows everything (file 06).** It reads like "disallow", but an empty value matches no path. RFC 9309 §2.2.2 says that when no rule matches, "the URI is allowed".

**Allow wins a tie (file 03).** When `Allow: /page` and `Disallow: /page` are the same length, RFC 9309 says the allow rule "SHOULD be used", and Google says it "uses the least restrictive rule". All five current parsers agreed. Python 3.12 did not.

**`$` only anchors the end of the path (file 05).** `Disallow: /*.pdf$` blocks `/files/report.pdf` and leaves `/files/report.pdf?v=2` allowed, because the query string is part of what gets matched. If you mean every PDF, drop the `$`.

## Python changed its answer between 3.12 and 3.13

Yesterday's [training-token lab](https://aiscan.site/blog/usage-ai-training-robots-txt-token) found that Python's parser takes the first matching rule, so `Disallow: /` above `Allow: /press-kit/` wins. That held on Python 3.11.18 and 3.12.3. On 3.13.18 the same module picks the longest match, lets Allow win ties, and understands `*` and `$`. The source shows it directly: `RuleLine.applies_to` now returns the match length, and `Entry.allowance` keeps the best one.

| Case | RFC 9309 / Google | Python 3.11.18 | Python 3.12.3 | Python 3.13.18 |
|---|---|---|---|---|
| 3: equal-length Allow and Disallow | allow | block | block | allow |
| 4: `/docs/` blocked, `/docs/public/` allowed | allow | block | block | allow |
| 5: `/*.pdf$` on `/files/report.pdf` | block | allow | allow | block |
| 7: `/tmp*` on `/tmp-cache/x` | block | allow | allow | block |
| 9: second GPTBot group's `/b/` | block | allow | allow | block |
| 1: `GPTBot/1.1` | block | allow | allow | allow |
| 12: BOM | block | allow | allow | allow |
| 17: over 500 KiB | allow | block | block | block |

The older builds also used only the first GPTBot group in file 07 and ignored the second. In practice, a checker on Python's standard library gives different answers depending on the server's Python version. Pin the version, or use a library that states its matching rules.

## Which parser should a checker behave like?

| If you are... | Behave like | Why |
|---|---|---|
| Checking Google's view of a file | Google's open-source matcher, or Google's spec page | It is the code Google published as the reference for its crawler |
| Checking OpenAI, Anthropic or Perplexity crawlers | RFC 9309, with Google's three choices as the likely reading | None of them publishes a parser; RFC 9309 is the shared baseline |
| Building in Python | Protego 0.7.0 or robotspy 0.13.0, plus a BOM strip and a 500 KiB cut | Both handle longest match, wildcards and merging; each missed 1 or 2 of the 3 hard cases |
| Building in Node | robots-parser 3.0.1, plus a 500 KiB cut | Only the size limit missed in our 21 cases |
| Using `urllib.robotparser` | Python 3.13 or later only | Older builds missed 8 of 21 |

The three fixes are small. Most of the remaining risk is in how a checker fetches the file (status codes, redirects, content type) rather than how it parses it.

## A spec-aware checker in about 50 lines

This checker applies every rule in the reference column. On our 21 cases it matched the reference 21 times. It uses only the Python standard library and does no fetching: give it the bytes you got from `/robots.txt`. Status handling stays your job, and [RFC 9309 §2.3.1](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1) says what to do with 4xx and 5xx responses.

```python
import re
from urllib.parse import quote, unquote, urlsplit

LIMIT = 500 * 1024  # RFC 9309 section 2.5: parse at least 500 KiB; Google stops there

def norm(path):
    # compare percent-encoded UTF-8, keep reserved characters as written
    return quote(unquote(path), safe="/?=&*$%:@!+,;~")

def groups(raw: bytes):
    text = raw[:LIMIT].decode("utf-8", "ignore").lstrip("")
    out, agents, rules_seen = [], [], False
    for line in re.split(r"\r\n|\r|\n", text):
        line = line.split("#", 1)[0].strip()
        if ":" not in line:
            continue                      # blank and junk lines never end a group
        key, val = (s.strip() for s in line.split(":", 1))
        key = key.lower()
        if key == "user-agent":
            if rules_seen:
                agents, rules_seen = [], False
            if not agents:
                out.append((agents, []))
            agents.append("*" if val.startswith("*") else re.match(r"[A-Za-z_-]*", val).group().lower())
        elif key in ("allow", "disallow") and out:
            rules_seen = True
            out[-1][1].append((key == "allow", norm(val)))
    return out

def rule_regex(pattern):
    end = pattern.endswith("$")
    body = re.escape(pattern.rstrip("$")).replace(r"\*", ".*")
    return re.compile(body + ("$" if end else ""))

def allowed(raw: bytes, agent: str, url: str) -> bool:
    agent = re.match(r"[A-Za-z_-]*", agent).group().lower()
    gs = groups(raw)
    rules = [r for a, rs in gs if agent in a for r in rs]          # merge every named group
    if not any(agent in a for a, _ in gs):
        rules = [r for a, rs in gs if "*" in a for r in rs]        # fall back to *
    parts = urlsplit(url)
    path = norm((parts.path or "/") + ("?" + parts.query if parts.query else ""))
    best_len, verdict = -1, True
    for allow, pattern in rules:
        if pattern and rule_regex(pattern).match(path):
            n = len(pattern)
            if n > best_len or (n == best_len and allow):          # longest wins, allow wins ties
                best_len, verdict = n, allow
    return verdict
```

Usage: `allowed(open("robots.txt", "rb").read(), "GPTBot", "https://example.com/docs/")`. Pass the bare product token (`GPTBot`), not the full user-agent header. Its limits: it measures rule length on the encoded pattern, it does not implement Google's special cases for `AdsBot-Google`, and it keeps `/robots.txt` subject to the rules, where RFC 9309 says that path is always allowed. Add that one line if you need it.

## Fix your own file: the mistakes behind every split row

| Mistake | What checkers do | Fix |
|---|---|---|
| `User-agent: GPTBot/1.1` (version pasted from bot docs) | 3 of 5 ignore the group | `User-agent: GPTBot` |
| File saved with a BOM | 3 of 5 drop the first group | Save as UTF-8 without BOM, or start with a comment line |
| Named AI groups at the end of a huge file | Google never reads them | Put named groups first, merge per-URL rules |
| `Disallow: /` above `Allow: /path/` | Python 3.12 and older ignore the Allow | Put Allow lines first, or rely on longest match only where you know the parser |
| Two separate GPTBot groups | Python 3.12 and older read only the first | Merge them into one group |

On WordPress, robots.txt is often generated by a plugin rather than edited by hand, so the BOM and the oversized file usually come from the generator. [ThinkRank](https://thinkrank.ai) edits robots.txt and robots meta from one screen instead of two plugins competing to write the file, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Yoast and Rank Math also have file editors; whichever you use, check the saved bytes, not the preview. For a ready-made set of AI crawler groups, start from the [robots.txt starter for AI crawlers](https://aiscan.site/blog/robots-txt-starter-for-ai-crawlers).

## What this post does not measure

A parser verdict is not a crawler's behaviour. We tested libraries, not GPTBot, ClaudeBot or Googlebot. The only evidence of what a bot actually fetched is in your access logs, and [finding AI crawlers in server logs](https://aiscan.site/blog/find-ai-crawlers-in-server-logs) shows how to read them. We did not run Google's C++ matcher (see the method section). We did not count how many live sites have each mistake: AIScan scans store check results, not robots.txt bodies, and we crawled no one else's files to build a count. The 15 files are minimal on purpose. Real files combine several of these cases, and combinations can fail in ways a single-case file does not show.

## Your next step

Run `npx aiscan-cli yoursite.com`, or paste your address at [aiscan.site](https://aiscan.site/), free and without an account. Read **B2** for your named AI crawler groups and **D1** for whether the file is served and readable. If two checkers still disagree about your file, look for a version number in a `User-agent` line, a BOM, or a file over 500 KiB, in that order. Every other bot access and discoverability fix is linked from [/guides](https://aiscan.site/guides).