Why robots.txt Checkers Disagree About GPTBot: 5 Parsers, 15 Files, Tested (2026)

Robots.txt checkers disagree about GPTBot because each one is built on a parser library, and the libraries do not agree on the rules. We ran 15 small robots.txt files through five parsers that checker tools are commonly built on. On 21 test cases all five matched RFC 9309 and Google's documented behaviour on 18. The three they missed are the ones that decide real verdicts: a version number in the User-agent line, a byte order mark at the start of the file, and a file over 500 KiB. Python's built-in parser also changed its answer on five cases between Python 3.12 and 3.13.

Below are the files, the raw verdicts, the RFC 9309 rule behind each row, and a short spec-aware checker you can copy. It matches the reference column on all 21 cases.

Quick summary

Short answer for a site owner: if two checkers disagree about your file, the file almost certainly has one of the three problems above, and each has a one-line fix. Run npx aiscan-cli yoursite.com (or paste your address at aiscan.site, free, no account) and read check B2 (explicit AI bot rules) and D1 (robots.txt served and readable) on the bot access and discoverability pages. Then fix the file with the table at the end of this post.

Short answer for someone building a checker: behave like Google's open-source matcher. Read the product token up to the first character that is not a letter, - or _. Strip a leading BOM. Stop parsing at 500 KiB. Merge every group that names the agent. Pick the longest matching rule, and let Allow win a tie.

Why this matters to anyone who checks AI crawler access

People ask one question about their robots.txt more than any other: is GPTBot (or ClaudeBot, or PerplexityBot) allowed? In the last month three public threads show people building tools to answer it: a resolver run over 122 well-known sites for 19 AI crawlers, a checker for 14 AI crawlers, and a robots.txt tester looking for feedback. Each sits on a parser. When the parser reads a file differently from the crawler, the tool gives a confident wrong answer.

The rules themselves are settled. RFC 9309 (September 2022) standardised the Robots Exclusion Protocol. Google's robots.txt specification documents how Google applies it, including choices the RFC leaves to the crawler, such as the 500 KiB limit. The gaps are in the libraries.

This post owns one question: how parsers match agents and paths. Three neighbouring questions have their own posts. What Python does with a 403 or 5xx response is covered in Python's robots.txt parser treats a 403 as disallow-all. The new AI-training policy token is covered in the __usage_ai-training token, tested. A leftover Disallow: / is covered in your robots.txt blocks everything by accident.

How we tested

Everything ran locally on 12 Nov 2026. We fetched nothing from anyone else's site, so every file is one we wrote, each isolating one edge case.

Method. Each file was saved as UTF-8 bytes and decoded the way a plain fetcher does (decode("utf-8"), no BOM stripping). Each parser then got the same text, the same agent product token and the same URL on https://example.com. That gave 21 cases (file × agent × path). The reference column applies RFC 9309 and, where the RFC leaves a choice, Google's specification page.

What we could not run. We meant to build Google's open-source matcher (the C++ google/robotstxt library) as a sixth column. The sandbox for this run does not allow compiling and running outside code, so we dropped it. We read its source for the three cases where parsers split, and we quote what it does below, but we did not run it.

The 15 files

The results, case by case

A ✗ marks a verdict that differs from the reference column.

Tally of departures from the reference, out of 21:

The good news comes first. Every parser handled case, CRLF, comments, stacked agents, empty Disallow:, the Sitemap: line inside a group, the non-ASCII path in both encodings, and the named-group-beats-* rule. If your checker is wrong, it is wrong on one of the three cases below.

Disagreement 1: a version number in the User-agent line

File 01 says User-agent: GPTBot/1.1 with Disallow: /, and a * group allows everything. Google's specification says "All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot)". Google's matcher implements this in ExtractUserAgent, which keeps only letters, - and _ from the start of the value. So for Google-style matching, GPTBot is blocked.

robotspy and robots-parser agreed and blocked GPTBot. urllib.robotparser, Protego and robotstxt treated GPTBot/1.1 as a different name, fell through to *, and said GPTBot was allowed.

RFC 9309 §2.2.1 says the product token in a user-agent line may contain only letters, underscores and hyphens, so GPTBot/1.1 is not a valid line. The fix belongs in the file: write User-agent: GPTBot. Bot documentation pages give full user-agent strings such as GPTBot/1.1, and people paste them in. The AI crawler user-agent list gives the bare token for each crawler, which is what goes in robots.txt.

Disagreement 2: a byte order mark before the first line

File 09 is two lines, User-agent: GPTBot and Disallow: /, saved with a UTF-8 byte order mark (the bytes EF BB BF). Many Windows editors add one silently. Google's specification says Google "ignores invalid lines in robots.txt files, including the Unicode Byte Order Mark (BOM) at the beginning", and its matcher skips the BOM byte by byte. For Google, GPTBot is blocked.

robotstxt and robots-parser coped. urllib.robotparser, Protego and robotspy read the first line as a field named user-agent, did not recognise it, dropped the whole group, and reported GPTBot as allowed.

So a site that blocks GPTBot in its first group can see a checker built on Scrapy's parser say the opposite. The fix is to save robots.txt as UTF-8 without a BOM. If the first group must stay first, a comment line at the top also works: a stray BOM then spoils only the comment.

Disagreement 3: everything after 500 KiB

File 13 is 630,054 bytes. It opens with User-agent: * / Allow: /, runs 9,000 comment lines, and ends with User-agent: GPTBot / Disallow: /. RFC 9309 §2.5 says "The parsing limit MUST be at least 500 kibibytes", so a crawler may stop there. Google says "Content which is after the maximum file size is ignored." For Google, the GPTBot group past byte 512,000 does not exist, and GPTBot falls back to Allow: /.

All five parsers read the whole file and blocked GPTBot. That is the only row where every library agreed with each other and not with Google. Other crawlers may set a higher limit (the RFC sets only a minimum), so this row has no single right answer for every bot. Within Google's documented behaviour, the checkers are wrong.

Files this large are rare and usually generated: one rule per URL, or a plugin appending to the file on every save. The fix is to put named AI crawler groups near the top and merge per-URL rules into directory rules. Our D1 check confirms the file is served and readable. It does not judge its size.

The surprises in the rows every parser got right

Eighteen rows agreed, but several of them surprise people who write robots.txt by hand. They are worth knowing before you edit a live file.

A blank line does not end a group (file 08). Many people separate User-agent: GPTBot from User-agent: * with a blank line and expect GPTBot to get a group with no rules, which would mean "allowed everywhere". The RFC 9309 grammar allows empty lines between user-agent lines, so the two lines form one group and GPTBot inherits Disallow: /. All five parsers blocked GPTBot, and so does Google's matcher, which ignores blank lines. If you want GPTBot allowed, give it a rule: Allow: /.

Two groups for the same bot are one group (file 07). RFC 9309 §2.2.1 says matching groups "MUST be combined into one group". A GPTBot group at the top and another at the bottom both apply, so /b/ stays blocked. Files grow this way when a plugin appends its own group and nobody merges them. Every current parser merged them; only Python 3.12 and older did not.

A named group replaces * and does not add to it (file 14). User-agent: GPTBot with Allow: / frees GPTBot from a site-wide Disallow: /, and Googlebot stays blocked. The reverse applies too: if you give GPTBot a group to block one folder, it no longer inherits anything from *, including your other Disallow lines. Google's spec says it directly: "User agent specific groups and global groups (*) are not combined."

An empty Disallow: allows everything (file 06). It reads like "disallow", but an empty value matches no path. RFC 9309 §2.2.2 says that when no rule matches, "the URI is allowed".

Allow wins a tie (file 03). When Allow: /page and Disallow: /page are the same length, RFC 9309 says the allow rule "SHOULD be used", and Google says it "uses the least restrictive rule". All five current parsers agreed. Python 3.12 did not.

$ only anchors the end of the path (file 05). Disallow: /*.pdf$ blocks /files/report.pdf and leaves /files/report.pdf?v=2 allowed, because the query string is part of what gets matched. If you mean every PDF, drop the $.

Python changed its answer between 3.12 and 3.13

Yesterday's training-token lab found that Python's parser takes the first matching rule, so Disallow: / above Allow: /press-kit/ wins. That held on Python 3.11.18 and 3.12.3. On 3.13.18 the same module picks the longest match, lets Allow win ties, and understands * and $. The source shows it directly: RuleLine.applies_to now returns the match length, and Entry.allowance keeps the best one.

The older builds also used only the first GPTBot group in file 07 and ignored the second. In practice, a checker on Python's standard library gives different answers depending on the server's Python version. Pin the version, or use a library that states its matching rules.

Which parser should a checker behave like?

The three fixes are small. Most of the remaining risk is in how a checker fetches the file (status codes, redirects, content type) rather than how it parses it.

A spec-aware checker in about 50 lines

This checker applies every rule in the reference column. On our 21 cases it matched the reference 21 times. It uses only the Python standard library and does no fetching: give it the bytes you got from /robots.txt. Status handling stays your job, and RFC 9309 §2.3.1 says what to do with 4xx and 5xx responses.

import re
from urllib.parse import quote, unquote, urlsplit

LIMIT = 500 * 1024  # RFC 9309 section 2.5: parse at least 500 KiB; Google stops there

def norm(path):
    # compare percent-encoded UTF-8, keep reserved characters as written
    return quote(unquote(path), safe="/?=&*$%:@!+,;~")

def groups(raw: bytes):
    text = raw[:LIMIT].decode("utf-8", "ignore").lstrip("")
    out, agents, rules_seen = [], [], False
    for line in re.split(r"\r\n|\r|\n", text):
        line = line.split("#", 1)[0].strip()
        if ":" not in line:
            continue                      # blank and junk lines never end a group
        key, val = (s.strip() for s in line.split(":", 1))
        key = key.lower()
        if key == "user-agent":
            if rules_seen:
                agents, rules_seen = [], False
            if not agents:
                out.append((agents, []))
            agents.append("*" if val.startswith("*") else re.match(r"[A-Za-z_-]*", val).group().lower())
        elif key in ("allow", "disallow") and out:
            rules_seen = True
            out[-1][1].append((key == "allow", norm(val)))
    return out

def rule_regex(pattern):
    end = pattern.endswith("$")
    body = re.escape(pattern.rstrip("$")).replace(r"\*", ".*")
    return re.compile(body + ("$" if end else ""))

def allowed(raw: bytes, agent: str, url: str) -> bool:
    agent = re.match(r"[A-Za-z_-]*", agent).group().lower()
    gs = groups(raw)
    rules = [r for a, rs in gs if agent in a for r in rs]          # merge every named group
    if not any(agent in a for a, _ in gs):
        rules = [r for a, rs in gs if "*" in a for r in rs]        # fall back to *
    parts = urlsplit(url)
    path = norm((parts.path or "/") + ("?" + parts.query if parts.query else ""))
    best_len, verdict = -1, True
    for allow, pattern in rules:
        if pattern and rule_regex(pattern).match(path):
            n = len(pattern)
            if n > best_len or (n == best_len and allow):          # longest wins, allow wins ties
                best_len, verdict = n, allow
    return verdict

Usage: allowed(open("robots.txt", "rb").read(), "GPTBot", "https://example.com/docs/"). Pass the bare product token (GPTBot), not the full user-agent header. Its limits: it measures rule length on the encoded pattern, it does not implement Google's special cases for AdsBot-Google, and it keeps /robots.txt subject to the rules, where RFC 9309 says that path is always allowed. Add that one line if you need it.

Fix your own file: the mistakes behind every split row

On WordPress, robots.txt is often generated by a plugin rather than edited by hand, so the BOM and the oversized file usually come from the generator. ThinkRank edits robots.txt and robots meta from one screen instead of two plugins competing to write the file, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Yoast and Rank Math also have file editors; whichever you use, check the saved bytes, not the preview. For a ready-made set of AI crawler groups, start from the robots.txt starter for AI crawlers.

What this post does not measure

A parser verdict is not a crawler's behaviour. We tested libraries, not GPTBot, ClaudeBot or Googlebot. The only evidence of what a bot actually fetched is in your access logs, and finding AI crawlers in server logs shows how to read them. We did not run Google's C++ matcher (see the method section). We did not count how many live sites have each mistake: AIScan scans store check results, not robots.txt bodies, and we crawled no one else's files to build a count. The 15 files are minimal on purpose. Real files combine several of these cases, and combinations can fail in ways a single-case file does not show.

Your next step

Run npx aiscan-cli yoursite.com, or paste your address at aiscan.site, free and without an account. Read B2 for your named AI crawler groups and D1 for whether the file is served and readable. If two checkers still disagree about your file, look for a version number in a User-agent line, a BOM, or a file over 500 KiB, in that order. Every other bot access and discoverability fix is linked from /guides.