Python's robots.txt Parser Treats a 403 as Disallow-All: Fix the False "Blocked" Result

A robots.txt that answers 401 or 403 to your script does not mean "closed" to most crawlers. RFC 9309 says a 4xx robots.txt is "unavailable" and the crawler MAY access any resource. Python's urllib.robotparser does the opposite for exactly those two codes: it sets disallow_all, so every can_fetch() call returns False. A Python-based checker, validator or CI gate then reports your whole site as blocked while Googlebot and AI crawlers read it as open.

In a local test on 8 Nov 2026 (Python 3.11.18), a server returning the same permissive robots.txt answered can_fetch("GPTBot", "/blog/") with True on a 200 and False on a 403.

Quick summary

Run the scan first: npx aiscan-cli yoursite.com reports check D1 (discoverability) and B2 (bot access) and shows whether robots.txt is readable. The curl command above is the by-hand version.

What the parser does with each status

The lab serves one robots.txt (User-agent: *, Allow: /, plus a GPTBot group that also allows everything) and changes only the status code. For each code it calls read() and then can_fetch().

Python 3.11.18, local http.server, 8 Nov 2026. Your Python version may differ on the last two rows, which is the point of the next paragraph.

The 401/403 rows come straight from the standard library. Its source carries the comment that access to robots.txt with that status "most likely" applies to the entire site. RFC 9309 section 2.3.1.3 reads the other way: for "unavailable" statuses, a 4xx, "the crawler MAY access any resources on the server."

The 5xx rows are different. In 3.11.18 the parser sets neither flag and never records a fetch time, so can_fetch() falls through to its "file not read yet" guard and returns False. Current CPython main handles 5xx explicitly with disallow_all, citing RFC 9309 section 2.3.1.4 ("the crawler MUST assume complete disallow"). Same answer, different route.

Why a site ends up with a 403 robots.txt for scripts only

The server does not need to block robots.txt on purpose. A web application firewall, bot-fight rule or security plugin can challenge or reject requests whose user agent is a library default, and /robots.txt is just another path to it. Python's urllib sends Python-urllib/3.x unless you set a header.

The second lab makes that concrete. A server returns 403 to any user agent starting with Python-urllib and 200 to everything else:

default read(): False  disallow_all = True
explicit fetch status: 200
parse() on that body: True

The default read() call is the one that fails. The same URL fetched with a different User-Agent returns the file. So the site is open to a real crawler and closed to your tooling, which means the tool is wrong, not the site.

This matters most in three places:

• CI gates that fail a deploy when can_fetch() is False for a bot you want to allow.
• Homegrown checkers that report "robots.txt blocks everything" after one default-agent fetch.
• Scrapers that quietly skip an entire domain because of one status code.

Fix it on the checking side

Fetch the file yourself so you control the user agent and the status handling, then hand the text to parse():

import urllib.request, urllib.error, urllib.robotparser

def robots_for(site, ua="MyChecker/1.0"):
    rp = urllib.robotparser.RobotFileParser()
    req = urllib.request.Request(site + "/robots.txt", headers={"User-Agent": ua})
    try:
        body = urllib.request.urlopen(req, timeout=10).read().decode("utf-8")
    except urllib.error.HTTPError as e:
        raise RuntimeError(f"robots.txt returned {e.code}; do not treat as a verdict")
    rp.parse(body.splitlines())
    return rp

Raising on any non-200 keeps a status code from becoming a policy answer. Report the status itself.

Fix it on the site side

If the 403 is real and a crawler that is not Python gets it too, the fix is on the server:

1. Find the layer that answers. Compare curl -I -A "Mozilla/5.0" https://yoursite.com/robots.txt with the same request using the bot's user agent. A difference means a user-agent rule.
2. Exempt /robots.txt from bot challenges and security-plugin rules. It is public by design, and no verified crawler can honour a permission it cannot read.
3. Keep the answer 200 with Content-Type: text/plain. Our guide on robots.txt status codes and content types covers the cases around that.
4. Re-run the curl check for each agent you care about.

On WordPress, ThinkRank is the plugin we point to for the robots.txt content itself: one plugin owns the file instead of several fighting over it, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress. It does not change what your firewall does to the request, so the status check above still applies. For the firewall side see a WordPress security plugin blocks AI crawlers.

What to do about 5xx

A 5xx is the dangerous one for real crawlers. RFC 9309 section 2.3.1.4 says that when robots.txt is unreachable because of server errors, the crawler MUST assume complete disallow, and may keep treating it that way if the file stays undefined for about 30 days. A temporary deploy error on robots.txt can therefore pause crawling. Make sure robots.txt is served by something that does not depend on the application being healthy.

Verify the fix

for ua in "Mozilla/5.0" "Python-urllib/3.11" "GPTBot/1.1" "ClaudeBot/1.0"; do
  printf "%-22s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://yoursite.com/robots.txt
done

Expected output: 200 on every line. Then run npx aiscan-cli yoursite.com and read the D1 and B2 results.

Next step

Check your own file with the command above, then run the scan and open the check's page, discoverability, for what D1 reads. More fixes are in the AIScan guides. The site-side starter for crawler rules is in robots.txt starter for AI crawlers, and if a specific assistant still reports it cannot reach you, see ChatGPT can't access my website.

Sources: RFC 9309, Robots Exclusion Protocol, section 2.3.1; CPython Lib/urllib/robotparser.py; the Python documentation for urllib.robotparser.