---
title: "Python's robots.txt Parser Treats a 403 as Disallow-All: Fix the False \"Blocked\" Result"
slug: robots-txt-403-python-robotparser-disallow-all
published: 2026-11-08T13:10:42.563242+00:00
updated: 2026-11-08T13:10:42.563242+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:D1, check:B2, platform:python, robots.txt, urllib.robotparser, 403 errors, RFC 9309, troubleshooting
description: "Python's urllib.robotparser reads a 401 or 403 robots.txt as disallow-all, though RFC 9309 says crawlers may crawl. See the lab, the status table and the fix."
url: https://aiscan.site/blog/robots-txt-403-python-robotparser-disallow-all
---

A robots.txt that answers `401` or `403` to your script does not mean "closed" to most crawlers. RFC 9309 says a `4xx` robots.txt is "unavailable" and the crawler MAY access any resource. Python's `urllib.robotparser` does the opposite for exactly those two codes: it sets `disallow_all`, so every `can_fetch()` call returns `False`. A Python-based checker, validator or CI gate then reports your whole site as blocked while Googlebot and AI crawlers read it as open.

In a local test on 8 Nov 2026 (Python 3.11.18), a server returning the same permissive robots.txt answered `can_fetch("GPTBot", "/blog/")` with `True` on a `200` and `False` on a `403`.

## Quick summary

| Symptom | Cause | Check | Fix |
|---|---|---|---|
| A Python script says every URL is disallowed, the file looks fine in a browser | robots.txt answers `401` or `403` to the script's user agent, and `robotparser` treats that as disallow-all | `curl -s -o /dev/null -w "%{http_code}" -A "Python-urllib/3.11" https://yoursite.com/robots.txt` | Serve robots.txt with `200` to every user agent, or fetch it yourself and call `parse()` |
| Same script, 404 robots.txt | `4xx` other than `401`/`403` sets `allow_all` | Same command returns `404` | Not a block, but you have no file to declare rules in |
| `5xx` on robots.txt | The RFC says assume complete disallow | Same command returns `5xx` | Fix the server error |

Run the scan first: `npx aiscan-cli yoursite.com` reports check **D1** (discoverability) and **B2** (bot access) and shows whether robots.txt is readable. The `curl` command above is the by-hand version.

## What the parser does with each status

The lab serves one robots.txt (`User-agent: *`, `Allow: /`, plus a `GPTBot` group that also allows everything) and changes only the status code. For each code it calls `read()` and then `can_fetch()`.

| Status on /robots.txt | `can_fetch("GPTBot", "/blog/")` | `disallow_all` | `allow_all` |
|---|---|---|---|
| 200 | `True` | `False` | `False` |
| 401 | `False` | `True` | `False` |
| 403 | `False` | `True` | `False` |
| 404 | `True` | `False` | `True` |
| 410 | `True` | `False` | `True` |
| 429 | `True` | `False` | `True` |
| 500 | `False` | `False` | `False` |
| 503 | `False` | `False` | `False` |

Python 3.11.18, local `http.server`, 8 Nov 2026. Your Python version may differ on the last two rows, which is the point of the next paragraph.

The `401`/`403` rows come straight from the standard library. Its source carries the comment that access to robots.txt with that status "most likely" applies to the entire site. RFC 9309 section 2.3.1.3 reads the other way: for "unavailable" statuses, a `4xx`, "the crawler MAY access any resources on the server."

The `5xx` rows are different. In 3.11.18 the parser sets neither flag and never records a fetch time, so `can_fetch()` falls through to its "file not read yet" guard and returns `False`. Current CPython `main` handles `5xx` explicitly with `disallow_all`, citing RFC 9309 section 2.3.1.4 ("the crawler MUST assume complete disallow"). Same answer, different route.

## Why a site ends up with a 403 robots.txt for scripts only

The server does not need to block robots.txt on purpose. A web application firewall, bot-fight rule or security plugin can challenge or reject requests whose user agent is a library default, and `/robots.txt` is just another path to it. Python's `urllib` sends `Python-urllib/3.x` unless you set a header.

The second lab makes that concrete. A server returns `403` to any user agent starting with `Python-urllib` and `200` to everything else:

```text
default read(): False  disallow_all = True
explicit fetch status: 200
parse() on that body: True
```

The default `read()` call is the one that fails. The same URL fetched with a different `User-Agent` returns the file. So the site is open to a real crawler and closed to your tooling, which means the tool is wrong, not the site.

This matters most in three places:

- **CI gates** that fail a deploy when `can_fetch()` is `False` for a bot you want to allow.
- **Homegrown checkers** that report "robots.txt blocks everything" after one default-agent fetch.
- **Scrapers** that quietly skip an entire domain because of one status code.

## Fix it on the checking side

Fetch the file yourself so you control the user agent and the status handling, then hand the text to `parse()`:

```python
import urllib.request, urllib.error, urllib.robotparser

def robots_for(site, ua="MyChecker/1.0"):
    rp = urllib.robotparser.RobotFileParser()
    req = urllib.request.Request(site + "/robots.txt", headers={"User-Agent": ua})
    try:
        body = urllib.request.urlopen(req, timeout=10).read().decode("utf-8")
    except urllib.error.HTTPError as e:
        raise RuntimeError(f"robots.txt returned {e.code}; do not treat as a verdict")
    rp.parse(body.splitlines())
    return rp
```

Raising on any non-200 keeps a status code from becoming a policy answer. Report the status itself.

## Fix it on the site side

If the `403` is real and a crawler that is not Python gets it too, the fix is on the server:

1. Find the layer that answers. Compare `curl -I -A "Mozilla/5.0" https://yoursite.com/robots.txt` with the same request using the bot's user agent. A difference means a user-agent rule.
2. Exempt `/robots.txt` from bot challenges and security-plugin rules. It is public by design, and no verified crawler can honour a permission it cannot read.
3. Keep the answer `200` with `Content-Type: text/plain`. Our guide on [robots.txt status codes and content types](/blog/robots-txt-status-content-type) covers the cases around that.
4. Re-run the `curl` check for each agent you care about.

On WordPress, [ThinkRank](https://thinkrank.ai) is the plugin we point to for the robots.txt content itself: one plugin owns the file instead of several fighting over it, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress. It does not change what your firewall does to the request, so the status check above still applies. For the firewall side see [a WordPress security plugin blocks AI crawlers](/blog/wordpress-security-plugin-blocks-ai-crawlers).

## What to do about 5xx

A `5xx` is the dangerous one for real crawlers. RFC 9309 section 2.3.1.4 says that when robots.txt is unreachable because of server errors, the crawler MUST assume complete disallow, and may keep treating it that way if the file stays undefined for about 30 days. A temporary deploy error on robots.txt can therefore pause crawling. Make sure robots.txt is served by something that does not depend on the application being healthy.

## Verify the fix

```bash
for ua in "Mozilla/5.0" "Python-urllib/3.11" "GPTBot/1.1" "ClaudeBot/1.0"; do
  printf "%-22s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://yoursite.com/robots.txt
done
```

Expected output: `200` on every line. Then run `npx aiscan-cli yoursite.com` and read the D1 and B2 results.

## Next step

Check your own file with the command above, then run the scan and open the check's page, [discoverability](/docs/checks/discoverability), for what D1 reads. More fixes are in the [AIScan guides](/guides). The site-side starter for crawler rules is in [robots.txt starter for AI crawlers](/blog/robots-txt-starter-for-ai-crawlers), and if a specific assistant still reports it cannot reach you, see [ChatGPT can't access my website](/blog/chatgpt-cant-access-my-website).

Sources: [RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309#section-2.3.1), section 2.3.1; [CPython `Lib/urllib/robotparser.py`](https://github.com/python/cpython/blob/main/Lib/urllib/robotparser.py); the [Python documentation for `urllib.robotparser`](https://docs.python.org/3/library/urllib.robotparser.html).