---
title: "ChatGPT Says It Can't Access Your Website: Five Causes and How to Tell Which"
slug: chatgpt-cant-access-my-website
published: 2026-10-31T13:11:26.742643+00:00
updated: 2026-11-02T13:34:17.572068+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:B2, check:D1, check:E3, platform:cloudflare, platform:wordpress, ChatGPT, robots.txt, AI crawlers, troubleshooting
description: "Five causes of ChatGPT not opening your site: robots.txt, firewall, empty JavaScript HTML, login walls, or no fetch. Run four curl checks, then verify."
url: https://aiscan.site/blog/chatgpt-cant-access-my-website
---

When ChatGPT says it cannot open your page, one of five things happened, and a short script of four `curl` requests sorts four of them in about a minute. The fifth is that ChatGPT never tried. This guide goes through the five in the order they are cheapest to rule out.

## Quick summary

| # | Cause | Signal in your own request | Where the fix lives |
|---|---|---|---|
| 1 | robots.txt names an OpenAI token | A `ChatGPT-User` or `OAI-SearchBot` group with `Disallow` | [robots.txt guide](https://aiscan.site/blog/robots-txt-starter-for-ai-crawlers) |
| 2 | CDN or firewall refuses the request | Bot identity gets 403 or 429, browser gets 200 | [Cloudflare setup](https://aiscan.site/blog/cloudflare-ai-crawl-control-setup) |
| 3 | HTML is an empty shell | 200, but 0 `<h1>` and almost no words | [rendering guide](https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers) |
| 4 | A login, cookie wall or geo redirect sits in front | 302 to another URL | Your auth or consent setup |
| 5 | The model never fetched the page | Nothing in your server logs | Retry with the exact URL |

**Direct answer.** [OpenAI documents three user agents](https://developers.openai.com/api/docs/bots) that matter here: `OAI-SearchBot` for search results, `GPTBot` for training, and `ChatGPT-User` for a fetch a person triggers. Blocking `GPTBot` does not touch the other two. OpenAI also says robots.txt rules "may not apply" to `ChatGPT-User`, so the edge is the layer that decides.

To run the same checks with a scan instead, use `npx aiscan-cli yoursite.com`. It reports `check:B2` (AI bot rules), `check:D1` (robots.txt) and `check:E3` (server-rendered text and headings), and marks a probe the firewall refused. The by-hand route is below and needs only `curl`.

## Run the four requests first

Save this as `triage.sh` and pass the page ChatGPT could not open. It sends a browser identity and a `ChatGPT-User` identity, which follows the string format on OpenAI's page. A forged string from your laptop will not match OpenAI's published IP ranges, so a rule that checks the source address can still treat the real fetcher differently. Read a pass as "no user-agent rule found", not as proof.

```bash
#!/usr/bin/env bash
URL="$1"; ORIGIN=$(echo "$URL" | sed -E 's#(https?://[^/]+).*#\1#')
UA_B='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'
UA_U='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot'
echo "== 1 robots.txt: groups that name an OpenAI token"
curl -s -m 20 -A "$UA_B" "$ORIGIN/robots.txt" | grep -iE -A2 '^user-agent: *(gptbot|chatgpt-user|oai-searchbot)'
echo "== 2 edge: status per identity"
for N in B U; do eval UA=\$UA_$N
  curl -s -m 20 -o /dev/null -A "$UA" -w "$N %{http_code} %{content_type} %{size_download}b\n" "$URL"; done
curl -s -m 20 -o /dev/null -D - -A "$UA_U" "$URL" | grep -iE '^(server|cf-mitigated)'
echo "== 3 rendering: headings and text in the raw HTML"
curl -s -m 20 -A "$UA_U" "$URL" > /tmp/t.html
echo "h1: $(grep -oi '<h1' /tmp/t.html | wc -l)  words: $(sed -E 's/<script[^>]*>.*<\/script>//g; s/<[^>]+>/ /g' /tmp/t.html | wc -w)"
echo "== 4 gate: redirects"
curl -s -m 20 -L -o /dev/null -A "$UA_U" -w "final %{url_effective} after %{num_redirects} redirects, %{http_code}\n" "$URL"
```

We ran it on 31 October 2026 against four local test servers we wrote, each built to show one cause. The output is below, trimmed to the lines that differ. These are lab results, not a survey of real sites.

| Test server | Step 2 (browser / bot) | Step 3 (h1, words) | Step 4 |
|---|---|---|---|
| Healthy page | `200` / `200` | 1 h1, 8 words | 0 redirects |
| Bot rule at the edge | `200` / `403`, `Server: cloudflare`, 78 bytes | 0 h1, 6 words | `403` |
| JavaScript shell | `200` / `200`, same size | 0 h1, 1 word | `200` |
| Login wall | `302` / `302`, 0 bytes | 0 h1, 0 words | ends at `/login` |

Now read your own output against each cause.

## Cause 1: robots.txt names an OpenAI token

Step 1 prints every group that names `GPTBot`, `ChatGPT-User` or `OAI-SearchBot`. The mistake we see in the data is assuming one group covers all three. In our [census of 451 robots.txt files](https://aiscan.site/blog/ai-crawler-declaration-census), 137 (30.4%) name `ChatGPT-User` and 113 (25.1%) name `OAI-SearchBot`. Of the 22 files that fully block `GPTBot`, 9 leave `ChatGPT-User` open, which is a deliberate split only if the owner wrote it that way.

Group matching follows [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309): a crawler obeys the group that names it, and falls back to `*` only when none does. We checked with Python's `robotparser` on 31 October 2026:

| robots.txt | GPTBot | OAI-SearchBot | ChatGPT-User |
|---|---|---|---|
| Only `GPTBot` blocked | blocked | allowed | allowed |
| `*` blocked, `GPTBot` allowed | allowed | blocked | blocked |
| `*` blocked, `OAI-SearchBot` allowed | blocked | allowed | blocked |

The second row is the common accident: allowing the training bot by name while a blanket block shuts out the other two. OpenAI says a robots.txt change can take about 24 hours to reach its search systems, so wait before retesting. On WordPress, [ThinkRank](https://thinkrank.ai) edits robots.txt from one plugin instead of three plugins fighting over the file, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress. The [blanket-block guide](https://aiscan.site/blog/robots-txt-disallow-all-by-accident) covers the lone `Disallow: /` case.

## Cause 2: a CDN or firewall refuses the request

In step 2, a `403` or `429` for the bot line next to a `200` for the browser line is a user-agent rule. The headers line names the layer, for example `Server: cloudflare`. [Cloudflare's AI Crawl Control setup](https://aiscan.site/blog/cloudflare-ai-crawl-control-setup) has the per-crawler steps; we do not repeat them here. WordPress security plugins and host-level firewalls can do the same, and their rule names differ by product, so look in the product's own log for the blocked request.

Two limits matter. All three lines returning `403` points to an IP range or site-wide challenge, so repeat the test from another network. And a refusal can hide inside your own scan: in our [712-site study](https://aiscan.site/blog/blocked-probe-share-of-failures), 85 sites answered with a 403, 429 or 5xx, and their E3 pass rate was 10.6% against 79.4% for the rest. The score measured the firewall.

## Cause 3: the page is an empty shell

If step 2 shows `200` for both identities but step 3 counts zero headings and a handful of words, the content arrives by JavaScript. OpenAI's page does not say whether its fetchers run scripts, so we make no claim either way. Your raw HTML is what you can control. See the [server-rendering guide](https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers) for the fixes by framework.

## Cause 4: a login, cookie wall or geo redirect

Step 4 prints where the request ends. A final URL such as `/login`, a consent page or a regional domain means the fetcher saw that page, not yours. Fix it by serving public pages without a session, or by exempting the public path from the consent redirect. Do not exempt the whole site.

## Cause 5: ChatGPT did not fetch the page

If all four steps pass, check your server logs for `ChatGPT-User` around the time of the chat. No entry means no request reached you, so nothing on your side blocked it. Paste the full URL again, one page at a time. We do not claim how ChatGPT decides to browse beyond what OpenAI documents.

## Verify the fix

Re-run the script. Success is a `200` for both identities, the same size within a few percent, one `<h1>` and a final URL equal to the one you gave. Then run `npx aiscan-cli yoursite.com` and look at `check:B2`, `check:D1` and `check:E3`. The by-hand version of that check is steps 1 and 3 above.

## Maintenance

Re-run the script after any change to your CDN rules, robots.txt, security plugin or framework upgrade. Vendors change user-agent version numbers, so copy the current string from [OpenAI's page](https://developers.openai.com/api/docs/bots) if yours looks stale.

## Your next step

Run the script now, then scan the site with `npx aiscan-cli yoursite.com` to see `check:B2`, `check:D1` and `check:E3` together. Our [curl agent guide](https://aiscan.site/blog/fetch-site-as-ai-agent-curl) shows the comparison with more identities, and the [bot access checks](https://aiscan.site/docs/checks/bot-access) explain each result. More fixes are in [all guides](https://aiscan.site/guides). For the OpenAI tokens by name, see the [user-agent list](https://aiscan.site/blog/ai-crawler-user-agent-list-2026).

If robots.txt allows the crawler but a WordPress firewall or security plugin still returns `403`, see [a WordPress security plugin is blocking GPTBot, ClaudeBot and PerplexityBot](https://aiscan.site/blog/wordpress-security-plugin-blocks-ai-crawlers).