ChatGPT Says It Can't Access Your Website: Five Causes and How to Tell Which

When ChatGPT says it cannot open your page, one of five things happened, and a short script of four curl requests sorts four of them in about a minute. The fifth is that ChatGPT never tried. This guide goes through the five in the order they are cheapest to rule out.

Quick summary

Direct answer. OpenAI documents three user agents that matter here: OAI-SearchBot for search results, GPTBot for training, and ChatGPT-User for a fetch a person triggers. Blocking GPTBot does not touch the other two. OpenAI also says robots.txt rules "may not apply" to ChatGPT-User, so the edge is the layer that decides.

To run the same checks with a scan instead, use npx aiscan-cli yoursite.com. It reports check:B2 (AI bot rules), check:D1 (robots.txt) and check:E3 (server-rendered text and headings), and marks a probe the firewall refused. The by-hand route is below and needs only curl.

Run the four requests first

Save this as triage.sh and pass the page ChatGPT could not open. It sends a browser identity and a ChatGPT-User identity, which follows the string format on OpenAI's page. A forged string from your laptop will not match OpenAI's published IP ranges, so a rule that checks the source address can still treat the real fetcher differently. Read a pass as "no user-agent rule found", not as proof.

#!/usr/bin/env bash
URL="$1"; ORIGIN=$(echo "$URL" | sed -E 's#(https?://[^/]+).*#\1#')
UA_B='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'
UA_U='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot'
echo "== 1 robots.txt: groups that name an OpenAI token"
curl -s -m 20 -A "$UA_B" "$ORIGIN/robots.txt" | grep -iE -A2 '^user-agent: *(gptbot|chatgpt-user|oai-searchbot)'
echo "== 2 edge: status per identity"
for N in B U; do eval UA=\$UA_$N
  curl -s -m 20 -o /dev/null -A "$UA" -w "$N %{http_code} %{content_type} %{size_download}b\n" "$URL"; done
curl -s -m 20 -o /dev/null -D - -A "$UA_U" "$URL" | grep -iE '^(server|cf-mitigated)'
echo "== 3 rendering: headings and text in the raw HTML"
curl -s -m 20 -A "$UA_U" "$URL" > /tmp/t.html
echo "h1: $(grep -oi '<h1' /tmp/t.html | wc -l)  words: $(sed -E 's/<script[^>]*>.*<\/script>//g; s/<[^>]+>/ /g' /tmp/t.html | wc -w)"
echo "== 4 gate: redirects"
curl -s -m 20 -L -o /dev/null -A "$UA_U" -w "final %{url_effective} after %{num_redirects} redirects, %{http_code}\n" "$URL"

We ran it on 31 October 2026 against four local test servers we wrote, each built to show one cause. The output is below, trimmed to the lines that differ. These are lab results, not a survey of real sites.

Now read your own output against each cause.

Cause 1: robots.txt names an OpenAI token

Step 1 prints every group that names GPTBot, ChatGPT-User or OAI-SearchBot. The mistake we see in the data is assuming one group covers all three. In our census of 451 robots.txt files, 137 (30.4%) name ChatGPT-User and 113 (25.1%) name OAI-SearchBot. Of the 22 files that fully block GPTBot, 9 leave ChatGPT-User open, which is a deliberate split only if the owner wrote it that way.

Group matching follows RFC 9309: a crawler obeys the group that names it, and falls back to * only when none does. We checked with Python's robotparser on 31 October 2026:

The second row is the common accident: allowing the training bot by name while a blanket block shuts out the other two. OpenAI says a robots.txt change can take about 24 hours to reach its search systems, so wait before retesting. On WordPress, ThinkRank edits robots.txt from one plugin instead of three plugins fighting over the file, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress. The blanket-block guide covers the lone Disallow: / case.

Cause 2: a CDN or firewall refuses the request

In step 2, a 403 or 429 for the bot line next to a 200 for the browser line is a user-agent rule. The headers line names the layer, for example Server: cloudflare. Cloudflare's AI Crawl Control setup has the per-crawler steps; we do not repeat them here. WordPress security plugins and host-level firewalls can do the same, and their rule names differ by product, so look in the product's own log for the blocked request.

Two limits matter. All three lines returning 403 points to an IP range or site-wide challenge, so repeat the test from another network. And a refusal can hide inside your own scan: in our 712-site study, 85 sites answered with a 403, 429 or 5xx, and their E3 pass rate was 10.6% against 79.4% for the rest. The score measured the firewall.

Cause 3: the page is an empty shell

If step 2 shows 200 for both identities but step 3 counts zero headings and a handful of words, the content arrives by JavaScript. OpenAI's page does not say whether its fetchers run scripts, so we make no claim either way. Your raw HTML is what you can control. See the server-rendering guide for the fixes by framework.

Cause 4: a login, cookie wall or geo redirect

Step 4 prints where the request ends. A final URL such as /login, a consent page or a regional domain means the fetcher saw that page, not yours. Fix it by serving public pages without a session, or by exempting the public path from the consent redirect. Do not exempt the whole site.

Cause 5: ChatGPT did not fetch the page

If all four steps pass, check your server logs for ChatGPT-User around the time of the chat. No entry means no request reached you, so nothing on your side blocked it. Paste the full URL again, one page at a time. We do not claim how ChatGPT decides to browse beyond what OpenAI documents.

Verify the fix

Re-run the script. Success is a 200 for both identities, the same size within a few percent, one <h1> and a final URL equal to the one you gave. Then run npx aiscan-cli yoursite.com and look at check:B2, check:D1 and check:E3. The by-hand version of that check is steps 1 and 3 above.

Maintenance

Re-run the script after any change to your CDN rules, robots.txt, security plugin or framework upgrade. Vendors change user-agent version numbers, so copy the current string from OpenAI's page if yours looks stale.

Your next step

Run the script now, then scan the site with npx aiscan-cli yoursite.com to see check:B2, check:D1 and check:E3 together. Our curl agent guide shows the comparison with more identities, and the bot access checks explain each result. More fixes are in all guides. For the OpenAI tokens by name, see the user-agent list.

If robots.txt allows the crawler but a WordPress firewall or security plugin still returns 403, see a WordPress security plugin is blocking GPTBot, ClaudeBot and PerplexityBot.