Fix: A WordPress Security Plugin Is Blocking GPTBot, ClaudeBot and PerplexityBot

If your robots.txt allows GPTBot, ClaudeBot and PerplexityBot but your logs show a wall of 403 for those same names, robots.txt is not the problem. A rule that runs before WordPress is. This guide shows how to confirm that with one curl per crawler, find which layer owns the rule, and fix it without opening the site to every bot.

In a local test on 2 Nov 2026, an nginx server with a permissive robots.txt returned 403 to all three crawler names and 200 to a Chrome user agent, on every URL including /robots.txt itself.

Quick summary

Run the scan first: npx aiscan-cli yoursite.com reports check B2 (bot access) and D1. The curl commands below are the by-hand version.

Why robots.txt cannot help here

robots.txt is a request. It only matters if the crawler can fetch it and chooses to obey it. A firewall rule is enforcement: it answers 403 before PHP starts, so WordPress, your SEO plugin and your robots.txt never see the request.

Think of a building with a security desk in the lobby. robots.txt is the sign on your office door saying visitors are welcome. The desk decides who reaches the door.

Three layers can sit in that position on a WordPress site:

• A CDN or WAF in front of the host. Cloudflare, Sucuri and similar services filter by user agent, IP and bot score.
• The web server. nginx or Apache rules, often added by the host or by a plugin that writes to .htaccess.
• A security plugin. It runs inside WordPress, so it sees the request after the web server. It can still return 403 before your theme loads.

Step 1: Reproduce the block with one request per crawler

Send the same request with a browser user agent and with each crawler name. Replace the domain.

for ua in "Mozilla/5.0 Chrome/130 Safari/537.36" \
          "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
          "Mozilla/5.0 (compatible; ClaudeBot/1.0)" \
          "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"; do
  printf '%s  ' "$(echo "$ua" | grep -o 'GPTBot\|ClaudeBot\|PerplexityBot\|Chrome')"
  curl -s -o /dev/null -w '%{http_code}\n' -A "$ua" https://yoursite.com/
done

OpenAI's crawler documentation publishes the GPTBot user-agent format, so you are testing the string real requests carry.

Expected output when a rule is blocking:

Chrome        200
GPTBot        403
ClaudeBot     403
PerplexityBot 403

If every row is 200, the user agent is not the trigger. Move to the IP or rate-limit cause in Step 3. If only some crawlers return 403, the rule names those crawlers specifically.

Limit of this test: it fakes the user agent from your own IP. A rule that checks the crawler's IP address, or a bot score, can still block the real crawler while letting your curl through. So a clean result here is not a clearance.

Step 2: Find which layer returned the 403

Look at the response, not the status code.

curl -s -D - -o /dev/null -A "GPTBot/1.4" https://yoursite.com/ | head -15

The order matters. Disable layers from the outside in: CDN first, then the host, then plugins. Changing a plugin setting while a CDN rule is the real cause wastes an afternoon.

For a second source, read your server access log by user agent. Find AI crawlers in your server logs shows the exact grep. In the lab, the log line for a blocked request looked like this:

127.0.0.1 "GET /robots.txt HTTP/1.1" 403 "GPTBot/1.2"

Step 3: Fix it at the layer that owns the rule

Do not search for a plugin setting from memory. Names and menu paths change between releases and between plugins. Open the plugin's own documentation, search for "user agent", "bot" or "blocked requests", and look for a block list that names crawlers, or a bot-protection mode that blocks unknown bots.

If the rule is in nginx or Apache. Remove the crawler names from the list. In the lab, the rule was an nginx map on $http_user_agent. After deleting GPTBot|ClaudeBot|PerplexityBot from the pattern and reloading, all three returned 200:

GPTBot        200
ClaudeBot     200
PerplexityBot 200
CCBot         403

CCBot stayed blocked, which is the point: you remove the crawlers you want and keep the ones you do not.

If the rule is in Cloudflare. Add an allow rule for the crawler that matches the user agent and its published IP ranges. Perplexity's documentation gives this pattern for Cloudflare: match the user agent together with the crawler's IP list, and set the action to Skip. OpenAI publishes its ranges as JSON files, for example openai.com/gptbot.json. A user agent alone is trivial to fake, so match both. See our Cloudflare bot-block guide for the Googlebot side effects.

If the rule is in a security plugin. Switch the bot or user-agent protection from "block" to "allow" for the crawlers you want, or add them to the plugin's allow list. If the plugin offers no per-bot control, the plugin's rate limiter or country block is usually the cause, and the fix is an IP allowlist.

Step 4: Keep fake crawlers out

Allowing a name is not the same as allowing a crawler. Anyone can send GPTBot in a header. Allow by user agent and IP range, and keep blocking the rest. Block fake AI crawlers and secret probes covers verification.

Verify the fix

Lead with the scan: npx aiscan-cli yoursite.com. Check B2 should pass. By hand, rerun the Step 1 loop. Expected output:

Chrome        200
GPTBot        200
ClaudeBot     200
PerplexityBot 200

Then confirm with real traffic: within a few days, your access log should show these names with 200, not 403. For the wider list of reasons ChatGPT cannot open a page, see ChatGPT says it can't access my website. The check is documented at /docs/checks/bot-access and every related guide is on /guides.

Maintenance

• Re-run the loop after every security plugin update. A plugin update can change what it blocks, so test again instead of assuming.
• Re-read the IP files monthly if you allow by IP; the published ranges change.
• Keep the robots.txt side tidy as well. On WordPress, ThinkRank edits robots.txt in one place, so a second plugin is not rewriting the same file.

Who does not need this

If every row in Step 1 already returns 200 and your logs show crawlers with 200, your block is somewhere else: an empty JavaScript shell, a login wall, or no fetch at all. Start with the ChatGPT guide above instead.