---
title: "Fix: A WordPress Security Plugin Is Blocking GPTBot, ClaudeBot and PerplexityBot"
slug: wordpress-security-plugin-blocks-ai-crawlers
published: 2026-11-02T13:31:20.203569+00:00
updated: 2026-11-02T13:31:20.203569+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:B2, check:D1, platform:wordpress, AI crawlers, 403 errors, WordPress security, GPTBot, troubleshooting
description: "robots.txt allows GPTBot, ClaudeBot and PerplexityBot but your logs show 403? Find the blocking firewall layer, test with curl, and allow crawlers safely."
url: https://aiscan.site/blog/wordpress-security-plugin-blocks-ai-crawlers
---

If your robots.txt allows GPTBot, ClaudeBot and PerplexityBot but your logs show a wall of `403` for those same names, robots.txt is not the problem. A rule that runs before WordPress is. This guide shows how to confirm that with one `curl` per crawler, find which layer owns the rule, and fix it without opening the site to every bot.

In a local test on 2 Nov 2026, an nginx server with a permissive robots.txt returned `403` to all three crawler names and `200` to a Chrome user agent, on every URL including `/robots.txt` itself.

## Quick summary

| Symptom | Cause | Check | Fix |
|---|---|---|---|
| robots.txt allows the bots, requests still get `403` | A user-agent rule in a firewall layer, before WordPress answers | `curl -I -A "GPTBot"` returns `403`, a browser UA returns `200` | Allow the crawler by user agent and published IP range at the layer that blocks it |
| `403` on `/robots.txt` too | Same rule, applied to every path | `curl -A "GPTBot" /robots.txt` | The crawler cannot even read your permission |
| Blocked only after many requests | A rate limit, not a user-agent rule | `429`, or `403` after a burst | Raise the limit for verified crawler IPs |

Run the scan first: `npx aiscan-cli yoursite.com` reports check **B2** (bot access) and **D1**. The `curl` commands below are the by-hand version.

## Why robots.txt cannot help here

robots.txt is a request. It only matters if the crawler can fetch it and chooses to obey it. A firewall rule is enforcement: it answers `403` before PHP starts, so WordPress, your SEO plugin and your robots.txt never see the request.

Think of a building with a security desk in the lobby. robots.txt is the sign on your office door saying visitors are welcome. The desk decides who reaches the door.

Three layers can sit in that position on a WordPress site:

- **A CDN or WAF in front of the host.** Cloudflare, Sucuri and similar services filter by user agent, IP and bot score.
- **The web server.** nginx or Apache rules, often added by the host or by a plugin that writes to `.htaccess`.
- **A security plugin.** It runs inside WordPress, so it sees the request after the web server. It can still return `403` before your theme loads.

## Step 1: Reproduce the block with one request per crawler

Send the same request with a browser user agent and with each crawler name. Replace the domain.

```bash
for ua in "Mozilla/5.0 Chrome/130 Safari/537.36" \
          "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
          "Mozilla/5.0 (compatible; ClaudeBot/1.0)" \
          "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"; do
  printf '%s  ' "$(echo "$ua" | grep -o 'GPTBot\|ClaudeBot\|PerplexityBot\|Chrome')"
  curl -s -o /dev/null -w '%{http_code}\n' -A "$ua" https://yoursite.com/
done
```

[OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) publishes the GPTBot user-agent format, so you are testing the string real requests carry.

Expected output when a rule is blocking:

```
Chrome        200
GPTBot        403
ClaudeBot     403
PerplexityBot 403
```

If every row is `200`, the user agent is not the trigger. Move to the IP or rate-limit cause in Step 3. If only some crawlers return `403`, the rule names those crawlers specifically.

**Limit of this test:** it fakes the user agent from your own IP. A rule that checks the crawler's IP address, or a bot score, can still block the real crawler while letting your `curl` through. So a clean result here is not a clearance.

## Step 2: Find which layer returned the 403

Look at the response, not the status code.

```bash
curl -s -D - -o /dev/null -A "GPTBot/1.4" https://yoursite.com/ | head -15
```

| What you see | Likely layer |
|---|---|
| `server: cloudflare` and a `cf-ray` header | The CDN or its WAF |
| `server: nginx` or `Apache` and a plain error page | The web server or host firewall |
| A branded block page from a security plugin, or a normal WordPress theme error | A plugin inside WordPress |
| The same status for `/robots.txt` | A global rule, not a path rule |

The order matters. Disable layers from the outside in: CDN first, then the host, then plugins. Changing a plugin setting while a CDN rule is the real cause wastes an afternoon.

For a second source, read your server access log by user agent. [Find AI crawlers in your server logs](https://aiscan.site/blog/find-ai-crawlers-in-server-logs) shows the exact grep. In the lab, the log line for a blocked request looked like this:

```
127.0.0.1 "GET /robots.txt HTTP/1.1" 403 "GPTBot/1.2"
```

## Step 3: Fix it at the layer that owns the rule

Do not search for a plugin setting from memory. Names and menu paths change between releases and between plugins. Open the plugin's own documentation, search for "user agent", "bot" or "blocked requests", and look for a block list that names crawlers, or a bot-protection mode that blocks unknown bots.

**If the rule is in nginx or Apache.** Remove the crawler names from the list. In the lab, the rule was an nginx [`map`](https://nginx.org/en/docs/http/ngx_http_map_module.html) on `$http_user_agent`. After deleting `GPTBot|ClaudeBot|PerplexityBot` from the pattern and reloading, all three returned `200`:

```
GPTBot        200
ClaudeBot     200
PerplexityBot 200
CCBot         403
```

`CCBot` stayed blocked, which is the point: you remove the crawlers you want and keep the ones you do not.

**If the rule is in Cloudflare.** Add an allow rule for the crawler that matches the user agent and its published IP ranges. [Perplexity's documentation](https://docs.perplexity.ai/guides/bots) gives this pattern for Cloudflare: match the user agent together with the crawler's IP list, and set the action to **Skip**. OpenAI publishes its ranges as JSON files, for example `openai.com/gptbot.json`. A user agent alone is trivial to fake, so match both. See [our Cloudflare bot-block guide](https://aiscan.site/blog/cloudflare-ai-bot-block-googlebot-2026) for the Googlebot side effects.

**If the rule is in a security plugin.** Switch the bot or user-agent protection from "block" to "allow" for the crawlers you want, or add them to the plugin's allow list. If the plugin offers no per-bot control, the plugin's rate limiter or country block is usually the cause, and the fix is an IP allowlist.

## Step 4: Keep fake crawlers out

Allowing a name is not the same as allowing a crawler. Anyone can send `GPTBot` in a header. Allow by user agent **and** IP range, and keep blocking the rest. [Block fake AI crawlers and secret probes](https://aiscan.site/blog/block-fake-ai-crawlers-secret-probes) covers verification.

## Verify the fix

Lead with the scan: `npx aiscan-cli yoursite.com`. Check **B2** should pass. By hand, rerun the Step 1 loop. Expected output:

```
Chrome        200
GPTBot        200
ClaudeBot     200
PerplexityBot 200
```

Then confirm with real traffic: within a few days, your access log should show these names with `200`, not `403`. For the wider list of reasons ChatGPT cannot open a page, see [ChatGPT says it can't access my website](https://aiscan.site/blog/chatgpt-cant-access-my-website). The check is documented at [/docs/checks/bot-access](https://aiscan.site/docs/checks/bot-access) and every related guide is on [/guides](https://aiscan.site/guides).

## Maintenance

- Re-run the loop after every security plugin update. A plugin update can change what it blocks, so test again instead of assuming.
- Re-read the IP files monthly if you allow by IP; the published ranges change.
- Keep the robots.txt side tidy as well. On WordPress, [ThinkRank](https://thinkrank.ai) edits robots.txt in one place, so a second plugin is not rewriting the same file.

## Who does not need this

If every row in Step 1 already returns `200` and your logs show crawlers with `200`, your block is somewhere else: an empty JavaScript shell, a login wall, or no fetch at all. Start with the ChatGPT guide above instead.