---
title: "Block Fake AI Crawlers From Probing .env and .git Files"
slug: block-fake-ai-crawlers-secret-probes
published: 2026-10-31T08:12:26.45521+00:00
updated: 2026-10-31T08:12:26.45521+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:B2, platform:nginx, platform:apache, platform:cloudflare, platform:wordpress, AI crawlers, user agent spoofing, security, AI readiness
description: "Scanners posing as GPTBot probe for .env and .git. Block secret paths on nginx, Apache or Cloudflare without breaking /.well-known/. Tested on real servers."
url: https://aiscan.site/blog/block-fake-ai-crawlers-secret-probes
---

Some of the traffic that calls itself GPTBot or ClaudeBot is not a crawler. It is a scanner that borrowed a famous name and asks your server for `/.env`, a `.git` folder or a `wp-config.php.bak`. The fix is one path rule per server layer, and in our local test it turned 6 of 6 secret-path requests into 404s while the home page, `robots.txt` and `/.well-known/` kept answering 200 for both a GPTBot user agent and a browser one.

The rule has to match the path. A user-agent rule cannot work, because the impostor and the real crawler send the same string. The second trap is the usual "deny every dotfile" snippet, which also takes down `/.well-known/`, where AI agents look for server cards and other discovery files.

## Quick summary

| If you run | Add this rule | Result in our test | Time |
|---|---|---|---|
| nginx | Two `location ~` blocks returning 404 | `.env`, `.git/config`, `.bak` 404; `.well-known` 200 | 5 min |
| Apache | Two `RedirectMatch 404` lines | Same as nginx | 5 min |
| Cloudflare | One custom rule, action Block, matching the path | Not run on a zone of ours; syntax from Cloudflare's docs | 10 min |
| WordPress on Apache | A plugin setting that writes the rules | Not installed here; behaviour from its docs | 3 min |

**Recommended order:** run `npx aiscan-cli yoursite.com` first, add the server rule that matches your stack, then scan again and confirm `/.well-known/` and `robots.txt` still pass.

## What the impostors are doing

[GreyNoise reported on 28 August 2026](https://www.greynoise.io/blog/threat-actors-posing-as-ai-crawlers) a cluster of scanners impersonating 13 AI crawlers from eight companies. They requested `.env` files, cloud access keys, private keys and password stores. Six of the names came from the same 824 addresses.

Our own logs show the same shape at small scale. In [63 days of server logs](/blog/fake-ai-crawler-traffic-own-logs), 1,297 of 2,107 requests (61.6%) came from 22 of 147 sources that each presented 10 to 15 different crawler names. Every one of those got a 404, because our site had nothing at those paths. That is why nothing worse happened.

The analogy is a visitor in a delivery uniform. The uniform proves nothing, so the guard checks which door the visitor is trying. A crawler reading your blog never needs the door marked "keys". Refuse that door for everyone, and the real crawler is unaffected.

## Why a user-agent rule fails

We ran a rule that returns 403 when the user agent contains `GPTBot`, and fetched six paths with a GPTBot string and a browser string.

| Rule | UA | Home page | `/robots.txt` | `/.env` | `/.git/config` |
|---|---|---|---|---|---|
| None | GPTBot | 200 | 200 | 200 | 200 |
| Block GPTBot by name | GPTBot | 403 | 403 | 403 | 403 |
| Block GPTBot by name | Browser | 200 | 200 | 200 | 200 |

The rule blocked the genuine crawler from your content and gave the impostor nothing to stop it: send a browser string instead and `/.env` answers 200. If you want to separate real from fake, [verify the source address](/blog/ai-crawler-user-agent-list-2026). OpenAI [publishes the address ranges](https://developers.openai.com/api/docs/bots) for GPTBot at `openai.com/gptbot.json`. That is a different job from protecting files.

## The test setup

nginx 1.24 and Apache 2.4.58 ran locally on 31 October 2026 with a document root holding six items: a real page, `robots.txt`, a decoy `.env`, a decoy `.git/config`, a decoy `wp-config.php.bak` and a decoy `/.well-known/mcp/server-card.json`. No real secret was involved. Each server ran three configurations: no rule, the "deny every dotfile" snippet, and the path rule below. Each was fetched with a GPTBot user agent and a Chrome user agent, 36 requests per server.

| Configuration | Real page | `.env` | `.git/config` | `.bak` | `.well-known` | `robots.txt` |
|---|---|---|---|---|---|---|
| No rule | 200 | 200 | 200 | 200 | 200 | 200 |
| Deny every dotfile | 200 | 404 | 404 | 200 | **404** | 200 |
| Path rule with carve-out | 200 | 404 | 404 | 404 | 200 | 200 |

nginx and Apache returned identical results, and the user agent made no difference in any row. The middle row is the trap: the common snippet hides `.env` and also hides the capability files that checks P1 to P4 look for. The [well-known folder guide](/blog/well-known-folder-404-fix) covers the other way that folder breaks.

## Path 1: nginx

Add these inside the `server { }` block, then run `nginx -t` and reload.

```nginx
location ~ /\.(?!well-known(/|$)) { return 404; }
location ~* \.(bak|old|orig|sql|swp)$ { return 404; }
```

1. Open the server config for your site, usually a file under `sites-enabled` or `conf.d`.
2. Paste both lines inside the `server { }` block, above any `location /` that proxies to your app.
3. Run `nginx -t`. It should print "syntax is ok".
4. Reload with `systemctl reload nginx`.

The first line matches any path segment starting with a dot, except `.well-known`. The lookahead is why `.well-known` survives. Regex locations in nginx are tried in file order, and a regex match beats a plain prefix like `location /`, so the rule applies even in front of a proxied app. See the [nginx `location` documentation](https://nginx.org/en/docs/http/ngx_http_core_module.html#location).

## Path 2: Apache

Put these in the virtual host or main config. We tested them there, not in `.htaccess`.

```apache
RedirectMatch 404 "/\.(?!well-known(/|$))"
RedirectMatch 404 "(?i)\.(bak|old|orig|sql|swp)$"
```

1. Confirm `mod_alias` is enabled (`apache2ctl -M | grep alias`).
2. Add both lines to the virtual host.
3. Run `apache2ctl configtest`, then reload.

`RedirectMatch` with status 404 sends a 404 and no redirect target, per the [mod_alias documentation](https://httpd.apache.org/docs/2.4/mod/mod_alias.html#redirectmatch). On shared hosting where you only have `.htaccess`, the same two lines usually work there if the host allows alias directives. We did not test that.

## Path 3: Cloudflare

Cloudflare can refuse the request before it reaches your server. We did not create this rule on a zone of ours, so the expression below comes from Cloudflare's [rules language reference](https://developers.cloudflare.com/ruleset-engine/rules-language/operators/), not from our test.

1. In the Cloudflare dashboard, open **Security rules**, then **Create rule > Custom rules**.
2. Name it "Refuse secret paths".
3. Switch to the expression editor and paste:

```
(http.request.uri.path contains "/." and not starts_with(http.request.uri.path, "/.well-known/"))
or ends_with(lower(http.request.uri.path), ".bak")
or ends_with(lower(http.request.uri.path), ".sql")
```

4. Under **Then take action**, pick **Block**. Cloudflare says that on plans without the Log action you should start with Managed Challenge, read **Security Events**, then switch to Block.
5. Select **Deploy**.

This uses `contains`, `starts_with`, `ends_with` and `lower`, so it needs no regular expression. The `matches` operator needs a Business or Enterprise plan, which is why we avoided it. If you also use Cloudflare's AI bot controls, [the crawl control guide](/blog/cloudflare-ai-crawl-control-setup) shows how to keep verified crawlers allowed while this rule runs on paths.

## Path 4: WordPress

If your site runs on Apache or LiteSpeed's paid edition, a plugin can write the rules for you. [BetterShield's "Hide private files from visitors"](https://bettershield.ai/docs/hide-sensitive-files/) answers requests for wp-config backups, logs, database dumps, `.git` and `.env` with a 404 through `.htaccess`. It is off until you turn it on, under **BetterShield > Protection > Hardening**. Its docs say it needs Apache or LiteSpeed's paid edition, so on nginx use Path 1. We did not install it for this post.

## Why 404 and not 403

All three server rules above answer 404. A 403 tells a scanner the file exists and is protected, which is a hint worth having if you are guessing at paths. A 404 gives the same answer whether the file is there or not. It also matches what a site with nothing at that path already says, so your logs stay uniform. Cloudflare's Block action returns its own block page instead, and we did not measure its status code. That is fine for this job, since the goal on that layer is to stop the request before it reaches your server.

One more reason to prefer the path over the name: the rule keeps working when the scanner changes its disguise. The GreyNoise cluster used 13 crawler names, and our logs showed single sources rotating through 10 to 15. A list of names is always one name behind. A list of paths that should never be public is short and does not change with the disguise.

## Verify it worked

**The AIScan route (recommended).** Run `npx aiscan-cli yoursite.com` or paste the URL at [aiscan.site](https://aiscan.site/), free and no account. Three things should stay green after the change: B2 (your `robots.txt` still answers and names the AI bots), P1 to P4 (your `/.well-known/` files still answer 200), and E1 (an unknown path still returns a real 404). The [bot access checks](/docs/checks/bot-access) page lists B2. None of those checks grades a request for `/.env`, so they cannot confirm the block itself. The by-hand route does.

**By hand.** Run these and expect the status codes shown.

```bash
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/.env                         # 404
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/.git/config                  # 404
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/robots.txt                   # 200
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/.well-known/mcp/server-card.json  # 200 if you publish one
curl -s -o /dev/null -w '%{http_code}\n' -A 'GPTBot' https://yoursite.com/                 # 200
```

If the fourth line returns 404, the carve-out failed. If the first returns 200, the rule is not in the server block that serves your domain.

## What to leave alone

- **Real AI crawlers.** Do not block by name. Control them in `robots.txt` and, where you need to, by [rate](/blog/rate-limit-ai-crawlers-without-blocking).
- **Files you publish on purpose.** A downloadable `.sql` or `.bak` sample will 404 too. Rename it.
- **The files themselves.** [OWASP's testing guide](https://wstg.owasp.org/v4.2/4-Web_Application_Security_Testing/02-Configuration_and_Deployment_Management_Testing/04-Review_Old_Backup_and_Unreferenced_Files_for_Sensitive_Information/) lists old backups and unreferenced files as a way to reach credentials and database access. A 404 hides them, and deleting them removes the risk. If a secret ever sat at a reachable path, rotate it.

Check the rule after server migrations and plugin changes, and re-test when you add a new `.well-known` file. A correct 404 for paths you do not have is covered in the [404 guide](/blog/correct-404-for-ai-agents).

## Next step

Scan your site with `npx aiscan-cli yoursite.com`, check B2, P1 to P4 and E1, then browse the rest of the [AIScan guides](/guides).