Block Fake AI Crawlers From Probing .env and .git Files

Some of the traffic that calls itself GPTBot or ClaudeBot is not a crawler. It is a scanner that borrowed a famous name and asks your server for /.env, a .git folder or a wp-config.php.bak. The fix is one path rule per server layer, and in our local test it turned 6 of 6 secret-path requests into 404s while the home page, robots.txt and /.well-known/ kept answering 200 for both a GPTBot user agent and a browser one.

The rule has to match the path. A user-agent rule cannot work, because the impostor and the real crawler send the same string. The second trap is the usual "deny every dotfile" snippet, which also takes down /.well-known/, where AI agents look for server cards and other discovery files.

Quick summary

Recommended order: run npx aiscan-cli yoursite.com first, add the server rule that matches your stack, then scan again and confirm /.well-known/ and robots.txt still pass.

What the impostors are doing

GreyNoise reported on 28 August 2026 a cluster of scanners impersonating 13 AI crawlers from eight companies. They requested .env files, cloud access keys, private keys and password stores. Six of the names came from the same 824 addresses.

Our own logs show the same shape at small scale. In 63 days of server logs, 1,297 of 2,107 requests (61.6%) came from 22 of 147 sources that each presented 10 to 15 different crawler names. Every one of those got a 404, because our site had nothing at those paths. That is why nothing worse happened.

The analogy is a visitor in a delivery uniform. The uniform proves nothing, so the guard checks which door the visitor is trying. A crawler reading your blog never needs the door marked "keys". Refuse that door for everyone, and the real crawler is unaffected.

Why a user-agent rule fails

We ran a rule that returns 403 when the user agent contains GPTBot, and fetched six paths with a GPTBot string and a browser string.

The rule blocked the genuine crawler from your content and gave the impostor nothing to stop it: send a browser string instead and /.env answers 200. If you want to separate real from fake, verify the source address. OpenAI publishes the address ranges for GPTBot at openai.com/gptbot.json. That is a different job from protecting files.

The test setup

nginx 1.24 and Apache 2.4.58 ran locally on 31 October 2026 with a document root holding six items: a real page, robots.txt, a decoy .env, a decoy .git/config, a decoy wp-config.php.bak and a decoy /.well-known/mcp/server-card.json. No real secret was involved. Each server ran three configurations: no rule, the "deny every dotfile" snippet, and the path rule below. Each was fetched with a GPTBot user agent and a Chrome user agent, 36 requests per server.

nginx and Apache returned identical results, and the user agent made no difference in any row. The middle row is the trap: the common snippet hides .env and also hides the capability files that checks P1 to P4 look for. The well-known folder guide covers the other way that folder breaks.

Path 1: nginx

Add these inside the server { } block, then run nginx -t and reload.

location ~ /\.(?!well-known(/|$)) { return 404; }
location ~* \.(bak|old|orig|sql|swp)$ { return 404; }

1. Open the server config for your site, usually a file under sites-enabled or conf.d.
2. Paste both lines inside the server { } block, above any location / that proxies to your app.
3. Run nginx -t. It should print "syntax is ok".
4. Reload with systemctl reload nginx.

The first line matches any path segment starting with a dot, except .well-known. The lookahead is why .well-known survives. Regex locations in nginx are tried in file order, and a regex match beats a plain prefix like location /, so the rule applies even in front of a proxied app. See the nginx location documentation.

Path 2: Apache

Put these in the virtual host or main config. We tested them there, not in .htaccess.

RedirectMatch 404 "/\.(?!well-known(/|$))"
RedirectMatch 404 "(?i)\.(bak|old|orig|sql|swp)$"

1. Confirm mod_alias is enabled (apache2ctl -M | grep alias).
2. Add both lines to the virtual host.
3. Run apache2ctl configtest, then reload.

RedirectMatch with status 404 sends a 404 and no redirect target, per the mod_alias documentation. On shared hosting where you only have .htaccess, the same two lines usually work there if the host allows alias directives. We did not test that.

Path 3: Cloudflare

Cloudflare can refuse the request before it reaches your server. We did not create this rule on a zone of ours, so the expression below comes from Cloudflare's rules language reference, not from our test.

1. In the Cloudflare dashboard, open Security rules, then Create rule > Custom rules.
2. Name it "Refuse secret paths".
3. Switch to the expression editor and paste:

(http.request.uri.path contains "/." and not starts_with(http.request.uri.path, "/.well-known/"))
or ends_with(lower(http.request.uri.path), ".bak")
or ends_with(lower(http.request.uri.path), ".sql")

1. Under Then take action, pick Block. Cloudflare says that on plans without the Log action you should start with Managed Challenge, read Security Events, then switch to Block.
2. Select Deploy.

This uses contains, starts_with, ends_with and lower, so it needs no regular expression. The matches operator needs a Business or Enterprise plan, which is why we avoided it. If you also use Cloudflare's AI bot controls, the crawl control guide shows how to keep verified crawlers allowed while this rule runs on paths.

Path 4: WordPress

If your site runs on Apache or LiteSpeed's paid edition, a plugin can write the rules for you. BetterShield's "Hide private files from visitors" answers requests for wp-config backups, logs, database dumps, .git and .env with a 404 through .htaccess. It is off until you turn it on, under BetterShield > Protection > Hardening. Its docs say it needs Apache or LiteSpeed's paid edition, so on nginx use Path 1. We did not install it for this post.

Why 404 and not 403

All three server rules above answer 404. A 403 tells a scanner the file exists and is protected, which is a hint worth having if you are guessing at paths. A 404 gives the same answer whether the file is there or not. It also matches what a site with nothing at that path already says, so your logs stay uniform. Cloudflare's Block action returns its own block page instead, and we did not measure its status code. That is fine for this job, since the goal on that layer is to stop the request before it reaches your server.

One more reason to prefer the path over the name: the rule keeps working when the scanner changes its disguise. The GreyNoise cluster used 13 crawler names, and our logs showed single sources rotating through 10 to 15. A list of names is always one name behind. A list of paths that should never be public is short and does not change with the disguise.

Verify it worked

The AIScan route (recommended). Run npx aiscan-cli yoursite.com or paste the URL at aiscan.site, free and no account. Three things should stay green after the change: B2 (your robots.txt still answers and names the AI bots), P1 to P4 (your /.well-known/ files still answer 200), and E1 (an unknown path still returns a real 404). The bot access checks page lists B2. None of those checks grades a request for /.env, so they cannot confirm the block itself. The by-hand route does.

By hand. Run these and expect the status codes shown.

curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/.env                         # 404
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/.git/config                  # 404
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/robots.txt                   # 200
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/.well-known/mcp/server-card.json  # 200 if you publish one
curl -s -o /dev/null -w '%{http_code}\n' -A 'GPTBot' https://yoursite.com/                 # 200

If the fourth line returns 404, the carve-out failed. If the first returns 200, the rule is not in the server block that serves your domain.

What to leave alone

• Real AI crawlers. Do not block by name. Control them in robots.txt and, where you need to, by rate.
• Files you publish on purpose. A downloadable .sql or .bak sample will 404 too. Rename it.
• The files themselves. OWASP's testing guide lists old backups and unreferenced files as a way to reach credentials and database access. A 404 hides them, and deleting them removes the risk. If a secret ever sat at a reachable path, rotate it.

Check the rule after server migrations and plugin changes, and re-test when you add a new .well-known file. A correct 404 for paths you do not have is covered in the 404 guide.

Next step

Scan your site with npx aiscan-cli yoursite.com, check B2, P1 to P4 and E1, then browse the rest of the AIScan guides.