Bot Access Checks
B1 Content Signals, B2 AI bot rules, B3 Web Bot Auth.
The default User-agent: * rule is too coarse for the AI era. Agents that train models, agents that ground answers in your live site, and agents that act on a user's behalf are three different things — and you probably feel differently about each. Bot Access is where you say so.
B1 — Content Signals in robots.txt (weight 5, optional)
Cloudflare's Content-Signal directive is an emerging proposal that lets you state, in one line, what crawlers may do with your content. AIScan flags it as informational because the standard is still in draft — it cannot lower your score.
How to fix
# robots.txt
Content-Signal: search=yes, ai-train=no, ai-input=yes- search — list this page in search results.
- ai-train — use this page to train models.
- ai-input — fetch this page to ground a single AI answer.
B2 — Explicit AI bot rules (weight 5)
AIScan looks for explicit User-agent: blocks naming the known AI crawlers. Four or more = pass, one to three = partial credit. The current watchlist:
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Bytespider, Amazonbot, meta-externalagent, CCBot, cohere-ai, Diffbot.
How to fix
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /Replace Allow: / with Disallow: / for any bot you'd rather not feed.
B3 — Web Bot Auth key directory (weight 3, optional)
Web Bot Auth is an IETF draft for cryptographically identifying bots via HTTP Message Signatures. AIScan checks for a JWKS-style directory at /.well-known/http-message-signatures-directory. Informational only — the standard is still experimental.
How to fix
If you operate an agent, publish your signing keys at the well-known path. Most site owners don't need to publish anything here.
Further reading
- Track ChatGPT, Perplexity, Claude and Gemini Referrals in GA4
GA4 can show you the visitors that ChatGPT, Claude, Perplexity and Gemini send to your site, but only if the session source is read correctly. Google now ships a default "AI Assistant" channel, and a…
- Content Signals in robots.txt: 117 of 799 Sites Declare One
Of 799 sites in our scan database on 2 November 2026, 117 serve a robots.txt with a ContentSignal line 14.6%. Among the 677 sites that serve a usable robots.txt at all, that is 17.3%. Another 682…
- Find AI Crawlers in Your Server Logs and Verify Them
Your analytics tool will not tell you which AI crawlers visit your site, because a crawler that fetches the HTML and leaves does not run its script. Only the server or the edge in front of it sees…
- ChatGPT Says It Can't Access Your Website: Five Causes and How to Tell Which
When ChatGPT says it cannot open your page, one of five things happened, and a short script of four curl requests sorts four of them in about a minute. The fifth is that ChatGPT never tried. This…
- Block Fake AI Crawlers From Probing .env and .git Files
Some of the traffic that calls itself GPTBot or ClaudeBot is not a crawler. It is a scanner that borrowed a famous name and asks your server for /.env, a .git folder or a wpconfig.php.bak. The fix is…
- Web Bot Auth Key Directories: 78 of 797 Sites Answer 200, and Most Are Soft 404s
Of 797 sites in our scan corpus, 78 answer HTTP 200 at the Web Bot Auth key directory path. That is 9.8%. The number sounds like early adoption, and it is not. 53 of those 78 sites also fail our…
- Your robots.txt blocks everything by accident: find the leftover Disallow: / and stop it coming back
If your robots.txt says Useragent: followed by Disallow: /, every crawler that obeys the file is told to stay off your whole site. It takes one line to cause and about a minute to find, and Google's…
- Fake AI Crawler Traffic: 63 Days of Our Own Server Logs
Of the 2,107 requests our own logger recorded under an AI crawler's name between 25 August and 27 October 2026, 61.6% came from 22 sources that each used 10 to 15 different crawler names, and not one…