Bot Access Scores 0 on 58% of 820 Sites, and One Check Decides It

Of 820 hosts in the AIScan corpus, 475 (57.9%) score 0 on the Bot Access dimension and 263 (32.1%) score 100. Only 82 sit anywhere between. The dimension is close to a coin flip, and the coin is a single check: B2, which counts how many known AI crawler names your robots.txt addresses. Naming four or more passes, naming one to three earns partial credit, and naming none fails.

Bot Access is also small. It carries an average of 5.6 weight points against 20.5 for content and 18.1 for discoverability. Small does not mean harmless: for the 475 zero-score hosts, full marks would add about 9 points to the overall score on average, and 72 of the 363 hosts below 60 would cross it.

Quick summary

Short answer for a site owner: Bot Access is decided by whether your robots.txt names AI crawlers, not by whether it allows or blocks them. If your latest scan shows "0 known AI agent(s) addressed in robots.txt", add explicit User-agent: groups for at least four of the watchlist names. Run npx aiscan-cli yoursite.com (or paste the address at aiscan.site, free, no account) and read checks B2, B1 and B3. If you would rather check by hand, the curl command is in the fix section.

What this study measures, and what it cannot

We took the latest site-scope scan of every host under rubric 2026.08.2, one scan per host. aiscan.site is excluded and no scanned host is named. That gives 820 hosts. Each scan stores a dimensions object with five entries (bot_access, capabilities, commerce, content, discoverability), each with an applicable flag, the points earned, a 0 to 100 score and a weight. Earlier corpus studies on this blog worked on single checks. This one reads the dimension layer.

Limits you should hold in mind:

• Self-selected sample. People scan their own sites, or sites they are about to work on. This is not a random draw from the web and it says nothing about what share of all sites name AI crawlers.
• Correlation, not cause. Sites that keep a tuned robots.txt tend to be tuned elsewhere. The overall-score gap between the zero and full groups is far larger than what Bot Access alone can explain, and we calculate the part it does explain below.
• Weights vary by host. Capabilities applies to 426 hosts and commerce to 289, so the denominator behind each overall score differs. We recompute using each host's own stored weights.
• Stored evidence is thin. B2 stores a count of agents, not their names, and not the file. We can say how many names a robots.txt carries, not which ones.
• No claim about crawler behaviour. We measure what a file says. Whether a given bot obeys it, or crawls more because it is named, is outside this data.
• Demand is not measured. We found no search query for this angle in Search Console and no keyword tool on our plan, so we do not quote a volume. Site owners do ask whether to allow or block AI crawlers. The gap this study fills is what the scoring does with the answer.

The five dimensions side by side

Three things stand out.

Bot Access is the only dimension that applies to every host and sits mostly at the extremes. Content and discoverability apply to every host too, but they spread out: only 6.0% of hosts score 0 on content and none score 0 on discoverability, which means every host in the sample earned at least some discoverability points.

Capabilities and commerce look polarised as well (42.3% and 33.9% at zero), but they apply only to some hosts, and the stored data does not say why each host was judged an app or a store, so read their shares as shares of scored hosts only.

Weights do not sum to 100. A host's applicable weights add up to about 51 to 64 points on average, depending on group, and the overall score is earned points divided by applicable points. That is why the 5.61 weight average understates the dimension's share: measured against each host's own applicable total, Bot Access carries 10.5% on average, with a range of 7.2% to 22.0%. The queue note that prompted this study said 5.6% of the points. That figure confused weight points with percentage, and the corrected number is about double.

One check decides the dimension

The Bot Access docs list three checks:

B1 is flagged informational because the standard is still a draft, and per the docs it cannot lower your score. B3 is informational too. That leaves B2 as the only check that can take a host from full marks to zero on its own, which is what the data shows.

Here is how the 475 zero-score hosts and the 263 full-score hosts differ on those checks:

Every one of the 475 zero hosts fails B2 with the same sentence, and none of them has a Content-Signal line or a key directory. Every one of the 263 full-score hosts passes B2. B2 is the whole story at the edges.

The 82 in between are the odd group. 55 of them fail B2 and still hold some Bot Access points, and 21 of the 82 pass B1. The stored report does not say which check supplied the points for each of those hosts, so we do not guess at a mechanism. What the numbers do show is that 530 hosts name no AI agent, and 475 of them (89.6%) end at zero while 55 do not.

How many names does it take

B2 stores a count, so we can plot score against count for every host:

The step is sharp. Hosts naming one to three agents average a Bot Access score near 60. Hosts naming four or more average 97 or higher at every count from 4 through 16. Past four names, naming more adds almost nothing. The practical edge is the fourth name. Only ten hosts in 820 sit in the one-to-three band. That suggests owners rarely stop at a partial list: they either do not touch the subject or they add a full block. We cannot see intent, so treat that as a reading of the counts, not a fact about owners.

The B2 watchlist, per the docs, is GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Bytespider, Amazonbot, meta-externalagent, CCBot, cohere-ai and Diffbot.

What the overall score does with it

The gap between the zero and full groups is 36.9 points. It would be easy to read that as "naming AI crawlers is worth 37 points". It is not, and the data lets us say how much of it is.

For each of the 475 zero hosts we took the stored earned points and weights, set Bot Access to its full weight, left everything else alone and recomputed the overall score. Our recomputation of the current score matched the stored one (42.7 against 42.8 on average), so the method reproduces what the scanner does. The result:

So roughly a quarter of the 36.9-point gap (9.4 of it) is the arithmetic of the dimension itself. The rest is who these sites are. Of the 475 zero hosts, 22.5% have both content and discoverability above 60. For the full-score hosts it is 77.6%. A site that already carries a tuned robots.txt is far more likely to have working structured data, a sitemap and server-rendered content too, and that group-level difference is not something a robots.txt edit creates.

A quarter of the gap is still the cheapest quarter on offer. Adding a robots.txt block takes minutes, needs no code deploy on most platforms, and touches a file that already exists. The highest-impact-fix study ranks checks by points recovered per host, and the check co-pass matrix shows which checks pass together. B2 is a text-file edit, which makes it a different kind of job from the checks that need markup or server changes.

Reading the counterfactual with care

The +9.4 figure is a ceiling for the mechanical effect, and three things keep it honest.

First, it assumes full Bot Access marks. A host that names exactly four agents averages 97 to 100 in this sample, so the real gain for a typical fixer is a hair under the ceiling, not half of it. Second, it holds every other dimension fixed. A host that edits robots.txt this week does not change its content or discoverability points. Third, it uses the stored weights, and those are rubric 2026.08.2's. A later rubric version can move them, and rescans under a new version will not be comparable to these numbers. The rescan study covers what happens to scores when the same host is scanned twice.

The 72 hosts that would cross 60 are the more useful number for planning. They are the sites where the missing Bot Access points are the margin between a middling report and a decent one. Another 291 of the 363 sub-60 hosts would still be under 60 after the fix, because their gaps are in content, capabilities or commerce. For those, B2 is worth doing and not worth stopping at.

Naming is not the same as allowing

B2 is a naming check. The docs say it looks for explicit User-agent: blocks "naming the known AI crawlers", and the fix text says to replace Allow: / with Disallow: / for any bot you would rather not feed. A site that blocks all sixteen agents by name passes the same as a site that allows all sixteen. The check scores whether the policy is unambiguous, not which policy you chose.

That matters for how you read your own result. If your report says "0 known AI agent(s) addressed in robots.txt" and you meant to block everything, a User-agent: * block with Disallow: / is a policy but it is not an explicit rule for any AI crawler as far as B2 is concerned. Under RFC 9309 a crawler follows the group that matches its own name most specifically and falls back to * only when none matches, so naming a bot is also the way to give it a different rule from everyone else. Google documents the same idea for its Google-Extended token.

A related finding from earlier in this series: 25 of 144 hosts that passed B2 later lost every named rule, all from exactly eight names to none. If your score used to be 100 and now reads 0, check for that before assuming the scoring changed. The AI crawler declaration census covers which names show up most in files that do name some.

How to fix it

Start with the scan, because it tells you which of the three states you are in. Run npx aiscan-cli yoursite.com or paste the address at aiscan.site. B2 reports the count it found. B1 and B3 will most likely read as informational. Fix B2 first, since it is the only one that moves the score.

If you would rather check by hand:

curl -s https://yoursite.com/robots.txt | grep -i -E 'GPTBot|ClaudeBot|PerplexityBot|Google-Extended|Applebot-Extended'

Count the distinct names that come back. Fewer than four means B2 is partial or failing. Then add one group per name. This is the block AIScan's own fix guidance recommends, using five of the watchlist names:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

Swap Allow: / for Disallow: / on any bot you do not want, and add path-level rules if your policy is mixed. Then re-scan. If robots.txt on a WordPress site is being written by an SEO plugin, edit it there rather than uploading a file, or the plugin will overwrite you. ThinkRank manages robots.txt, robots meta, schema and llms.txt from one plugin, which avoids two plugins fighting over the same file, and it imports settings from Rank Math, Yoast, AIOSEO and SEOPress. Whichever plugin you use, the check is the same: the live /robots.txt must contain the names.

What a re-scan will and will not tell you: it tells you the file now names the agents. It does not tell you that any crawler read it, honoured it, or visited more or less. For that, look at your server logs, as described in find AI crawlers in your server logs.

What to do next

Run the scan and read B2 first, then B1 and B3. If B2 says fewer than four agents, add the block above, deploy it and scan again. The rest of the Bot Access dimension is optional. For the full picture across all five dimensions, see the state of AI agent readiness and the platform league table. More fix guides are in /guides.