---
title: "Blocked Probes: 85 of 712 Sites Scored the Firewall, Not the Site"
slug: blocked-probe-share-of-failures
published: 2026-10-13T03:17:34.907996+00:00
updated: 2026-10-13T03:19:32.489101+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: corpus study, blocked probes, agent readiness, firewall, check:D1, check:C1, check:C2, check:B2, check:E3
description: "85 of 712 sites answered AIScan with a 403, 429 or 5xx. About 22% of C1, C2 and D1 failures were blocked probes, not missing files. How to tell which."
url: https://aiscan.site/blog/blocked-probe-share-of-failures
---

A failing score has two possible meanings. Either the site does not have the thing, or the scanner never got to look. A scan report prints both as the same red mark, and the owner reading it cannot tell which one they are holding.

We went looking for the second kind in our own data. AIScan stores the full report for every scan it has run, including the evidence string behind each check. For three of the checks that string carries the HTTP status code the probe received. On 13 October 2026 we queried production for those codes across 712 sites. We counted every failure whose evidence is a 403, a 429 or a 5xx, because a refusal or a server error is a statement about the request, not about the file. That gave 208 failing rows spread over 85 sites. Those 85 sites average a score of 21.9 against 60.4 for the other 627, and 75 of them sit in the "platform not detected" bucket. For a visible slice of the corpus, the score measures the firewall.

## Quick summary

| Question | Measured answer | Section |
|---|---|---|
| How many failing rows are a block or a server error? | 208 of 932 failures on C1, C2 and D1 (22.3%) | Per-check section |
| Which check is hit hardest? | D1 robots.txt: 59 of 116 failures (50.9%); C2 llms.txt 70 of 338 (20.7%); C1 Markdown 79 of 478 (16.5%) | Per-check section |
| How many sites are affected? | 85 of 712 (11.9%); 62 saw only 403 or 429, 23 saw only a 5xx | Host section |
| What do those sites score? | Mean 21.9 against 60.4 for the rest | Score section |
| What changes if blocked probes are set aside? | Corpus mean 55.8 to 56.8; the 85 sites move 21.9 to 30.6 on average, 29 of them by 10 points or more | Score section |
| Why is so much of the corpus "unknown" platform? | 75 of the 85 blocked sites are unknown, and 75 of 270 unknown sites (27.8%) are blocked | Platform section |
| Is a block permanent? | Of 74 sites blocked at least once, 50 were blocked every time and 24 were not | Repeat-scan section |

**What the data says:** about one site in eight returned a refusal or a server error to at least one of three probes, and for most of them (56 of 85) it did so on all three. A check that fails on a 403 has not measured the site's llms.txt, its robots.txt or its Markdown. It has measured a gate.

**Best for a site owner who sees HTTP 403 or 429 in a report:** treat it as a firewall finding first and a content finding second. **For anyone quoting an agent-readiness number:** ask whether the failures behind it were could-not-tell results counted as fails, because ours were.

## How we measured it

The limits decide how far to trust any of this, so they come first.

Every figure was queried on 13 October 2026 from AIScan's production database. The population is every site-scope scan on rubric 2026.08.2, which has been live since 24 August 2026. Scans in the table run from 24 August to 13 October. We reduced each URL to its host (no scheme, no leading `www.`) and kept the newest scan per host. `aiscan.site` is excluded: our own self-scans score 100 and would flatter every average. Page-scope scans are excluded. That leaves 712 hosts. No scanned site is named in this article.

The question needs a field that records what the probe received, and only some checks have one. When we grouped the evidence strings, three scored checks put a status code in the text of a failure:

| Check | What it probes | Failure evidence looks like |
|---|---|---|
| [C1](/docs/checks/content) Markdown content negotiation | The homepage with an `Accept: text/markdown` header | `Accept: text/markdown → 403 (text/html; charset=UTF-8)` |
| [C2](/docs/checks/content) /llms.txt | The `/llms.txt` path | `HTTP 429` |
| [D1](/docs/checks/discoverability) robots.txt present and sane | The `/robots.txt` path | `HTTP 403` |

Two more checks, P1 (API catalog) and P4 (OAuth discovery), also store codes such as `403 at /.well-known/api-catalog`, but they report mostly as informational and carry no weight, so they cannot move a score. We left them out of every number below. The other scored checks do not store a code in a failure string, which matters later.

We counted a failure as blocked only when the whole evidence string was a bare `HTTP` status or ended an arrow with a 403, 429 or 5xx code. The matching was strict on purpose. A looser pattern that finds "403" anywhere in the text would also catch byte counts and timestamps.

The count itself is one pass over the unnested checks:

```sql
WITH latest AS (
  SELECT DISTINCT ON (host) * FROM scans_with_host
  WHERE rubric_version = '2026.08.2' AND scope = 'site' AND host <> 'aiscan.site'
  ORDER BY host, created_at DESC
)
SELECT c->>'id' AS check, count(*) FILTER (WHERE c->>'status' = 'fail') AS fails,
       count(*) FILTER (WHERE c->>'status' = 'fail'
         AND (c->>'evidence' ~ '^HTTP (403|429|5[0-9][0-9])$'
           OR c->>'evidence' ~ '→ (403|429|5[0-9][0-9]) ')) AS blocked
FROM latest, jsonb_array_elements(result->'checks') c
WHERE c->>'id' IN ('C1', 'C2', 'D1') GROUP BY 1;
```

`scans_with_host` stands in for the `scans` table with the host expression applied. One more disclosure: the evidence does not record which user agent the probe sent, so this study cannot say which rule at which firewall produced a given 403.

## What a blocked row looks like

Across C1, C2 and D1 the 208 blocked failures break down as follows.

| Status | Failing rows | Share of the 208 | What it usually means |
|---|---|---|---|
| 403 Forbidden | 91 | 43.8% | The server or its firewall refused the request |
| 429 Too Many Requests | 61 | 29.3% | A rate limit answered before the file did |
| 530 | 36 | 17.3% | A Cloudflare error: the request reached the edge and not the origin |
| 526 | 9 | 4.3% | Cloudflare could not validate the origin's certificate |
| 500 | 6 | 2.9% | Origin server error |
| 525 | 3 | 1.4% | Cloudflare's TLS handshake with the origin failed |
| 502 | 2 | 1.0% | Bad gateway |

The first two rows, 403 and 429, are defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110) and [RFC 6585](https://www.rfc-editor.org/rfc/rfc6585) as answers about the request: you may not have this, or you have asked too often. Neither says the resource is missing. The Cloudflare codes (525, 526, 530) are listed in [Cloudflare's own 5xx error reference](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/) and signal a fault between the edge and the origin. A site that returns one of them to our scanner is probably returning it to a visitor too, which is a site problem. It is just a different problem from a missing llms.txt.

So the 208 split into 152 refusals (403 and 429) and 56 server-side errors. The second group is the one a scanner owes the owner an explanation for, because the fix is not "add a file".

## Per-check share: robots.txt takes the biggest hit

| Check | Failures | Blocked or server error | Share | 403 | 429 | 5xx |
|---|---|---|---|---|---|---|
| C1 Markdown negotiation | 478 | 79 | 16.5% | 36 | 21 | 22 |
| C2 llms.txt | 338 | 70 | 20.7% | 31 | 21 | 18 |
| D1 robots.txt | 116 | 59 | 50.9% | 24 | 19 | 16 |
| Total | 932 | 208 | 22.3% | 91 | 61 | 56 |

The D1 row is the one that should change how you read a report. Of the 116 sites that fail D1, only 48 answered HTTP 404, which is the real "no robots.txt here" result. Another 59 were blocked or errored. The remaining 9 failed for other reasons, such as a robots.txt that was present but not sane. If you read the D1 failure rate as "how many sites forgot robots.txt", you overstate the forgetting by more than a factor of two.

C1 and C2 behave differently because most of their failures are real. A site that answers 200 with an HTML page to an `Accept: text/markdown` request has not blocked anyone, it has declined to negotiate. Our [Markdown twin study](/blog/markdown-twin-beyond-docs) counted 381 such answers among 477 C1 failures, and its note on 78 blocked rows is the same effect measured earlier on a slightly different population. For C2 the bulk of failures are HTTP 404 (257 of 338), the honest "no llms.txt" result.

## Sites, not rows: 85 of 712

Rows overcount. One firewall that refuses the scanner will refuse it on all three probes. Counting hosts instead of rows:

| Group | Sites | Share of 712 |
|---|---|---|
| Blocked or errored on at least one of C1, C2, D1 | 85 | 11.9% |
| Of those: all three probes | 56 | 7.9% |
| Of those: two probes | 11 | 1.5% |
| Of those: one probe | 18 | 2.5% |
| Only 403 or 429, no 5xx | 62 | 8.7% |
| Only 5xx, no 403 or 429 | 23 | 3.2% |
| Both kinds | 0 | 0% |

No site showed both a refusal and a server error. That separation is clean enough to treat as two populations. The 62 refusal sites are deciding about the visitor, through a WAF rule, a bot-management product or a rate limit. The 23 server-error sites are failing for everyone, or at least failing for requests that look like ours.

The two populations score alike. The refusal group averages 21.5 and the error group 22.8, which suggests the score depends mostly on how many probes failed and little on why.

## What the score looks like without the blocked probes

The AIScan overall score is the sum of earned weight over the sum of applicable weight, taken across the scored dimensions. We rebuilt it from the stored dimension totals and matched the stored mean of 55.8 exactly, which is the check that the method is right.

Then we set the blocked C1, C2 and D1 probes aside, removing their weight from the dimension denominator and leaving every other check alone. This is what a "could not tell" status would do.

| Population | Sites | Mean score as stored | Mean score with blocked probes set aside |
|---|---|---|---|
| All sites | 712 | 55.8 | 56.8 |
| Sites with at least one blocked probe | 85 | 21.9 | 30.6 |
| Sites with no blocked probe | 627 | 60.4 | 60.4 |

The corpus average barely moves, up 1.0 point, because 88% of sites are untouched. The affected sites move a lot: 8.7 points on average, 75 of them by 5 points or more, 29 by 10 or more, with a maximum of 19.3. Eleven sites climb from under 40 to 40 or higher.

Read that as a floor, not an estimate. We only set aside three checks, and the evidence strings of the others cannot tell us whether they too were blocked. The next two sections show why the real figure is probably larger.

## The unknown platform bucket is partly a firewall bucket

AIScan detects a site's platform by reading its homepage. A scanner that is refused the homepage cannot fingerprint anything. In the platform table, restricted to platforms with at least 15 sites:

| Detected platform | Sites | With a blocked probe | Share blocked | Mean score, blocked | Mean score, not blocked |
|---|---|---|---|---|---|
| Not detected | 270 | 75 | 27.8% | 21.1 | 56.4 |
| WordPress | 187 | 4 | 2.1% | 34.0 | 67.3 |
| Next.js | 162 | 2 | 1.2% | 54.5 | 53.1 |
| Astro | 35 | 4 | 11.4% | 9.0 | 75.3 |
| Shopify | 17 | 0 | 0% | n/a | 66.6 |

Of the 85 blocked sites, 75 (88.2%) are in "not detected". Put the other way round, more than a quarter of the not-detected bucket is not a mysterious long tail of custom stacks. It is sites that turned the scanner away at the door. The not-detected bucket's unblocked mean of 56.4 is in line with the other platforms, so its low overall average in earlier studies came from this group and not from some weakness in the unknown stacks themselves.

Two cautions. The Astro row rests on four blocked sites, so its 9.0 is an anecdote. And the Shopify zero is a reminder that platform-hosted stores answer probes consistently, which is a different situation from a self-managed server behind a bot filter. Our [state of AI agent readiness study](/blog/state-of-ai-agent-readiness-2026) reported platform splits drawn from the same corpus. Those splits need this caveat attached.

## The blind spots downstream: B2 and E3

This is the part that makes 30.6 look conservative.

B2, the check for explicit AI crawler rules, reads robots.txt. D1 reads robots.txt. If the file returns a 403, both checks fail from the same refusal. We looked at the 59 sites whose D1 evidence is a bare blocked status. All 59 also fail B2, and none passes. B2's own failure string does not say why, so those 59 failures sit in our B2 numbers as "no AI bot rules" when the honest status is "could not read robots.txt".

The homepage tells the same story. E3 (server-rendered text and headings) reads the homepage body.

| Check | Pass rate, 85 blocked sites | Pass rate, 627 other sites |
|---|---|---|
| E3 text and headings | 9 of 85 (10.6%) | 79.4% |
| B2 AI bot rules | 6 of 85 (7.1%) | 36.5% |

An agent-readiness failure that low on E3, a check most sites pass four times out of five, is what you would expect if the homepage itself was refused. The 85 sites were not poor at extractable content. A lot of them were never read. We did not attempt a correction for these checks because their stored evidence cannot support one, and that gap is itself a finding, covered under product notes.

## Is a block permanent?

507 of the 712 hosts have been scanned more than once, so we can ask whether the same host gets blocked every time.

| Repeat-scan pattern | Sites |
|---|---|
| Scanned at least twice | 507 |
| Blocked on at least one scan | 74 |
| Blocked on every scan | 50 |
| Blocked on some scans, clean on others | 24 |

A third of the sites that ever saw a block (24 of 74) also saw a clean scan. That matches our [determinism audit](/blog/aiscan-check-determinism-self-audit), which traced 12 of 18 D1 flips to a 429 answering on one scan and not the next. It also means a single scan is a weak basis for a verdict on a site behind a bot filter. Our [repeat-scan study](/blog/repeat-scan-change-study) found that a lost pass can come from a changed file or a blocked probe, and the stored evidence can distinguish only some of the cases. The 50 always-blocked sites are the stable group, and for them re-scanning will not help until the owner changes a rule.

## What this does not show

- **It does not show the block is wrong.** A site may refuse unidentified automated requests on purpose, and some of these 152 refusals will be policy working as intended. The finding is about how a scanner should report the result, not about owners misconfiguring things.
- **It does not identify the cause.** The evidence has no user agent, no response headers beyond content type and no body, so a 403 from a WAF rule and a 403 from an origin permission bug look the same.
- **It undercounts.** Only C1, C2 and D1 store a usable code on a failure. B2, E3, D3, E2 and the rest do not, so every blocked-host total here is a floor.
- **It is one vantage point.** Every probe comes from the same scanning infrastructure, so a site that treats that address range differently from its real visitors is counted as blocked even if no visitor ever sees a refusal.
- **It is a self-selected corpus.** These are sites someone chose to scan, not a random sample of the web. For the wider picture, start with the [state of AI agent readiness](/blog/state-of-ai-agent-readiness-2026).

## Product notes for the scanner

Four changes follow from the data, and we have filed them as a site-issue request.

1. **Add a could-not-tell status.** A 403, 429 or 5xx on C1, C2 or D1 should not score as a fail. The report already has an informational status that is left out of the score, which is how P1 and P4 behave today.
2. **Say so in the report.** The sentence "your server answered 403 to this request, so we could not check" helps an owner more than a red mark.
3. **Make B2 inherit D1.** If robots.txt could not be read, B2 cannot have an opinion.
4. **Store the user agent and the response headers on a refusal.** A `cf-ray` or a `Retry-After` header turns an unexplained 429 into something an owner can act on.

## What to do if your report shows a 403 or a 429

Run the scan first. Use `npx aiscan-cli yoursite.com` or paste the address at [aiscan.site](https://aiscan.site/); both are free and need no account. Look at D1, C1 and C2. If the evidence reads `HTTP 403`, `HTTP 429` or an arrow followed by one of those codes, you have the finding this article is about.

Then confirm it by hand, if you would rather see the raw response.

```bash
for p in robots.txt llms.txt; do
  curl -s -o /dev/null -w "%{http_code}  /$p\n" "https://yoursite.com/$p"
done
curl -s -o /dev/null -w "%{http_code}  markdown\n" -H "Accept: text/markdown" https://yoursite.com/
```

A 200 or 404 from your own machine and a 403 in the scan report means a rule is treating the scanner differently from you. The place to look is your firewall or bot-management event log, filtered to the scan time. On Cloudflare the Security events view lists which rule matched. A 429 points to a rate limit, and the fix is to raise the limit on those three paths or exempt them, since robots.txt and llms.txt are cheap static files.

Three practical rules cover most cases:

1. **Keep robots.txt and llms.txt out of the challenge path.** Crawlers read robots.txt before anything else, so it is the file where a challenge page or a rate limit does the most damage, and both are cheap static files that rarely need protecting.
2. **Decide on purpose, per crawler.** If you want to refuse AI crawlers, say so in robots.txt, where [B2](/docs/checks/bot-access) can read it, not through silent 403s. A written rule can be audited and a silent block cannot.
3. **Scan twice.** The repeat-scan data shows a third of blocked sites answering normally on another day, so one refusal is a prompt to check, not a verdict.

On WordPress, the file is often served by an SEO plugin and the block by a security plugin or host-level firewall, which makes the cause easy to misread. [ThinkRank](https://thinkrank.ai) is the plugin we recommend for the file side, because it manages robots.txt, robots meta, schema and llms.txt from one place instead of three plugins competing over `/robots.txt`, and it imports settings from Rank Math, Yoast, AIOSEO and SEOPress. It will not change what your firewall does, so check both. Rank Math, Yoast and AIOSEO can also edit robots.txt, and the rule that matters is the same: only one of them should own the file. If you run a Shopify store, [StoreSEO](https://storeseo.com/) covers llms.txt, schema and AEO basics for stores. None of the 17 Shopify sites in our corpus returned a blocked probe, so the platform side is rarely the problem and the file side is where you have work to do. For llms.txt itself, the [llms.txt generator](/llms-txt-generator) writes a valid file in a minute.

If you do none of the above, you can still finish the job without us: the curl loop shows the status, your firewall log shows the rule and the file is plain text. The scan saves the steps and puts the evidence string next to the check ID.

## Run it on your own site

Paste your address into [aiscan.site](https://aiscan.site/) or run `npx aiscan-cli yoursite.com`, and read the status code in the evidence line for D1, C1 and C2. B2 and E3 deserve a second look if any of the three shows a block. The relevant pages are the [discoverability](/docs/checks/discoverability), [content](/docs/checks/content) and [bot access](/docs/checks/bot-access) check guides. The [guides](/guides) list a fix for every check. Scan again in a week.