---
title: "Web Bot Auth Key Directories: 78 of 797 Sites Answer 200, and Most Are Soft 404s"
slug: web-bot-auth-directory-adoption
published: 2026-10-31T03:09:57.573961+00:00
updated: 2026-11-05T03:11:55.665471+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: Web Bot Auth, HTTP Message Signatures, soft 404, AI readiness, corpus study, check:B3, check:E1
description: "Of 797 sites, 78 answer 200 at the Web Bot Auth key directory path, but 53 of them also fail the soft-404 check. Data from AIScan check B3, 31 Oct 2026."
url: https://aiscan.site/blog/web-bot-auth-directory-adoption
---

Of 797 sites in our scan corpus, 78 answer HTTP 200 at the Web Bot Auth key directory path. That is 9.8%. The number sounds like early adoption, and it is not. 53 of those 78 sites also fail our soft-404 check, which means they answer 200 at paths where nothing exists. Strip those out and the real count of sites that look like they publish a key directory is somewhere between 21 and 25, about 3% of the corpus.

The data below is first-party. It comes from check B3 in AIScan's own scan database, read on 31 October 2026, and every figure is a count of distinct hosts.

## Quick summary

**Direct answer:** Almost no website needs to publish a Web Bot Auth key directory. The directory belongs to operators of agents and crawlers that sign their outgoing requests. A site that only receives traffic has nothing to sign, so a 200 at `/.well-known/http-message-signatures-directory` on an ordinary site is far more likely a catch-all response than a published key set.

| Question | Answer from 797 hosts |
|---|---|
| Hosts answering HTTP 200 at the B3 path | 78 (9.8%) |
| Hosts reported as "not published" | 719 (90.2%) |
| Of the 78, how many also fail the soft-404 check (E1) | 53 (67.9%) |
| Of the 78, how many have a clean E1 | 21 pass, 4 partial |
| B3 status `pass` vs `partial` | 21 vs 57, identical evidence text |
| Share of E1 failures that answer 200 at the B3 path | 53 of 84 (63.1%) |
| Share of E1-clean hosts that answer 200 at the B3 path | 21 of 594 (3.5%) |
| Check that grades this | B3, bot_access dimension, weight 3, optional |

**Best for:** anyone who ran a scan, saw B3 report `info`, and wonders whether they are behind. You are not. Also useful if B3 shows `partial` or `pass` on a site that runs no agent, because that deserves a second look.

If you would rather skip the reading, run the scan first: `npx aiscan-cli yoursite.com`, or paste your address at [aiscan.site](https://aiscan.site/). It is free and needs no account. Look at B3 in the bot access group, then at E1 in discoverability. Read the two together, as this article explains.

## What B3 checks, and what it cannot see

Web Bot Auth is an effort to let a crawler or agent prove who it is. The operator signs each request using HTTP Message Signatures ([RFC 9421](https://www.rfc-editor.org/rfc/rfc9421)) and publishes the matching public keys in a directory the receiving site can fetch. The [architecture draft](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/) at the IETF describes the arrangement. Our explainer, [Web Bot Auth: cryptographic crawler identity](/blog/web-bot-auth-explained), covers the mechanism. This article does not repeat it. It counts who publishes.

[The documentation for B3](/docs/checks/bot-access) says AIScan looks for a JWKS-style directory at `/.well-known/http-message-signatures-directory`. The check has weight 3, is optional, and is marked informational, because the standard is still experimental. The same page says most site owners do not need to publish anything here.

Three limits shape everything that follows.

- **The evidence stores a status code and nothing else.** Every B3 result in the corpus carries the evidence string `HTTP 200` or the text "not published (optional / experimental)". There is no body, no key count, no content type. So no claim in this article says a directory is valid, signed correctly or in use.
- **A 200 is not a key set.** A host that returns 200 for any unknown path will answer 200 here too. The B3 result alone cannot separate a real directory from a catch-all page.
- **AIScan cannot see signing.** Whether a site's own agent signs outgoing requests is invisible to a scan of the site. B3 measures publishing, not use.

## How we counted

- **Source:** AIScan's production scan database, queried on 31 October 2026.
- **Rubric:** one rubric version only (2026.08.2), so a check's definition does not change mid-sample.
- **One scan per host:** the latest scan for each hostname. A site scanned fifty times counts once.
- **Exclusions:** aiscan.site is excluded, because it scores 100 and would flatter every average.
- **Result:** 797 hosts. B3 is scored on every one of them, since it does not depend on a blog or a store.
- **Privacy:** every figure is an aggregate. No scanned host is named here, and none will be.

**A limit you should know about.** This is not a random sample of the web. These are sites that someone chose to scan with AIScan, or that we scanned ourselves. Sites that care about agent readiness are probably over-represented. The adoption shares below describe this corpus, not the web.

## Finding 1: 78 sites answer 200, and the evidence cannot say why

Grouping the B3 evidence strings across the corpus gives exactly three rows:

| Status | Evidence | Hosts |
|---|---|---|
| info | not published (optional / experimental) | 719 |
| partial | HTTP 200 | 57 |
| pass | HTTP 200 | 21 |

Two statuses share one evidence string. That is a gap in what the report tells you: when B3 shows `partial` or `pass`, the line beneath it reads `HTTP 200` in both cases, and the report does not say what the check read that made it choose. This is a known gap in the report. Here we report the two statuses separately where the data allows it and together where it does not.

The statuses do behave differently, which is how we know they are not noise. Among the 21 `pass` hosts, 17 have a clean soft-404 check. Among the 57 `partial` hosts, only 4 do. That difference is large enough to say the check reads something beyond the status code, even though the evidence line does not show it.

## Finding 2: two thirds of the 200s come from sites that answer 200 to everything

Check E1 asks whether a site returns a real 404 for a path that does not exist. A site that serves its homepage, a branded error page or a search page with status 200 for a made-up URL fails E1. Our [soft-404 article](/blog/soft-404-agent-discovery-problem) explains why that matters for agents. The [discoverability docs](/docs/checks/discoverability) list the check.

Crossing B3 with E1 gives the table that changes the headline:

| E1 result | Hosts | Answer 200 at the B3 path | Share |
|---|---|---|---|
| Pass (real 404s) | 594 | 21 | 3.5% |
| Partial | 119 | 4 | 3.4% |
| Fail (soft 404) | 84 | 53 | 63.1% |
| All hosts | 797 | 78 | 9.8% |

A soft-404 host is eighteen times as likely to answer 200 at the B3 path as a host with clean 404s (63.1% against 3.5%). A site does not become more likely to run an agent because its error handling is loose. The simpler reading is that the 200 comes from the catch-all, not from a published directory.

Of the 57 `partial` hosts, 50 fail E1. That is 87.7%. Of the 21 `pass` hosts, 3 fail E1. So `pass` is mostly clean and `partial` is mostly a catch-all. If you want a working estimate of true publishers, take the 21 `pass` hosts as the upper end of what looks like a directory, and the 25 hosts with a non-failing E1 as the upper end of what could be one. We cannot confirm either number from the stored evidence.

The same pattern shows up on a second optional file: in [our study of MCP server cards](/blog/mcp-server-card-adoption-corpus), 52 of the 65 hosts that answer the card path with partial credit also fail E1. A related note for anyone building on this: a scanner that reads only the status code will overcount any well-known file on a soft-404 site. The same effect showed up in [our census of agent files](/blog/well-known-agent-files-corpus), and it is the reason this article reports the E1-clean subset next to the headline.

## Finding 3: the E1-clean publishers score higher and declare bot rules

Hosts that answer 200 at the B3 path with a clean E1 are a different group from the rest. Compare means and bot-rule behavior:

| Group | Hosts | Mean overall score | Pass B2 (named AI bot rules) |
|---|---|---|---|
| B3 answers 200, E1 pass | 21 | 86.1 | 17 (81.0%) |
| All E1-pass hosts | 594 | 62.3 | 236 (39.7%) |
| B3 answers 200, E1 fail | 53 | 63.3 | 17 (32.1%) |
| All E1-fail hosts | 84 | 53.7 | not computed |

The 21 look like deliberate sites. Their mean score is 86.1 against 62.3 for every clean-404 host, and 17 of 21 name AI bots in robots.txt. The 53 look like everyone else on a soft-404 stack. Their B2 rate, 32.1%, is close to the 32.7% of non-answering hosts that pass B2 (235 of 719).

Whole-corpus numbers, for reference. Among the 78 hosts answering 200, 34 pass B2 (43.6%) and 24 declare a Content-Signal line in robots.txt (B1). Among the 719 that answer "not published", 235 pass B2 (32.7%). The raw difference is real, and most of it comes from the 21.

Do not read this as "publishing a directory raises your score". B3 is informational and carries almost no weight. The likelier story is that a site careful enough to publish signing keys is careful about the rest of its machine-readable surface too.

## Finding 4: who the answering sites are

By platform, for the five platforms with at least 15 hosts, and then the rest as a group:

| Platform | Hosts | Answer 200 at B3 | Share |
|---|---|---|---|
| Not detected | 294 | 33 | 11.2% |
| WordPress | 199 | 22 | 11.1% |
| Next.js | 198 | 9 | 4.5% |
| Astro | 43 | 4 | 9.3% |
| Shopify | 16 | 1 | 6.3% |
| All other platforms (under 15 each) | 47 | 9 | 19.1% |

WordPress and unidentified stacks together supply 55 of the 78. That tracks their share of the corpus (493 of 797, 61.9%), so platform does not explain the 200s. Next.js stands out in the other direction: 4.5% against 9.8% overall.

The platform split changes once we remove soft-404 hosts. Of the 25 answering hosts with a non-failing E1, 11 are WordPress, 5 are Next.js, 4 are Astro, 3 are unidentified, 1 is Shopify and 1 is Vue. WordPress still leads, but the unidentified group collapses from 33 to 3. That is the pattern you would expect if custom or hosted-builder stacks are where the catch-all responses come from.

By agent-readiness level (the ladder on [the scoring page](/docs/scoring), 0 to 5):

| Level | Hosts | Answer 200 at B3 | Share |
|---|---|---|---|
| 0 Invisible | 31 | 0 | 0% |
| 1 Crawlable | 107 | 1 | 0.9% |
| 2 Indexable | 164 | 11 | 6.7% |
| 3 Readable | 240 | 27 | 11.3% |
| 4 Discoverable APIs | 151 | 25 | 16.6% |
| 5 Agent-Native | 104 | 14 | 13.5% |

The share rises with level and flattens at the top. The lower rungs are almost empty because a site at level 0 or 1 usually fails the broader checks that decide how a site looks to a crawler. The result is a mild correlation, with the usual limit: higher levels are also more likely to be sites that were built for the purpose.

## Who should publish a key directory

Our stance, given the data and the specification: publish only if you operate something that sends signed requests.

| You are | Publish a B3 directory? | Why |
|---|---|---|
| An AI company, search engine or crawler operator | Yes | The directory is how receiving sites verify your signatures. |
| A company running an agent that browses or buys on behalf of users | Yes, if you sign | A receiving site can only tell your agent from an impersonator if it can fetch your keys. |
| A site that wants to allow or block bots | No | You verify other people's signatures. You do not publish keys for that. |
| A blog, store or documentation site | No | There is nothing to sign. A B3 `info` result costs you nothing. |
| A site returning 200 at the B3 path by accident | Fix the catch-all | The 200 is an E1 problem, and it misleads every tool that reads well-known files. |

If the last row is you, the fix is not in B3 at all. Make unknown paths return 404 or 410, then re-scan. Cloudflare, nginx and Apache all let you do this without touching application code. Our [guides](/guides) list the E1 walkthroughs by platform.

The wider point is that the first ten percent of "adoption" in any well-known file is a measurement problem before it is a trend. Before anyone quotes an adoption figure for a path under `/.well-known/`, they should say how they excluded sites that answer 200 for everything.

## What to do if you scan and see B3 partial or pass

1. Request the path yourself: `curl -si https://yoursite.com/.well-known/http-message-signatures-directory | head -20`.
2. If the response is JSON with a `keys` array, you or your host publishes a directory. Check that the key IDs match what your signing agent uses.
3. If the response is your homepage or an error page with status 200, you have a soft 404. Fix E1 first.
4. If you do not run an agent and the response is a key set you did not create, find out who put it there. A hosting provider or a plugin may have.
5. Re-scan and read B3 and E1 together.

If you would rather check by hand across many files at once, the AIScan CLI prints every check in the bot access group in one run, and `npx aiscan-cli yoursite.com` lists E1 beside it.

## Your next action

1. Run `npx aiscan-cli yoursite.com` or paste your address at [aiscan.site](https://aiscan.site/).
2. Read B3 and E1 together. If B3 says `info`, you have nothing to do.
3. If B3 says `partial` or `pass` and you run no agent, fix E1 and re-scan.
4. If you do run a signing agent, read [how Web Bot Auth works](/blog/web-bot-auth-explained) and publish a directory at the path above.
5. Work through the other bot access checks in the [bot access docs](/docs/checks/bot-access), and browse the [AIScan guides](/guides) for the next fix.

## Frequently asked questions

**Do I need a Web Bot Auth key directory on my website?**
No, unless you operate a crawler or agent that signs its own requests. The directory publishes your signing keys so other sites can verify you. A site that only receives traffic has no keys to publish, and AIScan marks B3 as informational for that reason.

**My scan shows B3 as partial and I run no agent. Why?**
Most likely your site answers 200 for paths that do not exist. In our corpus 50 of the 57 partial hosts also fail the soft-404 check. Request the path with `curl`, and if you get a page that is not a key set, fix your 404 handling.

**What does the B3 evidence "HTTP 200" actually prove?**
Only that the path answered 200. It does not show a body, a key count or a content type, and it does not prove the response is a valid directory. `pass` and `partial` carry the same evidence text, so the status line cannot be audited from the report alone.

**Why do pass and partial show the same evidence?**
The report prints the status code in both cases and does not print the reason for the status. The statuses are not random: 17 of 21 pass hosts have a clean soft-404 check, against 4 of 57 partial hosts. The missing explanation is a known gap in the report.

**Does publishing a key directory improve my AIScan score?**
Barely. B3 has weight 3 and is optional, so its effect on the total is small. The higher scores among sites that answer 200 reflect careful sites in general, not a reward for this file.

**Which sites are most likely to have a real directory?**
Operators of agents and crawlers. In our data the strongest candidates are the 21 hosts that answer 200, pass the soft-404 check and score 86.1 on average. We cannot confirm any of them is valid from stored evidence.

**Is it safe to publish the directory at the well-known path?**
Yes, it is meant to be public. It carries public keys only. Keep the private signing key off the web server, and rotate it according to your agent's policy.

**How do I stop a catch-all route from answering 200 at well-known paths?**
Return 404 for unknown paths at your web server or framework router, and keep an explicit allow list for the well-known files you do serve. Then check with `curl -si` on a made-up path, and re-scan to confirm E1 reads pass.

## Sources and dates

- AIScan scan database, check B3 and E1 results, 797 hosts, rubric 2026.08.2, queried 31 October 2026 (first-party).
- [AIScan documentation: bot access checks (B3)](/docs/checks/bot-access), read 31 October 2026.
- [AIScan documentation: scoring and levels](/docs/scoring), read 31 October 2026.
- [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421), IETF.
- [HTTP Message Signatures Directory and Web Bot Auth architecture draft](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/), IETF datatracker.