Agent Discovery Files Census: P1, P3, P4 and E2 Across 835 Sites

Only 301 of the 835 sites in our scan corpus pass any one of four agent-discovery checks that depend on a /.well-known/ file or an API description. All four pass on 99 sites. Take out one platform and the picture changes: 79 of those 99 are Astro sites, which are 113 of the 835 hosts. This post is the census behind those numbers, built from what our scanner stored, with the limits of that evidence stated next to every figure.

The four checks are P1 (API catalog, /.well-known/api-catalog), P3 (Agent Skills bundle), P4 (OAuth discovery documents) and E2 (an OpenAPI description). They sit on the capabilities page of our check docs. Each one asks a site to publish a machine-readable file at a predictable address so that an agent does not have to guess.

TL;DR: what the census found

Start here: run npx aiscan-cli yoursite.com (or paste your address at aiscan.site, free, no account) and read the lines for P1, P3, P4 and E2. If you would rather check by hand, the curl commands are in the last section.

Best for a site with an API: publish an OpenAPI file first (E2), then point an API catalog at it (P1). The API catalog how-to walks through the file.

Best for a site with no API: do nothing for P4 and P1 beyond reading the product notes below. A missing file is the right answer there, and two of our four checks score it as a failure anyway.

What we counted, and what we did not

This is a census of stored scan results, not a live sweep of the web. We ran no new probes against third-party hosts for this post. An earlier plan for it called for a fresh sweep of every agentic well-known path across a named external population. That sweep needs thousands of requests to hosts we do not own, so we answered the question from the scans users and we had already run.

The population:

• Table scans, rubric version 2026.08.2 only, scope site.
• The latest scan per host, with www. and scheme stripped before deduplicating.
• aiscan.site excluded.
• 835 hosts, first scan stored 25 August 2026, read on 11 October 2026.
• No host is named anywhere in this post.

Every figure was aggregated in SQL. A representative query:

WITH latest AS (
  SELECT DISTINCT ON (lower(regexp_replace(url,'^https?://(www\.)?','')))
         url, platform, result
  FROM scans
  WHERE rubric_version = '2026.08.2' AND scope = 'site'
    AND url NOT ILIKE '%aiscan.site%'
  ORDER BY lower(regexp_replace(url,'^https?://(www\.)?','')), created_at DESC)
SELECT c->>'id' AS check_id, c->>'status' AS status, count(*)
FROM latest, jsonb_array_elements(result->'checks') c
WHERE c->>'id' IN ('P1','P3','P4','E2')
GROUP BY 1, 2 ORDER BY 1, 2;

Three limits shape everything below:

1. The corpus is not a random sample of the web. It is the set of sites that someone chose to scan. That skews toward people who care about agent readiness, and toward sites built with tools that advertise it. Read every percentage as "of scanned sites", never "of websites".
2. The evidence stores status codes, not bodies. For these four checks the stored evidence says things like 200 at /.well-known/api-catalog. It does not say whether the body parsed as the file the RFC describes. A soft 200 that returns an HTML page would look the same, and we cannot separate it from a real file without fetching the host again.
3. Check status has four values. pass, partial, fail and info, where info carries no weight. A missing optional file is often info, so a "pass rate" over the whole corpus mixes sites that failed with sites that were never asked. We give counts for every status.

The four paths, in plain terms

Four well-known paths, defined by four different documents. Calling them "the agentic web files" hides how different they are.

The first three live under the /.well-known/ prefix that RFC 8615 reserves for exactly this kind of discovery file. E2 is a convention about common filenames, and it only counts as well-known when a catalog points at it.

Two of the four are only meaningful if you have the thing they describe. A marketing site with no API has nothing to put in an API catalog, no OAuth server to point at and no OpenAPI file to publish. That matters when we get to the verdicts.

The headline table: statuses for all four checks

Each row sums to 835. Three observations before looking at combinations.

P3 cannot fail. Of 835 hosts, 164 pass and 671 are info. The evidence for the info rows is dominated by HTTP 404 (562 hosts), then HTTP 403 (45), HTTP 429 (19), HTTP 530 (12) and 14 rows that read not published (optional). A site that returns 404 for the Agent Skills path loses nothing. That is a defensible rule for an optional convention, and the next table shows it is not how the other three checks treat a missing file.

P3 probes one path. Our own notes from an earlier manual comparison record a large payments site that serves its skills index after a redirect to another host, and P3 reports the miss. The 164 passes are therefore a floor. Neither this post nor the rubric claims that 671 sites lack a skills file.

P1 and P4 count a 429 as a fail in the stored data. Four P1 fails read 429 at /.well-known/api-catalog and four P4 fails read auth-server: 429, protected-resource: 429. A host that rate-limited our scanner is recorded the same as a host without the file. Our determinism self-audit found the same pattern on two other checks and we filed it as a scanner issue.

One absence, three different verdicts

Here is how each check scores a plain "not there":

That is 506 failing readings across P1, P4 and E2 for a plain absence (one host can fail more than one). P3 is the only one of the four that treats a missing file as nothing.

Is that fair? It depends on the site. A store with a public product API and a sign-in flow should publish all of it. A brochure site with no API has nothing to describe, and the check cannot know that. The stored data carries no signal that separates "has an API and forgot to document it" from "has no API". In the rubric P1 is worth 4 points and P4 is worth 3.

For the reader: if your site has no API and no sign-in, a fail on P1, P4 or E2 is the scanner's limit, not your mistake. Publishing an empty catalog to turn the line green would help nobody, and an agent reading it learns nothing true. Treat the failing lines as questions: "Do I have an API?" If the answer is no, move on.

The Astro effect

Compare the platform groups:

Astro hosts are 13.5% of the corpus and supply 97 of the 238 P1 passes (40.8%), 81 of the 164 P3 passes (49.4%) and 96 of the 184 E2 passes (52.2%). Of the 114 E2 passes that rest on "api-catalog advertises a service-desc document", 86 are Astro.

Splitting the corpus in two shows how much of the headline sits in one group:

Outside Astro, 20 of 722 sites pass all four, and 56 of the 722 pass P4 and nothing else in the group.

We do not know why Astro is this far ahead. One reading is a shared starter template, theme or integration that generates all four files, so a few dozen sites made the same choice once. Another is that the people who build Astro sites and also run an agent-readiness scan are a self-selected group. The data cannot separate those. It says only that the Astro number should not be read as a property of the framework and that it lifts every overall figure for the corpus. When you read the ranking in our priority-ladder study, remember this group is in it.

For everyone else the rates are low and close together: P1 on 141 of 722 non-Astro sites (19.5%), P3 on 83 (11.5%), P4 on 125 (17.3%) and E2 on 88 (12.2%).

P4 is the odd one: one string, three statuses

The most useful query in this programme groups a check's evidence strings. For P4 it turns up something that cannot be a pure function of the evidence.

The string auth-server: 200, protected-resource: 200 appears on 232 hosts. Their statuses:

Identical evidence, three outcomes. Something other than the status codes decides the status, and the stored evidence does not record it. A guess is that the check also reads the body, or the response type, and only the status line is saved. A guess is all it is.

Two more patterns in the P4 data:

• Of the 213 passes, 188 have both documents answering 200. At least 22 pass with one document answering 200 and the other 404, so the check accepts either document alone.
• The sites that pass P4 and nothing else are not Astro sites. Of the 56 such hosts, 39 are WordPress, 11 are Shopify and 6 are unknown. P4 is also where WordPress and Shopify do best: 58 of 197 WordPress hosts and 13 of 17 Shopify hosts pass it.

A platform serving a 200 at an OAuth metadata path is not necessarily an OAuth deployment ready for agents. Without the body, "pass" here means "answered", and that is as far as the stored evidence goes. We have not fetched those documents to check them.

E2: the OpenAPI path and the Swagger path

Of 184 E2 passes, 114 come from an API catalog that advertises a service-desc document, which is the link relation RFC 9727 uses to point at an API description. The other 70 are direct hits on a filename.

For the direct hits, the grouped evidence shows /openapi.json answering 200 with application/json on 40 hosts, application/openapi+json on 11 and application/vnd.oai.openapi+json on 7. Those last two are the registered media types for an OpenAPI document, and a client can recognise them without opening the file.

The weakest passes are the seven (at least; this counts only the evidence strings that repeat three or more times) that rest on /swagger.json answering 200 with text/html. A Swagger path that returns HTML is more likely a documentation page or a single-page app falling back to its shell than a JSON file. They pass the check. Whether they should is a question about the check, and a different post, and it is the same pattern we found when C1 passed on status code alone.

On the failing side, 180 hosts show the single string no /openapi.json, /api/openapi.json or swagger document found. The probe list is short. An OpenAPI file at /api/v1/openapi.json, /docs/openapi.yaml or behind a link relation in the HTML head is invisible to it, so a site can document its API well and still fail E2.

How the four combine

Of the 835 hosts, the pass patterns are:

Three things stand out.

All-or-nothing. 633 of 835 hosts (75.8%) are either all four or none. A site that publishes one of these files usually has a reason to publish others, or a tool that does it for them.

P1 and E2 travel together. Of the 184 hosts that pass E2, 180 also pass P1. Only 4 pass E2 without P1. That fits the 114 catalog-advertised passes, where the catalog is the route to the OpenAPI file, but it also holds for direct-path hits.

P4 is the loner. 56 hosts pass P4 and nothing else. Only the none-at-all and all-four patterns are more common.

Do the files move the score?

Hosts that pass at least one of the four average an overall score of 83.0. All 835 average 60.9. That gap is large, and it is mostly Astro: the Astro group averages 88.7, the other 722 hosts average 56.5.

This is an association, not a cause. The scoring weights are public: P1 is worth 4 points, P3 4, P4 3, and the checks interact with every other file on the site. Sites that publish agent-discovery files also tend to publish llms.txt, a sitemap and server-rendered pages. Adding a catalog to a site that does none of the rest will not lift it to 83.

If you want the ordering by return for effort, the priority-ladder study ranks all checks by score delta. The four here are not at the top of that list. They matter most to sites that already have an API.

Product notes from this census

These came out of the grouped-evidence queries. We list them because publishing the bugs in your own checks is cheaper than having someone else find them.

1. P4 status is not derivable from its evidence. The same 200/200 string gives pass, fail and info. Store the reason, such as "body parsed as JSON with an issuer key".
2. Rate limits read as absence. P1 and P4 score a 429 as a fail. A 429 should read as "could not tell", the way P3 already treats it.
3. Absence is scored three ways. P3 gives info, the other three give fail. Either every optional file should be info when absent, or the rubric should say why not.
4. Status codes are stored without bodies. For P1, P4 and E2 the evidence should say whether the body matched the format, so a soft 200 is visible.
5. P3 probes one path. A redirected or relocated skills index is missed.
6. E2 probes a short list of filenames. A spec in a conventional but unlisted location fails the check.

Check your own site

The scan is the default way to check this. Run npx aiscan-cli yoursite.com and read four lines: P1, P3, P4 and E2. A status of info means the file is optional and absent; fail means the check expects it and could not find it. If you have an API, treat a fail on P1 and E2 as a to-do. If you have none, you can ignore all four.

If you would rather check by hand:

# P1: should return 200 and a linkset, not an HTML page
curl -sIL https://yoursite.com/.well-known/api-catalog | head -n 15
# P4: both documents, and look for "issuer" and "resource" in the JSON
curl -sL https://yoursite.com/.well-known/oauth-authorization-server | head -c 400
curl -sL https://yoursite.com/.well-known/oauth-protected-resource | head -c 400
# E2: content type should be an OpenAPI or JSON type, not text/html
curl -sIL https://yoursite.com/openapi.json | grep -i -E "^(HTTP|content-type)"

Always check the content-type and open the body. This census could only count statuses. You can do better.

For the how-to on each file, read the guides for the API catalog, the OAuth discovery documents, the Agent Skills index and the MCP server card. For a Shopify store, the commerce-side picture is in our UCP adoption study, and if you run a store, StoreSEO covers the llms.txt, schema and AEO side that these four files do not touch.

Next step

Scan your site, read the P1, P3, P4 and E2 lines, and decide for each whether you have the thing it describes. Then open the guides index for the fix that matches your platform, and scan again. The capabilities check docs list what each check requests.