The State of AI Agent Readiness in 2026: 473 Sites Measured

Half of the web's agent-readiness problem is already solved, and almost nobody has noticed which half. Across 473 real websites scanned by AIScan between 24 August and 3 September 2026, the median score is 55 out of 100. The discoverability layer that twenty-five years of SEO built (robots.txt, sitemaps, HTTPS, correct 404s) averages 77.5. Everything the agentic web actually added averages 39.3, and 252 of those 473 sites score exactly zero on it.

This is the first distribution study AIScan has published from its own corpus. Every number below comes from one database query against one rubric version, and the methodology section says plainly where the numbers are wrong.

Quick summary

The single most useful takeaway: a site that has done ordinary SEO well will land somewhere near 70 without touching anything agent-specific, and will still fail every check that an AI agent actually needs. 326 of the 473 sites pass both robots.txt and sitemap checks. Of those, 88 pass none of Markdown negotiation, llms.txt or explicit AI bot rules.

How the sample was built, and what it cannot tell you

The raw corpus is 2,922 scans of 787 distinct URLs recorded between 23 June and 3 September 2026, spanning three rubric versions. Mixing rubric versions would make the distribution meaningless, so the study population is narrower:

That last exclusion moves the mean from 61.6 to 56.4, which is a fair illustration of how easily a vendor's own corpus flatters the vendor.

Four limits matter more than the numbers do.

The sample is self-selected. These are sites whose owners chose to run an agent-readiness scan, so the population is more technical and far more likely to have heard of llms.txt than the web at large. According to Ahrefs, whose 15 June 2026 study of 137,210 domains carries the same warning in its own words, such customers "skew more technical and SEO-aware than the web at large". Treat every adoption figure here as an upper bound.

A readiness score is rubric-relative, not a property of a site. Our comparison work measured nextjs.org at 73 and Level 4 on this rubric and at Level 0, "Not Ready", on Cloudflare's scanner in the same hour, verified on 31 August 2026. A histogram of our scores is a histogram of our opinions.

Two of the five dimensions are conditional. The commerce checks ran on 175 of the 473 sites and the capabilities dimension applied to 230, because the rubric only grades those where it detects a relevant surface. Any commerce percentage in this article has a denominator of 175, never 473.

Our platform detector is known to misfire. It reported stripe.com as a Lovable app at 50% confidence, on the evidence, in its own words, of a /_next/static/ asset plus the string "lovable" in a JavaScript bundle. The platform breakdown below is the softest table here and should be read as indicative.

No individual scanned URL is named anywhere in this article. The corpus contains sites that other people submitted, and their scores are theirs.

The distribution: median 55, and a long tail at the bottom

Mean 56.4, median 55, 25th percentile 41, 75th percentile 75, 90th percentile 90.8. The lowest score recorded was 8.

The level bands, which are what the product actually shows a user, spread differently:

The shape is one broad hump centred on the low fifties with a thin bottom tail worth naming: 45 sites, roughly one in ten, score under 20. At that level a site is typically missing robots.txt, returning 200 for pages that do not exist, or shipping no server-rendered text at all.

For comparison, Agent Ready volunteers a percentile against its own corpus in every scan response. Fetched from its scan API on 3 September 2026, that corpus reports 1,528 sites, up from 1,410 on 29 August. It is the only other published baseline in this category, and it is roughly three times our sample size.

Discoverability is finished. Bot access has barely started.

Not one site in 473 scored zero on discoverability. Every one had at least a partial robots.txt, sitemap, HTTPS or 404 story, because those are the artefacts of ordinary search optimisation and the tooling has existed for two decades.

Bot access has a median of zero. 252 sites, 53.3% of the sample, earn nothing at all across B1, B2 and B3. That dimension asks three questions: does robots.txt carry Content Signals, does it name any AI crawler explicitly, is there a Web Bot Auth key directory. More than half of an unusually technical sample answers no to all three.

The gap between those two rows is the finding. Sites are discoverable. Sites have not decided anything about who may use what they find.

All 21 checks, ranked by how often sites pass them

Read top to bottom and the boundary is obvious: everything above 50% predates the agentic web, everything below 40% was proposed in the last two years. The one straddle is C2, llms.txt, at 44.6%, the only recent convention with anything close to majority adoption here.

188 of the 473 sites, 39.7%, pass none of C1, C2, B2 or P2. That is the cleanest single sentence in the dataset: two fifths of sites that deliberately ran an agent-readiness scan have adopted no agent-specific convention whatsoever.

The llms.txt number depends entirely on who you sample

C2 passes on 44.6% of sites here. Three other published measurements disagree, and none of them is wrong.

The spread runs from 8.7% to 44.6% across four honest measurements, and the ordering is exactly what you would predict from how each sample was drawn. The head of the web is full of banks, government sites and infrastructure hosts that will never publish an llms.txt. A traffic-weighted analytics panel skews technical. A queue of people who typed their URL into an agent-readiness scanner skews further still.

According to Rankability's own methodology note, verified on 3 September 2026, most of that gap is denominator choice: it reached a verdict on 549 of the top 1,000, recorded 451 as unknown, then "deliberately uses all 1,000 sites as its denominator" for the public headline.

According to the same Ahrefs study, 97% of the valid llms.txt files it found received zero requests in May 2026. Publishing the file and having it read are separate events, and only one of them is measured by any scanner including ours.

If you want to write one, AIScan's llms.txt generator builds a spec-shaped file from your sitemap. The eleven ways real files break the v2 spec are documented in our validator study of 39 live files.

Platform predicts the score better than anything else we record

Smaller groups (Vue 4, Framer 3, Wix 3, Bolt 3, SvelteKit 2, Nuxt 2, HubSpot 1) are excluded for sample size.

WordPress sites out-score Next.js sites by 15.8 points on the mean and 21 on the median. The median is the sharper number, because half of the Next.js sites in this sample sit below 46.

That is not a verdict on React. It is a verdict on defaults. WordPress serves rendered HTML, generates a sitemap, and has a plugin ecosystem that has been writing robots.txt for fifteen years. A Next.js app can do all of that and frequently does not, for the reasons in our Next.js rendering post: client-side data fetching, ssr: false, and shells that render for a person and fail for a crawler.

For WordPress, ThinkRank is the shortest path from this table's Next.js column to its WordPress column. The specific reason it fits here is the one this dataset keeps exposing: the checks that fail together (B1 Content Signals, B2 explicit AI bot rules, C2 llms.txt, C3 structured data, D2 sitemap) are owned by four different files that four different plugins fight over. One plugin owns all four surfaces in ThinkRank: the robots file, robots meta, schema and sitemaps, plus llms.txt. It also imports whatever Yoast, Rank Math, AIOSEO or SEOPress already had configured, so nothing has to be typed in twice. Yoast and Rank Math are both excellent at schema and sitemaps and are installed on vastly more sites; neither generates the llms.txt file. Running one of those and only missing the file? Upload a static one and pay nothing.

The 11 Shopify stores here sit differently again: the platform hands merchants part of the stack and none of the discovery layer. That missing half is what StoreSEO fills, generating llms.txt from products, collections, pages and articles, editing agents.md, and writing product schema. Its Shopify App Store listing showed a 5.0 rating across 685 reviews when checked on 27 August 2026. A free plan is available.

Four checks almost nobody passes

M3, the x402 payment signal: 0 of 175. Not one commerce-classified site signals a machine-payable price. This matches our 2 September 2026 measurement, where Cloudflare's scanner returned ucp pass and x402, mpp, acp and ap2 all fail on a real agent-enabled Shopify store. The protocol exists, the tooling exists, adoption is flat zero.

P2, the MCP Server Card: 3.4%. That number is wrong, and the next section says why.

B3, the Web Bot Auth key directory: 4.0%. This is a publisher-side check on a protocol only two crawler operators ship. Cloudflare's BotBase now validates Web Bot Auth signatures during operator onboarding, so the supply side is moving first.

E2 at 12.3% and P4 at 12.9%. Machine-readable API descriptions and OAuth discovery metadata are aimed at sites that intend to be used by an agent rather than read by one. Very few sites in any sample intend that yet.

According to Cloudflare, whose Radar sample of the top 200,000 domains was published on 17 April 2026, robots.txt reaches 78% of that population, Content Signals declarations only 4%, and Markdown content negotiation 3.9%. The pairing of an MCP Server Card with an API Catalog turned up on fewer than 15 domains out of 200,000. Our 21.1% on Content Signals and 30.9% on Markdown negotiation are five to eight times those rates, on a sample that selected itself for exactly this interest.

Where our own rubric is wrong, measured against itself

Four AIScan checks have confirmed defects, all found by our own publishing runs, all published before this article. All four are in the table above. Here is what each one does to the distribution.

The C1 defect can be sized from this dataset, and it is the most useful number here. C1 passes on 146 sites. 55 of those 146, or 37.7%, sit on hosts that do not cleanly pass E1, our own correct-404 check. A host that returns 200 for a path that does not exist returns 200 for /index.md too, and C1 reads that status code and passes. So up to 55 of the 146 C1 passes are the check congratulating a site for a file it does not have. Adjusting for it moves C1 from 30.9% to a floor of 19.2%, much closer to Cloudflare's independently measured 3.9%.

127 of the 473 sites, 26.8%, do not cleanly pass E1. Soft 404s are not a niche problem, and they contaminate every probe-a-well-known-path check in every scanner, not just ours.

P2 is worse in the other direction. On 2 September 2026 we measured 15 of 25 sampled Shopify storefronts answering POST /api/mcp with a valid JSON-RPC tools/list, several exposing ten tools including proceed_to_checkout, while P2 reported HTTP 404 and no server card. The full measurement is in our WebMCP study. If that ratio holds here, the real MCP-surface rate is several times 3.4%, and the Shopify row of the platform table is the least trustworthy number on this page.

None of this is a reason to withhold the distribution. It is a reason to publish the error bars with it.

Does scanning a site change it? 155 sites say mostly no

155 sites in the sample were scanned more than once under this rubric, a mean of 4.0 scans each over a mean span of 3.0 days.

Mean change from first scan to last: +5.0 points, from 56.1 to 61.1. Largest single improvement: +71 points. Largest regression: 29 points.

The honest reading is that seven sites in ten looked at a report and changed nothing measurable within three days. The mean gain is real but carried by a small group that acted decisively. This is the number a scanner vendor has the least incentive to publish and the most obligation to, because it is the only direct evidence anyone has that these reports lead to work.

Three days is a short window, so the effect is likely understated: a site that reads a report on Monday and ships the fix in the next sprint never appears in this cohort.

What the sites that did improve actually fixed

Among the 32 sites that improved, counting checks that moved from non-pass to pass between first and last scan:

The ordering is a surprise. B2 and C2 are the obvious first moves and they lead, but D3, a Link: header for discovery, is second. That is a CDN or server configuration change rather than a content change, which suggests the people acting fastest on these reports are infrastructure engineers rather than content teams.

No check regressed on more than one site, so these are additions rather than trade-offs. At the other end, exactly two checks were fixed by a single site each, B3 and M1, which is consistent with the pass-rate table: the checks almost nobody passes are also the checks almost nobody fixes.

Where AIScan fits, and where it doesn't

AIScan graded every number above, so the limits are worth stating directly.

What it sees: anything a server returns to an unauthenticated request. robots.txt and its directives (D1, B1, B2), sitemaps (D2), 404 behaviour (E1), HTTPS and canonical host (E4), Link: headers (D3), llms.txt (C2), Markdown negotiation (C1), server-rendered text and heading structure (E3), structured data (C3), feeds (E5), and the well-known paths behind the capabilities and commerce checks.

What it does not see: anything that only exists after JavaScript runs. AIScan does not render pages, so it cannot see WebMCP tools registered at runtime through navigator.modelContext, and it does not grade WebMCP at all: the word appears nowhere on our own capabilities checks page. It also cannot tell you whether any AI crawler ever read your llms.txt, the question Ahrefs answered with a flat 97% no.

Where competitors are ahead of us today: Agent Ready publishes a corpus percentile in every scan response and we do not, Cloudflare's free scanner returns a full evidence chain per check and ships a rendering step we lack, and both Agent Ready and AgentGrade ship a GitHub Action while AIScan does not. Our free tier is metered at 20 scans a month; AgentGrade, Cloudflare and IsAgentReady are all free and unmetered.

Where this rubric is genuinely ahead: E1 exists. Cloudflare's 22-check rubric has no soft-404 equivalent, and as the C1 cross-tab above shows, a soft 404 quietly corrupts every other check on the page.

Run the same measurement on your own site

Start with the scan, because it answers all 21 questions at once and needs no account:

npx aiscan-cli yoursite.com

Or paste the URL at aiscan.site. Both are free, need no account, and return the same rubric used for this study. The checks to read first, in the order this dataset says they matter:

1. B1 and B2 (bot access): the dimension with a median of zero. Add explicit AI crawler rules and Content Signals to robots.txt.
2. C2 (llms.txt): 217 sites here fail it outright. Start at the generator.
3. E1 (correct 404): 127 sites do not cleanly pass, and a soft 404 makes every other probe unreliable.
4. C1 (Markdown negotiation) and C3 (structured HTML): the content checks with the most partial results.

If you would rather check by hand, three commands cover most of it:

curl -s https://yoursite.com/robots.txt | grep -iE 'gptbot|claudebot|perplexity|content-signal'
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://yoursite.com/llms.txt
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/definitely-not-a-real-page-xyz   # must be 404

The third command is the one people skip. If it returns 200, every other probe you run against that host is meaningless, including the two above it.

Detail for each check ID is at /docs/checks/content and /docs/checks/bot-access; the setup guides by platform are at aiscan.site/guides.

This study will be repeated quarterly against the same rubric version, with the sample size, date range and known check defects published each time. Written by M Asif Rahman.