Which Agent-Readiness Checks Travel Together: 739 Sites, Three Bundles

Fix one agent-readiness check and a second one often moves with it. Fix another and nothing else changes. A scan report lists every check as its own line, so the reader cannot see which lines are really one job and which are separate jobs wearing separate labels.

We measured it. AIScan stores the full report for every scan, including the status of each check. On 15 October 2026 we queried production for the ten checks that every site gets scored on, across 739 sites, and counted how often each pair passes together. Most of the 45 pairs show only a weak link once site maturity is held constant. Three groups do not: robots.txt with the sitemap, the three AI-facing files (explicit AI bot rules, llms.txt and Markdown), and the homepage HTML checks. One pair does the opposite of what you would guess. Sites that pass the Markdown check are nine times more likely to fail the 404 check than sites that fail it, and the cause is a single server habit.

Quick summary

What the data says: the ten checks are closer to six decisions than ten. Fixing one member of a bundle is likely the cheapest way to move the next, and fixing something outside a bundle does not carry over.

Best for a site owner with a long red list: work bundle by bundle, not check by check. For anyone reading a pass rate: two checks that pass together are not two independent pieces of evidence about a site.

How we measured it

The limits decide how far to trust any of this, so they come first.

Every figure was queried on 15 October 2026 from AIScan's production database, and the rubric details were verified on 15 October 2026 against the live check guides. The population is every site-scope scan on rubric 2026.08.2, which has been live since 24 August 2026. We reduced each URL to its host, kept the newest scan per host and excluded aiscan.site, whose own scans score 100 and would skew every comparison. Page-scope scans are excluded. That leaves 739 hosts. No scanned site is named in this article.

Not every check is scored for every site. Our earlier work on the highest-impact fix found that several checks are info (weight zero) on most sites, and the platform league table showed that three capability checks switch between fail and info depending on whether the platform is detected as an app. A pair that includes a check absent for half the corpus cannot be compared fairly with a pair that does not. So the matrix uses only the ten checks that return pass, partial or fail for all 739 hosts:

A check counts as a pass only when its status is pass. A partial counts as not passing. That is a choice, and it matters most for E4, where 307 of 739 sites are partial and none fail, so E4 is split almost evenly between pass (432) and partial.

Three numbers describe each pair:

• Conditional pass rate. Of the sites that pass A, the share that also pass B, and the reverse. This is the number a site owner can use, and it is not symmetric.
• Lift. The share of sites passing both, divided by the share you would expect if the two were independent. Lift above 1 means they travel together. It is hard to read when both checks are common, because a check that 74% of sites pass cannot have a lift above 1.35 however tightly it follows another.
• φ (phi). The correlation between two yes-or-no variables, the phi coefficient, which runs from -1 to 1 and is not squeezed by common checks the way lift is. We rank by φ.

Raw φ has a problem we already ran into in the earlier study: mature sites pass nearly everything, so almost any two checks correlate because of the site, not because of each other. To separate the two we repeated the calculation inside four bands of the number of other checks a site passes (0 to 2 of the other eight, 3 to 4, 5 to 6, and 7 to 8) and averaged φ across the bands, weighted by band size. We call that the controlled φ. It is the same idea as the maturity control in the earlier post, applied to pairs. According to Wikipedia's entry on confounding, a third variable that drives both measures is the standard way a raw correlation overstates a link, and site maturity is that variable here.

The control has a known cost. Holding the count of other passes fixed pushes every φ down and can push a pair below zero for purely mechanical reasons, because a site that passes one check of a pair while passing a fixed number of the others is more likely to be failing the second. We treat small controlled values as noise, we only call a pair connected when its controlled φ stays at or above about 0.25, and we checked the one clearly negative pair directly (see the soft 404 section).

Everything here is observational, one scan per host. The scan carries a status for each check, not the reason the check passed, so when we say two checks share a cause we are proposing a mechanism, and we say which ones we could confirm in the stored evidence.

The matrix, strongest pairs first

Of the 45 pairs, the ten strongest by raw φ and two more we want to talk about:

The table has two kinds of strong pair, and the difference matters.

The first kind keeps its strength after the control: D1 with D2, C3 with E3, B2 with C2, E1 with E3, C1 with C2, B2 with C1. These are the bundles.

The second kind loses most of it: C2 with C3 falls from 0.506 to 0.157, C3 with D2 from 0.503 to 0.130, D2 with E3 from 0.492 to 0.107. These pairs are strong in the raw data because a site that passes the rarer check is almost certainly mature and so passes the common one. Look at the conditional columns: 95.8% of the sites with a valid llms.txt also pass D1, and 96.3% of the sites with good structured HTML also pass D2. That is a ladder, not a bundle. The sites that did the harder thing did the easier thing as well, and doing the easier thing does not make the harder one likely: only 60.5% of D1 passers have an llms.txt.

That ladder is useful in its own right. It says which checks to do first. It does not say which checks share a cause.

Bundle one: robots.txt and the sitemap

D1 and D2 are the strongest pair on every measure: 504 sites pass both, 94.7% of D1 passers also pass D2, and the controlled φ of 0.404 is the highest in the matrix. Only 28 sites pass D1 without D2, and only 49 pass D2 without D1.

The probable reason is in the rubric, not in the web. According to the D1 guide, D1 fetches /robots.txt, checks that it returns 200, checks it does not block everything for User-agent: *, and checks that it advertises a Sitemap: URL. D2 then looks for the sitemap at /sitemap.xml and /sitemap_index.xml. A site that writes a Sitemap: line in robots.txt is a site that has a sitemap to point to, and a site with a sitemap at the usual path has usually been through the one-page setup where an SEO plugin writes both. RFC 9309 defines the robots exclusion format, and Google's documentation on creating a robots.txt, fetched on 15 October 2026, lists the sitemap rule as a supported line, so the pairing is the standard pattern, not an AIScan quirk.

We cannot separate "the rubric ties them" from "the web ties them" with this data, because D1 stores one status and not which of its three conditions it failed. In the sitemap study we found that 48 of 60 sampled D2 failures had no sitemap at any common path, which says the file is genuinely missing and not a probing artefact. It does suggest the practical reading: when D2 fails, expect D1 to be failing too, and fix them in one sitting.

On WordPress, ThinkRank is the plugin we recommend for this bundle, because it manages robots.txt, robots meta, the sitemap, schema and llms.txt from one place instead of two or three plugins competing over /robots.txt. It imports settings from Rank Math, Yoast, AIOSEO and SEOPress, so switching does not mean re-entering them. Those plugins also write a sitemap and a Sitemap: line, and the rule that matters is the same for all of them: pick one owner for each file.

Bundle two: the three AI-facing files

B2 (explicit rules for AI crawlers in robots.txt), C2 (/llms.txt) and C1 (Markdown for agents) form a full clique: all three pairs keep a controlled φ between 0.282 and 0.333. They are also the three least common checks in the matrix, passed by 246, 336 and 235 of 739 sites, which is why their lifts (1.68 to 1.85) are the highest.

Two things stand out. Nearly four sites in ten fail all three, and nearly one in five passes all three. The middle is thin, and the biggest slice of it is C2 alone (82 sites). C2 looks like the entry point, and the order in which sites adopt the trio seems to run from llms.txt outward.

The conditional rates confirm it. Of the 246 sites that pass B2, 83.3% pass C2. Of the 336 that pass C2, only 61.0% pass B2. A site that has written explicit rules for AI crawlers has nearly always published an llms.txt too, but publishing an llms.txt does not imply the crawler rules. The same shape appears for C1: 76.6% of C1 passers pass C2, and 53.6% of C2 passers pass C1.

That asymmetry is the practical point. A site owner who has only done llms.txt looks like one of the 82 sites in the C2-only row. The next step the data suggests is the crawler rules in robots.txt, which the 65 sites in the B2-and-C2 row took; the 137 at the top took Markdown as well. The llms.txt format is a proposal and not a standard, a point we made in the Markdown study, so we describe this as an adoption order and not as a recommended one.

For WordPress, ThinkRank generates llms.txt and manages the crawler rules in robots.txt from the same screen, which suits this bundle. For a Shopify store, StoreSEO covers llms.txt, schema and AEO basics. If you want a file today without installing anything, the llms.txt generator writes a valid one, and the guides walk through the crawler rules.

There is a caution inside the bundle. The earlier Markdown study found that 38.1% of C1 passes in its sample returned a non-Markdown content type, because C1 passes on the status code of /index.md. Part of the C1 link to the other two checks may therefore be real adoption and part may be a server answering 200 for everything. The next section shows that second part is not small.

Bundle three: the homepage HTML chain

C3 (structured HTML), E3 (headings and server-rendered text) and E1 (404 handling) are linked, but as a chain. C3 and E3 sit at a controlled φ of 0.360, E1 and E3 at 0.310, and C3 and E1 only at 0.194, below our threshold.

C3 and E3 plausibly share a fetch. According to the content guide, C3 looks at the homepage for a single <h1>, a title, a meta description and a JSON-LD block, and E3 looks at the homepage text for 150 or more words and two or more <h2> headings. If the homepage is a near-empty shell until JavaScript runs, both fail together. Our empty-DOM study found 8.4% of sites failing C3 outright (61 of 739 in this corpus), and the E3 study found that every one of the 116 E3 failures had under 50 words and no <h2>. A server that sends no body text fails both, and one fix (render on the server) clears both.

The conditional rates show the direction: 95.5% of C3 passers also pass E3, while only 67.9% of E3 passers pass C3. E3 asks for words and headings; C3 asks for four metadata signals, one of which is a JSON-LD block that is easy to forget. The 170 sites that pass E3 but not C3 are, in the rubric's terms, sites with readable text and incomplete metadata, and for them the missing piece is often small, such as a JSON-LD block or a meta description.

E1 and E3 link for a different reason, and the next section shows why.

The pair that pulls apart: Markdown and 404 handling

C1 and E1 have a raw φ of -0.072 and a controlled φ of -0.267, the only clearly negative pair of the 45. A controlled value that low could be the mechanical effect described in the method section, so we went to the stored evidence.

Of the sites that pass C1, 26.8% fail E1. Of the sites that fail C1, 3.0% do. And the 63 C1 passers who fail E1 account for 63 of the 78 E1 failures in the corpus: 81% of all sites that fail the 404 check are sites that pass the Markdown check.

The evidence string for E1 on all 63 reads the same way: a probe at /aiscan-404-probe-<random> returned HTTP 200 and the check records a "soft 404", a missing page that answers as if it existed. According to Google's documentation on HTTP status codes and network errors, a soft 404 is a page that says "not found" while returning a success code. A server that answers 200 to a random path also answers 200 to /index.md. C1 passes on the status code of /index.md, so it passes. The C1 evidence agrees: for 55 of the 63, the stored text reads /index.md → 200 (text/html; charset=...), an HTML page and not a Markdown document.

So a visible part of the C1 pass rate in this corpus is a soft-404 server, not Markdown support. We said the same thing from the other side in the Markdown study, where C1 passed on status code regardless of content type; this is the first time we can attach a count to what that costs the E1 column. Passing C1 this way is also not harmless to the site: a server that returns 200 for every unknown path gives crawlers an unlimited supply of fake URLs.

The way out is to fix E1 first. A site that returns a real 404 for unknown paths and then adds a Markdown twin will pass C1 honestly, and the pair will stop pulling apart.

The loners: Link header and HTTPS

D3 (the Link header for discovery) and E4 (HTTPS and canonical host) do not belong to any bundle. Their best controlled φ with any other check is 0.187 for D3 (with C3) and 0.159 for E4 (with D3), against 0.333 for the best bundle pair. Their raw φs look respectable, 0.461 for D3 with C3 and 0.391 for E4 with D2, and almost all of that is maturity.

In plain terms, passing D3 tells you almost nothing about whether the site passes anything else once you account for how complete the site already is. The same goes for E4. Both are server configuration: a header added at the edge, a redirect rule that picks one host. That is a different job from writing a file, and the data agrees that it is a different job.

For the Link header study the finding was that 115 hosts pass D3 on the WordPress API pointer alone. That pointer is added by WordPress itself, which explains why D3 follows the platform and not the site's effort. E4, which has no failures at all and 307 partials, mostly measures whether the apex and www hosts both answer or whether HTTP upgrades to HTTPS; see the canonical host study for the figures.

What a complete bundle looks like in the score

We defined three flags per site: web (D1 and D2 both pass), html (C3, E3 and E1 all pass) and ai (B2, C2 and C1 all pass). Then we grouped the 739 sites by which flags they hold.

The order is clean. Holding no bundle at all is the largest single group (205 sites, 27.7%), and those sites pass 1.7 of the ten checks on average and score 28.0. Holding web and html but not ai is the largest group with two bundles: 185 sites at 61.9. Adding the AI bundle on top lifts the mean to 92.5 across 110 sites, a gap of about 30 points on a bundle that costs three small files.

Two details are worth a second look. First, the ai bundle almost never appears without the web bundle: of the 137 sites with the full ai trio, 133 also hold web. Nobody writes explicit AI crawler rules without a working robots.txt, and the data shows the dependency. Second, the 23 sites in the third row hold web and ai but not html, and they average 82.5. That is the best evidence in this corpus that the AI-facing files carry weight on their own, since those sites are missing at least one of the homepage checks.

As always, this is association. A site that fills in all three bundles is a site that someone is looking after, and the 92.5 average is the mark of that care, not proof that three files caused it. The earlier maturity-controlled study is the place to read the per-check effect with that adjustment applied.

How to use the bundles

Four rules follow from the matrix, and each can be tested against your own scan.

1. Fix a bundle in one sitting. If D1 fails, look at D2 in the same session; the 94.7% conditional rate says the work is usually the same file. If B2 fails and C2 passes, you are in the common middle of the corpus and the crawler rules are the next thing to write.
2. Fix E1 before C1. A soft-404 server can pass C1 without serving Markdown and, in this corpus, does so on 63 sites. A pass that comes from a 200 on every path is not a result you want to defend.
3. Treat D3 and E4 as separate tickets. They belong to the server or the CDN, not to the content, and fixing a file will not move them.
4. Do not read two passing checks as two confirmations. If D1 and D2 both pass, you have learned about one decision, not two. The same goes for C3 and E3.

To check these on your own site by hand, request the paths a scan would request:

curl -s -o /dev/null -w "%{http_code}  robots.txt\n"  https://yoursite.com/robots.txt
curl -s -o /dev/null -w "%{http_code}  sitemap.xml\n" https://yoursite.com/sitemap.xml
curl -s -o /dev/null -w "%{http_code}  llms.txt\n"    https://yoursite.com/llms.txt
curl -s -o /dev/null -w "%{http_code}  index.md\n"    https://yoursite.com/index.md
curl -s -o /dev/null -w "%{http_code}  missing page\n" https://yoursite.com/this-page-does-not-exist-$RANDOM

If the last line prints 200, your server has a soft 404, and the index.md line will print 200 for the same reason. A real 404 on the last line and a 200 on the Markdown line is the honest version of the pair.

What AIScan cannot see

The scan records one status per check, not the reason. It cannot tell us whether D1 failed because the file was missing, blocked everything or lacked a Sitemap: line, so the mechanism in the first bundle is a reading of the rubric and not a measurement. It stores a single homepage fetch for C3 and E3, so we cannot confirm that they share a response, only that their requirements overlap. It does not store response bodies, which is why the soft-404 finding rests on the evidence strings and not on a body parse. And about one site in eight answered one of the probes with a 403, 429 or 5xx; those results count as fails here, which pulls some pairs together because a blocked site fails several checks at once. Our own determinism study puts the flip rate for D1 at 5.9%, so a small part of every figure above is scan-to-scan noise.

Product notes from this work, for the AIScan team: C1 should require a Markdown content type before passing, since 55 of 63 soft-404 passes returned HTML; E1's evidence is a good model for the others because it names the probe path and the status; and D1 should store which of its three conditions failed.

Method

Every figure came from one query pattern: take the newest site-scope scan per host on rubric 2026.08.2, exclude aiscan.site, unnest result->'checks', keep the ten always-scored checks, and self-join on host.

with latest as (
  select distinct on (host) host, score, result
  from (select lower(regexp_replace(regexp_replace(url,'^https?://',''),'/.*$','')) host,
               score, result, created_at
        from scans
        where rubric_version = '2026.08.2' and scope = 'site') s
  where host not like '%aiscan.site%'
  order by host, created_at desc),
c as (
  select l.host, c->>'id' cid, (c->>'status' = 'pass') p
  from latest l, jsonb_array_elements(l.result->'checks') c
  where c->>'id' in ('B2','C1','C2','C3','D1','D2','D3','E1','E3','E4')),
pr as (
  select a.cid ca, b.cid cb, a.p pa, b.p pb
  from c a join c b on a.host = b.host and a.cid < b.cid)
select ca, cb, count(*) n,
       count(*) filter (where pa and pb) both_pass,
       count(*) filter (where pa) a_pass, count(*) filter (where pb) b_pass
from pr group by ca, cb;

Phi is computed from those four counts as (n·both − a·b) ÷ √(a·(n−a)·b·(n−b)). The controlled version adds the number of passes among the other eight checks to the pair table, groups by band, and averages φ across bands weighted by band size. No row-level data was selected and no host is named.

Run it on your own site

Paste your address into aiscan.site or run npx aiscan-cli yoursite.com, and read D1 and D2 together, then B2, C2 and C1 together, then C3, E3 and E1 together. D3 and E4 stand alone. If E1 fails, fix it before you trust a C1 pass. The discoverability, content and bot access check guides explain each fix, and the guides list a walkthrough for every check. Scan again in a week.