Link Header Discovery: Only 9.3% of 690 Sites Name an Agent File

Most sites send an HTTP Link header on their homepage. Almost none of them use it to tell an agent anything. On 6 October 2026 we queried AIScan's production database for the D3 check across 690 distinct hosts, one latest scan per host, and sorted every evidence string by what the header was actually spent on. 270 hosts (39.1%) send nothing usable. 361 (52.3%) pass. Among the passes, the number that name an agent discovery relation such as api-catalog, describedby or service-desc in the part of the evidence we can read is 64, or 9.3% of all hosts. The rest pass on something else.

That gap is the post. The Link header is a general-purpose RFC 8288 mechanism, and on the open web it is mostly carrying font preloads, preconnect hints and one WordPress REST API pointer. Our own how-to, Link headers for AI agent discovery, explains how to set the header up. This post is the denominator behind it: how many sites send one, what they put in it, and what that means for anyone reading a D3 result.

Quick summary

The short version of the advice is below, and the numbers that support it follow.

• If your homepage has no Link header, add one with two relations. It costs one line at the CDN or web server.
• If it already has one, read it. A header made of rel=preload and rel=preconnect entries scores partial on D3 and tells an agent nothing about your site.
• If you are on WordPress, you probably already pass D3. That pass does not mean agents can find your llms.txt or API catalog.

Why a header, when you already have files

A crawler that wants to know whether you publish an llms.txt, a sitemap or an API description can guess paths, or it can read what you told it. The Link header is the cheapest place to tell it, because the answer arrives with the first response. No second request, no parsing of HTML, no guessing at /.well-known/ conventions.

Three standards make this work. RFC 8288 (Web Linking, October 2017) defines the Link header and the registry of relation types. RFC 9727 (June 2025) registers the api-catalog relation and the /.well-known/api-catalog location, and shows the relation carried in a Link header on an ordinary response. The llms.txt proposal, in the version it shipped on 10 August 2026 per its own page at llmstxt.org, added rel="describedby" and rel="alternate"; type="text/markdown" as discovery relations, deliverable as HTML <link> tags or HTTP headers.

AIScan's D3 check reads the homepage response. According to the discoverability reference, D3 carries a weight of 3 and inspects the Link response header for an api-catalog or describedby entry. The live remediation text in the scan report adds that an HTML <link rel="api-catalog"> or <link rel="describedby"> tag also counts. That detail matters for the numbers below, because the two delivery routes show up differently in the evidence.

This is the next corpus post on discovery signals. The sitemap study measured D2, and the validator study measured whether sites let agents revalidate cheaply. D3 sits between them: it is the pointer that says where the other files live.

Method

Population. Every host in the AIScan scan table on rubric version 2026.08.2, reduced to one row per host by taking the latest scan, with aiscan.site excluded. That leaves 690 hosts. The population is people who chose to scan a site with AIScan, so it is not a random sample of the web. It skews toward owners who already care about agent readiness, which probably flatters every number in this post.

The query. D3 is one check in the unnested checks array of each stored report. We grouped by the check's status and evidence string. We did not read individual rows and we do not name any scanned site. The SQL shape is:

with latest as (
  select distinct on (lower(regexp_replace(url,'^https?://(www\.)?([^/]+).*$','\2')))
         url, platform, score, result
  from scans
  where rubric_version = '2026.08.2' and url not ilike '%aiscan.site%'
  order by lower(regexp_replace(url,'^https?://(www\.)?([^/]+).*$','\2')), created_at desc
)
select c->>'status' as status, c->>'evidence' as evidence, count(*)
from latest, jsonb_array_elements(result->'checks') c
where c->>'id' = 'D3'
group by 1, 2 order by 3 desc;

The classification. We sorted evidence strings into five groups by pattern. A string starting Link: came from the HTTP header. A string reading <link rel="…"> in HTML head came from the HTML route. Inside header strings, a match on describedby, llms-txt, api-catalog, service-desc, service-doc, sitemap, agent-skills, oauth- or a Markdown alternate counted as an agent discovery relation. A match on api.w.org counted as the WordPress REST pointer. A match on preconnect, dns-prefetch, preload or modulepreload counted as a resource hint.

Limitations, stated before the tables.

1. The evidence string is truncated. AIScan stores roughly the first 200 characters of the header. In 200 of the 257 header rows the string hit that cap. Any relation that appears after the cut is invisible to us. Every count of discovery relations in this post is therefore a lower bound, and the true figure for hosts that name one is higher by an amount we cannot measure from this table.
2. The HTML route does not name its relation. The evidence for a head <link> pass says only <link rel="…"> in HTML head. We cannot tell from the stored string which relation matched, so the 163 hosts that pass this way are counted separately and not classified further.
3. One scanner, one vantage point. AIScan probes from its own infrastructure. A site that varies headers by client would show us one version.
4. Association is not cause. Where we compare scores, we are reporting what sits next to what.

The headline: where 690 hosts land

Of the 690, 257 hosts (37.2%) send an HTTP Link header, and 163 (23.6%) pass through an HTML head tag instead. The remaining 270 send neither: the evidence reads no Link header or <link> tag.

Among the 257 that send a header, 198 pass (77.0%) and 59 are partial (23.0%). No header-sending host failed outright, because a header either carries something the check accepts or it carries only hints, and hints earn partial credit.

So D3 looks healthy at first glance: more than half of hosts pass. The next table is the reason that reading is incomplete.

What the header carries

We split every one of the 690 hosts by what the visible part of its evidence shows.

Four things stand out.

First, the agent discovery pass is small. 64 hosts is 9.3% of the population and 17.7% of the 361 passes. Only about one in eleven of the hosts we scanned tells an agent, in the readable part of its homepage header, where an API catalog, llms.txt, spec or sitemap lives.

Second, the largest pass group we can classify is a WordPress feature. WordPress core adds a header like this to a page response. We confirmed it on a live site on 6 October 2026:

link: <https://wordpress.org/wp-json/>; rel="https://api.w.org/"
link: <https://wordpress.org/wp-json/wp/v2/pages/457>; rel="alternate"; title="JSON"; type="application/json"
link: <https://w.org/>; rel=shortlink

The WordPress REST API handbook documents the api.w.org relation as the way a client discovers the REST root. That is a real, machine-readable pointer. It is also not an api-catalog and not a describedby, so it tells an agent how to talk to WordPress, not what your site offers. The scan report still scores these hosts as a pass, and the evidence shows the REST pointer and nothing else we recognise. 115 hosts (16.7% of the population) sit here. That is nearly twice the size of the group that names an agent relation.

We are not calling this a bug without more evidence. The check's own remediation text allows more than the documentation page describes, and the stored evidence cannot tell us which relation earned the points when a header carries several. What we can say is that a D3 pass and "an agent can find your llms.txt from the header" are two different statements, and the population shows how far apart they are.

Third, resource hints dominate the failures to reach full credit. 57 hosts send a Link header made entirely of preload, preconnect or modulepreload entries in the visible text, and all 57 score partial. The preload headers are not wrong. They are spending a discovery mechanism on performance. A representative evidence shape from the Framer family of sites is a pair of preconnect entries for a CDN host and nothing else. A Wix-hosted site's header is a column of preconnect entries for static asset domains. Next.js sites in the corpus send preload entries for font files and a stylesheet.

Fourth, six hosts pass on hints-only evidence. We read these as a truncation artefact. The visible 200 characters show only fonts, and a later entry in the same header probably carries an accepted relation. The 57 partials and the 6 passes have the same visible shape, which is why we do not trust the visible text alone for any single row. It is another reason the 64 is a floor.

Which relations show up

Across all 690 evidence strings, counting mentions in the visible text of the header (and so a lower bound):

The counts overlap, because a header can name several relations. A typical agent-aware header in the corpus looks like this, and several hosts send close to exactly this:

Link: </.well-known/api-catalog>; rel="api-catalog", </llms.txt>; rel="describedby"; type="text/plain"

Some go further. A handful add service-desc pointing at an OpenAPI file, and one site names an agent-skills index and an oauth-authorization-server document in the same header. One uses the older llms-txt relation name instead of describedby. A few send a Markdown alternate link, which is the other relation llms.txt v2 asks for.

api-catalog leads the discovery relations at 45 hosts, ahead of describedby at 39. That ordering is notable, because AIScan's own documentation puts it first and a lot of the guidance that circulates for agent readiness leads with llms.txt. Sites that have taken the trouble to add a header have more often chosen the API catalog. We cannot tell from this data whether that reflects what owners want or what a particular plugin, template or hosting default sends for them.

Sitemaps are a rounding error. Only 7 hosts advertise a sitemap through the Link header. The sitemap's usual home is a Sitemap: line in robots.txt, which D1 already checks, so a low number here is not a gap.

Platform: who gets to a pass without trying

AIScan labels each scan with a detected platform. The table below covers every platform with at least eight hosts. The six listed account for 663 of the 690 hosts.

WordPress passes almost by default. 172 of 184 hosts pass, and 125 of those 184 send a header, nearly all of it the REST pointer. Add the 163 hosts that pass through a head tag and it becomes clear how much of the population's D3 pass rate is WordPress core doing something for free.

Next.js is the split case. 66 pass, 23 are partial and 68 fail. The 23 partials include font and stylesheet preload headers that the framework or its host sends on its own. A Next.js site sends a Link header by default and spends it on performance, which is the exact pattern the how-to warns about. Fixing it means adding two entries to a header the site already sends, usually in next.config.js headers or at the CDN, not creating a new one. Our Next.js setup guide covers the framework side.

Shopify has the highest partial share. Of 17 hosts, 8 are partial, and the visible evidence is preconnect entries for cdn.shopify.com and similar. Shopify storefronts do not offer, to our knowledge, a merchant-facing setting for response headers, so for a store owner the file route is the practical one: StoreSEO generates llms.txt and schema for Shopify stores, and the <link> tag route is the one a theme can reach. The sample is 17 hosts, so treat the split as direction, not a rate.

The undetected group fails most. 174 of 260 (66.9%) send nothing. These are the static sites, custom builds and hosts where AIScan found no platform fingerprint, and they are also the majority of the failures: 174 of the 270 failing hosts. If you run a hand-built site, you are the likeliest to fail D3 and the likeliest to be able to fix it in one line.

The score gap

Pass, partial and fail sit at mean overall scores of 71.8, 48.8 and 34.6. That is a 37-point gap between pass and fail, and D3 itself is worth only 3 points of weight. The gap is not D3's doing. A host that has bothered to publish discovery pointers has usually done other work too, and hosts that fail D3 may well be hosts that fail other checks too. Treat D3 as a marker of a maintained site, and not as a lever that lifts a score 37 points on its own.

The partial group is informative the other way. A partial host averages 48.8, about 14 points above a failing host and 23 below a passing one. That puts the sites that spend their header on hints in the middle: they send headers, which is deliberate work, but the work went to performance, not discovery.

For a prioritised view of which checks move a score most, see the one agent-readiness fix that moves the score most. D3 carries a weight of 3, so we would not move it ahead of the heavier checks.

What this means if you run a site

The measured pattern suggests three groups of readers.

You send no header (270 of 690). This is the easiest case. Add a Link header with two entries on the homepage response, an api-catalog relation pointing at /.well-known/api-catalog and a describedby relation pointing at /llms.txt. If the files do not exist yet, the header points at nothing, so create them first. The API catalog how-to covers the file, and the llms.txt generator produces one. A minimal header is:

Link: </.well-known/api-catalog>; rel="api-catalog", </llms.txt>; rel="describedby"; type="text/plain"

You send a header of hints (57 of 690). Keep the hints and append the discovery relations. A header can carry any number of comma-separated entries. The partial status in the corpus does not come from sending hints. It comes from sending only hints. Put the two discovery entries first, so that a client or a log line that truncates the value still sees them. Our own truncation problem in this study is a reminder that the front of a header is the part that gets read.

You pass through the WordPress REST pointer (115 of 690). You are not broken, but you are not discoverable either. If you run WordPress, ThinkRank is the plugin we recommend for this job, because it manages llms.txt, robots.txt, robots meta, schema and sitemaps from one plugin instead of three that each try to own the same files, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. ThinkRank gives you the files the header should point at. The header itself is a web server or CDN setting, so add it in your server config or CDN rule as described in the Link header how-to. Rank Math, Yoast and AIOSEO also generate sitemaps and some of these files; if you already use one, what matters is that exactly one plugin writes each file. The WordPress setup guide covers the full path.

You pass through a head <link> tag (163 of 690). This is a legitimate route and the check accepts it. Open your page source and confirm the tag names api-catalog or describedby, because the stored evidence does not say which relation matched. If it names something else, such as canonical or alternate, the pass deserves a second look.

What the header should not become

Two traps showed up in the corpus.

A header that carries only performance hints. This is the 57-host partial group. It is a common pattern because frameworks and CDNs add preload entries automatically, and because the same header field serves two audiences with different needs.

A header that points at files that do not exist. We did not measure this in the 690, because D3 does not follow the targets. It is the natural next step in this series, since a pointer to a 404 is worse than no pointer: the agent trusts the header and gets nothing. The same soft-404 problem is described in the soft-404 discovery post. Until we publish a follow-up, check by hand that each target returns 200 with the right content type.

Where AIScan fits, and where it stops

Run npx aiscan-cli yoursite.com, or paste the URL at aiscan.site, and read the D3 line in the report. Free, no account. The evidence string tells you which route matched. Two related checks complete the picture: D1 (robots.txt, which should reference your sitemap) and D2 (the sitemap itself). If you publish an API catalog, the capabilities checks (P1) look at the catalog file itself, which D3 only points at.

What AIScan cannot see:

• Where a relation sits in the header. The scan stores the first part of the string. We showed above how that limits this very study.
• Whether the pointed-at file is correct. D3 confirms the pointer, not the destination.
• Pages other than the homepage. The check reads one response.
• Whether any agent reads the header. We have not verified that any major AI crawler acts on describedby or api-catalog today. These relations are the standards-track way to be found if clients adopt them, and this post measures supply, not demand.

If you would rather check by hand, request your homepage with curl -sI https://yoursite.com/ | grep -i '^link' and read what comes back. Check that discovery entries come before any resource hints, and that each target answers with a 200.

What we are changing in AIScan

This study turned up three issues in our own instrument.

1. Store the whole header. Truncating at about 200 characters made 200 of 257 header rows partly unreadable. Evidence for D3 should list every relation type in the header, even if it shortens each URL.
2. Name the relation on the HTML route. <link rel="…"> in HTML head should say which relation matched, so that a head-tag pass can be audited the way a header pass can.
3. Say why a pass passed. A WordPress REST pointer and an api-catalog entry both read as pass. The report should show which entries earned the credit and which were ignored.

We will re-run the same query after the evidence format changes and publish how much the 64-host floor moves.

Run it on your own site

Scan your site and read D3. If it says no Link header or <link> tag, add the two-entry header above. If it reads like a list of fonts, put your discovery entries first. If you are on WordPress and it passes on the REST pointer, confirm that your llms.txt and API catalog exist and add the header at the server. Then scan again, and read D1 and D2 while you are there.

The reference page for these checks is discoverability checks. The how-to guides, including the Link header setup, API catalog and llms.txt guides, are at /guides. For the wider picture of how sites score across all checks, see the state of AI agent readiness in 2026.