---
title: "Link Header Discovery: Only 9.3% of 690 Sites Name an Agent File"
slug: link-header-discovery-corpus
published: 2026-10-06T03:16:18.325827+00:00
updated: 2026-10-06T03:16:18.325827+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: check:D3, check:C2, check:P1, corpus study, link header, RFC 8288
description: "We sorted the Link header of 690 sites by what it carries. 52.3% pass D3, but only 9.3% name an agent discovery file; most spend it on fonts. See the fix."
url: https://aiscan.site/blog/link-header-discovery-corpus
---
Most sites send an HTTP `Link` header on their homepage. Almost none of them use it to tell an agent anything. On 6 October 2026 we queried AIScan's production database for the D3 check across 690 distinct hosts, one latest scan per host, and sorted every evidence string by what the header was actually spent on. 270 hosts (39.1%) send nothing usable. 361 (52.3%) pass. Among the passes, the number that name an agent discovery relation such as `api-catalog`, `describedby` or `service-desc` in the part of the evidence we can read is 64, or 9.3% of all hosts. The rest pass on something else.
That gap is the post. The `Link` header is a general-purpose RFC 8288 mechanism, and on the open web it is mostly carrying font preloads, preconnect hints and one WordPress REST API pointer. Our own how-to, [Link headers for AI agent discovery](/blog/link-headers-for-agent-discovery), explains how to set the header up. This post is the denominator behind it: how many sites send one, what they put in it, and what that means for anyone reading a D3 result.
## Quick summary
| If you want to know... | The measured answer | Where to look |
|---|---|---|
| How many sites pass D3? | 361 of 690 (52.3%); 59 partial (8.6%); 270 fail (39.1%) | Headline table |
| How many send an HTTP `Link` header at all? | 257 of 690 (37.2%) | Headline table |
| How many name an agent discovery relation? | 64 of 690 (9.3%), or 24.9% of the 257 that send a header | What the header carries |
| What do most headers actually contain? | Resource hints: preload, preconnect, modulepreload | What the header carries |
| Which platform passes most? | WordPress, 172 of 184 (93.5%), mostly on a REST API pointer | Platform table |
| Which platform fails most? | Hosts AIScan could not attribute to a platform, 174 of 260 (66.9%) | Platform table |
| Does D3 track the overall score? | Passing hosts average 71.8, failing hosts 34.6 | Score section |
| What should I send? | `api-catalog` and `describedby` first, then `service-desc` if you have a spec | Fix section |
| Can I check it myself? | Yes: `npx aiscan-cli yoursite.com`, check **D3** | Run it on your site |
The short version of the advice is below, and the numbers that support it follow.
- If your homepage has no `Link` header, add one with two relations. It costs one line at the CDN or web server.
- If it already has one, read it. A header made of `rel=preload` and `rel=preconnect` entries scores **partial** on D3 and tells an agent nothing about your site.
- If you are on WordPress, you probably already pass D3. That pass does not mean agents can find your llms.txt or API catalog.
## Why a header, when you already have files
A crawler that wants to know whether you publish an llms.txt, a sitemap or an API description can guess paths, or it can read what you told it. The `Link` header is the cheapest place to tell it, because the answer arrives with the first response. No second request, no parsing of HTML, no guessing at `/.well-known/` conventions.
Three standards make this work. [RFC 8288](https://www.rfc-editor.org/rfc/rfc8288) (Web Linking, October 2017) defines the `Link` header and the registry of relation types. [RFC 9727](https://www.rfc-editor.org/rfc/rfc9727) (June 2025) registers the `api-catalog` relation and the `/.well-known/api-catalog` location, and shows the relation carried in a `Link` header on an ordinary response. The llms.txt proposal, in the version it shipped on 10 August 2026 per its own page at [llmstxt.org](https://llmstxt.org/), added `rel="describedby"` and `rel="alternate"; type="text/markdown"` as discovery relations, deliverable as HTML `` tags or HTTP headers.
AIScan's D3 check reads the homepage response. According to the [discoverability reference](/docs/checks/discoverability), D3 carries a weight of 3 and inspects the `Link` response header for an `api-catalog` or `describedby` entry. The live remediation text in the scan report adds that an HTML `` or `` tag also counts. That detail matters for the numbers below, because the two delivery routes show up differently in the evidence.
This is the next corpus post on discovery signals. [The sitemap study](/blog/sitemap-reality-corpus-study) measured D2, and [the validator study](/blog/etag-last-modified-adoption-corpus) measured whether sites let agents revalidate cheaply. D3 sits between them: it is the pointer that says where the other files live.
## Method
**Population.** Every host in the AIScan scan table on rubric version 2026.08.2, reduced to one row per host by taking the latest scan, with `aiscan.site` excluded. That leaves 690 hosts. The population is people who chose to scan a site with AIScan, so it is not a random sample of the web. It skews toward owners who already care about agent readiness, which probably flatters every number in this post.
**The query.** D3 is one check in the unnested `checks` array of each stored report. We grouped by the check's status and evidence string. We did not read individual rows and we do not name any scanned site. The SQL shape is:
```sql
with latest as (
select distinct on (lower(regexp_replace(url,'^https?://(www\.)?([^/]+).*$','\2')))
url, platform, score, result
from scans
where rubric_version = '2026.08.2' and url not ilike '%aiscan.site%'
order by lower(regexp_replace(url,'^https?://(www\.)?([^/]+).*$','\2')), created_at desc
)
select c->>'status' as status, c->>'evidence' as evidence, count(*)
from latest, jsonb_array_elements(result->'checks') c
where c->>'id' = 'D3'
group by 1, 2 order by 3 desc;
```
**The classification.** We sorted evidence strings into five groups by pattern. A string starting `Link:` came from the HTTP header. A string reading ` in HTML head` came from the HTML route. Inside header strings, a match on `describedby`, `llms-txt`, `api-catalog`, `service-desc`, `service-doc`, `sitemap`, `agent-skills`, `oauth-` or a Markdown alternate counted as an agent discovery relation. A match on `api.w.org` counted as the WordPress REST pointer. A match on `preconnect`, `dns-prefetch`, `preload` or `modulepreload` counted as a resource hint.
**Limitations, stated before the tables.**
1. **The evidence string is truncated.** AIScan stores roughly the first 200 characters of the header. In 200 of the 257 header rows the string hit that cap. Any relation that appears after the cut is invisible to us. Every count of discovery relations in this post is therefore a **lower bound**, and the true figure for hosts that name one is higher by an amount we cannot measure from this table.
2. **The HTML route does not name its relation.** The evidence for a head `` pass says only ` in HTML head`. We cannot tell from the stored string which relation matched, so the 163 hosts that pass this way are counted separately and not classified further.
3. **One scanner, one vantage point.** AIScan probes from its own infrastructure. A site that varies headers by client would show us one version.
4. **Association is not cause.** Where we compare scores, we are reporting what sits next to what.
## The headline: where 690 hosts land
| D3 status | Hosts | Share of 690 | Mean overall score | Median overall score |
|---|---|---|---|---|
| Pass | 361 | 52.3% | 71.8 | 72 |
| Partial | 59 | 8.6% | 48.8 | 46 |
| Fail | 270 | 39.1% | 34.6 | 35 |
Of the 690, 257 hosts (37.2%) send an HTTP `Link` header, and 163 (23.6%) pass through an HTML head tag instead. The remaining 270 send neither: the evidence reads `no Link header or tag`.
Among the 257 that send a header, 198 pass (77.0%) and 59 are partial (23.0%). No header-sending host failed outright, because a header either carries something the check accepts or it carries only hints, and hints earn partial credit.
So D3 looks healthy at first glance: more than half of hosts pass. The next table is the reason that reading is incomplete.
## What the header carries
We split every one of the 690 hosts by what the visible part of its evidence shows.
| What D3 saw | Status | Hosts | Share of 690 |
|---|---|---|---|
| Nothing (`no Link header or tag`) | Fail | 270 | 39.1% |
| HTML head ``, relation not named | Pass | 163 | 23.6% |
| WordPress REST pointer (`api.w.org`), no agent relation visible | Pass | 115 | 16.7% |
| An agent discovery relation named in the header | Pass | 64 | 9.3% |
| Other header content, no relation we classify | Pass | 13 | 1.9% |
| Resource hints only in the visible text | Pass | 6 | 0.9% |
| Resource hints only in the visible text | Partial | 57 | 8.3% |
| Other or WordPress-only | Partial | 2 | 0.3% |
Four things stand out.
**First, the agent discovery pass is small.** 64 hosts is 9.3% of the population and 17.7% of the 361 passes. Only about one in eleven of the hosts we scanned tells an agent, in the readable part of its homepage header, where an API catalog, llms.txt, spec or sitemap lives.
**Second, the largest pass group we can classify is a WordPress feature.** WordPress core adds a header like this to a page response. We confirmed it on a live site on 6 October 2026:
```
link: ; rel="https://api.w.org/"
link: ; rel="alternate"; title="JSON"; type="application/json"
link: ; rel=shortlink
```
The [WordPress REST API handbook](https://developer.wordpress.org/rest-api/) documents the `api.w.org` relation as the way a client discovers the REST root. That is a real, machine-readable pointer. It is also not an `api-catalog` and not a `describedby`, so it tells an agent how to talk to WordPress, not what your site offers. The scan report still scores these hosts as a pass, and the evidence shows the REST pointer and nothing else we recognise. 115 hosts (16.7% of the population) sit here. That is nearly twice the size of the group that names an agent relation.
We are not calling this a bug without more evidence. The check's own remediation text allows more than the documentation page describes, and the stored evidence cannot tell us which relation earned the points when a header carries several. What we can say is that a D3 pass and "an agent can find your llms.txt from the header" are two different statements, and the population shows how far apart they are.
**Third, resource hints dominate the failures to reach full credit.** 57 hosts send a `Link` header made entirely of preload, preconnect or modulepreload entries in the visible text, and all 57 score partial. The preload headers are not wrong. They are spending a discovery mechanism on performance. A representative evidence shape from the Framer family of sites is a pair of `preconnect` entries for a CDN host and nothing else. A Wix-hosted site's header is a column of `preconnect` entries for static asset domains. Next.js sites in the corpus send `preload` entries for font files and a stylesheet.
**Fourth, six hosts pass on hints-only evidence.** We read these as a truncation artefact. The visible 200 characters show only fonts, and a later entry in the same header probably carries an accepted relation. The 57 partials and the 6 passes have the same visible shape, which is why we do not trust the visible text alone for any single row. It is another reason the 64 is a floor.
## Which relations show up
Across all 690 evidence strings, counting mentions in the visible text of the header (and so a lower bound):
| Relation | Hosts naming it | Share of 690 |
|---|---|---|
| `api-catalog` | 45 | 6.5% |
| `describedby` or `llms-txt` | 39 | 5.7% |
| `service-desc` | 19 | 2.8% |
| `sitemap` | 7 | 1.0% |
| `https://api.w.org/` | 121 | 17.5% |
The counts overlap, because a header can name several relations. A typical agent-aware header in the corpus looks like this, and several hosts send close to exactly this:
```
Link: ; rel="api-catalog", ; rel="describedby"; type="text/plain"
```
Some go further. A handful add `service-desc` pointing at an OpenAPI file, and one site names an `agent-skills` index and an `oauth-authorization-server` document in the same header. One uses the older `llms-txt` relation name instead of `describedby`. A few send a Markdown `alternate` link, which is the other relation llms.txt v2 asks for.
`api-catalog` leads the discovery relations at 45 hosts, ahead of `describedby` at 39. That ordering is notable, because AIScan's own documentation puts it first and a lot of the guidance that circulates for agent readiness leads with llms.txt. Sites that have taken the trouble to add a header have more often chosen the API catalog. We cannot tell from this data whether that reflects what owners want or what a particular plugin, template or hosting default sends for them.
Sitemaps are a rounding error. Only 7 hosts advertise a sitemap through the `Link` header. The sitemap's usual home is a `Sitemap:` line in robots.txt, which D1 already checks, so a low number here is not a gap.
## Platform: who gets to a pass without trying
AIScan labels each scan with a detected platform. The table below covers every platform with at least eight hosts. The six listed account for 663 of the 690 hosts.
| Platform | Hosts | Pass | Partial | Fail | Send a header | Mean score |
|---|---|---|---|---|---|---|
| Not detected | 260 | 70 (26.9%) | 16 | 174 (66.9%) | 39 | 46.0 |
| WordPress | 184 | 172 (93.5%) | 2 | 10 | 125 | 66.2 |
| Next.js | 157 | 66 (42.0%) | 23 | 68 (43.3%) | 61 | 52.1 |
| Astro | 34 | 25 (73.5%) | 0 | 9 | 8 | 67.4 |
| Shopify | 17 | 8 (47.1%) | 8 | 1 | 9 | 66.6 |
| Lovable | 11 | 9 (81.8%) | 0 | 2 | 3 | 77.2 |
**WordPress passes almost by default.** 172 of 184 hosts pass, and 125 of those 184 send a header, nearly all of it the REST pointer. Add the 163 hosts that pass through a head tag and it becomes clear how much of the population's D3 pass rate is WordPress core doing something for free.
**Next.js is the split case.** 66 pass, 23 are partial and 68 fail. The 23 partials include font and stylesheet preload headers that the framework or its host sends on its own. A Next.js site sends a `Link` header by default and spends it on performance, which is the exact pattern the how-to warns about. Fixing it means adding two entries to a header the site already sends, usually in `next.config.js` headers or at the CDN, not creating a new one. Our [Next.js setup guide](/blog/ai-readiness-setup-nextjs) covers the framework side.
**Shopify has the highest partial share.** Of 17 hosts, 8 are partial, and the visible evidence is `preconnect` entries for `cdn.shopify.com` and similar. Shopify storefronts do not offer, to our knowledge, a merchant-facing setting for response headers, so for a store owner the file route is the practical one: [StoreSEO](https://storeseo.com/) generates llms.txt and schema for Shopify stores, and the `` tag route is the one a theme can reach. The sample is 17 hosts, so treat the split as direction, not a rate.
**The undetected group fails most.** 174 of 260 (66.9%) send nothing. These are the static sites, custom builds and hosts where AIScan found no platform fingerprint, and they are also the majority of the failures: 174 of the 270 failing hosts. If you run a hand-built site, you are the likeliest to fail D3 and the likeliest to be able to fix it in one line.
## The score gap
Pass, partial and fail sit at mean overall scores of 71.8, 48.8 and 34.6. That is a 37-point gap between pass and fail, and D3 itself is worth only 3 points of weight. The gap is not D3's doing. A host that has bothered to publish discovery pointers has usually done other work too, and hosts that fail D3 may well be hosts that fail other checks too. Treat D3 as a **marker** of a maintained site, and not as a lever that lifts a score 37 points on its own.
The partial group is informative the other way. A partial host averages 48.8, about 14 points above a failing host and 23 below a passing one. That puts the sites that spend their header on hints in the middle: they send headers, which is deliberate work, but the work went to performance, not discovery.
For a prioritised view of which checks move a score most, see [the one agent-readiness fix that moves the score most](/blog/agent-readiness-highest-impact-fix). D3 carries a weight of 3, so we would not move it ahead of the heavier checks.
## What this means if you run a site
The measured pattern suggests three groups of readers.
**You send no header (270 of 690).** This is the easiest case. Add a `Link` header with two entries on the homepage response, an `api-catalog` relation pointing at `/.well-known/api-catalog` and a `describedby` relation pointing at `/llms.txt`. If the files do not exist yet, the header points at nothing, so create them first. The [API catalog how-to](/blog/publish-api-catalog-rfc-9727) covers the file, and the [llms.txt generator](/llms-txt-generator) produces one. A minimal header is:
```
Link: ; rel="api-catalog", ; rel="describedby"; type="text/plain"
```
**You send a header of hints (57 of 690).** Keep the hints and append the discovery relations. A header can carry any number of comma-separated entries. The partial status in the corpus does not come from sending hints. It comes from sending only hints. Put the two discovery entries first, so that a client or a log line that truncates the value still sees them. Our own truncation problem in this study is a reminder that the front of a header is the part that gets read.
**You pass through the WordPress REST pointer (115 of 690).** You are not broken, but you are not discoverable either. If you run WordPress, [ThinkRank](https://thinkrank.ai) is the plugin we recommend for this job, because it manages llms.txt, robots.txt, robots meta, schema and sitemaps from one plugin instead of three that each try to own the same files, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. ThinkRank gives you the files the header should point at. The header itself is a web server or CDN setting, so add it in your server config or CDN rule as described in the [Link header how-to](/blog/link-headers-for-agent-discovery). Rank Math, Yoast and AIOSEO also generate sitemaps and some of these files; if you already use one, what matters is that exactly one plugin writes each file. The [WordPress setup guide](/blog/ai-readiness-setup-wordpress) covers the full path.
**You pass through a head `` tag (163 of 690).** This is a legitimate route and the check accepts it. Open your page source and confirm the tag names `api-catalog` or `describedby`, because the stored evidence does not say which relation matched. If it names something else, such as `canonical` or `alternate`, the pass deserves a second look.
## What the header should not become
Two traps showed up in the corpus.
**A header that carries only performance hints.** This is the 57-host partial group. It is a common pattern because frameworks and CDNs add preload entries automatically, and because the same header field serves two audiences with different needs.
**A header that points at files that do not exist.** We did not measure this in the 690, because D3 does not follow the targets. It is the natural next step in this series, since a pointer to a 404 is worse than no pointer: the agent trusts the header and gets nothing. The same soft-404 problem is described in [the soft-404 discovery post](/blog/soft-404-agent-discovery-problem). Until we publish a follow-up, check by hand that each target returns 200 with the right content type.
## Where AIScan fits, and where it stops
Run `npx aiscan-cli yoursite.com`, or paste the URL at [aiscan.site](https://aiscan.site/), and read the **D3** line in the report. Free, no account. The evidence string tells you which route matched. Two related checks complete the picture: **D1** (robots.txt, which should reference your sitemap) and **D2** (the sitemap itself). If you publish an API catalog, the capabilities checks (**P1**) look at the catalog file itself, which D3 only points at.
What AIScan cannot see:
- **Where a relation sits in the header.** The scan stores the first part of the string. We showed above how that limits this very study.
- **Whether the pointed-at file is correct.** D3 confirms the pointer, not the destination.
- **Pages other than the homepage.** The check reads one response.
- **Whether any agent reads the header.** We have not verified that any major AI crawler acts on `describedby` or `api-catalog` today. These relations are the standards-track way to be found if clients adopt them, and this post measures supply, not demand.
If you would rather check by hand, request your homepage with `curl -sI https://yoursite.com/ | grep -i '^link'` and read what comes back. Check that discovery entries come before any resource hints, and that each target answers with a 200.
## What we are changing in AIScan
This study turned up three issues in our own instrument.
1. **Store the whole header.** Truncating at about 200 characters made 200 of 257 header rows partly unreadable. Evidence for D3 should list every relation type in the header, even if it shortens each URL.
2. **Name the relation on the HTML route.** ` in HTML head` should say which relation matched, so that a head-tag pass can be audited the way a header pass can.
3. **Say why a pass passed.** A WordPress REST pointer and an `api-catalog` entry both read as pass. The report should show which entries earned the credit and which were ignored.
We will re-run the same query after the evidence format changes and publish how much the 64-host floor moves.
## Run it on your own site
Scan your site and read **D3**. If it says `no Link header or tag`, add the two-entry header above. If it reads like a list of fonts, put your discovery entries first. If you are on WordPress and it passes on the REST pointer, confirm that your llms.txt and API catalog exist and add the header at the server. Then scan again, and read **D1** and **D2** while you are there.
The reference page for these checks is [discoverability checks](/docs/checks/discoverability). The how-to guides, including the Link header setup, API catalog and llms.txt guides, are at [/guides](/guides). For the wider picture of how sites score across all checks, see [the state of AI agent readiness in 2026](/blog/state-of-ai-agent-readiness-2026).