AIScan.site
Discoverability Checks

Discoverability Checks

D1 robots.txt, D2 XML sitemap, D3 Link header — what each means and how to fix it.

Discoverability answers a simple question: can an agent actually find your pages? If the agent is blocked at the front door or has to guess at your URL structure, nothing else matters.

D1 — robots.txt present & sane (weight 6)

AIScan fetches /robots.txt and checks three things: it returns 200, it doesn't catch-all blockUser-agent: * with Disallow: /, and it advertises your Sitemap: URL.

How to fix

Serve a non-blocking robots file that references your sitemap. Minimum viable:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

D2 — XML sitemap (weight 4)

AIScan looks at /sitemap.xml and /sitemap_index.xml. A valid sitemap lets agents discover every URL without crawling the entire site graph.

How to fix

  • WordPress — Yoast SEO, Rank Math, or core 5.5+ auto-generate one.
  • Shopify — generated automatically at /sitemap.xml.
  • Next.js / TanStack / SPA — generate one at build or via a server route.

Reference it from robots.txt so a single Sitemap: line covers discovery.

D3 — Link header for discovery (weight 3)

AIScan inspects the HTTP Link response header on your homepage. A rel=api-catalog or rel=describedby entry points agents at a machine-readable description of what your site exposes — without them having to guess.

How to fix

Add a header at your CDN or framework level:

Link: </.well-known/api-catalog>; rel="api-catalog"

If you don't have an API catalog yet, this check is informational on content sites — start with P1 API Catalog first.

E1 — Correct 404 handling (weight 4, essential)

AIScan requests a random path that cannot exist and reads the status line. A missing page must answer 404 (or 410). A soft 404 — HTTP 200 for a page that isn't there — teaches agents that every URL on your domain is valid, so they index error pages as content.

We also measure the body: an empty or near-empty 404 is a dead end. Full credit needs a body that names the site and links back to somewhere useful.

How to fix

  • Return a real 404 status from your framework's not-found handler.
  • Render recovery links: homepage, sitemap, docs, search.
  • Verify with curl -sI https://yoursite.com/nope-123 | head -n 1.

E4 — HTTPS & canonical host (weight 4, essential)

Agents resolve bare hostnames. AIScan probes the http:// variant and the opposite www/apex host, then reports the final origin, whether plain HTTP upgrades, whether the alternate host redirects to the canonical one, and whether Strict-Transport-Security is set.

Two live origins means every URL exists twice — duplicated crawl budget and ambiguous citations.

How to fix

# 301 http -> https, and www -> apex (or the reverse; pick one)
Strict-Transport-Security: max-age=31536000; includeSubDomains

An unreachable alternate host is fine — we only penalise a second host that serves its own copy.

Further reading