Your Sitemap Passes the Check and Still Fails Agents: 669 Sites Measured

A sitemap that returns HTTP 200 is not the same as a sitemap an agent can use. In AIScan's corpus of 669 sites, check D2 passes 499 of them (74.6%), and every one of those passes carries the same evidence string: a URL and "200 OK". We took 90 of the passing sites and opened the file. 87 were real sitemaps, but only 53 (58.9%) were also dated within the last 90 days and linked to pages that all resolved. That 58.9% is the number to fix.

Quick summary

Industry guidance on what makes a sitemap worth fetching: According to Google Search Central, a single sitemap is limited to 50MB uncompressed or 50,000 URLs, Google ignores <priority> and <changefreq>, and it uses <lastmod> only if the value is "consistently and verifiably" accurate. That last condition is the one most sites fail quietly.

Best for a quick check: run the scan, read D2 and D1, then run the three curl commands later in this post. Best for a WordPress site: pick one plugin to own sitemap, robots.txt and llms.txt so they stop contradicting each other. Best for a large site: split by content type and make lastmod come from the content database, not the build clock.

Why a sitemap gets graded as a file and used as a promise

An agent that wants to know what a site contains has two options. It can crawl links from the homepage, which is slow and misses orphaned pages, or it can read the sitemap, which is the site's own list of what matters. The sitemaps.org protocol defines that list, and RFC 9309 lets robots.txt point to it with a Sitemap: line.

The sitemap is useful only if three things hold. The agent can find it. It parses. And what it says is true: the URLs exist and the dates mean something. A check that only tests the first leg gives a site credit for a promise it has not made yet.

AIScan's D2 check is a good example, and we say that about our own check deliberately. Its failure string reads "no /sitemap.xml or /sitemap_index.xml resolved". That names exactly two paths. Its pass string is a URL followed by a status code. Grouping the evidence strings across all 669 sites shows no pass that mentions XML validity, URL counts, dates or link health. D2 is a presence check, and it is honest about being one, but the 74.6% headline reads as more than it is.

This post measures the gap. It extends the earlier piece on llms.txt, robots.txt and sitemap.xml and the job each one does by asking how many of the sitemaps that exist are doing that job. The soft 404 write-up covers a related trap: a path that answers 200 with a web page where a file should be.

The corpus and the three numbers D2 gives

Query: the latest scan of each host on rubric 2026.08.2, aiscan.site excluded, 669 hosts, scans dated 25 August to 1 October 2026.

E5 reports "not applicable" on 377 sites that show no periodical content section, so its denominator is 292, not 669. Of those 292, 175 pass (59.9%), 25 find a feed by convention but never declare it in <head>, and 92 have none.

D1 carries the part of this story that is easy to miss. Its pass string can end "declares Sitemap", which means robots.txt contains a Sitemap: line. That is the discoverability leg, and it comes from a different check than D2. A site can pass one and fail the other.

Two things this dataset is not. It is not a random sample of the web: these are sites someone chose to scan, which skews toward people who care about the question, and toward WordPress and small business sites. And one scan per host means a verdict is a snapshot. Half the point of the next section is how fast that snapshot ages.

The method: open the file, then open the pages

We drew 90 hosts from the 499 D2 passes and 60 from the 170 D2 fails, ordered by an MD5 hash of the hostname so the pick did not depend on scan order. On 1 October 2026 each host got this treatment, with a desktop Chrome user agent, redirects followed:

1. Fetch /robots.txt and collect every Sitemap: line.
2. Fetch /sitemap.xml and /sitemap_index.xml, plus any declared URL that differs.
3. Parse the response. Count it as a sitemap only if it contains a <urlset> or <sitemapindex> element. A 200 that returns HTML does not count.
4. For an index, open the first child sitemap and use that.
5. Record the newest <lastmod>, ignoring any date after 2 October 2026.
6. Pick five URLs at random from the sitemap and request each one. A URL counts as resolving only on 200 after redirects.

The limits are real. One cloud IP means a site that blocks datacentre traffic can look worse than it is; we saw this and name it where it happened. Five URLs per site is a spot check, not a crawl. For indexes we read only the first child sitemap, which can be the oldest one, so index freshness numbers lean low. A lastmod date is a claim the site makes. We did not compare it to the page's real modification time, which is the test Google describes.

Finding 1: the file is usually real

Of 90 passing sites, 87 serve valid sitemap XML. That is the good news, and it is worth stating plainly: a D2 pass is a reliable sign that a sitemap exists.

The 3 that did not parse, and what each looked like:

Half the valid files (44 of 87) are sitemap indexes rather than single files. Opening the first child worked for 43 of them. In the other the child failed to parse. Across the 60 failing sites, 7 more indexes turned up and 2 had a broken first child. So in 3 of 51 sitemap indexes the index existed and its first listed file did not.

A small observation on discovery. 81 of the 90 passing sites (90.0%) declare a sitemap in robots.txt, and 26 of them declare a URL other than the two conventional paths. For 23 of those, the conventional paths work too. For 3, the declared URL is the only place the sitemap lives. An agent that probes only /sitemap.xml misses those 3; an agent that reads robots.txt finds them. That is the argument for doing both.

Finding 2: a third of sitemaps cannot say when anything changed, or say it was long ago

Freshness is where the funnel narrows. Of the 87 valid sitemaps:

15 sites (17.2%) publish a list of URLs with no dates at all. For an agent deciding what to re-fetch, that is a list with no signal: every URL looks equally stale or equally new. The protocol does not require lastmod, so none of these sites is wrong, but each has given up the one field that tells a crawler where to spend its requests. This is the same logic as our earlier measurement of who can answer "nothing changed" with a 304: a site that cannot signal change makes every visit a full download.

Across the whole 87, 58 are fresh by our 90-day rule. The 14 stale ones split in half: 7 are 91 to 365 days old, 7 are older than a year. Stale does not mean wrong. A brochure site that has not changed in 18 months has an honest sitemap. The test is whether the date is accurate, and for a site with a blog and a newest lastmod of two years ago, it is not.

The oldest sitemap in the whole probe, from a site in the failing group, lists 16,741 URLs whose newest lastmod is October 2018. That file is large and confident about a date that cannot be right for a site still publishing in 2026. It is a good picture of the failure: a sitemap generated once and never touched.

Finding 3: the URLs mostly resolve, and the exceptions are instructive

We sampled five URLs from each of 86 valid sitemaps (one valid index had no readable child). 80 sites (93.0%) had all five return 200. That is a better number than the freshness one, and it deserves credit.

The other 6 split into two groups:

The 403 case matters for how you read every measurement here, ours included. A site that lists its own pages in a sitemap and then refuses to serve them to the client that read the sitemap has told an agent where to go and then shut the door. That could be a deliberate rule for unrecognised clients, or a CDN setting nobody reviewed. From outside, the effect is identical. The Cloudflare bot-block change shows how easily the second case arises.

The staging hostname case is the pattern we expected to see more of: a sitemap generated on one environment and deployed with absolute URLs from another. It is a one-line fix in most build tools, and it is invisible to a check that stops at the status code.

The funnel: from "has a sitemap" to "has a sitemap that works"

Sequential survivors among the 90 sites that D2 passed:

The sitemap funnel is the original framework here: discoverable, valid, fresh, resolving. A site clears it only if it survives all four stages. 58.9% is a conservative figure. It treats the 15 undated sitemaps as unproven rather than fresh, and it uses the first child of an index, which tends to be the least recently updated one.

The biggest single drop is not a broken file. It is the 15 sites that never say when anything changed. The smallest is the 3 that no longer serve a sitemap at all. The practical lesson is that the cheapest improvement for most sites is to emit lastmod correctly, not to build or repair a sitemap.

Finding 4: most D2 failures are real, and some are not

We probed 60 of the 170 D2 failures the same way, and added four more paths for the ones with no sitemap: /wp-sitemap.xml, /sitemap-index.xml, /sitemap/sitemap.xml and /sitemaps.xml.

Of those 48, none answered on any of the four alternative paths. That settles a worry we had going in: that D2 might be failing sites whose sitemap lives under a name it does not request. For this sample it was not. WordPress core publishes /wp-sitemap.xml, and not one of the 48 had it. D2's failures here are mostly genuine absences.

The 12 sites with a sitemap now are the interesting part. Their last scans were dated 28 August to 28 September: 9 were scanned before 15 September, and 3 within the last nine days. A site that scanned as a fail on 25 August and has a sitemap on 1 October most likely added one in the meantime, which is a fix and is good news. For the 3 recent ones the explanation is less comfortable: either the site fixed itself within days of the scan, or the scanner met a transient error. We cannot tell which from the evidence stored, and the evidence string, "no /sitemap.xml or /sitemap_index.xml resolved", does not record a status code. That is a limit of D2 worth fixing and we have logged it as a product note.

One more caution for readers who quote a failing D2 to someone else: re-run the scan first. A fail is a snapshot.

What AIScan checks, and what it cannot see

Run the scan first. npx aiscan-cli yoursite.com, or paste your URL at aiscan.site (free, no account). Three checks matter for this post:

Read the three together. A site that passes D2 and D1 has a sitemap and tells agents where it is. Add E5 for sites that publish regularly: a feed is the other freshness signal, and the guide to adding an RSS feed AI agents can find walks through it. Our own check cannot see the four legs of the funnel above. That is why we ran the probe ourselves, and why a clean scan is where this work starts, not where it ends.

If you would rather check by hand, three commands cover it:

# 1. Is it declared? (D1)
curl -sL https://yoursite.com/robots.txt | grep -i '^sitemap:'

# 2. Is it XML, and how many URLs? (D2, plus validity)
curl -sL https://yoursite.com/sitemap.xml | grep -c '<loc>'

# 3. Is it fresh? Newest lastmod in the file
curl -sL https://yoursite.com/sitemap.xml | grep -o '<lastmod>[^<]*' | sort | tail -1

For an index, repeat commands 2 and 3 against each child file it lists. To spot-check the links, take five <loc> values and request each one; anything but 200 is a finding.

How to fix each leg

Not discovered. Add Sitemap: https://yoursite.com/sitemap.xml to robots.txt, with an absolute URL. The line is independent of any User-agent group. If your sitemap lives somewhere unusual, this line is the only way an agent finds it, which is what happened to 3 of our 90 sites.

Not valid. If the URL returns 200 with an HTML page, your router is serving a catch-all page. Return a real 404 for files that do not exist and the XML for the file that does. The soft 404 guide covers the mechanics. Check size too: Google's limit is 50,000 URLs or 50MB uncompressed per file, and larger sites need an index.

Not fresh. Take lastmod from the content record's modification timestamp, not from the time the sitemap was built. Rebuilding on every deploy stamps every URL with today's date, which Google's guidance warns against by requiring dates to be verifiably accurate. If your tooling cannot give a true date, omit the field for those URLs rather than faking it. An absent date is honest; a wrong date trains crawlers to ignore all your dates.

Not resolving. Generate absolute URLs from your production hostname and fail the build if the sitemap host differs from it. Then add a post-deploy check that fetches a sample of the URLs. If a CDN rule returns 403 to non-browser clients, decide on purpose whether the listed pages should be reachable by crawlers, because the sitemap says they should be.

On WordPress. Core already serves /wp-sitemap.xml, but D2 does not request that path, so declare it in robots.txt or alias it to /sitemap.xml. ThinkRank is the plugin we recommend as the default here because it manages sitemap settings, robots.txt, robots meta, schema and llms.txt from one place, so there is one plugin deciding what each file says instead of three fighting over robots.txt, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress. Yoast, Rank Math and AIOSEO each generate sitemaps well, and any of them is fine if it already runs your site, provided only one is writing the file.

On Shopify. The platform generates /sitemap.xml for you, so the work is checking that robots.txt points to it. For stores, StoreSEO is the default recommendation for the half of the stack the platform does not touch: schema, llms.txt and the AEO side.

Why lastmod is the field worth getting right

Google states the rule precisely: the date is used if it is consistently and verifiably accurate, and the value should reflect the last significant change to the page. "Verifiably" is the word that matters. A crawler can request the page, read its Last-Modified header or compare the content to the last copy it saw, and decide whether your date was telling the truth. A site that stamps every URL with the build time fails that comparison on the second visit. After that the crawler has a reason to treat the whole field as noise, including the dates that were right.

You can run the same test on your own site with two requests. Take one URL and its sitemap date:

# date the sitemap claims for one URL
curl -sL https://yoursite.com/sitemap.xml | grep -A2 'yoursite.com/some-post' | grep -o '<lastmod>[^<]*'

# date the page itself reports
curl -sIL https://yoursite.com/some-post | grep -i '^last-modified'

If the two disagree by months, one of them is wrong, and an agent that checks will not guess which. If the page sends no Last-Modified header at all, the sitemap date is unverifiable from outside, which is the position most dynamic sites are in. That is a reason to keep the sitemap date honest, since it may be the only date you publish.

There is a cost argument as well. Our probe fetched five pages per site and that was enough to find broken hosts and bot walls. An agent working at scale fetches far more. Each stale or dateless entry makes it choose between re-fetching everything, which costs you bandwidth and server time, or trusting an old copy, which costs the reader accuracy. Accurate dates let it do neither.

What this study does not claim

Four limits are worth restating, because the headline number is easy to quote without them.

First, 90 passing and 60 failing sites is a spot check. The 58.9% funnel figure has a wide interval at this size, and it describes sites that were scanned, not the web. We would expect the funnel to look worse on a random sample, because people who run a scanner tend to have looked at their discovery files at least once.

Second, five sampled URLs per sitemap finds a site where nothing resolves and misses a site where 3% of a 20,000-URL file is dead. The 93.0% all-five-resolve figure is an upper bound on link health.

Third, we read the first child of each sitemap index. For a WordPress site that is often the posts file or the pages file, depending on the plugin. Either may be the least recently updated, so the freshness numbers for the 44 indexes lean pessimistic.

Fourth, we probed from one cloud address on one day. Three sites refused us outright. Some of those would answer a residential browser, and some would answer an AI crawler that the site has allow-listed. The measurement shows what an unknown client sees, which is the situation most agents are in on first contact.

None of this changes the direction of the findings. A passing D2 is a sound sign that a sitemap exists, and it is a weak sign that the sitemap is useful.

A ten-minute monthly review

The fastest fix we saw in the data was a site that scanned as a fail in late August and serves a valid, dated sitemap now. The slowest was a file that has not been touched since 2018. The difference is whether anyone looks. A review that takes ten minutes:

1. Run the scan and note D2, D1 and E5. Anything that changed since last month is the first thing to read.
2. Fetch robots.txt and confirm every Sitemap: URL returns 200 and parses.
3. Pull the newest lastmod and compare it with your newest published page. They should be within a day or two of each other.
4. Request five URLs from the sitemap, chosen at random, and confirm 200.
5. If you run a CDN or bot filter, repeat step 4 with a non-browser user agent and read the status codes.

Put the commands in a script and run it in CI after each deploy if you can. The guide to gating AI readiness in CI shows how to fail a build when a check regresses.

Common mistakes

Your next action

Run npx aiscan-cli yoursite.com and read D2, D1 and E5 in the discoverability and content results. Then run the three curl commands above. If the newest lastmod is older than your newest post, fix the date source first, because that was the largest drop in our funnel. The full set of fixes is in the AIScan guides, and the llms.txt generator covers the sibling file agents read next.