--- title: "Your Sitemap Passes the Check and Still Fails Agents: 669 Sites Measured" slug: sitemap-reality-corpus-study published: 2026-10-01T03:19:02.345827+00:00 updated: 2026-10-04T12:03:13.561784+00:00 author: "AIScan Content Team" author_url: https://aiscan.site/about category: "AI Readiness" tags: check:D2, check:D1, check:E5, sitemap, robots-txt, corpus-study description: "AIScan's D2 passes 74.6% of 669 sites on a 200 status alone. We opened 90 passing sitemaps: only 53 were valid, dated within 90 days and linked to live pages." url: https://aiscan.site/blog/sitemap-reality-corpus-study --- A sitemap that returns HTTP 200 is not the same as a sitemap an agent can use. In AIScan's corpus of 669 sites, check D2 passes 499 of them (74.6%), and every one of those passes carries the same evidence string: a URL and "200 OK". We took 90 of the passing sites and opened the file. 87 were real sitemaps, but only 53 (58.9%) were also dated within the last 90 days and linked to pages that all resolved. That 58.9% is the number to fix. ## Quick summary | Question | What we measured | Result | |---|---|---| | How many sites pass D2 (XML sitemap)? | 669 sites, rubric 2026.08.2, latest scan per host, aiscan.site excluded | 499 pass (74.6%), 170 fail (25.4%) | | What does a D2 pass assert? | Evidence strings, grouped | A URL and "200 OK", nothing else | | Is the passing file a sitemap? | 90 passing sites re-probed on 1 October 2026 | 87 serve valid sitemap XML (96.7%) | | Does it say when pages changed? | The same 87 sites | 72 carry `lastmod` (82.8%); 15 carry none | | Is it fresh? | Newest `lastmod`, same 87 | 58 within 90 days; 14 older (7 of them over a year) | | Do its URLs resolve? | 5 random URLs per sitemap, 86 sampled | 80 sites had all 5 return 200 (93.0%) | | Discoverable, valid, fresh and resolving at once | The funnel below | **53 of 90 (58.9%)** | | Are D2 failures real? | 60 failing sites re-probed | 48 (80%) serve no sitemap on any path we tried; 12 (20%) now serve one | **Industry guidance on what makes a sitemap worth fetching:** [**According to Google Search Central**](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap), a single sitemap is limited to 50MB uncompressed or 50,000 URLs, Google ignores `` and ``, and it uses `` only if the value is "consistently and verifiably" accurate. That last condition is the one most sites fail quietly. **Best for a quick check:** run the scan, read D2 and D1, then run the three curl commands later in this post. **Best for a WordPress site:** pick one plugin to own sitemap, robots.txt and llms.txt so they stop contradicting each other. **Best for a large site:** split by content type and make `lastmod` come from the content database, not the build clock. ## Why a sitemap gets graded as a file and used as a promise An agent that wants to know what a site contains has two options. It can crawl links from the homepage, which is slow and misses orphaned pages, or it can read the sitemap, which is the site's own list of what matters. [**The sitemaps.org protocol**](https://www.sitemaps.org/protocol.html) defines that list, and [**RFC 9309**](https://www.rfc-editor.org/rfc/rfc9309.html) lets robots.txt point to it with a `Sitemap:` line. The sitemap is useful only if three things hold. The agent can find it. It parses. And what it says is true: the URLs exist and the dates mean something. A check that only tests the first leg gives a site credit for a promise it has not made yet. [**AIScan's D2 check**](/docs/checks/discoverability) is a good example, and we say that about our own check deliberately. Its failure string reads "no /sitemap.xml or /sitemap_index.xml resolved". That names exactly two paths. Its pass string is a URL followed by a status code. Grouping the evidence strings across all 669 sites shows no pass that mentions XML validity, URL counts, dates or link health. D2 is a presence check, and it is honest about being one, but the 74.6% headline reads as more than it is. This post measures the gap. It extends the earlier piece on [llms.txt, robots.txt and sitemap.xml and the job each one does](/blog/llms-txt-vs-robots-txt-vs-sitemap) by asking how many of the sitemaps that exist are doing that job. The [soft 404 write-up](/blog/soft-404-agent-discovery-problem) covers a related trap: a path that answers 200 with a web page where a file should be. ## The corpus and the three numbers D2 gives Query: the latest scan of each host on rubric 2026.08.2, aiscan.site excluded, 669 hosts, scans dated 25 August to 1 October 2026. | Check | Pass | Partial | Fail | Evidence on a pass | |---|---|---|---|---| | D2, XML sitemap | 499 (74.6%) | none | 170 (25.4%) | A URL and "200 OK" | | D1, robots.txt present and sane | 479 (71.6%) | 86 (12.9%) | 104 (15.5%) | Status, byte size, "declares Sitemap" | | E5, content feed | 175 (26.2%) | 25 (3.7%) | 92 (13.8%) | Feed URL and how it was found | E5 reports "not applicable" on 377 sites that show no periodical content section, so its denominator is 292, not 669. Of those 292, 175 pass (59.9%), 25 find a feed by convention but never declare it in ``, and 92 have none. D1 carries the part of this story that is easy to miss. Its pass string can end "declares Sitemap", which means robots.txt contains a `Sitemap:` line. That is the discoverability leg, and it comes from a different check than D2. A site can pass one and fail the other. Two things this dataset is not. It is not a random sample of the web: these are sites someone chose to scan, which skews toward people who care about the question, and toward WordPress and small business sites. And one scan per host means a verdict is a snapshot. Half the point of the next section is how fast that snapshot ages. ## The method: open the file, then open the pages We drew 90 hosts from the 499 D2 passes and 60 from the 170 D2 fails, ordered by an MD5 hash of the hostname so the pick did not depend on scan order. On 1 October 2026 each host got this treatment, with a desktop Chrome user agent, redirects followed: 1. Fetch `/robots.txt` and collect every `Sitemap:` line. 2. Fetch `/sitemap.xml` and `/sitemap_index.xml`, plus any declared URL that differs. 3. Parse the response. Count it as a sitemap only if it contains a `` or `` element. A 200 that returns HTML does not count. 4. For an index, open the first child sitemap and use that. 5. Record the newest ``, ignoring any date after 2 October 2026. 6. Pick five URLs at random from the sitemap and request each one. A URL counts as resolving only on 200 after redirects. The limits are real. One cloud IP means a site that blocks datacentre traffic can look worse than it is; we saw this and name it where it happened. Five URLs per site is a spot check, not a crawl. For indexes we read only the first child sitemap, which can be the oldest one, so index freshness numbers lean low. A `lastmod` date is a claim the site makes. We did not compare it to the page's real modification time, which is the test Google describes. ## Finding 1: the file is usually real Of 90 passing sites, 87 serve valid sitemap XML. That is the good news, and it is worth stating plainly: a D2 pass is a reliable sign that a sitemap exists. The 3 that did not parse, and what each looked like: | Site | What we saw | Reading | |---|---|---| | A | HTTP 403 on both conventional paths | A bot wall on our IP; the file passed at scan time. We do not count this as dead | | B | `/sitemap.xml` and `/sitemap_index.xml` return 404; robots.txt declares a URL with a `.xmls` extension | A typo in the declared path, and the file at the conventional path is gone | | C | `/sitemap_index.xml` returns 200 with a web page, not XML | A soft 404: the status passes, the content is not a sitemap | Half the valid files (44 of 87) are sitemap indexes rather than single files. Opening the first child worked for 43 of them. In the other the child failed to parse. Across the 60 failing sites, 7 more indexes turned up and 2 had a broken first child. So in 3 of 51 sitemap indexes the index existed and its first listed file did not. A small observation on discovery. 81 of the 90 passing sites (90.0%) declare a sitemap in robots.txt, and 26 of them declare a URL other than the two conventional paths. For 23 of those, the conventional paths work too. For 3, the declared URL is the only place the sitemap lives. An agent that probes only `/sitemap.xml` misses those 3; an agent that reads robots.txt finds them. That is the argument for doing both. ## Finding 2: a third of sitemaps cannot say when anything changed, or say it was long ago Freshness is where the funnel narrows. Of the 87 valid sitemaps: | Freshness | Sites | Share of 87 | |---|---|---| | Newest `lastmod` within 30 days | 47 | 54.0% | | Newest `lastmod` 31 to 90 days ago | 11 | 12.6% | | Newest `lastmod` 91 to 365 days ago | 7 | 8.0% | | Newest `lastmod` over 365 days ago | 7 | 8.0% | | No `lastmod` anywhere | 15 | 17.2% | 15 sites (17.2%) publish a list of URLs with no dates at all. For an agent deciding what to re-fetch, that is a list with no signal: every URL looks equally stale or equally new. The protocol does not require `lastmod`, so none of these sites is wrong, but each has given up the one field that tells a crawler where to spend its requests. This is the same logic as our earlier measurement of [who can answer "nothing changed" with a 304](/blog/ai-crawler-traffic-waste-2026): a site that cannot signal change makes every visit a full download. Across the whole 87, 58 are fresh by our 90-day rule. The 14 stale ones split in half: 7 are 91 to 365 days old, 7 are older than a year. Stale does not mean wrong. A brochure site that has not changed in 18 months has an honest sitemap. The test is whether the date is *accurate*, and for a site with a blog and a newest `lastmod` of two years ago, it is not. The oldest sitemap in the whole probe, from a site in the failing group, lists 16,741 URLs whose newest `lastmod` is October 2018. That file is large and confident about a date that cannot be right for a site still publishing in 2026. It is a good picture of the failure: a sitemap generated once and never touched. ## Finding 3: the URLs mostly resolve, and the exceptions are instructive We sampled five URLs from each of 86 valid sitemaps (one valid index had no readable child). 80 sites (93.0%) had all five return 200. That is a better number than the freshness one, and it deserves credit. The other 6 split into two groups: | Result | Sites | Examples of what happened | |---|---|---| | Some of the 5 failed | 3 | One 404 among five; one 307 that never settled on a 200; four of five 404 on a staging-style hostname | | None of the 5 resolved | 3 | All five returned 403 (a bot wall); connection failures on all five; 404 or timeouts on all five | The 403 case matters for how you read every measurement here, ours included. A site that lists its own pages in a sitemap and then refuses to serve them to the client that read the sitemap has told an agent where to go and then shut the door. That could be a deliberate rule for unrecognised clients, or a CDN setting nobody reviewed. From outside, the effect is identical. The [Cloudflare bot-block change](/blog/cloudflare-ai-bot-block-googlebot-2026) shows how easily the second case arises. The staging hostname case is the pattern we expected to see more of: a sitemap generated on one environment and deployed with absolute URLs from another. It is a one-line fix in most build tools, and it is invisible to a check that stops at the status code. ## The funnel: from "has a sitemap" to "has a sitemap that works" Sequential survivors among the 90 sites that D2 passed: | Stage | Sites remaining | Share of 90 | Dropped | |---|---|---|---| | D2 passes | 90 | 100% | | | Serves valid sitemap XML today | 87 | 96.7% | 3 | | Declares any `lastmod` | 72 | 80.0% | 15 | | Newest `lastmod` within 90 days | 58 | 64.4% | 14 | | All 5 sampled URLs return 200 | 53 | 58.9% | 5 | **The sitemap funnel is the original framework here: discoverable, valid, fresh, resolving.** A site clears it only if it survives all four stages. 58.9% is a conservative figure. It treats the 15 undated sitemaps as unproven rather than fresh, and it uses the first child of an index, which tends to be the least recently updated one. The biggest single drop is not a broken file. It is the 15 sites that never say when anything changed. The smallest is the 3 that no longer serve a sitemap at all. The practical lesson is that the cheapest improvement for most sites is to emit `lastmod` correctly, not to build or repair a sitemap. ## Finding 4: most D2 failures are real, and some are not We probed 60 of the 170 D2 failures the same way, and added four more paths for the ones with no sitemap: `/wp-sitemap.xml`, `/sitemap-index.xml`, `/sitemap/sitemap.xml` and `/sitemaps.xml`. | Result | Sites | Share of 60 | |---|---|---| | No sitemap at either conventional path, none declared and valid, none at four alternatives | 48 | 80.0% | | Valid sitemap now served at `/sitemap.xml` or `/sitemap_index.xml` | 11 | 18.3% | | Valid sitemap served only at the URL robots.txt declares | 1 | 1.7% | Of those 48, none answered on any of the four alternative paths. That settles a worry we had going in: that D2 might be failing sites whose sitemap lives under a name it does not request. For this sample it was not. WordPress core publishes `/wp-sitemap.xml`, and not one of the 48 had it. D2's failures here are mostly genuine absences. The 12 sites with a sitemap now are the interesting part. Their last scans were dated 28 August to 28 September: 9 were scanned before 15 September, and 3 within the last nine days. A site that scanned as a fail on 25 August and has a sitemap on 1 October most likely added one in the meantime, which is a fix and is good news. For the 3 recent ones the explanation is less comfortable: either the site fixed itself within days of the scan, or the scanner met a transient error. We cannot tell which from the evidence stored, and the evidence string, "no /sitemap.xml or /sitemap_index.xml resolved", does not record a status code. That is a limit of D2 worth fixing and we have logged it as a product note. One more caution for readers who quote a failing D2 to someone else: re-run the scan first. A fail is a snapshot. ## What AIScan checks, and what it cannot see **Run the scan first.** `npx aiscan-cli yoursite.com`, or paste your URL at [aiscan.site](https://aiscan.site/) (free, no account). Three checks matter for this post: | Check | What it asserts | What it does not assert | |---|---|---| | D2, XML sitemap | A sitemap URL answers 200 at `/sitemap.xml` or `/sitemap_index.xml` | That it is valid XML, how many URLs it lists, how fresh it is, or whether the URLs resolve | | D1, robots.txt present and sane | robots.txt answers, and whether it declares a `Sitemap:` line | That the declared URL works | | E5, content feed | An RSS, Atom or JSON feed exists and whether `` declares it | That the feed's items are current | Read the three together. A site that passes D2 and D1 has a sitemap and tells agents where it is. Add E5 for sites that publish regularly: a feed is the other freshness signal, and the guide to [adding an RSS feed AI agents can find](/blog/add-rss-feed-declare-in-head) walks through it. Our own check cannot see the four legs of the funnel above. That is why we ran the probe ourselves, and why a clean scan is where this work starts, not where it ends. If you would rather check by hand, three commands cover it: ```bash # 1. Is it declared? (D1) curl -sL https://yoursite.com/robots.txt | grep -i '^sitemap:' # 2. Is it XML, and how many URLs? (D2, plus validity) curl -sL https://yoursite.com/sitemap.xml | grep -c '' # 3. Is it fresh? Newest lastmod in the file curl -sL https://yoursite.com/sitemap.xml | grep -o '[^<]*' | sort | tail -1 ``` For an index, repeat commands 2 and 3 against each child file it lists. To spot-check the links, take five `` values and request each one; anything but 200 is a finding. ## How to fix each leg **Not discovered.** Add `Sitemap: https://yoursite.com/sitemap.xml` to robots.txt, with an absolute URL. The line is independent of any `User-agent` group. If your sitemap lives somewhere unusual, this line is the only way an agent finds it, which is what happened to 3 of our 90 sites. **Not valid.** If the URL returns 200 with an HTML page, your router is serving a catch-all page. Return a real 404 for files that do not exist and the XML for the file that does. The [soft 404 guide](/blog/soft-404-agent-discovery-problem) covers the mechanics. Check size too: Google's limit is 50,000 URLs or 50MB uncompressed per file, and larger sites need an index. **Not fresh.** Take `lastmod` from the content record's modification timestamp, not from the time the sitemap was built. Rebuilding on every deploy stamps every URL with today's date, which Google's guidance warns against by requiring dates to be verifiably accurate. If your tooling cannot give a true date, omit the field for those URLs rather than faking it. An absent date is honest; a wrong date trains crawlers to ignore all your dates. **Not resolving.** Generate absolute URLs from your production hostname and fail the build if the sitemap host differs from it. Then add a post-deploy check that fetches a sample of the URLs. If a CDN rule returns 403 to non-browser clients, decide on purpose whether the listed pages should be reachable by crawlers, because the sitemap says they should be. **On WordPress.** Core already serves `/wp-sitemap.xml`, but D2 does not request that path, so declare it in robots.txt or alias it to `/sitemap.xml`. [**ThinkRank**](https://thinkrank.ai) is the plugin we recommend as the default here because it manages sitemap settings, robots.txt, robots meta, schema and llms.txt from one place, so there is one plugin deciding what each file says instead of three fighting over robots.txt, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress. Yoast, Rank Math and AIOSEO each generate sitemaps well, and any of them is fine if it already runs your site, provided only one is writing the file. **On Shopify.** The platform generates `/sitemap.xml` for you, so the work is checking that robots.txt points to it. For stores, [**StoreSEO**](https://storeseo.com/) is the default recommendation for the half of the stack the platform does not touch: schema, llms.txt and the AEO side. ## Why `lastmod` is the field worth getting right Google states the rule precisely: the date is used if it is consistently and verifiably accurate, and the value should reflect the last significant change to the page. "Verifiably" is the word that matters. A crawler can request the page, read its `Last-Modified` header or compare the content to the last copy it saw, and decide whether your date was telling the truth. A site that stamps every URL with the build time fails that comparison on the second visit. After that the crawler has a reason to treat the whole field as noise, including the dates that were right. You can run the same test on your own site with two requests. Take one URL and its sitemap date: ```bash # date the sitemap claims for one URL curl -sL https://yoursite.com/sitemap.xml | grep -A2 'yoursite.com/some-post' | grep -o '[^<]*' # date the page itself reports curl -sIL https://yoursite.com/some-post | grep -i '^last-modified' ``` If the two disagree by months, one of them is wrong, and an agent that checks will not guess which. If the page sends no `Last-Modified` header at all, the sitemap date is unverifiable from outside, which is the position most dynamic sites are in. That is a reason to keep the sitemap date honest, since it may be the only date you publish. There is a cost argument as well. Our probe fetched five pages per site and that was enough to find broken hosts and bot walls. An agent working at scale fetches far more. Each stale or dateless entry makes it choose between re-fetching everything, which costs you bandwidth and server time, or trusting an old copy, which costs the reader accuracy. Accurate dates let it do neither. ## What this study does not claim Four limits are worth restating, because the headline number is easy to quote without them. First, 90 passing and 60 failing sites is a spot check. The 58.9% funnel figure has a wide interval at this size, and it describes sites that were scanned, not the web. We would expect the funnel to look worse on a random sample, because people who run a scanner tend to have looked at their discovery files at least once. Second, five sampled URLs per sitemap finds a site where nothing resolves and misses a site where 3% of a 20,000-URL file is dead. The 93.0% all-five-resolve figure is an upper bound on link health. Third, we read the first child of each sitemap index. For a WordPress site that is often the posts file or the pages file, depending on the plugin. Either may be the least recently updated, so the freshness numbers for the 44 indexes lean pessimistic. Fourth, we probed from one cloud address on one day. Three sites refused us outright. Some of those would answer a residential browser, and some would answer an AI crawler that the site has allow-listed. The measurement shows what an unknown client sees, which is the situation most agents are in on first contact. None of this changes the direction of the findings. A passing D2 is a sound sign that a sitemap exists, and it is a weak sign that the sitemap is useful. ## A ten-minute monthly review The fastest fix we saw in the data was a site that scanned as a fail in late August and serves a valid, dated sitemap now. The slowest was a file that has not been touched since 2018. The difference is whether anyone looks. A review that takes ten minutes: 1. Run the scan and note D2, D1 and E5. Anything that changed since last month is the first thing to read. 2. Fetch robots.txt and confirm every `Sitemap:` URL returns 200 and parses. 3. Pull the newest `lastmod` and compare it with your newest published page. They should be within a day or two of each other. 4. Request five URLs from the sitemap, chosen at random, and confirm 200. 5. If you run a CDN or bot filter, repeat step 4 with a non-browser user agent and read the status codes. Put the commands in a script and run it in CI after each deploy if you can. The [guide to gating AI readiness in CI](/blog/aiscan-ci-gate-github-actions) shows how to fail a build when a check regresses. ## Common mistakes | Mistake | What happens | Fix | |---|---|---| | Sitemap built at deploy, `lastmod` set to build time | Every URL looks changed daily; agents learn to ignore the field | Source dates from content records | | Declared path has a typo | robots.txt promises a file that 404s | Fetch the declared URL after every robots.txt change | | Sitemap lists staging or `http://` URLs | Agents follow links to the wrong host | Build URLs from one production base setting | | CDN blocks unrecognised clients | Listed pages return 403 to the reader of the list | Allow the clients you want; test with curl | | One huge file | Over 50,000 URLs or 50MB breaks the limit | Split into an index of child files | | Never reviewed after launch | Dates freeze, pages disappear, the file stays | Add the three curl checks to a monthly review | ## Your next action Run `npx aiscan-cli yoursite.com` and read D2, D1 and E5 in the discoverability and content results. Then run the three curl commands above. If the newest `lastmod` is older than your newest post, fix the date source first, because that was the largest drop in our funnel. The full set of fixes is in the [AIScan guides](/guides), and the [llms.txt generator](/llms-txt-generator) covers the sibling file agents read next.