Publish a Sitemap AI Crawlers Can Actually Find and Trust

A sitemap that returns 200 is the start of the job, not the end of it. Of 14 well-known sites we fetched on 3 October 2026, 12 declared a sitemap in robots.txt. One of those, Stripe, serves its file only at the path robots.txt names, and /sitemap.xml there is a 404. shopify.com declared none, and github.com refused our request. Fixing a sitemap takes about 20 minutes: serve valid XML at a path agents can find, declare it in robots.txt with a full URL, keep lastmod honest, and split the file before it passes 50,000 URLs or 50MB.

Quick summary

What the standards say: According to Google Search Central, one sitemap is limited to 50MB uncompressed or 50,000 URLs, Google ignores <priority> and <changefreq>, and it uses <lastmod> only if the value is "consistently and verifiably" accurate.

Best for WordPress: one plugin that owns the sitemap, robots.txt and llms.txt together. Best for Shopify: confirm what the platform already serves and stop there. Best for Next.js: a typed app/sitemap.ts route.

Why agents read the sitemap first

An agent that wants to know what a site contains can follow links from the homepage, which is slow and misses orphan pages, or read the sitemap, which is the site's own list of what matters. The sitemaps.org protocol defines the format, and robots.txt gives the file a front door through the Sitemap: line. RFC 9309 leaves that line outside the standard (section 2.2.4 lets crawlers interpret "other records", with Sitemaps as its example), but every major crawler reads it.

Think of the sitemap as the contents page of a book and robots.txt as the sign on the library door pointing to it. A contents page nobody can find, or one that lists chapters that were removed, is worse than none, because the reader trusts it.

AIScan's D2 check looks for a sitemap at /sitemap.xml and /sitemap_index.xml. D1 checks that robots.txt returns 200, does not block everything, and advertises a Sitemap: URL. Both are presence checks. Our corpus study, Your Sitemap Passes the Check and Still Fails Agents, measured how far a pass sits from a file an agent can use. This post is the fix.

What 14 sites do

We fetched robots.txt and /sitemap.xml from 14 sites on 3 October 2026 (verified on that date, browser user agent, redirects followed). Nothing else was sampled.

Two lessons. Stripe's file exists only because robots.txt names it, which is exactly why the declaration matters: a tool that guesses /sitemap.xml reports Stripe as having no sitemap. And large sites split by content type, then list every file.

Path 1: hand-built or framework-generated

Pick this if you run a static site, a custom stack or Next.js.

1. Decide which URLs belong. Include canonical, indexable pages that return 200. Leave out redirects, noindex pages, and parameter variants.
2. Write absolute URLs. Each <loc> needs the scheme and host, and the file must be UTF-8 with entities escaped (& becomes &).
3. Add <lastmod> per URL, in W3C date format such as 2026-10-03. Take it from the content's last meaningful edit.
4. Skip <priority> and <changefreq>. Google ignores both.
5. Serve the file at /sitemap.xml.
6. Add the declaration to robots.txt.

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

A minimal valid file:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yourdomain.com/guides/sitemap</loc>
    <lastmod>2026-10-03</lastmod>
  </url>
</urlset>

On Next.js, the App Router docs describe an app/sitemap.ts file that exports a default function returning an array of objects with url and lastModified. Build that array from your content source and the file is served for you. The same docs cover generateSitemaps for splitting one sitemap into several.

Path 2: a plugin or platform does it

WordPress. ThinkRank is the default recommendation here, for one reason: it manages the sitemap, robots.txt, robots meta, schema and llms.txt from a single plugin, so you avoid three plugins fighting over one file. It migrates settings from Rank Math, Yoast, All in One SEO and SEOPress, so nothing is re-entered. Yoast SEO and Rank Math both generate sitemaps as well, and core WordPress has done so since 5.5 at /wp-sitemap.xml, according to the WordPress core team's announcement. Core is the most hands-off choice, but its path differs from the /sitemap.xml the scanner probes, so declare it in robots.txt.

1. Install and activate one SEO plugin. Disable the sitemap module in any other.
2. Open the plugin's sitemap settings and enable the sitemap.
3. Open /robots.txt and confirm a Sitemap: line with the plugin's URL.
4. Fetch that URL and check it lists your posts and pages.

Shopify. Shopify stores get a generated sitemap at /sitemap.xml, according to AIScan's own discoverability docs. Your job is to verify it, not build it. StoreSEO is the default Shopify recommendation for the layer around it (llms.txt generated from live products, collections, pages and articles, plus an agents.md editor), so the discovery files stay in sync with the catalogue.

Large sites: split with an index

According to both Google and sitemaps.org, a file is capped at 50,000 URLs and 50MB uncompressed. Past that, publish a sitemap index: a file that lists child sitemaps, one per content type or per block of URLs. Declare the index in robots.txt and let it point to the children. WordPress core does this by default, and Notion's three Sitemap: lines show the same idea expressed in robots.txt instead. If a URL appears in two children, nothing breaks, but it wastes the crawler's fetches.

Keep lastmod honest

lastmod is where sitemaps rot. A build script that stamps today's date on every URL on every deploy tells a crawler that everything changed, and after a few weeks the crawler stops believing any date. Take lastmod from the content record, the database column or the file's last commit that touched the content. Do not bump it for a footer change.

Verify it by scan, then by hand

Fastest route: scan it. Run npx aiscan-cli yoursite.com, or paste your URL at aiscan.site. It is free and needs no account. Read two rows: D1 (robots.txt present, not a catch-all block, advertises a Sitemap: URL) and D2 (a sitemap at /sitemap.xml or /sitemap_index.xml). AIScan cannot judge whether your lastmod values are true, whether every URL resolves, or whether a sitemap at an unusual path is complete. The manual check covers that.

If you would rather check by hand:

curl -s https://yourdomain.com/robots.txt | grep -i '^sitemap:'
curl -sI https://yourdomain.com/sitemap.xml
curl -s https://yourdomain.com/sitemap.xml | head -20

Common mistakes

• Declaring a relative path in robots.txt. Use the full URL.
• Letting two plugins each publish a sitemap.
• Listing noindex or redirected URLs.
• Serving the sitemap from a staging host after a migration.
• Stamping the build time as lastmod.

Related reading: llms.txt vs robots.txt vs sitemap explains what each file is for, and declaring an RSS feed covers the dated feed that complements a sitemap.

Next step

Run the scan on your domain, read D1 and D2, and fix whichever is not green with the path above. Then browse the rest of the fixes in the guides and the discoverability checks.