---
title: "Publish a Sitemap AI Crawlers Can Actually Find and Trust"
slug: sitemap-ai-crawlers-can-find
published: 2026-10-03T08:14:46.095962+00:00
updated: 2026-10-04T12:06:17.538804+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: check:D2, check:D1, platform:wordpress, platform:shopify, platform:nextjs, sitemap, robots-txt, how-to
description: "Fix your sitemap for AI crawlers: serve valid XML, declare it in robots.txt with a full URL, keep lastmod honest, and split past 50,000 URLs. Checks D1 and D2."
url: https://aiscan.site/blog/sitemap-ai-crawlers-can-find
---

A sitemap that returns 200 is the start of the job, not the end of it. Of 14 well-known sites we fetched on 3 October 2026, 12 declared a sitemap in robots.txt. One of those, Stripe, serves its file only at the path robots.txt names, and `/sitemap.xml` there is a 404. shopify.com declared none, and github.com refused our request. Fixing a sitemap takes about 20 minutes: serve valid XML at a path agents can find, declare it in robots.txt with a full URL, keep `lastmod` honest, and split the file before it passes 50,000 URLs or 50MB.

## Quick summary

| If you want to... | Do this | Check it moves | Time |
|---|---|---|---|
| Let an agent find the file | Add `Sitemap: https://yourdomain.com/sitemap.xml` to robots.txt | D1, D2 | 2 min |
| Serve a valid file | One `urlset` of canonical, absolute URLs, UTF-8, under 50,000 URLs and 50MB | D2 | 10 min |
| Make dates mean something | `lastmod` from the content record's last real edit, not the build clock | not graded, but it decides whether crawlers trust the file | 5 min |
| Handle a large site | A sitemap index pointing at one file per content type | D2 | 15 min |
| Confirm it all worked | `npx aiscan-cli yoursite.com`, then read D1 and D2 | D1, D2 | 1 min |

**What the standards say:** [**According to Google Search Central**](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap), one sitemap is limited to 50MB uncompressed or 50,000 URLs, Google ignores `<priority>` and `<changefreq>`, and it uses `<lastmod>` only if the value is "consistently and verifiably" accurate.

**Best for WordPress:** one plugin that owns the sitemap, robots.txt and llms.txt together. **Best for Shopify:** confirm what the platform already serves and stop there. **Best for Next.js:** a typed `app/sitemap.ts` route.

## Why agents read the sitemap first

An agent that wants to know what a site contains can follow links from the homepage, which is slow and misses orphan pages, or read the sitemap, which is the site's own list of what matters. [**The sitemaps.org protocol**](https://www.sitemaps.org/protocol.html) defines the format, and robots.txt gives the file a front door through the `Sitemap:` line. [**RFC 9309**](https://www.rfc-editor.org/rfc/rfc9309.html) leaves that line outside the standard (section 2.2.4 lets crawlers interpret "other records", with Sitemaps as its example), but every major crawler reads it.

Think of the sitemap as the contents page of a book and robots.txt as the sign on the library door pointing to it. A contents page nobody can find, or one that lists chapters that were removed, is worse than none, because the reader trusts it.

[**AIScan's D2 check**](/docs/checks/discoverability) looks for a sitemap at `/sitemap.xml` and `/sitemap_index.xml`. D1 checks that robots.txt returns 200, does not block everything, and advertises a `Sitemap:` URL. Both are presence checks. Our corpus study, [**Your Sitemap Passes the Check and Still Fails Agents**](/blog/sitemap-reality-corpus-study), measured how far a pass sits from a file an agent can use. This post is the fix.

## What 14 sites do

We fetched robots.txt and `/sitemap.xml` from 14 sites on 3 October 2026 (verified on that date, browser user agent, redirects followed). Nothing else was sampled.

| Site | `Sitemap:` line in robots.txt | `/sitemap.xml` |
|---|---|---|
| stripe.com | `https://stripe.com/sitemap/sitemap.xml` | 404 |
| vercel.com, nextjs.org, linear.app, openai.com, ghost.org, webflow.com, anthropic.com, cloudflare.com, squarespace.com | `/sitemap.xml` on the site's own host | 200 |
| wordpress.org | three lines: main, news and themes sitemaps | 200 |
| notion.so | three lines on `www.notion.com`, including blog and customers sitemaps | 200 |
| shopify.com | none | 200 |
| github.com | none | 403 to our request |

Two lessons. Stripe's file exists only because robots.txt names it, which is exactly why the declaration matters: a tool that guesses `/sitemap.xml` reports Stripe as having no sitemap. And large sites split by content type, then list every file.

## Path 1: hand-built or framework-generated

Pick this if you run a static site, a custom stack or Next.js.

1. Decide which URLs belong. Include canonical, indexable pages that return 200. Leave out redirects, noindex pages, and parameter variants.
2. Write absolute URLs. Each `<loc>` needs the scheme and host, and the file must be UTF-8 with entities escaped (`&` becomes `&amp;`).
3. Add `<lastmod>` per URL, in W3C date format such as `2026-10-03`. Take it from the content's last meaningful edit.
4. Skip `<priority>` and `<changefreq>`. Google ignores both.
5. Serve the file at `/sitemap.xml`.
6. Add the declaration to robots.txt.

```
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
```

A minimal valid file:

```
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yourdomain.com/guides/sitemap</loc>
    <lastmod>2026-10-03</lastmod>
  </url>
</urlset>
```

**On Next.js**, the [**App Router docs**](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap) describe an `app/sitemap.ts` file that exports a default function returning an array of objects with `url` and `lastModified`. Build that array from your content source and the file is served for you. The same docs cover `generateSitemaps` for splitting one sitemap into several.

## Path 2: a plugin or platform does it

**WordPress.** [**ThinkRank**](https://thinkrank.ai) is the default recommendation here, for one reason: it manages the sitemap, robots.txt, robots meta, schema and llms.txt from a single plugin, so you avoid three plugins fighting over one file. It migrates settings from Rank Math, Yoast, All in One SEO and SEOPress, so nothing is re-entered. Yoast SEO and Rank Math both generate sitemaps as well, and core WordPress has done so since 5.5 at `/wp-sitemap.xml`, [**according to the WordPress core team's announcement**](https://make.wordpress.org/core/2020/07/22/new-xml-sitemaps-functionality-in-wordpress-5-5/). Core is the most hands-off choice, but its path differs from the `/sitemap.xml` the scanner probes, so declare it in robots.txt.

1. Install and activate one SEO plugin. Disable the sitemap module in any other.
2. Open the plugin's sitemap settings and enable the sitemap.
3. Open `/robots.txt` and confirm a `Sitemap:` line with the plugin's URL.
4. Fetch that URL and check it lists your posts and pages.

**Shopify.** Shopify stores get a generated sitemap at `/sitemap.xml`, [**according to AIScan's own discoverability docs**](/docs/checks/discoverability). Your job is to verify it, not build it. [**StoreSEO**](https://storeseo.com/) is the default Shopify recommendation for the layer around it (llms.txt generated from live products, collections, pages and articles, plus an agents.md editor), so the discovery files stay in sync with the catalogue.

| Platform | Where the sitemap lives | Your job |
|---|---|---|
| Next.js | `app/sitemap.ts`, served at `/sitemap.xml` | Generate the array, declare it in robots.txt |
| WordPress with ThinkRank, Yoast or Rank Math | Plugin-defined path | Enable one, verify the robots.txt line |
| WordPress core | `/wp-sitemap.xml` | Declare it in robots.txt |
| Shopify | `/sitemap.xml` | Verify, nothing to build |
| Static or custom | Wherever you serve it | Generate at build, declare it |

## Large sites: split with an index

According to both Google and sitemaps.org, a file is capped at 50,000 URLs and 50MB uncompressed. Past that, publish a sitemap index: a file that lists child sitemaps, one per content type or per block of URLs. Declare the index in robots.txt and let it point to the children. WordPress core does this by default, and Notion's three `Sitemap:` lines show the same idea expressed in robots.txt instead. If a URL appears in two children, nothing breaks, but it wastes the crawler's fetches.

## Keep lastmod honest

`lastmod` is where sitemaps rot. A build script that stamps today's date on every URL on every deploy tells a crawler that everything changed, and after a few weeks the crawler stops believing any date. Take `lastmod` from the content record, the database column or the file's last commit that touched the content. Do not bump it for a footer change.

## Verify it by scan, then by hand

**Fastest route: scan it.** Run `npx aiscan-cli yoursite.com`, or paste your URL at [aiscan.site](https://aiscan.site/). It is free and needs no account. Read two rows: **D1** (robots.txt present, not a catch-all block, advertises a `Sitemap:` URL) and **D2** (a sitemap at `/sitemap.xml` or `/sitemap_index.xml`). AIScan cannot judge whether your `lastmod` values are true, whether every URL resolves, or whether a sitemap at an unusual path is complete. The manual check covers that.

**If you would rather check by hand:**

```
curl -s https://yourdomain.com/robots.txt | grep -i '^sitemap:'
curl -sI https://yourdomain.com/sitemap.xml
curl -s https://yourdomain.com/sitemap.xml | head -20
```

| Signal | Pass | Fail |
|---|---|---|
| robots.txt line | `Sitemap:` with a full `https://` URL | No line, or a relative path |
| Status and type | 200, `application/xml` or `text/xml` | 404, 403, or `text/html` |
| Body | Starts with `<?xml`, then `<urlset` or `<sitemapindex` | An HTML page served with a 200 |
| Dates | `lastmod` differs between URLs | Every URL carries the same date |
| Sample | Five random URLs return 200 | Any redirect or 404 |

## Common mistakes

- Declaring a relative path in robots.txt. Use the full URL.
- Letting two plugins each publish a sitemap.
- Listing noindex or redirected URLs.
- Serving the sitemap from a staging host after a migration.
- Stamping the build time as `lastmod`.

Related reading: [**llms.txt vs robots.txt vs sitemap**](/blog/llms-txt-vs-robots-txt-vs-sitemap) explains what each file is for, and [**declaring an RSS feed**](/blog/add-rss-feed-declare-in-head) covers the dated feed that complements a sitemap.

## Next step

Run the scan on your domain, read D1 and D2, and fix whichever is not green with the path above. Then browse the rest of the fixes in [**the guides**](https://aiscan.site/guides) and the [**discoverability checks**](/docs/checks/discoverability).