---
title: "Does robots.txt Need a Sitemap Line? 782 Sites Show What Happens"
slug: robots-txt-sitemap-line-corpus
published: 2026-10-29T03:14:10.693504+00:00
updated: 2026-10-29T03:14:10.693504+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:D1, check:D2, robots.txt, sitemap, corpus study, platform:wordpress, platform:nextjs, platform:astro, platform:shopify
description: "572 of 782 sites declare a sitemap in robots.txt. 95.3% of them have one at a conventional path, against 44.9% of the 89 that do not. Data and fix."
url: https://aiscan.site/blog/robots-txt-sitemap-line-corpus
---

Of 782 sites in our scan database on 29 October 2026, 572 serve a robots.txt that declares a sitemap, 89 serve a robots.txt with no Sitemap line, and 121 fail the robots.txt check outright. Declaring the sitemap goes with having one: 545 of the 572 sites that declare one (95.3%) also have a sitemap at a conventional path, against 40 of the 89 sites that do not declare one (44.9%). So if you leave the line out, you are betting that a crawler will guess your sitemap's location, and on our data that guess fails for 55 sites in 100.

This post crosses two AIScan checks, D1 (robots.txt) and D2 (XML sitemap), and answers one question: does robots.txt need a Sitemap line? It is written by M Asif Rahman for site owners who have a robots.txt and are not sure the extra line matters. The short answer is that the standard does not require it, you should add it anyway, and the cost is one line.

## Quick summary

| Question | Answer from our scans | Section |
|---|---|---|
| Does the robots.txt standard require a Sitemap line? | No. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html#name-other-records) says crawlers MAY read it | What the standard says |
| How many sites declare a sitemap in robots.txt? | 572 of 782 (73.1%) | The four cells |
| How many serve a robots.txt with no Sitemap line? | 89 (11.4%) | The four cells |
| Does a sitemap exist at a conventional path when the line is missing? | 40 of 89 (44.9%) | The four cells |
| And when the line is present? | 545 of 572 (95.3%) | The four cells |
| How big is a robots.txt without the line? | Median 103 to 121 bytes, against 445 to 477 bytes with it | File size |
| Should you add the line? | Yes, with a full URL, and test it | What to do |

**For site owners:** add one `Sitemap:` line with an absolute URL, then confirm the sitemap answers 200. **For anyone auditing a site:** a robots.txt of about 100 bytes with no Sitemap line is a stub, and in our data it is usually a stub nobody revisited.

## What the standard says, and what the check does

The robots.txt standard, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html#name-other-records), defines allow and disallow rules grouped by user agent. A Sitemap line is not part of those rules. Section 2.2.4 on other records says crawlers "MAY interpret other records that are not part of the robots.txt protocol", and it names Sitemaps as an example. That is permission, not a requirement, and it means no crawler is obliged to read the line.

The line itself is defined by the Sitemaps protocol. [sitemaps.org](https://www.sitemaps.org/protocol.html#informing) lists robots.txt as one of the ways to tell search engines where your sitemap lives: you add a `Sitemap:` line with the sitemap's full URL. It is independent of the user-agent groups, so you can put it anywhere in the file and it applies to every crawler that reads it.

AIScan checks both ends. According to the [discoverability check documentation](https://aiscan.site/docs/checks/discoverability), D1 fetches `/robots.txt` and wants three things: a 200 status, no catch-all block of `User-agent: *` with `Disallow: /`, and a `Sitemap:` URL. D2 looks at `/sitemap.xml` and `/sitemap_index.xml`. That second point matters for the numbers below. **D2 does not read the URL you declared.** It probes two conventional paths. So when a site declares a sitemap at `/wp-sitemap.xml` or `/sitemaps/main.xml` and nothing sits at the two conventional paths, D2 fails even though the declaration may be perfectly good. We come back to that in the 27-site cell.

## How we measured it, and what it cannot tell you

**The corpus.** Every AIScan report stores the result and the evidence string of each check. We took the latest site-scope scan per host under rubric 2026.08.2, excluded aiscan.site, and read D1 and D2 for each host. On 29 October 2026 that was 782 hosts. We name no scanned site. These are sites that people chose to scan, not a random sample of the web, and the mix leans toward WordPress and Next.js because those are the stacks of people who run a free agent-readiness scanner.

**What D1 stores.** The evidence reads `200 OK, N bytes, declares Sitemap` when the file answers 200 and carries a Sitemap line, and `200 OK, N bytes` when it answers 200 without one. The second form is scored as a partial. The evidence does not list a separate reason for a partial, so we read every `200 OK, N bytes` result as "no Sitemap line". If a few of those 89 are partial for another reason, such as a catch-all block, the evidence we store would not show it, and the cell counts would shift by a handful at most. The sizes support the reading: those files are tiny.

**What D1 failures are.** 119 sites return an HTTP error for `/robots.txt` and 2 return no response. Those are status-code problems, not Sitemap-line problems, and two earlier posts cover them: [serving robots.txt with the right status and content type](/blog/robots-txt-status-content-type) and [how much of a failure is a blocked probe](/blog/blocked-probe-share-of-failures). We count them here only so the totals add up.

**What we cannot say.** The database stores evidence strings and statuses, not file bodies. We can say whether a Sitemap line exists, not what URL it names, whether that URL resolves, or whether it is on the same host. Nothing here measures indexing or crawl behaviour. A correlation between declaring and having a sitemap is not proof that the line caused the sitemap.

## The four cells

Crossing the robots.txt result with the sitemap result gives four outcomes for the 661 sites whose robots.txt answered 200 (572 plus 89), plus the failures.

| robots.txt result | D2 sitemap found at a conventional path | D2 nothing at those paths | Sites |
|---|---:|---:|---:|
| 200 and declares a Sitemap | 545 (95.3%) | 27 (4.7%) | 572 |
| 200 and no Sitemap line | 40 (44.9%) | 49 (55.1%) | 89 |
| Fails D1 (HTTP error or no response) | 10 (8.3%) | 111 (91.7%) | 121 |
| All sites | 595 (76.1%) | 187 (23.9%) | 782 |

Read the table by rows.

**Declared, and the sitemap is where convention says (545).** This is the healthy majority. 69.7% of all sites in the corpus have both a Sitemap line and a sitemap at one of the two paths D2 probes. The median robots.txt in this cell is 477 bytes, which is room for a few crawler rules and the sitemap line.

**No Sitemap line, but a sitemap exists anyway (40).** These sites got lucky, or got the default from their platform. A crawler that knows the convention finds the sitemap without being told. A crawler that does not guess the convention finds nothing. The sitemap is there, the signpost is not.

**No Sitemap line, and no sitemap (49).** This is the cell that should worry the owners it describes. There is no declaration and nothing at the conventional paths, so an agent that wants a list of the site's pages has to crawl for them. Of the 89 sites with no Sitemap line, 49 land here. That is the basis for the 55-in-100 figure in the opening.

**Declared, but nothing at the conventional paths (27).** This is the cell to read with care, because of how D2 works. Because D2 only tries `/sitemap.xml` and `/sitemap_index.xml`, a site whose declared sitemap lives somewhere else fails D2 whether or not the declared file is fine. These 27 are a mix of two cases we cannot separate: sitemaps that are valid at a custom path, and declarations that point at a file that is broken or gone. The fix is the same for both: open the URL your robots.txt names and confirm it returns 200 with XML. If it does and you want D2 to pass, also serve it at `/sitemap.xml`, or redirect that path to it.

**robots.txt fails (121).** Of these, 111 also have no sitemap at the conventional paths. A site that cannot serve robots.txt correctly often has no sitemap either, which fits a site that nobody maintains for crawlers. The 10 that pass D2 despite a failing robots.txt show the two files are separate jobs.

## File size: the stub robots.txt

The byte counts tell a second story. The robots.txt files that declare a sitemap have a median of 477 bytes where D2 passes and 445 bytes where it fails. The files with no Sitemap line have a median of 102.5 bytes where D2 passes and 121 bytes where it fails.

| Cell | Median robots.txt size |
|---|---:|
| Declares Sitemap, sitemap found | 477 bytes |
| Declares Sitemap, nothing at conventional paths | 445 bytes |
| No Sitemap line, sitemap found | 103 bytes (102.5) |
| No Sitemap line, nothing at conventional paths | 121 bytes |

A file around 100 bytes holds roughly three or four short lines. That is the shape of a robots.txt that was created once to satisfy a requirement: a `User-agent: *` line, an empty `Disallow:` and maybe a comment. It is valid. It is also a sign that nobody has gone back to say anything useful in it. The sitemap line is the cheapest useful thing to add to a file like that.

The sizes do not prove cause. A site with a 477-byte robots.txt probably has an SEO plugin or a framework helper that writes the file, and that same tool writes the sitemap and the declaration together. The declared and the sitemap-found cells move as a pair because one tool generates both. That is the best argument for letting a tool own the file, which we get to in the platform section.

## By platform

Where a platform has at least 15 sites in the corpus, the split looks like this.

| Platform | Sites | Declares Sitemap | No Sitemap line | Fails D1 | Sitemap found at conventional path |
|---|---:|---:|---:|---:|---:|
| WordPress | 198 | 167 (84.3%) | 22 | 9 | 181 (91.4%) |
| Next.js | 193 | 170 (88.1%) | 11 | 12 | 173 (89.6%) |
| Astro | 41 | 36 (87.8%) | 0 | 5 | 35 (85.4%) |
| Shopify | 16 | 15 (93.8%) | 1 | 0 | 15 (93.8%) |
| Platform not detected | 288 | 149 (51.7%) | 46 | 93 | 153 (53.1%) |

Three things stand out.

**The named platforms look alike.** WordPress, Next.js, Astro and Shopify all declare a sitemap on 84% to 94% of their sites. The platforms that ship a sitemap generator also tend to write the line. Shopify generates `/sitemap.xml` automatically, according to the check documentation, and 15 of 16 Shopify sites pass D2. The smaller Shopify and Astro samples should be read as direction, not as percentages to quote.

**The no-line problem is concentrated on WordPress and on undetected stacks.** WordPress has 22 sites with a robots.txt and no Sitemap line, and 16 of those 22 still have a sitemap at a conventional path, since core WordPress and the common plugins create one. Next.js has 11. Astro has none, though it has 5 failures. That fits how the stacks work: on Astro and Next.js, the file is whatever the developer wrote or generated, so the line is present or the file is missing.

**Undetected platforms carry most of the failures.** The 288 hosts where AIScan could not name a platform hold 93 of the 121 D1 failures and 46 of the 89 no-line results. These are the sites most likely to be hand-built, behind a proxy that blocks the probe, or served by a platform AIScan does not recognise. Treat their numbers as a mix.

If you run WordPress, the practical point is that you can fix this without editing a file. [ThinkRank](https://thinkrank.ai) manages robots.txt, the sitemap and llms.txt from one plugin, so the Sitemap line and the sitemap it points to come from the same place and cannot drift apart, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress. The failure to avoid is two plugins and a hand-edited file each writing a different robots.txt.

## What to do

**1. Declare the sitemap.** Add a `Sitemap:` line with the full, absolute URL. The example in the Sitemaps protocol uses the full URL, scheme and host included, so write it that way rather than a bare `/sitemap.xml`. Put it on its own line:

```text
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml
```

If you have several sitemaps, either list each on its own `Sitemap:` line or declare one sitemap index. A sitemap index file is what most large sites do.

**2. Check it with the scan first.** Run `npx aiscan-cli yoursite.com`, or paste your address at [aiscan.site](https://aiscan.site/). It is free and needs no account. Look at **D1** (robots.txt) and **D2** (XML sitemap) in the report. D1 tells you whether the file answers 200 and carries a Sitemap line. D2 tells you whether a sitemap answers at one of the two conventional paths. If you would rather check by hand, two requests do the same job:

```bash
curl -sI https://example.com/robots.txt | head -n 1
curl -s https://example.com/robots.txt | grep -i '^sitemap:'
curl -sI https://example.com/sitemap.xml | head -n 1
```

**3. Make the declared URL and the conventional path agree.** If your sitemap lives at a custom path, keep the declaration, then redirect `/sitemap.xml` to it with a 301 so the conventional probe also works. That gets you both the declaration and a passing D2. [Which sitemap an AI crawler can find](/blog/sitemap-ai-crawlers-can-find) walks through the paths and the formats.

**4. Do not stop at the declaration.** A Sitemap line that points at a sitemap full of dead or redirecting URLs is worse than no line, because it advertises a list that does not hold. Our earlier [corpus study of sitemap files](/blog/sitemap-reality-corpus-study) covers what is inside them. If you need a starting robots.txt that also handles AI crawlers, use the [robots.txt starter for AI crawlers](/blog/robots-txt-starter-for-ai-crawlers).

**5. Re-check after any platform change.** A theme switch, a plugin swap or a move to a new host can regenerate robots.txt and silently drop the line. In our data the 100-byte stub is the file that results when nobody looks again.

## Should every site declare one? Our position

Yes, with one caveat. The caveat is that the standard makes the line optional, and a crawler that is good at finding sitemaps does not need it. If your sitemap is at `/sitemap.xml` and every crawler you care about checks that path, the line adds nothing for them.

We still say to add it, for three reasons that come from the table rather than from theory. First, the declared group has a sitemap at the conventional path 95.3% of the time, so the line is nearly always accurate where it exists. Second, the undeclared group has no sitemap at either conventional path 55.1% of the time, so for half the sites that skip the line there is nothing to fall back on. Third, the cost is one line in a file you already serve.

That is a position on a correlation, not a measured effect on crawling. We did not test whether any crawler fetched a sitemap because of the line. What the data does show is that omitting it correlates with the sitemap being missing as well, and that is a signal worth acting on.

## Next action

Run the scan on your own domain: `npx aiscan-cli yoursite.com` or paste it at [aiscan.site](https://aiscan.site/). Read **D1** and **D2** in the discoverability section, fix the Sitemap line first, then the sitemap path. The full list of checks and fixes is in the [discoverability checks documentation](https://aiscan.site/docs/checks/discoverability), and the rest of the AI-readiness guides are on [/guides](https://aiscan.site/guides).