---
title: "Serve robots.txt with the right status and content type"
slug: robots-txt-status-content-type
published: 2026-10-16T08:17:47.83465+00:00
updated: 2026-10-16T08:17:47.83465+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: how-to, robots.txt, http status, content type, rfc 9309, nginx, wordpress, nextjs, check:D1, platform:nginx, platform:wordpress, platform:nextjs
description: "A robots.txt that returns 403, 503 or an HTML page can be ignored or can stall crawlers. What RFC 9309 and Google do, plus a curl check and per-stack fixes."
url: https://aiscan.site/blog/robots-txt-status-content-type
---

Most robots.txt advice is about the rules inside the file. This guide is about the envelope: the HTTP status, the content type, the redirects and the size. A perfect set of rules does nothing if the response around it makes a crawler throw the file away, and in one case the response makes a crawler stop visiting your site altogether.

The reason is that a crawler never reads your file in isolation. It first reads the response, and the response decides whether the rules count. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), the Robots Exclusion Protocol, spells that out in a short section on access results. Most sites never look at it until something breaks.

## Quick summary

| What your server returns for /robots.txt | What the RFC says a crawler does | What Google says it does |
|---|---|---|
| 200 with `text/plain`, UTF-8 | Follows the rules it can parse | Same |
| 301 or 302 to another URL | Follows at least five hops, rules apply to the original host | Follows five hops, then treats it as a 404 |
| 404 or any other 4xx | "Unavailable": may crawl everything | Treats every 4xx except 429 as no file: no restrictions |
| 500, 503 or any 5xx | "Unreachable": must assume complete disallow | Stops crawling for 12 hours, then uses its last good copy for up to 30 days |
| 200 with an HTML page | Not defined; the parseable lines win | Parses it and ignores everything that is not a rule |

Three habits fix almost every case. Serve the file as `text/plain; charset=utf-8`. Return a plain 404 if you have no file, never a 200 with a page. Keep the file under 500 KiB and reachable on the exact host the crawler asked for. The scan check that covers this is `check:D1`, and [the discoverability docs](/docs/checks/discoverability) list what it grades.

## Why the status code is part of the rule

Think of the status code as the gatekeeper's first answer, given before the crawler reads a word of the rules. According to RFC 9309, section 2.3.1, there are four outcomes, and they are asymmetric.

A successful fetch means the crawler follows the rules it can parse. A status in the 400 range means the file is "unavailable", and the RFC says the crawler "MAY access any resources on the server". A status in the 500 range, or a network failure, means the file is "unreachable", and the crawler "MUST assume complete disallow". Redirects are followed for at least five consecutive hops, and the rules found at the end apply to the host the crawler started with.

That asymmetry is the whole story. A missing file is read as permission. A broken file is read as a locked door. So an error page that your WAF or an overloaded origin returns for a few minutes can change what a crawler does for hours, and it does so without any message to you.

Google's own page on its robots.txt handling adds detail. According to [Google's robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt), for the first 12 hours of a 5xx the crawler stops crawling and keeps retrying. After that it uses the last good copy for up to 30 days. If errors continue past 30 days, behaviour depends on whether the rest of the site is available. Google also says it generally caches the file for up to 24 hours, and RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable. The same Google page says a 403 is not a way to limit crawl rate, because it counts as no file.

Two consequences follow for sites that want to control AI crawlers:

- If your firewall answers a bot with 403 on `/robots.txt`, the bot reads that as "no rules". Your `Disallow` lines for it are never seen.
- If a deploy takes the origin down for ten minutes and `/robots.txt` returns 503 with nothing cached at the edge, well-behaved crawlers pause. That is the safe failure, but it is still a stall you caused.

## What real robots.txt files return

To see whether any of this matters in practice, I requested `/robots.txt` from 50 well-known documentation, framework, CMS and hosting sites on 16 October 2026, following up to ten redirects, with a browser user agent. This is a small hand-picked sample, not a survey. Every file was fetched from its own host over a datacentre connection. Three hosts refused or never answered (two returned 403, one gave no response), so 47 answered.

| Result | Hosts of 47 | Examples |
|---|---|---|
| 200, `text/plain`, no redirect | 40 | astro.build, nextjs.org, react.dev |
| Of those, with `charset=utf-8` declared | 27 | angular.dev, docs.stripe.com |
| One redirect, then `text/plain` | 3 | www.wikipedia.org, joomla.org, www.webflow.com |
| 404 with an HTML page | 3 | gohugo.io, rust-lang.org, tailwindcss.com |
| One redirect, then 200 `text/html` | 1 | www.nginx.com |

The useful rows are the last two. The three 404s are correct behaviour: no file, no restrictions, and a plain 404 is what RFC 9309 expects. The `www.nginx.com` row is the failure worth learning from. Its `/robots.txt` answers 301 to a product page on a different host, which returns 200 with a 370,035-byte HTML document and zero lines that look like robots.txt rules. A nonexistent path on the same host also returned 200 in my control request. A crawler will follow the redirect, parse a web page as a rules file, and find nothing to obey. The outcome is harmless here, but it is the exact shape that goes wrong when the page happens to contain a stray `Disallow:` string.

Our own file at `aiscan.site/robots.txt` returned 200, `text/plain; charset=utf-8`, no redirect and 1,239 bytes, and a nonexistent control path returned a real 404.

## Path 1: fix it at the server, with nginx

Use this path if you run your own nginx and serve a physical file. The directives below were verified on 16 October 2026 against the [nginx core module documentation](https://nginx.org/en/docs/http/ngx_http_core_module.html) and its [charset module page](https://nginx.org/en/docs/http/ngx_http_charset_module.html). I did not run this config here, so test it with `nginx -t` before you reload.

1. Put the file at the web root, for example `/var/www/example/robots.txt`, saved as UTF-8 with no byte order mark.
2. Add an exact-match location so nothing else can shadow the path:

   ```nginx
   location = /robots.txt {
       default_type text/plain;
       charset utf-8;
       access_log off;
   }
   ```

3. Make sure no catch-all rule above it rewrites unknown paths to your app. A `try_files $uri /index.html;` fallback is the usual cause of a 200 HTML page here, and the exact-match block keeps the file out of its reach.
4. Run `nginx -t`, then reload.
5. If you redirect `www` to the apex or `http` to `https`, send the robots.txt request through one redirect only. [The canonical host guide](/blog/canonical-host-redirects-for-agents) shows the single-hop version.

## Path 2: fix it in your framework or CMS

Use this path if a framework or plugin generates the file. The details differ, but the goal is the same: one source of truth that returns `text/plain`.

According to the docs, on Next.js the [robots.txt file convention](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots) (page last updated 1 May 2026) accepts either a static `app/robots.txt` or an `app/robots.ts` that returns a `Robots` object. Pick one. The same page notes that `robots.js` is a special Route Handler that is cached by default unless it uses a request-time API.

On WordPress, a virtual file is generated when no physical one exists. According to [the `do_robots()` source](https://developer.wordpress.org/reference/functions/do_robots/), core sends `Content-Type: text/plain; charset=utf-8` and passes the output through the `robots_txt` filter. If a physical `robots.txt` sits in the site root, your web server normally serves that file and WordPress is never asked, so edits in a plugin appear to do nothing. [ThinkRank](https://thinkrank.ai) is the one we suggest first for WordPress: it replaces the separate SEO plugins that each try to own robots.txt, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress so nothing is re-entered. When two SEO plugins both try to manage the file, one of them wins and the other's edits look like they did nothing.

1. Open `/robots.txt` on your site and note whether it is the virtual file or a physical one (the starter guide shows [how to tell](/blog/robots-txt-starter-for-ai-crawlers)).
2. Delete or rename a stray physical file if you want a plugin to own it.
3. Keep one plugin in charge of the file and deactivate the rest.
4. Reload `/robots.txt` and confirm the type, as in the next section.

Platform notes in one place:

| Stack | Where the file comes from | Common envelope bug |
|---|---|---|
| nginx or Apache | A file in the web root | SPA fallback returns 200 HTML |
| Next.js | `app/robots.txt` or `app/robots.ts` | A redirect or rewrite rule catches the path |
| WordPress | Virtual via `do_robots()` or a physical file | Physical file hides the plugin's rules |
| CDN or WAF in front | Passes through | 403 or a challenge page for bots |

## Check it with AIScan first

Run `npx aiscan-cli yoursite.com`, or paste your address at [aiscan.site](https://aiscan.site/), free and with no account. The relevant result is `check:D1`, which fetches `/robots.txt` and grades whether it returns 200, whether it avoids a catch-all `Disallow: /` for every agent, and whether it advertises a `Sitemap:` line. A 404, a 5xx or a challenge page shows up there as a D1 failure. The evidence line does not always state the response status, which is a known gap that [our evidence-string audit](/blog/evidence-string-audit) documents, so read the status yourself with the check below.

Be clear about what the scan cannot see. D1 does not grade the content type or the charset, it does not measure the 500 KiB limit, and it cannot tell you how a specific crawler treats a status. It sees one request from one place, so a WAF that only blocks certain bots will pass it. The manual check below covers those gaps.

## If you would rather check by hand

You can finish this task with `curl` alone. Save this as `check-robots.sh` and run it with your hostname:

```sh
#!/bin/sh
H="$1"
curl -s -L --max-redirs 5 -o /tmp/robots.out \
  -w 'status=%{http_code} type=%{content_type} hops=%{num_redirects} bytes=%{size_download} final=%{url_effective}\n' \
  "https://$H/robots.txt"
echo "rule lines: $(grep -ciE '^[[:space:]]*(user-agent|disallow|allow|sitemap):' /tmp/robots.out)"
echo "first bytes: $(head -c3 /tmp/robots.out | od -An -tx1)"
curl -s -L --max-redirs 5 -o /dev/null -w 'control status=%{http_code}\n' "https://$H/robots-control-$$.txt"
```

I ran it on 16 October 2026. For `aiscan.site` it printed `status=200 type=text/plain; charset=utf-8 hops=0 bytes=1239`, 39 rule lines and a control status of 404. For `www.nginx.com` it printed `status=200 type=text/html; charset=utf-8 hops=1 bytes=370035`, zero rule lines and a control status of 200. For `gohugo.io` it printed `status=404`, zero rule lines and a control status of 404.

Read the output like this:

| Line | Pass | Fail |
|---|---|---|
| `status` | 200 (or 404 if you truly have no file) | Any 5xx, 403 or 401 |
| `type` | `text/plain`, ideally with `charset=utf-8` | `text/html` or `application/json` |
| `hops` | 0 or 1, never above 5 | More than 5 |
| `bytes` | Under 512000 (500 KiB) | Over the limit |
| `rule lines` | One or more | Zero on a 200 |
| `first bytes` | Not `ef bb bf` | `ef bb bf` is a byte order mark |
| `control` | 404 | 200 means unknown paths are served pages |

Google says it ignores a leading byte order mark, so that row is a tidy-up, not an emergency. The RFC requires UTF-8, and a stray encoding is the safer thing to remove.

## Common failures and what they mean

- **Rules ignored for one bot.** Your firewall answers that bot with 403 or a challenge. The bot sees "no file". Allow `/robots.txt` through the firewall for every user agent.
- **Rules ignored for everyone.** The path returns 200 HTML from your app. Add the exact-match location above, or define the route in your framework.
- **Crawling stops after a deploy.** `/robots.txt` returned 503 during the restart. Serve it from the edge or a static host so it stays up while the origin restarts. The [rate-limit guide](/blog/rate-limit-ai-crawlers-without-blocking) covers the related 429 and 503 cases.
- **Status says 200 but nothing matches.** Check the final URL in the output. A redirect to a different host means the rules came from there. See [the soft-404 problem](/blog/soft-404-agent-discovery-problem) for how a 200 can hide a missing file.

## Next action

Run `npx aiscan-cli yoursite.com` and read the `check:D1` result, then run `check-robots.sh` against your own hostname and fix the first row that fails. Browse the rest of the series in [the AIScan guides](/guides).