Serve robots.txt with the right status and content type

Most robots.txt advice is about the rules inside the file. This guide is about the envelope: the HTTP status, the content type, the redirects and the size. A perfect set of rules does nothing if the response around it makes a crawler throw the file away, and in one case the response makes a crawler stop visiting your site altogether.

The reason is that a crawler never reads your file in isolation. It first reads the response, and the response decides whether the rules count. RFC 9309, the Robots Exclusion Protocol, spells that out in a short section on access results. Most sites never look at it until something breaks.

Quick summary

Three habits fix almost every case. Serve the file as text/plain; charset=utf-8. Return a plain 404 if you have no file, never a 200 with a page. Keep the file under 500 KiB and reachable on the exact host the crawler asked for. The scan check that covers this is check:D1, and the discoverability docs list what it grades.

Why the status code is part of the rule

Think of the status code as the gatekeeper's first answer, given before the crawler reads a word of the rules. According to RFC 9309, section 2.3.1, there are four outcomes, and they are asymmetric.

A successful fetch means the crawler follows the rules it can parse. A status in the 400 range means the file is "unavailable", and the RFC says the crawler "MAY access any resources on the server". A status in the 500 range, or a network failure, means the file is "unreachable", and the crawler "MUST assume complete disallow". Redirects are followed for at least five consecutive hops, and the rules found at the end apply to the host the crawler started with.

That asymmetry is the whole story. A missing file is read as permission. A broken file is read as a locked door. So an error page that your WAF or an overloaded origin returns for a few minutes can change what a crawler does for hours, and it does so without any message to you.

Google's own page on its robots.txt handling adds detail. According to Google's robots.txt specification, for the first 12 hours of a 5xx the crawler stops crawling and keeps retrying. After that it uses the last good copy for up to 30 days. If errors continue past 30 days, behaviour depends on whether the rest of the site is available. Google also says it generally caches the file for up to 24 hours, and RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable. The same Google page says a 403 is not a way to limit crawl rate, because it counts as no file.

Two consequences follow for sites that want to control AI crawlers:

• If your firewall answers a bot with 403 on /robots.txt, the bot reads that as "no rules". Your Disallow lines for it are never seen.
• If a deploy takes the origin down for ten minutes and /robots.txt returns 503 with nothing cached at the edge, well-behaved crawlers pause. That is the safe failure, but it is still a stall you caused.

What real robots.txt files return

To see whether any of this matters in practice, I requested /robots.txt from 50 well-known documentation, framework, CMS and hosting sites on 16 October 2026, following up to ten redirects, with a browser user agent. This is a small hand-picked sample, not a survey. Every file was fetched from its own host over a datacentre connection. Three hosts refused or never answered (two returned 403, one gave no response), so 47 answered.

The useful rows are the last two. The three 404s are correct behaviour: no file, no restrictions, and a plain 404 is what RFC 9309 expects. The www.nginx.com row is the failure worth learning from. Its /robots.txt answers 301 to a product page on a different host, which returns 200 with a 370,035-byte HTML document and zero lines that look like robots.txt rules. A nonexistent path on the same host also returned 200 in my control request. A crawler will follow the redirect, parse a web page as a rules file, and find nothing to obey. The outcome is harmless here, but it is the exact shape that goes wrong when the page happens to contain a stray Disallow: string.

Our own file at aiscan.site/robots.txt returned 200, text/plain; charset=utf-8, no redirect and 1,239 bytes, and a nonexistent control path returned a real 404.

Path 1: fix it at the server, with nginx

Use this path if you run your own nginx and serve a physical file. The directives below were verified on 16 October 2026 against the nginx core module documentation and its charset module page. I did not run this config here, so test it with nginx -t before you reload.

1. Put the file at the web root, for example /var/www/example/robots.txt, saved as UTF-8 with no byte order mark.

2. Add an exact-match location so nothing else can shadow the path:

   location = /robots.txt {
       default_type text/plain;
       charset utf-8;
       access_log off;
   }
   

3. Make sure no catch-all rule above it rewrites unknown paths to your app. A try_files $uri /index.html; fallback is the usual cause of a 200 HTML page here, and the exact-match block keeps the file out of its reach.

4. Run nginx -t, then reload.

5. If you redirect www to the apex or http to https, send the robots.txt request through one redirect only. The canonical host guide shows the single-hop version.

Path 2: fix it in your framework or CMS

Use this path if a framework or plugin generates the file. The details differ, but the goal is the same: one source of truth that returns text/plain.

According to the docs, on Next.js the robots.txt file convention (page last updated 1 May 2026) accepts either a static app/robots.txt or an app/robots.ts that returns a Robots object. Pick one. The same page notes that robots.js is a special Route Handler that is cached by default unless it uses a request-time API.

On WordPress, a virtual file is generated when no physical one exists. According to the do_robots() source, core sends Content-Type: text/plain; charset=utf-8 and passes the output through the robots_txt filter. If a physical robots.txt sits in the site root, your web server normally serves that file and WordPress is never asked, so edits in a plugin appear to do nothing. ThinkRank is the one we suggest first for WordPress: it replaces the separate SEO plugins that each try to own robots.txt, and it migrates from Rank Math, Yoast, AIOSEO and SEOPress so nothing is re-entered. When two SEO plugins both try to manage the file, one of them wins and the other's edits look like they did nothing.

1. Open /robots.txt on your site and note whether it is the virtual file or a physical one (the starter guide shows how to tell).
2. Delete or rename a stray physical file if you want a plugin to own it.
3. Keep one plugin in charge of the file and deactivate the rest.
4. Reload /robots.txt and confirm the type, as in the next section.

Platform notes in one place:

Check it with AIScan first

Run npx aiscan-cli yoursite.com, or paste your address at aiscan.site, free and with no account. The relevant result is check:D1, which fetches /robots.txt and grades whether it returns 200, whether it avoids a catch-all Disallow: / for every agent, and whether it advertises a Sitemap: line. A 404, a 5xx or a challenge page shows up there as a D1 failure. The evidence line does not always state the response status, which is a known gap that our evidence-string audit documents, so read the status yourself with the check below.

Be clear about what the scan cannot see. D1 does not grade the content type or the charset, it does not measure the 500 KiB limit, and it cannot tell you how a specific crawler treats a status. It sees one request from one place, so a WAF that only blocks certain bots will pass it. The manual check below covers those gaps.

If you would rather check by hand

You can finish this task with curl alone. Save this as check-robots.sh and run it with your hostname:

#!/bin/sh
H="$1"
curl -s -L --max-redirs 5 -o /tmp/robots.out \
  -w 'status=%{http_code} type=%{content_type} hops=%{num_redirects} bytes=%{size_download} final=%{url_effective}\n' \
  "https://$H/robots.txt"
echo "rule lines: $(grep -ciE '^[[:space:]]*(user-agent|disallow|allow|sitemap):' /tmp/robots.out)"
echo "first bytes: $(head -c3 /tmp/robots.out | od -An -tx1)"
curl -s -L --max-redirs 5 -o /dev/null -w 'control status=%{http_code}\n' "https://$H/robots-control-$$.txt"

I ran it on 16 October 2026. For aiscan.site it printed status=200 type=text/plain; charset=utf-8 hops=0 bytes=1239, 39 rule lines and a control status of 404. For www.nginx.com it printed status=200 type=text/html; charset=utf-8 hops=1 bytes=370035, zero rule lines and a control status of 200. For gohugo.io it printed status=404, zero rule lines and a control status of 404.

Read the output like this:

Google says it ignores a leading byte order mark, so that row is a tidy-up, not an emergency. The RFC requires UTF-8, and a stray encoding is the safer thing to remove.

Common failures and what they mean

• Rules ignored for one bot. Your firewall answers that bot with 403 or a challenge. The bot sees "no file". Allow /robots.txt through the firewall for every user agent.
• Rules ignored for everyone. The path returns 200 HTML from your app. Add the exact-match location above, or define the route in your framework.
• Crawling stops after a deploy. /robots.txt returned 503 during the restart. Serve it from the edge or a static host so it stays up while the origin restarts. The rate-limit guide covers the related 429 and 503 cases.
• Status says 200 but nothing matches. Check the final URL in the output. A redirect to a different host means the rules came from there. See the soft-404 problem for how a 200 can hide a missing file.

Next action

Run npx aiscan-cli yoursite.com and read the check:D1 result, then run check-robots.sh against your own hostname and fix the first row that fails. Browse the rest of the series in the AIScan guides.