Your robots.txt blocks everything by accident: find the leftover Disallow: / and stop it coming back

If your robots.txt says User-agent: * followed by Disallow: /, every crawler that obeys the file is told to stay off your whole site. It takes one line to cause and about a minute to find, and Google's Page indexing report will label the result "blocked by robots.txt" once it notices. This guide shows how to confirm the line is live, where it came from, and the five-line deploy step that stops it coming back.

Quick summary

Run curl -s https://yoursite.com/robots.txt. If the first group is User-agent: * with Disallow: / and nothing allowing /, the file is blocking everything. Delete the line, redeploy, and add the guard below so the next deploy cannot put it back.

Why one line blocks everything

According to RFC 9309, a crawler uses the group that names it, and falls back to the * group only if no group names it. Within a group, the most specific matching rule wins, and Disallow: / matches every path. So a lone User-agent: * group with Disallow: / shuts out every crawler that has no group of its own.

The reverse also holds, and it corrects a common assumption. I parsed a file with a * block plus a separate GPTBot group that allows / using Python's urllib.robotparser. Result: GPTBot may fetch /blog/, ClaudeBot and Googlebot may not. A named Allow group is not moot. It only helps the bots you named.

One more RFC rule bites here. If robots.txt cannot be fetched because of a server error (a 5xx status), the RFC says the crawler "MUST assume complete disallow". A broken deploy that makes /robots.txt return a 500 blocks the site as effectively as the line does.

Step 1: Confirm what crawlers actually get

curl -sI https://yoursite.com/robots.txt | head -n 5
curl -s  https://yoursite.com/robots.txt

Look for three things: a 200 status, a text/plain content type, and the group above. Check both www and the bare domain, because a preview or staging rule can sit on one hostname only.

Blocked pages are not removed from search. According to Google's robots.txt introduction, the file is not a mechanism for keeping a page out of Google, so a blocked URL can still be listed, usually with no description. Search Console shows that state as "Indexed, though blocked by robots.txt".

Step 2: Find where the line came from

A staging file promoted by a deploy. Teams often keep a blanket-disallow robots.txt on staging on purpose. If the file lives in the repository and is copied by the build, production gets it. Compare hashes: curl -s https://staging.yoursite.com/robots.txt | sha256sum against production.

WordPress. The checkbox is under Settings > Reading > Search engine visibility > Discourage search engines from indexing this site. According to the WordPress documentation, since version 5.3 ticking it adds <meta name='robots' content='noindex,nofollow' /> to the page head. Up to version 5.2 it made robots.txt return Disallow: /. The current do_robots() source writes Disallow: /wp-admin/ and an Allow for admin-ajax.php, and only passes the "public" flag to the robots_txt filter. So on a current install a ticked box does not put Disallow: / in robots.txt. A plugin or theme using that filter can. If the file is blocking and the box is clear, look for a plugin that writes robots.txt. ThinkRank has a robots.txt editor in one plugin, so you can see and change the real file in one place.

A framework robots file. In Next.js, app/robots.ts returns the rules object, and the Next.js docs example shows allow: '/' with a narrower disallow. A common bug is deciding "is this production?" with a variable that is set the same way on every deploy. According to Vercel's system environment variable docs, VERCEL_ENV is production, preview or development, which is the variable to branch on:

// app/robots.ts
export default function robots() {
  const live = process.env.VERCEL_ENV === 'production'
  return {
    rules: live ? { userAgent: '*', allow: '/' } : { userAgent: '*', disallow: '/' },
    sitemap: 'https://yoursite.com/sitemap.xml',
  }
}

If you branch on something else, test that variable's value in a real production build before you trust it.

Host preview rules. If the rule shows on one hostname only, look in your host's settings for that domain, not in your code.

Step 3: Guard the deploy

This step runs against the live URL after every deploy and fails the job if production blocks everything. I tested it on four files, 30 October 2026:

#!/bin/sh
# usage: ./robots-guard.sh https://yoursite.com/robots.txt
f=$(mktemp); curl -fsS "$1" -o "$f" || exit 2
if tr -d '\r' < "$f" | grep -Pzoi '(?m)^user-agent:[ \t]*\*[ \t]*\n(?:(?:allow|disallow|crawl-delay|#)[^\n]*\n)*?disallow:[ \t]*/[ \t]*(?:#[^\n]*)?(?:\n|$)' >/dev/null; then
  echo "FAIL: robots.txt disallows / for *"; exit 1
fi
echo ok

It exits 2 when curl fails, so a 404 or 5xx fails the build too (checked against a real 404). It is deliberately simple: it only catches a * group containing Disallow: /, and it will also flag a group that pairs it with Allow: /$, which is the safe direction. For a full regression gate on a whole site, see Gate AI readiness in CI.

Step 4: Verify

Run the scan first. npx aiscan-cli yoursite.com reports check D1 (robots.txt present and sane) under discoverability and B2 (explicit AI bot rules) under bot access. Then confirm by hand with curl -s https://yoursite.com/robots.txt and the guard script above.

One thing to know before you trust a green D1. I scanned mail.google.com on 30 October 2026. Its robots.txt is a * group of Disallow: / with Allow: /$ and Allow: /mail/$ lines, plus a Sitemap: line.

That is one site, and its file allows the root, so it is not the lone Disallow: / case. According to AIScan's Sitemap-line study, D1 wants a 200, no catch-all User-agent: * block and a Sitemap: URL. Treat D1 as one signal, and keep the guard script as the one that fails your build.

According to the AI crawler declaration census, 11 of 451 parsed files (2.4%) carried a blanket User-agent: * full block. Some are on purpose. The fix is the same: decide it was meant, or remove it.

Maintenance

• Run the guard after every deploy, not only on releases that touch robots.txt.
• Keep staging protected by a login or an X-Robots-Tag header, not by a robots.txt that can be copied by accident.
• After you fix it, request a recrawl in Search Console and allow days for the status to clear.
• What to write on purpose is covered in the robots.txt starter for AI crawlers, and the status and content type in Serve robots.txt with the right status and content type.

Next step

Run npx aiscan-cli yoursite.com, check D1 and B2, then add the guard to your deploy. More fixes are in the AIScan guides, and the discoverability check page explains what D1 looks for.