---
title: "Your robots.txt blocks everything by accident: find the leftover Disallow: / and stop it coming back"
slug: robots-txt-disallow-all-by-accident
published: 2026-10-30T13:20:21.727265+00:00
updated: 2026-10-30T13:20:21.727265+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:D1, check:B2, platform:wordpress, platform:nextjs
description: "Find the leftover Disallow: / in your robots.txt, trace it to staging, WordPress or Next.js, and add a deploy check so it cannot return. Checks D1 and B2."
url: https://aiscan.site/blog/robots-txt-disallow-all-by-accident
---

If your robots.txt says `User-agent: *` followed by `Disallow: /`, every crawler that obeys the file is told to stay off your whole site. It takes one line to cause and about a minute to find, and Google's Page indexing report will label the result "blocked by robots.txt" once it notices. This guide shows how to confirm the line is live, where it came from, and the five-line deploy step that stops it coming back.

## Quick summary

**Run `curl -s https://yoursite.com/robots.txt`.** If the first group is `User-agent: *` with `Disallow: /` and nothing allowing `/`, the file is blocking everything. Delete the line, redeploy, and add the guard below so the next deploy cannot put it back.

| Where it came from | How to tell | Fix |
|---|---|---|
| Staging file promoted by a deploy | Production file is byte-identical to staging's | Serve robots.txt from per-environment config |
| WordPress **Settings > Reading** box ticked | `noindex,nofollow` in the page `<head>`, robots.txt looks normal | Untick the box |
| Framework `robots` file keyed to the wrong variable | Local build is fine, deployed file is not | Key it to the deploy environment |
| Host preview rules on the custom domain | Rule appears only on one hostname | Check the host's per-domain settings |
| `robots.txt` returns 5xx | Status is 500 to 599 | Fix the server error |

## Why one line blocks everything

According to [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), a crawler uses the group that names it, and falls back to the `*` group only if no group names it. Within a group, the most specific matching rule wins, and `Disallow: /` matches every path. So a lone `User-agent: *` group with `Disallow: /` shuts out every crawler that has no group of its own.

The reverse also holds, and it corrects a common assumption. I parsed a file with a `*` block plus a separate `GPTBot` group that allows `/` using Python's `urllib.robotparser`. Result: GPTBot may fetch `/blog/`, ClaudeBot and Googlebot may not. A named `Allow` group is not moot. It only helps the bots you named.

One more RFC rule bites here. If robots.txt cannot be fetched because of a server error (a 5xx status), the RFC says the crawler "MUST assume complete disallow". A broken deploy that makes `/robots.txt` return a 500 blocks the site as effectively as the line does.

## Step 1: Confirm what crawlers actually get

```bash
curl -sI https://yoursite.com/robots.txt | head -n 5
curl -s  https://yoursite.com/robots.txt
```

Look for three things: a `200` status, a `text/plain` content type, and the group above. Check both `www` and the bare domain, because a preview or staging rule can sit on one hostname only.

Blocked pages are not removed from search. According to [Google's robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro), the file is not a mechanism for keeping a page out of Google, so a blocked URL can still be listed, usually with no description. Search Console shows that state as "Indexed, though blocked by robots.txt".

## Step 2: Find where the line came from

**A staging file promoted by a deploy.** Teams often keep a blanket-disallow robots.txt on staging on purpose. If the file lives in the repository and is copied by the build, production gets it. Compare hashes: `curl -s https://staging.yoursite.com/robots.txt | sha256sum` against production.

**WordPress.** The checkbox is under **Settings > Reading > Search engine visibility > Discourage search engines from indexing this site**. According to the [WordPress documentation](https://wordpress.org/documentation/article/settings-reading-screen/), since version 5.3 ticking it adds `<meta name='robots' content='noindex,nofollow' />` to the page head. Up to version 5.2 it made robots.txt return `Disallow: /`. The current [`do_robots()` source](https://developer.wordpress.org/reference/functions/do_robots/) writes `Disallow: /wp-admin/` and an `Allow` for `admin-ajax.php`, and only passes the "public" flag to the `robots_txt` filter. So on a current install a ticked box does not put `Disallow: /` in robots.txt. A plugin or theme using that filter can. If the file is blocking and the box is clear, look for a plugin that writes robots.txt. [ThinkRank](https://thinkrank.ai) has a robots.txt editor in one plugin, so you can see and change the real file in one place.

**A framework `robots` file.** In Next.js, `app/robots.ts` returns the rules object, and the [Next.js docs example](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots) shows `allow: '/'` with a narrower `disallow`. A common bug is deciding "is this production?" with a variable that is set the same way on every deploy. According to [Vercel's system environment variable docs](https://vercel.com/docs/environment-variables/system-environment-variables), `VERCEL_ENV` is `production`, `preview` or `development`, which is the variable to branch on:

```ts
// app/robots.ts
export default function robots() {
  const live = process.env.VERCEL_ENV === 'production'
  return {
    rules: live ? { userAgent: '*', allow: '/' } : { userAgent: '*', disallow: '/' },
    sitemap: 'https://yoursite.com/sitemap.xml',
  }
}
```

If you branch on something else, test that variable's value in a real production build before you trust it.

**Host preview rules.** If the rule shows on one hostname only, look in your host's settings for that domain, not in your code.

## Step 3: Guard the deploy

This step runs against the live URL after every deploy and fails the job if production blocks everything. I tested it on four files, 30 October 2026:

| Test file | Guard result |
|---|---|
| Lone `User-agent: *` + `Disallow: /` | FAIL, exit 1 |
| Same block plus a `GPTBot` group with `Allow: /` | FAIL, exit 1 |
| WordPress-style file (`Disallow: /wp-admin/`, a Sitemap line) | ok, exit 0 |
| Lowercase `user-agent: *` and `disallow: /` with trailing spaces | FAIL, exit 1 |


```sh
#!/bin/sh
# usage: ./robots-guard.sh https://yoursite.com/robots.txt
f=$(mktemp); curl -fsS "$1" -o "$f" || exit 2
if tr -d '\r' < "$f" | grep -Pzoi '(?m)^user-agent:[ \t]*\*[ \t]*\n(?:(?:allow|disallow|crawl-delay|#)[^\n]*\n)*?disallow:[ \t]*/[ \t]*(?:#[^\n]*)?(?:\n|$)' >/dev/null; then
  echo "FAIL: robots.txt disallows / for *"; exit 1
fi
echo ok
```

It exits 2 when `curl` fails, so a 404 or 5xx fails the build too (checked against a real 404). It is deliberately simple: it only catches a `*` group containing `Disallow: /`, and it will also flag a group that pairs it with `Allow: /$`, which is the safe direction. For a full regression gate on a whole site, see [Gate AI readiness in CI](/blog/aiscan-ci-gate-github-actions).

## Step 4: Verify

**Run the scan first.** `npx aiscan-cli yoursite.com` reports check D1 (robots.txt present and sane) under discoverability and B2 (explicit AI bot rules) under bot access. Then confirm by hand with `curl -s https://yoursite.com/robots.txt` and the guard script above.

One thing to know before you trust a green D1. I scanned `mail.google.com` on 30 October 2026. Its robots.txt is a `*` group of `Disallow: /` with `Allow: /$` and `Allow: /mail/$` lines, plus a `Sitemap:` line.

| Check | Status | Evidence AIScan printed |
|---|---|---|
| D1 robots.txt present and sane | pass | 200 OK, 130 bytes, declares Sitemap |
| B2 explicit AI bot rules | fail | 0 known AI agent(s) addressed in robots.txt |

That is one site, and its file allows the root, so it is not the lone `Disallow: /` case. According to AIScan's [Sitemap-line study](/blog/robots-txt-sitemap-line-corpus), D1 wants a 200, no catch-all `User-agent: *` block and a `Sitemap:` URL. Treat D1 as one signal, and keep the guard script as the one that fails your build.

According to the [AI crawler declaration census](/blog/ai-crawler-declaration-census), 11 of 451 parsed files (2.4%) carried a blanket `User-agent: *` full block. Some are on purpose. The fix is the same: decide it was meant, or remove it.

## Maintenance

- Run the guard after every deploy, not only on releases that touch robots.txt.
- Keep staging protected by a login or an `X-Robots-Tag` header, not by a robots.txt that can be copied by accident.
- After you fix it, request a recrawl in Search Console and allow days for the status to clear.
- What to write on purpose is covered in the [robots.txt starter for AI crawlers](/blog/robots-txt-starter-for-ai-crawlers), and the status and content type in [Serve robots.txt with the right status and content type](/blog/robots-txt-status-content-type).

## Next step

Run `npx aiscan-cli yoursite.com`, check D1 and B2, then add the guard to your deploy. More fixes are in the [AIScan guides](/guides), and the [discoverability check page](/docs/checks/discoverability) explains what D1 looks for.