Why AI Rules Vanish From robots.txt: 25 of 144 Passing Sites Lost Them

Of 144 hosts that passed the AI-crawler robots.txt check (B2) on their first scan, 25 failed it on their latest scan. All 25 had exactly 8 known AI agents named in robots.txt the first time and 0 the last time. Among the other 100 hosts that passed B2 with any other agent count, none regressed. That is the whole shape of this study: B2 does not decay at random, it fell off one pattern, and the falls bunch up in the scans taken after 28 September 2026.

Quick summary

Short answer for a site owner: if your robots.txt used to name AI crawlers and your latest scan says "0 known AI agent(s) addressed", the usual cause is that the rules are gone while the file still loads. Open your live /robots.txt, search for GPTBot, ClaudeBot and PerplexityBot, and put the block back. To see it from outside, run npx aiscan-cli yoursite.com (or paste the address at aiscan.site, free, no account) and read checks B2, D1 and B1. If you would rather check by hand, the curl command is in the fix section.

What this study measures, and what it cannot

We took every host with at least two site-scope scans under rubric 2026.08.2, at least one hour apart, and compared its first scan with its latest. aiscan.site is excluded and no scanned host is named. There are 574 such hosts. The count is lower than in the rescan study from 6 November because this query keeps only scans stored with site scope.

Five limits matter more than usual here:

• The stored report has no file body. B2's evidence is a count: "8 known AI agent(s) addressed in robots.txt". It does not say which eight, and it does not say what the file contains now. We can see that the count went from 8 to 0. We cannot see an edit.
• Two points in time. A host scanned five times appears once, first against latest. The regressed hosts averaged 3 to 5 scans each. A change made and reverted in between is invisible.
• Self-selected. People rescan after they act. These are sites whose owners came back.
• We cannot separate a site change from a scanner change. Every row uses the same rubric version, and the pattern is too uniform to be 25 owners making 25 separate decisions, so a shared cause is the likely explanation. The data cannot say whether that cause lives on the sites or in how we probe them. Read the result as a lead, not a verdict.
• Demand is thin. We found no search query for this topic in Search Console and no keyword tool on our plan. The gap is that existing guides explain how to add AI rules and say nothing about keeping them.

How B2 decides

B2 reads /robots.txt and counts how many known AI crawler names it addresses. A file that names enough of them passes and a file that names none fails. The stored evidence does not state the pass threshold, only the count. In this sample, hosts that passed named between 4 and 16. The evidence string audit found that B2 produces a single sentence of evidence whatever the failure, which is why the status is the reliable column here and the sentence is a count only.

The rule matters for reading the 25. A host that went from 8 agents to 0 did not lose "some" rules. It lost every rule B2 recognises, so the file now says nothing to AI crawlers by name.

The 25: one starting point

If agent counts were lost at random, the regressions would spread across rows. They sit in one. An 8-agent file is the shared shape of a default or a plugin template, and 44 hosts had it. Which template, we cannot say from stored evidence, and we are not going to guess a product name. What the table does show is that a block of exactly eight names is fragile in this sample and a block of any other size is not.

The regressed hosts are not one platform. Ten have no detected platform, seven are WordPress, five Next.js, and one each Lovable, Shopify and Astro. Platform is a weak lens: WordPress and Next.js hosts regress at similar counts, which fits a cause that is not tied to one CMS.

When it happened

Before the week of 28 September, all 12 rescanned 8-agent hosts still had 8. From that week to the end of October, 25 of 29 dropped to 0. The three in the week of 2 November kept their 8, and so did a few earlier. This is the pattern of something changing at one point in time, which is why we do not read it as owners acting independently.

Two explanations fit and the data cannot choose. One is a site-side change: a template, plugin or hosting default that rewrote robots.txt for many sites at once. The other is on our side: a change in how the scanner fetches or parses robots.txt that stopped counting a block that used to count. The seven 8-agent hosts that kept their count after 28 September argue against a blanket scanner fault, but they do not rule out a narrower one. We have not found a scanner change on those dates in our own notes, and we will say so if one turns up.

The file usually still loads

Twenty-two of the 25 (88%) still answer 200 with a readable file. So the usual failure is not "robots.txt disappeared". It is "robots.txt answers and names nobody". That is the hardest version to notice, because a browser shows a file, a status monitor shows 200, and nothing is red. The robots.txt status and content type post covers the other failure, where the file exists but is served wrongly.

Three hosts are different. Two now refuse the scanner with 403 and one answers 404. RFC 9309 separates the cases. Section 2.3.1.3 says a 4xx means the file is unavailable and the crawler MAY access any resources on the server. Section 2.3.1.4 says a 5xx or network error makes the file unreachable and the crawler MUST assume complete disallow. A 403 on /robots.txt therefore reads, to a standards-following crawler, as "no rules", which is the opposite of what an owner blocking bots usually wants. The blocked probe share study found that 11.9% of hosts refuse our scanner at some point and that those hosts fail more checks for that reason alone.

What else moved with it

Sixteen of the 25 regressed hosts pass E1 (a proper 404 on a missing page) on their latest scan, so most are healthy sites. Their mean score fell about 13 points. The co-pass matrix shows B2 sitting in a bundle with C1 and C2, so a fall in one is not always alone. We did not break the 13 points down per check for the 25, and the sample is too small for us to claim a cause.

The flip side is the 64 hosts that moved to pass. 63 of them had 0 agents on the first scan (one had 2), and on the latest they named between 4 and 16: 15 hosts at 15 agents, 8 at 9, 7 at 14, 6 at 16, 5 at 13, 5 at 7, and the rest spread across 4 to 12. Their average gap between scans was 23.9 days, against 30.9 days for the regressed. Adding rules is a deliberate act with a visible result. Losing them is quiet. The repeat scan study found the same asymmetry in October: B2 lost 25 of 89 passes then, and 21 of those 25 still answered 200.

What losing the block changes for a crawler

A robots.txt with no group for a crawler's name is not a ban. Under RFC 9309, a crawler follows the group that matches its user agent, and if none matches it falls back to the * group, if there is one. A site that listed eight AI crawlers to disallow them, then lost the list, is now open to those crawlers unless a * rule says otherwise. A site that listed eight to allow them explicitly loses less: the default for a missing rule is access. So the cost of the regression depends on which side the owner was on, and the B2 count does not tell us. It only tells us that the owner's stated intent is no longer in the file.

That is also why B2 is a check on a declaration, not on behaviour. The fake crawler traffic study measured our own server logs and found that many requests presenting AI crawler names never behaved like the real crawler. A robots.txt rule is a request that honest crawlers follow. It does not enforce anything, and losing the rule matters to the honest ones only.

How to read your own result

The scan gives you one of four outcomes for B2, and each points at a different job.

Start with D1 whenever both fail. In the October data, 59 of the hosts whose D1 probe was blocked also failed B2, which means the B2 failure was a symptom, not a separate fault (see the blocked probe share study).

What we are doing with this finding

The uniform 8-to-0 pattern is the part we cannot explain from stored evidence, and the honest next step is to find out whether it is on the sites or in our probe. Stored B2 evidence keeps only a count, so we cannot replay the original fetch. A future version of this check that stores the matched agent names would settle it, because we could then see whether the same eight names disappeared. We have noted that as a site issue for the scan team. If you run one of the 25 sites and the scan shows 0 where you expect rules, the quickest test is the curl command below: if your rules are in the live file, tell us, because that points at the scanner.

How to keep AI rules from disappearing

Treat the AI block in robots.txt as something with an owner, a copy and a test, because the failure here is silent.

1. Scan, then scan again on a schedule. Run npx aiscan-cli yoursite.com after any plugin, theme, CMS or hosting change and watch B2 and D1. AIScan's monitoring tools can track a site and rescan it automatically, so a drop from pass to fail reaches you instead of waiting for you to look. B2 counts recognised AI agents, D1 checks that robots.txt answers and declares a sitemap, and B1 checks Content Signals.
2. Keep the block in version control or a saved copy. If a generated file is rewritten by an update, you have the text to put back.
3. Test the live file, not the editor. Plugins show you the rules they intend to write. The scan shows what a crawler is served.
4. On WordPress, avoid three tools writing one file. ThinkRank manages robots.txt, robots meta, schema and llms.txt from one plugin and migrates settings from Rank Math, Yoast, AIOSEO and SEOPress, so there is one source of truth for the AI block. Yoast, Rank Math and AIOSEO each have a robots.txt editor too and do the job if you already run them, as long as only one of them is writing the file.
5. Know what a 403 or 5xx on robots.txt means to a crawler. A 4xx is read as no rules. A 5xx is read as disallow everything, for as long as it lasts.

If you would rather check by hand:

curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://yoursite.com/robots.txt
curl -s https://yoursite.com/robots.txt | grep -i -E 'GPTBot|ClaudeBot|PerplexityBot|Google-Extended|CCBot'

The first line should print 200 text/plain. The second should print at least one User-agent: line for each crawler you mean to address.

What AIScan can and cannot see

AIScan fetches /robots.txt as an anonymous client and counts the recognised agent names. It cannot tell you which names it counted, why a count dropped, whether a CDN served a different file to a different client, or whether your rules say allow or disallow. It does not read the file's history. The comparison above is two fetches, and everything in between is inference.

The full list of checks is in the bot access docs, and the guides index has the fix paths for each check.

Next step

Scan your site today and note B2, D1 and B1. If B2 says "0 known AI agent(s) addressed in robots.txt" and you expected rules there, restore the block and scan again. Then add a rescan to whatever process updates your CMS, theme or SEO plugin. The guides list every check with its fix.