Does AIScan Give the Same Score Twice? 303 Repeat Scans Measured

Every scanner owes its users one answer before it asks for trust: if I run you twice, do I get the same grade? Most tools never publish it. We can, because AIScan keeps every scan it has ever run, and a lot of people scan the same site twice in a row. On 5 October 2026 we queried our own production database for those repeat scans, 303 pairs of the same site scanned within ten minutes of each other, and compared the two reports check by check.

The headline is uncomfortable for a company that sells a score. Of those 303 back-to-back pairs, 235 produced an identical overall score and 68 did not. Thirteen differed by 30 points or more. Our own rubric is not deterministic, some checks are far less stable than others, and one of the least stable has a cause we can name and fix. This post gives the numbers, the evidence strings behind them, what they mean for anyone using a score as a gate, and the changes we are filing against our own scanner.

Quick summary

What the data says: about one pair in five changes at least one check verdict, and most of that movement sits in five checks that depend on what the target decides to serve a scanner at that moment. The checks that look rock solid are mostly the ones that rarely score anything at all.

Best for a site owner: treat a check that disagrees between two scans as a finding in itself, then fix the part you control. Best for a team with a CI gate: gate on a score floor with a few points of margin and one retry, not on a single exact number. The CI gate guide shows the wiring. Best for anyone comparing scanners: ask each vendor for this table.

How we measured it

Method first, because the limits decide how far you can trust the rest. Every figure here was queried on 5 October 2026 against the live production database.

The population is every site-scope scan stored under rubric version 2026.08.2, which ran from 24 August to 5 October 2026. A host is the URL with its scheme and any leading www. removed, and we excluded aiscan.site, which has scanned itself many times and would flatter the result. For each host we ordered its scans by time and paired every scan with the one immediately before it. A pair counts if the gap is ten minutes or less. That gave 303 pairs across 119 hosts. A wider bucket covers gaps of more than an hour and up to 24 hours, with 356 pairs, and the 10 minute to 1 hour bucket holds 110 more, which we use only for context.

Two caveats belong here rather than at the end. First, pairs are consecutive, so a host scanned three times inside ten minutes contributes two overlapping pairs. They are not independent samples. Second, we did not run these scans for the experiment. People ran them, for their own reasons, so we cannot separate a scanner that disagreed with itself from an owner who changed something between scans. The 24-hour bucket and the evidence strings help with that, and we use both below.

All of it is aggregate SQL. No row-level query was run and no individual host is named in this post. The core shape is a window function that pairs each scan with its predecessor, then a lateral unnest of the report's check array on both sides:

WITH s AS (
  SELECT lower(regexp_replace(url, '^https?://(www\.)?([^/]+).*$', '\2')) AS host,
         created_at, result
  FROM scans
  WHERE rubric_version = '2026.08.2' AND scope = 'site'
), p AS (
  SELECT host, created_at, result,
         lag(result)     OVER (PARTITION BY host ORDER BY created_at) AS pres,
         lag(created_at) OVER (PARTITION BY host ORDER BY created_at) AS prev
  FROM s WHERE host <> 'aiscan.site'
), pp AS (
  SELECT * FROM p
  WHERE prev IS NOT NULL AND created_at - prev <= interval '10 minutes'
)
SELECT c1->>'id' AS check_id,
       count(*) AS n,
       count(*) FILTER (WHERE c1->>'status' <> c2->>'status') AS flips
FROM pp
CROSS JOIN LATERAL jsonb_array_elements(pres->'checks') c1
JOIN LATERAL jsonb_array_elements(pp.result->'checks') c2
  ON c2->>'id' = c1->>'id'
GROUP BY 1 ORDER BY 1;

A report carries 18 checks, or 21 when commerce applies. M1, M3 and M4 therefore have 128 pairs, not 303, and we never divide a commerce check by the full sample. A "flip" means the status string changed between the two scans: pass, partial, fail, info (bonus checks, weight zero) or na (not applicable).

Does a second scan return the same score?

Mostly, and not always. Here is the distribution of absolute score difference across the 303 pairs.

The largest gap was 79 points. Among the 68 pairs that differed, 31 distinct hosts were involved out of 119, so roughly one host in four disagreed with itself at least once.

The distribution has two humps. A group of small movements is what you expect from one or two borderline checks. The 13 pairs at 30 points or more are a different animal, and they are the ones to worry about. A swing that size means several checks changed together, which points at the target returning a different page or an error on one of the two requests, not at a borderline threshold. Counting verdict changes per pair supports that.

The most checks to flip in a single pair was 14 of 18. When a scan returns 14 different verdicts ten minutes later, nobody changed 14 things on their website in ten minutes.

Spacing barely matters at the shortest end. Among the 157 pairs scanned within 30 seconds of each other, 19 differed in score, which is 12.1%. That is lower than the 22.4% for ten minutes but it is not zero, and a window of half a minute leaves almost no room for an owner to have edited anything. Of the numbers in this post, this is the one that comes closest to measuring pure scanner and network variance.

Check by check

This table is the core of the post: for every check, how often its status differed across the 303 pairs, and then across the 356 pairs scanned between one and 24 hours apart. Commerce checks use 128 and 185 pairs.

The authoritative definition of each check lives on its reference page: discoverability, content, bot access, capabilities and commerce.

Across the whole sample, about 200 individual check verdicts changed. Among the pass and fail movements, 77 went from fail to pass and 33 from pass to fail, so the first scan of a pair tends to be the worse one by more than two to one. We come back to that, because it has two competing explanations and the data cannot choose between them.

The volatile five

Five checks sit above 5% in the ten-minute table: E5, E3, C3, D1 and D3. Each one asks the target for something and decides on the answer. The flip rate is a measure of how much that answer varies.

D1: the 429 problem

D1 is the robots.txt check, and it has the one cause in this study we can state outright. The evidence string for a failing D1 is usually just HTTP plus a status code. Across the 303 pairs D1 changed status 18 times, and in 12 of those the code was 429 Too Many Requests. In total, 23 of the 303 pairs, 7.6%, had a 429 on at least one side, which includes 11 pairs where both scans were refused and nothing flipped.

The usual pattern is a robots.txt that answers 200 with a few hundred bytes on one scan and 429 on the next, or the reverse. Three more D1 flips involved a 403, a 404 or no response, and the remaining three were changes in the details of a pass or partial result. RFC 6585 defines 429 as the response for a client that has sent too many requests in a given time, and it says the server may include a Retry-After header. Two scans in quick succession from the same address can look like that kind of burst to a small site's rate limiter.

What the check does with the result is the part that is ours to fix. A 429 is not a statement about whether a site publishes a robots.txt. It means the server refused to answer. Today D1 records the status code and scores it as a failure, and every check that depends on the robots.txt body, B2 among them, inherits the problem. The soft 404 post argued that HTTP 200 alone is a weak signal of presence. The mirror image is true here: a 4xx alone is a weak signal of absence.

C3: the page that changed shape

C3 checks for one h1, a title, a meta description and JSON-LD. Its evidence string lists each signal as true or false, which makes its flips unusually readable. The commonest flip, five times, went from title: false, meta description: false, JSON-LD: false to all three true. The page the scanner received on the first request carried no head metadata at all, and the second request returned a complete one.

That pattern is what a bot-mitigation interstitial or a half-rendered shell looks like, and our empty-DOM study found 8.4% of a 663-host corpus failing C3 outright. The new number says some of those failures are not stable properties of the site. They are properties of one request. A smaller group of C3 flips changed only the JSON-LD signal, from true to false or the reverse, which is the signature of a page that injects schema with a script that does not always finish before the scanner reads the HTML.

E3: the word count that moves

E3 measures server-rendered text and heading structure. It is the second most volatile check, with 20 flips, and its evidence strings show why. Both sides of a flip look like N words of server-rendered text, N h2, N h3. The structure of the evidence is identical and only the numbers differ. The same site returned a different amount of text on two consecutive requests.

Threshold checks amplify small differences in a way presence checks do not. A page that is 40 words above the E3 pass line on one request and 40 below it on the next flips from pass to fail without anything dramatic having happened. The E3 flips run in both directions at about the same rate, with no lean toward either improvement or decline.

E5: an applicability decision, not just a verdict

E5 looks for an RSS, Atom or JSON feed, and it is the most volatile check in the rubric at 6.9%. It also has the odd property that it can answer na, meaning no periodical content section was detected. Of its 21 flips, 9 crossed between na and a scored verdict, including fail to na and na to pass.

That matters because of what applicability does to a score. A check that answers na drops out of the denominator. A site whose second scan finds a blog section the first scan missed picks up a new check that it can pass or fail. In the cases we saw, the feed evidence on one side named an RSS feed declared in the page head, and on the other side said no feed was found at the conventional paths. The page the scanner saw on one request had the link rel="alternate" tag and the page on the other request did not. We are reading evidence strings and cannot see the target's side, but that is what the strings say.

D3: the header that is there and then is not

D3 looks for discovery relations in a Link: response header or in <link> tags in the HTML head, and its flips are the cleanest of the five to describe. In 14 of 16, one scan reports no Link header or <link> tag and the other reports a real header or head tag, including rel="api-catalog", rel="describedby" for llms.txt, and in one case an ard.json relation. In ten of those 14 the second scan found the relations the first one missed, and in four the order was reversed.

A response header can differ between requests for ordinary reasons: a CDN edge that adds headers only to some responses, a cache that holds an older copy, or an origin that adds the header on a cold path. The Link-header how-to assumes a site sets it consistently. The data says that for some sites it is not set consistently, which is a finding the owner would want to know about and the single score hides.

B2 and C1: flips that are partly real and partly ours

Two of the middle-volatility checks need their own caveat, because their flips do not all look like noise.

B2 has 13 flips and 11 of them went from fail to pass. The evidence is the same sentence on both sides, N known AI agents addressed in robots.txt, with a different count. The simplest reading is that the robots.txt changed between the scans, and the earlier post on which fix moves your score most names B2 as the cheapest first fix, at about 15 minutes of work. An owner who scans, adds the rules and scans again would produce exactly this pattern. We cannot tell that apart from a target that served different robots.txt bodies to two requests, so we do not count B2's flips as scanner variance. This is also the best candidate explanation for the 77 to 33 skew toward improvement.

C1 has 13 flips and its evidence shows the mechanism most clearly. C1 tries more than one route for a Markdown version of the page. In at least five of the thirteen, the first scan reported only the Accept: text/markdown probe, which returned HTML, and failed. The second scan reported a pass on /index.md or on the header. Our documentation-site study found that no site in its population answered all four Markdown routes, so a check that tries routes in a variable order or stops early on a timeout would produce just this. We have not pinned down which. It is the item on our list we understand least.

C2, the llms.txt check, belongs in the same family as D1. Of its 10 flips, 7 went from HTTP plus a code to a clean 200 with the expected headings, sections and links, and 1 went from no response to a pass. The llms.txt 404 study covers what a valid file contains. This one is about whether the scanner reached it.

Why the stable checks look stable

At the other end of the table, M1 and M3 never flipped in 128 pairs and B3 flipped twice in 303. Two things sit behind these numbers, and neither is "the check is excellent".

First, many of these checks answer info on most sites, which means a bonus check with weight zero. A check that returns the same non-scoring verdict on nearly every scan will look perfectly stable, and it is. Every B1, B3, P2 and P3 flip involved info: 4 of 4 for B1, 2 of 2 for B3 and 4 of 4 for both P2 and P3. Their stability is a property of their rarity. Across the whole table, 23 flips were info to pass or pass to info, which is the scanner deciding on one request that a bonus check applies and on the next that it does not.

Second, stable does not mean right. M1 is a check where an earlier AIScan audit of our own rubric found that it decides on a status code alone; the commerce protocols study covers the protocols involved. A check that decides on a status code alone will give the same answer every time the status code is the same. Determinism and correctness are different properties, and a rubric can be perfectly repeatable and wrong. This post measures only the first one.

Evidence strings are a third trap. E1 differed in its evidence in 286 of 303 pairs while its verdict flipped in only 8. The reason is by design: the evidence quotes the randomly named probe path the check requested, for example a GET to a made-up address ending in a random token. A tool that diffed evidence strings to detect instability would call E1 the least stable check in the rubric, when it is among the better ones. We compared verdicts.

Does time make it worse?

If the flips came mostly from owners changing their sites, the rate should climb as the gap between scans grows, because a day gives an owner far more room to edit than ten minutes. The table suggests the opposite for most checks. D1 is 5.9% at ten minutes and 2.2% at one to 24 hours. E5 is 6.9% and 3.7%. E4 is 3.0% and 1.1%. Only a few checks rise: C1 from 4.3% to 5.1%, P1 from 3.6% to 4.8%, E2 from 2.0% to 3.9%, and E3 stays high at 5.3%.

For the checks that fall, noise is the better explanation. A rate limiter that bites on a rapid repeat does not bite a day later, so D1 and C2 improve with spacing exactly as the 429 finding predicts. For the checks that rise, a real change on the site is more plausible, and those are the checks where an owner is likely to be iterating: Markdown routes, API descriptions and discovery relations. The two groups behave differently, which is evidence that the data contains both effects.

One more note on timing. The 10-minute bucket is also where users plausibly re-scan right after changing something, and that makes our ten-minute flip rates an upper bound on scanner-only variance. The 30-second pairs, with a 12.1% score-difference rate, are the closest thing we have to a lower bound, and 12% is still not small.

What AIScan cannot see here

• We cannot see the target's side. Every explanation above is inferred from the evidence string our scanner stored. We cannot see whether a CDN returned a challenge page, whether a rate limiter fired or whether a cache served stale HTML. A 429 in the evidence tells us the server said no, not why.
• We cannot separate a changed site from a moody one. Our data has no flag for "the owner edited something". The 30-second pairs and the falling rates over time narrow the gap, and B2 shows where it stays open.
• Self-selected repeat scans. People who scan twice in ten minutes are not a random sample of sites. A host that errors on the first scan is more likely to be rescanned, which can overstate flakiness and may explain the lopsided direction of the changes.
• One scanner location. All scans come from one infrastructure, so a rate limiter that likes our address less than someone else's would show up in every row.
• Not a measure of correctness. A check can return the same wrong answer every time and score 0% here.
• One rubric version. The numbers apply to 2026.08.2 only. Earlier versions are excluded, and a rubric change resets all of this.

What to do with this

You can use a score in three ways, and each needs a different response.

For a one-off audit, scan twice and read both reports. A check that flips between them is information. If D1 fails once with HTTP 429, your robots.txt is behind a rate limiter, and a crawler that hits the limit sees the same thing our scanner did. If C3 loses its metadata on one request, find out whether your bot protection, your edge cache or your JavaScript is serving the stripped version. Run npx aiscan-cli yoursite.com or paste the URL at aiscan.site, which is free and needs no account. The checks to compare are D1, C3, E3, D3 and E5, since those five flipped most, and the report gives the evidence string for each. The terminal audit guide explains how to read them.

If you would rather check by hand, request the same URL several times in a row and look at the status code and headers:

for i in 1 2 3 4 5; do
  curl -s -o /dev/null -D - https://yoursite.com/robots.txt | head -1
done

A 429 on any of the five repeats is the D1 problem reproduced from your own terminal. For D3, curl -sI https://yoursite.com/ | grep -i '^link:' run several times shows whether the header is set every time.

For a CI gate, do not gate on an exact score. In the data, 22.4% of pairs moved at least one point and 7.3% moved 15 or more. A pipeline that fails on any drop will fail on noise. Gate on a floor that leaves a few points of margin, retry once on failure before you block a deploy, and log the check IDs that changed so you can tell a regression from a repeat of the same flip. The CI gate guide has the workflow file.

For fixing what flips, most of what flips is within your control, and the fixes are the unglamorous ones. Allow-list your own monitoring or accept that crawlers face the same limits. Serve robots.txt, llms.txt and sitemap.xml from a path your rate limiter exempts, or from static storage, so they answer with 200 every time. On WordPress, ThinkRank manages robots.txt, robots meta, schema, sitemaps and llms.txt from one plugin, which removes the usual fight between plugins over one robots.txt. It migrates from Rank Math, Yoast, All in One SEO and SEOPress, each of which covers part of that file set, so the real question is whether you want one plugin or several writing to the same files. On Shopify, StoreSEO generates llms.txt and schema for stores. The llms.txt generator produces a file you can host anywhere that answers every time.

What we are changing in AIScan

A determinism post that blames the targets and not the instrument would be dishonest, so here is the list that comes out of the data.

1. D1 and C2 should not score a 429 as an answer. A rate-limit response should trigger a short wait, honour Retry-After when present, retry once, and report rate limited as its own state when the retry fails too. Twelve of D1's 18 flips would likely have resolved.
2. Scans should record the retry. If a second request was needed, the evidence string should say so, so that a score built on a retry is visible as one.
3. C1 should probe every route it names, in a fixed order, and say which it skipped. Today the evidence names the routes it reached and not the ones it did not.
4. Applicability should be shown separately from the verdict. A site that flips between na and a scored E5 verdict changes its score denominator, and the report should say so.
5. A repeat scan inside ten minutes should show the previous result. If a user scans twice and sees a different grade, the second report should name which checks differ.

We will report how many of these flips disappear after the retry change, using the same query, in a follow-up. The SQL above is the measurement. If the rate for D1 does not drop, the diagnosis was wrong.

Run it on your own site

Scan the site once, then again, and compare the check lists. The scan covers D1, C3, E3, D3 and E5, the five checks that flipped most, and the evidence line for each shows what the scanner saw on that request. Use npx aiscan-cli yoursite.com or paste the URL at aiscan.site. If two scans disagree on a check, work on that check first, because an inconsistent answer is the easiest kind of problem for a crawler to hit and the hardest for an owner to notice.

Every check named here has a reference page: discoverability checks for D1 and D3, content checks for C1, C2, C3 and E3, bot access checks for B2, and capabilities checks for the P-series. The full set of how-to guides is at /guides.