Monitor AI Readiness Over Time and Catch Silent Regressions

Discovery files rot without anyone touching them. A deploy overwrites robots.txt, a plugin update regenerates llms.txt, a CDN rule starts answering every unknown path with 200. Nothing errors, the pages still load, and the first sign is a score that fell weeks ago. This guide sets up a watch that catches the change within a day: a scheduled AIScan re-scan, plus a 40-line script that asserts what a status code cannot.

Quick summary

Why readiness regresses without an error

A discovery file is a promise made to a stranger. Crawlers and agents read robots.txt, a sitemap and llms.txt at conventional paths, and they never tell you when the answer changes. The site owner sees only the pages, so a missing file looks the same as a working one from the dashboard.

Think of a smoke alarm with the battery removed. The ceiling looks fine, the room is quiet, and quiet is what you would see in a fire's first minute too. A file monitor is the battery test: it makes a silent failure audible.

There are three ways the file goes wrong, and each needs a different assertion.

The second row is the dangerous one. Our CI gate guide measured a live Replit app where /llms.txt and an invented path both returned 200, text/html, 5,712 bytes, with the same MD5. A monitor that asserts only "status is 200" reports that site as healthy forever. The soft 404 explainer covers why hosts behave this way.

Path one: let AIScan watch the site

This is the recommended route because it needs no server and no code. AIScan re-scans a tracked domain on a schedule, saves each result to your history, and emails you when the grade drops.

According to the AIScan monitored sites docs, fetched on 9 October 2026, the cadence is weekly by default with a daily option, and alerts cover grade changes and noticeable score drops. The pricing page, fetched the same day, lists the plan differences:

The pricing page also states that card checkout is not live yet and that plans are activated manually on request. Check it before you plan around Pro.

1. Run a scan of your site at aiscan.site while signed in to a free account.
2. Open the report and click Track this site at the top.
3. Open your profile and find the site under Monitored sites.
4. Set the cadence there. Weekly catches a regression within seven days, daily within one.
5. Open the alert preferences linked in any alert email and choose digest or per-change mail.
6. Write down today's score as your baseline. An alert only means something against a known-good number.

AIScan can also track from an assistant. According to the MCP server docs, fetched on 9 October 2026, the server has a track_site tool and a get_domain_history tool that returns the score trend with a note when a rubric change, not your site, moved the number. That note matters: a score can shift because the rubric version changed, and the history says so.

Path two: a cron job and a snapshot, if you would rather run it yourself

The script below fetches three files, plus a path that cannot exist, and compares them with the previous run. It uses only the Python standard library.

#!/usr/bin/env python3
import hashlib, json, os, random, sys, urllib.request, urllib.error

site = sys.argv[1].rstrip("/")
snap = sys.argv[2] if len(sys.argv) > 2 else "snapshot.json"
PATHS = ["/robots.txt", "/sitemap.xml", "/llms.txt"]
UA = {"User-Agent": "Mozilla/5.0 (compatible; readiness-watch/1.0)"}

class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, *a, **k):
        return None

opener = urllib.request.build_opener(NoRedirect)

def get(path):
    try:
        r = opener.open(urllib.request.Request(site + path, headers=UA), timeout=20)
    except urllib.error.HTTPError as e:
        r = e
    except Exception as e:
        return {"status": 0, "type": str(e)[:60], "sha": ""}
    body = r.read()
    return {"status": r.code,
            "type": r.headers.get("content-type", "").split(";")[0],
            "sha": hashlib.sha256(body).hexdigest()[:12]}

ctrl = get("/no-such-file-%d.txt" % random.randint(10**6, 10**7))
problems = []
if ctrl["status"] != 404:
    problems.append("control path returned %s: every 200 below is untrustworthy" % ctrl["status"])

now = {p: get(p) for p in PATHS}
old = json.load(open(snap)) if os.path.exists(snap) else {}
for p, cur in now.items():
    if cur["status"] == 200 and cur["sha"] == ctrl["sha"]:
        problems.append("%s is byte-identical to the control path (soft 200)" % p)
    prev = old.get(p)
    if prev and prev["status"] == 200 and cur["status"] != 200:
        problems.append("%s went from 200 to %s" % (p, cur["status"]))
    elif prev and prev["type"] != cur["type"]:
        problems.append("%s content-type %s -> %s" % (p, prev["type"], cur["type"]))
json.dump(now, open(snap, "w"), indent=1)
for line in problems:
    print("FLIP:", line)
print("control", ctrl["status"], "| checked", len(now), "files |", len(problems), "problem(s)")
sys.exit(1 if problems else 0)

Save it as watch.py, then schedule it:

1. Save the script on any machine that stays on, such as a small VPS or a home server.
2. Run python3 watch.py https://example.com /var/lib/watch/example.json once. The first run writes the baseline and should print 0 problem(s).
3. Add a cron line: 15 6 * * * python3 /opt/watch.py https://example.com /var/lib/watch/example.json || your-alert-command.
4. Replace your-alert-command with whatever reaches you: mail, a Slack webhook with curl, or a push service.
5. Pick an odd minute, not :00. GitHub documents that scheduled workflows can be delayed at the start of every hour, and the same crowding applies to any shared cron service.

We ran it on 9 October 2026 against aiscan.site twice: both runs printed control 404 | checked 3 files | 0 problem(s) and exited 0. Then I tested the two failures on a local server. Deleting llms.txt printed FLIP: /llms.txt went from 200 to 404 and exited 1. A server that answered everything with 200 and an HTML shell printed four problems, one for the control path and one for each file, and exited 1.

If you would rather use GitHub Actions

A scheduled workflow can run the AIScan CLI instead of the script. The command from the CLI docs is npx aiscan-cli https://example.com --min-score 85 --fail-on essential. Exit code 0 means the gate passed, 1 means a gate failed, 2 is bad usage and 3 is a network or API error, so a failed run is a regression or an outage and never a silent pass. According to GitHub's workflow events documentation, fetched on 9 October 2026, the shortest schedule interval is five minutes, and in a public repository scheduled workflows are disabled automatically after 60 days without repository activity. A watch that quietly turns itself off is the failure you were trying to prevent, so use a private repository or a server cron for anything you rely on.

Which path fits

You can run two of these together. The CI gate stops a bad deploy, and the scheduled scan catches the changes no deploy caused, such as a plugin auto-update or a CDN setting changed in a dashboard.

Where the regressions come from

On WordPress, the first row has a known shape: several plugins each want to own robots.txt. ThinkRank manages robots.txt, robots meta, schema, sitemaps and llms.txt from one plugin, so one update changes one file owner instead of three, and it imports settings from Rank Math, Yoast, All in One SEO and SEOPress. Other SEO plugins handle some of these files well. The monitor is the same either way. On Shopify, StoreSEO generates llms.txt from live products, collections, pages and articles, so the file tracks the catalogue instead of going stale.

What the watch cannot see

AIScan and the script both fetch from one network location with one identity. They do not see a CDN that blocks GPTBot and serves you, a page that renders differently for a crawler, or anything behind a login. They grade the homepage on the public scan, and per-page scans are a Pro feature. Treat a green result as "the files are there and honest", not as proof that every crawler gets through. For the bot side, see the robots.txt starter.

When an alert fires

1. Open the latest report and read the row that changed, not the total.
2. Fetch the path yourself with curl -sI and compare it with a path that cannot exist.
3. Check what shipped in the last 24 hours: deploys, plugin updates, CDN changes.
4. Fix the source, not the symptom. If a plugin rewrote the file, change its setting, not the file on disk.
5. Re-scan with npx aiscan-cli yoursite.com --fresh to bypass the five-minute cache, and confirm the score returned to your baseline.

Next step

Scan your site at aiscan.site, then click Track this site to start the watch. The rows to read first are D1, D2, C2 and E1; the discoverability checks page explains each one. The rest of the AIScan guides cover the checks that sit around them.