Audit Your Own Agent Readiness From a Terminal in Ten Minutes

Paste your URL at aiscan.site or run npx aiscan-cli yoursite.com and you get a score in under thirty seconds. Fine for a first read, not enough if you actually want to know what that number is testing, or whether you can trust it where it disagrees with what you expected. Every check behind that score started as a manual probe against a real population of sites, and every one of those probes is a single curl call you can run yourself. This guide hands you ten of them, in the order that makes them mean something, each traced back to the study that measured it at scale.

Quick summary

Probe 1: does your site actually 404?

curl -s -o /dev/null -w '%{http_code} %{size_download}\n' https://yoursite.com/this-page-does-not-exist-zz9
curl -s -o /dev/null -w '%{http_code} %{size_download}\n' https://yoursite.com/this-page-does-not-exist-zz9.md

Everything below this line is meaningless if either line comes back 200. About a quarter of real sites do exactly that: the server answers with its own homepage, app shell or a generic "not found" page dressed up as a 200, and every check downstream reads it as content. We measured it at corpus scale (733 sites, 25 September 2026): 75.3% cleanly 404, 24.7% don't, and running the same test with a .md suffix instead of the plain path catches hosts that fail one form and not the other, since some frameworks hard-404 a normal path and soft-200 a Markdown one, a distinct bug the first line alone would miss. Two live examples make the failure concrete rather than abstract: one platform's invented-URL page grew from 2,518,049 bytes to 2,638,026 over two weeks, and another's grew from 155,640 to 562,981 over the same window, both fully rendered HTML shells, both served at 200, neither host having noticed. Verified on 28 September 2026, aiscan.site itself still returns a clean, small 404 on both forms of this test, which is the baseline every check below assumes. Run the two-request control (the real path against a deliberately invented one) before trusting any check that asserts a file or page exists, because a 200 alone never tells you whether the body is the thing you asked for or the app's default catch-all. The full population, both live examples and the framework-level explanation are in Your site returns 200 for pages that do not exist, which is also where /docs/checks/discoverability (check E1) is explained end to end.

Probe 2: what does a crawler get that a browser doesn't?

curl -s -o /dev/null -w '%{http_code}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/140.0 Safari/537.36' https://yoursite.com/
curl -s -o /dev/null -w '%{http_code}\n' -A 'ClaudeBot/1.0' https://yoursite.com/

A single differential proves nothing on its own; it could be a rate limit, an A/B test or a CDN hiccup rather than a policy. The reliable version fetches the same URL as a browser control, then as an operator's training identity and its user-driven identity separately, because those two tokens are answered differently on a meaningful share of hosts, and re-checks any difference it finds two more times, spaced apart, before treating it as real. On 75 hosts checked this way, roughly a third that serve a browser cleanly refuse at least one AI identity outright, and a further slice answer selectively rather than uniformly: some hosts let the training identity through and block the user-driven one, the opposite of what most publishers think they've configured when they write one blanket rule for a vendor's name. Reading only the status code isn't enough either, since two identities can both come back 200 while the actual page content differs by user agent, so a serious version of this probe also diffs the presence of <title> and the meta description between the two fetches, not only the response code. The matched-pair method, the reproduction protocol and the full breakdown by identity are in Stealth Crawling vs User-Driven Fetching in 2026.

Probe 3: does your robots.txt actually stop anything?

curl -s https://yoursite.com/robots.txt | grep -A2 -i 'user-agent'
curl -s -o /dev/null -w '%{http_code}\n' -A 'GPTBot/1.0' https://yoursite.com/some-real-page

This is the gap in the audit that surprises people most: a declared rule is not the same fact as an enforced one. The same 75-host sweep behind probe 2 also resolved each host's effective robots.txt policy per identity and then checked whether the server's own behavior agreed with it, and a full Disallow: / line was actually backed by a matching refusal on only 45%-67% of the hosts that published one. A rule that exists only in a text file a crawler is choosing to honor is a request, not a control, and the same post above shows the arithmetic on both the compliant and the non-compliant share. It's also worth checking your file's semantics, not just its presence: according to RFC 9309's own grouping rules, the record with the most specific matching path wins, and a robots.txt with rules ordered on the assumption that the first match wins rather than the most specific one can declare exactly the opposite policy from the one its author intended, a parsing mistake that never shows up unless you read the spec's own logic against your file line by line.

Probe 4: can your server say "nothing changed"?

curl -sD - -o /dev/null https://yoursite.com/ | grep -iE 'etag|last-modified'
curl -s -o /dev/null -w '%{http_code}\n' -H 'If-None-Match: "test"' https://yoursite.com/

If the first line prints nothing, the second one can never return 304: there's no validator for a client to send back. That matters more for agent traffic than for a browser, because an agent that re-checks a page on every session has no way to confirm nothing moved, so it re-downloads the full payload every time. Across 240 round-trip probes on 60 hosts, 68.2% shipped a validator and 51.8% of those answered 304 when it was echoed back, meaning the honest floor, counting hosts that never emit one at all, is well under half. The split isn't random noise either: a follow-up pass on the same hosts found roughly a fifth answering 304 on every surface they served and a similar share answering on none, which tracks how the site is built far more than anything the operator configured deliberately. We dogfooded this ourselves while writing it up. Verified on 28 September 2026, aiscan.site ships neither ETag nor Last-Modified on any route we checked, including this one, so run the same two lines against us and you'll see the same gap we're reporting, on the house. The full funnel (emits a validator, honors it, actually returns 304) and the byte savings a working conditional request produces are in AI Crawlers Are Wasting Your Bandwidth, and Here's the Proof.

Probe 5: are your discovery files actually there, and actually valid?

for f in robots.txt sitemap.xml llms.txt feed.xml; do
  curl -s -o /dev/null -w "$f -> %{http_code}\n" "https://yoursite.com/$f"
done

A 200 on all four is the start of the check, not the end of it. llms.txt is the one worth reading rather than just fetching: of 50 URLs we probed for it, 39 returned a file and 38 of the 39 broke the v2 llms.txt spec somewhere, usually a missing blockquote summary or headings out of the order the spec defines. Run curl -s https://yoursite.com/llms.txt and read it against the spec at llms.txt Validator: The Mistakes Almost Everyone Makes, or generate one that matches it with our free llms.txt generator. On WordPress specifically, this whole probe tends to fail for a boring reason: robots.txt, robots meta and llms.txt end up split across two or three plugins that were never told about each other, so a rule one of them writes gets silently undone by another on the next save. ThinkRank manages all three from one plugin for exactly that reason, and migrates existing settings in from Rank Math, Yoast, All in One SEO or SEOPress rather than asking you to re-enter them. If your site serves content under a path prefix rather than at the root (a docs subdomain, a /blog mount), check the prefix too: a file that exists at /docs/llms.txt and nowhere else still reads as "missing" to a check that only probes the origin root, exactly the gap llms.txt vs robots.txt vs Sitemap: What Each One Actually Controls works through surface by surface, mapped against /docs/checks/discoverability (D1, D2) and /docs/checks/content (C2). The feed is the one people forget to actually open: a static-site generator on a default configuration often serves it at a path a generic feed check doesn't expect, /index.xml rather than /feed.xml on one popular generator and /blog/rss.xml on another, so a 404 on the exact filename above is worth one more try at your generator's own default before you conclude the feed doesn't exist at all.

Probe 6: does a Markdown twin of the page exist?

curl -s -H 'Accept: text/markdown' -o /dev/null -w 'header:  %{http_code} %{content_type}\n' https://yoursite.com/some-page
curl -s -o /dev/null -w '.md:     %{http_code} %{content_type}\n' https://yoursite.com/some-page.md
curl -s -o /dev/null -w 'index:   %{http_code} %{content_type}\n' https://yoursite.com/some-page/index.md

Run all three and expect them to disagree. Across 52 real documentation sites, the header route worked on 29, the .md suffix on 27, /index.md on 12, and not one site answered all three the same way a fourth convention would ask. Two rules make this probe worth the extra two commands instead of one. Check the content type, not just the status code, because a meaningful share of "passes" turn out to be a normal HTML page returned with a 200 under the Markdown route, so the check is satisfied and the agent still gets a wall of tags. And treat a 200 on a path that never existed as its own failure mode, which probe 1 already primed you to expect. The full 52-site breakdown, the platform-by-platform pattern (Starlight and GitBook serve it consistently, hand-rolled documentation stacks mostly don't) and the two-request control that tells a real Markdown twin from a soft 200 wearing the right content type are in Docs Sites and AI Agents: Four Markdown Routes, 52 Sites Tested, which maps to /docs/checks/content (C1).

Probe 7: is there a live agent endpoint behind the well-known path?

curl -s -X POST https://yoursite.com/api/mcp -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/.well-known/agent-skills/index.json
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/.well-known/skills/index.json

Two separate gaps live in this one probe. First, a scanner that only checks the well-known server-card path can miss a real, answering endpoint entirely: on one populated storefront sample, /.well-known/mcp/server-card.json returned real JSON on 12 of 60 hosts while a direct POST to the platform's own MCP route answered on far more, meaning the card and the endpoint are two different questions and a scanner asking only one of them under-reports adoption. Second, the Agent Skills convention itself has two published locations, a newer one and a legacy one, and only 4 of 11 publishing sites serve both, so a check (or a reader) probing a single path finds roughly half of what's actually out there. Read the body fetched from the tools/list response too, not just the status: a live answer isn't necessarily a stable one, since the same endpoint on the same store has answered with five tools one day and one tool four days later with no code change in between, which is why a single successful probe reads as "an endpoint exists today," not "here is its permanent surface." Both gaps, the exact hosts and the tool-list responses are worked through in Do You Need WebMCP Yet?, which sits under /docs/checks/capabilities (P2, P3).

Probe 8: does your platform already declare commerce capabilities?

curl -s https://yourstore.com/.well-known/ucp | python3 -m json.tool

Read the key, not the status code. A 200 here can be a valid Universal Commerce Protocol profile, or it can be a Next.js error page that also happens to answer 200; we found a live storefront doing exactly that on three unrelated well-known paths at once, all serving the identical error shell under a status line that told every automated checker the file existed. Piping the response through a JSON parser and asserting the specific key is the entire fix, and it costs one extra line wherever you've already written the curl. If the body parses and carries ucp.capabilities, your commerce platform has already published what an agent can query or transact without you configuring anything: on a 30-storefront sample, 21 served a valid profile and 19 separately answered a working POST /api/mcp, though the exact tool set offered is not stable per store over time and needs at least two timestamped probes to report honestly. This is also the one place in the audit where the platform, not the merchant, owns the gap. Shopify has already turned this on for a meaningful share of stores, and the merchant-facing half most stores are missing, a validated llms.txt, an agents.md file and product schema an agent can actually parse, is exactly what StoreSEO builds for Shopify specifically, because Shopify's own rollout stops at the protocol layer and does not touch it. The full protocol map (four layers, one of them deployed at scale, the other three mostly draft) is in Agentic Commerce Protocols, Explained, under /docs/checks/commerce (M1).

Probe 9: what are your own pages telling Google?

curl -s https://yoursite.com/some-real-article | grep -oiE '<meta[^>]*name="(robots|googlebot)"[^>]*>'
curl -s https://yoursite.com/some-real-article | grep -c 'application/ld+json'

This probe has nothing to do with AI crawlers fetching your site directly; it's about what your pages say to the crawler that feeds Google's AI Overviews and AI Mode, a different lever from robots.txt entirely. According to Google's own robots-meta-tag documentation, nosnippet is named as a control that reaches specifically that surface, not just the classic search snippet, the one publisher-side lever Google documents by name for it, worked through in full in the sibling post linked below. Across 63 real article pages we checked, zero carried nosnippet, ten carried max-snippet (all set to -1, which changes nothing about AI Overviews eligibility), and the lever publishers pull instead, blocking Google-Extended in robots.txt, governs training data, not what shows up in an AI Overview today. The same sample shipped application/ld+json on 85.7% of pages, so the second grep above will come back non-zero for almost anyone reading this, which makes the near-universal absence of the one tag that actually does something for this specific surface the more interesting finding, not the schema count. The full breakdown of which lever does what, sourced to Google's own pages, is in Google AI Overviews: What You Can Actually Control.

Probe 10: where do you actually land?

npx aiscan-cli yoursite.com

That's the whole audit in one line, scored against every check named above plus the ones this guide didn't have room for: the full /docs/checks/bot-access dimension (B1-B3), the rest of discoverability and content, and the rest of commerce. Read the score with one caveat in mind: our own platform detector still returns unknown for a large minority of sites it scans, and three real, common platforms (Webflow, Docusaurus and Hugo) currently have zero rows in our corpus despite being live in the wild, a gap in our own coverage rather than a fact about those platforms. If your site runs on one of those three, read the per-check results rather than leaning on any platform-level comparison. Run a second opinion from Cloudflare's isitagentready.com scanner alongside your own score, because the two rubrics measure genuinely different things rather than agreeing or disagreeing on the same ten checks:

Neither number invalidates the other. A site can legitimately score well on one and middling on the other because they're not grading the same rubric. That comparison, the full check-by-check table and two AIScan bugs we found and fixed while building it are in AIScan vs. Lighthouse's Agentic-Browsing Audits.

Bank the work: gate it in CI

A one-time audit tells you where you stand today. The version worth keeping is the one that fails a pull request when a regression ships: a redirect that breaks llms.txt, a rewrite rule that starts soft-404ing, a caching change that drops your sitemap.xml. None of the ten probes above need anything more exotic than curl and an exit code to become a CI gate: assert the status and the size on the real path, assert the same on the deliberately invented one from probe 1, and fail the job the moment they stop disagreeing the way they should. The same soft-404 control-path check from probe 1, wired into a GitHub Actions job with byte-count and status comparisons instead of eyeballing curl output, is the working example in Gate AI Readiness in CI So a Regression Fails the Build; copy the workflow file, point it at your own domain, and the next silent regression fails a build instead of shipping quietly and waiting for someone to notice a scan score dropped.

What this audit doesn't tell you

None of the ten probes above are a substitute for reading the rubric honestly, including where it's incomplete.

Grouping the evidence strings our own rubric records behind the scenes shows the identical HTTP 200 string producing both a partial and a pass verdict on more than one check, which means two sites can read as different grades while the only thing the rubric actually looked at, on both of them, was a status code. None of that is a reason to skip the automated score; it's a reason to also run the commands above at least once, because the two together see more than either alone. Probes 2 and 4 aren't gaps we're leaving alone either. A crawler-identity parity check and a conditional-request check are both on our own backlog, precisely because a single extra request per scan is enough to grade what a manual probe like this currently has to do by hand. Until they ship, this guide is the only way to see either signal, from us or from anyone else's scanner.

Run it now

Start with probe 1: if it fails, fix that before anything else means anything, then work down the list against your own domain. None of it needs an account, a paid tool or anything beyond curl and ten minutes, which is exactly the point: the score is a shortcut to this same information, not a gate in front of it. When you're done, run npx aiscan-cli yoursite.com or paste the URL at aiscan.site for the scored version, and check /guides for the fix-it walkthroughs behind whichever checks came back short.