AI Readiness Blog
Practical guides on making your website discoverable, parseable, and actionable for AI agents and LLM-powered search.
How to ship one h1, title, meta description and JSON-LD on Webflow
Webflow's SEO panel now writes three of the four signals that AIScan's C3 check grades, and the fourth one is not in any panel. On a paid Site plan you can set a page's title tag, its meta…
Publish an MCP Server Card So Agents Can Find Your Tools
On 10 September 2026 we probed every host running a remote MCP server listed in the official MCP Registry: 79 domains, three requests each. Seven of them publish a parseable server card at…
Docs Sites and AI Agents: Four Markdown Routes, 52 Sites Tested
Ask an AI assistant how to configure a webhook, add a database index, or set a cache header, and it does not go looking for a blog post. It goes to the vendor's documentation. Docs are the…
How to ship one h1, title, meta description and JSON-LD on Next.js
Next.js hands you a Metadata API that writes your <title and your meta description into the head automatically. It does not write your JSONLD, and it does not write your <h1. Those two are ordinary…
Gate AI readiness in CI so a regression fails the build
A green pipeline is supposed to mean the site is fine. On an AI readiness check it often means something narrower: that the gate could not tell a file from a phantom. This step, which appears in a…
Stealth Crawling vs User-Driven Fetching in 2026: What 75 Sites Actually Do
On 4 August 2025 Cloudflare removed Perplexity from its verified bot list and published the evidence. Perplexity's answer was that fetching a page because a person asked for it is not the same act as…
How to ship one h1, title, meta description and JSON-LD on WordPress
WordPress sites fail AIScan's C3 check more often than their owners expect, and almost never for the reason they expect. Across the 129 WordPress sites in our scan corpus, verified on 8 September…
The complete AI readiness setup for Wix in 2026
Every other platform in this series asks you to create something. Wix has already created it. Before you open a single panel, a Wix site is serving a robots.txt, a sitemap index, serverrendered HTML…
The Agentic Commerce Stack in 2026: UCP, ACP, AP2 and x402, Measured on 30 Storefronts
Five protocol names now compete for the same sentence in every agentic commerce explainer: UCP, ACP, AP2, x402 and MPP. Almost every explainer treats them as rivals. They are not. Four of the five…
How to declare allow/deny rules for named AI crawlers in robots.txt on Wix
Wix writes your robots.txt for you. Fetched on 7 September 2026, the file at www.wix.com/robots.txt ran to 90 nonempty lines: one Useragent: group, 87 Disallow rules, and a single Sitemap: line at…
The complete AI readiness setup for Docusaurus in 2026
Docusaurus is the framework most likely to be sitting between an AI agent and the answer it is looking for. It runs a very large share of the developer documentation on the web, and developer…
Google AI Overviews in 2026: What You Can Actually Control, Measured on 110 Publishers
Search Console will tell you that AI Overviews sent you traffic. It will not tell you how to turn them off, because there is no setting to turn off. Google publishes exactly four controls that reach…
How to declare allow/deny rules for named AI crawlers in robots.txt on Docusaurus
Docusaurus gives you somewhere to put a robots.txt in about ten seconds. Making sure a crawler ever reads it is the harder half, because the most common Docusaurus deployment shape files the rules at…
Cloudflare Pay Per Crawl in 2026: Should You Charge AI Crawlers? 119 Sites Measured
Stack Overflow will sell you its homepage for fifty cents. Point an AI crawler at it and the server answers HTTP 402 Payment Required, crawlerprice: USD 0.5, and a JSON body naming Cloudflare's Pay…
The complete AI readiness setup for Hugo in 2026
Hugo hands you every byte your site emits. No plan tier decides which files exist, no editor panel hides a toggle, no vendor writes a robots.txt on your behalf. That ownership is the whole appeal,…
How to publish a valid llms.txt on Wix
Every other guide in this series starts by asking you to create a file. Wix already wrote yours. Since Wix released llms.txt to all eligible sites, the file is generated and maintained for you from…
The complete AI readiness setup for Squarespace in 2026
Squarespace hands you a finished website and a finished discovery layer at the same time. The robots.txt is written for you, the sitemap is generated for you, and the RSS feed already exists and is…
AI Crawler Traffic in 2026: Only 51.8% of Sites Can Answer "Nothing Changed"
Verified 5 September 2026. All measurements in this article were taken on that date. Every AI crawler that visits your site asks the same handful of questions over and over. Where is your robots.txt.…
How to publish a valid llms.txt on Docusaurus
A Docusaurus site can publish a completely valid llms.txt and still score zero on it. The reason is one config value. On GitHub Pages, the platform's most common deployment target, a project site…
The complete AI readiness setup for Framer in 2026
A brandnew Framer site scores better on an agent readiness scan than a new site on almost any other hosted builder, and that is exactly what makes the remaining gaps hard to see. Framer prerenders…
llms.txt vs robots.txt vs sitemap.xml in 2026: Six Files, Six Jobs
Verified 4 September 2026. Every figure below was measured or fetched on that date. Three files keep getting compared as if they were competing for the same job: robots.txt, sitemap.xml and llms.txt.…
How to publish a valid llms.txt on Hugo
Hugo has no llms.txt setting, no plugin and no checkbox. What it has instead is an output format system that already builds robots.txt and sitemap.xml the same way, which means the file can be…
The State of AI Agent Readiness in 2026: 473 Sites Measured
Half of the web's agentreadiness problem is already solved, and almost nobody has noticed which half. Across 473 real websites scanned by AIScan between 24 August and 3 September 2026, the median…
How to publish a valid llms.txt on Framer
Verified on 2 September 2026. Every path, plan limit and status code below was either read from Framer's own help centre or measured live against www.framer.com on that date. On Framer, llms.txt is…
The complete AI readiness setup for Replit in 2026
Verified 2 September 2026. Every command, config key and file path below was run against live Replit apps or fetched from Replit's own documentation on that date. On most platforms a missing…
WebMCP in 2026: Do You Actually Need It Yet? 85 Sites Measured
Verified 2 September 2026. Every number below was measured on this date against live sites. The honest answer to "do I need WebMCP yet" turned out to depend on a question nobody asks first: is it…
How to publish a valid llms.txt on Replit
Verified 1 September 2026. On Replit, publishing /llms.txt is not one job. It is two, and which one you have depends on how the app is published. An app built by Agent runs on an Autoscale Deployment…
The complete AI readiness setup for Lovable in 2026
Two Lovable apps can look identical in a browser and differ by a factor of fifty in what an AI crawler actually receives. On 1 September 2026 we fetched fifteen published Lovable apps twice each,…
Is Your llms.txt Valid? 39 Real Files Tested, and the 11 Mistakes That Break Them
Verified 1 September 2026. Every figure here comes from a live fetch on that date. We fetched /llms.txt from 50 wellknown sites, kept the 39 that returned a real file, and ran every one against the…
How to publish a valid llms.txt on Squarespace
On Squarespace, llms.txt is not a file. There is nowhere to put one. The platform stores every upload on a CDN it controls, and its own documentation says you cannot move or redirect those URLs. What…
The complete AI readiness setup for Ghost in 2026
Verified 31 August 2026 against Ghost's own source and three live Ghost 6.61 publications. Ghost hands you more agent readiness for free than any platform in this series, then stops at a wall you…
Cloudflare Agent Readiness vs Chrome's Agentic Browsing Audit in 2026: 28 Checks, One Overlap
Verified 31 August 2026. Every score, status code and audit weight below was produced on this date by running both tools live. The Lighthouse figures come from Lighthouse 13.4.1 driving Chrome for…
How to publish a valid llms.txt on Ghost
Verified 30 August 2026. Ghost has no setting for llms.txt and no folder you can drop a file into that lands on the site root. The fix takes about fifteen minutes: one route in routes.yaml, one…
The complete AI readiness setup for Webflow in 2026
Verified 30 August 2026. A Webflow site passes the two hardest AI readiness checks before you touch anything, then loses four cheap points at the root path. This guide closes those four in about 20…
AI Readiness Scanner Pricing in 2026: Every Price, Verified and Dated
Verified 30 August 2026. Every price below was fetched from the vendor's own pricing page on this date. Where a vendor publishes no price, that is recorded as a finding rather than filled in with a…
How to publish a valid llms.txt on Webflow
Verified 29 August 2026. Webflow added a native llms.txt upload on 23 July 2025, so publishing one now takes about five minutes and no custom code. Go to Site settings SEO LLMs.txt, click Upload…
The complete AI readiness setup for Astro in 2026
Verified 29 August 2026 against Astro 7.2.9. Most platform guides open by telling you to fix your rendering. On Astro you can skip that. Astro ships static HTML by default, so the two checks that…
The 10 Best AI Agent Readiness Scanners in 2026, Tested on One URL
Verified 29 August 2026. Every score below was captured from the tool's own public endpoint on that date, and every disagreement was adjudicated by hand against the live site. A category of tool has…
How to publish a valid llms.txt on Astro
Verified 28 August 2026. A valid llms.txt on Astro is either a twominute file drop or a twentyline endpoint that rebuilds itself from your content collections every time you run astro build. AIScan's…
The complete AI readiness setup for Next.js in 2026
Next.js gives you five of the files AI crawlers look for, and four of them are one file away. The setup below takes about 40 minutes on an App Router project and moves a typical Next.js site from a…
Your Next.js Site Ranks on Google and Is Invisible to ChatGPT: How to Prove It
Googlebot runs your JavaScript. The crawlers that build ChatGPT's and Claude's indexes, as far as anyone has measured, do not. That gap is why a React or Next.js site can hold page one on Google and…
How to publish a valid llms.txt on Next.js
AIScan's C2 check is worth 6 points, and on Next.js you can earn it in about two minutes. C2 passes when /llms.txt returns 200 and the file contains an H1, at least one section, and markdown links.…
The complete AI readiness setup for Shopify in 2026
Shopify hands you more AI readiness for free than any other platform on this blog. Serverrendered HTML, a sitemap, an Atom feed, product JSONLD and a live /llms.txt all ship without you touching a…
Every AI Crawler User-Agent in 2026: The Verified List and How to Spot a Fake
Verified 27 August 2026. Every useragent below was read from the operator's own documentation on that date. Every IP range file was fetched and counted. Every reverseDNS result was tested live.…
How to publish a valid llms.txt on Shopify
If your store runs a Liquid theme, Shopify already serves /llms.txt for you. We checked four live storefronts on 26 August 2026 Allbirds, Brooklinen, Kotn and Red Bull Shop US and all four returned…
The complete AI readiness setup for WordPress in 2026
Updated August 2026. A default WordPress install already passes more AIreadiness checks than most handbuilt sites. It ships serverrendered HTML, a real XML sitemap and an RSS feed without you…
Does llms.txt Actually Work in 2026? Google Ignores It, Chrome Audits It, 97% Get Zero Requests
Updated 26 August 2026. In June, Google added a line to its own AI search documentation saying that Google Search ignores llms.txt files. Two months earlier, a different Google team shipped a…
Search, Agent, Training: What Cloudflare's Three AI Crawler Categories Actually Control
Updated 25 August 2026. Cloudflare now sorts crawler behaviour into three categories you can configure — Search, Agent, and Training — and the setting you pick for each decides whether Google indexes…
Cloudflare's AI Bot Block Is About to Catch Googlebot: What Changes on 15 September 2026
Updated 25 August 2026. On 15 September, Cloudflare changes how it enforces AI crawler rules. If you have ever ticked "Block AI bots" on a Cloudflare zone and left it on, Googlebot and Bingbot start…
Best AI Readiness Checkers in 2026 | AIScan
Your site can rank on page one of Google and still be invisible to ChatGPT, Claude, and Perplexity. The two systems read the web differently, and most sites have never been tested for the second one.…
Best AI Agent-Ready Tools in 2026 | AIScan
Search stopped being about ten blue links. ChatGPT now has roughly 900 million weekly active users according to OpenAI's February 2026 figureshttps://www.omnibound.ai/blog/aisearchstatistics,…