# AIScan.site — Full Reference

> Long-form companion to /llms.txt. Describes every public surface of AIScan.site so an AI agent can reason about the product end-to-end.

## What AIScan.site does

AIScan grades how prepared a website is for AI agents (ChatGPT, Claude, Perplexity, Gemini, and others). It detects the underlying platform (WordPress, Shopify, Lovable, Replit, Bolt, v0, Framer, Wix, Webflow, Ghost, Squarespace, HubSpot, Next.js) and runs a 5-dimension rubric — Discoverability, Content, Bot Access, Capabilities, Commerce — returning a 0–100 score, a letter grade, and per-check plain-English remediation.

## Core pages
- [Home](https://aiscan.site/): Scanner + community feed
- [How it works](https://aiscan.site/how-it-works): Methodology and rubric
- [Docs](https://aiscan.site/docs): Full documentation
- [Changelog](https://aiscan.site/changelog): Release history
- [Extension](https://aiscan.site/extension): Chrome extension
- [llms.txt Generator](https://aiscan.site/llms-txt-generator): Free tool

## Tools & APIs
- [Claude connector](https://claude.ai/directory/aiscan): Official Claude connector directory listing. Add AIScan in one click, sign in with an AIScan account, and run scans, retrieve fixes and manage monitoring without an API key.
- [MCP Server, OAuth](https://aiscan.site/mcp): Authenticated MCP connector (OAuth 2.1 + dynamic client registration). 13 tools: scan_website, get_fixes, get_grade, scan_page (Pro), whoami, list_my_scans, get_report, get_domain_history, list_tracked_sites, track_site, untrack_site, list_checks, get_check_guidance
- [MCP Endpoint, legacy](https://aiscan.site/api/mcp): Same tools with an API key bearer token; also blog CRUD
- [REST API](https://aiscan.site/api/public/scan): POST {url} or GET ?url= — full ScanResult JSON
- [Agent Skill](https://aiscan.site/aiscan-skill.json): Claude Agent Skills manifest
- [CLAUDE.md](https://aiscan.site/CLAUDE.md): Plain-English instructions for Claude Code
- [MCP server card](https://aiscan.site/.well-known/mcp/server-card.json): MCP discovery metadata
- [Sitemap](https://aiscan.site/sitemap.xml): Sitemap index of pages, docs, use cases and blog posts (scan reports are not listed)

## Blog

[Blog index](https://aiscan.site/blog) — all articles on AI readiness. Every post is also available as raw markdown at `/blog/<slug>.md`.

### Fix soft 404s on Astro, Docusaurus, Lovable, Next.js, Webflow and Wix
- URL: https://aiscan.site/blog/fix-soft-404s-for-ai-agents
- Markdown: https://aiscan.site/blog/fix-soft-404s-for-ai-agents.md
- Published 2026-10-27 · Category: AI Readiness · By M Asif Rahman

A soft 404 is a page that says "not found" while the status line says 200. On the six platforms below, the default behaviour ranges from a correct 404 with a deadend body Webflow, Wix, Docusaurus…

### Add ETag and Last-Modified so crawlers can send conditional requests
- URL: https://aiscan.site/blog/conditional-requests-etag-last-modified
- Markdown: https://aiscan.site/blog/conditional-requests-etag-last-modified.md
- Published 2026-10-27 · Category: AI Readiness · By M Asif Rahman

To make a page answer 304 Not Modified, send an ETag or a LastModified header on the first response, then answer a repeat request that carries IfNoneMatch or IfModifiedSince with status 304 and no…

### Fake AI Crawler Traffic: 63 Days of Our Own Server Logs
- URL: https://aiscan.site/blog/fake-ai-crawler-traffic-own-logs
- Markdown: https://aiscan.site/blog/fake-ai-crawler-traffic-own-logs.md
- Published 2026-10-27 · Category: AI Readiness · By M Asif Rahman

Of the 2,107 requests our own logger recorded under an AI crawler's name between 25 August and 27 October 2026, 61.6% came from 22 sources that each used 10 to 15 different crawler names, and not one…

### Why /.well-known/ files return 404 or 403 on your host, and how to fix it
- URL: https://aiscan.site/blog/well-known-folder-404-fix
- Markdown: https://aiscan.site/blog/well-known-folder-404-fix.md
- Published 2026-10-26 · Category: AI Readiness · By M Asif Rahman

Your /.wellknown/ file is returning 404 or 403 because something between your repository and the browser treats a folder that starts with a dot as private. In our tests on 26 October 2026, a default…

### llms.txt on Astro, Docusaurus, Framer, Ghost, Hugo, Next.js, Replit, Shopify, Squarespace, Webflow and Wix
- URL: https://aiscan.site/blog/llms-txt-setup-every-platform
- Markdown: https://aiscan.site/blog/llms-txt-setup-every-platform.md
- Published 2026-10-24 · Category: AI Readiness · By M Asif Rahman

Updated on 26 October 2026. Every platform section was checked on the dates shown inside it. Squarespace, Webflow and Wix joined this guide on 26 October 2026, so all eleven platforms that had their…

### Set Up Cloudflare AI Crawl Control: Allow, Block, Verify
- URL: https://aiscan.site/blog/cloudflare-ai-crawl-control-setup
- Markdown: https://aiscan.site/blog/cloudflare-ai-crawl-control-setup.md
- Published 2026-10-21 · Category: AI Readiness · By M Asif Rahman

If your site sits behind Cloudflare, you can allow or block each AI crawler from one table in the dashboard, and Cloudflare turns every block into a single WAF custom rule on your zone. The setting…

### AI readiness setup guide by platform (2026)
- URL: https://aiscan.site/blog/ai-readiness-setup-guide
- Markdown: https://aiscan.site/blog/ai-readiness-setup-guide.md
- Published 2026-10-17 · Category: AI Readiness · By M Asif Rahman

Verified 17, 18, 20 and 22 October 2026. Updated on 22 October 2026. An AIready site needs six things: a robots.txt that names AI crawlers D1, B2, a sitemap D2, a declared feed E5, an llms.txt C2,…

### How to publish schema.org Offer JSON-LD on your pricing page on Ghost
- URL: https://aiscan.site/blog/pricing-schema-markup-ghost
- Markdown: https://aiscan.site/blog/pricing-schema-markup-ghost.md
- Published 2026-10-16 · Category: AI Readiness · By M Asif Rahman

Ghost writes its own structured data for every page, and none of it is pricing. On Ghost 6.71 the home page carries WebSite and Organization, and a page carries Article, Person and Organization. No…

### Serve robots.txt with the right status and content type
- URL: https://aiscan.site/blog/robots-txt-status-content-type
- Markdown: https://aiscan.site/blog/robots-txt-status-content-type.md
- Published 2026-10-16 · Category: AI Readiness · By M Asif Rahman

Most robots.txt advice is about the rules inside the file. This guide is about the envelope: the HTTP status, the content type, the redirects and the size. A perfect set of rules does nothing if the…

### What a Failing AIScan Check Really Says: 2,961 Fail Lines Audited
- URL: https://aiscan.site/blog/evidence-string-audit
- Markdown: https://aiscan.site/blog/evidence-string-audit.md
- Published 2026-10-16 · Category: AI Readiness · By M Asif Rahman

A failing check on a scan report is supposed to be an instruction. The site owner reads the line, knows what the scanner asked for, knows what came back, and knows what to change. We audited whether…

### Rate-limit AI crawlers without blocking them
- URL: https://aiscan.site/blog/rate-limit-ai-crawlers-without-blocking
- Markdown: https://aiscan.site/blog/rate-limit-ai-crawlers-without-blocking.md
- Published 2026-10-15 · Category: AI Readiness · By M Asif Rahman

Most advice on rate limiting AI crawlers stops at "add a limit". The part that costs you is where the limit applies and what it answers. A limit that covers /robots.txt and /llms.txt makes your own…

### Which Agent-Readiness Checks Travel Together: 739 Sites, Three Bundles
- URL: https://aiscan.site/blog/check-co-pass-matrix
- Markdown: https://aiscan.site/blog/check-co-pass-matrix.md
- Published 2026-10-15 · Category: AI Readiness · By M Asif Rahman

Fix one agentreadiness check and a second one often moves with it. Fix another and nothing else changes. A scan report lists every check as its own line, so the reader cannot see which lines are…

### Return HTTP 402 with an x402 payload on a paid resource
- URL: https://aiscan.site/blog/return-402-x402-for-agents
- Markdown: https://aiscan.site/blog/return-402-x402-for-agents.md
- Published 2026-10-14 · Category: AI Readiness · By M Asif Rahman

Most sites should not return a 402. If you sell nothing per request, skip this guide and keep your score. If you do charge for an API response, a report or a dataset, this is how to return an x402…

### Platform League Table: WordPress, Next.js, Astro and Shopify Across 728 Sites
- URL: https://aiscan.site/blog/platform-league-table-agent-readiness
- Markdown: https://aiscan.site/blog/platform-league-table-agent-readiness.md
- Published 2026-10-14 · Category: AI Readiness · By M Asif Rahman

Every few weeks someone asks which platform produces the most agentready sites. WordPress, Next.js, Astro and Shopify all have fans, and a ranked table is the answer people expect. We have the data…

### 10 Best Instagram Profile Analyzers in 2026 (Tested)
- URL: https://aiscan.site/blog/best-instagram-profile-analyzers-2026
- Markdown: https://aiscan.site/blog/best-instagram-profile-analyzers-2026.md
- Published 2026-10-13 · Category: AI Readiness Reports · By M Asif Rahman

Comparison of 10 Instagram profile analyzers with free plans, prices and AIScan AIreadiness scores, October 2026 Updated October 13, 2026. Prices, freetier limits and AIScan scores were all checked…

### Publish a UCP Profile for a Store That Is Not on Shopify
- URL: https://aiscan.site/blog/publish-ucp-profile-non-shopify
- Markdown: https://aiscan.site/blog/publish-ucp-profile-non-shopify.md
- Published 2026-10-13 · Category: AI Readiness · By M Asif Rahman

A store that is not on Shopify gets no UCP profile by default. Shopify serves one for its merchants. WooCommerce, BigCommerce, and handbuilt carts publish one only if somebody puts a JSON file at…

### Blocked Probes: 85 of 712 Sites Scored the Firewall, Not the Site
- URL: https://aiscan.site/blog/blocked-probe-share-of-failures
- Markdown: https://aiscan.site/blog/blocked-probe-share-of-failures.md
- Published 2026-10-13 · Category: AI Readiness · By M Asif Rahman

A failing score has two possible meanings. Either the site does not have the thing, or the scanner never got to look. A scan report prints both as the same red mark, and the owner reading it cannot…

### How to Publish an OpenAPI File AI Agents Can Find
- URL: https://aiscan.site/blog/publish-openapi-json-for-agents
- Markdown: https://aiscan.site/blog/publish-openapi-json-for-agents.md
- Published 2026-10-12 · Category: AI Readiness · By M Asif Rahman

An agent that wants to call your API has to know what it can ask for. An OpenAPI file answers that in one fetch: every path, every parameter, every response shape, in JSON a program can read.…

### Which Checks Do Sites Fix? 349 Repeat Scans Compared
- URL: https://aiscan.site/blog/repeat-scan-change-study
- Markdown: https://aiscan.site/blog/repeat-scan-change-study.md
- Published 2026-10-12 · Category: AI Readiness · By M Asif Rahman

Most scanner posts tell you what to fix. Fewer tell you what people actually fixed. AIScan keeps every scan it has ever run, and many sites get scanned more than once, so we can answer a plainer…

### Redirect www and http to one canonical host, the way an agent needs it
- URL: https://aiscan.site/blog/canonical-host-redirects-for-agents
- Markdown: https://aiscan.site/blog/canonical-host-redirects-for-agents.md
- Published 2026-10-11 · Category: AI Readiness · By M Asif Rahman

Type http://www.example.com/pricing into a terminal and count the hops it takes to reach the page. On a clean setup the answer is one, a single 301 straight to the canonical address. On many sites it…

### Agent Discovery Files Census: P1, P3, P4 and E2 Across 835 Sites
- URL: https://aiscan.site/blog/well-known-agent-files-corpus
- Markdown: https://aiscan.site/blog/well-known-agent-files-corpus.md
- Published 2026-10-11 · Category: AI Readiness · By M Asif Rahman

Only 301 of the 835 sites in our scan corpus pass any one of four agentdiscovery checks that depend on a /.wellknown/ file or an API description. All four pass on 99 sites. Take out one platform and…

### Declare your content-use policy to AI: Content Signals, ai.txt, TDMRep
- URL: https://aiscan.site/blog/declare-content-use-policy-to-ai
- Markdown: https://aiscan.site/blog/declare-content-use-policy-to-ai.md
- Published 2026-10-10 · Category: AI Readiness · By M Asif Rahman

Blocking or allowing a crawler answers one question: may it fetch this page. It says nothing about what the crawler may do afterwards. This guide covers the three places a site can state that second…

### UCP profiles: Shopify serves one, most other stores don't
- URL: https://aiscan.site/blog/ucp-profile-adoption-shopify-vs-rest
- Markdown: https://aiscan.site/blog/ucp-profile-adoption-shopify-vs-rest.md
- Published 2026-10-10 · Category: Research · By M Asif Rahman

Shopify stores answer on /.wellknown/ucp. Almost nobody else does. We checked 238 commerceactive sites in the AIScan corpus on 10 October 2026: 12 of 17 Shopify catalogue stores pass check M1 the…

### How to publish /.well-known/mcp/server-card.json on Docusaurus
- URL: https://aiscan.site/blog/mcp-server-card-docusaurus
- Markdown: https://aiscan.site/blog/mcp-server-card-docusaurus.md
- Published 2026-10-09 · Category: AI Readiness · By M Asif Rahman

Docusaurus serves anything in static/ asis, so the MCP server card needs one file and no plugin. This guide covers check P2 on Docusaurus. I built a site on Docusaurus 3.10.2 and ran the dev server…

### Monitor AI Readiness Over Time and Catch Silent Regressions
- URL: https://aiscan.site/blog/monitor-ai-readiness-over-time
- Markdown: https://aiscan.site/blog/monitor-ai-readiness-over-time.md
- Published 2026-10-09 · Category: AI Readiness · By M Asif Rahman

Discovery files rot without anyone touching them. A deploy overwrites robots.txt, a plugin update regenerates llms.txt, a CDN rule starts answering every unknown path with 200. Nothing errors, the…

### Server-Rendered Text and Headings: 16.6% of 699 Sites Fail E3
- URL: https://aiscan.site/blog/server-rendered-text-headings-corpus
- Markdown: https://aiscan.site/blog/server-rendered-text-headings-corpus.md
- Published 2026-10-09 · Category: AI Readiness · By M Asif Rahman

On 9 October 2026 we queried the AIScan database for check E3, the heading and serverrendered text check, across 699 distinct hosts. 498 of them 71.2% pass. 85 12.2% score partial. 116 16.6% fail,…

### How to publish /.well-known/mcp/server-card.json on Hugo
- URL: https://aiscan.site/blog/mcp-server-card-hugo
- Markdown: https://aiscan.site/blog/mcp-server-card-hugo.md
- Published 2026-10-08 · Category: AI Readiness · By M Asif Rahman

Hugo has two ways to put an MCP server card at /.wellknown/mcp/servercard.json, and one of them can build cleanly while writing the file to the wrong place. This guide covers the card for check P2 on…

### How to Test Whether an AI Agent Can Complete a Task on Your Site
- URL: https://aiscan.site/blog/test-agent-can-complete-task
- Markdown: https://aiscan.site/blog/test-agent-can-complete-task.md
- Published 2026-10-08 · Category: AI Readiness · By M Asif Rahman

Finding an MCP endpoint is the easy half. The half that matters is whether an agent can finish a job with it: ask a question, get an answer it can parse, and act on it. A server can answer tools/list…

### Markdown for AI Agents Beyond Docs: Only 19.3% of 701 Sites Serve It
- URL: https://aiscan.site/blog/markdown-twin-beyond-docs
- Markdown: https://aiscan.site/blog/markdown-twin-beyond-docs.md
- Published 2026-10-08 · Category: AI Readiness · By M Asif Rahman

On 8 October 2026 we queried the AIScan database for check C1, the Markdown availability check, across 701 distinct hosts. 135 of them 19.3% returned a real text/markdown response on one of the two…

### How to publish /.well-known/mcp/server-card.json on Lovable
- URL: https://aiscan.site/blog/mcp-server-card-lovable
- Markdown: https://aiscan.site/blog/mcp-server-card-lovable.md
- Published 2026-10-07 · Category: AI Readiness · By M Asif Rahman

Lovable builds two kinds of project, and an MCP server card lives in a different place in each. Get the template wrong and you either edit a file that never ships or create a route that answers 404.…

### Control which AI crawlers reach your site: a robots.txt starter
- URL: https://aiscan.site/blog/robots-txt-starter-for-ai-crawlers
- Markdown: https://aiscan.site/blog/robots-txt-starter-for-ai-crawlers.md
- Published 2026-10-07 · Category: AI Readiness · By M Asif Rahman

Most sites that "block AI" with a robots.txt file end up with one of two outcomes: a file that blocks nothing because a bot name is misspelled, or a file that blocks the crawlers that would have sent…

### Duplicate www and apex Hosts: 38.2% of 696 Sites Fail the Canonical Check
- URL: https://aiscan.site/blog/canonical-host-https-corpus
- Markdown: https://aiscan.site/blog/canonical-host-https-corpus.md
- Published 2026-10-07 · Category: AI Readiness · By M Asif Rahman

On 7 October 2026 we queried the AIScan database for check E4, the canonical host and HTTPS check, across 696 distinct hosts. 405 pass. The other 291 are partial, and not one fails outright. The…

### How to publish /.well-known/mcp/server-card.json on Astro
- URL: https://aiscan.site/blog/mcp-server-card-astro
- Markdown: https://aiscan.site/blog/mcp-server-card-astro.md
- Published 2026-10-06 · Category: AI Readiness · By M Asif Rahman

Astro serves a file under public/ as it is and builds anything in src/pages/ through its own router, and an MCP server card can live in either place. The two behave differently in ways that only show…

### FAQ and Q&A schema that answers an agent's questions
- URL: https://aiscan.site/blog/faq-qa-schema-for-agents
- Markdown: https://aiscan.site/blog/faq-qa-schema-for-agents.md
- Published 2026-10-06 · Category: AI Readiness · By M Asif Rahman

Most FAQ schema advice is two years out of date. Google limited its FAQ rich result to government and health sites in 2023, then stopped showing it at all in May 2026. Plenty of guides still tell you…

### Link Header Discovery: Only 9.3% of 690 Sites Name an Agent File
- URL: https://aiscan.site/blog/link-header-discovery-corpus
- Markdown: https://aiscan.site/blog/link-header-discovery-corpus.md
- Published 2026-10-06 · Category: AI Readiness · By M Asif Rahman

Most sites send an HTTP Link header on their homepage. Almost none of them use it to tell an agent anything. On 6 October 2026 we queried AIScan's production database for the D3 check across 690…

### How to serve an MCP server card on Next.js
- URL: https://aiscan.site/blog/mcp-server-card-nextjs
- Markdown: https://aiscan.site/blog/mcp-server-card-nextjs.md
- Published 2026-10-05 · Category: AI Readiness · By M Asif Rahman

Next.js will serve an MCP server card without any plugin, but two things catch people out: where the file lives, and the proxy file that quietly redirects the request to your login page. This guide…

### Does AIScan Give the Same Score Twice? 303 Repeat Scans Measured
- URL: https://aiscan.site/blog/aiscan-check-determinism-self-audit
- Markdown: https://aiscan.site/blog/aiscan-check-determinism-self-audit.md
- Published 2026-10-05 · Category: AI Readiness · By M Asif Rahman

Every scanner owes its users one answer before it asks for trust: if I run you twice, do I get the same grade? Most tools never publish it. We can, because AIScan keeps every scan it has ever run,…

### Fetch Your Own Site the Way an AI Agent Sees It
- URL: https://aiscan.site/blog/fetch-site-as-ai-agent-curl
- Markdown: https://aiscan.site/blog/fetch-site-as-ai-agent-curl.md
- Published 2026-10-05 · Category: AI Readiness · By M Asif Rahman

Your site has at least three audiences: people in a browser, search crawlers, and AI agents. Servers can treat each differently, and the only way to know is to ask as each of them and compare the…

### How to serve text/markdown to AI agents on Docusaurus
- URL: https://aiscan.site/blog/markdown-for-ai-agents-docusaurus
- Markdown: https://aiscan.site/blog/markdown-for-ai-agents-docusaurus.md
- Published 2026-10-04 · Category: AI Readiness · By M Asif Rahman

An AI agent that sends Accept: text/markdown to a Docusaurus page gets HTML back, because docusaurus build writes files and a file cannot read a request header. AIScan check C1 grades exactly that…

### The Agent-Readiness Fix That Moves Your Score Most (2026): 681 Sites Ranked
- URL: https://aiscan.site/blog/agent-readiness-highest-impact-fix
- Markdown: https://aiscan.site/blog/agent-readiness-highest-impact-fix.md
- Published 2026-10-04 · Category: AI Readiness · By M Asif Rahman

Every agentreadiness checklist ends the same way: twentyodd items, all marked "important", no order. A site owner with one free afternoon cannot do twenty things, and nobody tells them which one to…

### agents.md: What It Is and How It Differs from llms.txt
- URL: https://aiscan.site/blog/agents-md-explained
- Markdown: https://aiscan.site/blog/agents-md-explained.md
- Published 2026-10-04 · Category: AI Readiness · By M Asif Rahman

Ask three people what agents.md is and you get three answers. One says it is the file coding agents read in a repository. One says it is the Markdown page a store publishes for shopping agents. One…

### How to serve text/markdown to AI agents on Hugo
- URL: https://aiscan.site/blog/markdown-for-ai-agents-hugo
- Markdown: https://aiscan.site/blog/markdown-for-ai-agents-hugo.md
- Published 2026-10-03 · Category: AI Readiness · By M Asif Rahman

An AI agent that sends Accept: text/markdown to a Hugo page gets HTML back, because Hugo builds files and a file cannot read a request header. AIScan check C1 grades exactly that request. This guide…

### Publish a Sitemap AI Crawlers Can Actually Find and Trust
- URL: https://aiscan.site/blog/sitemap-ai-crawlers-can-find
- Markdown: https://aiscan.site/blog/sitemap-ai-crawlers-can-find.md
- Published 2026-10-03 · Category: AI Readiness · By M Asif Rahman

A sitemap that returns 200 is the start of the job, not the end of it. Of 14 wellknown sites we fetched on 3 October 2026, 12 declared a sitemap in robots.txt. One of those, Stripe, serves its file…

### Can Your Site Say "Nothing Changed"? ETag and Last-Modified on 300 Hosts
- URL: https://aiscan.site/blog/etag-last-modified-adoption-corpus
- Markdown: https://aiscan.site/blog/etag-last-modified-adoption-corpus.md
- Published 2026-10-03 · Category: AI Readiness · By M Asif Rahman

We probed 300 popular hosts on 3 October 2026, and only 77 of the 152 usable homepages sent a cache validator at all. That is 50.7%. Until a site sends an ETag or a LastModified header, no crawler,…

### How to serve text/markdown to AI agents on Astro
- URL: https://aiscan.site/blog/markdown-for-ai-agents-astro
- Markdown: https://aiscan.site/blog/markdown-for-ai-agents-astro.md
- Published 2026-10-02 · Category: AI Readiness · By M Asif Rahman

An AI agent that sends Accept: text/markdown to an Astro page gets the rendered HTML back, because Astro does not negotiate content types on its own. AIScan check C1 grades exactly that request. This…

### JSON-LD for AI: The Schema Types Worth Adding (C3, M4)
- URL: https://aiscan.site/blog/json-ld-schema-types-for-ai
- Markdown: https://aiscan.site/blog/json-ld-schema-types-for-ai.md
- Published 2026-10-02 · Category: AI Readiness · By M Asif Rahman

Most guides to structured data list thirty types and leave you to guess which ones matter. For a normal site, six are worth adding, and they fall into three groups: types that identify the site…

### What Sites Tell GPTBot in robots.txt, and Whether They Mean It
- URL: https://aiscan.site/blog/ai-crawler-declaration-census
- Markdown: https://aiscan.site/blog/ai-crawler-declaration-census.md
- Published 2026-10-02 · Category: AI Readiness · By M Asif Rahman

We pulled robots.txt from 538 hosts in the AIScan scan corpus on 2 October 2026 and parsed 451 of them with a group parser that follows RFC 9309. The headline: 173 of the 451 files 38.4% name at…

### How to serve text/markdown to AI agents on WordPress
- URL: https://aiscan.site/blog/markdown-for-ai-agents-wordpress
- Markdown: https://aiscan.site/blog/markdown-for-ai-agents-wordpress.md
- Published 2026-10-01 · Category: AI Readiness · By M Asif Rahman

An AI agent that sends Accept: text/markdown to a WordPress page gets the full theme HTML back, because WordPress core does not do content negotiation for Markdown. AIScan check C1 grades exactly…

### Make a Page Answerable by AI: C1, C3 and E3 Fixes
- URL: https://aiscan.site/blog/make-a-page-answerable-by-ai
- Markdown: https://aiscan.site/blog/make-a-page-answerable-by-ai.md
- Published 2026-10-01 · Category: AI Readiness · By M Asif Rahman

An AI assistant that answers a question about your business does not read your page the way a visitor does. It receives the HTML your server sends, strips it down to text, and lifts whatever looks…

### Your Sitemap Passes the Check and Still Fails Agents: 669 Sites Measured
- URL: https://aiscan.site/blog/sitemap-reality-corpus-study
- Markdown: https://aiscan.site/blog/sitemap-reality-corpus-study.md
- Published 2026-10-01 · Category: AI Readiness · By M Asif Rahman

A sitemap that returns HTTP 200 is not the same as a sitemap an agent can use. In AIScan's corpus of 669 sites, check D2 passes 499 of them 74.6%, and every one of those passes carries the same…

### How to serve text/markdown to AI agents on Next.js
- URL: https://aiscan.site/blog/markdown-for-ai-agents-nextjs
- Markdown: https://aiscan.site/blog/markdown-for-ai-agents-nextjs.md
- Published 2026-09-30 · Category: AI Readiness · By M Asif Rahman

An agent that sends Accept: text/markdown to a Next.js site gets the full HTML shell back unless you build the second representation yourself. Next.js does not do content negotiation for you. This…

### llms-full.txt: When a Full-Text File Beats a Link List
- URL: https://aiscan.site/blog/llms-full-txt-when-to-use
- Markdown: https://aiscan.site/blog/llms-full-txt-when-to-use.md
- Published 2026-09-30 · Category: AI Readiness · By M Asif Rahman

An agent that reads your llms.txt gets a map. To answer one question it then has to fetch three or four of the linked pages, one request each. llmsfull.txt removes that loop: it is one file with the…

### How Many Sites Show an AI Crawler a Blank Page: 663 Measured
- URL: https://aiscan.site/blog/empty-dom-render-blindness-corpus
- Markdown: https://aiscan.site/blog/empty-dom-render-blindness-corpus.md
- Published 2026-09-30 · Category: AI Readiness · By M Asif Rahman

Of 663 sites in the AIScan corpus, 195 send an AI crawler an initial HTML document with no <h1 in it. That is 29.4% of the population, and it is the number the singleframework guides never add up.…

### How to publish an RSS/Atom/JSON feed and declare it in the head on Docusaurus
- URL: https://aiscan.site/blog/rss-feed-docusaurus
- Markdown: https://aiscan.site/blog/rss-feed-docusaurus.md
- Published 2026-09-29 · Category: AI Readiness · By M Asif Rahman

Docusaurus is the one platform in this series where the feed usually exists before you ask for it. @docusaurus/presetclassic includes the blog plugin, and the blog plugin writes RSS and Atom files on…

### Link Headers for AI Agent Discovery: Set Up RFC 8288 Relations
- URL: https://aiscan.site/blog/link-headers-for-agent-discovery
- Markdown: https://aiscan.site/blog/link-headers-for-agent-discovery.md
- Published 2026-09-29 · Category: AI Readiness · By M Asif Rahman

An agent that lands on your homepage has already paid for the request. If that response can also say where your llms.txt, sitemap and API catalog live, the agent skips the guessing entirely. The…

### What a Shopping Agent Can Read About Your Prices: 282 Sites Measured
- URL: https://aiscan.site/blog/machine-readable-pricing-corpus-study
- Markdown: https://aiscan.site/blog/machine-readable-pricing-corpus-study.md
- Published 2026-09-29 · Category: AI Readiness · By M Asif Rahman

An agent asked to compare three tools by price has two ways to answer. It can read a structured field, or it can read your marketing prose and guess. The howto on pricing…

### How to publish an RSS/Atom/JSON feed and declare it in the head on Replit
- URL: https://aiscan.site/blog/rss-feed-replit
- Markdown: https://aiscan.site/blog/rss-feed-replit.md
- Published 2026-09-28 · Category: AI Readiness · By M Asif Rahman

A Replit app can serve /feed.xml two completely different ways, and which one applies to you depends on a setting you may never have opened. Get it backwards and you'll ship a feed that 404s in…

### OAuth Discovery for AI Agents: RFC 8414 and RFC 9728 Explained
- URL: https://aiscan.site/blog/oauth-discovery-for-agents
- Markdown: https://aiscan.site/blog/oauth-discovery-for-agents.md
- Published 2026-09-28 · Category: AI Readiness · By M Asif Rahman

An agent calls your API cold, with no token. Your server returns 401, and the agent has nowhere to go from there unless something on your domain names the authorization server it should talk to. Two…

### Audit Your Own Agent Readiness From a Terminal in Ten Minutes
- URL: https://aiscan.site/blog/agent-readiness-audit-from-the-terminal
- Markdown: https://aiscan.site/blog/agent-readiness-audit-from-the-terminal.md
- Published 2026-09-28 · Category: AI Readiness · By M Asif Rahman

Paste your URL at aiscan.sitehttps://aiscan.site/ or run npx aiscancli yoursite.com and you get a score in under thirty seconds. Fine for a first read, not enough if you actually want to know what…

### Publish an Agent Skills Index for Your Website
- URL: https://aiscan.site/blog/publish-agent-skills-index
- Markdown: https://aiscan.site/blog/publish-agent-skills-index.md
- Published 2026-09-27 · Category: AI Readiness · By M Asif Rahman

| If you want to… | Do this | Time | What it changes | ||||| | Give an agent a reusable, ondemand capability | Write a SKILL.md with name + description frontmatter, plus any scripts or reference…

### The AI-Crawler Statistics Everyone Quotes, and Which You Can Actually Check
- URL: https://aiscan.site/blog/ai-crawl-statistics-reproducibility
- Markdown: https://aiscan.site/blog/ai-crawl-statistics-reproducibility.md
- Published 2026-09-27 · Category: AI Readiness · By M Asif Rahman

Every AIreadiness article, including plenty of ours, opens with a number: 78% of sites have a robots.txt, 28% publish an llms.txt, AI crawlers can't read JavaScript, one crawler is spoofed 17 times…

### How to publish an RSS/Atom/JSON feed and declare it in the head on Lovable
- URL: https://aiscan.site/blog/rss-feed-lovable
- Markdown: https://aiscan.site/blog/rss-feed-lovable.md
- Published 2026-09-26 · Category: AI Readiness · By M Asif Rahman

A sitemap tells a crawler what exists. It does not tell an agent what changed since last week, and that gap is what check E5 grades: whether the site publishes a dated content feed at all. Run…

### Web Bot Auth: Cryptographic Crawler Identity, Explained
- URL: https://aiscan.site/blog/web-bot-auth-explained
- Markdown: https://aiscan.site/blog/web-bot-auth-explained.md
- Published 2026-09-26 · Category: AI Readiness · By M Asif Rahman

Across the 499 sites in our own scan corpus, exactly 19 publish a working Web Bot Auth key directory. That's 3.8%. If you run any kind of automated client that fetches other people's sites a scraping…

### How Stale Are AI Crawler IP Lists? We Checked All 15 Files
- URL: https://aiscan.site/blog/ai-crawler-ip-list-staleness
- Markdown: https://aiscan.site/blog/ai-crawler-ip-list-staleness.md
- Published 2026-09-26 · Category: AI Readiness · By M Asif Rahman

Every guide on verifying an AI crawler tells you to check its request against the operator's published IP range file. Almost none of them tell you when that file was last regenerated. The answer is…

### How to ship one h1, title, meta description and JSON-LD on Wix
- URL: https://aiscan.site/blog/title-meta-schema-wix
- Markdown: https://aiscan.site/blog/title-meta-schema-wix.md
- Published 2026-09-25 · Category: AI Readiness · By M Asif Rahman

Ask AIScan's check C3 to grade a Wix page and it looks for four specific things in the raw HTML: exactly one <h1 tag, a <title, a name="description" meta tag, and at least one application/ld+json…

### Publish an API Catalog Agents Can Discover (RFC 9727)
- URL: https://aiscan.site/blog/publish-api-catalog-rfc-9727
- Markdown: https://aiscan.site/blog/publish-api-catalog-rfc-9727.md
- Published 2026-09-25 · Category: AI Readiness · By M Asif Rahman

Agents that want to call your API have to guess the URL today. RFC 9727https://www.rfceditor.org/rfc/rfc9727.html gives them a fixed place to look instead: /.wellknown/apicatalog, a small JSON…

### The Soft 404 Problem: Why HTTP 200 Breaks Every Agent Discovery Check
- URL: https://aiscan.site/blog/soft-404-agent-discovery-problem
- Markdown: https://aiscan.site/blog/soft-404-agent-discovery-problem.md
- Published 2026-09-25 · Category: AI Readiness · By M Asif Rahman

Every convention on the agentic web works the same way: an agent asks for a file at a wellknown address, and a 200 means the file is there. /llms.txt, /robots.txt, a .md suffix on a documentation…

### How to ship one h1, title, meta description and JSON-LD on Replit
- URL: https://aiscan.site/blog/title-meta-schema-replit
- Markdown: https://aiscan.site/blog/title-meta-schema-replit.md
- Published 2026-09-24 · Category: AI Readiness · By M Asif Rahman

| Question | Answer | ||| | Where does Replit check this? | The SEO Rating card in the Growth pane, calculated after every publish | | Does that rating cover all four C3 signals? | No. Title, meta…

### Add an RSS Feed AI Agents Can Actually Find
- URL: https://aiscan.site/blog/add-rss-feed-declare-in-head
- Markdown: https://aiscan.site/blog/add-rss-feed-declare-in-head.md
- Published 2026-09-24 · Category: AI Readiness · By M Asif Rahman

Two staticsite generators build you a feed with zero configuration. One declares it in <head on every site we tested. The other only manages it on five of every eight. Same defaulton feature, same…

### Your llms.txt Is Sending AI Agents to 404s: We Measured It on Our Own Site
- URL: https://aiscan.site/blog/llms-txt-agent-404-parsing-errors
- Markdown: https://aiscan.site/blog/llms-txt-agent-404-parsing-errors.md
- Published 2026-09-24 · Category: AI Readiness · By M Asif Rahman

| Question | Short answer | ||| | What broke? | Our own llms.txt, parsed the way most link extractors parse Markdown, sent roughly 55% of nonscan agent traffic to a 404. | | Why? | The standard…

### How to ship one h1, title, meta description and JSON-LD on Lovable
- URL: https://aiscan.site/blog/title-meta-schema-lovable
- Markdown: https://aiscan.site/blog/title-meta-schema-lovable.md
- Published 2026-09-23 · Category: AI Readiness · By M Asif Rahman

| C3 signal | Where it lives on Lovable | How you fix it | |||| | Title + meta description | Generated per page while Lovable builds your app | Ask in the project chat, there is no settings field | |…

### What a real 404 should return to an AI agent
- URL: https://aiscan.site/blog/correct-404-for-ai-agents
- Markdown: https://aiscan.site/blog/correct-404-for-ai-agents.md
- Published 2026-09-23 · Category: AI Readiness · By M Asif Rahman

You already know the first rule of a good 404: return status 404, not 200. That half of the check is a status line, and our own CIgate guidehttps://aiscan.site/blog/aiscancigategithubactions already…

### Does Blocking AI Training Protect Your Content?
- URL: https://aiscan.site/blog/does-blocking-ai-training-protect-content
- Markdown: https://aiscan.site/blog/does-blocking-ai-training-protect-content.md
- Published 2026-09-23 · Category: AI Readiness · By M Asif Rahman

Too long, didn't read? Here's the honest version. | If you're asking... | The evidence says | What to do instead | |||| | Will blocking crawlers remove content already used for training? | No.…

### Make your pricing machine-readable for shopping agents
- URL: https://aiscan.site/blog/machine-readable-pricing-schema
- Markdown: https://aiscan.site/blog/machine-readable-pricing-schema.md
- Published 2026-09-22 · Category: AI Readiness · By M Asif Rahman

| If you want to… | Do this | Where | AIScan check | ||||| | Let an agent answer "what does this cost" without reading prose | Add Offer or AggregateOffer JSONLD to the pricing page | A <script…

### How to ship one h1, title, meta description and JSON-LD on Ghost
- URL: https://aiscan.site/blog/title-meta-schema-ghost
- Markdown: https://aiscan.site/blog/title-meta-schema-ghost.md
- Published 2026-09-22 · Category: AI Readiness · By M Asif Rahman

| Signal | Who owns it on Ghost | Where you set it | |||| | <h1 | The theme, from your post's literal title | Nothing to set. It's the title field itself | | <title tag | The {{metatitle}} helper |…

### How to ship one h1, title, meta description and JSON-LD on Webflow
- URL: https://aiscan.site/blog/title-meta-schema-webflow
- Markdown: https://aiscan.site/blog/title-meta-schema-webflow.md
- Published 2026-09-10 · Category: AI Readiness · By M Asif Rahman

Webflow's SEO panel now writes three of the four signals that AIScan's C3 check grades, and the fourth one is not in any panel. On a paid Site plan you can set a page's title tag, its meta…

### Publish an MCP Server Card So Agents Can Find Your Tools
- URL: https://aiscan.site/blog/publish-mcp-server-card
- Markdown: https://aiscan.site/blog/publish-mcp-server-card.md
- Published 2026-09-10 · Category: AI Readiness · By M Asif Rahman

On 10 September 2026 we probed every host running a remote MCP server listed in the official MCP Registry: 79 domains, three requests each. Seven of them publish a parseable server card at…

### Docs Sites and AI Agents: Four Markdown Routes, 52 Sites Tested
- URL: https://aiscan.site/blog/documentation-sites-ai-agent-readiness
- Markdown: https://aiscan.site/blog/documentation-sites-ai-agent-readiness.md
- Published 2026-09-10 · Category: AI Readiness · By M Asif Rahman

Ask an AI assistant how to configure a webhook, add a database index, or set a cache header, and it does not go looking for a blog post. It goes to the vendor's documentation. Docs are the…

### How to ship one h1, title, meta description and JSON-LD on Next.js
- URL: https://aiscan.site/blog/title-meta-schema-nextjs
- Markdown: https://aiscan.site/blog/title-meta-schema-nextjs.md
- Published 2026-09-09 · Category: AI Readiness · By M Asif Rahman

Next.js hands you a Metadata API that writes your <title and your meta description into the head automatically. It does not write your JSONLD, and it does not write your <h1. Those two are ordinary…

### Gate AI readiness in CI so a regression fails the build
- URL: https://aiscan.site/blog/aiscan-ci-gate-github-actions
- Markdown: https://aiscan.site/blog/aiscan-ci-gate-github-actions.md
- Published 2026-09-09 · Category: AI Readiness · By M Asif Rahman

A green pipeline is supposed to mean the site is fine. On an AI readiness check it often means something narrower: that the gate could not tell a file from a phantom. This step, which appears in a…

### Stealth Crawling vs User-Driven Fetching in 2026: What 75 Sites Actually Do
- URL: https://aiscan.site/blog/stealth-crawling-user-driven-fetching-debate
- Markdown: https://aiscan.site/blog/stealth-crawling-user-driven-fetching-debate.md
- Published 2026-09-09 · Category: AI Readiness · By M Asif Rahman

On 4 August 2025 Cloudflare removed Perplexity from its verified bot list and published the evidence. Perplexity's answer was that fetching a page because a person asked for it is not the same act as…

### How to ship one h1, title, meta description and JSON-LD on WordPress
- URL: https://aiscan.site/blog/title-meta-schema-wordpress
- Markdown: https://aiscan.site/blog/title-meta-schema-wordpress.md
- Published 2026-09-08 · Category: AI Readiness · By M Asif Rahman

WordPress sites fail AIScan's C3 check more often than their owners expect, and almost never for the reason they expect. Across the 129 WordPress sites in our scan corpus, verified on 8 September…

### The Agentic Commerce Stack in 2026: UCP, ACP, AP2 and x402, Measured on 30 Storefronts
- URL: https://aiscan.site/blog/agentic-commerce-protocols-explained
- Markdown: https://aiscan.site/blog/agentic-commerce-protocols-explained.md
- Published 2026-09-08 · Category: AI Readiness · By M Asif Rahman

Five protocol names now compete for the same sentence in every agentic commerce explainer: UCP, ACP, AP2, x402 and MPP. Almost every explainer treats them as rivals. They are not. Four of the five…

### How to declare allow/deny rules for named AI crawlers in robots.txt on Wix
- URL: https://aiscan.site/blog/robots-txt-ai-bots-wix
- Markdown: https://aiscan.site/blog/robots-txt-ai-bots-wix.md
- Published 2026-09-07 · Category: AI Readiness · By M Asif Rahman

Wix writes your robots.txt for you. Fetched on 7 September 2026, the file at www.wix.com/robots.txt ran to 90 nonempty lines: one Useragent: group, 87 Disallow rules, and a single Sitemap: line at…

### Google AI Overviews in 2026: What You Can Actually Control, Measured on 110 Publishers
- URL: https://aiscan.site/blog/google-ai-overviews-what-you-can-control
- Markdown: https://aiscan.site/blog/google-ai-overviews-what-you-can-control.md
- Published 2026-09-07 · Category: AI Readiness · By M Asif Rahman

Search Console will tell you that AI Overviews sent you traffic. It will not tell you how to turn them off, because there is no setting to turn off. Google publishes exactly four controls that reach…

### How to declare allow/deny rules for named AI crawlers in robots.txt on Docusaurus
- URL: https://aiscan.site/blog/robots-txt-ai-bots-docusaurus
- Markdown: https://aiscan.site/blog/robots-txt-ai-bots-docusaurus.md
- Published 2026-09-06 · Category: AI Readiness · By M Asif Rahman

Docusaurus gives you somewhere to put a robots.txt in about ten seconds. Making sure a crawler ever reads it is the harder half, because the most common Docusaurus deployment shape files the rules at…

### Cloudflare Pay Per Crawl in 2026: Should You Charge AI Crawlers? 119 Sites Measured
- URL: https://aiscan.site/blog/cloudflare-pay-per-crawl-should-you-charge
- Markdown: https://aiscan.site/blog/cloudflare-pay-per-crawl-should-you-charge.md
- Published 2026-09-06 · Category: AI Readiness · By M Asif Rahman

Stack Overflow will sell you its homepage for fifty cents. Point an AI crawler at it and the server answers HTTP 402 Payment Required, crawlerprice: USD 0.5, and a JSON body naming Cloudflare's Pay…

### AI Crawler Traffic in 2026: Only 51.8% of Sites Can Answer "Nothing Changed"
- URL: https://aiscan.site/blog/ai-crawler-traffic-waste-2026
- Markdown: https://aiscan.site/blog/ai-crawler-traffic-waste-2026.md
- Published 2026-09-05 · Category: AI Readiness · By M Asif Rahman

Verified 5 September 2026. All measurements in this article were taken on that date. Every AI crawler that visits your site asks the same handful of questions over and over. Where is your robots.txt.…

### llms.txt vs robots.txt vs sitemap.xml in 2026: Six Files, Six Jobs
- URL: https://aiscan.site/blog/llms-txt-vs-robots-txt-vs-sitemap
- Markdown: https://aiscan.site/blog/llms-txt-vs-robots-txt-vs-sitemap.md
- Published 2026-09-04 · Category: AI Readiness · By M Asif Rahman

Verified 4 September 2026. Every figure below was measured or fetched on that date. Three files keep getting compared as if they were competing for the same job: robots.txt, sitemap.xml and llms.txt.…

### The State of AI Agent Readiness in 2026: 473 Sites Measured
- URL: https://aiscan.site/blog/state-of-ai-agent-readiness-2026
- Markdown: https://aiscan.site/blog/state-of-ai-agent-readiness-2026.md
- Published 2026-09-03 · Category: AI Readiness · By M Asif Rahman

Half of the web's agentreadiness problem is already solved, and almost nobody has noticed which half. Across 473 real websites scanned by AIScan between 24 August and 3 September 2026, the median…

### WebMCP in 2026: Do You Actually Need It Yet? 85 Sites Measured
- URL: https://aiscan.site/blog/webmcp-do-you-need-it-yet
- Markdown: https://aiscan.site/blog/webmcp-do-you-need-it-yet.md
- Published 2026-09-02 · Category: AI Readiness · By M Asif Rahman

Verified 2 September 2026. Every number below was measured on this date against live sites. The honest answer to "do I need WebMCP yet" turned out to depend on a question nobody asks first: is it…

### Is Your llms.txt Valid? 39 Real Files Tested, and the 11 Mistakes That Break Them
- URL: https://aiscan.site/blog/llms-txt-validator-common-mistakes
- Markdown: https://aiscan.site/blog/llms-txt-validator-common-mistakes.md
- Published 2026-09-01 · Category: AI Readiness · By M Asif Rahman

Verified 1 September 2026. Every figure here comes from a live fetch on that date. We fetched /llms.txt from 50 wellknown sites, kept the 39 that returned a real file, and ran every one against the…

### Cloudflare Agent Readiness vs Chrome's Agentic Browsing Audit in 2026: 28 Checks, One Overlap
- URL: https://aiscan.site/blog/cloudflare-agent-readiness-vs-lighthouse-agentic-browsing
- Markdown: https://aiscan.site/blog/cloudflare-agent-readiness-vs-lighthouse-agentic-browsing.md
- Published 2026-08-31 · Category: AI Readiness · By M Asif Rahman

Verified 31 August 2026. Every score, status code and audit weight below was produced on this date by running both tools live. The Lighthouse figures come from Lighthouse 13.4.1 driving Chrome for…

### AI Readiness Scanner Pricing in 2026: Every Price, Verified and Dated
- URL: https://aiscan.site/blog/ai-readiness-scanner-pricing-2026
- Markdown: https://aiscan.site/blog/ai-readiness-scanner-pricing-2026.md
- Published 2026-08-30 · Category: AI Readiness · By M Asif Rahman

Verified 30 August 2026. Every price below was fetched from the vendor's own pricing page on this date. Where a vendor publishes no price, that is recorded as a finding rather than filled in with a…

### The 10 Best AI Agent Readiness Scanners in 2026, Tested on One URL
- URL: https://aiscan.site/blog/best-ai-agent-readiness-scanners-2026
- Markdown: https://aiscan.site/blog/best-ai-agent-readiness-scanners-2026.md
- Published 2026-08-29 · Category: AI Readiness · By M Asif Rahman

Verified 29 August 2026. Every score below was captured from the tool's own public endpoint on that date, and every disagreement was adjudicated by hand against the live site. A category of tool has…

### Your Next.js Site Ranks on Google and Is Invisible to ChatGPT: How to Prove It
- URL: https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers
- Markdown: https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers.md
- Published 2026-08-28 · Category: AI Readiness · By M Asif Rahman

Googlebot runs your JavaScript. The crawlers that build ChatGPT's and Claude's indexes, as far as anyone has measured, do not. That gap is why a React or Next.js site can hold page one on Google and…

### Every AI Crawler User-Agent in 2026: The Verified List and How to Spot a Fake
- URL: https://aiscan.site/blog/ai-crawler-user-agent-list-2026
- Markdown: https://aiscan.site/blog/ai-crawler-user-agent-list-2026.md
- Published 2026-08-27 · Category: AI Readiness · By M Asif Rahman

Verified 27 August 2026. Every useragent below was read from the operator's own documentation on that date. Every IP range file was fetched and counted. Every reverseDNS result was tested live.…

### Does llms.txt Actually Work in 2026? Google Ignores It, Chrome Audits It, 97% Get Zero Requests
- URL: https://aiscan.site/blog/does-llms-txt-actually-work-2026
- Markdown: https://aiscan.site/blog/does-llms-txt-actually-work-2026.md
- Published 2026-08-26 · Category: AI Readiness · By M Asif Rahman

Updated 26 August 2026. In June, Google added a line to its own AI search documentation saying that Google Search ignores llms.txt files. Two months earlier, a different Google team shipped a…

### Search, Agent, Training: What Cloudflare's Three AI Crawler Categories Actually Control
- URL: https://aiscan.site/blog/cloudflare-search-agent-training-crawler-categories
- Markdown: https://aiscan.site/blog/cloudflare-search-agent-training-crawler-categories.md
- Published 2026-08-25 · Category: AI Readiness · By M Asif Rahman

Updated 25 August 2026. Cloudflare now sorts crawler behaviour into three categories you can configure — Search, Agent, and Training — and the setting you pick for each decides whether Google indexes…

### Cloudflare's AI Bot Block Is About to Catch Googlebot: What Changes on 15 September 2026
- URL: https://aiscan.site/blog/cloudflare-ai-bot-block-googlebot-2026
- Markdown: https://aiscan.site/blog/cloudflare-ai-bot-block-googlebot-2026.md
- Published 2026-08-25 · Category: AI Readiness · By M Asif Rahman

Update — 28 September 2026: the 15 September migration landed, and it did not go the way this article predicted. Cloudflare's own followup post, "Introducing an Accountable designation for AI…

### Best AI Readiness Checkers in 2026 | AIScan
- URL: https://aiscan.site/blog/best-ai-readiness-checkers-2026
- Markdown: https://aiscan.site/blog/best-ai-readiness-checkers-2026.md
- Published 2026-06-28 · Category: AI Readiness · By M Asif Rahman

Your site can rank on page one of Google and still be invisible to ChatGPT, Claude, and Perplexity. The two systems read the web differently, and most sites have never been tested for the second one.…

### Best AI Agent-Ready Tools in 2026 | AIScan
- URL: https://aiscan.site/blog/best-ai-agent-ready-tools-2026
- Markdown: https://aiscan.site/blog/best-ai-agent-ready-tools-2026.md
- Published 2026-06-28 · Category: AI Readiness · By M Asif Rahman

Search stopped being about ten blue links. ChatGPT now has roughly 900 million weekly active users according to OpenAI's February 2026 figureshttps://www.omnibound.ai/blog/aisearchstatistics,…


## Discoverability files
- /robots.txt — crawler rules
- /sitemap.xml — dynamic sitemap index: /sitemap-pages.xml, /sitemap-docs.xml, /sitemap-use-cases.xml, /sitemap-blog.xml (scan reports are not listed; browse them at /scans)
- /llms.txt — short pointer file (dynamic, auto-includes new posts)
- /llms-full.txt — this document