AI readiness setup guide by platform (2026)

Verified 17, 18, 20 and 22 October 2026. Updated on 22 October 2026.

An AI-ready site needs six things: a robots.txt that names AI crawlers (D1, B2), a sitemap (D2), a declared feed (E5), an llms.txt (C2), and server-rendered HTML that holds the article text (C3, E3). Which of those your platform writes for you decides the work. Static-first frameworks such as Astro, Docusaurus and Hugo generate the sitemap and the feed but no AI-aware robots.txt or llms.txt, hosted builders such as Framer, Ghost, Squarespace, Webflow and Wix write some files and block others, WordPress ships the HTML, sitemap and feed but names no AI crawler, and each platform has one trap that can hide the whole set from crawlers.

TL;DR

• Every platform gets the same six checks; only the missing files differ.
• Astro: HTML is static by default, so C3 and E3 pass unless you wrap content in client:only. Extension-bearing endpoints 404 behind a trailing slash.
• Docusaurus: sitemap and feed come free, robots.txt and llms.txt do not, and a route can ship an empty __docusaurus div. A baseUrl subpath makes robots.txt void.
• Hugo: robots.txt is off until enableRobotsTXT = true, and the default is 13 bytes naming nobody. 0 of 15 live Hugo sites named an AI crawler.
• Next.js: app/robots.ts and app/sitemap.ts generate the files, but the framework's bot list names no AI crawler, so a client-only fetch leaves the page empty to them.
• Replit: an Agent-built app answers 200 with the homepage HTML for files that do not exist. 10 of 12 apps did.
• Shopify: the default robots.txt blocks /policies/. One robots.txt.liquid template fixes it and adds the AI bot groups.
• Squarespace: the robots.txt names 26 AI crawlers in the wildcard group and gives none a rule. llms.txt is a settings field on 7.1 only.
• Framer: robots.txt, sitemap and a Markdown twin come free; a custom robots.txt upload replaces the generated one, Sitemap: line included.
• Ghost: only theme files and a routes.yaml route reach the root, .md and .json never do, and a disabled llms.txt answers 302, not 404.
• Lovable: TanStack Start renders for everyone; React + Vite renders only for crawlers Lovable verifies. 7 of 9 Vite apps returned under 50 words.
• Webflow: static HTML passes C3 and E3; robots.txt, sitemap and llms.txt are Site settings switches and need a Site plan. llms.txt is not published to staging. .well-known is Enterprise only.
• Wix: files are generated for you. Editing llms.txt stops its automatic updates for good, NLWeb cannot be uninstalled, and no AI crawler is named until you add a group.
• WordPress: a physical robots.txt shadows every plugin rule, so check it first. Name the AI bots and add Content Signals; the sitemap and feed are free.
• Run npx aiscan-cli yoursite.com first. It names the failing rows in one pass.

Pick your platform

All thirteen platforms are folded in. The separate Webflow, Wix and WordPress guides now redirect to their sections below.

What every platform shares

The retrieval crawlers behind ChatGPT, Claude and Perplexity do not run JavaScript. Per Vercel and MERJ, 17 December 2024: "none of the major AI crawlers currently render JavaScript," covering GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. So the HTML your server sends is the page, as far as they are concerned.

The six checks, which the sections below refer to by ID:

Naming crawlers matters more than the value you give them. Google's robots.txt specification says only the most specific matching group applies, so a bot with its own group never reads your * group. Consecutive User-agent lines form one group, so put a blank line between groups that need different rules. Take the tokens from the current crawler list rather than from memory, because operators rename them.

An empty file does not count either. RFC 9309 section 2.2.1 says: "If no group matches the product token and there is no group with a user-agent line with the * value, or no groups are present at all, no rules apply." The robots.txt rules must also sit at the top-level path of the origin, so a file served only under a subpath is void.

Astro

Verified 17 October 2026 against the Astro docs. First written against Astro 7.2.9 on 29 August 2026.

Astro ships static HTML by default, so C3 and E3 usually pass before you touch anything. It does not create the discovery files an agent looks for, and one routing rule can break three of them at once. Budget 25 minutes.

Astro's own documentation site shows there is no upstream example to copy. On 17 October 2026 docs.astro.build/robots.txt still returns two lines, User-agent: * and Allow: /, with no AI crawler group and no Sitemap: line, and docs.astro.build/llms.txt still returns 404.

Rendering. Astro's islands architecture renders most of the page to static HTML and adds JavaScript only where needed (Astro docs, islands), so a React component inside an Astro page is rendered to HTML at build time. One directive undoes that. Per Astro's template directives reference, client:only={string} "skips HTML server rendering, and renders only on the client." A component marked client:only="react" leaves an empty element in the built HTML. Grep first:

grep -rn 'client:only' src/

1. Set your deployed URL. Add site: 'https://example.com' to astro.config.mjs. Three of the four files need it for absolute URLs.

2. Write robots.txt. A plain public/robots.txt works. Generating it from site keeps the sitemap line from drifting. Create src/pages/robots.txt.ts:

import type { APIRoute } from 'astro';

const getRobotsTxt = (sitemapURL: URL) => `User-agent: *
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Sitemap: ${sitemapURL.href}
`;

export const GET: APIRoute = ({ site }) => {
  const sitemapURL = new URL('sitemap-index.xml', site);
  return new Response(getRobotsTxt(sitemapURL));
};

To permit retrieval but refuse training, give OAI-SearchBot, Claude-SearchBot and PerplexityBot Allow: /, and GPTBot, ClaudeBot, Google-Extended and Applebot-Extended Disallow: /.

3. Add the sitemap. Run npx astro add sitemap. The build writes sitemap-index.xml and sitemap-0.xml. It cannot generate entries for dynamic routes in SSR mode, and nothing links the file for you, so add <link rel="sitemap" href="/sitemap-index.xml" /> to your layout head.

4. Add the feed and declare it. Install @astrojs/rss, export a GET from src/pages/rss.xml.js that returns rss({ title, description, site: context.site, items }), and build items from getCollection(). E5 fails on the declaration far more often than on the feed, so add this to the head:

<link rel="alternate" type="application/rss+xml" title="Your Site" href={new URL("rss.xml", Astro.site)} />

The RSS helper adds trailing slashes to links by default. If your config sets trailingSlash: "never", pass trailingSlash: false to rss().

5. Add llms.txt. An endpoint at src/pages/llms.txt.ts can call getCollection() and emit an H1, ## sections and Markdown links from the same source your pages use. The llms.txt generator drafts one from a live URL. Starlight docs sites can use the starlight-llms-txt plugin instead. No Astro plugin writes robots.txt with AI crawler groups, so step 2 stays manual.

The trailing-slash trap. The file exists, the build succeeds, and the URL still 404s. From Astro's endpoints guide, checked 17 October 2026: endpoints whose URLs include a file extension "can only be accessed without a trailing slash (e.g. /sitemap.xml), regardless of your build.trailingSlash configuration." Every file above is such an endpoint. /robots.txt/, /rss.xml/ and /llms.txt/ return 404 while the slashless forms return 200. If your host or middleware appends a trailing slash, it hands every crawler a 404 for your whole discovery layer. Test the slashless URL and exempt those paths from any slash-adding rule.

Check it.

curl -sI https://example.com/robots.txt | head -1        # expect 200
curl -s  https://example.com/robots.txt | grep -i gptbot # expect a match
curl -sI https://example.com/sitemap-index.xml | head -1 # expect 200
curl -s  https://example.com/ | grep -o 'rel="alternate"[^>]*rss[^>]*'
curl -s  https://example.com/ | grep -c '<h1'            # expect 1

A single-page Astro marketing site needs neither a sitemap nor a feed. Do step 2 and stop. If the front end reads from a headless WordPress CMS, that CMS domain has its own robots.txt, and the work above does not touch it.

Docusaurus

Verified 17 October 2026 against the Docusaurus docs. Site measurements taken 7 September 2026.

Docusaurus runs a large share of developer documentation, which is the material retrieval systems reach for. The discovery files are the easy half. The hard half is whether a given page contains any words, and Docusaurus says so itself. Budget 30 minutes.

Why a page can be empty and still be "static". Docusaurus is a React single-page application that emits static HTML as an optimisation. Its Static Site Generation page, rechecked 17 October 2026, says: "Docusaurus is ultimately a single-page application, so static site generation is only an optimization (progressive enhancement, as it's called), but our functionality does not fully depend on those HTML files." A route that escapes server rendering ships <div id="__docusaurus"></div> empty.

On 7 September 2026 we fetched the Cypress documentation home page with a desktop browser user agent. It returned 6,826 bytes with that empty div, data-has-hydrated="false" and two words of visible text. Our scanner grades it 72, Level 4, with C3 and E3 failing on "2 words of server-rendered text, 0 <h2>, 0 <h3>". The same site's robots.txt names eight AI crawlers. The failure was one route, not the site: two other documentation pages on that host returned 2,196 and 3,897 words. On Docusaurus, C3 and E3 are a per-route outcome.

What 15 live Docusaurus sites publish (30 candidate documentation domains fingerprinted, 15 confirmed, 6 paths each, 90 requests, 7 September 2026):

Four of the nine robots.txt files are byte-identical at 1,248 bytes and contain no User-agent, Allow, Disallow or Sitemap line, only a licence preamble. Under RFC 9309 that means no rules apply. The one site that names AI crawlers stacks all eight into a single group with *, so no named agent gets a rule of its own. That is harmless when everything is allowed, and wrong the moment the rules differ.

Path 1: the static folder. Docusaurus copies the contents of static/ into the root of the build output, with the directory structure preserved.

1. Create static/robots.txt with real groups separated by blank lines and a Sitemap: line:

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /

Sitemap: https://example.org/sitemap.xml

1. Add static/llms.txt with an H1, a blockquote summary and H2 sections of [title](url) links. The llms.txt generator writes the skeleton.
2. For .well-known files, static/.well-known/ works. On GitHub Pages, add an empty static/.nojekyll.
3. Run npm run build and confirm the files landed in build/.

Path 2: generate the files from your routes. The postBuild(props) plugin hook runs when a production build finishes and receives siteConfig, outDir, baseUrl and routesPaths. Create plugins/discovery-files.js:

import fs from 'node:fs/promises';
import path from 'node:path';

export default function discoveryFiles() {
  return {
    name: 'discovery-files',
    async postBuild({siteConfig, outDir, baseUrl, routesPaths}) {
      if (baseUrl !== '/') {
        throw new Error(
          `robots.txt is void under baseUrl "${baseUrl}". ` +
          'Deploy at a domain root, or remove the robots.txt half.'
        );
      }
      const origin = siteConfig.url.replace(/\/$/, '');
      await fs.writeFile(
        path.join(outDir, 'robots.txt'),
        `User-agent: *\nAllow: /\n\nSitemap: ${origin}/sitemap.xml\n`,
      );
      const docs = routesPaths.filter((r) => r.startsWith('/docs/'));
      await fs.writeFile(
        path.join(outDir, 'llms.txt'),
        [`# ${siteConfig.title}`, '', `> ${siteConfig.tagline}`, '',
         '## Docs', '',
         ...docs.map((r) => `- [${r}](${origin}${r})`), ''].join('\n'),
      );
    },
  };
}

Register it with plugins: ['./plugins/discovery-files.js'] in docusaurus.config.js, then run npm run build, not npm start, because postBuild fires only on a production build. The link list is derived from the route table, so it cannot drift, and the baseUrl guard turns a silent deployment failure into a failed build. docusaurus-plugin-llms is a community package for the llms.txt half; pin its version.

The baseUrl trap. static/ is served under baseUrl. For a site at https://my-org.com/my-project/, baseUrl is /my-project/, so a valid static/robots.txt answers at /my-project/robots.txt while the origin root returns 404. For llms.txt that is survivable, since the specification allows a subpath file. For robots.txt it is fatal: RFC 9309 section 2.3 says "The rules MUST be accessible in a file named /robots.txt (all lowercase) in the top-level path of the service." On 7 September 2026 neither facebook.github.io nor microsoft.github.io served a robots.txt at all, because the origin root belongs to the GitHub account and no repository can write to it. Three ways out are in the Docusaurus robots.txt guide, and the llms.txt half is in publishing llms.txt on Docusaurus.

Sitemap and feed are already handled. @docusaurus/plugin-sitemap ships in preset-classic and is active only in production, which is why a local check finds nothing. noIndex results in no sitemap, so check it before anything else when sitemap.xml is missing on a live build. The blog plugin defaults feedOptions to RSS and Atom, 20 posts, and injects the autodiscovery tags. Setting feedOptions.type to null is the only way to end up with no feed.

Audit the half that can break.

1. Search for browser-only rendering: grep -rn "BrowserOnly\|useIsBrowser\|typeof window" src/.
2. Build, then check the built HTML: grep -c 'id="__docusaurus"></div>' build/index.html. A match means that page shipped an empty root.
3. Repeat on the routes that matter. Custom landing pages written as React components are where this goes wrong; generated documentation routes almost never do.
4. Strip scripts and count words on the built file. A few dozen words on a page that should have hundreds is the signal.

The identical failure on another stack is written up in React apps that go invisible to crawlers.

Next.js

Verified 20 October 2026: the html-bots source file and the robots.txt metadata reference. First written against Next.js 16.3.3 on 28 August 2026.

Next.js generates robots.txt and the sitemap from code, serves llms.txt from public/, and renders on the server by default. Four of the five discovery files are one file away, and the fifth question, whether the article body is in the HTML at all, decides whether the rest matters. Budget 40 minutes on an App Router project.

Why a Next.js site can pass SEO and still be blank to AI crawlers. The retrieval crawlers read only what the server put in the response. If the content arrives from a useEffect call, a next/dynamic import with ssr: false, or markup behind a typeof window guard, the crawler gets the layout and nothing else. Next.js streams metadata by default and keeps a list of user agents that get blocking metadata instead, HTML_LIMITED_BOT_UA_RE in packages/next/src/shared/lib/router/utils/html-bots.ts. The file read on 20 October 2026 names no AI crawler: GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot are absent, so the framework treats them as browsers. They never execute what they receive.

1. robots.txt with AI bot rules (D1, B2). Create app/robots.ts. Each entry in rules becomes its own User-Agent group:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: '/api/' },
      { userAgent: ['OAI-SearchBot', 'Claude-SearchBot', 'PerplexityBot'], allow: '/' },
      { userAgent: ['GPTBot', 'ClaudeBot', 'CCBot'], allow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

Search bots sit in a separate group from training bots, so changing your mind about training never costs the retrieval index. A bot with its own group never reads the * group, so put everything a named bot needs inside its own block. A static app/robots.txt collides with robots.ts on the same route; delete one.

2. Sitemap (D2). app/sitemap.ts exports an array of URL objects, with lastModified taken from your content. Next.js caches robots and sitemap route handlers by default unless they use a request-time API, so a sitemap built from a CMS stays stale until the next deploy. Set revalidate on the fetch that loads the posts.

3. llms.txt (C2). Put a Markdown file in public/llms.txt: an H1 with the site name, a one-line blockquote, then H2 sections of [title](url): note links. Anything in public/ is served from the root. To generate it from the same data as the sitemap, use a route handler at app/llms.txt/route.ts that returns text/plain. The Next.js llms.txt guide covers the generated version and the llms.txt generator drafts a first file. Ahrefs looked at 137,210 domains in June 2026: 28% publish a valid llms.txt and 97% of those files got zero requests in May, and Google says its search ignores the file. It costs ten minutes and agentic browsers are audited against it.

4. Feed, declared (E5). Return the XML from app/rss.xml/route.ts with Content-Type: text/xml, then declare it in the root layout, because a feed nothing links to is a feed nothing finds:

export const metadata = {
  alternates: {
    types: { 'application/rss+xml': 'https://example.com/rss.xml' },
  },
}

5. Body in the response (C3, E3). 'use client' is not the cause. The Next.js documentation says client components and the RSC payload are both used to prerender HTML. The causes are narrower: data fetched in useEffect, next/dynamic with { ssr: false }, and typeof window guards. Move the fetch into the server component, or into a loading.tsx boundary that renders real content instead of a skeleton. The identical failure on another stack is written up in React apps that go invisible to crawlers.

Pages Router or output: 'export'. The app/ metadata conventions do not exist, so everything becomes a static file in public/: a hand-written robots.txt with the same named groups, llms.txt, a sitemap.xml generated in a prebuild script (or pages/sitemap.xml.tsx with getServerSideProps when not exporting), rss.xml written in the same build step, and the <link rel="alternate"> tag in pages/_document.tsx. Pages that use neither getStaticProps nor getServerSideProps and fetch only on the client ship an empty shell. Those are the ones to audit.

Check it.

curl -s https://example.com/robots.txt | grep -iE 'gptbot|claudebot|perplexity|oai-'
curl -sI https://example.com/sitemap.xml | head -1
curl -sI https://example.com/llms.txt | grep -i content-type
curl -sL https://example.com/ | grep -c -- '<!--$!-->'    # above 0: a Suspense boundary reached the crawler as nothing
curl -sL --compressed https://example.com/ \
| perl -0777 -pe 's/<(script|style|noscript|svg)\b.*?<\/\1>//gsi; s/<!--.*?-->//gs; s/<[^>]+>/ /gs' \
| tr -s '[:space:]' ' ' | wc -w                            # close to what the page shows

Use the Perl version for the word count. A sed 's/<[^>]*>//g' version cannot strip multi-line <script> blocks and counts the JavaScript bundle as prose; on nextjs.org the difference was 626 words against 1,041. If you also run WordPress next to the app, ThinkRank handles robots.txt, schema, sitemaps and llms.txt from one plugin. A purely internal app behind auth needs none of this.

Framer

Measured 4 September 2026 on two live Framer sites. Framer's help articles are rendered by script and could not be re-fetched on 18 October, so those quotes carry their September date.

Framer does more for you than any other hosted builder in this guide, which is why its gaps are easy to miss. It writes a robots.txt and a sitemap for every published site, pre-renders every page, and serves a Markdown copy of each optimized page. What it leaves out is what a default cannot express: which AI crawlers you name (B2), an llms.txt (C2) and a feed (E5). Budget 20 minutes.

Framer's help article on AI agents says: "Although Framer sites are built with React, every page is pre-rendered to HTML on our servers before it is served." The Markdown copies are small. On 4 September, www.framer.com/pricing was 1,470,695 bytes of HTML and 11,651 bytes of Markdown (126 times smaller), and www.linkpreview.app/ was 245,109 against 5,106 (48 times). Both Markdown answers were 200 text/markdown; charset=utf-8.

Why B2 fails on a good default. The generated file for www.linkpreview.app is 72 bytes, rechecked on 18 October: User-agent: *, Allow: /, and a Sitemap: line. Allowing everything is a legitimate decision, but a wildcard says nothing about AI crawlers, and B2 asks whether you stated a preference. Google's robots.txt specification applies only the most specific matching group, so a bot with its own group never reads the * group.

Path 1: upload the files (Pro and Enterprise).

1. Open Site Settings, then Hosting, then Files. Older tutorials point at the Domains dashboard, where these settings used to live.
2. Click Add and upload the file.
3. Set the file path without the file name: / for the domain root, /.well-known for /.well-known/security.txt.
4. Publish.

Supported formats are PDF, TXT, CSV, JSON, XML, MD, VCARD and JS. Because JSON and .well-known paths work, capability files are reachable on Framer's paid plans, which Webflow and Ghost do not allow.

The upload trap. Framer's article states that a custom robots.txt replaces the generated one. It does not merge, and the Sitemap: line goes with the old file. Start from the generated file, not a blank one:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Content-Signal: ai-train=yes, search=yes, ai-input=yes

Sitemap: https://example.com/sitemap.xml

Then upload llms.txt at path /. Framer's own file at www.framer.com/llms.txt (5,900 bytes on 4 September) is a good model: a one-line description, then H2 sections of annotated links pointing at the ?md version of each page. The llms.txt generator writes a starting point, and llms.txt on Framer covers the contents.

Path 2: answer in front of Framer (any plan). Free and Basic sites cannot upload static files. Framer documents reverse-proxy hosting for this, with Cloudflare, CloudFront, Vercel, Netlify and NGINX named.

1. Add a custom domain to the Framer site to act as the origin, for example origin.example.com. Do not use a *.framer.app domain: Framer warns that those are not indexed by search engines.
2. Point the proxy at that origin and add routes for /robots.txt, /llms.txt and your feed, returning static text with the right content type. Pass everything else through untouched.
3. Set the Canonical URL in General settings to the public address.
4. Publish, then request each path on the public domain and confirm the proxy answered.

Do not modify or remove tags inside <head> and do not touch <div id="main">. Adding head tags is supported. This route also covers E5: on 4 September /rss.xml, /feed.xml and /blog/rss.xml all returned 404 on www.framer.com, so a Framer feed is an uploaded XML file you regenerate on publish, or a proxy route that reads your CMS.

Structured data. Schema is neither automatic nor gated. Put a <script type="application/ld+json"> tag in Custom Code in the head. On CMS detail pages, {{Field | json}} escapes a collection field safely, and {{Created}} and {{Updated}} exist on every CMS item, so BlogPosting gets datePublished and dateModified without extra fields. Avoid {{field | unsafeRaw}} unless you control the stored value, since it escapes nothing.

Check it.

curl -sI https://example.com/ | grep -i server-timing     # expect ssg-status;desc="optimized"
curl -s  https://example.com/robots.txt                    # named groups plus a Sitemap: line
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' -H 'Accept: text/markdown' https://example.com/

Framer says Markdown is generated only for optimized pages, so a page whose server-timing header does not read optimized has no Markdown twin and C1 will fail without saying why. The third command should answer 200 text/markdown; charset=utf-8.

Ghost

Measured 31 August 2026 on three live Ghost 6.61 publications, against Ghost's source at version 6.61.1. The source file could not be re-fetched on 18 October; the theme and routing documentation was, and still says what is quoted below.

Ghost gives you server-rendered HTML, a sitemap, an RSS feed and Article schema switched on, then stops at a wall: it has no static file root and no file manager, so every other discovery file arrives through one of two doors. Budget 30 minutes.

A 404 Media article fetched with plain curl on 31 August returned 1,866 words, one <h1> and one application/ld+json block typed Article. Ghost's default robots.txt is 203 bytes and matches platformer.news, 404media.co and thebrowser.com byte for byte:

User-agent: *
Sitemap: {{blog-url}}/sitemap.xml
Disallow: /ghost/
Disallow: /email/
Disallow: /members/api/comments/counts/
Disallow: /r/
Disallow: /webmentions/receive/
Disallow: /.ghost/analytics/api/

The fall-through rule. Ghost serves root-path files from the active theme folder through a middleware called static-theme. It denies the extensions .hbs, .md, .json, .lock and .log (except manifest.json, assetlinks.json and anything under /assets/). The paths /robots.txt, /sitemap.xml, /sitemap.xsl, /llms.txt, /llms-full.txt and the two /.well-known/ llms variants fall through: a theme file wins if present, and Ghost's generated one answers if not. So a discovery file works on Ghost if its extension is not .md or .json, and it is easy if the path is on that list.

Path 1: ship the files in your theme. Ghost's theme documentation says a theme can include a robots.txt "which overrides the default robots.txt provided by Ghost" (rechecked 18 October).

1. Download the active theme as a zip from Ghost admin, or clone it.
2. Find the folder holding index.hbs, post.hbs and package.json.
3. Create robots.txt beside package.json, not in /assets/.
4. Write the whole file. It replaces Ghost's default, so copy the seven lines above, swap {{blog-url}} for your domain, and add your AI groups underneath. Drop the Sitemap: line and you fix B2 while losing D2.
5. Give each crawler its own group and repeat Disallow: and Sitemap: inside it:

User-agent: GPTBot
Disallow: /members/
Allow: /
Sitemap: https://example.com/sitemap.xml

1. Create llms.txt beside it: an H1, a one-line blockquote, then H2 link lists. See publishing llms.txt on Ghost.
2. Re-zip and upload. Ghost runs GScan on upload and rejects fatal errors, so a broken theme cannot take the site down.

Path 2: routes.yaml, when the theme is not yours to edit.

1. In admin, go to Settings, then Labs, and download routes.yaml.
2. Add a route with a custom mime type. The routing documentation says routes generally render HTML but a content_type property overrides it.
3. Use a trailing slash on the route key. Ghost "automatically forces trailing slashes" (rechecked 18 October).
4. Create the matching .hbs template with a {{#get}} query over your posts.
5. Upload the file back through Labs. Uploading applies it at once; editing on disk needs a restart.

routes:
  /llms.txt/:
    template: llms
    content_type: text/plain

{{#get}} reads posts, tags, authors, tiers and newsletters but not pages, so a generated index misses About and Pricing.

The 302 that reads as a pass. Ghost 6.61 registers four llms.txt routes and, when the feature is off, answers them with res.redirect(302, '/'). On 31 August all three publications returned 302 to / on exactly those four paths, while /humans.txt returned a real 404. A redirect-following check lands on the homepage and records 200. Read the final URL and the content type:

curl -sL -o /dev/null -w '%{http_code} %{content_type} %{url_effective}\n' https://example.com/llms.txt
# no file:   200 text/html; charset=utf-8 https://example.com/
# real file: 200 text/plain; charset=utf-8 https://example.com/llms.txt

What Ghost cannot serve. agents.md, /.well-known/mcp/server-card.json, /.well-known/api-catalog, /.well-known/agent-skills/index.json and OAuth metadata all hit the .md and .json denial. On 31 August /agents.md returned a hard 404 and /AGENTS.md 301ed to lowercase before the same 404. Host those on a domain whose web server you control. Appending .md to a post URL returns the post's HTML with a 200, so a C1 test that reads only the status passes a Ghost site with no Markdown at all.

Check it.

SITE=https://yourpublication.com
curl -s "$SITE/robots.txt" | grep -iE 'gptbot|claudebot|perplexitybot|oai-searchbot'
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' "$SITE/sitemap.xml"
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' "$SITE/rss/"
curl -s "$SITE/" | grep -o 'rel="alternate" type="application/rss+xml"'

Hugo

Verified 18 October 2026 against Hugo's configuration, robots.txt and RSS pages. Site measurements taken 6 September 2026.

Hugo owns every byte your site emits, so no plan tier or editor panel decides which files exist. It generates a sitemap and an RSS feed without being asked and generates no robots.txt unless you ask, the inverse of the hosted builders. Budget 30 minutes.

Hugo's configuration reference describes enableRobotsTXT as "Whether to enable generation of a robots.txt file. Default is false." Sitemaps have no switch: Hugo writes sitemap.xml from an embedded template unless you add sitemap to disableKinds. The RSS page says Hugo generates feeds for home, section, taxonomy and term pages. When you do switch robots.txt on, the embedded template outputs only User-agent: *, 13 bytes, with no Sitemap: line and no crawler named.

What 15 live Hugo sites publish (30 candidate domains fingerprinted by the generator tag, 15 confirmed, versions 0.74.0 to 0.165.0, 75 probes, 6 September 2026):

Six of the 11 robots.txt files were the 13 or 14 byte default. One of the four sites with none is gohugo.io itself. Hugo sites lose points on the one file the build leaves to them, not because the build is wrong.

Path 1: the template.

1. Add enableRobotsTXT = true to hugo.toml (enableRobotsTXT: true in YAML).
2. Create layouts/robots.txt. The lookup is /layouts/robots.txt, then /themes/<THEME>/layouts/robots.txt, so a project file overrides a theme.
3. Write the policy:

User-agent: *
Allow: /
Content-Signal: search=yes, ai-train=no, use=reference

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.org/sitemap.xml

1. Run hugo and read public/robots.txt before deploying.

Consecutive User-agent: lines form one group, so stacking names above one rule block gives them all the same rule. Separate blocks with a blank line when you want different treatment. In RFC 9309 the Sitemap: line is in section 2.2.4, "Other Records", an extension that is universally supported. Take crawler tokens from the user-agent list.

Path 2: the static folder. Set enableRobotsTXT = false, put the file in static/robots.txt, and build. Hugo's robots page says it copies everything in static to the root of public. A static file cannot read site variables, but it cannot break the build either, which suits an inherited theme or a site that ships twice a year. Use one route or the other, never both.

Wire the feed Hugo already built. Hugo writes /index.xml at the first build, and its embedded HTML templates do not link it. Of the eight sites serving a valid feed, two had no reference in <head> and one pointed rel="alternate" at the homepage. Hugo's RSS page publishes the fix; put it in the head partial:

{{ with .OutputFormats.Get "rss" }}
  {{ printf `<link rel=%q type=%q href=%q title=%q>` .Rel .MediaType.Type .Permalink site.Title | safeHTML }}
{{ end }}

If the feed moved to /feed.xml by renaming the output format, the tag must follow. To drop the feed on purpose, add rss to disableKinds.

llms.txt. Define a custom output format with baseName = "llms", or write static/llms.txt once by hand. The output-format configuration and the template naming change in v0.146.0 are in llms.txt on Hugo.

Check it.

curl -s https://example.org/robots.txt | wc -c                                 # over about 100 bytes
curl -s https://example.org/robots.txt | grep -icE 'gptbot|claudebot|perplexity'
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.org/sitemap.xml
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.org/index.xml
curl -s https://example.org/ | grep -o '<link[^>]*rss[^>]*>'
curl -s -o /dev/null -w '%{http_code}\n' https://example.org/no-such-path-xyz  # expect 404

A scanner caveat: on 6 September our E5 check missed gohugo.io's working feed because it did not probe /index.xml. Your feed is fine in that case; confirm the head tag exists.

Lovable

Verified 18 October 2026 against Lovable's SEO and TanStack Start upgrade pages. Measurements taken 1 September 2026.

Two Lovable apps can look identical in a browser and differ by a factor of fifty in what an AI crawler receives. The template decides, and the fix is one confirmation dialog. Budget 20 minutes, plus the migration if you need it.

What 15 published Lovable apps returned (fetched twice each on 1 September 2026, with a desktop browser user agent and a GPTBot user agent, scripts stripped before counting):

Every app returned the same word count to both user agents. The stack label is Lovable's own tech_stack field. The sample is one operator's portfolio, so it measures platform behaviour, not how Lovable users configure sites.

Lovable's SEO page says older projects use "on-request pre-rendering on deployed public URLs, served only to verified search and AI crawlers", and that "Third-party SEO scanners and other unverified agents see the regular single-page app." Its upgrade page says TanStack Start "renders each page into finished HTML on the server before sending it. Everyone receives the same complete page, every time." Sending a crawler's user agent does not get you onto the verified list. Vercel and MERJ (17 December 2024) found none of the major AI crawlers execute JavaScript.

Step 1: find your template. Ask the chat "What stack is this project on?", or look for Project settings, then General, then Project actions, then Migrate to TanStack Start. If the option is missing, the project is probably already on TanStack Start.

Step 2, path A: migrate.

1. Type / in the editor chat and pick Migrate to TanStack Start, or use the Project actions menu and select Migrate.
2. Confirm on the card. The cost is "10 to 35 credits, depending on how large your project is."
3. A large project can pause partway; ask it to continue.
4. Review in the preview. The live site does not change until you publish.
5. Publish. A blank page afterwards usually means a browser-only library; tell Lovable which page failed.

To revert, restore the version before the upgrade from version history. The published site is unaffected.

Step 2, path B: stay on React + Vite. Lovable's FAQ says you do not have to upgrade. If you stay: you cannot verify rendering with curl or a scanner (use Google's URL Inspection tool or a link debugger); any crawler outside Lovable's unpublished verified list gets the shell; and the <title>, description and Open Graph tags are in the initial HTML on both templates, so put anything a crawler must read there.

Step 3: add the discovery files, on either template.

Anything in public/ is served from the site root, and dotted directories work. aiscan.site itself serves public/.well-known/mcp/server-card.json and public/.well-known/agent-skills/index.json (checked 1 September). That makes Lovable the one hosted builder in this guide where the capabilities dimension is reachable on any plan. For routes, TanStack Router's file naming uses square brackets to escape special characters: llms[.]txt.ts answers at /llms.txt and a folder named [.well-known] produces /.well-known/.

For B2, name the crawlers. lovable.dev/robots.txt carries separate groups for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended, with a comment that a named group replaces the * group outright.

Content outside the app. If posts live in WordPress or Contentful behind a Lovable front end, the Lovable work does not touch that domain's robots.txt, sitemap or llms.txt. Those are graded separately.

Check it on your custom domain, not the preview host:

URL=https://yoursite.com
curl -sL --compressed "$URL" \
| perl -0777 -pe 's/<(script|style|noscript|svg)\b.*?<\/\1>//gsi; s/<[^>]+>/ /gs' \
| tr -s '[:space:]' ' ' | wc -w         # under 100 means the shell, not the page
curl -sI "$URL" | grep -i x-robots-tag  # noindex means you measured a preview URL
curl -s "$URL/robots.txt" | grep -iE 'gptbot|claudebot|perplexity'
curl -sI "$URL/llms.txt" | head -1

On a React + Vite app the scanner reads the single-page shell by design, so a low C3 there means an unverified client sees nothing, which is not the same as Google seeing nothing.

Replit

Measured 2 September 2026 on twelve published Replit apps. Replit's deployment-types page moved and could not be re-fetched on 20 October, so its quotes carry their September date.

On most platforms a missing robots.txt returns 404 and you find out at once. On a Replit app built by Agent it returns HTTP 200 with your homepage HTML inside. Ten of twelve apps answered 200 for a path that does not exist, and six returned the same HTML shell, byte for byte, for /robots.txt, /sitemap.xml, /llms.txt, /agents.md and a nonsense URL. Budget 15 minutes once you know which deployment target serves the app.

The mechanism. An Agent-built app is a React front end and an Express server. In production the server serves the built static files, then hands everything it did not recognise to React. Replit's generated server/vite.ts ends with a wildcard route that sends index.html for any unmatched path. A browser is fine with that, because the router draws the right page. A crawler asked for /llms.txt, got 200 text/html, and cannot tell the file is absent. E1 fails, C2 and D2 report a file present when nothing is there, and any checker that decides on status code alone records a pass.

Which deployment target serves you. The response headers say so in one request:

curl -sI https://yourapp.replit.app/ | grep -i '^server\|^x-powered-by'

server: Google Frontend with x-powered-by: Express is Autoscale, with your Express server and its catch-all answering. server: nginx with no x-powered-by is Static Deployment, where files come off disk and missing paths return a real 404. The [deployment] block in .replit says the same: deploymentTarget = "autoscale" (older apps read "cloudrun", which maps to Autoscale) or "static" with a publicDir key.

Path 1: Static Deployment. Best for landing pages, portfolios and documentation, and the only target where discovery files behave as on the rest of the web. Replit's deployment-types page, fetched 2 September 2026, said: "Static Deployments are not compatible with Replit Apps created using Agent. Agent builds full-stack apps that need a backend server, so use Autoscale or Reserved VM for those." So this path is open if the project has no backend, or if you split the marketing pages into a second project.

1. Open the Publishing tool, then Adjust settings, then the Deployment type dropdown, and choose Static.
2. Set the public directory to the build output, for a Vite client dist/public, which appears in .replit as publicDir = "dist/public".
3. Put robots.txt, sitemap.xml and llms.txt in the source folder copied into it, client/public/ for Vite.
4. Create a 404.html in the root directory of the Replit App. Per Replit's static configuration reference it renders for paths matching no file or rule.
5. For a specific content type, add a header block to .replit:

[[deployment.responseHeaders]]
path = "/llms.txt"
name = "Content-Type"
value = "text/plain; charset=utf-8"

1. Republish. Changes to .replit do not apply until you do. A real file always wins over a matching [[deployment.rewrites]] rule (Replit calls it shadowing), which keeps robots.txt serving as itself.

Path 2: Autoscale, which is what Agent gives you. Three edits.

1. Put the flat files where the build copies them, client/public/ in an Agent-built Vite project. On visa4.travel that folder held exactly robots.txt, llms.txt and three image files, and all of them served from the root.
2. Register real routes for anything generated, before the catch-all. A sitemap built from a database is a route, not a file:

// server/routes.ts: must be registered BEFORE the wildcard handler
app.get("/sitemap.xml", async (_req, res) => {
  const urls = await buildUrlList();
  res.type("application/xml").send(renderSitemap(urls));
});

1. Give unmatched file-like paths a real 404, after your API and file routes and before the SPA catch-all:

app.use((req, res, next) => {
  if (/\.(txt|xml|json|md)$/.test(req.path)) {
    return res.status(404).type("text/plain").send("Not found");
  }
  next();
});

The third edit is the one nobody makes, and it is the difference between a scanner telling you the truth and one telling you everything is fine.

The AI bot rules. Replit ships no robots.txt of its own, so what you write is the whole file. Named groups replace the wildcard group, so repeat the rules you still want:

User-agent: *
Allow: /
Content-Signal: search=yes, ai-train=no, use=reference

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Sitemap: https://yourapp.com/sitemap.xml

GPTBot collects training data and OAI-SearchBot fetches for search answers, so blocking the first and allowing the second is usually what "no training" means. Reference: bot access checks.

What twelve published Replit apps returned (five paths each plus the homepage, desktop browser user agent, scripts and styles stripped, 2 September 2026; one operator's portfolio, so it measures platform behaviour, not adoption):

The two exceptions are the same app in two forms, both Static Deployments built with Astro. visa4.travel is the instructive middle case: a real robots.txt, a real llms.txt and a working sitemap route, and still eight visible words on the homepage and 200 for a URL that does not exist. Fixing the files does not fix the render.

Check it.

curl -so /dev/null -w '%{http_code} %{content_type} %{size_download}\n' https://yourapp.com/llms.txt
curl -so /dev/null -w '%{http_code} %{content_type} %{size_download}\n' https://yourapp.com/zzz-not-real

Pass is 200 text/plain at your file's size for the first and 404 for the second. Identical numbers on both mean both are the shell. Under about 50 visible words on the homepage means an unauthenticated crawler reads an empty page, which is not the same as Google reading one, because Google renders JavaScript. The file itself is covered in publishing llms.txt on Replit.

Shopify

Verified 20 October 2026: Shopify's robots.txt.liquid documentation and the live robots.txt of allbirds.com. Storefront measurements taken 27 August 2026.

Shopify gives you more for free than any other platform in this guide. Liquid renders every page on Shopify's servers, so C3 and E3 pass. A sitemap, an Atom feed, product JSON-LD and a live /llms.txt ship without code. What it does not give agents is the pages a shopping assistant needs: the default robots.txt contains Disallow: /policies/, which blocks your shipping, refund and terms pages from every crawler, including the ones answering "does this store ship to Canada" inside ChatGPT. On 20 October 2026 allbirds.com/robots.txt still carried that rule, plus /*/policies/. Budget 20 minutes in the theme code editor.

Free: D1 robots.txt, D2 sitemap, C2 llms.txt, C3 and E3 rendered HTML, E5 the Atom feed, M4 machine-readable pricing on any theme using Shopify's structured-data filter. To add: B2 explicit AI bot rules, and the policy-page unblock.

Shopify also serves agent-facing files natively. On 27 August 2026 allbirds.com/llms.txt and /agents.md both returned 200 with content-type: text/markdown; charset=utf-8, and the sitemap index listed sitemap_agentic_discovery.xml, whose only entry is /agents.md. The default llms.txt is Shopify's own text: it describes the Shop skill and the store's Universal Commerce Protocol endpoint at /.well-known/ucp, and says nothing specific about what you sell.

Path 1: the theme code editor. From the admin go to Online Store, then Themes, find the published theme, click ..., then Edit code.

Step 1, take control of robots.txt. No template exists in the theme until you create one. Per Shopify's documentation: right-click the Templates folder, click New File, name it robots.txt.liquid. The template supports six Liquid objects: robots, group, rule, user_agent, sitemap and request. Shopify recommends printing the default groups through Liquid rather than replacing them with static text, because Shopify updates those defaults. To drop only the policies rule:

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {%- unless rule.directive == 'Disallow' and rule.value == '/policies/' -%}
      {{ rule }}
    {%- endunless -%}
  {%- endfor -%}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Step 2, explicit AI bot rules (B2). The default file names no AI crawler. Because a bot with its own group never reads the * group, named groups make your position explicit instead of accidental. Paste this below the Liquid block, as plain text:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes that answer shopping questions. ChatGPT-User and Claude-User fetch a page because a customer asked right now. GPTBot and ClaudeBot collect training data. OpenAI's bot documentation says "each setting is independent of the others", so you can decline training and keep the shopping traffic. Blocking the live-fetch agents is the one move that costs sales. Every token is in the AI crawler user-agent list. Changes are live at once; crawlers cache the file for roughly 24 hours.

Step 3, replace the generic llms.txt (C2). Create templates/llms.txt.liquid the same way. Only the request and agents objects are available inside it, so shop details are written literally:

# Northwind Supply

> Independent outdoor gear. Ships to the US and Canada. Free returns within 60 days.

## Catalogue
- [All products](https://northwind.example/collections/all): full catalogue, prices in USD
- [Size guide](https://northwind.example/pages/sizing): measurements for every model

## Policies
- [Shipping and returns](https://northwind.example/policies/refund-policy)

An H1 naming the project is the only required part of the format Jeremy Howard of Answer.AI proposed on 3 September 2024. The Shopify llms.txt guide covers how llms.txt.liquid, agents.md.liquid and llms-full.txt.liquid inherit from each other.

Step 4, machine-readable prices (M4). Search sections/main-product.liquid for structured_data. In Dawn, Shopify's reference theme, it reads {{ product | structured_data }}, which emits schema.org JSON-LD: a Product for single-variant items, a ProductGroup for variants, with price, priceCurrency and availability inside an Offer. If your theme lacks it, add it inside a <script type="application/ld+json"> tag in the product section. Without it an agent reads the price out of prose and sometimes gets it wrong. Theme upgrades replace section files, and this is the line that quietly goes missing.

Path 2: an app. A Shopify SEO app can write most of this. StoreSEO generates llms.txt and agents.md from the live catalogue and keeps them in step as products change, which is the part manual editing gets wrong over time. A WordPress blog beside the store is a separate surface; ThinkRank manages its robots.txt, schema and llms.txt from one plugin.

Check it.

curl -s https://yourstore.com/robots.txt | grep -iE 'policies|GPTBot|SearchBot|Sitemap'
curl -sI https://yourstore.com/llms.txt | grep -i content-type     # text/markdown; charset=utf-8
curl -sI https://yourstore.com/blogs/news.atom | grep -i content-type  # application/atom+xml
curl -s https://yourstore.com/products/YOUR-PRODUCT | grep -o '"priceCurrency":"[A-Z]*"'

If the last command returns nothing, Step 4 has not taken effect. The scan reports D1, B2, C2, D2, E5 and M4; the Shopify-specific fixes are on the Shopify platform page and the commerce checks. It cannot say whether the llms.txt links are the right links or whether product descriptions answer what buyers ask. It grades machine readability, not merchandising.

Squarespace

Measured 5 September 2026 on two Squarespace-hosted sites. Squarespace's help articles are script-rendered and could not be re-fetched on 20 October, so every quote carries its September or earlier date.

A Squarespace site has no folder you can put a file in, so none of its robots.txt, sitemap or RSS feed can be opened, edited or replaced. Squarespace's sitemap guide answers "Can I edit my sitemap for SEO?" with "No. You can't edit the sitemap or a page's source code." Your AI readiness is decided by two checkboxes, one text field and one code box that three of the four plans include. Budget 25 minutes.

The sitemap updates within an hour of adding a page, with priorities of 1.00 for the homepage, 0.75 for other pages and 0.50 for blog posts, and a sitemap fetched on 5 September carried exactly those values. The cost of that model: there is no upload panel, so /.well-known/mcp/server-card.json, /.well-known/agent-skills/index.json, agents.md and llms-full.txt all returned a hard 404 on www.squarespace.com, and no setting changes that.

Your robots.txt names 26 AI crawlers and gives none of them a rule. The default file opens with # Squarespace Robots Txt and stacks User-agent: lines: AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider and YouBot, then three Google AdsBot agents, then User-agent: *. Consecutive User-agent: lines with no blank line between them form one group, so all of those names share the wildcard's rules. GPTBot receives exactly what every other visitor receives. Squarespace's guide "Request that AI models exclude your site" (updated 17 July 2026) lists the same 26 bots and says the checkbox "updates your robots.txt file to tell the following bots not to crawl your site." The checkbox adds the exclusion; the names are there either way. A checker that greps for GPTBot reports a policy where none exists, so B2 looks for a rule, not a name. The crawler tokens are in the AI crawler user-agent list.

Path 1: the built-in switches, in order. None needs code.

1. Settings, then Crawlers. Two checkboxes, both unchecked by default: "Block Search engine crawlers" and "Block known artificial intelligence crawlers."
2. Decide the AI checkbox on purpose, and usually leave it unchecked. Checking it asks all 26 bots to stay out, with no distinction between training and answering. Squarespace defaults it off because it does not "want to potentially impact your site's traffic by excluding it from chat answers and sources."
3. Turn on llms.txt. SEO/AI Visibility, then SEO Settings, then the LLMS.txt tab. Squarespace's guide (updated 24 July 2026) says "the llms.txt file is disabled by default. To enable it, add text for your llms.txt file to the text field," then Save. It is a text field, not an upload, and only 7.1 sites can do it, so 7.0 sites have no route to C2. The URL behaviour is in publishing llms.txt on Squarespace.
4. Draft the contents elsewhere. The box does no validation, so malformed Markdown saves silently, and Squarespace says it "can't offer specific advice on creating text for your llms.txt file." Draft it in the llms.txt generator and keep the source. www.squarespace.com/llms.txt (2,088 bytes on 5 September) is a good shape: an H1, a one-paragraph blockquote, then H2 sections of [name](url): description bullets.
5. Pages, System Pages, 404 Page: confirm it is still the default.
6. Confirm the feed. Any blog, products, events, gallery or album page has a feed at its own URL with ?format=rss appended.

Path 2: code injection. Squarespace's guide says it is "available in the Core, Plus, Advanced, and some legacy billing plans", so Basic has none. The Header field goes in <head> on every page and the Footer field before </body>.

Squarespace already emits schema: a blog post fetched on 5 September carried Article, BlogPosting, WebPage, Organization and ImageObject across two JSON-LD blocks. Add a block only for what the platform does not model, such as Offer pricing on a services page. Add a <link rel="alternate"> for a second feed if you publish more than one collection, since only the current page's feed is declared. Header injection cannot create a URL, serve /llms-full.txt, answer a /.well-known/ path or change robots.txt.

The 404 trap. Squarespace returns a real 404 by default; five nonexistent paths on www.squarespace.com all returned 404 with one error page. The risk is self-inflicted. Its guide "Creating a custom 404 page" (updated 17 August 2026) warns a custom 404 can trigger a Soft 404 alert in Search Console, because Google sees "a contradiction: an indexable page that says it doesn't exist", and recommends the system default or a noindex tag. One of the two sites probed answered every unknown path with HTTP 200 and a byte-identical copy of the homepage. That is per-site configuration, invisible from the editor.

Check it.

Fetch the homepage the same way and compare byte counts. Identical sizes mean the site answers every request with the homepage, and no discovery file you add will be believed. A scan cannot see which checkbox produced a result, so it will not say the Crawlers panel is untouched. No response measured on 5 September carried a Link: discovery relation, and there is no Markdown alternative: Accept: text/markdown returned text/html and /index.md returned 404. Framer answers both, which is why its section reads so differently.

Webflow

Verified 30 August 2026 against Webflow's help centre and a live webflow.com. The help articles were not re-fetched on 22 October, so every quote carries its August date.

A Webflow site passes the two hardest checks, C3 and E3, before you touch anything, then loses four cheap points at the root path. The Designer compiles to static HTML at publish time, so there is no empty <div id="root"> problem. Fetched with plain curl on 30 August 2026, webflow.com/blog/robots-txt returned 2,722 visible words and two application/ld+json blocks with no hydration. What bites is ownership of the root path: Webflow has no public/ folder, every root file is a switch in Site settings, and Webflow decides which switches exist. Budget 20 minutes.

robots.txt, sitemap.xml and llms.txt need a Site plan or a paid Workspace plan. Two quotes from Webflow's help centre (30 August 2026) shape the order of work. The llms.txt upload article says the file "won't be published to your Webflow staging domain", so you cannot verify it before a production publish. The sitemap article says "it isn't possible to delete the /sitemap.xml page once you've created your sitemap and published your site", so treat that toggle as one-way.

The Designer route. Every step is Save, then Publish; a changed setting is not served until the site is republished.

1. Site settings > SEO > Indexing. Add your robots.txt rules. Webflow adds a link to your sitemap by default, per its robots.txt article; leave that on.
2. Add a named group per AI crawler in the same field. A lone User-agent: * group does not satisfy B2. Decide per bot: GPTBot, ClaudeBot and CCBot collect training data, while OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot fetch pages to answer a question someone is asking now. Blocking the second group removes you from answers.
3. Site settings > SEO > Sitemap. Set Auto-generate sitemap to Yes, Save, Publish. To control what is listed, set it to No and paste XML into Custom sitemap.xml.
4. Site settings > SEO > LLMs.txt > Upload file, then Save changes. The file must be UTF-8 and under 100 KB. The llms.txt generator builds one from a live URL, and llms.txt on Webflow covers what belongs inside.
5. Pages panel, hover the Collection page, click the cog. Under RSS Feed Settings switch Enable RSS 2.0 feed on and fill in Channel Title, Channel Description, Item Title, Item Description and Item Publication Date. Webflow's own feed lives at /blog/rss.xml and returned application/rss+xml on 30 August 2026.
6. Site settings > Custom code > Head code. A feed nothing points at fails E5. Paste <link rel="alternate" type="application/rss+xml" title="Blog" href="https://example.com/blog/rss.xml">, Save, Publish.

The API route, for .well-known. Everything under /.well-known/ goes through one Data API endpoint, PUT /v2/sites/:site_id/well_known, whose reference says it "requires an Enterprise workspace". Files must be .txt, .json or .noext, each "smaller than 100kb", with "less than 30 total files". An API catalog (P1) and an MCP server card (P2) both live there. Off Enterprise those checks are out of reach and no Designer workaround exists, so record that rather than hunting for a plugin.

Webflow Cloud is different. Webflow Cloud deploys an Astro or Next.js app from GitHub at a mount path under your domain, so files for that path follow the framework's rules: see Astro and Next.js. Root-level robots.txt, sitemap.xml and llms.txt still come from Site settings.

Check it.

A 404 for llms.txt on a .webflow.io address proves nothing, because the file is excluded there. If the feed resolves but E5 still fails, the rel="alternate" tag is missing or its href 404s. A scan cannot see your plan, and a robots.txt directive is a request, not enforcement: Webflow gives you no edge rules and no request logs of your own.

Wix

Measured 8 September 2026 on a live Wix-hosted site; click paths read from Wix's help centre the same day and not re-fetched on 22 October. One site, not a population.

Wix has already created your files. Before you open a panel, a Wix site serves a robots.txt, a sitemap index, server-rendered HTML and, on paid plans, a generated llms.txt. The work is understanding what happens the first time you overrule one. There is no static file root, and the only code path that answers a public URL is a Velo HTTP function at <baseUrl>/_functions/<functionName>, so every root path is off limits by construction. Budget 10 minutes.

The four switches live in SEO & GEO in the dashboard, under Tools and settings, and they do not un-switch alike:

Wix says of llms.txt: "Once you edit the file, it will stop updating automatically so your changes are preserved." Add a product afterwards and the file will not learn about it. Robots.txt is the opposite: it is regenerated after a page is published even after an edit. On NLWeb, the help centre says it is "not currently possible to disconnect your site from NLWeb after installation". llms.txt also needs a paid plan, a connected custom domain and search indexing enabled, so a free wixsite.com site has none and a 404 there is a billing question, not a bug.

1. Open SEO & GEO, scroll to Tools and settings.
2. Robots.txt Editor > View File. Add directives under This is your current file, then Save Changes and Save. This is the only place on Wix to name an AI crawler (B2). Naming a bot only matters if it gets a group of its own, which the Wix robots.txt guide covers; the group-matching rule is RFC 9309 section 2.2.1.
3. To undo a bad edit, return to the same screen and click Reset to Default, then Reset.
4. Go to llms.txt > View File. Read it before touching it. A missing button means a precondition is unmet. To edit: More Actions > Edit file > Continue, change the text, Save Changes. The llms.txt generator drafts a replacement, and llms.txt on Wix has the platform detail.
5. NLWeb > Install NLWeb only once you want it permanently. To pause agent querying, switch off the ASK toggle.
6. For structured data, open a page's SEO Settings and click +Add New Markup under Structured Data Markup. Wix takes JSON-LD only, up to five markups per page, each under 7,000 characters, on static pages only.

Check it.

The first-party probe, 8 September 2026: /robots.txt 200, 2,797 B, one group, 87 Disallow lines, one Sitemap:; /llms.txt 200, 36,670 B; /sitemap.xml 200, 2,213,477 B. C3 and E3 pass without configuration, at about 1,053 bytes of HTML per visible word. A scan cannot tell a maintained llms.txt from one frozen six months ago, because they look identical over HTTP, and it cannot see whether NLWeb is installed.

WordPress

Updated August 2026; menu paths and plugin versions are as of that month and were not re-fetched on 22 October.

A default WordPress install passes three checks without help: it ships server-rendered HTML, a real XML sitemap and a declared RSS feed. The gap is narrower than "your site is invisible to AI" suggests, and on most installs it is four items you can close in about 40 minutes without PHP.

Step 0: find out which robots.txt you serve. WordPress keeps no robots.txt on disk; it generates one on request, and code can change it through the robots_txt filter (WordPress 3.0). But the web server checks the filesystem first. If Apache or Nginx finds a real file at /robots.txt, the request never reaches PHP and every rule your SEO plugin publishes does nothing. Run curl -s https://yoursite.com/robots.txt and compare it with the plugin's preview. If they differ, look next to wp-config.php over SFTP and either delete the file or commit to editing it directly. Pick one owner. Across Cloudflare's top 200,000 domains, 78% have a robots.txt but only 4% declare any AI usage preference.

Path A, the plugin route (about 20 minutes).

1. Take control of robots.txt. In Rank Math, switch on Advanced Mode, then Rank Math SEO > General Settings > Edit robots.txt. In Yoast, Yoast SEO > Tools > File editor. If Yoast's File editor is missing, DISALLOW_FILE_EDIT is true in wp-config.php or the file is not writable; edit over SFTP. Rank Math refuses while a physical file exists, and its documentation says to remove that file first.
2. Add named AI bot rules. An explicit baseline that stays readable by answer engines and opts out of training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

GPTBot, ClaudeBot and Google-Extended are training crawlers; OAI-SearchBot and PerplexityBot fetch for answers, so blocking them removes your citations. See Cloudflare's three crawler categories.

1. Add Content Signals. They ride inside a user-agent group as a Content-Signal: line, in the form Cloudflare's managed file writes:

User-agent: *
Content-Signal: search=yes, ai-train=no, use=reference
Allow: /

The signals are search, ai-input (RAG and grounding) and ai-train. The AI Content Signals plugin (slug ai-content-signals, version 1.3.2, updated 20 August 2026) writes them to both physical and virtual robots.txt and sends the matching X-Robots-Tag. Weigh it honestly: Google's John Mueller has said no crawler he knows of uses content-signal directives, and Cloudflare's documentation says compliance "is voluntary".

1. Generate llms.txt. ThinkRank AI SEO (slug thinkrank, version 2.0.2, WordPress 6.0 or later) handles robots.txt, robots meta, schema and llms.txt from one plugin and migrates settings from Rank Math, Yoast, All in One SEO and SEOPress. It is our own plugin, so weigh that. If you only want llms.txt, Website LLMs.txt (slug website-llms-txt, updated 18 August 2026) integrates with Yoast, Rank Math, SEOPress and AIOSEO, with settings under Settings > LLMs.txt. Or upload a file from our llms.txt generator by hand (Path B).
2. Confirm your sitemap. Core serves /wp-sitemap.xml (switchable with wp_sitemaps_enabled); Yoast and Rank Math serve /sitemap_index.xml. Request both, see which returns 200, and list that one in robots.txt.

Path B, the manual route (about 20 minutes, one file). Create wp-content/mu-plugins/ai-readiness.php. Must-use plugins load automatically and survive a theme switch, which functions.php does not.

<?php
/**
 * Plugin Name: AI Readiness Rules
 */
add_filter( 'robots_txt', function ( $output ) {
    $rules  = "\n# AI crawler rules\n";
    $rules .= "User-agent: GPTBot\nDisallow: /\n\n";
    $rules .= "User-agent: ClaudeBot\nDisallow: /\n\n";
    $rules .= "User-agent: OAI-SearchBot\nAllow: /\n\n";
    $rules .= "Sitemap: " . home_url( '/wp-sitemap.xml' ) . "\n";
    return $output . $rules;
}, 10, 1 );

add_action( 'wp_head', function () {
    printf(
        '<link rel="describedby" href="%s">' . "\n",
        esc_url( home_url( '/llms.txt' ) )
    );
} );

add_action( 'send_headers', function () {
    header( 'Link: <' . home_url( '/llms.txt' ) . '>; rel="describedby"', false );
} );

The send_headers hook also satisfies D3, a Link header for discovery. Upload llms.txt to the web root beside wp-config.php; it is a static file. Leave the feed alone, since feed_links() has hooked to wp_head by default since WordPress 2.8.

Check it.

Headless WordPress, a REST backend behind a React or Next.js front end, is a different problem: C3 and E3 then depend on that front end rendering on the server (see Next.js). The Interactivity API (WordPress 6.5) stays server-rendered. A five-page brochure site can skip llms.txt, and a site whose host manages robots.txt at the CDN should make these choices there. Ahrefs found 28% of roughly 137,000 domains publish a valid llms.txt and 97% of those files received zero requests in May 2026; Google has said in writing that Search ignores it. Publish one because directed tools and agentic browsers can use it. Re-run the checks after any host migration, CDN change or security-plugin install, and note that Cloudflare's managed robots.txt prepends to yours, so you can end up with two contradicting User-agent: * groups. Cloudflare's crawler defaults were scheduled to change on 15 September 2026 (not re-checked since): read what the change meant for Googlebot.

Confirm it with a scan

Run the scan. It is free, needs no account, and covers every check in this guide in one pass:

npx aiscan-cli yoursite.com

A URL pasted into the scanner does the same without a terminal. Results sit under discoverability for D1 and D2, bot access for B2, and content for C2, C3, E3 and E5. Each row prints its evidence string, so you see the actual fetch.

What a scan cannot tell you is whether your policy is the one you meant, whether the sitemap lists every page, or whether the text answers a reader's question. A perfect score means an agent can reach and parse your site, not that it will find your answer worth citing.

Next steps

Fix the files the scan names, in checklist order, then re-scan. Re-run it after any change to your adapter, host redirect rules, trailingSlash or baseUrl, since those break discovery files without touching the code that generates them. Every platform guide is indexed at aiscan.site/guides.