---
title: "AI readiness setup guide by platform (2026)"
slug: ai-readiness-setup-guide
published: 2026-10-17T13:09:33.080532+00:00
updated: 2026-10-18T03:08:46.941507+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: platform:astro, platform:docusaurus, platform:framer, platform:ghost, platform:hugo, platform:lovable, check:D1, check:D2, check:B2, check:C2, check:C3, check:E3, check:E5, AI readiness, robots.txt, llms.txt, setup guide, Framer, Ghost, Hugo, Lovable, check:C1, check:E1
description: "Fix AI readiness on Astro, Docusaurus, Hugo, Framer, Ghost and Lovable: robots.txt, sitemap, feed, llms.txt and rendering, with the traps that hide them."
url: https://aiscan.site/blog/ai-readiness-setup-guide
---

**Verified 17 and 18 October 2026.** Updated on 18 October 2026.

An AI-ready site needs six things: a robots.txt that names AI crawlers (D1, B2), a sitemap (D2), a declared feed (E5), an llms.txt (C2), and server-rendered HTML that holds the article text (C3, E3). Which of those your platform writes for you decides the work. Static-first frameworks such as Astro, Docusaurus and Hugo generate the sitemap and the feed but no AI-aware robots.txt or llms.txt, hosted builders such as Framer and Ghost write some files and block others, and each platform has one trap that can hide the whole set from crawlers.

> **TL;DR**
> - Every platform gets the same six checks; only the missing files differ.
> - **Astro:** HTML is static by default, so C3 and E3 pass unless you wrap content in `client:only`. Extension-bearing endpoints 404 behind a trailing slash.
> - **Docusaurus:** sitemap and feed come free, robots.txt and llms.txt do not, and a route can ship an empty `__docusaurus` div. A `baseUrl` subpath makes robots.txt void.
> - **Hugo:** robots.txt is off until `enableRobotsTXT = true`, and the default is 13 bytes naming nobody. 0 of 15 live Hugo sites named an AI crawler.
> - **Framer:** robots.txt, sitemap and a Markdown twin come free; a custom robots.txt upload replaces the generated one, `Sitemap:` line included.
> - **Ghost:** only theme files and a `routes.yaml` route reach the root, `.md` and `.json` never do, and a disabled llms.txt answers 302, not 404.
> - **Lovable:** TanStack Start renders for everyone; React + Vite renders only for crawlers Lovable verifies. 7 of 9 Vite apps returned under 50 words.
> - Run `npx aiscan-cli yoursite.com` first. It names the failing rows in one pass.

## Pick your platform

| Platform | Section | Time | Status |
|---|---|---|---|
| Astro | [Astro](#astro) | 25 min | in this guide |
| Docusaurus | [Docusaurus](#docusaurus) | 30 min | in this guide |
| Next.js | [Next.js setup](https://aiscan.site/blog/ai-readiness-setup-nextjs) | | full guide |
| Hugo | [Hugo](#hugo) | 30 min | in this guide |
| Ghost | [Ghost](#ghost) | 30 min | in this guide |
| Framer | [Framer](#framer) | 25 min | in this guide |
| Lovable | [Lovable](#lovable) | 20 min | in this guide |
| Replit | [Replit setup](https://aiscan.site/blog/ai-readiness-setup-replit) | | full guide |
| Shopify | [Shopify setup](https://aiscan.site/blog/ai-readiness-setup-shopify) | | full guide |
| Squarespace | [Squarespace setup](https://aiscan.site/blog/ai-readiness-setup-squarespace) | | full guide |
| Webflow | [Webflow setup](https://aiscan.site/blog/ai-readiness-setup-webflow) | | full guide |
| Wix | [Wix setup](https://aiscan.site/blog/ai-readiness-setup-wix) | | full guide |
| WordPress | [WordPress setup](https://aiscan.site/blog/ai-readiness-setup-wordpress) | | full guide |

Seven platforms are folded in so far. The rest move into this page one run at a time, and their separate guides stay live until each one does.

## What every platform shares

The retrieval crawlers behind ChatGPT, Claude and Perplexity do not run JavaScript. Per [Vercel and MERJ, 17 December 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler): "none of the major AI crawlers currently render JavaScript," covering GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. So the HTML your server sends is the page, as far as they are concerned.

The six checks, which the sections below refer to by ID:

| Check | What it tests |
|---|---|
| D1 | `/robots.txt` exists and says something |
| B2 | at least one AI crawler is named in its own group |
| D2 | a sitemap resolves and parses |
| C2 | `/llms.txt` has an H1, sections and Markdown links |
| C3, E3 | the served HTML carries headings and body text |
| E5 | an RSS or Atom feed is declared in `<head>` |

Naming crawlers matters more than the value you give them. Google's robots.txt specification says only the most specific matching group applies, so a bot with its own group never reads your `*` group. Consecutive `User-agent` lines form one group, so put a blank line between groups that need different rules. Take the tokens from the [current crawler list](https://aiscan.site/blog/ai-crawler-user-agent-list-2026) rather than from memory, because operators rename them.

An empty file does not count either. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) section 2.2.1 says: "If no group matches the product token and there is no group with a user-agent line with the `*` value, or no groups are present at all, no rules apply." The robots.txt rules must also sit at the top-level path of the origin, so a file served only under a subpath is void.

## Astro

*Verified 17 October 2026 against the Astro docs. First written against Astro 7.2.9 on 29 August 2026.*

Astro ships static HTML by default, so C3 and E3 usually pass before you touch anything. It does not create the discovery files an agent looks for, and one routing rule can break three of them at once. Budget 25 minutes.

| Goal | What to add on Astro | Check | Time |
|---|---|---|---|
| A robots.txt exists | `public/robots.txt`, or `src/pages/robots.txt.ts` | D1 | 3 min |
| AI crawlers get their own rules | Named `User-agent` groups | B2 | 5 min |
| Agents can enumerate pages | `npx astro add sitemap`, plus `site` in `astro.config.mjs` | D2 | 5 min |
| A feed exists and is declared | `@astrojs/rss` at `src/pages/rss.xml.js`, plus `<link rel="alternate">` | E5 | 7 min |
| A machine-readable summary | `src/pages/llms.txt.ts` built from your content collections | C2 | 5 min |
| Real content in the HTML | Avoid `client:only` on article text | C3, E3 | 0 min if clean |

Astro's own documentation site shows there is no upstream example to copy. On 17 October 2026 `docs.astro.build/robots.txt` still returns two lines, `User-agent: *` and `Allow: /`, with no AI crawler group and no `Sitemap:` line, and `docs.astro.build/llms.txt` still returns 404.

**Rendering.** Astro's islands architecture renders most of the page to static HTML and adds JavaScript only where needed ([Astro docs, islands](https://docs.astro.build/en/concepts/islands/)), so a React component inside an Astro page is rendered to HTML at build time. One directive undoes that. Per Astro's template directives reference, `client:only={string}` "skips HTML server rendering, and renders only on the client." A component marked `client:only="react"` leaves an empty element in the built HTML. Grep first:

```bash
grep -rn 'client:only' src/
```

**1. Set your deployed URL.** Add `site: 'https://example.com'` to `astro.config.mjs`. Three of the four files need it for absolute URLs.

**2. Write robots.txt.** A plain `public/robots.txt` works. Generating it from `site` keeps the sitemap line from drifting. Create `src/pages/robots.txt.ts`:

```ts
import type { APIRoute } from 'astro';

const getRobotsTxt = (sitemapURL: URL) => `User-agent: *
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Sitemap: ${sitemapURL.href}
`;

export const GET: APIRoute = ({ site }) => {
  const sitemapURL = new URL('sitemap-index.xml', site);
  return new Response(getRobotsTxt(sitemapURL));
};
```

To permit retrieval but refuse training, give OAI-SearchBot, Claude-SearchBot and PerplexityBot `Allow: /`, and GPTBot, ClaudeBot, Google-Extended and Applebot-Extended `Disallow: /`.

**3. Add the sitemap.** Run `npx astro add sitemap`. The build writes `sitemap-index.xml` and `sitemap-0.xml`. It cannot generate entries for dynamic routes in SSR mode, and nothing links the file for you, so add `<link rel="sitemap" href="/sitemap-index.xml" />` to your layout head.

**4. Add the feed and declare it.** Install `@astrojs/rss`, export a `GET` from `src/pages/rss.xml.js` that returns `rss({ title, description, site: context.site, items })`, and build `items` from `getCollection()`. E5 fails on the declaration far more often than on the feed, so add this to the head:

```astro
<link rel="alternate" type="application/rss+xml" title="Your Site" href={new URL("rss.xml", Astro.site)} />
```

The RSS helper adds trailing slashes to links by default. If your config sets `trailingSlash: "never"`, pass `trailingSlash: false` to `rss()`.

**5. Add llms.txt.** An endpoint at `src/pages/llms.txt.ts` can call `getCollection()` and emit an H1, `##` sections and Markdown links from the same source your pages use. The [llms.txt generator](https://aiscan.site/llms-txt-generator) drafts one from a live URL. Starlight docs sites can use the `starlight-llms-txt` plugin instead. No Astro plugin writes robots.txt with AI crawler groups, so step 2 stays manual.

**The trailing-slash trap.** The file exists, the build succeeds, and the URL still 404s. From Astro's endpoints guide, checked 17 October 2026: endpoints whose URLs include a file extension "can only be accessed without a trailing slash (e.g. `/sitemap.xml`), regardless of your `build.trailingSlash` configuration." Every file above is such an endpoint. `/robots.txt/`, `/rss.xml/` and `/llms.txt/` return 404 while the slashless forms return 200. If your host or middleware appends a trailing slash, it hands every crawler a 404 for your whole discovery layer. Test the slashless URL and exempt those paths from any slash-adding rule.

**Check it.**

```bash
curl -sI https://example.com/robots.txt | head -1        # expect 200
curl -s  https://example.com/robots.txt | grep -i gptbot # expect a match
curl -sI https://example.com/sitemap-index.xml | head -1 # expect 200
curl -s  https://example.com/ | grep -o 'rel="alternate"[^>]*rss[^>]*'
curl -s  https://example.com/ | grep -c '<h1'            # expect 1
```

A single-page Astro marketing site needs neither a sitemap nor a feed. Do step 2 and stop. If the front end reads from a headless WordPress CMS, that CMS domain has its own robots.txt, and the work above does not touch it.

## Docusaurus

*Verified 17 October 2026 against the Docusaurus docs. Site measurements taken 7 September 2026.*

Docusaurus runs a large share of developer documentation, which is the material retrieval systems reach for. The discovery files are the easy half. The hard half is whether a given page contains any words, and Docusaurus says so itself. Budget 30 minutes.

| Check | What Docusaurus gives you | What you do |
|---|---|---|
| D1 robots.txt | Nothing | Put one in `static/`, or write it in `postBuild` |
| D2 sitemap | `sitemap.xml` via `preset-classic` | Nothing, unless `noIndex` is on |
| B2 AI bot rules | Nothing | Named groups in your robots.txt |
| C2 llms.txt | Nothing in core | `static/llms.txt`, or generate it from real routes |
| C3 / E3 content | Static HTML per route, if the route renders on the server | Audit the routes that do not |
| E5 feed | RSS and Atom plus the head tags, if you run the blog plugin | Nothing |
| E1 real 404 | A real 404 on every host measured | Nothing |

**Why a page can be empty and still be "static".** Docusaurus is a React single-page application that emits static HTML as an optimisation. Its Static Site Generation page, rechecked 17 October 2026, says: "Docusaurus is ultimately a single-page application, so static site generation is only an optimization (progressive enhancement, as it's called), but our functionality does not fully depend on those HTML files." A route that escapes server rendering ships `<div id="__docusaurus"></div>` empty.

On 7 September 2026 we fetched the Cypress documentation home page with a desktop browser user agent. It returned 6,826 bytes with that empty div, `data-has-hydrated="false"` and two words of visible text. Our scanner grades it 72, Level 4, with C3 and E3 failing on "2 words of server-rendered text, 0 `<h2>`, 0 `<h3>`". The same site's robots.txt names eight AI crawlers. The failure was one route, not the site: two other documentation pages on that host returned 2,196 and 3,897 words. On Docusaurus, C3 and E3 are a per-route outcome.

**What 15 live Docusaurus sites publish** (30 candidate documentation domains fingerprinted, 15 confirmed, 6 paths each, 90 requests, 7 September 2026):

| Surface | Result across 15 sites |
|---|---|
| Generator meta tag | 15 of 15 |
| `/sitemap.xml` returns 200 | 15 of 15 |
| Real 404 for an unknown `.txt` path | 15 of 15 |
| `/robots.txt` returns 200 | 9 of 15 |
| robots.txt naming any AI crawler | 1 of 15 |
| `/llms.txt` returns 200 | 6 of 15 |
| `/blog/rss.xml` returns 200 | 9 of 15 |
| Feed declared in `<head>` | 9 of 9 |

Four of the nine robots.txt files are byte-identical at 1,248 bytes and contain no `User-agent`, `Allow`, `Disallow` or `Sitemap` line, only a licence preamble. Under RFC 9309 that means no rules apply. The one site that names AI crawlers stacks all eight into a single group with `*`, so no named agent gets a rule of its own. That is harmless when everything is allowed, and wrong the moment the rules differ.

**Path 1: the static folder.** Docusaurus copies the contents of `static/` into the root of the build output, with the directory structure preserved.

1. Create `static/robots.txt` with real groups separated by blank lines and a `Sitemap:` line:

```
User-agent: *
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /

Sitemap: https://example.org/sitemap.xml
```

2. Add `static/llms.txt` with an H1, a blockquote summary and H2 sections of `[title](url)` links. The [llms.txt generator](https://aiscan.site/llms-txt-generator) writes the skeleton.
3. For `.well-known` files, `static/.well-known/` works. On GitHub Pages, add an empty `static/.nojekyll`.
4. Run `npm run build` and confirm the files landed in `build/`.

**Path 2: generate the files from your routes.** The `postBuild(props)` plugin hook runs when a production build finishes and receives `siteConfig`, `outDir`, `baseUrl` and `routesPaths`. Create `plugins/discovery-files.js`:

```js
import fs from 'node:fs/promises';
import path from 'node:path';

export default function discoveryFiles() {
  return {
    name: 'discovery-files',
    async postBuild({siteConfig, outDir, baseUrl, routesPaths}) {
      if (baseUrl !== '/') {
        throw new Error(
          `robots.txt is void under baseUrl "${baseUrl}". ` +
          'Deploy at a domain root, or remove the robots.txt half.'
        );
      }
      const origin = siteConfig.url.replace(/\/$/, '');
      await fs.writeFile(
        path.join(outDir, 'robots.txt'),
        `User-agent: *\nAllow: /\n\nSitemap: ${origin}/sitemap.xml\n`,
      );
      const docs = routesPaths.filter((r) => r.startsWith('/docs/'));
      await fs.writeFile(
        path.join(outDir, 'llms.txt'),
        [`# ${siteConfig.title}`, '', `> ${siteConfig.tagline}`, '',
         '## Docs', '',
         ...docs.map((r) => `- [${r}](${origin}${r})`), ''].join('\n'),
      );
    },
  };
}
```

Register it with `plugins: ['./plugins/discovery-files.js']` in `docusaurus.config.js`, then run `npm run build`, not `npm start`, because `postBuild` fires only on a production build. The link list is derived from the route table, so it cannot drift, and the `baseUrl` guard turns a silent deployment failure into a failed build. `docusaurus-plugin-llms` is a community package for the llms.txt half; pin its version.

**The baseUrl trap.** `static/` is served under `baseUrl`. For a site at `https://my-org.com/my-project/`, `baseUrl` is `/my-project/`, so a valid `static/robots.txt` answers at `/my-project/robots.txt` while the origin root returns 404. For llms.txt that is survivable, since the specification allows a subpath file. For robots.txt it is fatal: RFC 9309 section 2.3 says "The rules MUST be accessible in a file named `/robots.txt` (all lowercase) in the top-level path of the service." On 7 September 2026 neither `facebook.github.io` nor `microsoft.github.io` served a robots.txt at all, because the origin root belongs to the GitHub account and no repository can write to it. Three ways out are in the [Docusaurus robots.txt guide](https://aiscan.site/blog/robots-txt-ai-bots-docusaurus), and the llms.txt half is in [publishing llms.txt on Docusaurus](https://aiscan.site/blog/llms-txt-docusaurus).

**Sitemap and feed are already handled.** `@docusaurus/plugin-sitemap` ships in `preset-classic` and is active only in production, which is why a local check finds nothing. `noIndex` results in no sitemap, so check it before anything else when `sitemap.xml` is missing on a live build. The blog plugin defaults `feedOptions` to RSS and Atom, 20 posts, and injects the autodiscovery tags. Setting `feedOptions.type` to `null` is the only way to end up with no feed.

**Audit the half that can break.**

| What you see | What it means | Where to look |
|---|---|---|
| E3 reports a handful of words | That route escaped server rendering | The page component, for `<BrowserOnly>` or a `window` reference |
| `sitemap.xml` returns 404 on a live build | `noIndex` is on, or the preset was replaced | `docusaurus.config.js` |
| `robots.txt` 404s at the origin root | Deployed under a `baseUrl` subpath | The deployment target |
| `llms.txt` is 200 locally, 404 in CI | `postBuild` did not run | Use `docusaurus build` |

1. Search for browser-only rendering: `grep -rn "BrowserOnly\|useIsBrowser\|typeof window" src/`.
2. Build, then check the built HTML: `grep -c 'id="__docusaurus"></div>' build/index.html`. A match means that page shipped an empty root.
3. Repeat on the routes that matter. Custom landing pages written as React components are where this goes wrong; generated documentation routes almost never do.
4. Strip scripts and count words on the built file. A few dozen words on a page that should have hundreds is the signal.

The identical failure on another stack is written up in [React apps that go invisible to crawlers](https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers).

## Framer

*Measured 4 September 2026 on two live Framer sites. Framer's help articles are rendered by script and could not be re-fetched on 18 October, so those quotes carry their September date.*

Framer does more for you than any other hosted builder in this guide, which is why its gaps are easy to miss. It writes a robots.txt and a sitemap for every published site, pre-renders every page, and serves a Markdown copy of each optimized page. What it leaves out is what a default cannot express: which AI crawlers you name (B2), an llms.txt (C2) and a feed (E5). Budget 20 minutes.

| Check | A fresh Framer site | What you do |
|---|---|---|
| D1 robots.txt, D2 sitemap | Generated on publish, sitemap declared | Nothing |
| C3, E3 server-rendered HTML | Pre-rendered before it is served | Nothing |
| C1 Markdown | Passes: `Accept: text/markdown` and `?md` return real Markdown | Confirm the page is optimized |
| E1 real 404 | Real 404 on unknown paths | Nothing |
| B2 AI crawler rules | Fails: the default file names no AI crawler | Upload a custom robots.txt |
| C2 llms.txt | Fails: `/llms.txt` returns 404 | Upload the file, or serve it in front |
| E5 feed | No native feed | Upload an XML file, or generate it upstream |

Framer's help article on AI agents says: "Although Framer sites are built with React, every page is pre-rendered to HTML on our servers before it is served." The Markdown copies are small. On 4 September, `www.framer.com/pricing` was 1,470,695 bytes of HTML and 11,651 bytes of Markdown (126 times smaller), and `www.linkpreview.app/` was 245,109 against 5,106 (48 times). Both Markdown answers were `200 text/markdown; charset=utf-8`.

**Why B2 fails on a good default.** The generated file for `www.linkpreview.app` is 72 bytes, rechecked on 18 October: `User-agent: *`, `Allow: /`, and a `Sitemap:` line. Allowing everything is a legitimate decision, but a wildcard says nothing about AI crawlers, and B2 asks whether you stated a preference. Google's robots.txt specification applies only the most specific matching group, so a bot with its own group never reads the `*` group.

**Path 1: upload the files (Pro and Enterprise).**

1. Open **Site Settings**, then **Hosting**, then **Files**. Older tutorials point at the Domains dashboard, where these settings used to live.
2. Click **Add** and upload the file.
3. Set the file path without the file name: `/` for the domain root, `/.well-known` for `/.well-known/security.txt`.
4. Publish.

Supported formats are PDF, TXT, CSV, JSON, XML, MD, VCARD and JS. Because JSON and `.well-known` paths work, capability files are reachable on Framer's paid plans, which Webflow and Ghost do not allow.

**The upload trap.** Framer's article states that a custom robots.txt replaces the generated one. It does not merge, and the `Sitemap:` line goes with the old file. Start from the generated file, not a blank one:

```
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Content-Signal: ai-train=yes, search=yes, ai-input=yes

Sitemap: https://example.com/sitemap.xml
```

Then upload `llms.txt` at path `/`. Framer's own file at `www.framer.com/llms.txt` (5,900 bytes on 4 September) is a good model: a one-line description, then H2 sections of annotated links pointing at the `?md` version of each page. The [llms.txt generator](https://aiscan.site/llms-txt-generator) writes a starting point, and [llms.txt on Framer](https://aiscan.site/blog/llms-txt-framer) covers the contents.

**Path 2: answer in front of Framer (any plan).** Free and Basic sites cannot upload static files. Framer documents reverse-proxy hosting for this, with Cloudflare, CloudFront, Vercel, Netlify and NGINX named.

1. Add a custom domain to the Framer site to act as the origin, for example `origin.example.com`. Do not use a `*.framer.app` domain: Framer warns that those are not indexed by search engines.
2. Point the proxy at that origin and add routes for `/robots.txt`, `/llms.txt` and your feed, returning static text with the right content type. Pass everything else through untouched.
3. Set the Canonical URL in General settings to the public address.
4. Publish, then request each path on the public domain and confirm the proxy answered.

Do not modify or remove tags inside `<head>` and do not touch `<div id="main">`. Adding head tags is supported. This route also covers E5: on 4 September `/rss.xml`, `/feed.xml` and `/blog/rss.xml` all returned 404 on `www.framer.com`, so a Framer feed is an uploaded XML file you regenerate on publish, or a proxy route that reads your CMS.

**Structured data.** Schema is neither automatic nor gated. Put a `<script type="application/ld+json">` tag in **Custom Code** in the head. On CMS detail pages, `{{Field | json}}` escapes a collection field safely, and `{{Created}}` and `{{Updated}}` exist on every CMS item, so `BlogPosting` gets `datePublished` and `dateModified` without extra fields. Avoid `{{field | unsafeRaw}}` unless you control the stored value, since it escapes nothing.

**Check it.**

```bash
curl -sI https://example.com/ | grep -i server-timing     # expect ssg-status;desc="optimized"
curl -s  https://example.com/robots.txt                    # named groups plus a Sitemap: line
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' -H 'Accept: text/markdown' https://example.com/
```

Framer says Markdown is generated only for optimized pages, so a page whose `server-timing` header does not read `optimized` has no Markdown twin and C1 will fail without saying why. The third command should answer `200 text/markdown; charset=utf-8`.

## Ghost

*Measured 31 August 2026 on three live Ghost 6.61 publications, against Ghost's source at version 6.61.1. The source file could not be re-fetched on 18 October; the theme and routing documentation was, and still says what is quoted below.*

Ghost gives you server-rendered HTML, a sitemap, an RSS feed and `Article` schema switched on, then stops at a wall: it has no static file root and no file manager, so every other discovery file arrives through one of two doors. Budget 30 minutes.

| Check | Ghost's default | What you do |
|---|---|---|
| D1 robots.txt | Present, generated by Ghost | Nothing |
| B2 AI crawlers | None named | Ship a `robots.txt` in your theme |
| D2 sitemap | Generated and linked from robots.txt | Nothing |
| E5 feed | `/rss/` with `rel="alternate"` in the head | Nothing |
| C3, E3 body copy | Handlebars renders on the server | Nothing |
| C2 llms.txt | Route exists and redirects | Theme file, or a `routes.yaml` route |
| C1 `.md` | Returns HTML with a 200 (a false pass) | Nothing fixable in admin |
| P1 to P4 `.well-known` JSON | Blocked by the theme file filter | Not reachable on Ghost |

A 404 Media article fetched with plain curl on 31 August returned 1,866 words, one `<h1>` and one `application/ld+json` block typed `Article`. Ghost's default robots.txt is 203 bytes and matches platformer.news, 404media.co and thebrowser.com byte for byte:

```
User-agent: *
Sitemap: {{blog-url}}/sitemap.xml
Disallow: /ghost/
Disallow: /email/
Disallow: /members/api/comments/counts/
Disallow: /r/
Disallow: /webmentions/receive/
Disallow: /.ghost/analytics/api/
```

**The fall-through rule.** Ghost serves root-path files from the active theme folder through a middleware called `static-theme`. It denies the extensions `.hbs`, `.md`, `.json`, `.lock` and `.log` (except `manifest.json`, `assetlinks.json` and anything under `/assets/`). The paths `/robots.txt`, `/sitemap.xml`, `/sitemap.xsl`, `/llms.txt`, `/llms-full.txt` and the two `/.well-known/` llms variants fall through: a theme file wins if present, and Ghost's generated one answers if not. So a discovery file works on Ghost if its extension is not `.md` or `.json`, and it is easy if the path is on that list.

**Path 1: ship the files in your theme.** Ghost's theme documentation says a theme can include a robots.txt "which overrides the default robots.txt provided by Ghost" (rechecked 18 October).

1. Download the active theme as a zip from Ghost admin, or clone it.
2. Find the folder holding `index.hbs`, `post.hbs` and `package.json`.
3. Create `robots.txt` beside `package.json`, not in `/assets/`.
4. Write the whole file. It replaces Ghost's default, so copy the seven lines above, swap `{{blog-url}}` for your domain, and add your AI groups underneath. Drop the `Sitemap:` line and you fix B2 while losing D2.
5. Give each crawler its own group and repeat `Disallow:` and `Sitemap:` inside it:

```
User-agent: GPTBot
Disallow: /members/
Allow: /
Sitemap: https://example.com/sitemap.xml
```

6. Create `llms.txt` beside it: an H1, a one-line blockquote, then H2 link lists. See [publishing llms.txt on Ghost](https://aiscan.site/blog/llms-txt-ghost).
7. Re-zip and upload. Ghost runs GScan on upload and rejects fatal errors, so a broken theme cannot take the site down.

**Path 2: `routes.yaml`, when the theme is not yours to edit.**

1. In admin, go to **Settings, then Labs**, and download `routes.yaml`.
2. Add a route with a custom mime type. The routing documentation says routes generally render HTML but a `content_type` property overrides it.
3. Use a trailing slash on the route key. Ghost "automatically forces trailing slashes" (rechecked 18 October).
4. Create the matching `.hbs` template with a `{{#get}}` query over your posts.
5. Upload the file back through Labs. Uploading applies it at once; editing on disk needs a restart.

```yaml
routes:
  /llms.txt/:
    template: llms
    content_type: text/plain
```

`{{#get}}` reads posts, tags, authors, tiers and newsletters but not pages, so a generated index misses About and Pricing.

**The 302 that reads as a pass.** Ghost 6.61 registers four llms.txt routes and, when the feature is off, answers them with `res.redirect(302, '/')`. On 31 August all three publications returned 302 to `/` on exactly those four paths, while `/humans.txt` returned a real 404. A redirect-following check lands on the homepage and records 200. Read the final URL and the content type:

```bash
curl -sL -o /dev/null -w '%{http_code} %{content_type} %{url_effective}\n' https://example.com/llms.txt
# no file:   200 text/html; charset=utf-8 https://example.com/
# real file: 200 text/plain; charset=utf-8 https://example.com/llms.txt
```

**What Ghost cannot serve.** `agents.md`, `/.well-known/mcp/server-card.json`, `/.well-known/api-catalog`, `/.well-known/agent-skills/index.json` and OAuth metadata all hit the `.md` and `.json` denial. On 31 August `/agents.md` returned a hard 404 and `/AGENTS.md` 301ed to lowercase before the same 404. Host those on a domain whose web server you control. Appending `.md` to a post URL returns the post's HTML with a 200, so a C1 test that reads only the status passes a Ghost site with no Markdown at all.

**Check it.**

```bash
SITE=https://yourpublication.com
curl -s "$SITE/robots.txt" | grep -iE 'gptbot|claudebot|perplexitybot|oai-searchbot'
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' "$SITE/sitemap.xml"
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' "$SITE/rss/"
curl -s "$SITE/" | grep -o 'rel="alternate" type="application/rss+xml"'
```

## Hugo

*Verified 18 October 2026 against Hugo's configuration, robots.txt and RSS pages. Site measurements taken 6 September 2026.*

Hugo owns every byte your site emits, so no plan tier or editor panel decides which files exist. It generates a sitemap and an RSS feed without being asked and generates no robots.txt unless you ask, the inverse of the hosted builders. Budget 30 minutes.

| Symptom | Hugo change | Check | Time |
|---|---|---|---|
| No robots.txt | Set `enableRobotsTXT = true` | D1 | 1 min |
| robots.txt says nothing | Write `layouts/robots.txt` with named AI groups | B2, D1 | 10 min |
| Sitemap not declared | Add a `Sitemap:` line to that template | D1, D2 | 2 min |
| Feed exists, nothing links it | Paste Hugo's `.OutputFormats.Get "rss"` snippet into `<head>` | E5 | 5 min |
| No llms.txt | Custom output format, or a file in `static/` | C2 | 10 min |

Hugo's configuration reference describes `enableRobotsTXT` as "Whether to enable generation of a robots.txt file. Default is false." Sitemaps have no switch: Hugo writes `sitemap.xml` from an embedded template unless you add `sitemap` to `disableKinds`. The RSS page says Hugo generates feeds for home, section, taxonomy and term pages. When you do switch robots.txt on, the embedded template outputs only `User-agent: *`, 13 bytes, with no `Sitemap:` line and no crawler named.

**What 15 live Hugo sites publish** (30 candidate domains fingerprinted by the generator tag, 15 confirmed, versions 0.74.0 to 0.165.0, 75 probes, 6 September 2026):

| Surface | Result across 15 sites |
|---|---|
| `/sitemap.xml` returns 200 | 15 of 15 |
| Real 404 for an unknown path | 15 of 15 |
| `/robots.txt` returns 200 | 11 of 15 |
| `/index.xml` returns 200 | 8 of 15 |
| `/llms.txt` returns 200 | 2 of 15 |
| Any AI crawler named in robots.txt | 0 of 15 |
| A `Sitemap:` line in robots.txt | 2 of 11 |

Six of the 11 robots.txt files were the 13 or 14 byte default. One of the four sites with none is gohugo.io itself. Hugo sites lose points on the one file the build leaves to them, not because the build is wrong.

**Path 1: the template.**

1. Add `enableRobotsTXT = true` to `hugo.toml` (`enableRobotsTXT: true` in YAML).
2. Create `layouts/robots.txt`. The lookup is `/layouts/robots.txt`, then `/themes/<THEME>/layouts/robots.txt`, so a project file overrides a theme.
3. Write the policy:

```
User-agent: *
Allow: /
Content-Signal: search=yes, ai-train=no, use=reference

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.org/sitemap.xml
```

4. Run `hugo` and read `public/robots.txt` before deploying.

Consecutive `User-agent:` lines form one group, so stacking names above one rule block gives them all the same rule. Separate blocks with a blank line when you want different treatment. In RFC 9309 the `Sitemap:` line is in section 2.2.4, "Other Records", an extension that is universally supported. Take crawler tokens from the [user-agent list](https://aiscan.site/blog/ai-crawler-user-agent-list-2026).

**Path 2: the static folder.** Set `enableRobotsTXT = false`, put the file in `static/robots.txt`, and build. Hugo's robots page says it copies everything in `static` to the root of `public`. A static file cannot read site variables, but it cannot break the build either, which suits an inherited theme or a site that ships twice a year. Use one route or the other, never both.

**Wire the feed Hugo already built.** Hugo writes `/index.xml` at the first build, and its embedded HTML templates do not link it. Of the eight sites serving a valid feed, two had no reference in `<head>` and one pointed `rel="alternate"` at the homepage. Hugo's RSS page publishes the fix; put it in the head partial:

```
{{ with .OutputFormats.Get "rss" }}
  {{ printf `<link rel=%q type=%q href=%q title=%q>` .Rel .MediaType.Type .Permalink site.Title | safeHTML }}
{{ end }}
```

If the feed moved to `/feed.xml` by renaming the output format, the tag must follow. To drop the feed on purpose, add `rss` to `disableKinds`.

**llms.txt.** Define a custom output format with `baseName = "llms"`, or write `static/llms.txt` once by hand. The output-format configuration and the template naming change in v0.146.0 are in [llms.txt on Hugo](https://aiscan.site/blog/llms-txt-hugo).

**Check it.**

```bash
curl -s https://example.org/robots.txt | wc -c                                 # over about 100 bytes
curl -s https://example.org/robots.txt | grep -icE 'gptbot|claudebot|perplexity'
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.org/sitemap.xml
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.org/index.xml
curl -s https://example.org/ | grep -o '<link[^>]*rss[^>]*>'
curl -s -o /dev/null -w '%{http_code}\n' https://example.org/no-such-path-xyz  # expect 404
```

A scanner caveat: on 6 September our E5 check missed gohugo.io's working feed because it did not probe `/index.xml`. Your feed is fine in that case; confirm the head tag exists.

## Lovable

*Verified 18 October 2026 against Lovable's SEO and TanStack Start upgrade pages. Measurements taken 1 September 2026.*

Two Lovable apps can look identical in a browser and differ by a factor of fifty in what an AI crawler receives. The template decides, and the fix is one confirmation dialog. Budget 20 minutes, plus the migration if you need it.

| If your project is on | What crawlers get | What to do |
|---|---|---|
| TanStack Start (default for new projects since 13 May 2026) | Fully rendered HTML on every request | Add the discovery files (D1, D2, C2) |
| React + Vite (older projects) | Pre-rendered HTML, only for crawlers Lovable verifies | Migrate, then add the files (10 to 35 credits) |
| Either, on a preview URL | `x-robots-tag: noindex, nofollow` | Connect a custom domain before measuring |

**What 15 published Lovable apps returned** (fetched twice each on 1 September 2026, with a desktop browser user agent and a GPTBot user agent, scripts stripped before counting):

| Template | Apps | Body words | Under 50 words |
|---|---|---|---|
| TanStack Start | 6 | 246 to 1,284 (median 470) | 0 |
| React + Vite | 9 | 9 to 210 (median 22) | 7 |

Every app returned the same word count to both user agents. The stack label is Lovable's own `tech_stack` field. The sample is one operator's portfolio, so it measures platform behaviour, not how Lovable users configure sites.

Lovable's SEO page says older projects use "on-request pre-rendering on deployed public URLs, served only to verified search and AI crawlers", and that "Third-party SEO scanners and other unverified agents see the regular single-page app." Its upgrade page says TanStack Start "renders each page into finished HTML on the server before sending it. Everyone receives the same complete page, every time." Sending a crawler's user agent does not get you onto the verified list. Vercel and MERJ (17 December 2024) found none of the major AI crawlers execute JavaScript.

**Step 1: find your template.** Ask the chat "What stack is this project on?", or look for **Project settings, then General, then Project actions, then Migrate to TanStack Start**. If the option is missing, the project is probably already on TanStack Start.

**Step 2, path A: migrate.**

1. Type `/` in the editor chat and pick **Migrate to TanStack Start**, or use the Project actions menu and select **Migrate**.
2. Confirm on the card. The cost is "10 to 35 credits, depending on how large your project is."
3. A large project can pause partway; ask it to continue.
4. Review in the preview. The live site does not change until you publish.
5. Publish. A blank page afterwards usually means a browser-only library; tell Lovable which page failed.

To revert, restore the version before the upgrade from version history. The published site is unaffected.

**Step 2, path B: stay on React + Vite.** Lovable's FAQ says you do not have to upgrade. If you stay: you cannot verify rendering with curl or a scanner (use Google's URL Inspection tool or a link debugger); any crawler outside Lovable's unpublished verified list gets the shell; and the `<title>`, description and Open Graph tags are in the initial HTML on both templates, so put anything a crawler must read there.

**Step 3: add the discovery files, on either template.**

| Need | Check | Mechanism | Where |
|---|---|---|---|
| robots.txt with named AI bots | D1, B2 | Static file | `public/robots.txt` |
| llms.txt | C2 | Static file | `public/llms.txt` |
| Markdown twin of the homepage | C1 | Static file | `public/index.md` |
| MCP server card, API catalog | P1, P2 | Static file | `public/.well-known/…` |
| Sitemap built from a database | D2 | Server route | `sitemap[.]xml.ts` |

Anything in `public/` is served from the site root, and dotted directories work. aiscan.site itself serves `public/.well-known/mcp/server-card.json` and `public/.well-known/agent-skills/index.json` (checked 1 September). That makes Lovable the one hosted builder in this guide where the capabilities dimension is reachable on any plan. For routes, TanStack Router's file naming uses square brackets to escape special characters: `llms[.]txt.ts` answers at `/llms.txt` and a folder named `[.well-known]` produces `/.well-known/`.

For B2, name the crawlers. `lovable.dev/robots.txt` carries separate groups for `GPTBot`, `OAI-SearchBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended` and `Applebot-Extended`, with a comment that a named group replaces the `*` group outright.

**Content outside the app.** If posts live in WordPress or Contentful behind a Lovable front end, the Lovable work does not touch that domain's robots.txt, sitemap or llms.txt. Those are graded separately.

**Check it** on your custom domain, not the preview host:

```bash
URL=https://yoursite.com
curl -sL --compressed "$URL" \
| perl -0777 -pe 's/<(script|style|noscript|svg)\b.*?<\/\1>//gsi; s/<[^>]+>/ /gs' \
| tr -s '[:space:]' ' ' | wc -w         # under 100 means the shell, not the page
curl -sI "$URL" | grep -i x-robots-tag  # noindex means you measured a preview URL
curl -s "$URL/robots.txt" | grep -iE 'gptbot|claudebot|perplexity'
curl -sI "$URL/llms.txt" | head -1
```

On a React + Vite app the scanner reads the single-page shell by design, so a low C3 there means an unverified client sees nothing, which is not the same as Google seeing nothing.

## Confirm it with a scan

Run the scan. It is free, needs no account, and covers every check in this guide in one pass:

```bash
npx aiscan-cli yoursite.com
```

A URL pasted into [the scanner](https://aiscan.site/) does the same without a terminal. Results sit under [discoverability](https://aiscan.site/docs/checks/discoverability) for D1 and D2, [bot access](https://aiscan.site/docs/checks/bot-access) for B2, and [content](https://aiscan.site/docs/checks/content) for C2, C3, E3 and E5. Each row prints its evidence string, so you see the actual fetch.

What a scan cannot tell you is whether your policy is the one you meant, whether the sitemap lists every page, or whether the text answers a reader's question. A perfect score means an agent can reach and parse your site, not that it will find your answer worth citing.

## Next steps

Fix the files the scan names, in checklist order, then re-scan. Re-run it after any change to your adapter, host redirect rules, `trailingSlash` or `baseUrl`, since those break discovery files without touching the code that generates them. Every platform guide is indexed at [aiscan.site/guides](https://aiscan.site/guides).