AI readiness setup guide by platform (2026)

Verified 17 and 18 October 2026. Updated on 18 October 2026.

An AI-ready site needs six things: a robots.txt that names AI crawlers (D1, B2), a sitemap (D2), a declared feed (E5), an llms.txt (C2), and server-rendered HTML that holds the article text (C3, E3). Which of those your platform writes for you decides the work. Static-first frameworks such as Astro, Docusaurus and Hugo generate the sitemap and the feed but no AI-aware robots.txt or llms.txt, hosted builders such as Framer and Ghost write some files and block others, and each platform has one trap that can hide the whole set from crawlers.

TL;DR

• Every platform gets the same six checks; only the missing files differ.
• Astro: HTML is static by default, so C3 and E3 pass unless you wrap content in client:only. Extension-bearing endpoints 404 behind a trailing slash.
• Docusaurus: sitemap and feed come free, robots.txt and llms.txt do not, and a route can ship an empty __docusaurus div. A baseUrl subpath makes robots.txt void.
• Hugo: robots.txt is off until enableRobotsTXT = true, and the default is 13 bytes naming nobody. 0 of 15 live Hugo sites named an AI crawler.
• Framer: robots.txt, sitemap and a Markdown twin come free; a custom robots.txt upload replaces the generated one, Sitemap: line included.
• Ghost: only theme files and a routes.yaml route reach the root, .md and .json never do, and a disabled llms.txt answers 302, not 404.
• Lovable: TanStack Start renders for everyone; React + Vite renders only for crawlers Lovable verifies. 7 of 9 Vite apps returned under 50 words.
• Run npx aiscan-cli yoursite.com first. It names the failing rows in one pass.

Pick your platform

Seven platforms are folded in so far. The rest move into this page one run at a time, and their separate guides stay live until each one does.

What every platform shares

The retrieval crawlers behind ChatGPT, Claude and Perplexity do not run JavaScript. Per Vercel and MERJ, 17 December 2024: "none of the major AI crawlers currently render JavaScript," covering GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. So the HTML your server sends is the page, as far as they are concerned.

The six checks, which the sections below refer to by ID:

Naming crawlers matters more than the value you give them. Google's robots.txt specification says only the most specific matching group applies, so a bot with its own group never reads your * group. Consecutive User-agent lines form one group, so put a blank line between groups that need different rules. Take the tokens from the current crawler list rather than from memory, because operators rename them.

An empty file does not count either. RFC 9309 section 2.2.1 says: "If no group matches the product token and there is no group with a user-agent line with the * value, or no groups are present at all, no rules apply." The robots.txt rules must also sit at the top-level path of the origin, so a file served only under a subpath is void.

Astro

Verified 17 October 2026 against the Astro docs. First written against Astro 7.2.9 on 29 August 2026.

Astro ships static HTML by default, so C3 and E3 usually pass before you touch anything. It does not create the discovery files an agent looks for, and one routing rule can break three of them at once. Budget 25 minutes.

Astro's own documentation site shows there is no upstream example to copy. On 17 October 2026 docs.astro.build/robots.txt still returns two lines, User-agent: * and Allow: /, with no AI crawler group and no Sitemap: line, and docs.astro.build/llms.txt still returns 404.

Rendering. Astro's islands architecture renders most of the page to static HTML and adds JavaScript only where needed (Astro docs, islands), so a React component inside an Astro page is rendered to HTML at build time. One directive undoes that. Per Astro's template directives reference, client:only={string} "skips HTML server rendering, and renders only on the client." A component marked client:only="react" leaves an empty element in the built HTML. Grep first:

grep -rn 'client:only' src/

1. Set your deployed URL. Add site: 'https://example.com' to astro.config.mjs. Three of the four files need it for absolute URLs.

2. Write robots.txt. A plain public/robots.txt works. Generating it from site keeps the sitemap line from drifting. Create src/pages/robots.txt.ts:

import type { APIRoute } from 'astro';

const getRobotsTxt = (sitemapURL: URL) => `User-agent: *
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Sitemap: ${sitemapURL.href}
`;

export const GET: APIRoute = ({ site }) => {
  const sitemapURL = new URL('sitemap-index.xml', site);
  return new Response(getRobotsTxt(sitemapURL));
};

To permit retrieval but refuse training, give OAI-SearchBot, Claude-SearchBot and PerplexityBot Allow: /, and GPTBot, ClaudeBot, Google-Extended and Applebot-Extended Disallow: /.

3. Add the sitemap. Run npx astro add sitemap. The build writes sitemap-index.xml and sitemap-0.xml. It cannot generate entries for dynamic routes in SSR mode, and nothing links the file for you, so add <link rel="sitemap" href="/sitemap-index.xml" /> to your layout head.

4. Add the feed and declare it. Install @astrojs/rss, export a GET from src/pages/rss.xml.js that returns rss({ title, description, site: context.site, items }), and build items from getCollection(). E5 fails on the declaration far more often than on the feed, so add this to the head:

<link rel="alternate" type="application/rss+xml" title="Your Site" href={new URL("rss.xml", Astro.site)} />

The RSS helper adds trailing slashes to links by default. If your config sets trailingSlash: "never", pass trailingSlash: false to rss().

5. Add llms.txt. An endpoint at src/pages/llms.txt.ts can call getCollection() and emit an H1, ## sections and Markdown links from the same source your pages use. The llms.txt generator drafts one from a live URL. Starlight docs sites can use the starlight-llms-txt plugin instead. No Astro plugin writes robots.txt with AI crawler groups, so step 2 stays manual.

The trailing-slash trap. The file exists, the build succeeds, and the URL still 404s. From Astro's endpoints guide, checked 17 October 2026: endpoints whose URLs include a file extension "can only be accessed without a trailing slash (e.g. /sitemap.xml), regardless of your build.trailingSlash configuration." Every file above is such an endpoint. /robots.txt/, /rss.xml/ and /llms.txt/ return 404 while the slashless forms return 200. If your host or middleware appends a trailing slash, it hands every crawler a 404 for your whole discovery layer. Test the slashless URL and exempt those paths from any slash-adding rule.

Check it.

curl -sI https://example.com/robots.txt | head -1        # expect 200
curl -s  https://example.com/robots.txt | grep -i gptbot # expect a match
curl -sI https://example.com/sitemap-index.xml | head -1 # expect 200
curl -s  https://example.com/ | grep -o 'rel="alternate"[^>]*rss[^>]*'
curl -s  https://example.com/ | grep -c '<h1'            # expect 1

A single-page Astro marketing site needs neither a sitemap nor a feed. Do step 2 and stop. If the front end reads from a headless WordPress CMS, that CMS domain has its own robots.txt, and the work above does not touch it.

Docusaurus

Verified 17 October 2026 against the Docusaurus docs. Site measurements taken 7 September 2026.

Docusaurus runs a large share of developer documentation, which is the material retrieval systems reach for. The discovery files are the easy half. The hard half is whether a given page contains any words, and Docusaurus says so itself. Budget 30 minutes.

Why a page can be empty and still be "static". Docusaurus is a React single-page application that emits static HTML as an optimisation. Its Static Site Generation page, rechecked 17 October 2026, says: "Docusaurus is ultimately a single-page application, so static site generation is only an optimization (progressive enhancement, as it's called), but our functionality does not fully depend on those HTML files." A route that escapes server rendering ships <div id="__docusaurus"></div> empty.

On 7 September 2026 we fetched the Cypress documentation home page with a desktop browser user agent. It returned 6,826 bytes with that empty div, data-has-hydrated="false" and two words of visible text. Our scanner grades it 72, Level 4, with C3 and E3 failing on "2 words of server-rendered text, 0 <h2>, 0 <h3>". The same site's robots.txt names eight AI crawlers. The failure was one route, not the site: two other documentation pages on that host returned 2,196 and 3,897 words. On Docusaurus, C3 and E3 are a per-route outcome.

What 15 live Docusaurus sites publish (30 candidate documentation domains fingerprinted, 15 confirmed, 6 paths each, 90 requests, 7 September 2026):

Four of the nine robots.txt files are byte-identical at 1,248 bytes and contain no User-agent, Allow, Disallow or Sitemap line, only a licence preamble. Under RFC 9309 that means no rules apply. The one site that names AI crawlers stacks all eight into a single group with *, so no named agent gets a rule of its own. That is harmless when everything is allowed, and wrong the moment the rules differ.

Path 1: the static folder. Docusaurus copies the contents of static/ into the root of the build output, with the directory structure preserved.

1. Create static/robots.txt with real groups separated by blank lines and a Sitemap: line:

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /

Sitemap: https://example.org/sitemap.xml

1. Add static/llms.txt with an H1, a blockquote summary and H2 sections of [title](url) links. The llms.txt generator writes the skeleton.
2. For .well-known files, static/.well-known/ works. On GitHub Pages, add an empty static/.nojekyll.
3. Run npm run build and confirm the files landed in build/.

Path 2: generate the files from your routes. The postBuild(props) plugin hook runs when a production build finishes and receives siteConfig, outDir, baseUrl and routesPaths. Create plugins/discovery-files.js:

import fs from 'node:fs/promises';
import path from 'node:path';

export default function discoveryFiles() {
  return {
    name: 'discovery-files',
    async postBuild({siteConfig, outDir, baseUrl, routesPaths}) {
      if (baseUrl !== '/') {
        throw new Error(
          `robots.txt is void under baseUrl "${baseUrl}". ` +
          'Deploy at a domain root, or remove the robots.txt half.'
        );
      }
      const origin = siteConfig.url.replace(/\/$/, '');
      await fs.writeFile(
        path.join(outDir, 'robots.txt'),
        `User-agent: *\nAllow: /\n\nSitemap: ${origin}/sitemap.xml\n`,
      );
      const docs = routesPaths.filter((r) => r.startsWith('/docs/'));
      await fs.writeFile(
        path.join(outDir, 'llms.txt'),
        [`# ${siteConfig.title}`, '', `> ${siteConfig.tagline}`, '',
         '## Docs', '',
         ...docs.map((r) => `- [${r}](${origin}${r})`), ''].join('\n'),
      );
    },
  };
}

Register it with plugins: ['./plugins/discovery-files.js'] in docusaurus.config.js, then run npm run build, not npm start, because postBuild fires only on a production build. The link list is derived from the route table, so it cannot drift, and the baseUrl guard turns a silent deployment failure into a failed build. docusaurus-plugin-llms is a community package for the llms.txt half; pin its version.

The baseUrl trap. static/ is served under baseUrl. For a site at https://my-org.com/my-project/, baseUrl is /my-project/, so a valid static/robots.txt answers at /my-project/robots.txt while the origin root returns 404. For llms.txt that is survivable, since the specification allows a subpath file. For robots.txt it is fatal: RFC 9309 section 2.3 says "The rules MUST be accessible in a file named /robots.txt (all lowercase) in the top-level path of the service." On 7 September 2026 neither facebook.github.io nor microsoft.github.io served a robots.txt at all, because the origin root belongs to the GitHub account and no repository can write to it. Three ways out are in the Docusaurus robots.txt guide, and the llms.txt half is in publishing llms.txt on Docusaurus.

Sitemap and feed are already handled. @docusaurus/plugin-sitemap ships in preset-classic and is active only in production, which is why a local check finds nothing. noIndex results in no sitemap, so check it before anything else when sitemap.xml is missing on a live build. The blog plugin defaults feedOptions to RSS and Atom, 20 posts, and injects the autodiscovery tags. Setting feedOptions.type to null is the only way to end up with no feed.

Audit the half that can break.

1. Search for browser-only rendering: grep -rn "BrowserOnly\|useIsBrowser\|typeof window" src/.
2. Build, then check the built HTML: grep -c 'id="__docusaurus"></div>' build/index.html. A match means that page shipped an empty root.
3. Repeat on the routes that matter. Custom landing pages written as React components are where this goes wrong; generated documentation routes almost never do.
4. Strip scripts and count words on the built file. A few dozen words on a page that should have hundreds is the signal.

The identical failure on another stack is written up in React apps that go invisible to crawlers.

Framer

Measured 4 September 2026 on two live Framer sites. Framer's help articles are rendered by script and could not be re-fetched on 18 October, so those quotes carry their September date.

Framer does more for you than any other hosted builder in this guide, which is why its gaps are easy to miss. It writes a robots.txt and a sitemap for every published site, pre-renders every page, and serves a Markdown copy of each optimized page. What it leaves out is what a default cannot express: which AI crawlers you name (B2), an llms.txt (C2) and a feed (E5). Budget 20 minutes.

Framer's help article on AI agents says: "Although Framer sites are built with React, every page is pre-rendered to HTML on our servers before it is served." The Markdown copies are small. On 4 September, www.framer.com/pricing was 1,470,695 bytes of HTML and 11,651 bytes of Markdown (126 times smaller), and www.linkpreview.app/ was 245,109 against 5,106 (48 times). Both Markdown answers were 200 text/markdown; charset=utf-8.

Why B2 fails on a good default. The generated file for www.linkpreview.app is 72 bytes, rechecked on 18 October: User-agent: *, Allow: /, and a Sitemap: line. Allowing everything is a legitimate decision, but a wildcard says nothing about AI crawlers, and B2 asks whether you stated a preference. Google's robots.txt specification applies only the most specific matching group, so a bot with its own group never reads the * group.

Path 1: upload the files (Pro and Enterprise).

1. Open Site Settings, then Hosting, then Files. Older tutorials point at the Domains dashboard, where these settings used to live.
2. Click Add and upload the file.
3. Set the file path without the file name: / for the domain root, /.well-known for /.well-known/security.txt.
4. Publish.

Supported formats are PDF, TXT, CSV, JSON, XML, MD, VCARD and JS. Because JSON and .well-known paths work, capability files are reachable on Framer's paid plans, which Webflow and Ghost do not allow.

The upload trap. Framer's article states that a custom robots.txt replaces the generated one. It does not merge, and the Sitemap: line goes with the old file. Start from the generated file, not a blank one:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Content-Signal: ai-train=yes, search=yes, ai-input=yes

Sitemap: https://example.com/sitemap.xml

Then upload llms.txt at path /. Framer's own file at www.framer.com/llms.txt (5,900 bytes on 4 September) is a good model: a one-line description, then H2 sections of annotated links pointing at the ?md version of each page. The llms.txt generator writes a starting point, and llms.txt on Framer covers the contents.

Path 2: answer in front of Framer (any plan). Free and Basic sites cannot upload static files. Framer documents reverse-proxy hosting for this, with Cloudflare, CloudFront, Vercel, Netlify and NGINX named.

1. Add a custom domain to the Framer site to act as the origin, for example origin.example.com. Do not use a *.framer.app domain: Framer warns that those are not indexed by search engines.
2. Point the proxy at that origin and add routes for /robots.txt, /llms.txt and your feed, returning static text with the right content type. Pass everything else through untouched.
3. Set the Canonical URL in General settings to the public address.
4. Publish, then request each path on the public domain and confirm the proxy answered.

Do not modify or remove tags inside <head> and do not touch <div id="main">. Adding head tags is supported. This route also covers E5: on 4 September /rss.xml, /feed.xml and /blog/rss.xml all returned 404 on www.framer.com, so a Framer feed is an uploaded XML file you regenerate on publish, or a proxy route that reads your CMS.

Structured data. Schema is neither automatic nor gated. Put a <script type="application/ld+json"> tag in Custom Code in the head. On CMS detail pages, {{Field | json}} escapes a collection field safely, and {{Created}} and {{Updated}} exist on every CMS item, so BlogPosting gets datePublished and dateModified without extra fields. Avoid {{field | unsafeRaw}} unless you control the stored value, since it escapes nothing.

Check it.

curl -sI https://example.com/ | grep -i server-timing     # expect ssg-status;desc="optimized"
curl -s  https://example.com/robots.txt                    # named groups plus a Sitemap: line
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' -H 'Accept: text/markdown' https://example.com/

Framer says Markdown is generated only for optimized pages, so a page whose server-timing header does not read optimized has no Markdown twin and C1 will fail without saying why. The third command should answer 200 text/markdown; charset=utf-8.

Ghost

Measured 31 August 2026 on three live Ghost 6.61 publications, against Ghost's source at version 6.61.1. The source file could not be re-fetched on 18 October; the theme and routing documentation was, and still says what is quoted below.

Ghost gives you server-rendered HTML, a sitemap, an RSS feed and Article schema switched on, then stops at a wall: it has no static file root and no file manager, so every other discovery file arrives through one of two doors. Budget 30 minutes.

A 404 Media article fetched with plain curl on 31 August returned 1,866 words, one <h1> and one application/ld+json block typed Article. Ghost's default robots.txt is 203 bytes and matches platformer.news, 404media.co and thebrowser.com byte for byte:

User-agent: *
Sitemap: {{blog-url}}/sitemap.xml
Disallow: /ghost/
Disallow: /email/
Disallow: /members/api/comments/counts/
Disallow: /r/
Disallow: /webmentions/receive/
Disallow: /.ghost/analytics/api/

The fall-through rule. Ghost serves root-path files from the active theme folder through a middleware called static-theme. It denies the extensions .hbs, .md, .json, .lock and .log (except manifest.json, assetlinks.json and anything under /assets/). The paths /robots.txt, /sitemap.xml, /sitemap.xsl, /llms.txt, /llms-full.txt and the two /.well-known/ llms variants fall through: a theme file wins if present, and Ghost's generated one answers if not. So a discovery file works on Ghost if its extension is not .md or .json, and it is easy if the path is on that list.

Path 1: ship the files in your theme. Ghost's theme documentation says a theme can include a robots.txt "which overrides the default robots.txt provided by Ghost" (rechecked 18 October).

1. Download the active theme as a zip from Ghost admin, or clone it.
2. Find the folder holding index.hbs, post.hbs and package.json.
3. Create robots.txt beside package.json, not in /assets/.
4. Write the whole file. It replaces Ghost's default, so copy the seven lines above, swap {{blog-url}} for your domain, and add your AI groups underneath. Drop the Sitemap: line and you fix B2 while losing D2.
5. Give each crawler its own group and repeat Disallow: and Sitemap: inside it:

User-agent: GPTBot
Disallow: /members/
Allow: /
Sitemap: https://example.com/sitemap.xml

1. Create llms.txt beside it: an H1, a one-line blockquote, then H2 link lists. See publishing llms.txt on Ghost.
2. Re-zip and upload. Ghost runs GScan on upload and rejects fatal errors, so a broken theme cannot take the site down.

Path 2: routes.yaml, when the theme is not yours to edit.

1. In admin, go to Settings, then Labs, and download routes.yaml.
2. Add a route with a custom mime type. The routing documentation says routes generally render HTML but a content_type property overrides it.
3. Use a trailing slash on the route key. Ghost "automatically forces trailing slashes" (rechecked 18 October).
4. Create the matching .hbs template with a {{#get}} query over your posts.
5. Upload the file back through Labs. Uploading applies it at once; editing on disk needs a restart.

routes:
  /llms.txt/:
    template: llms
    content_type: text/plain

{{#get}} reads posts, tags, authors, tiers and newsletters but not pages, so a generated index misses About and Pricing.

The 302 that reads as a pass. Ghost 6.61 registers four llms.txt routes and, when the feature is off, answers them with res.redirect(302, '/'). On 31 August all three publications returned 302 to / on exactly those four paths, while /humans.txt returned a real 404. A redirect-following check lands on the homepage and records 200. Read the final URL and the content type:

curl -sL -o /dev/null -w '%{http_code} %{content_type} %{url_effective}\n' https://example.com/llms.txt
# no file:   200 text/html; charset=utf-8 https://example.com/
# real file: 200 text/plain; charset=utf-8 https://example.com/llms.txt

What Ghost cannot serve. agents.md, /.well-known/mcp/server-card.json, /.well-known/api-catalog, /.well-known/agent-skills/index.json and OAuth metadata all hit the .md and .json denial. On 31 August /agents.md returned a hard 404 and /AGENTS.md 301ed to lowercase before the same 404. Host those on a domain whose web server you control. Appending .md to a post URL returns the post's HTML with a 200, so a C1 test that reads only the status passes a Ghost site with no Markdown at all.

Check it.

SITE=https://yourpublication.com
curl -s "$SITE/robots.txt" | grep -iE 'gptbot|claudebot|perplexitybot|oai-searchbot'
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' "$SITE/sitemap.xml"
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' "$SITE/rss/"
curl -s "$SITE/" | grep -o 'rel="alternate" type="application/rss+xml"'

Hugo

Verified 18 October 2026 against Hugo's configuration, robots.txt and RSS pages. Site measurements taken 6 September 2026.

Hugo owns every byte your site emits, so no plan tier or editor panel decides which files exist. It generates a sitemap and an RSS feed without being asked and generates no robots.txt unless you ask, the inverse of the hosted builders. Budget 30 minutes.

Hugo's configuration reference describes enableRobotsTXT as "Whether to enable generation of a robots.txt file. Default is false." Sitemaps have no switch: Hugo writes sitemap.xml from an embedded template unless you add sitemap to disableKinds. The RSS page says Hugo generates feeds for home, section, taxonomy and term pages. When you do switch robots.txt on, the embedded template outputs only User-agent: *, 13 bytes, with no Sitemap: line and no crawler named.

What 15 live Hugo sites publish (30 candidate domains fingerprinted by the generator tag, 15 confirmed, versions 0.74.0 to 0.165.0, 75 probes, 6 September 2026):

Six of the 11 robots.txt files were the 13 or 14 byte default. One of the four sites with none is gohugo.io itself. Hugo sites lose points on the one file the build leaves to them, not because the build is wrong.

Path 1: the template.

1. Add enableRobotsTXT = true to hugo.toml (enableRobotsTXT: true in YAML).
2. Create layouts/robots.txt. The lookup is /layouts/robots.txt, then /themes/<THEME>/layouts/robots.txt, so a project file overrides a theme.
3. Write the policy:

User-agent: *
Allow: /
Content-Signal: search=yes, ai-train=no, use=reference

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.org/sitemap.xml

1. Run hugo and read public/robots.txt before deploying.

Consecutive User-agent: lines form one group, so stacking names above one rule block gives them all the same rule. Separate blocks with a blank line when you want different treatment. In RFC 9309 the Sitemap: line is in section 2.2.4, "Other Records", an extension that is universally supported. Take crawler tokens from the user-agent list.

Path 2: the static folder. Set enableRobotsTXT = false, put the file in static/robots.txt, and build. Hugo's robots page says it copies everything in static to the root of public. A static file cannot read site variables, but it cannot break the build either, which suits an inherited theme or a site that ships twice a year. Use one route or the other, never both.

Wire the feed Hugo already built. Hugo writes /index.xml at the first build, and its embedded HTML templates do not link it. Of the eight sites serving a valid feed, two had no reference in <head> and one pointed rel="alternate" at the homepage. Hugo's RSS page publishes the fix; put it in the head partial:

{{ with .OutputFormats.Get "rss" }}
  {{ printf `<link rel=%q type=%q href=%q title=%q>` .Rel .MediaType.Type .Permalink site.Title | safeHTML }}
{{ end }}

If the feed moved to /feed.xml by renaming the output format, the tag must follow. To drop the feed on purpose, add rss to disableKinds.

llms.txt. Define a custom output format with baseName = "llms", or write static/llms.txt once by hand. The output-format configuration and the template naming change in v0.146.0 are in llms.txt on Hugo.

Check it.

curl -s https://example.org/robots.txt | wc -c                                 # over about 100 bytes
curl -s https://example.org/robots.txt | grep -icE 'gptbot|claudebot|perplexity'
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.org/sitemap.xml
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.org/index.xml
curl -s https://example.org/ | grep -o '<link[^>]*rss[^>]*>'
curl -s -o /dev/null -w '%{http_code}\n' https://example.org/no-such-path-xyz  # expect 404

A scanner caveat: on 6 September our E5 check missed gohugo.io's working feed because it did not probe /index.xml. Your feed is fine in that case; confirm the head tag exists.

Lovable

Verified 18 October 2026 against Lovable's SEO and TanStack Start upgrade pages. Measurements taken 1 September 2026.

Two Lovable apps can look identical in a browser and differ by a factor of fifty in what an AI crawler receives. The template decides, and the fix is one confirmation dialog. Budget 20 minutes, plus the migration if you need it.

What 15 published Lovable apps returned (fetched twice each on 1 September 2026, with a desktop browser user agent and a GPTBot user agent, scripts stripped before counting):

Every app returned the same word count to both user agents. The stack label is Lovable's own tech_stack field. The sample is one operator's portfolio, so it measures platform behaviour, not how Lovable users configure sites.

Lovable's SEO page says older projects use "on-request pre-rendering on deployed public URLs, served only to verified search and AI crawlers", and that "Third-party SEO scanners and other unverified agents see the regular single-page app." Its upgrade page says TanStack Start "renders each page into finished HTML on the server before sending it. Everyone receives the same complete page, every time." Sending a crawler's user agent does not get you onto the verified list. Vercel and MERJ (17 December 2024) found none of the major AI crawlers execute JavaScript.

Step 1: find your template. Ask the chat "What stack is this project on?", or look for Project settings, then General, then Project actions, then Migrate to TanStack Start. If the option is missing, the project is probably already on TanStack Start.

Step 2, path A: migrate.

1. Type / in the editor chat and pick Migrate to TanStack Start, or use the Project actions menu and select Migrate.
2. Confirm on the card. The cost is "10 to 35 credits, depending on how large your project is."
3. A large project can pause partway; ask it to continue.
4. Review in the preview. The live site does not change until you publish.
5. Publish. A blank page afterwards usually means a browser-only library; tell Lovable which page failed.

To revert, restore the version before the upgrade from version history. The published site is unaffected.

Step 2, path B: stay on React + Vite. Lovable's FAQ says you do not have to upgrade. If you stay: you cannot verify rendering with curl or a scanner (use Google's URL Inspection tool or a link debugger); any crawler outside Lovable's unpublished verified list gets the shell; and the <title>, description and Open Graph tags are in the initial HTML on both templates, so put anything a crawler must read there.

Step 3: add the discovery files, on either template.

Anything in public/ is served from the site root, and dotted directories work. aiscan.site itself serves public/.well-known/mcp/server-card.json and public/.well-known/agent-skills/index.json (checked 1 September). That makes Lovable the one hosted builder in this guide where the capabilities dimension is reachable on any plan. For routes, TanStack Router's file naming uses square brackets to escape special characters: llms[.]txt.ts answers at /llms.txt and a folder named [.well-known] produces /.well-known/.

For B2, name the crawlers. lovable.dev/robots.txt carries separate groups for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended, with a comment that a named group replaces the * group outright.

Content outside the app. If posts live in WordPress or Contentful behind a Lovable front end, the Lovable work does not touch that domain's robots.txt, sitemap or llms.txt. Those are graded separately.

Check it on your custom domain, not the preview host:

URL=https://yoursite.com
curl -sL --compressed "$URL" \
| perl -0777 -pe 's/<(script|style|noscript|svg)\b.*?<\/\1>//gsi; s/<[^>]+>/ /gs' \
| tr -s '[:space:]' ' ' | wc -w         # under 100 means the shell, not the page
curl -sI "$URL" | grep -i x-robots-tag  # noindex means you measured a preview URL
curl -s "$URL/robots.txt" | grep -iE 'gptbot|claudebot|perplexity'
curl -sI "$URL/llms.txt" | head -1

On a React + Vite app the scanner reads the single-page shell by design, so a low C3 there means an unverified client sees nothing, which is not the same as Google seeing nothing.

Confirm it with a scan

Run the scan. It is free, needs no account, and covers every check in this guide in one pass:

npx aiscan-cli yoursite.com

A URL pasted into the scanner does the same without a terminal. Results sit under discoverability for D1 and D2, bot access for B2, and content for C2, C3, E3 and E5. Each row prints its evidence string, so you see the actual fetch.

What a scan cannot tell you is whether your policy is the one you meant, whether the sitemap lists every page, or whether the text answers a reader's question. A perfect score means an agent can reach and parse your site, not that it will find your answer worth citing.

Next steps

Fix the files the scan names, in checklist order, then re-scan. Re-run it after any change to your adapter, host redirect rules, trailingSlash or baseUrl, since those break discovery files without touching the code that generates them. Every platform guide is indexed at aiscan.site/guides.