The complete AI readiness setup for Astro in 2026

Verified 29 August 2026 against Astro 7.2.9.

Most platform guides open by telling you to fix your rendering. On Astro you can skip that. Astro ships static HTML by default, so the two checks that sink most React and Vue sites, C3 and E3, usually pass before you touch anything. What Astro does not do is create the four discovery files an AI agent looks for, and one routing rule quietly breaks three of them at once. Expect 25 minutes to pass all six checks below.

Quick summary

Astro's own documentation site is the honest illustration. On 29 August 2026, docs.astro.build/robots.txt returns exactly two lines, User-agent: * and Allow: /, with no AI crawler group and no Sitemap: line, and docs.astro.build/llms.txt returns 404. There is no upstream example to copy, so every file below comes from the primary documentation.

What Astro's architecture already gives you

Islands architecture is the reason Astro starts ahead. Astro describes the pattern as "rendering the majority of your page to fast, static HTML with smaller 'islands' of JavaScript added when interactivity or personalization is needed on the page" (Astro docs, islands architecture). A React component in an Astro page is server-rendered to HTML during the build and hydrated in the browser afterwards, so the crawler that never runs JavaScript still receives the text.

That matters because the retrieval crawlers behind ChatGPT, Claude and Perplexity do not run JavaScript. Per Vercel and MERJ, 17 December 2024: "none of the major AI crawlers currently render JavaScript," covering GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. Nothing published since contradicts it.

One directive undoes that advantage. From Astro's template directives reference, client:only={string} "skips HTML server rendering, and renders only on the client." A component marked client:only="react" produces an empty element in the built HTML. Put your article body or pricing table inside one and every non-rendering crawler gets a blank page while your browser looks perfect. That is the only Astro-specific way to fail C3 and E3, so grep for it first:

grep -rn 'client:only' src/

If nothing in that list wraps content a reader would quote, your rendering is fine and the rest of the work is files.

Path 1: the four discovery files, by hand

This path needs no dependencies beyond two official Astro packages.

1. Set your deployed URL. In astro.config.mjs, add site: 'https://example.com'. Three of the four files need it to build absolute URLs, the sitemap integration requires it, and it must begin with http:// or https://.

2. Write robots.txt. The static option is a plain file at public/robots.txt, which Astro copies to the root untouched. The better option, adapted from Astro's sitemap integration docs, generates it from the site value so the two cannot drift apart. Create src/pages/robots.txt.ts:

import type { APIRoute } from 'astro';

const getRobotsTxt = (sitemapURL: URL) => `User-agent: *
Allow: /

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Sitemap: ${sitemapURL.href}
`;

export const GET: APIRoute = ({ site }) => {
  const sitemapURL = new URL('sitemap-index.xml', site);
  return new Response(getRobotsTxt(sitemapURL));
};

The second group is what B2 looks for. Naming those crawlers matters more than the value you give them, because Google's robots.txt specification says only the most specific matching group applies and all other groups are ignored. A bot with its own group never reads your * group, so a named Allow is a statement rather than a redundancy. To permit retrieval while refusing training, split them: Allow: / for OAI-SearchBot, Claude-SearchBot and PerplexityBot, Disallow: / for GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. Check the current crawler tokens first, because five operators now run three separate bots each.

3. Add the sitemap. Run npx astro add sitemap, which installs @astrojs/sitemap (3.7.3, published 26 May 2026) and adds sitemap() to integrations. The build then writes sitemap-index.xml and sitemap-0.xml. Two documented limits catch people out: it "cannot generate sitemap entries for dynamic routes in SSR mode," and nothing links the file automatically. Add the link in your layout <head>:

<link rel="sitemap" href="/sitemap-index.xml" />

4. Add the feed and declare it. Install @astrojs/rss (4.0.19, published 30 June 2026) and create src/pages/rss.xml.js exporting a GET that returns rss({ title, description, site: context.site, items }), building items from getCollection(). E5 fails on the declaration far more often than on the feed itself, so add this to your <head> too:

<link rel="alternate" type="application/rss+xml" title="Your Site" href={new URL("rss.xml", Astro.site)} />

One documented mismatch: the RSS helper "produces links with a trailing slash by default, no matter what value you have configured for trailingSlash." If your config sets trailingSlash: "never", pass trailingSlash: false to rss().

5. Add llms.txt. Content collections make this a build step rather than a file you maintain: an endpoint at src/pages/llms.txt.ts can call getCollection() and emit an H1, ## sections and Markdown links from the same source your pages use. The full version of this step is in How to publish a valid llms.txt on Astro, and the llms.txt generator will draft one from a live URL.

Path 2: integrations, if you would rather not write endpoints

Documentation sites built on Starlight have a plugin for the hardest part. starlight-llms-txt (0.11.0, published 1 July 2026) is maintained by Starlight maintainer Chris Swithinbank and generates llms.txt, llms-full.txt and llms-small.txt from your existing docs. Install with npm i starlight-llms-txt, add starlightLlmsTxt() to the plugins array inside starlight(), set site, then preview at localhost:4321/llms.txt. Steps 3 and 4 are unchanged.

No Astro plugin writes robots.txt with AI crawler groups for you, so step 2 stays manual either way.

The trailing-slash rule that breaks three endpoints at once

This trap produces the strangest scan results, because the file exists, the build succeeds, and the URL still 404s. From Astro's endpoints guide: "endpoints whose URLs include a file extension (e.g. src/pages/sitemap.xml.ts) can only be accessed without a trailing slash (e.g. /sitemap.xml), regardless of your build.trailingSlash configuration."

Every file in Path 1 is an extension-bearing endpoint. /robots.txt/, /rss.xml/ and /llms.txt/ all return 404 while their slashless forms return 200. If your host, CDN or middleware appends a trailing slash to normalise URLs, it hands every crawler a 404 for your whole discovery layer at once. Test the slashless form, and exempt those three paths from any slash-adding redirect rule.

The related setting is prerender. Astro's endpoints guide states that routes "will be rendered on demand by default in server mode," and that in static mode you opt out per endpoint. If you added an adapter and these files started behaving dynamically, that flag is where to look.

Confirm it with a scan

Run the scan first. It checks all six of these at once and names the row that failed:

npx aiscan-cli example.com

No account, no key, and you can paste the URL at aiscan.site instead if you prefer a browser. The rows that matter here are D1 robots.txt present, B2 explicit AI bot rules, D2 sitemap, C2 llms.txt, C3 and E3 server-rendered HTML, and E5 feed declared.

If you would rather check by hand, these five commands cover the same ground:

curl -sI https://example.com/robots.txt | head -1        # expect 200
curl -s  https://example.com/robots.txt | grep -i gptbot # expect a match
curl -sI https://example.com/sitemap-index.xml | head -1 # expect 200
curl -s  https://example.com/ | grep -o 'rel="alternate"[^>]*rss[^>]*'
curl -s  https://example.com/ | grep -c '<h1'            # expect 1

The last one is the rendering test. If the headings are missing while the browser shows them, client:only is the cause.

Where AIScan fits, and where it doesn't

The right column is editorial and no scanner closes it. A perfect score means an agent can reach and parse your site, not that it will find your answer worth citing.

A ten-minute checklist

• site in astro.config.mjs matches the deployed origin
• grep -rn 'client:only' src/ returns nothing wrapping article text
• /robots.txt returns 200 with at least one named AI crawler group
• /sitemap-index.xml returns 200 and <link rel="sitemap"> is in the layout head
• /rss.xml returns 200 and <link rel="alternate" type="application/rss+xml"> is in the head
• /llms.txt returns 200 with an H1, one ## section and Markdown links
• None of those four URLs redirect to a trailing-slash form

Re-run it after any change to your adapter, your host's redirect rules or trailingSlash. Those three break discovery files without touching the code that generates them.

Who can skip most of this

A single-page Astro marketing site needs neither a sitemap nor a feed, and llms.txt on a five-page site adds nothing an agent could not read directly. Do step 2 and stop. If your Astro front end pulls from a headless WordPress CMS, that CMS domain has its own robots.txt, sitemap and schema, and none of the work above touches it. ThinkRank handles all four from one WordPress plugin, which avoids two or three SEO plugins each claiming ownership of robots.txt, and it migrates settings from Rank Math, Yoast, All in One SEO and SEOPress so nothing is re-entered.

Start here

Run npx aiscan-cli against your Astro site and read the D1, B2, D2, C2, E5 and C3 rows. Fix whichever files it names, in checklist order, then re-scan. The same six checks apply to every framework: the Next.js version of this setup covers the same ground with app/robots.ts and app/sitemap.ts, and the teardown of a Next.js site that ranks on Google while being invisible to ChatGPT shows what a rendering failure looks like. Full check definitions live on discoverability, content and bot access, and every platform guide is indexed at aiscan.site/guides.