--- title: "The complete AI readiness setup for Astro in 2026" slug: ai-readiness-setup-astro published: 2026-08-29T14:34:15.901667+00:00 updated: 2026-08-29T14:34:15.901667+00:00 author: "Asif Rahman" author_url: https://masifrahman.com category: "AI Readiness" tags: platform:astro, check:D1, check:D2, check:B2, check:C2, check:C3, check:E3, check:E5, Astro, AI readiness, robots.txt, llms.txt, sitemap, RSS description: "Astro ships static HTML, so C3 and E3 pass by default. Add robots.txt, sitemap, RSS and llms.txt in 25 minutes, plus the trailing-slash rule that 404s them." url: https://aiscan.site/blog/ai-readiness-setup-astro --- **Verified 29 August 2026 against Astro 7.2.9.** Most platform guides open by telling you to fix your rendering. On Astro you can skip that. Astro ships static HTML by default, so the two checks that sink most React and Vue sites, C3 and E3, usually pass before you touch anything. What Astro does not do is create the four discovery files an AI agent looks for, and one routing rule quietly breaks three of them at once. Expect 25 minutes to pass all six checks below. ## Quick summary | Goal | What to add on Astro | Check | Time | |---|---|---|---| | A robots.txt exists at the root | `public/robots.txt`, or a `src/pages/robots.txt.ts` endpoint | D1 | 3 min | | AI crawlers get their own rules | Named `User-agent` groups for GPTBot, ClaudeBot, PerplexityBot and the rest | B2 | 5 min | | Agents can enumerate every page | `npx astro add sitemap`, plus `site` in `astro.config.mjs` | D2 | 5 min | | A feed exists and is declared | `@astrojs/rss` at `src/pages/rss.xml.js`, plus `` | E5 | 7 min | | A machine-readable summary at the root | `src/pages/llms.txt.ts` built from your content collections | C2 | 5 min | | The HTML carries real content | Avoid `client:only` on anything that holds article text | C3, E3 | 0 min if clean | Astro's own documentation site is the honest illustration. On 29 August 2026, `docs.astro.build/robots.txt` returns exactly two lines, `User-agent: *` and `Allow: /`, with no AI crawler group and no `Sitemap:` line, and `docs.astro.build/llms.txt` returns 404. There is no upstream example to copy, so every file below comes from the primary documentation. ## What Astro's architecture already gives you Islands architecture is the reason Astro starts ahead. Astro describes the pattern as "rendering the majority of your page to fast, static HTML with smaller 'islands' of JavaScript added when interactivity or personalization is needed on the page" ([Astro docs, islands architecture](https://docs.astro.build/en/concepts/islands/)). A React component in an Astro page is server-rendered to HTML during the build and hydrated in the browser afterwards, so the crawler that never runs JavaScript still receives the text. That matters because the retrieval crawlers behind ChatGPT, Claude and Perplexity do not run JavaScript. Per [Vercel and MERJ, 17 December 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler): "none of the major AI crawlers currently render JavaScript," covering GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot. Nothing published since contradicts it. One directive undoes that advantage. From Astro's template directives reference, `client:only={string}` "**skips** HTML server rendering, and renders only on the client." A component marked `client:only="react"` produces an empty element in the built HTML. Put your article body or pricing table inside one and every non-rendering crawler gets a blank page while your browser looks perfect. That is the only Astro-specific way to fail C3 and E3, so grep for it first: ```bash grep -rn 'client:only' src/ ``` If nothing in that list wraps content a reader would quote, your rendering is fine and the rest of the work is files. ## Path 1: the four discovery files, by hand This path needs no dependencies beyond two official Astro packages. **1. Set your deployed URL.** In `astro.config.mjs`, add `site: 'https://example.com'`. Three of the four files need it to build absolute URLs, the sitemap integration requires it, and it must begin with `http://` or `https://`. **2. Write robots.txt.** The static option is a plain file at `public/robots.txt`, which Astro copies to the root untouched. The better option, adapted from [Astro's sitemap integration docs](https://docs.astro.build/en/guides/integrations-guide/sitemap/), generates it from the `site` value so the two cannot drift apart. Create `src/pages/robots.txt.ts`: ```ts import type { APIRoute } from 'astro'; const getRobotsTxt = (sitemapURL: URL) => `User-agent: * Allow: / User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / Sitemap: ${sitemapURL.href} `; export const GET: APIRoute = ({ site }) => { const sitemapURL = new URL('sitemap-index.xml', site); return new Response(getRobotsTxt(sitemapURL)); }; ``` The second group is what B2 looks for. Naming those crawlers matters more than the value you give them, because Google's robots.txt specification says only the most specific matching group applies and all other groups are ignored. A bot with its own group never reads your `*` group, so a named `Allow` is a statement rather than a redundancy. To permit retrieval while refusing training, split them: `Allow: /` for OAI-SearchBot, Claude-SearchBot and PerplexityBot, `Disallow: /` for GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. Check the [current crawler tokens](https://aiscan.site/blog/ai-crawler-user-agent-list-2026) first, because five operators now run three separate bots each. **3. Add the sitemap.** Run `npx astro add sitemap`, which installs `@astrojs/sitemap` (3.7.3, published 26 May 2026) and adds `sitemap()` to `integrations`. The build then writes `sitemap-index.xml` and `sitemap-0.xml`. Two documented limits catch people out: it "cannot generate sitemap entries for dynamic routes in SSR mode," and nothing links the file automatically. Add the link in your layout ``: ```astro ``` **4. Add the feed and declare it.** Install `@astrojs/rss` (4.0.19, published 30 June 2026) and create `src/pages/rss.xml.js` exporting a `GET` that returns `rss({ title, description, site: context.site, items })`, building `items` from `getCollection()`. E5 fails on the declaration far more often than on the feed itself, so add this to your `` too: ```astro ``` One documented mismatch: the RSS helper "produces links with a trailing slash by default, no matter what value you have configured for `trailingSlash`." If your config sets `trailingSlash: "never"`, pass `trailingSlash: false` to `rss()`. **5. Add llms.txt.** Content collections make this a build step rather than a file you maintain: an endpoint at `src/pages/llms.txt.ts` can call `getCollection()` and emit an H1, `##` sections and Markdown links from the same source your pages use. The full version of this step is in [How to publish a valid llms.txt on Astro](https://aiscan.site/blog/llms-txt-astro), and the [llms.txt generator](https://aiscan.site/llms-txt-generator) will draft one from a live URL. ## Path 2: integrations, if you would rather not write endpoints Documentation sites built on Starlight have a plugin for the hardest part. `starlight-llms-txt` (0.11.0, published 1 July 2026) is maintained by Starlight maintainer Chris Swithinbank and generates `llms.txt`, `llms-full.txt` and `llms-small.txt` from your existing docs. Install with `npm i starlight-llms-txt`, add `starlightLlmsTxt()` to the `plugins` array inside `starlight()`, set `site`, then preview at `localhost:4321/llms.txt`. Steps 3 and 4 are unchanged. No Astro plugin writes robots.txt with AI crawler groups for you, so step 2 stays manual either way. ## The trailing-slash rule that breaks three endpoints at once This trap produces the strangest scan results, because the file exists, the build succeeds, and the URL still 404s. From [Astro's endpoints guide](https://docs.astro.build/en/guides/endpoints/): "endpoints whose URLs include a file extension (e.g. `src/pages/sitemap.xml.ts`) can only be accessed without a trailing slash (e.g. `/sitemap.xml`), regardless of your `build.trailingSlash` configuration." Every file in Path 1 is an extension-bearing endpoint. `/robots.txt/`, `/rss.xml/` and `/llms.txt/` all return 404 while their slashless forms return 200. If your host, CDN or middleware appends a trailing slash to normalise URLs, it hands every crawler a 404 for your whole discovery layer at once. Test the slashless form, and exempt those three paths from any slash-adding redirect rule. The related setting is `prerender`. Astro's endpoints guide states that routes "will be rendered on demand by default in `server` mode," and that in `static` mode you opt out per endpoint. If you added an adapter and these files started behaving dynamically, that flag is where to look. ## Confirm it with a scan Run the scan first. It checks all six of these at once and names the row that failed: ```bash npx aiscan-cli example.com ``` No account, no key, and you can paste the URL at [aiscan.site](https://aiscan.site/) instead if you prefer a browser. The rows that matter here are **D1** robots.txt present, **B2** explicit AI bot rules, **D2** sitemap, **C2** llms.txt, **C3** and **E3** server-rendered HTML, and **E5** feed declared. If you would rather check by hand, these five commands cover the same ground: ```bash curl -sI https://example.com/robots.txt | head -1 # expect 200 curl -s https://example.com/robots.txt | grep -i gptbot # expect a match curl -sI https://example.com/sitemap-index.xml | head -1 # expect 200 curl -s https://example.com/ | grep -o 'rel="alternate"[^>]*rss[^>]*' curl -s https://example.com/ | grep -c '` | Whether the items carry usable descriptions | | llms.txt has an H1, sections and links | Whether those links resolve or the summaries are true | | The served HTML carries headings and body text | Whether that text answers the reader's question | The right column is editorial and no scanner closes it. A perfect score means an agent can reach and parse your site, not that it will find your answer worth citing. ## A ten-minute checklist - [ ] `site` in `astro.config.mjs` matches the deployed origin - [ ] `grep -rn 'client:only' src/` returns nothing wrapping article text - [ ] `/robots.txt` returns 200 with at least one named AI crawler group - [ ] `/sitemap-index.xml` returns 200 and `` is in the layout head - [ ] `/rss.xml` returns 200 and `` is in the head - [ ] `/llms.txt` returns 200 with an H1, one `##` section and Markdown links - [ ] None of those four URLs redirect to a trailing-slash form Re-run it after any change to your adapter, your host's redirect rules or `trailingSlash`. Those three break discovery files without touching the code that generates them. ## Who can skip most of this A single-page Astro marketing site needs neither a sitemap nor a feed, and llms.txt on a five-page site adds nothing an agent could not read directly. Do step 2 and stop. If your Astro front end pulls from a headless WordPress CMS, that CMS domain has its own robots.txt, sitemap and schema, and none of the work above touches it. [ThinkRank](https://thinkrank.ai) handles all four from one WordPress plugin, which avoids two or three SEO plugins each claiming ownership of robots.txt, and it migrates settings from Rank Math, Yoast, All in One SEO and SEOPress so nothing is re-entered. ## Start here Run `npx aiscan-cli` against your Astro site and read the D1, B2, D2, C2, E5 and C3 rows. Fix whichever files it names, in checklist order, then re-scan. The same six checks apply to every framework: the [Next.js version of this setup](https://aiscan.site/blog/ai-readiness-setup-nextjs) covers the same ground with `app/robots.ts` and `app/sitemap.ts`, and [the teardown of a Next.js site that ranks on Google while being invisible to ChatGPT](https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers) shows what a rendering failure looks like. Full check definitions live on [discoverability](https://aiscan.site/docs/checks/discoverability), [content](https://aiscan.site/docs/checks/content) and [bot access](https://aiscan.site/docs/checks/bot-access), and every platform guide is indexed at [aiscan.site/guides](https://aiscan.site/guides).