---
title: "The complete AI readiness setup for Next.js in 2026"
slug: ai-readiness-setup-nextjs
published: 2026-08-28T08:08:31.410332+00:00
updated: 2026-08-28T08:08:31.410332+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: platform:nextjs, check:D1, check:D2, check:B2, check:C2, check:C3, check:E3, check:E5, Next.js, App Router, robots.txt, llms.txt, sitemap, AI crawlers, server-side rendering
description: "Set up AI readiness on Next.js: robots.ts with AI bot rules, sitemap.ts, llms.txt, an RSS feed, and the render check that decides if crawlers see anything."
url: https://aiscan.site/blog/ai-readiness-setup-nextjs
---
Next.js gives you five of the files AI crawlers look for, and four of them are one file away. The setup below takes about 40 minutes on an App Router project and moves a typical Next.js site from a partial AIScan score to a clean pass on B2, C2, D1, D2, E5, plus the one that actually decides whether any of it matters: C3/E3.
## Quick summary
| What an agent looks for | Check | Where it lives in Next.js | Time |
|---|---|---|---|
| robots.txt exists | D1 | `app/robots.ts` | 5 min |
| Explicit AI bot rules | B2 | rules array in `app/robots.ts` | 10 min |
| XML sitemap | D2 | `app/sitemap.ts` | 10 min |
| llms.txt at the root | C2 | `public/llms.txt` | 10 min |
| Feed, declared in `` | E5 | `app/rss.xml/route.ts` + `alternates.types` | 10 min |
| Article body in the HTML response | C3 / E3 | wherever your data fetching happens | varies |
Verified against the Next.js documentation at version 16.3.3, 28 August 2026. Paths are App Router unless the Pages Router is called out.
## The mechanism: why a Next.js site can pass SEO and still be blank
The retrieval crawlers that build the index ChatGPT, Claude and Perplexity answer from do not run JavaScript. As last measured (Vercel and MERJ, December 2024, with no operator publishing anything since to change it), GPTBot, OAI-SearchBot, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot all fetch your page and read only what the server already put in the response. Googlebot renders, so Gemini renders. The others read HTML and stop.
That splits Next.js sites into two groups that look identical in a browser. If your content is in the server render, every crawler sees it. If it arrives from a `useEffect` call, a `next/dynamic` import with `ssr: false`, or a component gated behind `typeof window`, the crawler gets your layout and nothing else.
Here is a detail almost nobody has checked. Next.js streams metadata by default and keeps a regex of user agents that need blocking metadata instead, because they parse the `` before the stream finishes. That list is `HTML_LIMITED_BOT_UA_RE` in `packages/next/src/shared/lib/router/utils/html-bots.ts`. Read it on 28 August 2026 and it names Google crawlers, Bingbot, applebot, Twitterbot, LinkedInBot, Slackbot, Discordbot and about a dozen more. **GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot are not in it.** Next.js treats them as DOM-capable clients, because from the framework's point of view they are. They just never execute what they receive.
So the framework's own default assumes a crawler that behaves like a browser, and the crawlers that matter most for AI visibility do not. That is the gap this setup closes.
## Path 1: App Router (Next.js 13.4 and later)
### 1. Generate robots.txt with AI bot rules (D1, B2)
Create `app/robots.ts`. The `rules` property takes an array, and each entry becomes its own `User-Agent` group:
```ts
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{ userAgent: '*', allow: '/', disallow: '/api/' },
{ userAgent: ['OAI-SearchBot', 'Claude-SearchBot', 'PerplexityBot'], allow: '/' },
{ userAgent: ['GPTBot', 'ClaudeBot', 'CCBot'], allow: '/' },
],
sitemap: 'https://example.com/sitemap.xml',
}
}
```
Two things make this pass B2 rather than just D1. The AI bots are named explicitly, because a scanner cannot tell an intentional allow from an absent rule if you only ship `User-agent: *`. And search bots sit in a separate group from training bots, so you can change your mind about training without losing the retrieval index. Google's robots.txt spec matches one group per crawler, the most specific one, so a bot with its own group never reads your `*` group. Put everything a named bot needs inside its own block.
If you have a static `app/robots.txt` instead, delete it before adding `robots.ts`; they collide on the same route.
### 2. Generate the sitemap (D2)
`app/sitemap.ts`, exporting an array of URL objects:
```ts
import type { MetadataRoute } from 'next'
export default async function sitemap(): MetadataRoute.Sitemap {
const posts = await getPosts()
return [
{ url: 'https://example.com', lastModified: new Date(), changeFrequency: 'weekly', priority: 1 },
...posts.map((p) => ({
url: `https://example.com/blog/${p.slug}`,
lastModified: p.updatedAt,
})),
]
}
```
Both `robots.js` and `sitemap.js` are special Route Handlers that Next.js caches by default unless they use a request-time API. A sitemap built from a CMS will go stale until the next deploy, so set a `revalidate` on the fetch that loads your posts.
### 3. Publish llms.txt (C2)
An llms.txt file is a Markdown index of your best pages. The spec requires exactly one thing: an H1 with the site name. Everything after that is optional.
Static file, `public/llms.txt`:
```md
# Example
> One sentence on what this site is for.
## Docs
- [Getting started](https://example.com/docs/start): install and first run
- [API reference](https://example.com/docs/api): every endpoint
## Blog
- [Latest post](https://example.com/blog/latest): what it argues
```
Anything in `public/` is served from the base URL, so this lands at `/llms.txt` with no route needed. If your list changes with your content, generate it from the same data source your sitemap uses. A route handler at `app/llms.txt/route.ts` returning `text/plain` works the same way. The [Next.js llms.txt guide](https://aiscan.site/blog/llms-txt-nextjs) covers the generated version line by line, and our [llms.txt generator](https://aiscan.site/llms-txt-generator) will build a first draft from a URL.
Be honest about what this buys. Ahrefs looked at 137,210 domains in June 2026: 28% publish a valid llms.txt and 97% of those files got zero requests in May, and Google says its search ignores the file. It costs ten minutes, and it is the file agentic browsers are now audited against. That is the case for shipping one.
### 4. Add a feed and declare it (E5)
Route handlers can return anything. The Next.js docs show a feed at `app/rss.xml/route.ts`:
```ts
export async function GET() {
return new Response(
`Examplehttps://example.com
What this site publishes`,
{ headers: { 'Content-Type': 'text/xml' } }
)
}
```
The half everyone skips is declaring it. In your root `app/layout.tsx`:
```ts
export const metadata = {
alternates: {
canonical: 'https://example.com',
types: { 'application/rss+xml': 'https://example.com/rss.xml' },
},
}
```
That emits `` into the head. A feed nothing links to is a feed nothing finds.
### 5. Make sure the body is in the response (C3, E3)
This is the step that decides whether the other four matter. `'use client'` is not the problem. The Next.js docs are explicit that client components and the RSC payload are both used to prerender HTML. The real causes are narrower: data fetched in `useEffect`, `next/dynamic` with `{ ssr: false }`, and markup behind a `typeof window` guard. Move the fetch into the server component, or into a `loading.tsx` boundary that renders real content rather than a skeleton.
The framework's own caching documentation puts the failure mode plainly: a page that loads for a person can fail to render for a crawler. Our [Next.js invisibility teardown](https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers) walks through proving it on a live site.
## Path 2: Pages Router or `output: 'export'`
The `app/` metadata conventions do not exist here. Everything becomes a static file in `public/`, which ships as-is:
1. Write `public/robots.txt` by hand, with the same named AI bot groups from step 1 and a `Sitemap:` line.
2. Write `public/llms.txt` by hand, or generate it in your build script before `next build`.
3. Generate `public/sitemap.xml` in a prebuild script from your content directory, or serve it from `pages/sitemap.xml.tsx` with `getServerSideProps` if you are not exporting statically.
4. Write the feed to `public/rss.xml` in the same build step, and add the `` tag in `pages/_document.tsx`.
5. For the body: Pages Router sites get their HTML from `getStaticProps` or `getServerSideProps`. A page using neither, with client-side fetching only, ships an empty shell. That is the one to audit.
Static export is the safest of these for crawlers, since every route is already HTML on disk, as long as the data was fetched at build time.
## How to verify it worked
**Start with a scan.** Run `npx aiscan-cli example.com`, or paste the URL at [aiscan.site](https://aiscan.site/). It is free, no account, and it returns each check by ID: **D1** robots.txt present, **B2** explicit AI bot rules, **D2** sitemap, **C2** llms.txt, **E5** feed discovery, and **C3/E3** server-rendered content with the word count and heading count it actually extracted. The word count is the number to look at. If the browser shows 1,800 words and the scan reports 90, you have a rendering problem, not a content problem.
**If you would rather check by hand**, four commands cover the same ground:
```bash
curl -s https://example.com/robots.txt | grep -iE 'gptbot|claudebot|perplexity|oai-'
curl -sI https://example.com/sitemap.xml | head -1
curl -sI https://example.com/llms.txt | grep -i content-type
curl -sL --compressed https://example.com/ \
| perl -0777 -pe 's/<(script|style|noscript|svg)\b.*?<\/\1>//gsi; s///gs; s/<[^>]+>/ /gs' \
| tr -s '[:space:]' ' ' | wc -w
```
Pass marks: the first prints your AI bot lines, the next two return 200, and the last returns a number close to what you see on the page. Use that Perl one-liner rather than the `sed 's/<[^>]*>//g'` version that circulates. `sed` cannot strip multi-line `