The complete AI readiness setup for Next.js in 2026

Table of contents

Quick summary
The mechanism: why a Next.js site can pass SEO and still be blank
Path 1: App Router (Next.js 13.4 and later)
1. Generate robots.txt with AI bot rules (D1, B2)
2. Generate the sitemap (D2)
3. Publish llms.txt (C2)
4. Add a feed and declare it (E5)
5. Make sure the body is in the response (C3, E3)
Path 2: Pages Router or `output: 'export'`
How to verify it worked
Maintenance, and who does not need this
Next step

Next.js gives you five of the files AI crawlers look for, and four of them are one file away. The setup below takes about 40 minutes on an App Router project and moves a typical Next.js site from a partial AIScan score to a clean pass on B2, C2, D1, D2, E5, plus the one that actually decides whether any of it matters: C3/E3.

Quick summary

What an agent looks for	Check	Where it lives in Next.js	Time
robots.txt exists	D1	`app/robots.ts`	5 min
Explicit AI bot rules	B2	rules array in `app/robots.ts`	10 min
XML sitemap	D2	`app/sitemap.ts`	10 min
llms.txt at the root	C2	`public/llms.txt`	10 min
Feed, declared in `<head>`	E5	`app/rss.xml/route.ts` + `alternates.types`	10 min
Article body in the HTML response	C3 / E3	wherever your data fetching happens	varies

Verified against the Next.js documentation at version 16.3.3, 28 August 2026. Paths are App Router unless the Pages Router is called out.

The mechanism: why a Next.js site can pass SEO and still be blank

The retrieval crawlers that build the index ChatGPT, Claude and Perplexity answer from do not run JavaScript. As last measured (Vercel and MERJ, December 2024, with no operator publishing anything since to change it), GPTBot, OAI-SearchBot, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot all fetch your page and read only what the server already put in the response. Googlebot renders, so Gemini renders. The others read HTML and stop.

That splits Next.js sites into two groups that look identical in a browser. If your content is in the server render, every crawler sees it. If it arrives from a useEffect call, a next/dynamic import with ssr: false, or a component gated behind typeof window, the crawler gets your layout and nothing else.

Here is a detail almost nobody has checked. Next.js streams metadata by default and keeps a regex of user agents that need blocking metadata instead, because they parse the <head> before the stream finishes. That list is HTML_LIMITED_BOT_UA_RE in packages/next/src/shared/lib/router/utils/html-bots.ts. Read it on 28 August 2026 and it names Google crawlers, Bingbot, applebot, Twitterbot, LinkedInBot, Slackbot, Discordbot and about a dozen more. GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot are not in it. Next.js treats them as DOM-capable clients, because from the framework's point of view they are. They just never execute what they receive.

So the framework's own default assumes a crawler that behaves like a browser, and the crawlers that matter most for AI visibility do not. That is the gap this setup closes.

Path 1: App Router (Next.js 13.4 and later)

1. Generate robots.txt with AI bot rules (D1, B2)

Create app/robots.ts. The rules property takes an array, and each entry becomes its own User-Agent group:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: '/api/' },
      { userAgent: ['OAI-SearchBot', 'Claude-SearchBot', 'PerplexityBot'], allow: '/' },
      { userAgent: ['GPTBot', 'ClaudeBot', 'CCBot'], allow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

Two things make this pass B2 rather than just D1. The AI bots are named explicitly, because a scanner cannot tell an intentional allow from an absent rule if you only ship User-agent: *. And search bots sit in a separate group from training bots, so you can change your mind about training without losing the retrieval index. Google's robots.txt spec matches one group per crawler, the most specific one, so a bot with its own group never reads your * group. Put everything a named bot needs inside its own block.

If you have a static app/robots.txt instead, delete it before adding robots.ts; they collide on the same route.

2. Generate the sitemap (D2)

app/sitemap.ts, exporting an array of URL objects:

import type { MetadataRoute } from 'next'

export default async function sitemap(): MetadataRoute.Sitemap {
  const posts = await getPosts()
  return [
    { url: 'https://example.com', lastModified: new Date(), changeFrequency: 'weekly', priority: 1 },
    ...posts.map((p) => ({
      url: `https://example.com/blog/${p.slug}`,
      lastModified: p.updatedAt,
    })),
  ]
}

Both robots.js and sitemap.js are special Route Handlers that Next.js caches by default unless they use a request-time API. A sitemap built from a CMS will go stale until the next deploy, so set a revalidate on the fetch that loads your posts.

3. Publish llms.txt (C2)

An llms.txt file is a Markdown index of your best pages. The spec requires exactly one thing: an H1 with the site name. Everything after that is optional.

Static file, public/llms.txt:

# Example

> One sentence on what this site is for.

## Docs
- [Getting started](https://example.com/docs/start): install and first run
- [API reference](https://example.com/docs/api): every endpoint

## Blog
- [Latest post](https://example.com/blog/latest): what it argues

Anything in public/ is served from the base URL, so this lands at /llms.txt with no route needed. If your list changes with your content, generate it from the same data source your sitemap uses. A route handler at app/llms.txt/route.ts returning text/plain works the same way. The Next.js llms.txt guide covers the generated version line by line, and our llms.txt generator will build a first draft from a URL.

Be honest about what this buys. Ahrefs looked at 137,210 domains in June 2026: 28% publish a valid llms.txt and 97% of those files got zero requests in May, and Google says its search ignores the file. It costs ten minutes, and it is the file agentic browsers are now audited against. That is the case for shipping one.

4. Add a feed and declare it (E5)

Route handlers can return anything. The Next.js docs show a feed at app/rss.xml/route.ts:

export async function GET() {
  return new Response(
    `<?xml version="1.0" encoding="UTF-8" ?><rss version="2.0"><channel>
      <title>Example</title><link>https://example.com</link>
      <description>What this site publishes</description>
    </channel></rss>`,
    { headers: { 'Content-Type': 'text/xml' } }
  )
}

The half everyone skips is declaring it. In your root app/layout.tsx:

export const metadata = {
  alternates: {
    canonical: 'https://example.com',
    types: { 'application/rss+xml': 'https://example.com/rss.xml' },
  },
}

That emits <link rel="alternate" type="application/rss+xml" href="…"> into the head. A feed nothing links to is a feed nothing finds.

5. Make sure the body is in the response (C3, E3)

This is the step that decides whether the other four matter. 'use client' is not the problem. The Next.js docs are explicit that client components and the RSC payload are both used to prerender HTML. The real causes are narrower: data fetched in useEffect, next/dynamic with { ssr: false }, and markup behind a typeof window guard. Move the fetch into the server component, or into a loading.tsx boundary that renders real content rather than a skeleton.

The framework's own caching documentation puts the failure mode plainly: a page that loads for a person can fail to render for a crawler. Our Next.js invisibility teardown walks through proving it on a live site.

Path 2: Pages Router or `output: 'export'`

The app/ metadata conventions do not exist here. Everything becomes a static file in public/, which ships as-is:

Write public/robots.txt by hand, with the same named AI bot groups from step 1 and a Sitemap: line.
Write public/llms.txt by hand, or generate it in your build script before next build.
Generate public/sitemap.xml in a prebuild script from your content directory, or serve it from pages/sitemap.xml.tsx with getServerSideProps if you are not exporting statically.
Write the feed to public/rss.xml in the same build step, and add the <link rel="alternate"> tag in pages/_document.tsx.
For the body: Pages Router sites get their HTML from getStaticProps or getServerSideProps. A page using neither, with client-side fetching only, ships an empty shell. That is the one to audit.

Static export is the safest of these for crawlers, since every route is already HTML on disk, as long as the data was fetched at build time.

How to verify it worked

Start with a scan. Run npx aiscan-cli example.com, or paste the URL at aiscan.site. It is free, no account, and it returns each check by ID: D1 robots.txt present, B2 explicit AI bot rules, D2 sitemap, C2 llms.txt, E5 feed discovery, and C3/E3 server-rendered content with the word count and heading count it actually extracted. The word count is the number to look at. If the browser shows 1,800 words and the scan reports 90, you have a rendering problem, not a content problem.

If you would rather check by hand, four commands cover the same ground:

curl -s https://example.com/robots.txt | grep -iE 'gptbot|claudebot|perplexity|oai-'
curl -sI https://example.com/sitemap.xml | head -1
curl -sI https://example.com/llms.txt | grep -i content-type
curl -sL --compressed https://example.com/ \
| perl -0777 -pe 's/<(script|style|noscript|svg)\b.*?<\/\1>//gsi; s/<!--.*?-->//gs; s/<[^>]+>/ /gs' \
| tr -s '[:space:]' ' ' | wc -w

Pass marks: the first prints your AI bot lines, the next two return 200, and the last returns a number close to what you see on the page. Use that Perl one-liner rather than the sed 's/<[^>]*>//g' version that circulates. sed cannot strip multi-line <script> blocks and counts your JavaScript bundle as prose. On nextjs.org the difference is 626 words versus 1,041.

One more, specific to React: curl -sL https://example.com/ | grep -c -- ''. That comment marks a Suspense boundary React gave up on and handed to the browser. Any count above zero means part of your page reached the crawler as nothing at all.

What AIScan cannot see. It reads one URL as an anonymous client. It cannot tell you whether your CDN serves a different response to GPTBot's IP range, whether your sitemap.ts cache is stale, or whether the pages listed in llms.txt are the ones worth citing. Those are judgement calls and log-file questions.

Maintenance, and who does not need this

Re-scan after any change to data fetching, after a Next.js major upgrade, and after adding a CDN or bot-management layer in front of the app. That last one changes what crawlers receive without touching your code. If you also run a WordPress site alongside the app, ThinkRank handles the same surface there from one plugin: robots.txt, robots meta, schema, sitemaps and llms.txt together, instead of three plugins overwriting each other's robots.txt. It also imports existing settings from Rank Math, Yoast, All in One SEO and SEOPress, so nothing gets re-entered. Running a Shopify storefront instead? StoreSEO builds llms.txt from live products, collections, pages and articles and includes an agents.md editor; the Shopify setup guide has that path in full.

A purely internal app behind auth needs none of this. Neither does a site you deliberately keep out of AI answers, though blocking training and blocking retrieval are separate decisions, and only one of them costs you citations.

Next step

Scan the site and read six rows: D1, B2, D2 on discoverability, C2 and C3/E3 on content, and the bot rules on bot access. Fix whichever fails using the matching step above. Platform-specific notes live on the Next.js platform page, and every other setup walkthrough is indexed at aiscan.site/guides.

Frequently asked questions

Does 'use client' stop AI crawlers from seeing my content?

No. The Next.js documentation is explicit that client components and the RSC payload are both used to prerender HTML, so a client component still produces server-rendered markup. The directive marks a hydration boundary, not an SSR opt-out. The real causes of an empty response are data fetched in useEffect, next/dynamic with { ssr: false }, and markup wrapped in a typeof window check.

My scan says C3 failed and reports 90 words, but the page clearly has 1,800. What is wrong?

The scanner reads the raw HTML response without running JavaScript, so 90 words means the server sent a shell. Run curl -sL --compressed on the URL and count the words yourself; if you get a similar low number, the article body is arriving client-side. Check for useEffect data fetching in the page component and for a next/dynamic import with ssr: false in the render path.

I added app/robots.ts but /robots.txt still returns the old file. Why?

Two causes. A static app/robots.txt or a public/robots.txt collides with the generated route on the same path, so delete whichever one you are not using. Second, robots.js is a special Route Handler that Next.js caches by default, so a stale build output can persist until you redeploy. Rebuild, then re-request with a cache-busting query string.

Should I block GPTBot and ClaudeBot from training on my content?

That is a business decision, but keep it separate from retrieval. OpenAI states that each setting is independent, so you can allow OAI-SearchBot to appear in ChatGPT search answers while disallowing GPTBot for training. Anthropic warns that disabling Claude-User prevents its system retrieving your content for a user query, which may reduce visibility. Blocking training costs you nothing in citations; blocking the search and user bots does.

Do I need llms.txt on a Next.js site?

It is optional and cheap. Ahrefs found across 137,210 domains in June 2026 that 28% publish a valid llms.txt and 97% of those files received zero requests in May 2026, and Google has said its search ignores the file. It costs about ten minutes as a static file in public/, and Chrome's Lighthouse Agentic Browsing category audits for it, which is the argument for shipping one anyway.

My sitemap.ts works locally but the deployed sitemap is missing new posts. How do I fix it?

sitemap.js is a Route Handler cached by default unless it uses a request-time API or a dynamic config option. If it builds its list from a CMS fetch, that fetch is cached at build time and the sitemap freezes at the last deploy. Add a revalidate option to the fetch that loads your posts, or mark the route dynamic, then redeploy and re-request the URL.

Does the Pages Router support app/robots.ts and app/sitemap.ts?

No. Those file conventions live in the app directory and are App Router only. On the Pages Router, put robots.txt, llms.txt and rss.xml in public/ as static files, generate sitemap.xml in a prebuild script or serve it from pages/sitemap.xml.tsx with getServerSideProps, and add the feed link tag in pages/_document.tsx.

Are AI crawlers on Next.js's htmlLimitedBots list?

No. The default list is the HTML_LIMITED_BOT_UA_RE regex in packages/next/src/shared/lib/router/utils/html-bots.ts, and as read on 28 August 2026 it names Google crawlers, Bingbot, applebot, Twitterbot, LinkedInBot, Slackbot, Discordbot and similar preview bots. GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot are absent, so Next.js streams metadata to them as it would to a browser. Overriding the config replaces the default list entirely, so most sites should leave it alone and fix the render path instead.

The complete AI readiness setup for Next.js in 2026

Quick summary

The mechanism: why a Next.js site can pass SEO and still be blank

Path 1: App Router (Next.js 13.4 and later)

1. Generate robots.txt with AI bot rules (D1, B2)

2. Generate the sitemap (D2)

3. Publish llms.txt (C2)

4. Add a feed and declare it (E5)

5. Make sure the body is in the response (C3, E3)

Path 2: Pages Router or `output: 'export'`

How to verify it worked

Maintenance, and who does not need this

Next step

Frequently asked questions

Related guides

The State of AI Agent Readiness in 2026: 473 Sites Measured

How to publish a valid llms.txt on Framer

The complete AI readiness setup for Replit in 2026

How to publish a valid llms.txt on Replit

Quick summary

The mechanism: why a Next.js site can pass SEO and still be blank

Path 1: App Router (Next.js 13.4 and later)

1. Generate robots.txt with AI bot rules (D1, B2)

2. Generate the sitemap (D2)

3. Publish llms.txt (C2)

4. Add a feed and declare it (E5)

5. Make sure the body is in the response (C3, E3)

Path 2: Pages Router or output: 'export'

How to verify it worked

Maintenance, and who does not need this

Next step

Frequently asked questions

Related guides

The State of AI Agent Readiness in 2026: 473 Sites Measured

How to publish a valid llms.txt on Framer

The complete AI readiness setup for Replit in 2026

How to publish a valid llms.txt on Replit

Path 2: Pages Router or `output: 'export'`