Your Next.js Site Ranks on Google and Is Invisible to ChatGPT: How to Prove It

Googlebot runs your JavaScript. The crawlers that build ChatGPT's and Claude's indexes, as far as anyone has measured, do not. That gap is why a React or Next.js site can hold page one on Google and still be missing from the answer an AI assistant gives about your own product.

You can find out which side of the gap you are on in about thirty seconds, and the fix is usually a configuration change rather than a rewrite. This guide covers the mechanism, the exact commands, four live sites measured on 28 August 2026 (one of them ours, and it failed), and the seven patterns in Next.js that cause it.

Quick summary

The short answer: if a plain curl of your page returns fewer than about 150 words of readable text and no <h2> headings, retrieval crawlers are getting an empty page from you, whatever Google shows.

The honest caveat, stated up front: the last first-party measurement of AI-crawler JavaScript execution is Vercel and MERJ's study from 17 December 2024. No crawler operator publishes a rendering statement, and no independent 2026 replication exists. The correct claim is "as last measured, they did not render, and nobody has said otherwise since", not "AI crawlers cannot render JavaScript."

Why Google sees your React site and ChatGPT does not

Two pipelines, one page.

Google's is documented and has two stages. Per Google's JavaScript SEO basics (last updated 4 March 2026): "Googlebot queues all pages with a 200 HTTP status code for rendering… Once Google's resources allow, a headless Chromium renders the page and executes the JavaScript." And then the sentence that matters: "Google also uses the rendered HTML to index the page."

So Google indexes the DOM after your bundle has run. Whatever your React app paints, Google reads.

The retrieval crawlers behind AI assistants take one stage. Vercel and MERJ instrumented nextjs.org plus two job boards and reported, verbatim: "The results consistently show that none of the major AI crawlers currently render JavaScript. This includes: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ByteDance (Bytespider), Perplexity (PerplexityBot)."

They also measured the near miss: "while ChatGPT and Claude crawlers do fetch JavaScript files (ChatGPT: 11.50%, Claude: 23.84% of requests), they don't execute them. They can't read client-side rendered content."

Two crawlers sit on the Google side of the line. Vercel found that "Google's Gemini leverages Googlebot's infrastructure, enabling full JavaScript rendering", and that "AppleBot renders JavaScript through a browser-based crawler, similar to Googlebot." Google's own docs explain why: Google-Extended "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." It is a permission token, not a crawler, so Gemini inherits Googlebot's renderer.

The distinction people get wrong

A retrieval crawler is not an agentic browser, and conflating them produces bad advice in both directions.

Both statements are true at once. Your React site can be perfectly usable by an agent that opens it in a browser and completely absent from the index ChatGPT answers from when no browser is involved. The second case is the one that decides whether you get mentioned at all.

Step 1: run the scan

The fastest route is a scan, because the checks are already written down and it reads the page the way a non-rendering fetcher does.

Paste your URL at aiscan.site, or run it locally with no account:

npx aiscan-cli yoursite.com

Four checks answer this question directly, and they are worth reading in this order:

• E3 — heading hierarchy and server-rendered text is the one that matters here. AIScan strips scripts and styles from the HTML your server actually returns, counts the remaining words and the <h2>/<h3> outline, and scores 2 points for 150+ words of server-rendered text (1 point for 50+), plus 2 points for two or more <h2> sections. A client-rendered app scores 0 or 1.
• C3 — structured HTML wants a single <h1>, a non-empty <title>, a meta description and at least one JSON-LD block. A shell page fails most of these because the components that emit them never run on the server.
• C1 — Markdown content negotiation re-requests your homepage with Accept: text/markdown. Passing it is the clean escape hatch from this whole problem, and the section below shows a site that does it.
• C2 — llms.txt is a curated index of your content in Markdown. Worth 6 points and about two minutes on Next.js, covered in our Next.js llms.txt guide.

Step 2: check it by hand

If you would rather see the bytes yourself, three commands do it. Every one below was run on 28 August 2026 and the outputs quoted are real.

Count the words a non-JavaScript fetcher receives. Use perl, not sed. The sed 's/<[^>]*>/ /g' one-liner that circulates in SEO posts cannot strip multi-line <script> blocks, so it counts your JavaScript as prose. On nextjs.org the sed version reported 626 words and the correct version reported 1,041.

URL='https://yoursite.com/'
curl -sL --compressed "$URL" \
| perl -0777 -pe 's/<(script|style|noscript|svg)\b.*?<\/\1>//gsi; s/<!--.*?-->//gs; s/<[^>]+>/ /gs' \
| tr -s '[:space:]' ' ' | wc -w

Under 150 words on a content page means you have the problem.

Look for a failed server render. React encodes Suspense boundaries as HTML comments. The markers are defined in React's own source, in ReactFizzConfigDOM.js, under the comment "Suspense boundaries are encoded as comments":

curl -sL --compressed "$URL" | grep -o -e '<!--\$-->' -e '<!--\$?-->' -e '<!--\$!-->' | sort | uniq -c

Any count above zero for <!--$!--> is a bug, not a style choice. It is the single highest-signal line in this entire article and it is undocumented outside the React source.

Find out whether your text is real HTML or only flight data. This is the check almost nobody runs, and it is where confident conclusions go wrong.

NEEDLE='Your exact H1 text'
RAW=$(curl -sL --compressed "$URL")
echo "$RAW" | grep -c "$NEEDLE"                                                  # anywhere in source
echo "$RAW" | perl -0777 -pe 's/<script\b.*?<\/script>//gsi' | grep -c "$NEEDLE"  # outside <script>

If the first number is above zero and the second is zero, your content exists only inside the React Server Components flight payload, the self.__next_f.push([1,"…"]) blocks Next.js writes into <script> tags.

Be precise about what that means, because both overclaims are wrong. The payload is in the response body, so a naive "is my text in the HTML" test passes. Vercel's own caveat cuts in your favour: content in the initial response "like JSON data or delayed React Server Components, may still be indexed since AI models can interpret non-HTML content." But every standard extraction library (Readability, trafilatura, anything built on innerText) strips <script> before it does anything else, and escaped JSON carries no <h1>, no <article>, no <time>. Flight-payload-only content is unreliable, not invisible. Real tags are the outcome you control.

Four sites, measured

All four fetched 28 August 2026 with curl, scripts and styles stripped, words counted.

Apple's page is the demonstration. It returns HTTP 200 and about 21 KB, and the string "applebot" appears zero times in the entire response, including inside script tags. The 666 readable words are the Apple Developer navigation and footer. One command reproduces it:

curl -sL --compressed 'https://search.developer.apple.com/help/applebot' | grep -ci applebot
# → 0

nextjs.org does the thing almost nobody does. Request it with a GPTBot, ClaudeBot or PerplexityBot user agent and Vercel returns content-type: text/markdown with x-matched-path: /llms.md: about 4 KB of clean Markdown with frontmatter pointing at /llms.txt, instead of 283 KB of HTML. The response carries vary: Accept, and sending Accept: text/markdown from a normal browser gets the same thing. That is AIScan's C1 check passing in the wild, done by the people who wrote the framework.

curl -sIL 'https://nextjs.org/' -A 'GPTBot/1.4' | grep -i content-type
# → content-type: text/markdown; charset=utf-8

Our own failure, published because it is the strongest evidence we have. Until 25 August 2026 every post on this blog shipped 73 words and zero headings to any fetcher that did not run JavaScript, while looking perfect in a browser. The cause was one import: isomorphic-dompurify pulled in jsdom at module top level, which threw TypeError: Cannot read properties of undefined (reading 'bind') in the Cloudflare Worker runtime. React discarded the server render. E3 scored 1 out of 4 and C3 reported h1: 0. Swapping to sanitize-html took the same page to 5,536 words, 12 <h2> sections and a score of 100. Nothing about the page looked different to a human at any point.

The seven patterns that cause it in Next.js

Verified against the live Next.js documentation on 28 August 2026, current version 16.3.3.

Start by clearing up the one people blame first. 'use client' is not the cause. Per Next.js's Server and Client Components guide, "Client Components and the RSC Payload are used to prerender HTML." A 'use client' component is still server-rendered into the initial response by default. The directive marks a hydration boundary, not an SSR opt-out.

These are the real causes:

1. Data fetched in useEffect. Nothing exists at response time, under either router. This is the most common cause by a wide margin. Fix: fetch in a Server Component, or getServerSideProps/getStaticProps on the Pages Router.
2. dynamic(() => import('./X'), { ssr: false }) around a whole route. The lazy-loading docs are explicit: "If you want to disable prerendering for a Client Component, you can use the ssr option set to false." Scope it to the one widget that genuinely cannot render on the server, never the page.
3. typeof window !== 'undefined' around the content branch. The server takes the undefined path and emits nothing. Guard the browser-only API call, not the markup.
4. Client-only i18n providers. Translations resolve after hydration, so the HTML ships raw keys like home.hero.title or empty nodes. Resolve messages on the server.
5. Auth or session wrappers gating public content on a client-side session a crawler never has. Split the route: public content server-rendered, personalised parts behind the boundary.
6. Third-party widgets for reviews, pricing, job listings or docs search that inject through a script tag. Invisible to every non-rendering fetcher. Mirror the data server-side if it is the reason the page exists.
7. A Cache Components shell whose data is not available at request time. New in Next.js 16, and the caching documentation states the failure mode plainly: "If part of your shell depends on inputs that only exist while prerendering… a page that loads for a person can fail to render for a crawler."

The Next.js 16 detail worth knowing

Next.js already tries to protect you, and the protection is a list. From the streaming guide: "HTML-limited bots and crawlers need metadata to be available in the <head> of the initial HTML. Next.js detects them by their user agent and waits for generateMetadata to resolve before streaming the page content." With Cache Components, "HTML-limited bots skip the prerendered shell and render the page dynamically so metadata can be placed in the <head>."

That behaviour is governed by the htmlLimitedBots config option, which is a user-agent allow-list. A crawler that is not on the list takes the DOM-capable path, which assumes it can finish the job in a browser. If you rely on this mechanism, add the AI crawler tokens you care about to htmlLimitedBots rather than assuming they are covered. The current user-agent list has 33 documented tokens; five operators now run three separate bots each.

Other stacks, same trap

Two notes on the last two rows. For headless WordPress, ThinkRank handles robots.txt, robots meta, schema, sitemaps and llms.txt from one plugin, which matters when your front end is a separate origin and you do not want three plugins arguing over a single robots.txt file. It migrates settings from Rank Math, Yoast, All in One SEO and SEOPress, so switching costs nothing in re-entered configuration. Rank Math and Yoast are both strong on classic on-page SEO and Yoast has the deeper content-analysis tooling; neither ships llms.txt generation as a first-class feature. For Shopify, StoreSEO generates llms.txt from products, collections, pages and articles and includes an agents.md editor, which is the store-shaped version of the same job.

How to verify the fix worked

Re-run the same measurements and compare against thresholds, not against feelings.

One more comparison is worth doing once. Open Google Search Console, run URL Inspection > Test Live URL > View Tested Page > HTML, and put it side by side with your curl output. The Search Console pane is post-render Chromium output, so it tells you what Google sees and, by construction, nothing about what GPTBot sees. curl is the closer proxy. If those two panes disagree, the disagreement is the article. Google's own JavaScript SEO video series covers the rendering side of it in more depth than any blog post.

What AIScan cannot see

Being clear about this matters more than the score.

• It reads one URL at a time. A homepage that server-renders while your product pages do not will still score well. Scan the templates that carry your money pages.
• It does not run your bundle, on purpose. It cannot tell you what your React app would have painted, only what arrived without it. That is the measurement you want, but it means AIScan will never explain why a boundary failed. <!--$!--> tells you where; your server logs tell you why.
• It cannot see behind auth. Content behind a login is invisible to it and to every crawler, which is usually correct and occasionally a surprise.
• It cannot tell you whether any given crawler acted on what it fetched. Nobody can. Vercel measured what was requested and executed, not what was retained.
• It does not know what your bot logs say. If you want to know whether GPTBot is even reaching you, that answer is in your access logs, not in any scanner.

When this does not matter

Honest scoping, because a rewrite is expensive and most of them are unnecessary.

An internal dashboard, an authenticated app, an admin panel or a tool with no public content has nothing to gain here. Neither does a site whose entire public surface is a handful of marketing pages that already server-render. If your curl word count already matches what a reader sees and there are no <!--$!--> markers, you are done, and no amount of llms.txt will improve on that.

The sites that should care are the ones whose public content is the product: documentation, guides, pricing pages, catalogues, job boards, anything you would want quoted back to a person who asked an assistant about your category.

Maintenance

This regresses silently, which is the whole difficulty. It regressed on us through a dependency import that nobody reviewed as a rendering change, and it looked perfect in a browser the entire time.

Two habits catch it. Add the word-count and <!--$!--> commands to CI so a pull request that breaks server rendering fails the build rather than shipping. And re-scan after any dependency upgrade, framework major version, or move between hosting runtimes, because a Node-only dependency in a render path passes locally and fails in a Workers or edge runtime. That is precisely how ours broke.

Your next step

Run the scan and read four rows:

npx aiscan-cli yoursite.com

E3 tells you how many words and headings a non-rendering crawler received. C3 tells you whether the page has a single <h1>, a title, a meta description and JSON-LD. C1 tells you whether you offer Markdown to anything that asks for it. C2 tells you whether you publish an llms.txt at all. Fix them in that order, because E3 is the one that decides whether the other three have anything to describe.

If you want the wider context first, our llms.txt evidence review covers what that file does and does not do, and the full guide library has platform-specific walkthroughs for Next.js, WordPress, Shopify and the rest.