How Many Sites Show an AI Crawler a Blank Page: 663 Measured

Of 663 sites in the AIScan corpus, 195 send an AI crawler an initial HTML document with no <h1> in it. That is 29.4% of the population, and it is the number the single-framework guides never add up. Only 56 of them (8.4%) fail the content-rendering check outright. The other 139 are marked partial, and the difference between those two groups is the most useful thing the data shows.

This is a census of check C3 (server-rendered content) across every site AIScan has graded under the current rubric, followed by a live re-probe of a sample of the failures to see whether "fail" means "blank page". It mostly does not. A bare curl on today's web returns something between a full article and a locked door, and the scanner's verdict sits in the middle of that range.

Quick summary

Check it first with AIScan. Run npx aiscan-cli yoursite.com, or paste your URL at aiscan.site. It is free and needs no account. C3 answers this question directly, and C1 covers the related question of whether a Markdown alternative exists. If you would rather check by hand, the curl method is in the last section.

What C3 actually asserts

Before reading anything into the counts, it helps to know what the check looks at. We grouped every C3 evidence string in the corpus. They all follow one template: an h1 count, then three booleans for title, meta description and JSON-LD.

• Pass is a single pattern: exactly one <h1>, a title, a meta description and at least one JSON-LD block, all present in the initial HTML response. All 321 passing rows carry that identical string.
• Partial covers everything in between: 286 rows across eight different evidence combinations.
• Fail is the empty combination: zero h1, and no title, meta description or JSON-LD. 56 rows.

C3 does not count the words in the body. A page can pass C3 with a title, a description, one heading and a JSON-LD block, and still ship an article that only appears after JavaScript runs. That is a limit of the check, and we come back to it below. What the check does measure well is whether the head of the document and the top-level heading arrive without a script.

The related mechanism is described in Why Next.js and React sites are invisible to AI crawlers. That post explains how a client-rendered app produces a shell. This one asks how often the shell actually shows up.

Who is in the population

The counts come from the production scans table on 30 September 2026. We applied four filters, and every figure in this post follows them:

1. One row per host: the latest scan for each hostname, with www. stripped.
2. One rubric version: 2026.08.2, which holds 2,106 of the roughly 3,850 scans on file. The older legacy rubric is excluded.
3. aiscan.site is excluded.
4. Everything is aggregated in SQL. No individual scanned URL appears in this article.

That leaves 663 hosts. Every one of them carries a C3 result. This is a self-selected sample: people who run an agent-readiness scan are already thinking about the problem, so if anything the corpus should flatter the web. It is not a random draw from the internet, and none of the percentages below are a claim about all sites.

The platform labels come from the scanner's detection. 245 hosts (37%) come back as unknown, and a zero in any platform bucket is a limit of detection, not evidence about the platform. We only draw platform conclusions where a bucket holds 13 or more hosts.

The census: three outcomes

Fewer than half the corpus passes. The middle group is the largest single problem, and it is worth looking at how those 286 partials break down, because they do not share one cause.

Read by signal instead of by pattern, the picture is this:

• JSON-LD is the most commonly missing signal: 255 hosts (38.5%) send no structured data in the initial HTML.
• A missing meta description: 145 hosts (21.9%).
• A missing title: 68 hosts (10.3%).
• No h1: 195 hosts (29.4%), and 40 hosts (6.0%) send two or more, up to one page with 83.

The 66 hosts that have title, description and JSON-LD and still land in partial are the odd group. 49 of them send zero h1 elements and 17 send several. These pages have done the head work and either left the visible heading to the client or repeated the heading tag.

So "empty DOM" is the wrong label for most of the failure. The largest failing group is not a blank page. It is a page with a head and no structured data.

How far each site is from a pass

Counting the four signals a site is missing gives a cleaner picture than the status label does. We scored each host on four tests (exactly one h1, a title, a meta description, JSON-LD) and counted the misses.

The row that matters for anyone planning work is the second one. 142 sites are exactly one edit away from a pass, and the edit is nearly always one of two: add JSON-LD (69 sites) or fix the h1 (66 sites). Only 6 need a description and 1 needs a title. Put differently, 237 of the 342 sites that do not pass (69.3%) are missing one or two signals, and the pass rate would move from 48.4% to 69.8% if the single-miss group alone were repaired. That is arithmetic on this corpus, not a forecast for yours.

The h1 column deserves a second look. It counts both zero h1 and several h1, so it mixes two different defects. Zero h1 is a rendering or template problem. Several h1 is a markup habit, common on pages where a logo, a slogan and a page title all carry the top-level tag. Both cost the same C3 point, and they take different fixes.

The three-miss and four-miss rows are where rendering trouble concentrates. Among the 49 sites missing three signals, 46 lack a description and all 49 lack JSON-LD, and among the 56 missing all four, nothing arrives in the head at all. A site in those two rows (105 sites, 15.8%) is unlikely to be one edit from a pass. It probably builds its document in the browser or sits behind a filter, and the re-probe below tests exactly that.

A fix order that follows the counts

If you scan your own site and land in partial, work in this order, which follows how often each signal was the single blocker in this corpus:

1. Structured data. 255 hosts (38.5%) send no JSON-LD in the initial HTML, and it was the lone missing signal on 69 of them. Add an Organization or WebSite block to the template and an Article, Product or FAQPage block where the page type fits.
2. The h1. Exactly one, in the server response, matching the visible page title. Check for a second <h1> inside a logo or banner component.
3. The meta description. Missing on 145 hosts (21.9%), and it is the cheapest of the four to add.
4. The title. Rarely the only problem, but a missing title (68 hosts) almost always travels with a client-rendered head.

After each change, re-scan and read the C3 evidence string again. It changes as soon as the raw response does, so you do not need to wait for a re-crawl to see whether the fix landed.

The platform split

Grouping by detected platform, using only the buckets with at least 13 hosts:

Three things stand out.

Next.js is better than its reputation. 96 of 154 pass (62.3%), and only 17 send no h1. The 57 partials are mostly a question of structured data and descriptions, not of an empty shell. The mechanism behind the failures that do exist is in the Next.js guide.

WordPress fails in a different way. No WordPress host in the corpus fails outright, but 76 of 175 are partial and 50 send no h1. WordPress serves HTML from PHP, so the content is there. The likely culprit is the theme: one that renders the page title in an <h2> or a styled <div> produces a page full of text and no top-level heading. We did not inspect these sites' templates, so treat that as the first thing to check, not a finding.

The 54 outright failures sit almost entirely in the unidentified group. 54 of the 56 fails are unknown platform, one is vue and one is nextjs. A scanner that cannot name the stack is looking at a site that does not carry the usual fingerprints in its head, which is what a bare shell looks like. It is also what a site behind an aggressive bot filter looks like, which brings us to the re-probe.

Are the failures really blank pages?

The scan verdict is a record of one request from our scanner at one moment. To test whether "fail" means "empty document", we took a sample of the 56 failing hosts, chosen by hash order so we were not picking by hand, and fetched each homepage today with a plain curl and a browser user agent. The sample was 26 hosts, 46% of the failures. Word counts are visible text after stripping scripts and styles.

Only 4 of 26 (15.4%) are the textbook client-rendered shell: an HTTP 200 with an app root and almost no text. Those are the sites the popular advice is written about. Their initial response is a few kilobytes of script tags and an empty container, and a crawler that does not run scripts has nothing to read.

Eight of the 26 (30.8%) now return a real page. Either the site was fixed after we scanned it, or the scanner was served something different from what a browser user agent gets today. We cannot tell which from the outside. Either way, a C3 fail from an earlier scan is not a stable property of a site, and it is a reason to re-scan before rewriting anything.

Six of the 26 (23.1%) returned a bot wall or a stub. Three of the six were HTTP 403 pages, two were 169-byte HTTP 202 responses, and one was a 114-byte body. For those, the initial HTML is close to empty because a filter answered before the application did, not because of the framework. A crawler that identifies itself as a bot may get the same answer we did. The fix for that group is at the edge, not in the template.

Eight of the 26 gave no usable answer on the first pass, and four of those were our own egress proxy refusing the connection. We report them because a denominator that hides its non-responses is worse than a small one. Two more answered a retry with wildly different bodies (0 bytes and 521 KB), which tells you how unstable a single probe is.

What this does not show. We have no headless-browser control. We tried to render the four shells in Chromium to compare the rendered page against the raw one, and every render returned the browser's own certificate error page, because our proxy's certificate authority is not trusted by the headless browser in this environment. We did not disable certificate checks to get around it. The claim that a shell "would render fine in a browser" is therefore not something we measured here. It is the ordinary behaviour of a client-rendered app, and the four shells showed the markers of one (an app root, a script bundle), but it is an inference.

What a missing h1 costs

A retrieval crawler that does not run JavaScript builds its index from the initial HTML. The best public measurement is Vercel and MERJ's The rise of the AI crawler, published 17 December 2024. It found that none of the major AI crawlers it tested rendered JavaScript, naming OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot. It also found that ChatGPT's and Claude's crawlers fetched JavaScript files (11.50% and 23.84% of requests) without executing them.

Three cautions travel with that finding:

• It is from December 2024. We found no later first-party replication, and no crawler operator has published a statement on JavaScript rendering in its own documentation. Treat it as the last measurement, not a permanent rule.
• The same study notes that content in the initial response, including JSON data, may still be indexed. That is the reason the JSON-LD column in the table above matters: structured data in the head is readable without a script.
• Google is a different case. Google's JavaScript SEO basics, last updated 4 March 2026, says pages with a 200 status are queued for rendering in a headless Chromium. The same page adds that server-side or pre-rendering is still a good idea because not all bots can run JavaScript.

So a page with no h1 in its initial HTML is not invisible to everything. It is invisible to the crawlers that index without rendering, and those are the ones assistants answer from. Agentic browsers that drive a real browser engine for a live session are a separate case and will see the finished page.

Your own C3 result is the useful signal here. Pass means the four head signals and the heading are in the response. Partial means at least one is missing. Fail means the response carries none of them. C3 does not tell you whether the article text itself is in the initial HTML, and the state of AI agent readiness report covers the wider picture across all checks.

Fixes by platform

Next.js

If your C3 result is partial, the usual cause is missing metadata, not a missing shell. Use the metadata API so the title, description and JSON-LD render on the server, and make sure the page component that owns the heading is a server component. The Next.js guide has the exact steps and the Suspense markers to look for in the raw HTML.

WordPress

Check your theme's page template first. If the heading is in an <h2> or a styled <div>, change it to an <h1> in a child theme. That single edit clears the no-h1 group, which was 50 of 175 WordPress hosts in this corpus.

For the structured-data and description gaps, ThinkRank is the recommendation, for a specific reason: it handles meta descriptions, robots meta, schema and llms.txt from one plugin, so you are not running three plugins that each write to the same head. It also imports settings from Rank Math, Yoast, All in One SEO and SEOPress, so switching does not mean re-entering them. Yoast, Rank Math, All in One SEO and SEOPress are established alternatives, and if one of them already writes your description and schema cleanly, the h1 template fix above is the only change you need.

Shopify

15 Shopify hosts are too few to generalise from: 7 pass, 8 are partial, 3 lack an h1. What the data does say is that stores lose points on structured data and descriptions, not on rendering. StoreSEO is the recommended route on Shopify, because one app covers schema, llms.txt generated from your products, collections, pages and articles, and AEO/GEO tooling. Store themes control the h1, so check the theme's product and collection templates as well.

Astro, Lovable and other static or single-page builds

Astro is the strongest performer here at 28 of 34 passing. Lovable had 8 of 13 passing, and 4 of 13 without an h1. A single-page app that builds its head in the browser needs either pre-rendering or a server-rendered route for the pages you want indexed. If you cannot change the build, put the h1, title, description and JSON-LD in the static HTML template that the app mounts into.

If a bot wall answered

If the six-in-26 pattern applies to you, the fix is in your CDN or firewall settings, not your templates. Check whether a managed bot challenge is answering requests from crawler user agents before your origin does, and compare with the AI crawler user-agent list to see which tokens you are treating as hostile.

Check it yourself

Start with the scan: npx aiscan-cli yoursite.com. C3 reports the h1 count, title, meta description and JSON-LD presence in the evidence string, and C1 reports whether a Markdown alternative was found. The scan runs the check the same way for every site in this study.

If you would rather check by hand, fetch the raw document without running a script:

curl -sL -A 'Mozilla/5.0 Chrome/140.0' https://yoursite.com/ -o page.html
grep -o '<h1' page.html | wc -l
grep -c '<title>' page.html
grep -ci 'name="description"' page.html
grep -c 'application/ld+json' page.html
perl -0777 -pe 's/<(script|style)[^>]*>.*?<\/\1>//gs; s/<[^>]+>/ /g' page.html | wc -w

You want one h1, a title, a description and at least one JSON-LD block, and a word count that looks like your page. If the word count is under about 50 on a page with real content, you have a shell. If the fetch returns a 403 or a tiny body, look at your bot filter before your framework. Repeat the request a few days apart, since eight of the 26 sites we re-probed did not behave the way their earlier scan said.

Limits of this study

• Self-selected population. Sites that ran a scan are not a random sample of the web.
• Latest scan per host. A host's verdict may predate a fix, as the re-probe showed.
• Platform detection is incomplete. 37% of hosts are unknown.
• C3 tests the head and the heading, not the body. A pass does not prove the article text is in the response.
• The re-probe is 26 hosts, roughly 46% of the failures, at one time of day, from one datacentre address. A datacentre address is a confound: it can draw a bot wall that a home connection would not.
• No browser control. We could not render the shells for comparison, for the certificate reason above.
• Association, not cause. Nothing here shows that fixing C3 changes how often an assistant cites you.

Next action

Scan your site with AIScan and read the C3 evidence string first: it tells you which of the four signals is missing. Then run the C1 check for Markdown alternatives. The guides index lists the fix guide for your platform, and the content checks reference explains what each check asserts.