---
title: "Google AI Overviews in 2026: What You Can Actually Control, Measured on 110 Publishers"
slug: google-ai-overviews-what-you-can-control
published: 2026-09-07T03:23:36.134162+00:00
updated: 2026-09-07T03:26:01.722272+00:00
author: "Asif Rahman"
author_url: https://masifrahman.com
category: "AI Readiness"
tags: AI Overviews, Google Search, robots meta tag, nosnippet, Google-Extended, AEO, check:B1, check:B2, check:D1, check:E1, platform:wordpress, platform:shopify
description: "No AI Overviews opt-out exists. We measured 110 publishers: 38.5% block Google-Extended, and not one set the snippet control Google says actually works."
url: https://aiscan.site/blog/google-ai-overviews-what-you-can-control
---
Search Console will tell you that AI Overviews sent you traffic. It will not tell you how to turn them off, because there is no setting to turn off. Google publishes exactly four controls that reach AI features in Search, and every one of them costs you something in ordinary search results.
On 7 September 2026 we fetched `robots.txt`, a homepage and one article page from 110 news, reference, health, finance and retail sites, and read what they had actually configured. The gap between what publishers set and what Google documents as effective is the whole story: **37 of 96 readable robots.txt files block Google-Extended, a token Google says has no effect on Search. Zero of 63 readable article pages carried `nosnippet`, the one directive Google's own documentation says prevents content being used as a direct input for AI Overviews.**
## Quick summary
| If you want to… | The control | Does it reach AI Overviews? | What it costs you |
|---|---|---|---|
| Stop your text appearing in an AI Overview | `nosnippet` robots meta | Yes, verbatim in Google's docs | Your ordinary search snippet, and video previews |
| Cap how much can be quoted | `max-snippet:[n]` | Yes, with a documented exception | Snippet length everywhere, including Discover |
| Exclude one part of a page | `data-nosnippet` on `span`, `div`, `section` | Yes, for that element only | Nothing, if the HTML is valid |
| Disappear from Search entirely | `noindex` | Yes | All Google search traffic |
| Stop Gemini training and grounding | `Google-Extended` in robots.txt | **No** | Nothing, and it changes nothing in Search |
| Charge an AI crawler for a fetch | Cloudflare Pay Per Crawl | **Not for Googlebot** | Search inclusion, if you try |
**Verified on 7 September 2026** against developers.google.com: there is no AI-Overviews-specific opt-out, and Google's page saying so has not been updated since 10 December 2025.
**Measured on 110 publishers the same day:** 38.5% block Google-Extended. 0% set `nosnippet`. Of the ten article pages that did carry a `max-snippet` directive, all ten set it to `-1`, which means *no limit*.
**The practical read:** the AI Overviews levers are the snippet levers, they are blunt, and almost nobody is pulling them. Before you decide whether to, scan your own site with `npx aiscan-cli yoursite.com` and read checks D1, B1 and B2, which report what your robots.txt actually declares.
## There is no AI Overviews switch, and Google's own page says so
The page publishers should read first is [Google's AI features and your website](https://developers.google.com/search/docs/appearance/ai-features). Its control section says, verbatim:
> "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search. To limit the information shown from your pages in Search, use `nosnippet`, `data-nosnippet`, `max-snippet`, or `noindex` controls."
Read that again, because the sentence is doing something people miss. It does not say "here is how to opt out of AI Overviews". It says the control for AI features is the control for Search, and they are the same control. There is no second dial. If you want Googlebot to stop feeding an AI Overview, you are asking Googlebot to stop feeding Search.
That page carries a footer reading `Last updated 2025-12-10`. On the day we checked, that was **271 days** without an edit. Its sibling page, the [AI features and your website optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), which exists mainly to tell you what *not* to bother building, reads `Last updated 2026-07-10`. Google actively maintains the page listing things that will not help you and has left the page listing your actual controls untouched for nine months.
Google's documentation-updates feed, checked the same day, carries zero September 2026 entries. Its three August 2026 entries cover dataset licensing, favicon file formats and site-reputation enforcement. None of them touch AI crawling.
## The four controls that do reach AI Overviews
The scope language lives on the [robots meta tag reference](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag), last updated 2026-03-24, and it is unusually explicit for Google.
According to that page, `nosnippet` does this, in Google's own words:
> "Do not show a text snippet or video preview in the search results for this page… This applies to all forms of search results (at Google: web search, Google Images, Discover, AI Overviews, AI Mode) and will also prevent the content from being used as a direct input for AI Overviews and AI Mode."
`max-snippet: [number]`:
> "This applies to all forms of search results (such as Google web search, Google Images, Discover, Assistant, AI Overviews, AI Mode) and will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode."
Those two sentences are the strongest first-party statement Google has published about controlling AI Overviews, and they sit on a page most publishers read only when they are debugging a title tag.
| Directive | Where it goes | Granularity | Documented AI Overviews effect |
|---|---|---|---|
| `nosnippet` | `` or `X-Robots-Tag` | Whole page | Prevents use as a direct input |
| `max-snippet:0` | Same | Whole page | Equivalent to `nosnippet` |
| `max-snippet:-1` | Same | Whole page | **No limit.** Google picks the length |
| `data-nosnippet` | HTML attribute on `span`, `div`, `section` | One element | Excludes that element from snippets |
| `noindex` | Meta tag or header | Whole page | Removes the page from Search altogether |
Three details from the same page that decide whether your implementation works:
* `max-snippet:0` is documented as equivalent to `nosnippet`, and `max-snippet:-1` explicitly means Google chooses the length it believes is most effective.
* `data-nosnippet` is valid only on `span`, `div` and `section`. Google's own example marks `` as NOT VALID.
* `data-nosnippet` is a boolean attribute, so `data-nosnippet="false"` still excludes the element. Google's example comment reads: all values are ignored.
## Google-Extended is not one of them
The most common piece of AI Overviews advice on the web is to add `Google-Extended` to robots.txt. [Google's crawler documentation](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), last updated 2026-07-14, describes exactly what that buys you:
> "Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding… Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
The same page notes that Google-Extended has no user agent string of its own: "Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." It is a permission token attached to Googlebot's fetch, not a separate crawler you can watch in your logs.
Google-Extended governs Gemini. AI Overviews are a Search feature. Blocking the first does not touch the second, and Google says so on the page that defines the token.
## How we measured 110 publishers
The population is news, reference, technical documentation, health, personal finance, food, travel, entertainment, sport and retail domains, chosen because these are the categories AI Overviews summarise most often. Every request went out on 7 September 2026 with a desktop Chrome user agent from a datacentre IP address.
Three fetches per host:
1. `GET /robots.txt`, following redirects, checking that the body is a real robots file and not an HTML page served under HTTP 200.
2. `GET /` on the homepage.
3. `GET` one article page, chosen from the homepage's own internal links by preferring URLs containing a four-digit year and a long hyphenated final segment.
Then a fourth pass: `GET /robots.txt` again with a self-declared Googlebot user agent, to test whether the file we were reading is the file Google is reading.
Counts, so nothing is hidden in a denominator:
| Fetch | Attempted | Usable | Why the rest failed |
|---|---|---|---|
| robots.txt | 110 | **96** | 4 × 403, 1 × 406, 5 × 418, 1 connection failure, 2 HTML error pages under HTTP 200, 1 excluded as unreproducible |
| Homepage | 110 | **76** | 19 × 403, 6 × 402, 2 × 401, 2 × 429, 1 × 406, 1 × 202, 2 connection failures, 1 non-HTML |
| Article page | 70 | **63** | 7 × 403 |
| robots.txt as Googlebot | 110 | 91 comparable | 6 explicit refusals, 2 connection failures, plus hosts already failing on the browser pass |
One host, The Telegraph, returned a robots.txt that disallowed Googlebot while allowing `*`. An identical refetch twenty minutes later returned HTTP 503 from an Akamai error page, and its homepage returned 402, so we could not reproduce the file and have excluded the host rather than publish a claim we cannot stand behind. Across the remaining 96, **zero disallow Googlebot**.
## What publishers actually did: 37 block training, none limit snippets
| Signal in robots.txt | Hosts | Share of 96 |
|---|---|---|
| Names `Google-Extended` at all | 41 | 42.7% |
| `Google-Extended` with `Disallow: /` | **37** | **38.5%** |
| Names `CCBot` | 55 | 57.3% |
| Names `GPTBot` | 51 | 53.1% |
| Names `ClaudeBot` | 50 | 52.1% |
| Names `GPTBot` but not `Google-Extended` | 18 | 18.8% |
| Names no AI crawler at all | 34 | 35.4% |
| Carries a `Content-Signal` line | 1 | 1.0% |
| Disallows `Googlebot` | **0** | **0%** |
The median file names 19 user-agent tokens. The mean is 36.9, dragged up by slate.com at 358 distinct tokens, which is a robots.txt maintained the way a firewall is maintained.
Five more hosts do carry a `Content-Signal` line, and all five are Stack Exchange properties whose `robots.txt` is served with **HTTP 418**, so a compliant crawler is entitled to ignore the file entirely. That is the same behaviour we recorded in the [Pay Per Crawl measurement](https://aiscan.site/blog/cloudflare-pay-per-crawl-should-you-charge), reproduced a day later.
Now the page-level half, which is where the argument lands:
| Snippet control | Homepages (n=76) | Article pages (n=63) |
|---|---|---|
| `nosnippet` present | **0 (0%)** | **0 (0%)** |
| `max-snippet` present | 8 (10.5%) | 10 (15.9%) |
| …of which set an actual limit | **0** | **0** |
| `noarchive` present | 4 (5.3%) | 8 (12.7%) |
| `data-nosnippet` used anywhere | 9 (11.8%) | 12 (19.0%) |
| JSON-LD structured data present | 61 (80.3%) | 54 (85.7%) |
Then the cross-tab. Of the 37 hosts that block Google-Extended outright, 23 also returned a readable article page. Among those 23:
* **0 set `nosnippet`.**
* **0 set a `max-snippet` limit.**
* **23 of 23, 100%, ship JSON-LD structured data.**
Every one of them pulled the lever Google documents as having no effect on Search, and not one pulled the lever Google documents as limiting AI Overviews directly.
## The max-snippet result is the one that surprised us
Eighteen `max-snippet` directives across both samples. Every single value was `-1`.
`max-snippet:-1` is not a limit. Google's reference defines it as: Google will choose the snippet length that it believes is most effective. It is the value publishers set when they want Google News-style long snippets and are worried a plugin default might clamp them. It is the opposite of restraint, and it is the only value anyone in this sample used.
So the honest summary of publisher behaviour in September 2026 is not "publishers are locking down AI Overviews". It is: publishers are blocking the training token that Google says changes nothing, and simultaneously granting unlimited snippet length through the directive that Google says feeds AI Overviews directly.
## The structured-data door Google leaves open
The `max-snippet` documentation carries a qualification that almost no coverage of AI Overviews mentions:
> "However, this limit does not apply in cases where a publisher has separately granted permission for use of content. For instance, if the publisher supplies content in the form of in-page structured data or has a license agreement with Google…"
In-page structured data is a separate grant of permission, and it overrides your snippet limit.
Which matters, because 85.7% of the article pages we could read ship JSON-LD, and 100% of the Google-Extended blockers do. A publisher who sets `max-snippet:20` on an article whose `Recipe` or `Article` schema contains the full method has capped one door and left the other one open. Google's sentence says the cap does not apply to what you handed over in the schema.
None of this makes structured data a mistake. Rich results, Discover eligibility and knowledge panels all depend on it, and stripping it to constrain an AI Overview is almost certainly a bad trade. It does mean that "I set max-snippet, so I am covered" is not true, and that the audit worth running is: what does my schema actually contain that my snippet limit is trying to withhold?
## The pages AI Overviews summarise are getting harder to fetch
A quieter result from the same sweep: **34 of 110 homepages did not return a readable page** to an automated fetch carrying an ordinary Chrome user agent. Nineteen returned 403, six returned 402, two 401, two 429, one 406, one 202, and two refused the connection.
Two hosts returned **HTTP 200 with an HTML error page at `/robots.txt`**:
```
www.thetimes.com/robots.txt 200 text/html (a web page, not a robots file)
www.creditkarma.com/robots.txt 200 text/html (a web page, not a robots file)
```
A checker reading status codes alone records those as a robots.txt that exists. It does not exist. This is the same soft-200 failure our own E1 check exists to catch, landing on the single file every crawler reads first.
None of that is an AI decision. It is ordinary bot mitigation reacting to a datacentre IP address, and Googlebot is allowlisted through it by IP verification. But it has a consequence for anyone auditing their own controls: the tooling you would naturally reach for is being refused by the sites you would naturally compare yourself against, and you cannot tell a policy from a firewall by reading a status code. Read the body, and run a browser control before concluding anything about bot policy.
## You cannot audit this by pretending to be Googlebot
The obvious way to check what Google sees is to send a Googlebot user agent and read the response. We tried it on all 110 hosts. It does not work, and the failure is instructive.
Of the 91 hosts where both identities returned a real file, **88 were byte-identical**. One, Lonely Planet, genuinely differed: it returns a different set of `Sitemap:` lines to Googlebot than to a browser, reproduced four times over. The other two differed only because our browser-agent fetch received an HTML error page.
More interesting are the refusals. **Six hosts served a browser and refused a self-declared Googlebot outright**, and two more failed to connect:
```
www.nbcnews.com 403 en.wikipedia.org 403
www.webmd.com 403 www.lowes.com 403
www.reddit.com 403 www.etsy.com 429
```
Those refusals are correct behaviour. A user agent string is a claim, not an identity, and these sites verify Googlebot by reverse DNS and IP range rather than by trusting the header. We know that verification works because we [tested it across nine crawlers](https://aiscan.site/blog/ai-crawler-user-agent-list-2026) and Google is one of only four operators publishing a hostname suffix that makes it possible.
So the correct audit path is the one Google names in its own troubleshooting section: use the URL Inspection tool in Search Console to see the HTML Googlebot received. Google adds a warning worth quoting for anyone who changes a preview control and expects a same-day result:
> "Allow time for Google to recrawl and process the change in preview controls. Remember that crawling can take anywhere from several days to several months, depending on how often our systems determine a page needs to be refreshed."
## What Pew measured, and what Google says back
The number driving most publisher anxiety comes from [Pew Research Center](https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/), published 22 July 2025 from browsing data collected in March 2025 across 900 US adults. Its findings, in its own words:
* Users who encountered an AI summary clicked a traditional search result in **8% of visits**. Users who did not encounter one clicked in **15%**, nearly twice as often.
* Users clicked a link *inside* the summary in **1% of visits**.
* Sessions ended entirely on **26%** of pages with an AI summary, against **16%** without.
* **18%** of the Google searches in the study produced an AI summary at all, and 88% of those summaries cited three or more sources. The median summary ran 67 words.
Two caveats travel with those figures. The data is March 2025, which is eighteen months old, and according to Pew's own methodology the panel is 900 US adults who agreed to share their browsing data. Nobody has published a 2026 replication.
Google's counter-claim sits on the AI features page, verbatim: "We've seen that when people click from search results pages with AI Overviews, these clicks are higher quality (meaning, users are more likely to spend more time on the site)." Google publishes no number behind that sentence and no methodology. Both claims can be true at once: fewer clicks, better clicks. Neither side has published data that settles the trade, and any article telling you it is settled is selling something.
## Everyone else can be charged. Google cannot.
While Google's control page sat unedited, publishers built a control surface of their own. In a [119-host sweep on 6 September 2026](https://aiscan.site/blog/cloudflare-pay-per-crawl-should-you-charge), 16 publishers answered an AI crawler with HTTP 402, and six Stack Exchange properties quoted a real price of USD 0.50 per page. Cloudflare Pay Per Crawl is live, it bills per successful retrieval, and it charges different crawler tokens different amounts.
None of that reaches AI Overviews, for a structural reason. Every other AI system fetches with its own token, so a publisher can price GPTBot, block CCBot and allow OAI-SearchBot from one dashboard. AI Overviews are fed by Googlebot, the crawler that also feeds the ten blue links. Charge it and you leave Search. Refuse it and you leave Search. There is no token to negotiate with, which is exactly why the only levers Google offers are snippet levers.
That asymmetry is the real 2026 story: every AI consumer of your content can now be charged, negotiated with or refused, except the largest one.
## Setting the controls on WordPress and Shopify
The directives are three lines of HTML. Getting them onto the right pages, and only the right pages, is the work.
**WordPress.** Page-level robots meta is owned by your SEO plugin, and the classic failure is two plugins writing conflicting tags into the same ``. [ThinkRank](https://thinkrank.ai) is the one to start with, for a reason specific to this task: a single plugin owns every crawler-facing surface at once, so the snippet directive on a post, the site's robots rules, the schema block and the sitemap entry cannot contradict each other. That matters here more than usual, because the schema exception above means your snippet limit and your structured data have to be decided together. Migration paths exist from Rank Math, Yoast, All in One SEO and SEOPress, so nothing configured today has to be typed in twice. The alternatives are real: Yoast has the stronger multi-author editorial workflow, and Rank Math gives away more in its free tier than anyone else. Both expose per-post advanced robots settings and will set these directives correctly. What none of them do for you is the schema question above, which stays a manual audit.
For a single template, the tag is:
```php
add_action('wp_head', function () {
if (is_singular('post') && has_category('wire-copy')) {
echo '' . "\n";
}
});
```
**Shopify.** Theme-level control lives in `theme.liquid`, and product descriptions are the content most often reused verbatim:
```liquid
{% if template contains 'product' %}
{% endif %}
```
For the rest of the crawler-facing stack on a store, [StoreSEO](https://storeseo.com/) is where to start. Snippet directives are only the Google half of this problem; product and collection schema, agents.md and a store-wide llms.txt are the half that decides what every non-Google system can quote, and StoreSEO writes all of them from the Shopify admin. Its marketplace listing carried a 5.0 rating across 737 reviews and the Built for Shopify badge when we read it on 7 September 2026, on [its Shopify App Store page](https://apps.shopify.com/storeseo). Whatever you use, confirm the tag on a live product URL rather than in the theme editor preview.
## Five ways this goes wrong
1. **Blocking Google-Extended and expecting AI Overviews to change.** It governs Gemini training and grounding. Google documents that it does not affect Search inclusion or ranking, and AI Overviews are a Search feature.
2. **Wrapping content in a tag `data-nosnippet` does not support.** Only `span`, `div` and `section` count. Google marks anything else NOT VALID, and the attribute is silently ignored.
3. **Leaving a `data-nosnippet` container unclosed.** Google's own example warns that an unclosed `div` "will include all content afterwards", so one missing tag can suppress the rest of your page from every snippet.
4. **Writing `data-nosnippet="false"` to turn it off.** It is a boolean attribute; all values are ignored, including that one. Remove the attribute.
5. **Setting `max-snippet` and ignoring your schema.** Google states the limit does not apply where permission was granted separately, and names in-page structured data as an example.
## Check your own controls in one pass
Start with the scan, because it reads the files a crawler reads and tells you what they declare:
```bash
npx aiscan-cli yoursite.com
```
Free, no account, and it runs in a terminal or from the paste box at [aiscan.site](https://aiscan.site/). The rows to read for this article are **D1** (robots.txt present and valid), **B1** (content signals declared), **B2** (explicit AI bot rules) and **E1** (does the host return a real 404, or a 200 for files that do not exist). E1 is the one that catches the failure two hosts in our sample are living with right now.
If you would rather check by hand, four commands cover it:
```bash
# 1. What does your robots.txt actually say about AI crawlers?
curl -sL https://yoursite.com/robots.txt | grep -iE 'google-extended|gptbot|claudebot|ccbot|content-signal'
# 2. Is /robots.txt a real file, or an HTML error page under HTTP 200?
curl -sLo /dev/null -w '%{http_code} %{content_type} %{size_download}\n' https://yoursite.com/robots.txt
# 3. What robots meta does a real article page carry?
curl -sL https://yoursite.com/some-article | grep -oiE ']+name="(robots|googlebot)"[^>]*>'
# 4. How much structured data is on that page?
curl -sL https://yoursite.com/some-article | grep -c 'application/ld+json'
```
Command 2 is the control most audits skip. If `size_download` for `/robots.txt` looks like a web page rather than a text file, you have The Times' problem and every crawler-facing conclusion below it is wrong.
## Where AIScan fits, and where it doesn't
AIScan grades what your site declares to a crawler. It reads robots.txt and reports which AI tokens you address (B2), whether you carry content signals (B1), whether the file exists at all (D1), and whether your host distinguishes a real page from a missing one (E1). Those are the checks that decide whether the controls in this article are even legible.
What it does not do, and should not be sold as doing:
* **It does not grade snippet directives.** `nosnippet`, `max-snippet` and `data-nosnippet` are Google-specific presentation controls, and no check in our rubric reads them. The measurement in this post was run by hand, and the commands above are the whole method.
* **It cannot see what Google actually did with your page.** Only Search Console can, through the URL Inspection tool and the Performance report.
* **It fetches with our own identity**, so it cannot see per-crawler policy. A site can score 100 on our rubric and answer an AI crawler with a 402.
* **It has no opinion on whether you should restrict anything.** That is a business decision about the trade Pew and Google disagree about.
Everything in this article is doable with `curl` and a text editor. The scan is the faster route, not the only one.
## Your next action on AI Overviews
Do this in the order below, because step one changes what steps two and three mean.
1. **Run `npx aiscan-cli yoursite.com`** and read D1, B1, B2 and E1. If E1 is not clean, fix that first: a host that answers 200 for missing files makes every other finding unreliable.
2. **Read a real article page's robots meta** with command 3 above. If it says `max-snippet:-1`, you have granted unlimited snippet length, whatever your robots.txt says about training.
3. **Count your JSON-LD** and decide whether the schema contains the content your snippet limit is meant to withhold.
4. **Decide the trade deliberately.** `nosnippet` removes you from AI Overviews as a direct input and removes your ordinary snippet with it. That is the deal Google publishes. There is no version of it that costs nothing.
More on the checks in this article at [/docs/checks/bot-access](https://aiscan.site/docs/checks/bot-access) and [/docs/checks/discoverability](https://aiscan.site/docs/checks/discoverability), the platform pages for [WordPress](https://aiscan.site/docs/platforms/wordpress) and [Shopify](https://aiscan.site/docs/platforms/shopify), and the full library at [aiscan.site/guides](https://aiscan.site/guides). If llms.txt is next on your list, Google says plainly that Search ignores it, which we covered in [does llms.txt actually work](https://aiscan.site/blog/does-llms-txt-actually-work-2026); the [llms.txt generator](https://aiscan.site/llms-txt-generator) will still build you a valid one for the systems that do read it. For where the wider web sits on all of this, the [473-site readiness study](https://aiscan.site/blog/state-of-ai-agent-readiness-2026) has the distribution.