---
title: "Does noindex Keep a Page Out of ChatGPT, Claude, Perplexity and Copilot? What Each Vendor Documents (2026)"
slug: noindex-ai-assistants-what-each-vendor-documents
published: 2026-11-22T13:12:38.972366+00:00
updated: 2026-11-22T13:12:38.972366+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:B2, noindex, robots.txt, AI crawlers, OAI-SearchBot, Claude-SearchBot, PerplexityBot
description: "Does noindex keep a page out of ChatGPT, Claude, Perplexity and Copilot answers? What nine vendor docs say, and the robots.txt lever that works instead (2026)."
url: https://aiscan.site/blog/noindex-ai-assistants-what-each-vendor-documents
---

Mostly no. A `noindex` tag only works for an assistant that answers from a search index whose crawler reads the tag, and on 22 November 2026 only Google documents that link for its AI features. OpenAI, Anthropic, Perplexity, DuckDuckGo and Mistral document robots.txt as their control and say nothing about `noindex` on their crawler pages, so the reliable way to stay out of an AI answer is a robots.txt group for that vendor's search agent, plus a 401 or 403 for anything that must never be fetched.

We read nine vendor documentation pages on 22 November 2026 and counted how many times each one mentions `noindex`. Seven never mention it; the two that do are both Google's. This guide gives you that table, the mechanism behind it, and the exact lever for each of three goals: out of the answer, out of training, or not fetched at all.

## TL;DR

| Your goal | The lever that is documented | Works on |
|---|---|---|
| Keep a page out of Google AI Overviews and AI Mode | `noindex` or `nosnippet` on a crawlable page | Google only |
| Keep a page out of ChatGPT, Claude, Perplexity, DuckAssist and Mistral answers | `Disallow` for OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, MistralAI-Index | Each vendor named |
| Keep a page out of Copilot grounding | Bing's `ai-grounding` preference | Bing and Copilot |
| Keep content out of model training | `Disallow` for GPTBot, ClaudeBot, Google-Extended, MistralAI-Training | Each vendor named |
| Never let an assistant read the page | 401 or 403 at the server or firewall | Everyone, including user-triggered fetches |

**The short answer:** `noindex` is an indexing instruction. It can only keep you out of an answer built from an index that honours it, and outside Google no assistant vendor says its index does.

## Why noindex misses most AI answers

A robots meta tag is a note pinned inside the page. A crawler has to fetch the page to read it, and only an indexer acts on it, because the rule is about what goes into an index. [**Google's robots meta tag specification**](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) (last updated 24 March 2026) defines it exactly that way: "Do not show this page, media, or resource in search results."

AI assistants get pages two ways. Some answer from an index their own search crawler built (OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot). Others fetch the page live when a person asks (ChatGPT-User, Claude-User, Perplexity-User). A live fetch puts nothing into an index, so a tag that says "do not index" has nothing to act on. Perplexity says so plainly: [**according to its crawler page**](https://docs.perplexity.ai/guides/bots), "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Google writes the same about its own user-triggered fetchers.

Think of `noindex` as a "do not file" stamp. A filing clerk obeys it. A courier sent to collect the document for one reader never looks at filing stamps.

## What each vendor documents (read 22 November 2026)

Every page below was fetched from the vendor's own site with `curl` on 22 November 2026 and its text searched for `noindex`. A cell is quoted from the vendor's own page or marked not documented. The pages: [OpenAI crawlers](https://developers.openai.com/api/docs/bots), [Anthropic's crawler article](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity crawlers](https://docs.perplexity.ai/guides/bots), [DuckAssistBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot), [Mistral robots](https://docs.mistral.ai/robots), Google's [robots meta tag spec](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag), [AI features page](https://developers.google.com/search/docs/appearance/ai-features) and [common crawlers list](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), and Microsoft's announcement.

| Assistant | Agent that builds the answer | `noindex` mentions on the vendor page | What the vendor documents instead |
|---|---|---|---|
| ChatGPT search | OAI-SearchBot (index), ChatGPT-User (live) | 0 | robots.txt: opted-out sites "will not be shown in ChatGPT search answers, though can still appear as navigational links" |
| Claude | Claude-SearchBot (index), Claude-User (live) | 0 | robots.txt per agent; blocking Claude-User "prevents our system from retrieving your content in response to a user query" |
| Perplexity | PerplexityBot (index), Perplexity-User (live) | 0 | robots.txt for PerplexityBot; Perplexity-User "generally ignores robots.txt rules" |
| Copilot | Bing's index | 0 in the 30 Oct 2026 post | `ai-grounding` preference, "without changing eligibility for Bing search results" |
| AI Overviews, AI Mode | Googlebot (Google's index) | Defined in the spec | `noindex`, `nosnippet`, `max-snippet`, `data-nosnippet` |
| Gemini apps | Google-Extended (a robots.txt token, no own crawler) | 0 on the crawler page | robots.txt token for "AI training and grounding in some of Google's other systems" |
| DuckAssist | DuckAssistBot (live, cites sources) | 0 | robots.txt; takes effect "after 72 hours" |
| Mistral (Vibe) | MistralAI-Index, MistralAI-User | 0 | robots.txt per agent |

Two rows need a note. OpenAI's publishers FAQ (help.openai.com article 12627856), which search results still show recommending a `noindex` tag, redirected on 22 November 2026 to the "ChatGPT Atlas retirement" article, so no live OpenAI page states that rule and the cell stays at 0. Meta's crawler page answered HTTP 400 to our request, so Meta AI is left out rather than guessed.

[**According to Microsoft's 30 October 2026 announcement**](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/), the grounding preference covers "web-grounded responses in Bing, Copilot, and Microsoft 365" and is separate from search eligibility. On Bing the AI answer has its own switch, which is the opposite of relying on `noindex`.

## How to pick the right lever

| If the page should be... | Do this | Do not rely on |
|---|---|---|
| In search, out of AI answers | Disallow each vendor's search agent; Bing `ai-grounding: disallow`; Google `nosnippet` | `noindex` alone |
| In answers, out of training | Disallow GPTBot, ClaudeBot, Google-Extended, MistralAI-Training | Blocking search agents |
| Unreadable by any assistant | Authentication, or 403 by user agent at the edge | robots.txt alone |

## The fix, step by step

### Goal 1: out of AI answers

1. Open your robots.txt. On WordPress, [ThinkRank](https://thinkrank.ai) edits it from one plugin instead of three fighting over one file (Yoast, Rank Math and AIOSEO also have editors); elsewhere it is the file in your web root.
2. Add one group per search agent you want out of the path:

```txt
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
Disallow: /private-reports/
```

3. Add the live-fetch agents if a person asking about the page should also get nothing: `ChatGPT-User`, `Claude-User`, `MistralAI-User`. OpenAI's crawler page, in its own words, says that for ChatGPT-User "robots.txt rules may not apply", so treat this as a request.
4. For Copilot, set Bing's page-level grounding preference as shown in [our Bing ai-training and ai-grounding guide](https://aiscan.site/blog/bing-ai-training-ai-grounding-every-platform).
5. For Google's AI features, keep the page crawlable and use `nosnippet` or `noindex`. The trade-offs are in [what you can control in Google AI Overviews](https://aiscan.site/blog/google-ai-overviews-what-you-can-control).

### Goal 2: out of training only

Disallow GPTBot, ClaudeBot, Google-Extended and MistralAI-Training, and leave the search agents allowed. The page-level tags for this are covered in [noai and noimageai meta tags](https://aiscan.site/blog/noai-noimageai-meta-tag), and the full starter file is in [the robots.txt starter for AI crawlers](https://aiscan.site/blog/robots-txt-starter-for-ai-crawlers).

### Goal 3: never fetched

Return 401 behind a login, or 403 for the listed user agents at your firewall. This is the only lever that also stops Perplexity-User, and [the stealth crawling debate](https://aiscan.site/blog/stealth-crawling-user-driven-fetching-debate) explains why a user agent string is still only a claim.

## The trap: Disallow plus noindex

Do not put a page behind both a robots.txt `Disallow` and a `noindex` tag for the same crawler. According to Google's specification, if a page is disallowed in robots.txt, "any information about indexing or serving rules will not be found and will therefore be ignored." The crawler never sees the tag, and the URL can still be listed from links elsewhere. Pick one lever per crawler: `noindex` for Googlebot on a crawlable page, `Disallow` for the AI search agents.

## Verify the fix

1. Run the scan: `npx aiscan-cli yoursite.com`, or paste the URL into [AIScan](https://aiscan.site/). Check **B2 (Explicit AI bot rules)** reads your robots.txt for named AI agent groups, so every agent you added should appear there by name.
2. By hand, confirm the file is served and the groups are there:

```bash
curl -s https://yoursite.com/robots.txt | grep -A3 -i "OAI-SearchBot"
curl -sI https://yoursite.com/private-reports/page | grep -i x-robots-tag
```

Expected: the first command prints your group with `Disallow: /private-reports/`; the second prints nothing for pages you blocked in robots.txt, and `x-robots-tag: noindex` only where you chose the Google route.

3. Wait out the vendors' own delays before testing an answer: about 24 hours for OAI-SearchBot, according to OpenAI's crawler page, and 72 hours for DuckAssistBot, according to DuckDuckGo's help page.

## Who does not need this

If every page you publish is meant to be read, quoted and linked, change nothing: our own robots.txt allows every AI agent by name. This is for gated reports, staging copies, customer pages and anything you would not want summarised.

## Next step

Scan your site with [AIScan](https://aiscan.site/) and read the B2 result, check the rubric on [/docs/checks/bot-access](https://aiscan.site/docs/checks/bot-access), and see how pacing rules fare in [Crawl-delay for AI crawlers](https://aiscan.site/blog/crawl-delay-ai-crawlers). More fix guides are in [/guides](https://aiscan.site/guides). The rules themselves follow [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html).

