Table of contents
Mostly no. A noindex tag only works for an assistant that answers from a search index whose crawler reads the tag, and on 22 November 2026 only Google documents that link for its AI features. OpenAI, Anthropic, Perplexity, DuckDuckGo and Mistral document robots.txt as their control and say nothing about noindex on their crawler pages, so the reliable way to stay out of an AI answer is a robots.txt group for that vendor's search agent, plus a 401 or 403 for anything that must never be fetched.
We read nine vendor documentation pages on 22 November 2026 and counted how many times each one mentions noindex. Seven never mention it; the two that do are both Google's. This guide gives you that table, the mechanism behind it, and the exact lever for each of three goals: out of the answer, out of training, or not fetched at all.
TL;DR
| Your goal | The lever that is documented | Works on |
|---|---|---|
| Keep a page out of Google AI Overviews and AI Mode | noindex or nosnippet on a crawlable page | Google only |
| Keep a page out of ChatGPT, Claude, Perplexity, DuckAssist and Mistral answers | Disallow for OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, MistralAI-Index | Each vendor named |
| Keep a page out of Copilot grounding | Bing's ai-grounding preference | Bing and Copilot |
| Keep content out of model training | Disallow for GPTBot, ClaudeBot, Google-Extended, MistralAI-Training | Each vendor named |
| Never let an assistant read the page | 401 or 403 at the server or firewall | Everyone, including user-triggered fetches |
The short answer: noindex is an indexing instruction. It can only keep you out of an answer built from an index that honours it, and outside Google no assistant vendor says its index does.
Why noindex misses most AI answers
A robots meta tag is a note pinned inside the page. A crawler has to fetch the page to read it, and only an indexer acts on it, because the rule is about what goes into an index. Google's robots meta tag specification (last updated 24 March 2026) defines it exactly that way: "Do not show this page, media, or resource in search results."
AI assistants get pages two ways. Some answer from an index their own search crawler built (OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot). Others fetch the page live when a person asks (ChatGPT-User, Claude-User, Perplexity-User). A live fetch puts nothing into an index, so a tag that says "do not index" has nothing to act on. Perplexity says so plainly: according to its crawler page, "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." Google writes the same about its own user-triggered fetchers.
Think of noindex as a "do not file" stamp. A filing clerk obeys it. A courier sent to collect the document for one reader never looks at filing stamps.
What each vendor documents (read 22 November 2026)
Every page below was fetched from the vendor's own site with curl on 22 November 2026 and its text searched for noindex. A cell is quoted from the vendor's own page or marked not documented. The pages: OpenAI crawlers, Anthropic's crawler article, Perplexity crawlers, DuckAssistBot, Mistral robots, Google's robots meta tag spec, AI features page and common crawlers list, and Microsoft's announcement.
| Assistant | Agent that builds the answer | noindex mentions on the vendor page | What the vendor documents instead |
|---|---|---|---|
| ChatGPT search | OAI-SearchBot (index), ChatGPT-User (live) | 0 | robots.txt: opted-out sites "will not be shown in ChatGPT search answers, though can still appear as navigational links" |
| Claude | Claude-SearchBot (index), Claude-User (live) | 0 | robots.txt per agent; blocking Claude-User "prevents our system from retrieving your content in response to a user query" |
| Perplexity | PerplexityBot (index), Perplexity-User (live) | 0 | robots.txt for PerplexityBot; Perplexity-User "generally ignores robots.txt rules" |
| Copilot | Bing's index | 0 in the 30 Oct 2026 post | ai-grounding preference, "without changing eligibility for Bing search results" |
| AI Overviews, AI Mode | Googlebot (Google's index) | Defined in the spec | noindex, nosnippet, max-snippet, data-nosnippet |
| Gemini apps | Google-Extended (a robots.txt token, no own crawler) | 0 on the crawler page | robots.txt token for "AI training and grounding in some of Google's other systems" |
| DuckAssist | DuckAssistBot (live, cites sources) | 0 | robots.txt; takes effect "after 72 hours" |
| Mistral (Vibe) | MistralAI-Index, MistralAI-User | 0 | robots.txt per agent |
Two rows need a note. OpenAI's publishers FAQ (help.openai.com article 12627856), which search results still show recommending a noindex tag, redirected on 22 November 2026 to the "ChatGPT Atlas retirement" article, so no live OpenAI page states that rule and the cell stays at 0. Meta's crawler page answered HTTP 400 to our request, so Meta AI is left out rather than guessed.
According to Microsoft's 30 October 2026 announcement, the grounding preference covers "web-grounded responses in Bing, Copilot, and Microsoft 365" and is separate from search eligibility. On Bing the AI answer has its own switch, which is the opposite of relying on noindex.
How to pick the right lever
| If the page should be... | Do this | Do not rely on |
|---|---|---|
| In search, out of AI answers | Disallow each vendor's search agent; Bing ai-grounding: disallow; Google nosnippet | noindex alone |
| In answers, out of training | Disallow GPTBot, ClaudeBot, Google-Extended, MistralAI-Training | Blocking search agents |
| Unreadable by any assistant | Authentication, or 403 by user agent at the edge | robots.txt alone |
The fix, step by step
Goal 1: out of AI answers
- Open your robots.txt. On WordPress, ThinkRank edits it from one plugin instead of three fighting over one file (Yoast, Rank Math and AIOSEO also have editors); elsewhere it is the file in your web root.
- Add one group per search agent you want out of the path:
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
Disallow: /private-reports/
- Add the live-fetch agents if a person asking about the page should also get nothing:
ChatGPT-User,Claude-User,MistralAI-User. OpenAI's crawler page, in its own words, says that for ChatGPT-User "robots.txt rules may not apply", so treat this as a request. - For Copilot, set Bing's page-level grounding preference as shown in our Bing ai-training and ai-grounding guide.
- For Google's AI features, keep the page crawlable and use
nosnippetornoindex. The trade-offs are in what you can control in Google AI Overviews.
Goal 2: out of training only
Disallow GPTBot, ClaudeBot, Google-Extended and MistralAI-Training, and leave the search agents allowed. The page-level tags for this are covered in noai and noimageai meta tags, and the full starter file is in the robots.txt starter for AI crawlers.
Goal 3: never fetched
Return 401 behind a login, or 403 for the listed user agents at your firewall. This is the only lever that also stops Perplexity-User, and the stealth crawling debate explains why a user agent string is still only a claim.
The trap: Disallow plus noindex
Do not put a page behind both a robots.txt Disallow and a noindex tag for the same crawler. According to Google's specification, if a page is disallowed in robots.txt, "any information about indexing or serving rules will not be found and will therefore be ignored." The crawler never sees the tag, and the URL can still be listed from links elsewhere. Pick one lever per crawler: noindex for Googlebot on a crawlable page, Disallow for the AI search agents.
Verify the fix
- Run the scan:
npx aiscan-cli yoursite.com, or paste the URL into AIScan. Check B2 (Explicit AI bot rules) reads your robots.txt for named AI agent groups, so every agent you added should appear there by name. - By hand, confirm the file is served and the groups are there:
curl -s https://yoursite.com/robots.txt | grep -A3 -i "OAI-SearchBot"
curl -sI https://yoursite.com/private-reports/page | grep -i x-robots-tag
Expected: the first command prints your group with Disallow: /private-reports/; the second prints nothing for pages you blocked in robots.txt, and x-robots-tag: noindex only where you chose the Google route.
- Wait out the vendors' own delays before testing an answer: about 24 hours for OAI-SearchBot, according to OpenAI's crawler page, and 72 hours for DuckAssistBot, according to DuckDuckGo's help page.
Who does not need this
If every page you publish is meant to be read, quoted and linked, change nothing: our own robots.txt allows every AI agent by name. This is for gated reports, staging copies, customer pages and anything you would not want summarised.
Next step
Scan your site with AIScan and read the B2 result, check the rubric on /docs/checks/bot-access, and see how pacing rules fare in Crawl-delay for AI crawlers. More fix guides are in /guides. The rules themselves follow RFC 9309.
Frequently asked questions
Does a noindex tag keep my page out of ChatGPT answers?
Not by any rule OpenAI documents today. OpenAI's crawler page names robots.txt for OAI-SearchBot as the control for ChatGPT search and does not mention noindex; the publishers FAQ that once mentioned it redirected to an Atlas retirement article on 22 November 2026. Disallow OAI-SearchBot for the path instead.
I added noindex but my page still shows up in Perplexity. Why?
Perplexity documents robots.txt, not noindex, and its Perplexity-User fetcher "generally ignores robots.txt rules" because a person asked for the page. Disallow PerplexityBot to leave its index, and return a 403 to Perplexity-User if the page must never be read.
My page is blocked in robots.txt and has noindex, but Google still lists the URL. What went wrong?
The block hides the tag. Google's specification says that when a page is disallowed in robots.txt, indexing rules on it "will not be found and will therefore be ignored." Remove the Disallow for Googlebot so it can read the noindex.
Claude quoted a page I marked noindex. Is that a bug?
No. Anthropic's crawler article documents robots.txt groups for ClaudeBot, Claude-SearchBot and Claude-User and never mentions noindex. A Claude-User fetch made for one person reads the page as served, so use a Disallow for Claude-User or a 403.
Does noindex stop my content being used for AI training?
No vendor we read says so. Training has its own tokens: GPTBot, ClaudeBot, Google-Extended and MistralAI-Training in robots.txt, and Bing's ai-training preference. A page can stay in search results and still opt out of training.
How do I keep a page out of Copilot answers but still in Bing search?
Use Bing's ai-grounding preference, announced on 30 October 2026. Microsoft says it applies to web-grounded responses in Bing, Copilot and Microsoft 365 without changing eligibility for Bing search results.
My robots.txt change has not taken effect in ChatGPT after an hour. Is it broken?
Probably not. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust, and DuckDuckGo gives 72 hours for DuckAssistBot. Confirm the file is served with curl, then wait.
Which AIScan check covers this?
Check B2, Explicit AI bot rules, reads your robots.txt for named groups for AI agents. No AIScan check grades noindex, because no assistant except Google documents honouring it. Run npx aiscan-cli yoursite.com to see your B2 result.

About the author
M Asif Rahman
Entrepreneur, curious about AI
I have been building companies for a long time, and lately most of my curiosity goes to AI. AIScan is a small personal project I built on a flight from Dubai to Narita. I use it to understand how ready the web is for AI agents, and I share what the scans teach me.
Related guides
AI Crawler User-Agents 2026: Verified List and IP Ranges
Updated on 12 November 2026: every IP range file refetched and recounted, and every reverseDNS test rerun. The short answer: the AI crawlers that matter in 2026 come from OpenAI, Anthropic, Google,…
Search, Agent, Training: What Cloudflare's Three AI Crawler Categories Actually Control
Updated 25 August 2026. Cloudflare now sorts crawler behaviour into three categories you can configure — Search, Agent, and Training — and the setting you pick for each decides whether Google indexes…
Does Crawl-delay Slow GPTBot, ClaudeBot or PerplexityBot? What Each Vendor Documents (2026)
Only some do. Of the AI crawler operators whose documentation we read on 20 November 2026, Anthropic ClaudeBot and Common Crawl CCBot say they honour Crawldelay, and Microsoft's Bing documents its…
