Table of contents
Bing reads two new page-level values from 1 February 2027: ai-training (allow or disallow) and ai-grounding (allow, disallowextended or disallow). Put each one in its own <meta name="Bingbot"> tag, or on its own X-Robots-Tag: Bingbot: header line. We tested eight setups on Next.js 16.4.6, Astro 7.4.12 and a WordPress hook stub under PHP 8.3.6 on 14 November 2026, and in three of them a value went missing before the response left the server. This guide shows where, and the config that keeps both.
Quick summary
| Your goal | What to publish | Where | Bing honours it from |
|---|---|---|---|
| Keep content out of training, whole site | User-Agent: __usage_ai-training + Disallow: / | robots.txt | 1 Feb 2027 |
| Keep one page out of training | <meta name="Bingbot" content="ai-training:disallow"> | that page | 1 Feb 2027 (overrides robots.txt) |
| Allow Copilot and Bing answers, refuse third-party grounding APIs | <meta name="Bingbot" content="ai-grounding:disallowextended"> | every page you want covered | 1 Feb 2027 |
| Refuse all grounding | ai-grounding:disallow | every page | 1 Feb 2027 |
| Keep everything allowed | Nothing | Nowhere | Already the default |
| You used NoArchive or NoCache for this | Publish the new values and record a temporary grounding preference in Bing Webmaster Tools | pages + BWT | Before 1 Mar 2027 |
Safest form on every stack we tested: two separate meta tags in the page template. Header rules work, but each stack has its own way of dropping or folding a second line, so check the served response, not the config file.
What Bing reads, and what it ignores
According to Microsoft's announcement of 30 October 2026 (retrieved 14 November 2026), training and grounding are two independent choices. Training uses the shared robots.txt group built on the IETF AI Preferences working group's draft, and Bing extends it to single pages. Grounding means using your page as input for a generated answer in Bing, Copilot or Microsoft 365, and it exists only at page level: "There is no site-wide robots.txt grounding preference."
Think of robots.txt as the sign on the building and the meta tag as a label on one box inside. Bing's AI preferences FAQ (Q12) says the box label wins for training: "a page-level preference overrides the applicable robots.txt preference for that page." Grounding has no building sign at all, so every page that needs a grounding value has to carry its own label.
Four rules from those two pages decide everything below:
- Both values default to Allow. A page with nothing on it is allowed for both.
- These govern use, not crawling. An Allow does not undo a robots.txt rule that blocks Bingbot.
- Nothing changes before 1 February 2027, and a value only counts once Bingbot recrawls the page.
- From that date, a site that publishes any new value has its old NoArchive and NoCache ignored for training and grounding. After 1 March 2027 they stop counting for these uses on every site.
Bing's pages show one value per meta tag and one per header line. Neither page shows a combined form such as ai-training:disallow, ai-grounding:disallow in one content attribute, so we do not use one here.
Pick your values before you touch config
Decide per section of the site, not per stack:
- A publisher that wants citations in Copilot but no model training: robots.txt training group set to Disallow, grounding left at the default. No page tags needed.
- Paid or licensed content:
ai-training:disallowandai-grounding:disallowon those pages only. Expect those pages to stop appearing in AI answers. - Content you license to other AI firms:
ai-grounding:disallowextended, which keeps Bing, Copilot and Microsoft 365 answers but refuses the grounding Microsoft sells to third parties, such as Web IQ and the grounding API in Foundry (Microsoft's own examples). - Everything else: leave it. Allow is the default.
To find out which branch you are on, ask whoever signs your content licences. If nobody does, you are almost certainly on the last one.
What our lab showed
Every setup below ran locally on 14 November 2026. A 33-line Python parser fetched each URL, collected the X-Robots-Tag lines and Bingbot meta tags, and printed the effective values (the script is in the verification section).
| Stack and setup (8 URLs tested) | Header Bing would see | Meta tags in the HTML | Parser result |
|---|---|---|---|
Next.js 16.4.6, two X-Robots-Tag entries in headers() | Only the last one | none | training lost |
Next.js 16.4.6, proxy.js with two append() calls | One line, comma-joined | none | both, folded |
Next.js 16.4.6, metadata.other array | n/a | Two Bingbot tags | both |
Next.js 16.4.6, child page sets its own other.Bingbot | n/a | Child's value only | layout's value replaced |
| Astro 7.4.12, middleware, prerendered page | None at all | Two tags from the layout | both, via meta only |
Astro 7.4.12, middleware, prerender = false | One line, comma-joined | Two tags | both |
WordPress hooks, header(..., false) twice | Two lines | One tag | both |
Same, plus a later plugin calling header() without false | That plugin's line only | One tag | grounding lost |
For context, on the same day we ran the parser on our own five sites (aiscan.site, xspeedcache.com, storeseo.com, bettershield.ai, thinkrank.ai). All five returned allow (default) for both values, with no __usage_ai-training group and no NoArchive or NoCache. That is a deliberate choice for us: we want AI answers to cite our pages.
Next.js (App Router)
Next.js's headers() documentation states the trap directly: "If two headers match the same path and set the same header key, the last header key will override the first." Our test confirmed it inside a single rule too. So use metadata for page values.
- In
app/layout.js, set the site-wide training value:export const metadata = { other: { Bingbot: 'ai-training:disallow' }, }; - On a page that also needs a grounding value, repeat the training value, because a page's
other.Bingbotreplaces the layout'sBingbotkey (other keys merge):
The array renders as twoexport const metadata = { other: { Bingbot: ['ai-training:disallow', 'ai-grounding:disallowextended'] }, };<meta name="Bingbot">tags, as the metadata docs describe forother. - If you need a header (for PDFs or other non-HTML files), send one value per route in
headers(). For two values on one route, useproxy.jswithres.headers.append(); expect one comma-joined line on the wire, which HTTP treats as equal to two lines (RFC 9110, section 5.3), though Bing has not documented that form.
The robots field in Next.js metadata has a googleBot key and no Bing key, so other is the route.
Astro
Astro's middleware guide says middleware runs at build time for prerendered pages. Headers you add there never reach a visitor or Bingbot on a static page. Our prerendered URLs came back with no X-Robots-Tag at all.
- Put the tags in your base layout, which works for static and on-demand pages alike:
--- const { grounding } = Astro.props; --- <meta name="Bingbot" content="ai-training:disallow" /> {grounding && <meta name="Bingbot" content={`ai-grounding:${grounding}`} />} - Pass
grounding="disallowextended"(ordisallow) from the pages that need it. - Use middleware headers only on routes with
export const prerender = false. For a static build deployed to Netlify or Cloudflare Pages, write the header into the host's_headersfile instead (below).
WordPress
For the robots.txt half (site-wide training), ThinkRank is the plugin we point WordPress sites to first: its Robots.txt screen takes custom rules, so the __usage_ai-training group goes in one place instead of three plugins fighting over the file, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Its per-post robots controls and the thinkrank_robots_meta filter (per its developer docs, checked 14 November 2026) edit the standard robots meta tag, not a Bingbot tag, so the page values need a small snippet with any SEO plugin.
- Create
wp-content/mu-plugins/bing-ai-prefs.php:<?php add_action('wp_head', function () { echo '<meta name="Bingbot" content="ai-training:disallow">' . "\n"; if (is_singular('post')) { echo '<meta name="Bingbot" content="ai-grounding:disallowextended">' . "\n"; } }, 1); - If you prefer headers, use the
send_headershook and always passfalseas the second argument ofheader():add_action('send_headers', function () { header('X-Robots-Tag: Bingbot: ai-training:disallow', false); }); - Search your plugins for
X-Robots-Tag. In our test, a later hook callingheader('X-Robots-Tag: noindex')withoutfalsereplaced both Bing lines.
Cloudflare
Cloudflare's response header transform docs separate Add (keeps existing headers of the same name) from Set (replaces them). Pick Add, or your origin's X-Robots-Tag disappears.
- In the dashboard, go to Rules > Overview, then Create rule > Response Header Transform Rule.
- Under When incoming requests match, enter a custom filter such as
starts_with(http.request.uri.path, "/blog/"). - Under Modify response header, choose Add static, header name
X-Robots-Tag, valueBingbot: ai-grounding:disallowextended. Add a second Add static row for the training value if you want it. - Deploy. Through the API, the same rule in the
http_response_headers_transformphase looks like this:{ "expression": "starts_with(http.request.uri.path, \"/blog/\")", "action": "rewrite", "action_parameters": { "headers": { "X-Robots-Tag": { "operation": "add", "value": "Bingbot: ai-grounding:disallowextended" } } } }
We wrote this rule from Cloudflare's docs and did not deploy it.
Nginx and Apache
Neither server was installed for this run, so these follow their own docs. Nginx's add_header sends one line per directive; add always so error pages carry it too:
add_header X-Robots-Tag "Bingbot: ai-training:disallow" always;
add_header X-Robots-Tag "Bingbot: ai-grounding:disallowextended" always;
Server-level add_header lines are inherited only when a location has none of its own. Repeat both lines in any location that sets its own headers, or use add_header_inherit merge on Nginx 1.29.3 and later.
Apache mod_headers: Header always add keeps existing lines, while Header set replaces them.
Header always add X-Robots-Tag "Bingbot: ai-training:disallow"
Header always add X-Robots-Tag "Bingbot: ai-grounding:disallowextended"
Netlify and Vercel
Netlify's headers docs say two lines with the same name in _headers are collapsed into one comma-joined header, the same folding we saw in Next.js and Astro:
/blog/*
X-Robots-Tag: Bingbot: ai-training:disallow
X-Robots-Tag: Bingbot: ai-grounding:disallowextended
In vercel.json, a headers rule takes source and a list of key/value pairs. Vercel does not document what happens to two pairs with the same key, so on both hosts prefer the meta tags and run the curl check after the first deploy.
Check that it worked
Start with a scan: run npx aiscan-cli yoursite.com or paste your URL at aiscan.site, free and with no account. Check B2 (explicit AI bot rules) reads your robots.txt groups, so it confirms the file still parses after you add the __usage_ai-training group. AIScan does not score the page-level Bing values, so confirm those by hand. The bot access checks page explains what B2 looks at.
If you would rather check by hand:
curl -sI https://yoursite.com/blog/post | grep -i '^x-robots-tag'
curl -s https://yoursite.com/blog/post | grep -io '<meta name="bingbot"[^>]*>'
Then run the parser over several URLs, one per template:
#!/usr/bin/env python3
import re, sys, urllib.request
VALUES = {"ai-training": {"allow", "disallow"},
"ai-grounding": {"allow", "disallow", "disallowextended"}}
def prefs(url):
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0 bingprefs"})
with urllib.request.urlopen(req) as r:
tags = r.headers.get_all("X-Robots-Tag") or []
html = r.read().decode("utf-8", "replace")
found = {"header": [], "meta": []}
for line in tags:
for part in re.split(r",\s*(?=bingbot\s*:)", line, flags=re.I):
m = re.match(r"\s*bingbot\s*:\s*(.+)", part, re.I)
if m: found["header"] += [d.strip().lower() for d in m.group(1).split(",")]
for m in re.finditer(r'<meta\s+name=["\']bingbot["\']\s+content=["\']([^"\']+)', html, re.I):
found["meta"] += [d.strip().lower() for d in m.group(1).split(",")]
out = {}
for key, ok in VALUES.items():
seen = {(s, d.split(":", 1)[1]) for s, ds in found.items() for d in ds if d.startswith(key + ":")}
vals = {v for _, v in seen}
out[key] = ("INVALID " + ",".join(vals - ok)) if vals - ok else \
("CONFLICT " + ",".join(sorted(vals))) if len(vals) > 1 else \
(vals.pop() + " (" + "+".join(sorted(s for s, _ in seen)) + ")") if vals else "allow (default)"
return out
for u in sys.argv[1:]:
p = prefs(u); print(f"{u}\n training: {p['ai-training']}\n grounding: {p['ai-grounding']}")
Pass means: every URL prints the values you chose, with no CONFLICT and no INVALID. A page that should carry grounding but prints allow (default) lost its tag or header somewhere between config and response; the lab table above lists the usual places.
Mistakes to avoid
- Writing
ai-groundinginto robots.txt. Bing reads no grounding value there. - Sending
disallowextendedas a training value. It exists only for grounding. - Removing NoArchive or NoCache before the new values are live and recrawled. The full migration, with a checker that shows each page before and after the switch, is in Bing NoArchive and NoCache stop controlling AI on 1 March 2027. If you used them to keep content out of grounding, Microsoft asks you to publish page values and record a temporary preference in Bing Webmaster Tools before 1 March 2027; that option opens on 15 December 2026, only for sites already using NoArchive or NoCache.
- Expecting a search ranking change. Bing says these values do not change eligibility for Bing results, and that Disallow or DisallowExtended for grounding will reduce your visibility in AI answers.
Who can skip this: any site happy for Bing to train on and cite its pages. Both values already default to Allow.
Maintenance
Re-run the parser after any framework upgrade, SEO plugin change or new CDN rule, because each of the losses in our lab came from a layer added on top of a working config. Bing also suggests IndexNow to ask for a recrawl once the values are live; it raises the chance of a recrawl but sets no date.
For the robots.txt half on every platform, see Add the usage AI-training token on every platform, and for what the token means, the __usage_ai-training robots.txt token. The older noai tags are covered in noai and noimageai meta tags, and the same per-stack header mechanics for .md copies are in X-Robots-Tag on .md copies and llms.txt.
Your next step
Scan your site at aiscan.site and confirm check B2 still passes after you add the robots.txt group, then run the parser on one URL per template. For the setup guides for every other check, see /guides.
Frequently asked questions
I added two X-Robots-Tag entries in next.config headers() and only one shows up. Why?
Next.js keeps the last header with the same key on a route; its headers() docs say the last one overrides the first. Our Next.js 16.4.6 test sent only the grounding line. Use metadata.other with an array, which renders two Bingbot meta tags, or proxy.js with headers.append().
My Astro middleware sets X-Robots-Tag but curl shows nothing. What is wrong?
The page is prerendered. Astro runs middleware at build time for prerendered pages, so the header never reaches a request. Put the Bingbot meta tags in your layout, set prerender = false on that route, or add the header in your host's _headers file.
curl shows one X-Robots-Tag line with both Bingbot values joined by a comma. Is that broken?
Not by HTTP rules: RFC 9110 lets a server fold repeated header lines into one comma-joined line with the same meaning. Netlify, Next.js proxy and Astro on-demand pages all did this in our tests. Bing has not documented the folded form, so if you want no doubt, use two meta tags.
My WordPress Bingbot header disappeared after I installed a security plugin. Why?
Another hook probably called header('X-Robots-Tag: ...') without false as the second argument, which replaces every earlier X-Robots-Tag line. Pass false in your own call, search the plugin for X-Robots-Tag, or switch to the wp_head meta tags, which a header call cannot remove.
Can I set ai-grounding in robots.txt?
No. Bing's FAQ says AI Grounding cannot be set in robots.txt. Only training has a robots.txt form, the User-Agent: __usage_ai-training group. Grounding goes on each page as a meta tag or an X-Robots-Tag header.
Will ai-training:disallow or ai-grounding:disallow remove my pages from Bing search?
No. Microsoft says these values do not change eligibility for Bing search results. Disallow or DisallowExtended for grounding will reduce how often your pages appear in AI answers and Copilot.
When does Bing start reading these values?
From 1 February 2027, as Bingbot crawls each page. A value only counts once the page or robots.txt is recrawled. Nothing changes before that date.
I use NoArchive to keep pages out of Copilot. What do I need to do?
Publish the new ai-grounding value on those pages and, between 15 December 2026 and 1 March 2027, record a temporary grounding preference in Bing Webmaster Tools. After 1 March 2027 NoArchive and NoCache stop counting for training and grounding.

About the author
M Asif Rahman
Entrepreneur, curious about AI
I have been building companies for a long time, and lately most of my curiosity goes to AI. AIScan is a small personal project I built on a flight from Dubai to Narita. I use it to understand how ready the web is for AI agents, and I share what the scans teach me.
Related guides
Add the __usage_ai-training Token on 10 Platforms (Tested 2026)
To add the usageaitraining token, put a separate UserAgent: usageaitraining group in whatever file or setting actually produces your robots.txt: a template on Shopify, an editor on Wix and Webflow, a…
Content Signals in robots.txt: 117 of 799 Sites Declare One
Of 799 sites in our scan database on 2 November 2026, 117 serve a robots.txt with a ContentSignal line 14.6%. Among the 677 sites that serve a usable robots.txt at all, that is 17.3%. Another 682…
AI readiness setup guide by platform (2026)
Verified 17, 18, 20 and 22 October 2026. Updated on 22 October 2026. An AIready site needs six things: a robots.txt that names AI crawlers D1, B2, a sitemap D2, a declared feed E5, an llms.txt C2,…
