The __usage_ai-training robots.txt Token, Tested in 3 Parsers (2026)

__usage_ai-training is a robots.txt "policy token": you write User-Agent: __usage_ai-training followed by Disallow: and Allow: paths, and participating AI companies read that group as your preference about AI training. It is not a crawl rule, so on its own it stops nothing from fetching your pages. Google says it will support the token in 2027 and OpenAI says "in the coming months", so today it changes no crawler's behaviour unless you put it in the wrong place.

We ran eight robots.txt files through three deployed parsers to see what "the wrong place" means. Two placements quietly change crawling for every bot, and one common parser gets the draft's own example wrong.

Quick summary

Short answer for a site owner: add the token as its own group, separated by a blank line, below your existing crawler groups. Keep the crawler groups you have. Then run npx aiscan-cli yoursite.com (or paste the address at aiscan.site, free, no account) and read check B2 to confirm your named crawler groups survived the edit. If you would rather check by hand, the parser test further down runs in a few seconds.

Why one token, and why now

Until now, opting out of AI training meant one robots.txt group per company: Google-Extended for Google, GPTBot for OpenAI, and so on, each with its own name and its own meaning. Our user-agent list tracks those names, and it keeps growing. A site that wanted "no training, but keep me in search" had to know which name meant training and which meant search for every vendor.

The draft's answer is one shared name for one purpose. Its author, K. Kelley of Anthropic, filed it on 30 October 2026 for the IETF AI Preferences (aipref) working group, and it expires on 3 May 2027 unless it is revised. The same day, Google's Search Central blog said "We aim to support this new control along with other AI providers in 2027", and OpenAI said "We aim to implement the signal in the coming months."

Update, 12 Nov 2026: Microsoft has confirmed it first-party. Bing's post of 30 October 2026 says that "Starting February 1, 2027, Microsoft will honor the new preferences as it crawls", with the same User-Agent: __usage_ai-training group in robots.txt. It adds page-level controls of its own (<meta name="Bingbot" content="ai-training:disallow">, or the same value in an X-Robots-Tag: Bingbot: header, plus a separate ai-grounding preference), and says these replace NoArchive and NoCache as its AI controls, with a 1 March 2027 deadline to switch. Update, 13 Nov 2026: Anthropic has confirmed it first-party too. Its crawler help page (last updated 4 November 2026) says: "As of October 30, 2026, Anthropic has begun recognizing a machine-readable signal that lets website owners express a preference about whether their content may be used to train AI models", and that it is "rolling this out across all of our services over time". It also states its precedence rule: a robots.txt rule that names ClaudeBot is treated as more specific than the general signal and wins for that path; if ClaudeBot is not named, the general setting applies. So Anthropic is the only one of the four that says it has started, Microsoft starts on 1 February 2027, and Google and OpenAI give 2027 and "the coming months".

What the draft actually says

The whole mechanism fits in five rules. We read them from the draft text itself, version 00, on 11 Nov 2026.

Training has a defined meaning here, taken from the working group's vocabulary draft (draft-ietf-aipref-vocab-08, 13 Sep 2026): "Using an asset to modify the learned parameters of a generative AI model." Models used only for classification, ranking or scoring are out of scope. So the token is about training generative models. It does not address search indexing, and it does not address an assistant fetching your page to answer one question. Those still belong to the crawler groups and to the separate signals we covered in how to declare your content-use policy to AI.

The draft's own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That reads as "no training on anything, except the press kit, which you may train on".

The parser lab: what deployed software does with the token

The draft's design goal 4 is "Unchanged legacy behavior", and it says this "holds for parsers that are actually deployed, not only for idealized RFC 9309 parsers." That is a testable claim, so we tested it.

Method. Eight small robots.txt files, five agent names (__usage_ai-training, Googlebot, GPTBot, Google-Extended, ClaudeBot), and two paths (/ and /press-kit/logo.png). Each file went through three open-source parsers that real crawlers and tools use: Python 3.11.18's standard-library urllib.robotparser, Protego 0.7.0 (the parser Scrapy uses), and the npm package robots-parser 3.0.1. Run locally on 11 Nov 2026, no network, no third-party site fetched. Eighty answers per parser.

The eight files and the results

"Blocked" for the token row means the parser answered can_fetch = false when asked about the token as if it were a bot. A spec-aware reader would translate that into "no training". ClaudeBot and Google-Extended matched Googlebot in every file except where named.

Finding 1: generic parsers read User-agent: * as a training preference

In file 1 every parser said the token was blocked, because every one of them falls back to the * group for a name it does not know. That is the normal RFC 9309 behaviour for crawlers, and §4.2 tells consumers to do the opposite for this token. The reverse also holds: on a file containing only User-agent: * / Allow: /, all three parsers answered "allowed" for the token, where the draft says no preference exists.

What this means: a company that reuses its crawler's parser to read the preference, without the §4.2 exception, would treat every site with a wildcard Disallow: / as having opted out of training and every wildcard Allow: / as having opted in. Neither is true. You cannot fix that from your side, but you can avoid depending on it: if you care about training, write the token group explicitly instead of trusting the wildcard.

Finding 2: stacking the token on User-agent: * blocks every crawler

File 3 is the mistake we expect to see most, because it looks tidy. RFC 9309 §2.2 (the section the draft cites) treats consecutive User-agent lines as one group, so the token and * share the same Disallow: /. All three parsers blocked Googlebot, GPTBot, ClaudeBot and Google-Extended from the whole site. The result is a site that has left search because its owner wanted to opt out of training. We have written before about a robots.txt that disallows everything by accident; this is a new way to get there.

Finding 3: stacking it on a crawler group changes that crawler's rules too

File 4 is legal under §4.1, and every parser did what the RFC says: GPTBot was blocked from crawling. If your intent was "OpenAI may not train on my site, but GPTBot may still read it", this file says the opposite for the crawl. Mixing is fine only when the crawler rule and the training rule really are the same paths. Most people want them separate.

Finding 4: Python's standard parser gets the draft's own example wrong

File 5 is the draft's Figure 1. Protego and robots-parser both returned "allowed" for /press-kit/logo.png, because they apply RFC 9309 §2.2.2's longest-match rule. Python's urllib.robotparser returned "blocked": it walks the rules in file order and stops at the first match, and Disallow: / comes first. Reversing the two lines (Allow: /press-kit/ above Disallow: /) made all three agree. This is not a token problem. It is an old parser behaviour that path exceptions expose, and we have seen the same library cause trouble in our post on Python's robotparser and a 403 robots.txt.

Matching is not the only place parsers split. Our five-parser robots.txt lab found Python 3.13 already picks the longest match, and that version tokens, a byte order mark and files over 500 KiB divide the libraries.

Practical rule: put Allow: exceptions above the Disallow: they carve out. Longest-match parsers do not care about order, and first-match parsers then get it right too.

Finding 5: case and blank lines are not the problem

The upper-case token (file 7) matched in all three parsers, as §4.1 requires. File 8, with no blank line before the next User-agent, still split into two groups in all three, because a User-agent line after rules starts a new group. Blank lines are for people. Use them anyway: they make file 3's mistake visible.

A spec-aware checker you can run

No parser we found implements §4.2 yet, so here is a small one. It reads only groups that name the token, ignores *, returns None when there is no token line, and applies longest match with Allow winning a tie. It gave the expected answer on all eight lab files, including "training allowed" for the press kit in file 5 and None for file 1.

import re

TOKEN = "__usage_ai-training"

def parse_groups(robots_txt):
    groups, agents, rules, seen_rule = [], [], [], False
    for raw in robots_txt.splitlines():
        line = raw.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        key, value = (part.strip() for part in line.split(":", 1))
        key = key.lower()
        if key == "user-agent":
            if seen_rule:  # a User-agent line after rules starts a new group
                groups.append((agents, rules))
                agents, rules, seen_rule = [], [], False
            agents.append(value.lower())
        elif key in ("allow", "disallow"):
            seen_rule = True
            rules.append((key, value))
    if agents:
        groups.append((agents, rules))
    return groups

def matches(pattern, path):
    rx = re.escape(pattern).replace(r"\*", ".*")
    if rx.endswith(r"\$"):
        rx = rx[:-2] + "$"
    return re.match(rx, path) is not None

def training_preference(robots_txt, path):
    groups = [g for g in parse_groups(robots_txt) if TOKEN in g[0]]
    if not groups:          # no token line, or only "*": no preference (s4.2)
        return None
    best = None             # longest match wins, Allow wins a tie (RFC 9309 s2.2.2)
    for _, rules in groups:
        for kind, pattern in rules:
            if pattern and matches(pattern, path):
                rank = (len(pattern), kind == "allow")
                if best is None or rank > best[0]:
                    best = (rank, kind)
    if best is None or best[1] == "allow":
        return "training allowed"
    return "no training"

To test your own file: curl -s https://yoursite.com/robots.txt > robots.txt, then in Python print(training_preference(open("robots.txt").read(), "/")). It is a test helper, not a full RFC 9309 implementation: it skips the 500 KiB limit and percent-encoding rules.

Four details in the draft that most how-tos skip

The file that counts is the one live when your page was fetched. The draft defines the asset as "the content retrieved from a path" and says the preferences that apply are "those expressed in the robots.txt file in effect when the asset was retrieved". Adding the token next year does not reach back to copies collected this year. That is one more reason to add it now, even though nobody reads it yet: from the day the first company ships, every fetch it makes of your pages is covered by the file as it stands.

Private agreements win. §4.5 says "Private agreements MAY override preferences expressed via this mechanism." If you have licensed your content to an AI company, the contract decides, not the robots.txt line. If you have not, the line is the statement you have.

Vendor tokens can outrank the shared one, but only if the vendor says so. §4.5 lets a consumer "document that they interpret rules that match proprietary policy tokens as being more specific". Google has documented exactly that for Google-Extended. OpenAI's 30 Oct page says its existing crawler controls "will continue to apply" but does not say which wins when its own group and the token disagree. Until it does, do not write files where they disagree unless you are ready for either reading.

File size still matters. The draft lists "No large-scale duplication" as a goal, noting that robots.txt files are size-constrained under RFC 9309 §2.5 (parsers must read at least the first 500 kibibytes). The token adds one group, not one group per vendor, which is the point. If your file is already huge because a plugin writes a group per bot, the token is a chance to trim it later, once support ships, not now.

Timeline as of 11 Nov 2026

How the token combines with the controls you already have

Here is what each combination means, built from the draft and from what Google and OpenAI wrote on 30 Oct 2026. Where a company has not said how it resolves a conflict, the cell says so.

Two things stand out. First, the token is additive: nothing in it removes the need for crawler groups if you want to control fetching, and our robots.txt starter for AI crawlers still applies line for line. Second, Content Signals and the token are separate signals with separate readers. Cloudflare's Content-Signal line (contentsignals.org) is not mentioned in the draft, and a file that uses only one says nothing in the other's terms. If you already publish ai-train=no, add the token group too; they do not conflict. If your signal line itself is the problem, see Content-Signal syntax not understood.

What to write: four decisions, four files

The first file in full, for a site that allows search and named crawlers but opts out of training:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Allow: /

User-Agent: __usage_ai-training
Disallow: /

Sitemap: https://example.com/sitemap.xml

The Google-Extended line is there because Google-Extended already works today, documented in Google's common crawlers list, while the token does not yet. Keep it until Google ships support, then decide whether you still need both. If you would rather allow Gemini grounding, set it to Allow: / as in the third row.

Blocking training crawlers has limits that a robots.txt line cannot fix: content already collected stays collected, and only compliant bots read the file at all. We went through that in does blocking AI training protect your content.

How to make the edit on WordPress and other sites

On WordPress, ThinkRank is the plugin we recommend for this. It edits robots.txt, robots meta, llms.txt, schema and sitemaps in one place, so you have one plugin writing the file instead of two or three overwriting each other, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Add the token group as its own block in the robots.txt editor, below your crawler groups. Yoast SEO and Rank Math both have robots.txt editors too, and either works for this edit if it is already the plugin you run. Whichever you use, check the live file afterwards, because a cache or a second plugin can serve an older copy.

For the exact place to type it on Shopify, Wix, Webflow, Cloudflare, Next.js, Astro, Hugo and Docusaurus, see where to add the token on every platform.

On other stacks, edit the static robots.txt at your web root, or the template your framework generates it from. Keep one source of truth: if a CDN rule or a host setting also writes robots.txt, the token group can disappear on the next deploy. We saw named groups vanish that way in our robots.txt regression study.

Check it with AIScan, then by hand

Run npx aiscan-cli yoursite.com, or paste your address at aiscan.site. The checks that matter after this edit:

• B2, explicit AI bot rules: counts how many of sixteen named AI crawlers your file addresses. Four or more pass. The token is not on the watchlist, so it does not add to the count, but a bad edit that removed or merged your crawler groups will show up here as a drop.
• B1, Content Signals: shown as informational. It tells you whether your Content-Signal line is present, which is a separate statement from the token.
• The robots.txt fetch itself: a file that stops answering 200 as text/plain breaks every rule in it. See robots.txt status and content type.

Details for each check are on /docs/checks/bot-access.

If you would rather check by hand, run the parser test from the lab above against your live file:

python3 -c "import urllib.robotparser as r; p=r.RobotFileParser('https://yoursite.com/robots.txt'); p.read(); print(p.can_fetch('Googlebot','https://yoursite.com/'))"

True means Googlebot can still crawl your homepage. If it prints False after you added the token, you have made the file 3 mistake. If it prints False before any edit, your server may be refusing Python's default user agent, which this parser reads as "disallow everything"; that case has its own post.

Common mistakes

What this post does not measure

We cannot tell you how many sites already use the token. AIScan scans store check results and evidence strings, not the robots.txt body, so there is no adoption count in our data, and we did not crawl other people's files to make one. The lab tests three parsers, not every crawler; Google's own parser and the vendors' training pipelines are not public in their final form, and nobody has shipped support yet. The draft is version 00 and may change before it expires on 3 May 2027. We will re-run the lab when Google or OpenAI ship.

Your next step

Add the token as its own group, keep your crawler groups, put any Allow: exceptions first, then run npx aiscan-cli yoursite.com and confirm B2 still passes. For the rest of the bot access fixes and every other guide on this site, start at /guides.