Google-Extended Allow Overrides Your __usage_ai-training Opt-Out: The Precedence Table

If your robots.txt has a Google-Extended group with Allow: /, adding the new __usage_ai-training opt-out does nothing at Google. Google said on 30 Oct 2026 that it will read the token as a Google-Extended disallow only when "Google-Extended is not explicitly specified". A named group wins. Many files name it with Allow: /, and that includes the B2 example block AIScan has been recommending.

Why the named group wins

The shared token comes from an Internet-Draft, draft-kelley-aipref-training-preference-signal-00, filed 30 Oct 2026. Its §4.5 sets the precedence rule in three lines: "Specific expressions SHOULD take precedence over general ones", consumers "MAY document that they interpret rules that match proprietary policy tokens as being more specific", and private agreements may override both.

Google is the first consumer to use that permission. Its Search Central post of 30 Oct 2026 says the token opt-out counts "as if the publisher had disallowed Google-Extended" only when Google-Extended is not named, and adds: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." Google's common crawlers page now carries a note that it intends to support the token "in early 2027". So Google-Extended stays Google's own switch, and the token only fills the gap when that switch is missing.

That is a reasonable design. It is also a trap for anyone who copied a starter file a year ago and is now adding one line to opt out of training everywhere.

The precedence table: nine files, one reading each

We wrote nine robots.txt files, one per combination, and ran each through a 39-line checker on Python 3.11.18 on 11 Nov 2026. The checker groups lines the way RFC 9309 §2.1 and §2.2 define, and it does not let User-agent: * stand in for the token, as draft §4.2 requires. The outcome column quotes or follows Google's wording. Where Google has said nothing, the cell says so.

Row 8 is the one that catches people. Two more cases from the same lab are worth knowing:

• A partial Google-Extended group still counts as "specified". With the token on Disallow: / and Google-Extended on Disallow: /private/ only, the checker reads / as allowed for Google-Extended. Under RFC 9309 §2.2.2 a path with no matching rule is allowed. This is our reading of Google's wording plus the RFC, not something Google has spelled out.
• Stacking the two tokens in one group is fine here. User-Agent: __usage_ai-training directly above User-agent: Google-Extended, then Disallow: /, gives both tokens the same rule. It is a different story with User-agent: *, which our policy token guide covers in its lab.

Other providers are a separate question. The draft's precedence rule is a SHOULD and a MAY, and OpenAI's 30 Oct page does not say whether GPTBot or the token wins when they disagree. This table is about Google only, and it describes Google's stated plan for 2027, not behaviour you can observe today.

Find out which row your file is in

Step 1. Read the served file, not the one in your repo. A CDN or plugin can rewrite it.

curl -s https://yoursite.com/robots.txt | grep -n -i -A3 -E '^user-agent: *(google-extended|__usage_ai-training)'

If Google-Extended appears with Allow: under it, you are in row 2, 5 or 8.

Step 2. Run the checker for an exact verdict. It is the lab script, cut to the part you need:

import sys, urllib.request

def groups(text):
    out, agents, rules, last = [], [], [], None
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        key, val = (s.strip() for s in line.split(":", 1))
        if key.lower() == "user-agent":
            if last == "rule":
                out.append((agents, rules)); agents, rules = [], []
            agents.append(val.lower()); last = "ua"
        elif key.lower() in ("allow", "disallow") and agents:
            rules.append((key.lower(), val)); last = "rule"
    if agents:
        out.append((agents, rules))
    return out

def verdict(text, token, path="/"):
    rules = [r for a, rs in groups(text) if token in a for r in rs]
    if not rules:
        return None
    hits = [(len(v), k == "allow") for k, v in rules if v and path.startswith(v)]
    return max(hits)[1] if hits else True

text = urllib.request.urlopen(sys.argv[1]).read().decode()
print("token:", verdict(text, "__usage_ai-training"), "google-extended:", verdict(text, "google-extended"))

Run it as python3 check.py https://yoursite.com/robots.txt. None means no group names that token, True means allowed, False means disallowed. Expected output for a correct opt-out:

token: False google-extended: False

token: False google-extended: True is row 8.

The fix

You have two clean options. Both reach the same state at Google.

Option A: say it twice, the same way. Keep the named group and make it agree with the token:

User-Agent: __usage_ai-training
Disallow: /

User-agent: Google-Extended
Disallow: /

Option B: delete the Google-Extended group and let the token speak for Google from 2027. Until Google ships support, though, nothing in your file opts you out of Gemini training, so Option A is the safer choice today.

Keep the blank line between groups. Without it, both tokens join one group, which is harmless here but turns harmful the moment someone adds User-agent: * to the stack.

If you want the opposite outcome, training out but Gemini grounding in, Google's wording gives you no split. One Google-Extended group controls both, so you have to pick one. That is the honest trade-off behind row 8, and some sites will choose Allow: / on purpose. The point is to choose it, not inherit it.

On WordPress, edit robots.txt with ThinkRank, which keeps the file in one place instead of three plugins fighting over it, and migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Yoast and Rank Math also have robots.txt editors if you already use them. On any other stack, edit the file your server sends, which our platform-by-platform token guide maps out.

About AIScan's own example. Our B2 guidance has shown a block with User-agent: Google-Extended and Allow: /. That block is right if you want to be in Gemini, and wrong if you add the token to leave training. B2 passes on explicit named rules either way, so the scan will not flag row 8 for you. The checker above will.

Verify it with AIScan, then by hand

Run npx aiscan-cli yoursite.com and look at B2, Explicit AI bot rules. It should list Google-Extended among your named groups. Then run the checker and confirm google-extended: False if you meant to opt out. For the full starter file with search crawlers kept open, see the robots.txt starter for AI crawlers, and for what Google-Extended does not control, read what you can control in Google AI Overviews.

Your next step

Scan your site at aiscan.site, open the B2 check page, and fix row 8 before 2027 makes it matter. More short fixes are in the guides.