Add the __usage_ai-training Token on 10 Platforms (Tested 2026)

To add the __usage_ai-training token, put a separate User-Agent: __usage_ai-training group in whatever file or setting actually produces your robots.txt: a template on Shopify, an editor on Wix and Webflow, a filter or editor on WordPress, a source file on static site generators. Squarespace is the one platform in this guide where we found no documented way to add it. Every platform below can also break the file in the same way, by stacking the token on User-agent: *, so each section ends with the same check.

This is the where-to-type-it companion to our explainer on what the token means and how three parsers read it. We built the framework cases ourselves on 11 Nov 2026 and ran every generated file through two parsers.

Quick summary

Short answer: whatever the platform, the group you add is the same four lines, it sits below your existing groups with a blank line above it, and any Allow: exception goes above the Disallow: it carves out. Then run npx aiscan-cli yoursite.com and confirm check B2 still passes.

The group you are adding, and the one mistake every platform allows

The Internet-Draft (version 00, read 11 Nov 2026) defines __usage_ai-training as a policy token. Google (30 Oct 2026) says it will support it in 2027; OpenAI says "in the coming months". It expresses a training preference and is not a crawl rule, so adding it today changes no crawler's behaviour, as long as it lives in its own group.

User-Agent: __usage_ai-training
Allow: /press-kit/
Disallow: /

Drop the Allow: line if you have no exception. Think of robots.txt groups like labelled envelopes: each User-agent line is a name on the envelope, and every rule inside goes to every name on it. RFC 9309 treats consecutive User-agent lines as one group. Write the token line directly under User-agent: * and the Disallow: / meant for training now tells Googlebot to leave too. Our pillar post measured that in three parsers; this post found a framework API that writes it for you (the Next.js section).

What our lab builds produced

On 11 Nov 2026 we built each framework case locally, with no network fetches of anyone else's site, and parsed the output with Python 3.11.18's urllib.robotparser and Protego 0.7.0 (the parser behind Scrapy). Four agents and two paths per file.

Three results matter. The Next.js array form, which looks like a tidy way to write "same rules for both", emits the token stacked on User-agent: * and blocks the whole site for every crawler. Astro copied public/robots.txt byte for byte (cmp reported no difference). And the Cloudflare case exposed an old parser gap that matters more on Cloudflare than anywhere else, covered in its section.

We also fetched the live robots.txt of our own four sites (aiscan.site, xspeedcache.com, storeseo.com, bettershield.ai) on 11 Nov 2026: none carries the token yet, and none shows a Cloudflare managed block.

WordPress

WordPress serves a virtual robots.txt built by do_robots() unless a real robots.txt file sits in your web root, in which case the web server sends that file and WordPress never runs. Find out which you have before editing: if your hosting file manager shows a robots.txt next to wp-config.php, edit that file.

Path 1: an SEO plugin editor (recommended). ThinkRank is the plugin we recommend here. It edits robots.txt, robots meta, llms.txt, schema and sitemaps from one place, so one plugin owns the file instead of two overwriting each other, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress.

1. Open the plugin's robots.txt editor.
2. Scroll to the end of the existing content.
3. Add a blank line, then the token group.
4. Save, then fetch https://yoursite.com/robots.txt and confirm the group is there.

Yoast SEO and Rank Math have robots.txt editors too, and the same four steps apply if one of them already runs your site. Use only one.

Path 2: the robots_txt filter (no plugin). WordPress passes the generated text and the "Search engine visibility" setting through the robots_txt filter. Put this in a small must-use plugin (wp-content/mu-plugins/ai-training-token.php) so a theme switch does not drop it:

<?php
add_filter( 'robots_txt', function ( $output, $public ) {
    if ( ! $public ) {
        return $output; // "Discourage search engines" is on: leave the file alone
    }
    $output .= "\nUser-Agent: __usage_ai-training\nAllow: /press-kit/\nDisallow: /\n";
    return $output;
}, 20, 2 );

We ran this callback with PHP 8.3.6 against the core default output (User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php) using a stub in place of WordPress; the result parsed as expected. Priority 20 runs it after plugins that use the default 10. If an SEO plugin also writes robots.txt, this filter may append to its output or be replaced by it, so pick one path.

Path 3: a physical file. Upload a robots.txt to the web root with your full rules plus the group. This overrides every plugin and filter.

Shopify

Shopify generates robots.txt and lets you replace it with a theme template. Shopify calls this an "unsupported customization": its support team will not help with edits, and a bad file "can result in loss of all traffic".

1. Save a copy of your current https://yourstore.com/robots.txt. Shopify's developer docs ask for this, because the template's output "doesn't always match Shopify's generated" file.
2. From your Shopify admin, go to Online Store. For your live theme, click … > Edit code.
3. Click Add a new template, select robots, then click Create template.
4. Leave the Liquid loop over robots.default_groups as it is. Shopify's docs place custom groups outside that loop, so add the token group after the closing {% endfor %}, with a blank line before it.
5. Save, then compare the live file with your saved copy. Every default group should still be there.

Shopify's help center also notes that agentic storefronts such as ChatGPT receive product data through Shopify Catalog, "independently of /robots.txt", so this token does not reach that channel.

Wix

Wix maintains the file itself and regenerates its * group around your additions, which our Wix robots.txt post covered in detail.

1. Open your Wix dashboard and select SEO & GEO.
2. Under Tools and settings, click Robots.txt Editor.
3. Click View File.
4. Under This is your current file, add a blank line and the token group at the end.
5. Click Save Changes, then Save.

If anything goes wrong, the same screen has Reset to Default.

Squarespace

Squarespace writes robots.txt for you and does not let you edit it, as we found when we set up a Squarespace site for AI agents. Its help article on AI models (read 11 Nov 2026) documents one control: Settings > Crawlers > Block known artificial intelligence crawlers, which adds a fixed list of named crawlers. We found no documented way to add a custom User-agent line, so the token cannot be added on Squarespace today. That checkbox blocks crawling by the listed bots, which is a stronger and different statement than a training preference. If you need the token specifically, ask Squarespace support for it.

Webflow

1. Open Site settings > SEO > Indexing.
2. In the robots.txt field, add a blank line and the token group after your existing rules.
3. Save and publish the site.

Webflow's own examples write groups without blank lines. That works (a User-agent line after rules starts a new group), but blank lines make a stacked token easy to spot. The same page has Traffic control toggles that generate rules for groups of known bots; check the published file to make sure those and your group sit in separate groups.

Cloudflare

If you use Cloudflare's managed robots.txt, Cloudflare prepends its own block above your origin file whenever the origin answers 200, "combining both into a single response". The managed block shown in Cloudflare's docs on 11 Nov 2026 contained a Content-signal line and named crawler groups, and no __usage_ai-training line. So the token goes in your origin file, wherever that file comes from (one of the other sections here). See our Cloudflare AI Crawl Control setup guide for the dashboard side.

The combined file has two User-agent: * groups: Cloudflare's, then yours. RFC 9309 §2.2.1 says groups matching the same agent are combined. Protego did that. Python's urllib.robotparser used only the first * group, so an origin rule Disallow: /lp was ignored for Googlebot (urllib: allowed; Protego: blocked). The token group itself parsed correctly in both. To verify on Cloudflare:

1. Fetch the served file: curl -s https://yoursite.com/robots.txt > served.txt.
2. Fetch the origin file directly, bypassing Cloudflare, into origin.txt.
3. Run diff origin.txt served.txt. The only difference should be the managed block at the top; your token group should appear once, below # END Cloudflare Managed Content.

Next.js

Path 1: a static file. Put a plain app/robots.txt in the app directory with the token group in it. Next.js serves it as written.

Path 2: app/robots.ts. The userAgent field is typed string | string[], so the token string type-checks. Give it its own rule object:

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/' },
      { userAgent: 'Google-Extended', disallow: '/' },
      { userAgent: '__usage_ai-training', allow: '/press-kit/', disallow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

next build on 16.4.6 prerendered this as a static route and wrote each object as a separate group, Allow above Disallow. Do not write userAgent: ['*', '__usage_ai-training'] with one disallow: '/': the build emitted both User-Agent lines over a single Disallow: /, and both parsers then blocked every crawler from the whole site. The array form is fine for crawlers that should share rules; never put the token in an array with * or with a crawler you want to keep.

Astro

Astro copies everything in public/ to the build output unprocessed.

1. Create or open public/robots.txt.
2. Add a blank line and the token group below your crawler groups.
3. Run astro build and confirm dist/robots.txt is unchanged from the source.

If you generate robots.txt from an endpoint or an integration instead, add the group there and check the built file.

Hugo

With enableRobotsTXT = true, Hugo's built-in template writes only User-agent: * (that was the full output of our 0.169.0 build). To add the token:

1. Keep enableRobotsTXT = true in hugo.toml.
2. Create layouts/robots.txt with your groups, the token group and Sitemap: {{ "sitemap.xml" | absURL }}.
3. Run hugo and read public/robots.txt.

Our template produced exactly the groups we wrote, with the sitemap resolved to the site's baseURL. A plain static/robots.txt with enableRobotsTXT left off also works.

Docusaurus

Docusaurus copies every file in static/ "into the root of the generated build folder". Add the token group to static/robots.txt (create it if missing), run the build, and check build/robots.txt. Our Docusaurus robots.txt post covers the crawler groups that usually sit above it.

Verify the edit: AIScan first, then a parser

The fastest check is a scan: run npx aiscan-cli yoursite.com or paste the address at aiscan.site. It is free and needs no account. Read three results:

• B2 (explicit AI bot rules) passes when your file names at least four of the sixteen AI crawlers on AIScan's watchlist. The training token is not on that watchlist. If B2 was passing yesterday and fails after your edit, a crawler group got merged with the token or deleted.
• B1 (Content Signals) is informational and only reports a Content-Signal line. The token and that line are independent; having one does not satisfy the other.
• The file itself has to come back as HTTP 200 with text/plain, or crawlers ignore all of it (robots.txt status and content type explains the failure modes).

The rubric for each is on /docs/checks/bot-access. What AIScan cannot see: whether Google or OpenAI has read your preference, because neither had shipped support on 11 Nov 2026.

For a manual check, ask Python whether Googlebot may still fetch your homepage:

python3 -c "import urllib.robotparser as r; p=r.RobotFileParser('https://yoursite.com/robots.txt'); p.read(); print(p.can_fetch('Googlebot','https://yoursite.com/'))"

True is the pass. False straight after this edit is the stacked-token mistake, a modern route to a robots.txt that blocks the whole site. To read the training preference itself rather than crawl access, use the spec-aware checker in the token explainer.

Mistakes by platform

Your next step

Find the platform section that matches how your robots.txt is produced, add the group, then run npx aiscan-cli yoursite.com and confirm B2 still passes. For the crawler groups themselves, start from our robots.txt starter for AI crawlers; for the other signals that state how AI may use your content, see how to declare your content-use policy to AI. Every other guide is at /guides.