---
title: "Add the __usage_ai-training Token on 10 Platforms (Tested 2026)"
slug: usage-ai-training-token-every-platform
published: 2026-11-11T08:15:09.653845+00:00
updated: 2026-11-11T08:15:09.653845+00:00
author: "M Asif Rahman"
author_url: https://ar.bd
category: "AI Readiness"
tags: check:B2, platform:wordpress, platform:shopify, platform:cloudflare, platform:nextjs, robots.txt, AI training
description: "Where to add the __usage_ai-training robots.txt group on WordPress, Shopify, Wix, Webflow, Cloudflare, Next.js, Astro, Hugo and Docusaurus, lab-tested."
url: https://aiscan.site/blog/usage-ai-training-token-every-platform
---

To add the `__usage_ai-training` token, put a separate `User-Agent: __usage_ai-training` group in whatever file or setting actually produces your robots.txt: a template on Shopify, an editor on Wix and Webflow, a filter or editor on WordPress, a source file on static site generators. Squarespace is the one platform in this guide where we found no documented way to add it. Every platform below can also break the file in the same way, by stacking the token on `User-agent: *`, so each section ends with the same check.

This is the where-to-type-it companion to [our explainer on what the token means and how three parsers read it](/blog/usage-ai-training-robots-txt-token). We built the framework cases ourselves on 11 Nov 2026 and ran every generated file through two parsers.

## Quick summary

| Platform | Where the group goes | Can you add it? | Source, read 11 Nov 2026 |
|---|---|---|---|
| WordPress | SEO plugin robots.txt editor, the `robots_txt` filter, or a physical `robots.txt` | Yes | [WordPress `do_robots()` reference](https://developer.wordpress.org/reference/functions/do_robots/) |
| Shopify | `templates/robots.txt.liquid`, after the default-groups loop | Yes, as an unsupported customization | [Shopify developer docs](https://shopify.dev/docs/storefronts/themes/seo/robots-txt) |
| Wix | **SEO & GEO > Tools and settings > Robots.txt Editor** | Yes | [Wix Help Center](https://support.wix.com/en/article/editing-your-sites-robotstxt-file) |
| Squarespace | Nowhere: only a "Block known artificial intelligence crawlers" checkbox | No | [Squarespace Help Center](https://support.squarespace.com/hc/en-us/articles/360022347072) |
| Webflow | **Site settings > SEO > Indexing**, robots.txt field | Yes | [Webflow Help Center](https://help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules) |
| Cloudflare (managed robots.txt on) | Your origin file. Cloudflare prepends its own block above it | Yes, at the origin | [Cloudflare docs](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) |
| Next.js | `app/robots.ts` as its own rule object, or a static `app/robots.txt` | Yes, tested on 16.4.6 | [Next.js docs](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots) |
| Astro | `public/robots.txt` | Yes, tested on 7.4.10 | [Astro project structure](https://docs.astro.build/en/basics/project-structure/) |
| Hugo | `layouts/robots.txt` with `enableRobotsTXT = true`, or `static/robots.txt` | Yes, tested on 0.169.0 | [Hugo robots.txt templates](https://gohugo.io/templates/robots/) |
| Docusaurus | `static/robots.txt` | Yes | [Docusaurus static assets](https://docusaurus.io/docs/static-assets) |

**Short answer:** whatever the platform, the group you add is the same four lines, it sits below your existing groups with a blank line above it, and any `Allow:` exception goes above the `Disallow:` it carves out. Then run `npx aiscan-cli yoursite.com` and confirm check B2 still passes.

## The group you are adding, and the one mistake every platform allows

The [Internet-Draft](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) (version 00, read 11 Nov 2026) defines `__usage_ai-training` as a policy token. Google ([30 Oct 2026](https://developers.google.com/search/blog/2026/10/ai-training-control)) says it will support it in 2027; OpenAI says "in the coming months". It expresses a training preference and is not a crawl rule, so adding it today changes no crawler's behaviour, as long as it lives in its own group.

```
User-Agent: __usage_ai-training
Allow: /press-kit/
Disallow: /
```

Drop the `Allow:` line if you have no exception. Think of robots.txt groups like labelled envelopes: each `User-agent` line is a name on the envelope, and every rule inside goes to every name on it. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) treats consecutive `User-agent` lines as one group. Write the token line directly under `User-agent: *` and the `Disallow: /` meant for training now tells Googlebot to leave too. Our pillar post measured that in three parsers; this post found a framework API that writes it for you (the Next.js section).

## What our lab builds produced

On 11 Nov 2026 we built each framework case locally, with no network fetches of anyone else's site, and parsed the output with Python 3.11.18's `urllib.robotparser` and Protego 0.7.0 (the parser behind Scrapy). Four agents and two paths per file.

| Build | Token at `/` | Token at `/press-kit/logo.png` | Googlebot at `/` | GPTBot at `/` | Google-Extended at `/` | Both parsers agree? |
|---|---|---|---|---|---|---|
| Next.js 16.4.6, `app/robots.ts`, token as its own rule object | no training | allowed | allowed | allowed | blocked (by design) | yes |
| Next.js 16.4.6, `userAgent: ['*', '__usage_ai-training']` | no training | no training | **blocked** | **blocked** | blocked | yes |
| Astro 7.4.10, `public/robots.txt` | no training | allowed | allowed | allowed | allowed | yes |
| Hugo 0.169.0, `layouts/robots.txt` template | no training | allowed | allowed | allowed | allowed | yes |
| WordPress `robots_txt` filter (PHP 8.3.6, core output stubbed) | no training | allowed | allowed | allowed | allowed | yes |
| Cloudflare managed block prepended to an origin file with the token | no training | allowed | allowed | blocked (managed) | blocked (managed) | yes at `/`; **no** on a second `*` rule |

Three results matter. The Next.js array form, which looks like a tidy way to write "same rules for both", emits the token stacked on `User-agent: *` and blocks the whole site for every crawler. Astro copied `public/robots.txt` byte for byte (`cmp` reported no difference). And the Cloudflare case exposed an old parser gap that matters more on Cloudflare than anywhere else, covered in its section.

We also fetched the live robots.txt of our own four sites (aiscan.site, xspeedcache.com, storeseo.com, bettershield.ai) on 11 Nov 2026: none carries the token yet, and none shows a Cloudflare managed block.

## WordPress

WordPress serves a virtual robots.txt built by `do_robots()` unless a real `robots.txt` file sits in your web root, in which case the web server sends that file and WordPress never runs. Find out which you have before editing: if your hosting file manager shows a `robots.txt` next to `wp-config.php`, edit that file.

**Path 1: an SEO plugin editor (recommended).** [**ThinkRank**](https://thinkrank.ai) is the plugin we recommend here. It edits robots.txt, robots meta, llms.txt, schema and sitemaps from one place, so one plugin owns the file instead of two overwriting each other, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress.

1. Open the plugin's robots.txt editor.
2. Scroll to the end of the existing content.
3. Add a blank line, then the token group.
4. Save, then fetch `https://yoursite.com/robots.txt` and confirm the group is there.

Yoast SEO and Rank Math have robots.txt editors too, and the same four steps apply if one of them already runs your site. Use only one.

**Path 2: the `robots_txt` filter (no plugin).** WordPress passes the generated text and the "Search engine visibility" setting through the `robots_txt` filter. Put this in a small must-use plugin (`wp-content/mu-plugins/ai-training-token.php`) so a theme switch does not drop it:

```php
<?php
add_filter( 'robots_txt', function ( $output, $public ) {
    if ( ! $public ) {
        return $output; // "Discourage search engines" is on: leave the file alone
    }
    $output .= "\nUser-Agent: __usage_ai-training\nAllow: /press-kit/\nDisallow: /\n";
    return $output;
}, 20, 2 );
```

We ran this callback with PHP 8.3.6 against the core default output (`User-agent: *`, `Disallow: /wp-admin/`, `Allow: /wp-admin/admin-ajax.php`) using a stub in place of WordPress; the result parsed as expected. Priority 20 runs it after plugins that use the default 10. If an SEO plugin also writes robots.txt, this filter may append to its output or be replaced by it, so pick one path.

**Path 3: a physical file.** Upload a `robots.txt` to the web root with your full rules plus the group. This overrides every plugin and filter.

## Shopify

Shopify generates robots.txt and lets you replace it with a theme template. Shopify calls this an "unsupported customization": its support team will not help with edits, and a bad file "can result in loss of all traffic".

1. Save a copy of your current `https://yourstore.com/robots.txt`. Shopify's developer docs ask for this, because the template's output "doesn't always match Shopify's generated" file.
2. From your Shopify admin, go to **Online Store**. For your live theme, click **…** > **Edit code**.
3. Click **Add a new template**, select **robots**, then click **Create template**.
4. Leave the Liquid loop over `robots.default_groups` as it is. Shopify's docs place custom groups outside that loop, so add the token group after the closing `{% endfor %}`, with a blank line before it.
5. Save, then compare the live file with your saved copy. Every default group should still be there.

Shopify's help center also notes that agentic storefronts such as ChatGPT receive product data through Shopify Catalog, "independently of /robots.txt", so this token does not reach that channel.

## Wix

Wix maintains the file itself and regenerates its `*` group around your additions, which [our Wix robots.txt post](/blog/robots-txt-ai-bots-wix) covered in detail.

1. Open your Wix dashboard and select **SEO & GEO**.
2. Under **Tools and settings**, click **Robots.txt Editor**.
3. Click **View File**.
4. Under **This is your current file**, add a blank line and the token group at the end.
5. Click **Save Changes**, then **Save**.

If anything goes wrong, the same screen has **Reset to Default**.

## Squarespace

Squarespace writes robots.txt for you and does not let you edit it, as we found when we [set up a Squarespace site for AI agents](/blog/ai-readiness-setup-squarespace). Its help article on AI models (read 11 Nov 2026) documents one control: **Settings > Crawlers > Block known artificial intelligence crawlers**, which adds a fixed list of named crawlers. We found no documented way to add a custom `User-agent` line, so the token cannot be added on Squarespace today. That checkbox blocks crawling by the listed bots, which is a stronger and different statement than a training preference. If you need the token specifically, ask Squarespace support for it.

## Webflow

1. Open **Site settings > SEO > Indexing**.
2. In the robots.txt field, add a blank line and the token group after your existing rules.
3. Save and publish the site.

Webflow's own examples write groups without blank lines. That works (a `User-agent` line after rules starts a new group), but blank lines make a stacked token easy to spot. The same page has **Traffic control** toggles that generate rules for groups of known bots; check the published file to make sure those and your group sit in separate groups.

## Cloudflare

If you use Cloudflare's managed robots.txt, Cloudflare prepends its own block above your origin file whenever the origin answers 200, "combining both into a single response". The managed block shown in Cloudflare's docs on 11 Nov 2026 contained a `Content-signal` line and named crawler groups, and no `__usage_ai-training` line. So the token goes in your origin file, wherever that file comes from (one of the other sections here). See [our Cloudflare AI Crawl Control setup guide](/blog/cloudflare-ai-crawl-control-setup) for the dashboard side.

The combined file has two `User-agent: *` groups: Cloudflare's, then yours. RFC 9309 §2.2.1 says groups matching the same agent are combined. Protego did that. Python's `urllib.robotparser` used only the first `*` group, so an origin rule `Disallow: /lp` was ignored for Googlebot (urllib: allowed; Protego: blocked). The token group itself parsed correctly in both. To verify on Cloudflare:

1. Fetch the served file: `curl -s https://yoursite.com/robots.txt > served.txt`.
2. Fetch the origin file directly, bypassing Cloudflare, into `origin.txt`.
3. Run `diff origin.txt served.txt`. The only difference should be the managed block at the top; your token group should appear once, below `# END Cloudflare Managed Content`.

## Next.js

**Path 1: a static file.** Put a plain `app/robots.txt` in the app directory with the token group in it. Next.js serves it as written.

**Path 2: `app/robots.ts`.** The `userAgent` field is typed `string | string[]`, so the token string type-checks. Give it its own rule object:

```ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/' },
      { userAgent: 'Google-Extended', disallow: '/' },
      { userAgent: '__usage_ai-training', allow: '/press-kit/', disallow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}
```

`next build` on 16.4.6 prerendered this as a static route and wrote each object as a separate group, `Allow` above `Disallow`. **Do not** write `userAgent: ['*', '__usage_ai-training']` with one `disallow: '/'`: the build emitted both `User-Agent` lines over a single `Disallow: /`, and both parsers then blocked every crawler from the whole site. The array form is fine for crawlers that should share rules; never put the token in an array with `*` or with a crawler you want to keep.

## Astro

Astro copies everything in `public/` to the build output unprocessed.

1. Create or open `public/robots.txt`.
2. Add a blank line and the token group below your crawler groups.
3. Run `astro build` and confirm `dist/robots.txt` is unchanged from the source.

If you generate robots.txt from an endpoint or an integration instead, add the group there and check the built file.

## Hugo

With `enableRobotsTXT = true`, Hugo's built-in template writes only `User-agent: *` (that was the full output of our 0.169.0 build). To add the token:

1. Keep `enableRobotsTXT = true` in `hugo.toml`.
2. Create `layouts/robots.txt` with your groups, the token group and `Sitemap: {{ "sitemap.xml" | absURL }}`.
3. Run `hugo` and read `public/robots.txt`.

Our template produced exactly the groups we wrote, with the sitemap resolved to the site's `baseURL`. A plain `static/robots.txt` with `enableRobotsTXT` left off also works.

## Docusaurus

Docusaurus copies every file in `static/` "into the root of the generated build folder". Add the token group to `static/robots.txt` (create it if missing), run the build, and check `build/robots.txt`. [Our Docusaurus robots.txt post](/blog/robots-txt-ai-bots-docusaurus) covers the crawler groups that usually sit above it.

## Verify the edit: AIScan first, then a parser

The fastest check is a scan: run `npx aiscan-cli yoursite.com` or paste the address at [aiscan.site](https://aiscan.site/). It is free and needs no account. Read three results:

- **B2 (explicit AI bot rules)** passes when your file names at least four of the sixteen AI crawlers on AIScan's watchlist. The training token is not on that watchlist. If B2 was passing yesterday and fails after your edit, a crawler group got merged with the token or deleted.
- **B1 (Content Signals)** is informational and only reports a `Content-Signal` line. The token and that line are independent; having one does not satisfy the other.
- The file itself has to come back as HTTP 200 with `text/plain`, or crawlers ignore all of it ([robots.txt status and content type](/blog/robots-txt-status-content-type) explains the failure modes).

The rubric for each is on [/docs/checks/bot-access](https://aiscan.site/docs/checks/bot-access). What AIScan cannot see: whether Google or OpenAI has read your preference, because neither had shipped support on 11 Nov 2026.

For a manual check, ask Python whether Googlebot may still fetch your homepage:

```
python3 -c "import urllib.robotparser as r; p=r.RobotFileParser('https://yoursite.com/robots.txt'); p.read(); print(p.can_fetch('Googlebot','https://yoursite.com/'))"
```

`True` is the pass. `False` straight after this edit is the stacked-token mistake, a modern route to [a robots.txt that blocks the whole site](/blog/robots-txt-disallow-all-by-accident). To read the training preference itself rather than crawl access, use the spec-aware checker in [the token explainer](/blog/usage-ai-training-robots-txt-token).

## Mistakes by platform

| Platform | Mistake | What happens | Fix |
|---|---|---|---|
| Next.js | `userAgent: ['*', '__usage_ai-training']` | Whole site blocked for all crawlers (lab, 2 of 2 parsers) | One rule object per group |
| Any editor | Token line typed under `User-agent: *` | Same as above | Blank line, then the token on its own |
| WordPress | Filter plus SEO plugin editor | One overwrites the other | Pick one path |
| Shopify | Token placed inside the `default_groups` loop | Repeated under every default group | Put it after `{% endfor %}` |
| Cloudflare | Expecting the managed block to add the token | It does not (docs read 11 Nov 2026) | Add it at the origin |
| Hugo | Only `enableRobotsTXT = true` | File contains `User-agent: *` and nothing else | Add `layouts/robots.txt` |
| Any | `Disallow: /` above `Allow: /press-kit/` | First-match parsers ignore the exception | `Allow:` first |

## Your next step

Find the platform section that matches how your robots.txt is produced, add the group, then run `npx aiscan-cli yoursite.com` and confirm B2 still passes. For the crawler groups themselves, start from our [robots.txt starter for AI crawlers](/blog/robots-txt-starter-for-ai-crawlers); for the other signals that state how AI may use your content, see [how to declare your content-use policy to AI](/blog/declare-content-use-policy-to-ai). Every other guide is at [/guides](https://aiscan.site/guides).