---
title: "Declare your content-use policy to AI: Content Signals, ai.txt, TDMRep"
slug: declare-content-use-policy-to-ai
published: 2026-10-10T08:15:29.843219+00:00
updated: 2026-10-10T08:15:44.06545+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: check:B1, platform:wordpress, Content Signals, TDMRep, robots.txt, AI crawlers, bot access
description: "Say what AI may do with your content: add a Content-Signal line, publish a tdmrep.json file, and skip ai.txt for now. Includes tests and a B1 scan check."
url: https://aiscan.site/blog/declare-content-use-policy-to-ai
---

Blocking or allowing a crawler answers one question: may it fetch this page. It says nothing about what the crawler may do afterwards. This guide covers the three places a site can state that second answer, which are a `Content-Signal` line in robots.txt, a `/ai.txt` file, and the W3C TDM Reservation Protocol (`/.well-known/tdmrep.json`). The robots.txt line takes about five minutes. The TDMRep file takes about ten. None of the three is enforcement, and the guide says so before you spend time on them.

## Quick summary

| If you want to... | Use | Where it lives | Time |
|---|---|---|---|
| Say "index me, do not train on me" to AI crawlers | `Content-Signal: search=yes, ai-train=no` | `/robots.txt` | 5 min |
| Reserve text and data mining rights in the EU sense | `tdm-reservation: 1` | `/.well-known/tdmrep.json` or an HTTP header | 10 min |
| Publish an `ai.txt` file | Nothing reliable yet | `/ai.txt` | Skip |
| Refuse a crawler outright | A firewall or CDN rule | Your edge | See the [robots.txt starter](/blog/robots-txt-starter-for-ai-crawlers) |

**What the sources say:** [Cloudflare's Content Signals announcement](https://blog.cloudflare.com/content-signals-policy/) states that "content signals express preferences; they are not technical countermeasures against scraping." The [TDMRep specification](https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/) is a W3C Community Group report, not a W3C Standard, and ties its opt-out to Article 4 of EU Directive 2019/790.

Check where you stand first: `npx aiscan-cli yoursite.com` reports check:B1 (Content Signals) in about 20 seconds. B1 is informational, so it cannot lower your score.

## Why a block list is not a policy

robots.txt gates access. A policy gates use. Cloudflare's post describes the gap: a robots.txt file "does not, however, let them know what they are able to do with your content after accessing it." A search crawler that indexes you and an AI trainer that copies you look identical to a `Disallow` rule.

Think of a library. A door lock decides who enters. A sign on the shelf says "reading room only, no photocopying." The sign has no lock behind it, but it matters to anyone who reads it and it gives you something to point to later. The three formats below are signs of three different kinds.

## The three surfaces compared

| | Content Signals | TDMRep | ai.txt |
|---|---|---|---|
| Where | `Content-Signal:` line in robots.txt | `/.well-known/tdmrep.json`, HTTP header, or HTML meta | `/ai.txt` |
| Vocabulary | `search`, `ai-input`, `ai-train` = yes or no | `tdm-reservation` 1 or 0, optional `tdm-policy` URL | No spec found that is maintained |
| Spec status | Cloudflare policy, released CC0 | W3C Community Group final report, 10 May 2024 | Spawning's page was "Under Maintenance" on 10 Oct 2026 |
| Aimed at | Crawlers and AI products | Text and data mining under EU law | Unclear |

IETF is working on a shared vocabulary. The [AI Preferences working group](https://datatracker.ietf.org/wg/aipref/about/) lists `draft-ietf-aipref-vocab` (updated 13 Sep 2026) and `draft-ietf-aipref-attach` (updated 18 Aug 2026), both still Internet-Drafts. Its charter puts technical enforcement out of scope. Expect these formats to converge, and expect today's lines to need a revisit.

## Path 1: Content Signals in robots.txt

The syntax is one comma-separated line inside a `User-agent` group. This example, from Cloudflare's announcement, allows search, disallows training and expresses no view on `ai-input`:

```
User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /
```

1. Open your robots.txt. On a custom site it is a file at the web root. On WordPress, edit it in your SEO plugin.
2. Under the `User-agent: *` group, add one `Content-Signal` line. Use `yes` or `no` for each of `search`, `ai-input` and `ai-train`.
3. Leave a signal out if you have no preference. Per Cloudflare's text, a missing signal "neither grants nor restricts permission."
4. Optionally paste the human-readable policy comments Cloudflare publishes at [contentsignals.org](https://contentsignals.org/) above the line. Crawlers ignore comments, but people reading the file see the terms.
5. Deploy, then fetch `https://yoursite.com/robots.txt` and confirm the line is present once, inside a group.

For WordPress, [ThinkRank](https://thinkrank.ai) manages robots.txt, robots meta, llms.txt and schema in one plugin, so the policy line lives next to the rules it qualifies. It also imports from Rank Math, Yoast, AIOSEO and SEOPress. Paste the line into its robots.txt editor, save, and re-fetch the live file to confirm it survived. Other SEO plugins can do this too; the re-fetch is the test for any of them. On Shopify, the store's robots.txt is a template file, covered in [the robots.txt starter](/blog/robots-txt-starter-for-ai-crawlers).

**Do not copy another site's line without reading it.** On 10 Oct 2026 AIScan fetched robots.txt from 15 well-known sites. Four carried a `Content-Signal` line. cloudflare.com publishes `search=yes, ai-input=yes, ai-train=no`. stackoverflow.com publishes `search=no, ai-train=no`. aiscan.site publishes `search=yes, ai-train=yes, ai-input=yes` (permissive, on purpose). vercel.com publishes `ai-train=yes, search=yes, ai-input=yes`. Same syntax, opposite policies.

## Path 2: TDMRep for rights reservation

TDMRep exists because Article 4 of the EU copyright directive lets a rightsholder opt out of commercial text and data mining "in an appropriate manner, such as machine-readable means." You can express that three ways. Use one.

**Site-wide file.**

1. Create `/.well-known/tdmrep.json`. It must be a JSON array, even for one rule.
2. Add a rule with `location` and `tdm-reservation`:

```
[
  { "location": "/", "tdm-reservation": 1 }
]
```

3. Optional: add `"tdm-policy": "https://yoursite.com/tdm-policy.json"`. The spec says a machine-readable policy uses `application/json` or `application/ld+json`.
4. Serve it as JSON with a 200. Fetch it with `curl -i` and read the content type.

**HTTP header.** Send `tdm-reservation: 1` (and optionally `tdm-policy: <url>`) on responses. The spec calls the header the preferred technique and says header values override the site file.

**HTML meta.** Add `<meta name="tdm-reservation" content="1">` in the head of each page. This is the only option on hosts where you cannot set headers or serve a well-known file.

The spec asks publishers to use one technique. If you mix them, the later one overrides the earlier: header over file, meta over header.

**Who uses it.** AIScan fetched `/.well-known/tdmrep.json` from the same 15 sites on 10 Oct 2026. Three returned a real file: springer.com and nature.com (reservation 1 with a policy at dev.springernature.com) and elsevier.com (reservation 1, plus `tdm-reservation` and `tdm-policy` response headers). The other 12 returned 404 or 403. All three adopters are publishers who license content. A blog or SaaS site has no licence to offer, so `tdm-reservation: 1` with no policy is the honest minimal form.

## Path 3: ai.txt

An `ai.txt` file is often listed next to the two above. No maintained spec turned up. Spawning's `ai.txt` page returned "Source.Plus Spawning Under Maintenance" on 10 Oct 2026. Of the 15 sites fetched, vercel.com and springer.com returned 200 at `/ai.txt`, but both bodies were HTML pages (vercel.com's marketing shell, springer.com a "Client Challenge" page), not policy files. A status check alone would score both as present.

If you want a human-readable statement of terms, put it at a URL you control and link it from `tdm-policy`. That has a spec behind it.

## Verify it landed

Scan first: `npx aiscan-cli yoursite.com` (or paste the URL at [aiscan.site](https://aiscan.site/)). It reads check:B1 for the `Content-Signal` line and check:B2 for explicit AI bot groups. AIScan does not look for `tdmrep.json`, `ai.txt` or the `tdm-*` headers, so test those by hand:

| Check | Command | Pass signal |
|---|---|---|
| Signal line present | `curl -s https://yoursite.com/robots.txt \| grep -i content-signal` | One line, values you chose |
| TDMRep file | `curl -si https://yoursite.com/.well-known/tdmrep.json` | 200, JSON content type, body starts with `[` |
| Control path | `curl -si https://yoursite.com/.well-known/zz-not-real.json` | 404. If it returns 200, your server soft-200s and the first check proves nothing |
| Header form | `curl -sI https://yoursite.com/ \| grep -i '^tdm-'` | `tdm-reservation: 1` |

The control path is the same trap described in [correct 404s for AI agents](/blog/correct-404-for-ai-agents).

## When it does not work

- **"The scan still says B1 is not found."** The scanner reads `robots.txt` at the root. A redirect to an HTML page or a line outside any `User-agent` group will not register. Re-fetch the live file.
- **"My `tdmrep.json` returns 200 but a crawler ignores it."** Open it in a JSON validator. A bare object instead of an array is the usual cause. The spec says an agent that cannot parse it must treat the reservation as unset.
- **"A crawler I said `ai-train=no` to still trained."** Signals are requests. Per Cloudflare, combine them with WAF rules and bot management if you need refusal. [Does blocking AI training protect your content?](/blog/does-blocking-ai-training-protect-content) measures how often blocks are actually enforced.
- **"Cloudflare's managed robots.txt overwrote my edit."** Managed robots.txt serves its own file with `search=yes, ai-train=no`. Turn the feature off or edit the managed content instead.
- **"I am outside the EU. Does TDMRep matter?"** The spec says the opt-out mechanism comes from the EU directive and that outside the EU the exception does not apply. A reservation still states your position in a form a crawler can read. It does not create rights.

## Maintenance

- Re-read the IETF AIPREF drafts each quarter. If they become RFCs, switch to the standard vocabulary.
- After any SEO plugin, CDN or theme change, re-run the scan and the four checks above.
- Keep the policy consistent with your blocks. Allowing `ClaudeBot` in robots.txt while saying `ai-train=no` is a contradiction a lawyer will notice.

## What to do next

Run `npx aiscan-cli yoursite.com`, read B1, add one `Content-Signal` line, then scan again. For the wider set of crawler decisions, read the [AI crawler user-agent list](/blog/ai-crawler-user-agent-list-2026) and browse the [AI readiness guides](/guides). The [bot access checks](/docs/checks/bot-access) page documents B1, B2 and B3.