Declare your content-use policy to AI: Content Signals, ai.txt, TDMRep

Blocking or allowing a crawler answers one question: may it fetch this page. It says nothing about what the crawler may do afterwards. This guide covers the three places a site can state that second answer, which are a Content-Signal line in robots.txt, a /ai.txt file, and the W3C TDM Reservation Protocol (/.well-known/tdmrep.json). The robots.txt line takes about five minutes. The TDMRep file takes about ten. None of the three is enforcement, and the guide says so before you spend time on them.

Quick summary

What the sources say: Cloudflare's Content Signals announcement states that "content signals express preferences; they are not technical countermeasures against scraping." The TDMRep specification is a W3C Community Group report, not a W3C Standard, and ties its opt-out to Article 4 of EU Directive 2019/790.

Check where you stand first: npx aiscan-cli yoursite.com reports check:B1 (Content Signals) in about 20 seconds. B1 is informational, so it cannot lower your score.

Why a block list is not a policy

robots.txt gates access. A policy gates use. Cloudflare's post describes the gap: a robots.txt file "does not, however, let them know what they are able to do with your content after accessing it." A search crawler that indexes you and an AI trainer that copies you look identical to a Disallow rule.

Think of a library. A door lock decides who enters. A sign on the shelf says "reading room only, no photocopying." The sign has no lock behind it, but it matters to anyone who reads it and it gives you something to point to later. The three formats below are signs of three different kinds.

The three surfaces compared

IETF is working on a shared vocabulary. The AI Preferences working group lists draft-ietf-aipref-vocab (updated 13 Sep 2026) and draft-ietf-aipref-attach (updated 18 Aug 2026), both still Internet-Drafts. Its charter puts technical enforcement out of scope. Expect these formats to converge, and expect today's lines to need a revisit.

Path 1: Content Signals in robots.txt

The syntax is one comma-separated line inside a User-agent group. This example, from Cloudflare's announcement, allows search, disallows training and expresses no view on ai-input:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

1. Open your robots.txt. On a custom site it is a file at the web root. On WordPress, edit it in your SEO plugin.
2. Under the User-agent: * group, add one Content-Signal line. Use yes or no for each of search, ai-input and ai-train.
3. Leave a signal out if you have no preference. Per Cloudflare's text, a missing signal "neither grants nor restricts permission."
4. Optionally paste the human-readable policy comments Cloudflare publishes at contentsignals.org above the line. Crawlers ignore comments, but people reading the file see the terms.
5. Deploy, then fetch https://yoursite.com/robots.txt and confirm the line is present once, inside a group.

For WordPress, ThinkRank manages robots.txt, robots meta, llms.txt and schema in one plugin, so the policy line lives next to the rules it qualifies. It also imports from Rank Math, Yoast, AIOSEO and SEOPress. Paste the line into its robots.txt editor, save, and re-fetch the live file to confirm it survived. Other SEO plugins can do this too; the re-fetch is the test for any of them. On Shopify, the store's robots.txt is a template file, covered in the robots.txt starter.

Do not copy another site's line without reading it. On 10 Oct 2026 AIScan fetched robots.txt from 15 well-known sites. Four carried a Content-Signal line. cloudflare.com publishes search=yes, ai-input=yes, ai-train=no. stackoverflow.com publishes search=no, ai-train=no. aiscan.site publishes search=yes, ai-train=yes, ai-input=yes (permissive, on purpose). vercel.com publishes ai-train=yes, search=yes, ai-input=yes. Same syntax, opposite policies.

Path 2: TDMRep for rights reservation

TDMRep exists because Article 4 of the EU copyright directive lets a rightsholder opt out of commercial text and data mining "in an appropriate manner, such as machine-readable means." You can express that three ways. Use one.

Site-wide file.

1. Create /.well-known/tdmrep.json. It must be a JSON array, even for one rule.
2. Add a rule with location and tdm-reservation:

[
  { "location": "/", "tdm-reservation": 1 }
]

1. Optional: add "tdm-policy": "https://yoursite.com/tdm-policy.json". The spec says a machine-readable policy uses application/json or application/ld+json.
2. Serve it as JSON with a 200. Fetch it with curl -i and read the content type.

HTTP header. Send tdm-reservation: 1 (and optionally tdm-policy: <url>) on responses. The spec calls the header the preferred technique and says header values override the site file.

HTML meta. Add <meta name="tdm-reservation" content="1"> in the head of each page. This is the only option on hosts where you cannot set headers or serve a well-known file.

The spec asks publishers to use one technique. If you mix them, the later one overrides the earlier: header over file, meta over header.

Who uses it. AIScan fetched /.well-known/tdmrep.json from the same 15 sites on 10 Oct 2026. Three returned a real file: springer.com and nature.com (reservation 1 with a policy at dev.springernature.com) and elsevier.com (reservation 1, plus tdm-reservation and tdm-policy response headers). The other 12 returned 404 or 403. All three adopters are publishers who license content. A blog or SaaS site has no licence to offer, so tdm-reservation: 1 with no policy is the honest minimal form.

Path 3: ai.txt

An ai.txt file is often listed next to the two above. No maintained spec turned up. Spawning's ai.txt page returned "Source.Plus Spawning Under Maintenance" on 10 Oct 2026. Of the 15 sites fetched, vercel.com and springer.com returned 200 at /ai.txt, but both bodies were HTML pages (vercel.com's marketing shell, springer.com a "Client Challenge" page), not policy files. A status check alone would score both as present.

If you want a human-readable statement of terms, put it at a URL you control and link it from tdm-policy. That has a spec behind it.

Verify it landed

Scan first: npx aiscan-cli yoursite.com (or paste the URL at aiscan.site). It reads check:B1 for the Content-Signal line and check:B2 for explicit AI bot groups. AIScan does not look for tdmrep.json, ai.txt or the tdm-* headers, so test those by hand:

The control path is the same trap described in correct 404s for AI agents.

When it does not work

• "The scan still says B1 is not found." The scanner reads robots.txt at the root. A redirect to an HTML page or a line outside any User-agent group will not register. Re-fetch the live file.
• "My tdmrep.json returns 200 but a crawler ignores it." Open it in a JSON validator. A bare object instead of an array is the usual cause. The spec says an agent that cannot parse it must treat the reservation as unset.
• "A crawler I said ai-train=no to still trained." Signals are requests. Per Cloudflare, combine them with WAF rules and bot management if you need refusal. Does blocking AI training protect your content? measures how often blocks are actually enforced.
• "Cloudflare's managed robots.txt overwrote my edit." Managed robots.txt serves its own file with search=yes, ai-train=no. Turn the feature off or edit the managed content instead.
• "I am outside the EU. Does TDMRep matter?" The spec says the opt-out mechanism comes from the EU directive and that outside the EU the exception does not apply. A reservation still states your position in a form a crawler can read. It does not create rights.

Maintenance

• Re-read the IETF AIPREF drafts each quarter. If they become RFCs, switch to the standard vocabulary.
• After any SEO plugin, CDN or theme change, re-run the scan and the four checks above.
• Keep the policy consistent with your blocks. Allowing ClaudeBot in robots.txt while saying ai-train=no is a contradiction a lawyer will notice.

What to do next

Run npx aiscan-cli yoursite.com, read B1, add one Content-Signal line, then scan again. For the wider set of crawler decisions, read the AI crawler user-agent list and browse the AI readiness guides. The bot access checks page documents B1, B2 and B3.