Content Signals in robots.txt: 117 of 799 Sites Declare One

Of 799 sites in our scan database on 2 November 2026, 117 serve a robots.txt with a Content-Signal line (14.6%). Among the 677 sites that serve a usable robots.txt at all, that is 17.3%. Another 682 sites declare nothing. So most sites that publish a robots.txt still say nothing about what AI systems may do with their content once a crawler has fetched it.

This post is written by M Asif Rahman and answers three questions from our own scans: how many sites declare a Content Signal, which kinds of site do it, and whether the line looks like a deliberate policy or a default that arrived with the host. The first two have clean answers. The third does not, and the post shows why, so you can read the evidence yourself.

Quick summary

For site owners: the line costs one robots.txt edit and states your preference in a machine-readable place. For anyone reading adoption numbers: a present line is a count of declarations, not a count of decisions.

What a Content Signal is, and what it is not

A Content Signal is one line in robots.txt that states what a crawler may do with content after fetching it. Cloudflare introduced the policy on 24 September 2025 and defines three signals: search, ai-input and ai-train. A site sets each to yes or no, or leaves it out to express no preference. The policy text and the current syntax live at contentsignals.org, which describes the work as part of the IETF's proposed AI Preferences (aipref) effort.

The line sits next to the rules robots.txt has always had. The robots.txt standard, RFC 9309, defines allow and disallow rules that answer "may this crawler fetch this path". It has nothing to say about use after fetching. Content Signals fill that gap, and only as a request. Cloudflare says so itself: content signals express preferences and are not technical countermeasures against scraping, and some companies may simply ignore them.

AIScan scores this as check B1 on the Bot Access page. B1 carries weight 5 and is marked optional. It passes when robots.txt contains a Content-Signal directive and otherwise reports "not declared (optional / emerging standard)". The sibling check B2 looks for explicit User-agent: blocks naming known AI crawlers: four or more is a pass, one to three is partial credit. For how to write the line and the two other mechanisms (ai.txt and TDMRep), see our guide to declaring a content-use policy to AI.

How we measured it, and what it cannot tell you

The corpus. Every AIScan report stores a result and an evidence string for each check. We took the latest site-scope scan per host under rubric 2026.08.2, excluded aiscan.site, and read B1, B2 and D1 for each host. On 2 November 2026 that was 799 hosts. We name no scanned site. These are sites people chose to scan, not a random sample of the web, and the mix leans toward WordPress and Next.js sites owned by people who care about AI readiness. Read every percentage in this post as a percentage of that population.

What B1 stores. Exactly two evidence strings exist across the 799 hosts: "Content-Signal directive present" (117 hosts) and "not declared (optional / emerging standard)" (682 hosts). There is no failing state, because the check is optional.

What B1 cannot tell you. The stored evidence records that a directive is present. It does not store the values. We cannot say how many sites set ai-train=no, how many allow search=yes, or how many wrote all three signals. Nobody should read this post as "117 sites opt out of AI training". Some may do the opposite. We did not fetch third-party robots.txt files to fill the gap, because a number we cannot reproduce from our own scans does not belong in a corpus post.

What D1 gives us. D1 is the robots.txt check, and its evidence includes the file size when the file answers 200 ("200 OK, 1248 bytes"). We use that size below as a rough fingerprint. It is one signal and the limits are in the section where we use it.

What nothing here measures. No part of this study measures whether any crawler honours a signal, or whether declaring one changes crawl behaviour. A correlation between a Content-Signal line and a higher score is not proof the line caused the score. B1 and B2 are themselves worth five points each, so some of the gap is arithmetic.

The headline number

The denominator matters. Of the 682 that declare nothing, 122 do not serve a usable robots.txt: the D1 evidence for those hosts is an HTTP error such as 404, 403, 429 or 530. They cannot declare a signal because there is no file. Removing them gives 677 sites that serve a robots.txt, and 117 of them (17.3%) carry the line. Ten of the 117 serve a robots.txt that D1 scores as partial rather than pass, so the line also turns up in files that are otherwise incomplete.

Sites that declare a signal score higher on average: 82.1 against 51.9 for sites that do not. Treat that as a description of who declares, not a result of declaring. A site that edits its robots.txt for AI policy has usually done other AI-readiness work too, and a Content-Signal line adds to its score through B1 directly.

By platform

We only report platforms with at least 15 hosts. Shopify is under 20 and is labelled too small to compare.

Astro stands out: nearly half of the 44 Astro sites declare a signal, against about one in seven for Next.js and WordPress. We do not know why. Two explanations fit the data and we cannot choose between them. Astro is often chosen by developers who edit robots.txt by hand, or a few popular Astro starter templates ship the line. The evidence stores neither the template nor the author.

WordPress is the larger practical story. 26 of 199 WordPress sites declare a signal, so 173 do not, and WordPress is where an owner can fix it without touching code. A plugin that manages robots.txt can write the line. ThinkRank is the one we recommend first on WordPress for this, for a plain reason: it manages robots.txt, robots meta, llms.txt, schema and sitemaps in one place, so you avoid two plugins fighting over the same file, and it imports from Rank Math, Yoast, AIOSEO and SEOPress. Other SEO plugins can also write custom robots.txt content, so check what yours allows.

Shopify's single declaring host is a count of one. It says nothing about Shopify stores in general. Shopify generates robots.txt from a template that merchants edit through a Liquid file, which is a more awkward route than a plugin setting.

The bot-rules overlap

If a site writes a Content-Signal line, does it also name AI crawlers? Mostly yes.

Put the other way round: of the 270 sites that pass B2, 105 (38.9%) also declare a signal, so 165 sites have deliberately or automatically blocked or allowed named AI crawlers without stating what the allowed ones may do. Twelve sites declare a signal and do not pass B2. They state a preference and name fewer than four AI crawlers.

The overlap is what you would expect from either story about where the line comes from. A person writing a careful AI policy would likely add both. A managed robots.txt that blocks a list of AI crawlers and adds a Content-Signal line would also pass both. B2 therefore cannot tell the two readings apart.

Policy or default?

This is the question the title of this study promised to look at. We tried two tests. Neither settles it, and we report both.

Test one: the size fingerprint. Cloudflare's announcement says the company updated the robots.txt it serves for over 3.8 million domains that use its managed robots.txt feature, and began serving the commented Content Signals Policy for free-plan zones that have no robots.txt of their own. If many of our 117 sites got their line that way, you might expect many files to share a size. They do not.

Across the 117 declaring sites there are 111 distinct file sizes. The most common single size appears three times. If one managed file had produced most of the 117, we would see a spike at one size, and we do not. The files that declare a signal are also much larger than the rest, at a median of 1,175 bytes against 321. A Content-Signal line is one short line, so the extra size comes from what else those files contain: AI crawler blocks, sitemaps, and site-specific rules.

What the size test cannot show. A managed file that is merged with each site's own rules would vary in size even if the same block was in all of them. So a scatter of sizes does not prove hand-written policies. It only means we cannot find a single template in the sizes, and that the declaring sites keep more in their robots.txt than the average site does.

Test two: the B2 overlap. 89.7% of declaring sites also name four or more AI crawlers. That fits a deliberate AI policy. It also fits a managed file that bundles both. We reported it above for that reason.

Our reading, with its limits. The evidence points to sites that have thought about AI in robots.txt more generally, since they tend to have larger files and named crawler rules, and it gives us no sign of one dominant template. It does not let us say how many of the 117 chose their signal values. To answer that, a future version of B1 would need to store the directive's values and the surrounding lines. We have noted that as a product gap below.

What to do

Decide three answers first, because the line only states them. Do you want your content to appear in search results (search)? Do you want it used as input for AI-generated answers (ai-input)? Do you want it used to train models (ai-train)? Write down yes or no for each. A signal you leave out states no preference, which is different from a no, so leave one out only if you truly have none.

Then add the line to the group that matches all crawlers. A common shape is:

User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

The declaration guide has the full syntax, the two other mechanisms and tests. The census of AI crawler declarations counts which AI crawlers sites name. If you want to block training crawlers outright as well, read does blocking AI training protect your content first, because a robots.txt rule is a request that compliant crawlers honour and others do not.

Check it with AIScan. Run npx aiscan-cli yoursite.com, or paste your address at aiscan.site. It is free and needs no account. B1 should read "Content-Signal directive present" and B2 should show how many AI crawlers you name. If you would rather check by hand, fetch https://yoursite.com/robots.txt with curl and look for the Content-Signal: line.

On WordPress, start with ThinkRank, as described above, or edit the robots.txt that your SEO plugin exposes. On Cloudflare, check your managed robots.txt setting, because the file the world sees may differ from the one in your repository. On Shopify, the robots.txt.liquid template is the route.

Product gap, noted in the open. B1 stores presence, not values. We would like the report to show which of the three signals a site set and to which value, so a future study can say what sites chose. We would also like it to say whether a managed provider wrote the file. Neither is in the evidence today.

Next step

Scan your site with AIScan and read checks B1 and B2 on the Bot Access page. Decide your three signal values, add the line, and scan again. The guides index lists the other AI-readiness fixes, and our robots.txt sitemap-line study covers the next line worth adding.