The complete AI readiness setup for WordPress in 2026

Updated August 2026.

A default WordPress install already passes more AI-readiness checks than most hand-built sites. It ships server-rendered HTML, a real XML sitemap and an RSS feed without you touching anything. The gap is narrower than the "your site is invisible to AI" pitch suggests. On most installs it comes down to four fixable items, and you can close all four in about 40 minutes without writing a line of PHP.

Below: what WordPress gives you free, what it does not, and two complete paths to fix the difference. Every menu path is the current one as of August 2026.

Quick summary

Three of the seven are already done. The work is in the bottom three rows.

Why WordPress starts ahead, and where it slips

WordPress assembles pages in PHP on the server and sends finished HTML to whatever asked for it. A retrieval crawler that never runs JavaScript sees the same article a human sees. That single architectural fact is why a plain WordPress blog usually scores better on structured HTML than a modern React site does.

The slip happens in two places.

The first is robots.txt precedence. WordPress does not keep a robots.txt file on disk. It generates one on request through do_robots(), which other code can modify through the robots_txt filter (available since WordPress 3.0). But your web server checks the filesystem first. If Apache or Nginx finds a real file at /robots.txt, it serves that file and the request never reaches PHP, so every rule your SEO plugin thinks it is publishing quietly does nothing. This is the most common reason a WordPress site's bot rules are not live.

The second is that nobody ships AI-crawler defaults. WordPress core has no opinion about GPTBot or ClaudeBot, and neither do Yoast SEO or Rank Math out of the box. Across Cloudflare's top 200,000 domains, 78% have a robots.txt but only 4% declare any AI usage preference. A file that only says Disallow: /wp-admin/ tells an AI crawler nothing about whether it may use your content for answers or for training.

That split matters, because the fix for the first problem is deletion and the fix for the second is addition. People who skip the mechanism try to add rules on top of a shadowed file, then wonder why nothing changed.

Step 0: find out which robots.txt you are actually serving

Do this before anything else. Run curl -s https://yoursite.com/robots.txt and compare the output against what your SEO plugin shows you in the admin. If the two differ, a physical file is winning. Connect over SFTP or your host's file manager, look in the web root next to wp-config.php, and if a robots.txt file is sitting there, either delete it so WordPress takes over, or commit to editing that file directly from now on. Pick one owner. Two owners is how sites end up publishing rules nobody can find.

Path A: the plugin route (about 20 minutes, no code)

Best if you already run an SEO plugin and would rather not touch PHP.

1. Take control of robots.txt. In Rank Math, turn on Advanced Mode from the top right of the Rank Math screen, then go to Rank Math SEO → General Settings → Edit robots.txt. In Yoast SEO, go to Yoast SEO → Tools → File editor. If Yoast's File editor is missing, your install has DISALLOW_FILE_EDIT set to true in wp-config.php, or the file is not writable; edit over SFTP instead. Rank Math will likewise refuse if a physical robots.txt exists, and its documentation is explicit that you have to remove that file first.

2. Add named AI bot rules. Paste rules for the crawlers you have an actual opinion about. A permissive-but-explicit baseline that keeps you readable by answer engines while opting out of training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Decide deliberately. GPTBot, ClaudeBot and Google-Extended are training crawlers; OAI-SearchBot and PerplexityBot fetch for search and answers, so blocking that second group removes you from the citations you probably want. Our breakdown of Cloudflare's three crawler categories explains which bot belongs where.

3. Add Content Signals. Cloudflare's vocabulary states intent rather than access, and it rides inside a normal user-agent group as a Content-Signal: line. This is the exact form Cloudflare's own managed file writes:

User-agent: *
Content-Signal: search=yes, ai-train=no, use=reference
Allow: /

The signals are search (building a search index), ai-input (feeding content into a model at answer time, meaning RAG and grounding) and ai-train (training or fine-tuning). The AI Content Signals plugin (slug ai-content-signals, ~700 active installs, version 1.3.2, last updated 20 August 2026) writes these across both physical and virtual robots.txt and emits the matching X-Robots-Tag header. Be honest with yourself about weight: Google's John Mueller has said flatly that no crawler he knows of uses content-signal directives, and Cloudflare's own documentation says compliance "is voluntary." It is a stated reservation of rights, not enforcement.

4. Generate llms.txt. If you followed step 1, you already know the problem with bolting on another plugin: every extra plugin that wants an opinion about robots.txt is another thing to reconcile. ThinkRank AI SEO (slug thinkrank, ~8,000 active installs, version 2.0.2, requires WordPress 6.0 or later) is the one we build and the one we would install here, because it handles robots.txt, robots meta, schema and llms.txt generation from a single plugin — which means steps 1 through 4 stop being four settings screens in four places. It migrates existing directives from Rank Math, Yoast, All in One SEO and SEOPress, so you are not retyping anything.

If you only want llms.txt and already have an SEO plugin you are happy with, Website LLMs.txt (slug website-llms-txt, ~40,000 active installs, last updated 18 August 2026) does that one job well and integrates with Yoast SEO, Rank Math, SEOPress and AIOSEO; its settings live under Settings → LLMs.txt. And if you would rather not install anything at all, our llms.txt generator produces a file you can upload by hand — that is Path B below.

5. Confirm your sitemap is the one you think it is. WordPress core serves /wp-sitemap.xml (introduced in 5.5, via the WP_Sitemaps class, switchable with the wp_sitemaps_enabled filter). Yoast SEO and Rank Math both serve theirs at /sitemap_index.xml; Rank Math's settings are under Rank Math SEO → Sitemap Settings, Yoast's toggle under Yoast SEO → Settings → Site features → Technical SEO. Request both URLs, see which returns 200, and put that one in robots.txt.

Path B: the manual route (about 20 minutes, one file)

Best if you run no SEO plugin, or you want these rules in version control instead of the database.

1. Create an mu-plugin. Make wp-content/mu-plugins/ai-readiness.php. Must-use plugins load automatically and cannot be deactivated by accident, which is the right home for infrastructure rules. Do not put this in your theme's functions.php, because a theme switch would silently drop it.

2. Filter the virtual robots.txt:

<?php
/**
 * Plugin Name: AI Readiness Rules
 */
add_filter( 'robots_txt', function ( $output ) {
    $rules  = "\n# AI crawler rules\n";
    $rules .= "User-agent: GPTBot\nDisallow: /\n\n";
    $rules .= "User-agent: ClaudeBot\nDisallow: /\n\n";
    $rules .= "User-agent: OAI-SearchBot\nAllow: /\n\n";
    $rules .= "Sitemap: " . home_url( '/wp-sitemap.xml' ) . "\n";
    return $output . $rules;
}, 10, 1 );

3. Declare llms.txt from the page itself. The describedby link relation points a crawler from an HTML page to the file describing it. wp_head (since WordPress 1.5.0) prints into <head>, and send_headers (since 2.1.0) fires before headers go out:

add_action( 'wp_head', function () {
    printf(
        '<link rel="describedby" href="%s">' . "\n",
        esc_url( home_url( '/llms.txt' ) )
    );
} );

add_action( 'send_headers', function () {
    header( 'Link: <' . home_url( '/llms.txt' ) . '>; rel="describedby"', false );
} );

That second one also satisfies D3 Link header for discovery, which almost nobody bothers with.

4. Upload llms.txt to your web root, beside wp-config.php. It is a static file, no PHP involved.

5. Leave the feed alone. feed_links() has been hooked to wp_head by default since WordPress 2.8 and already emits your <link rel="alternate"> tags. Check your rendered <head> for one rather than adding a second by hand.

How to verify it worked

Start with a scan. You have just changed seven things across robots.txt, llms.txt, sitemaps, feeds and rendering, and checking them one at a time is how one gets missed. AIScan is free, needs no account, and grades all seven as individually named checks — D1 robots.txt, D2 sitemap, B1 Content Signals, B2 explicit AI bot rules, C2 llms.txt, C3 structured HTML, E3 server-rendered text and E5 content feed — so a partial score tells you exactly which row is still open rather than that something, somewhere, is wrong:

npx aiscan-cli yoursite.com

Or paste the URL at aiscan.site and read the report in the browser. Our WordPress platform page lists the same fixes in the same order as this guide, and the bot access checks page explains what B1 and B2 look for. If you monitor the site, a re-scan after your next plugin update will catch a regression before it costs you anything.

If you would rather check by hand, all four are one-liners. Each has an unambiguous pass or fail:

SITE=https://yoursite.com
curl -s $SITE/robots.txt | grep -Ei 'gptbot|claudebot|content-signal'
curl -sL -o /dev/null -w '%{http_code}\n' $SITE/llms.txt
curl -sL -o /dev/null -w '%{http_code}\n' $SITE/wp-sitemap.xml
curl -s $SITE | grep -o 'rel="alternate"[^>]*rss[^>]*'

Pass: robots.txt returns your rules over HTTP, not just in the plugin's preview pane; llms.txt returns 200 with content-type: text/plain; one sitemap path returns 200; the feed link appears in <head>.

Fail, and what it means. robots.txt returns rules you did not write: a physical file is shadowing the virtual one, so go back to Step 0. llms.txt returns 404: the file went to the wrong directory, or a caching layer has not purged. Both sitemap paths return 404: a plugin disabled core sitemaps without enabling its own.

What no scanner can see, ours included: whether your llms.txt content is accurate, whether anything ever fetches it, or whether your stated Content Signals are honoured by anyone. Ahrefs found 28% of roughly 137,000 domains publish a valid llms.txt and 97% of those files received zero requests in May 2026. Google has said in writing that Search ignores llms.txt entirely, neither helping nor harming rankings. Publish one because directed tools and agentic browsers can use it, not because a green tick appeared.

Who can skip most of this

A five-page brochure site with no blog does not need llms.txt, because the sitemap already describes everything. A site whose host manages robots.txt at the CDN edge should make these choices in that dashboard instead, or the two will disagree.

And if you run headless WordPress, with WordPress as a REST backend behind a React or Next.js front end, the C3 and E3 rows do not apply to you. Your rendering happens on the client, and that is a different fix. WordPress's own Interactivity API (WordPress 6.5) stays server-rendered by default, so block themes are fine.

Maintenance

Re-run the four verification commands after any host migration, CDN change or security-plugin install, because all three can start serving a physical robots.txt you did not create. Cloudflare's managed robots.txt setting does not replace yours, it prepends to it, so a site behind Cloudflare can end up with two User-agent: * groups whose rules contradict each other. Read the whole file, not the first ten lines.

Cloudflare's crawler defaults change on 15 September 2026, and on existing zones a Training block will begin returning 403 to Googlebot, Applebot and Bingbot. If your site sits behind Cloudflare, read what changes on 15 September before then. For the evidence on whether llms.txt is worth maintaining at all, we went through it in Does llms.txt actually work in 2026.

More platform walkthroughs are in our guides index.