--- title: "The complete AI readiness setup for WordPress in 2026" slug: ai-readiness-setup-wordpress published: 2026-08-26T08:43:56.178391+00:00 updated: 2026-08-27T08:48:00.894833+00:00 author: "Asif Rahman" author_url: https://masifrahman.com category: "AI Readiness" tags: platform:wordpress, check:D1, check:D2, check:D3, check:B1, check:B2, check:C2, check:C3, check:E3, check:E5, WordPress, robots.txt, llms.txt, AI crawlers, AI readiness description: "WordPress passes three AI readiness checks by default. Add AI bot rules, Content Signals and llms.txt in about 40 minutes, with exact plugin paths and code." url: https://aiscan.site/blog/ai-readiness-setup-wordpress --- *Updated August 2026.* A default WordPress install already passes more AI-readiness checks than most hand-built sites. It ships server-rendered HTML, a real XML sitemap and an RSS feed without you touching anything. The gap is narrower than the "your site is invisible to AI" pitch suggests. On most installs it comes down to four fixable items, and you can close all four in about 40 minutes without writing a line of PHP. Below: what WordPress gives you free, what it does not, and two complete paths to fix the difference. Every menu path is the current one as of August 2026. ## Quick summary | Check | What WordPress does by default | What you have to do | |---|---|---| | **D1 robots.txt present & valid** | Generates a virtual one via `do_robots()` | Usually nothing, unless a physical file is overriding it | | **D2 XML sitemap** | Ships `/wp-sitemap.xml` since WordPress 5.5 | Nothing, unless a plugin disabled it and serves its own | | **E5 Content feed** | Serves RSS and declares it in `` via `feed_links()` | Nothing. WordPress wins this one outright | | **C3 Structured HTML / E3 Heading hierarchy** | Renders on the server in PHP | Nothing on a classic stack. Headless WordPress is a different story | | **B2 Explicit AI bot rules** | Nothing | Add named rules for AI crawlers | | **B1 Content Signals** | Nothing | Add `search` / `ai-input` / `ai-train` to robots.txt | | **C2 llms.txt** | Nothing | Generate one, or decide deliberately not to | Three of the seven are already done. The work is in the bottom three rows. ## Why WordPress starts ahead, and where it slips WordPress assembles pages in PHP on the server and sends finished HTML to whatever asked for it. A retrieval crawler that never runs JavaScript sees the same article a human sees. That single architectural fact is why a plain WordPress blog usually scores better on [structured HTML](https://aiscan.site/docs/checks/content) than a modern React site does. The slip happens in two places. **The first is robots.txt precedence.** WordPress does not keep a robots.txt file on disk. It generates one on request through `do_robots()`, which other code can modify through the `robots_txt` filter (available since WordPress 3.0). But your web server checks the filesystem first. If Apache or Nginx finds a real file at `/robots.txt`, it serves that file and the request never reaches PHP, so every rule your SEO plugin thinks it is publishing quietly does nothing. This is the most common reason a WordPress site's bot rules are not live. **The second is that nobody ships AI-crawler defaults.** WordPress core has no opinion about GPTBot or ClaudeBot, and neither do Yoast SEO or Rank Math out of the box. Across Cloudflare's top 200,000 domains, 78% have a robots.txt but only 4% declare any AI usage preference. A file that only says `Disallow: /wp-admin/` tells an AI crawler nothing about whether it may use your content for answers or for training. That split matters, because the fix for the first problem is deletion and the fix for the second is addition. People who skip the mechanism try to add rules on top of a shadowed file, then wonder why nothing changed. ## Step 0: find out which robots.txt you are actually serving Do this before anything else. Run `curl -s https://yoursite.com/robots.txt` and compare the output against what your SEO plugin shows you in the admin. **If the two differ, a physical file is winning.** Connect over SFTP or your host's file manager, look in the web root next to `wp-config.php`, and if a `robots.txt` file is sitting there, either delete it so WordPress takes over, or commit to editing that file directly from now on. Pick one owner. Two owners is how sites end up publishing rules nobody can find. ## Path A: the plugin route (about 20 minutes, no code) Best if you already run an SEO plugin and would rather not touch PHP. **1. Take control of robots.txt.** In Rank Math, turn on Advanced Mode from the top right of the Rank Math screen, then go to **Rank Math SEO → General Settings → Edit robots.txt**. In Yoast SEO, go to **Yoast SEO → Tools → File editor**. If Yoast's File editor is missing, your install has `DISALLOW_FILE_EDIT` set to true in `wp-config.php`, or the file is not writable; edit over SFTP instead. Rank Math will likewise refuse if a physical robots.txt exists, and its documentation is explicit that you have to remove that file first. **2. Add named AI bot rules.** Paste rules for the crawlers you have an actual opinion about. A permissive-but-explicit baseline that keeps you readable by answer engines while opting out of training: ``` User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / ``` Decide deliberately. GPTBot, ClaudeBot and Google-Extended are training crawlers; OAI-SearchBot and PerplexityBot fetch for search and answers, so blocking that second group removes you from the citations you probably want. Our breakdown of [Cloudflare's three crawler categories](https://aiscan.site/blog/cloudflare-search-agent-training-crawler-categories) explains which bot belongs where. **3. Add Content Signals.** Cloudflare's vocabulary states intent rather than access, and it rides inside a normal user-agent group as a `Content-Signal:` line. This is the exact form Cloudflare's own managed file writes: ``` User-agent: * Content-Signal: search=yes, ai-train=no, use=reference Allow: / ``` The signals are `search` (building a search index), `ai-input` (feeding content into a model at answer time, meaning RAG and grounding) and `ai-train` (training or fine-tuning). The **AI Content Signals** plugin (slug `ai-content-signals`, ~700 active installs, version 1.3.2, last updated 20 August 2026) writes these across both physical and virtual robots.txt and emits the matching `X-Robots-Tag` header. Be honest with yourself about weight: Google's John Mueller has said flatly that no crawler he knows of uses content-signal directives, and Cloudflare's own documentation says compliance "is voluntary." It is a stated reservation of rights, not enforcement. **4. Generate llms.txt.** If you followed step 1, you already know the problem with bolting on another plugin: every extra plugin that wants an opinion about robots.txt is another thing to reconcile. [**ThinkRank AI SEO**](https://thinkrank.ai) (slug `thinkrank`, ~8,000 active installs, version 2.0.2, requires WordPress 6.0 or later) is the one we build and the one we would install here, because it handles robots.txt, robots meta, schema and llms.txt generation from a single plugin — which means steps 1 through 4 stop being four settings screens in four places. It migrates existing directives from Rank Math, Yoast, All in One SEO and SEOPress, so you are not retyping anything. If you only want llms.txt and already have an SEO plugin you are happy with, **Website LLMs.txt** (slug `website-llms-txt`, ~40,000 active installs, last updated 18 August 2026) does that one job well and integrates with Yoast SEO, Rank Math, SEOPress and AIOSEO; its settings live under **Settings → LLMs.txt**. And if you would rather not install anything at all, our [llms.txt generator](https://aiscan.site/llms-txt-generator) produces a file you can upload by hand — that is Path B below. **5. Confirm your sitemap is the one you think it is.** WordPress core serves `/wp-sitemap.xml` (introduced in 5.5, via the `WP_Sitemaps` class, switchable with the `wp_sitemaps_enabled` filter). Yoast SEO and Rank Math both serve theirs at `/sitemap_index.xml`; Rank Math's settings are under **Rank Math SEO → Sitemap Settings**, Yoast's toggle under **Yoast SEO → Settings → Site features → Technical SEO**. Request both URLs, see which returns 200, and put that one in robots.txt. ## Path B: the manual route (about 20 minutes, one file) Best if you run no SEO plugin, or you want these rules in version control instead of the database. **1. Create an mu-plugin.** Make `wp-content/mu-plugins/ai-readiness.php`. Must-use plugins load automatically and cannot be deactivated by accident, which is the right home for infrastructure rules. Do not put this in your theme's `functions.php`, because a theme switch would silently drop it. **2. Filter the virtual robots.txt:** ```php `, and `send_headers` (since 2.1.0) fires before headers go out: ```php add_action( 'wp_head', function () { printf( '' . "\n", esc_url( home_url( '/llms.txt' ) ) ); } ); add_action( 'send_headers', function () { header( 'Link: <' . home_url( '/llms.txt' ) . '>; rel="describedby"', false ); } ); ``` That second one also satisfies **D3 Link header for discovery**, which almost nobody bothers with. **4. Upload `llms.txt` to your web root**, beside `wp-config.php`. It is a static file, no PHP involved. **5. Leave the feed alone.** `feed_links()` has been hooked to `wp_head` by default since WordPress 2.8 and already emits your `` tags. Check your rendered `` for one rather than adding a second by hand. ## How to verify it worked **Start with a scan.** You have just changed seven things across robots.txt, llms.txt, sitemaps, feeds and rendering, and checking them one at a time is how one gets missed. [AIScan](https://aiscan.site) is free, needs no account, and grades all seven as individually named checks — D1 robots.txt, D2 sitemap, B1 Content Signals, B2 explicit AI bot rules, C2 llms.txt, C3 structured HTML, E3 server-rendered text and E5 content feed — so a partial score tells you exactly which row is still open rather than that something, somewhere, is wrong: ```bash npx aiscan-cli yoursite.com ``` Or paste the URL at [aiscan.site](https://aiscan.site) and read the report in the browser. Our [WordPress platform page](https://aiscan.site/docs/platforms/wordpress) lists the same fixes in the same order as this guide, and the [bot access checks](https://aiscan.site/docs/checks/bot-access) page explains what B1 and B2 look for. If you monitor the site, a re-scan after your next plugin update will catch a regression before it costs you anything. **If you would rather check by hand**, all four are one-liners. Each has an unambiguous pass or fail: ```bash SITE=https://yoursite.com curl -s $SITE/robots.txt | grep -Ei 'gptbot|claudebot|content-signal' curl -sL -o /dev/null -w '%{http_code}\n' $SITE/llms.txt curl -sL -o /dev/null -w '%{http_code}\n' $SITE/wp-sitemap.xml curl -s $SITE | grep -o 'rel="alternate"[^>]*rss[^>]*' ``` **Pass:** robots.txt returns your rules over HTTP, not just in the plugin's preview pane; llms.txt returns 200 with `content-type: text/plain`; one sitemap path returns 200; the feed link appears in ``. **Fail, and what it means.** robots.txt returns rules you did not write: a physical file is shadowing the virtual one, so go back to Step 0. llms.txt returns 404: the file went to the wrong directory, or a caching layer has not purged. Both sitemap paths return 404: a plugin disabled core sitemaps without enabling its own. **What no scanner can see, ours included:** whether your llms.txt content is accurate, whether anything ever fetches it, or whether your stated Content Signals are honoured by anyone. Ahrefs found 28% of roughly 137,000 domains publish a valid llms.txt and 97% of those files received zero requests in May 2026. Google has said in writing that Search ignores llms.txt entirely, neither helping nor harming rankings. Publish one because directed tools and agentic browsers can use it, not because a green tick appeared. ## Who can skip most of this A five-page brochure site with no blog does not need llms.txt, because the sitemap already describes everything. A site whose host manages robots.txt at the CDN edge should make these choices in that dashboard instead, or the two will disagree. And if you run **headless WordPress**, with WordPress as a REST backend behind a React or Next.js front end, the C3 and E3 rows do not apply to you. Your rendering happens on the client, and that is a different fix. WordPress's own Interactivity API (WordPress 6.5) stays server-rendered by default, so block themes are fine. ## Maintenance Re-run the four verification commands after any host migration, CDN change or security-plugin install, because all three can start serving a physical robots.txt you did not create. Cloudflare's managed robots.txt setting does not replace yours, it prepends to it, so a site behind Cloudflare can end up with two `User-agent: *` groups whose rules contradict each other. Read the whole file, not the first ten lines. Cloudflare's crawler defaults change on **15 September 2026**, and on existing zones a Training block will begin returning 403 to Googlebot, Applebot and Bingbot. If your site sits behind Cloudflare, read [what changes on 15 September](https://aiscan.site/blog/cloudflare-ai-bot-block-googlebot-2026) before then. For the evidence on whether llms.txt is worth maintaining at all, we went through it in [Does llms.txt actually work in 2026](https://aiscan.site/blog/does-llms-txt-actually-work-2026). More platform walkthroughs are in [our guides index](https://aiscan.site/guides).