Platform Guides
Platform-specific fixes for WordPress, Shopify, Lovable, Replit, and Next.js.
The same check needs a different fix on every platform. WordPress writes robots through a plugin, Shopify compiles it from a Liquid template, and Next.js owns it as a static file. Pick your stack:
- WordPress — robots.txt, llms.txt, sitemap, Yoast/Rank Math, AI bot rules.
- Shopify — robots.txt.liquid, product feeds, structured data, theme limits.
- Lovable — public/ files, server routes, MCP endpoints, llms.txt.
- Replit — static assets, robots, llms.txt, deployment caveats.
- Next.js / Vercel — App Router metadata, route handlers, public/ files, edge headers.
Don't see your platform? Most static-site generators (Hugo, Eleventy, Astro, Jekyll) work like Next.js for AIScan purposes — drop files into public/ or static/ and they're served at the root. Longer walkthroughs for each stack live in the guide library.
Further reading
- Content Signals in robots.txt: 117 of 799 Sites Declare One
Of 799 sites in our scan database on 2 November 2026, 117 serve a robots.txt with a ContentSignal line 14.6%. Among the 677 sites that serve a usable robots.txt at all, that is 17.3%. Another 682…
- Your RSS Feed Returns 403 to Feed Readers and Agents: Cloudflare and Firewall Rules That Block It
Cloudflare's Bot Fight Mode and its "block AI bots" switch can answer your RSS feed with a 403 challenge page, and a feed reader or AI agent has no way to solve one. On 1 November 2026 we requested…
- Find AI Crawlers in Your Server Logs and Verify Them
Your analytics tool will not tell you which AI crawlers visit your site, because a crawler that fetches the HTML and leaves does not run its script. Only the server or the edge in front of it sees…
- ChatGPT Says It Can't Access Your Website: Five Causes and How to Tell Which
When ChatGPT says it cannot open your page, one of five things happened, and a short script of four curl requests sorts four of them in about a minute. The fifth is that ChatGPT never tried. This…
- Block Fake AI Crawlers From Probing .env and .git Files
Some of the traffic that calls itself GPTBot or ClaudeBot is not a crawler. It is a scanner that borrowed a famous name and asks your server for /.env, a .git folder or a wpconfig.php.bak. The fix is…
- Your robots.txt blocks everything by accident: find the leftover Disallow: / and stop it coming back
If your robots.txt says Useragent: followed by Disallow: /, every crawler that obeys the file is told to stay off your whole site. It takes one line to cause and about a minute to find, and Google's…
- Does robots.txt Need a Sitemap Line? 782 Sites Show What Happens
Of 782 sites in our scan database on 29 October 2026, 572 serve a robots.txt that declares a sitemap, 89 serve a robots.txt with no Sitemap line, and 121 fail the robots.txt check outright. Declaring…
- JSON-LD in Google Tag Manager is missing from your served HTML: test it and move it server-side
If your JSONLD is added by a Google Tag Manager Custom HTML tag, a plain HTTP fetch of your page contains zero schema blocks. Google can still read it, because Googlebot renders JavaScript. Crawlers…