Fetch Your Own Site the Way an AI Agent Sees It

Your site has at least three audiences: people in a browser, search crawlers, and AI agents. Servers can treat each differently, and the only way to know is to ask as each of them and compare the answers. Four requests and one comparison take about five minutes and show you the status code, content type and body size each identity receives.

This guide is the do-it-yourself version. It covers what the commands send, what the results mean, and what to do when the three answers disagree.

Quick summary

The short answer: run AIScan first for the declared rules, then fetch your homepage under three user-agent strings and compare status, content-type and byte size. A difference between the browser and a bot is the finding. A spoofed request cannot prove what the real bot receives, because many sites also check IP address, so treat a clean result as "no user-agent rule found", not as proof.

Why the same URL can return three different pages

A web server does not hand out one page per URL. It makes a decision for every request, using the user-agent header, the source IP, the Accept header, cookies and rate counters. A CDN or bot-management layer in front of your origin can answer before your application ever sees the request.

Think of a venue with a door list. The host checks your name against several lists, and a person on none of them may be turned away or walked to a different room. Your site has the same lists spread across at least four places: robots.txt (a request, not a lock, as RFC 9309 says), your CDN's bot rules, your application and your hosting firewall. Reading the rules tells you what you intended. Fetching the page tells you what happens.

Our own measurements show why the second step matters. On 5 October 2026 we fetched ten homepages under a browser string, a ClaudeBot string and a GPTBot string, from one datacenter IP, one request per identity:

Three lessons sit in that table. nextjs.org answers the two bot strings with a 3.9 KB Markdown version of the page and everyone else with 352 KB of HTML, so an agent and a person get different documents at one URL. openai.com blocked the browser string and the GPTBot string but let the ClaudeBot string through from the same IP, so the rule is not simply "datacenter IP blocked". And nytimes.com refused all three, so a user-agent rule alone does not explain it.

One more result is a warning. On allbirds.com the first run returned 200 to all three strings, with the ClaudeBot response 671,269 bytes against 714,750 for the others. On a repeat run the browser string got 429. A rate limit looks exactly like a block, so never trust a single request.

Path 1: ask the scan first

AIScan is free and needs no account. Run it before anything else, because it covers the declared half of the picture in one step.

1. Run npx aiscan-cli yoursite.com, or paste the URL at aiscan.site. The CLI docs list the options.
2. Read check B2. It looks for explicit User-agent: blocks naming known AI crawlers in robots.txt. Four or more named bots pass, one to three is partial credit.
3. Read check C3, which tests whether the HTML your server returns carries real content, not an empty shell that needs JavaScript to fill in.
4. Read the other bot access checks for Content Signals (B1) and Web Bot Auth (B3).

What AIScan cannot see: it does not impersonate ClaudeBot or GPTBot, so it cannot tell you that your CDN challenges one of them. It grades the homepage from the public scan, and it cannot see a rule that depends on the requester's IP. That gap is why the manual path exists.

Path 2: fetch it by hand with curl

If you would rather check by hand, or you need the bot-versus-browser comparison the scan does not make, this path is complete without any tool of ours. You need a terminal with curl.

1. Set three identities. The strings below follow the formats published by Anthropic and OpenAI. Vendors change version numbers, so copy the current string from their page if you see a mismatch.

UA_BROWSER='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'
UA_CLAUDE='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)'
UA_GPT='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'

1. Fetch the same URL under each identity and print the facts that matter. Replace the URL with your own homepage, then a second page such as a product or article.

URL=https://yoursite.com/
for NAME in BROWSER CLAUDE GPT; do
  eval UA=\$UA_$NAME
  curl -s -L -m 25 -A "$UA" -o "out_$NAME.html" \
    -w "$NAME %{http_code} %{content_type} %{size_download} bytes, %{num_redirects} redirects\n" "$URL"
  echo "  h1 tags: $(grep -ao '<h[1]' out_$NAME.html | wc -l)"
done

1. Add a control path that cannot exist, such as https://yoursite.com/zz-no-such-page-8841. A healthy server answers 404. If it answers 200, your "pass" on any other path means little, and this is the same control-path method our agents.md guide uses.
2. Run the browser request twice more. If the second or third one returns 429, you are rate limited and the comparison is void until you wait.
3. Compare the three lines. Status and content-type are the first signals, byte size the second, the h1 count the third.
4. Look at the body when sizes differ: head -c 600 out_CLAUDE.html. A challenge page is short and mentions a check or a captcha. A Markdown reply starts with front matter or a heading.

How to read the results

Finding the layer. Look at the response headers with curl -sI -A "$UA_CLAUDE" $URL. Cloudflare, Vercel and most CDNs add identifying headers such as server or a vendor-specific one. If the block page names a vendor, the rule lives in that vendor's dashboard. If there is no vendor header, check your application and hosting firewall.

WordPress and robots.txt. If your goal is to change what robots.txt declares, ThinkRank manages robots.txt, robots meta, schema and llms.txt from one WordPress plugin, so you are not juggling three plugins that each write the file, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Rank Math and Yoast both also edit robots.txt and are reasonable if you already run them. Remember that robots.txt does not stop a request from being answered. Only the CDN or server layer can do that.

What a spoofed request cannot prove

A request carrying a ClaudeBot string from your laptop is not ClaudeBot. Big operators publish IP ranges so sites can verify real traffic, and Cloudflare keeps a verified bots directory. A site that allows ClaudeBot only from Anthropic's addresses will refuse your spoofed request, even though the real bot gets in. In the other direction, a site that trusts the string alone will let your spoof through, even though the real bot would be blocked for another reason.

So read results this way:

• A block on the bot string but not the browser string proves a user-agent rule exists.
• A pass on every string proves only that no user-agent rule exists.
• Real access logs are the ground truth. Search your server or CDN logs for ClaudeBot and GPTBot and read the status codes they actually received.

For the verification methods, see the AI crawler user-agent list and the explainer on Web Bot Auth.

Who does not need this

If your site is a small brochure page on shared hosting with no CDN and no bot plugin, a clean scan is probably enough. The comparison pays off when you run Cloudflare or another bot-management layer, a firewall plugin, a headless front end, or any setup where several systems can answer before your application does.

Keep it from drifting

Bot rules change when someone turns on a new managed ruleset or a security plugin updates. Re-run the loop after any CDN or firewall change, and put it in a monthly reminder. Keep a plain text file of the last results so you can diff the sizes.

Your next step

Run npx aiscan-cli yoursite.com and read checks B2 and C3, then paste the curl loop above and compare the three lines. When you want more guides in this series, browse all AIScan guides.