---
title: "Fetch Your Own Site the Way an AI Agent Sees It"
slug: fetch-site-as-ai-agent-curl
published: 2026-10-05T08:15:32.665527+00:00
updated: 2026-10-05T08:17:21.940114+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: check:B2, check:C3, AI crawlers, user agent, curl, ClaudeBot, GPTBot, AI readiness
description: "Fetch your homepage as a browser, ClaudeBot and GPTBot with curl, then compare status, content type and size to see what AI agents really receive today."
url: https://aiscan.site/blog/fetch-site-as-ai-agent-curl
---

Your site has at least three audiences: people in a browser, search crawlers, and AI agents. Servers can treat each differently, and the only way to know is to ask as each of them and compare the answers. Four requests and one comparison take about five minutes and show you the status code, content type and body size each identity receives.

This guide is the do-it-yourself version. It covers what the commands send, what the results mean, and what to do when the three answers disagree.

## Quick summary

| If you want to... | Do this | What you learn | Time |
|---|---|---|---|
| Check the declared rules | Run the AIScan scan (checks B2 and C3) | Whether robots.txt names AI bots, and whether your served HTML has real content | 1 minute |
| See what each identity actually gets | Fetch the page with a browser, ClaudeBot and GPTBot user-agent, then compare | Status, content type and size per identity | 5 minutes |
| Rule out a rate limit | Repeat the browser request twice | Whether a 429 is blocking you rather than a bot rule | 1 minute |
| Fix a mismatch | Change the rule at the layer that returned the block | Whether the bot, the browser or both now get the page | 10 to 30 minutes |

**The short answer:** run [AIScan](https://aiscan.site/) first for the declared rules, then fetch your homepage under three user-agent strings and compare status, `content-type` and byte size. A difference between the browser and a bot is the finding. A spoofed request cannot prove what the real bot receives, because many sites also check IP address, so treat a clean result as "no user-agent rule found", not as proof.

## Why the same URL can return three different pages

A web server does not hand out one page per URL. It makes a decision for every request, using the user-agent header, the source IP, the `Accept` header, cookies and rate counters. A CDN or bot-management layer in front of your origin can answer before your application ever sees the request.

Think of a venue with a door list. The host checks your name against several lists, and a person on none of them may be turned away or walked to a different room. Your site has the same lists spread across at least four places: robots.txt (a request, not a lock, as [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) says), your CDN's bot rules, your application and your hosting firewall. Reading the rules tells you what you intended. Fetching the page tells you what happens.

Our own measurements show why the second step matters. On 5 October 2026 we fetched ten homepages under a browser string, a ClaudeBot string and a GPTBot string, from one datacenter IP, one request per identity:

| Site | Browser | ClaudeBot string | GPTBot string |
|---|---|---|---|
| aiscan.site | 200, HTML, 163,251 bytes | same | same |
| stripe.com | 200, HTML, 667,934 bytes | same | same |
| nextjs.org | 200, HTML, 351,854 bytes | 200, `text/markdown`, 3,939 bytes | 200, `text/markdown`, 3,939 bytes |
| openai.com | 403, 9,893 bytes | 200, HTML, 466,175 bytes | 403, 9,872 bytes |
| nytimes.com | 403 | 403 | 403 |

Three lessons sit in that table. nextjs.org answers the two bot strings with a 3.9 KB Markdown version of the page and everyone else with 352 KB of HTML, so an agent and a person get different documents at one URL. openai.com blocked the browser string and the GPTBot string but let the ClaudeBot string through from the same IP, so the rule is not simply "datacenter IP blocked". And nytimes.com refused all three, so a user-agent rule alone does not explain it.

One more result is a warning. On allbirds.com the first run returned 200 to all three strings, with the ClaudeBot response 671,269 bytes against 714,750 for the others. On a repeat run the browser string got `429`. A rate limit looks exactly like a block, so never trust a single request.

## Path 1: ask the scan first

[AIScan](https://aiscan.site/) is free and needs no account. Run it before anything else, because it covers the declared half of the picture in one step.

1. Run `npx aiscan-cli yoursite.com`, or paste the URL at [aiscan.site](https://aiscan.site/). The [CLI docs](https://aiscan.site/docs/cli) list the options.
2. Read check **B2**. It looks for explicit `User-agent:` blocks naming known AI crawlers in robots.txt. Four or more named bots pass, one to three is partial credit.
3. Read check **C3**, which tests whether the HTML your server returns carries real content, not an empty shell that needs JavaScript to fill in.
4. Read the other [bot access checks](https://aiscan.site/docs/checks/bot-access) for Content Signals (B1) and Web Bot Auth (B3).

**What AIScan cannot see:** it does not impersonate ClaudeBot or GPTBot, so it cannot tell you that your CDN challenges one of them. It grades the homepage from the public scan, and it cannot see a rule that depends on the requester's IP. That gap is why the manual path exists.

## Path 2: fetch it by hand with curl

If you would rather check by hand, or you need the bot-versus-browser comparison the scan does not make, this path is complete without any tool of ours. You need a terminal with `curl`.

1. Set three identities. The strings below follow the formats published by [Anthropic](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) and [OpenAI](https://developers.openai.com/api/docs/bots). Vendors change version numbers, so copy the current string from their page if you see a mismatch.

```
UA_BROWSER='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'
UA_CLAUDE='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)'
UA_GPT='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'
```

2. Fetch the same URL under each identity and print the facts that matter. Replace the URL with your own homepage, then a second page such as a product or article.

```
URL=https://yoursite.com/
for NAME in BROWSER CLAUDE GPT; do
  eval UA=\$UA_$NAME
  curl -s -L -m 25 -A "$UA" -o "out_$NAME.html" \
    -w "$NAME %{http_code} %{content_type} %{size_download} bytes, %{num_redirects} redirects\n" "$URL"
  echo "  h1 tags: $(grep -ao '<h[1]' out_$NAME.html | wc -l)"
done
```

3. Add a control path that cannot exist, such as `https://yoursite.com/zz-no-such-page-8841`. A healthy server answers `404`. If it answers `200`, your "pass" on any other path means little, and this is the same control-path method [our agents.md guide](https://aiscan.site/blog/agents-md-explained) uses.
4. Run the browser request twice more. If the second or third one returns `429`, you are rate limited and the comparison is void until you wait.
5. Compare the three lines. Status and `content-type` are the first signals, byte size the second, the h1 count the third.
6. Look at the body when sizes differ: `head -c 600 out_CLAUDE.html`. A challenge page is short and mentions a check or a captcha. A Markdown reply starts with front matter or a heading.

## How to read the results

| What you see | Most likely cause | First fix to try |
|---|---|---|
| All three 200, similar sizes | No user-agent rule found | Nothing. Re-check from a different network |
| Bot 403, browser 200 | A bot rule at the CDN, firewall or application | Find the layer that returned the 403 (see below) |
| All three 403 | The block is on the IP range or a site-wide challenge | Test from a residential network to separate the two |
| Bot 200, size far smaller | Content negotiation or a stripped bot version | Open the body. It may be intentional Markdown |
| Bot 200, h1 count 0 | The HTML is an empty shell | Read the [server-rendering guide](https://aiscan.site/blog/nextjs-react-invisible-to-ai-crawlers) |
| `429` on any identity | Rate limiting | Wait, then repeat |
| `200` on the control path | Soft 404 | Fix that first, then redo the comparison |

**Finding the layer.** Look at the response headers with `curl -sI -A "$UA_CLAUDE" $URL`. Cloudflare, Vercel and most CDNs add identifying headers such as `server` or a vendor-specific one. If the block page names a vendor, the rule lives in that vendor's dashboard. If there is no vendor header, check your application and hosting firewall.

**WordPress and robots.txt.** If your goal is to change what robots.txt declares, [ThinkRank](https://thinkrank.ai) manages robots.txt, robots meta, schema and llms.txt from one WordPress plugin, so you are not juggling three plugins that each write the file, and it migrates settings from Rank Math, Yoast, AIOSEO and SEOPress. Rank Math and Yoast both also edit robots.txt and are reasonable if you already run them. Remember that robots.txt does not stop a request from being answered. Only the CDN or server layer can do that.

## What a spoofed request cannot prove

A request carrying a ClaudeBot string from your laptop is not ClaudeBot. Big operators publish IP ranges so sites can verify real traffic, and Cloudflare keeps a [verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) directory. A site that allows ClaudeBot only from Anthropic's addresses will refuse your spoofed request, even though the real bot gets in. In the other direction, a site that trusts the string alone will let your spoof through, even though the real bot would be blocked for another reason.

So read results this way:

- A **block on the bot string but not the browser string** proves a user-agent rule exists.
- A **pass on every string** proves only that no user-agent rule exists.
- **Real access logs** are the ground truth. Search your server or CDN logs for `ClaudeBot` and `GPTBot` and read the status codes they actually received.

For the verification methods, see the [AI crawler user-agent list](https://aiscan.site/blog/ai-crawler-user-agent-list-2026) and the explainer on [Web Bot Auth](https://aiscan.site/blog/web-bot-auth-explained).

## Who does not need this

If your site is a small brochure page on shared hosting with no CDN and no bot plugin, a clean scan is probably enough. The comparison pays off when you run Cloudflare or another bot-management layer, a firewall plugin, a headless front end, or any setup where several systems can answer before your application does.

## Keep it from drifting

Bot rules change when someone turns on a new managed ruleset or a security plugin updates. Re-run the loop after any CDN or firewall change, and put it in a monthly reminder. Keep a plain text file of the last results so you can diff the sizes.

## Your next step

Run `npx aiscan-cli yoursite.com` and read checks B2 and C3, then paste the curl loop above and compare the three lines. When you want more guides in this series, browse [all AIScan guides](https://aiscan.site/guides).