How to Test Whether an AI Agent Can Complete a Task on Your Site

Finding an MCP endpoint is the easy half. The half that matters is whether an agent can finish a job with it: ask a question, get an answer it can parse, and act on it. A server can answer tools/list with a clean 200 and still fail every real task. This guide gives you a five-step test that takes about ten minutes and ends in a plain pass or fail, using curl and a short script, plus the AIScan scan that covers the discovery half.

On 8 October 2026 I ran this test against two live servers, www.allbirds.com and aiscan.site. Both passed, but they failed a bad call in two different ways, and a script written for one failed the other until it learned to read both. That is the kind of detail only a task test finds.

Quick summary

Short answer: discovery, listing, calling and failing correctly are four separate tests. AIScan's P2 check grades the first. The other three need a request that completes a task, and a pass means the response body parses and holds the key you asked for. A status code alone proves nothing.

Why a green status code is not a passing test

An MCP server speaks JSON-RPC 2.0 over HTTP. According to the Model Context Protocol specification (version 2025-06-18), two requests matter here. It defines tools/list, which returns the tools a server offers, and tools/call, which runs one of them with arguments.

Think of a restaurant. tools/list is the menu. tools/call is ordering a dish. A menu on the door tells you nothing about whether the kitchen can cook. Most checks, including most scanners, read the menu. A task test orders the dish and checks what arrives.

Two details of the protocol make status codes misleading:

• Errors travel in the body, in two shapes. On 8 October 2026 I called a tool that does not exist. www.allbirds.com/api/mcp replied HTTP 200 with a JSON-RPC error object ("Tool not found: no_such_tool", code -32602). aiscan.site/api/mcp also replied HTTP 200, but with a normal result carrying "isError": true and the text MCP error -32602: Tool no_such_tool not found. Both are valid. A script that checks only the status code calls both a pass.
• The body may not be plain JSON. The same request to aiscan.site/api/mcp returned content-type: text/event-stream, with the JSON-RPC message inside a data: line. A parser that expects raw JSON fails on a healthy server.

Here is what the same three requests returned on 8 October 2026, fetched from each endpoint with a desktop browser user agent:

According to the specification's tools page, tool execution errors are reported inside the result with isError, while protocol errors such as an unknown tool use the JSON-RPC error object. Both servers are inside the spec, and a test has to accept both.

So a task test counts success only when the body parses, the response has a result and no error, and the key you need is present inside result. The Streamable HTTP transport lets a server choose either shape, which is why the script below handles both.

Path 1: scan first, then test by hand with curl

AIScan is the quickest way to check the discovery half. Run npx aiscan-cli example.com or paste the URL at aiscan.site. It is free and needs no account. The relevant row is P2 (MCP Server Card) in the capabilities dimension. According to the P2 guidance fetched from the AIScan MCP on 8 October 2026, the check covers the MCP Server Card, a file at /.well-known/mcp/server-card.json, so it tells you whether an agent can find your server and read its declared tools. If you have not published a card yet, publish an MCP server card first.

What AIScan does not do is call your tools. It cannot tell you whether search_catalog returns products or an empty array. That is what the by-hand test is for. If you would rather check everything yourself, this path is complete without the scan.

1. Set the endpoint. Use the URL from your server card's transport entry, or your own path. Shopify stores expose theirs at /api/mcp on the storefront domain.

   EP=https://example.com/api/mcp
   H='Content-Type: application/json'
   A='Accept: application/json, text/event-stream'
   

2. List the tools. Send a tools/list request and read the names, not just the status.

   curl -sL -X POST "$EP" -H "$H" -H "$A" \
     -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' \
     -w '\n%{http_code} %{content_type}\n'
   

   Pass: result.tools is an array with at least one entry, and each entry has a name and an inputSchema. Fail: HTML, an empty array, or an error object.

3. Pick one real task and call the tool. Choose the job a visitor's agent would do first. For a store, that is a policy question. On www.allbirds.com on 8 October 2026 the list held one tool, search_shop_policies_and_faqs, which needs a query.

   curl -sL -X POST "$EP" -H "$H" -H "$A" \
     -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_shop_policies_and_faqs","arguments":{"query":"What is your return policy?","policy_types":["refund"]}}}'
   

   Pass: the reply has result.content, and the text answers the question. Fail: an error, an empty content array, or isError: true in the result, which the specification uses for tool-level failures.

4. Read the answer against your own site. This is the step people skip. The Allbirds structured answer said returns are accepted "within 31 days days of delivery", while the written refund policy it returned in the same reply says 30 days. A call can succeed and still hand an agent a contradiction. Compare the answer with the page a human would read.

5. Run a control call. Ask for a tool that does not exist.

   curl -sL -X POST "$EP" -H "$H" -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"no_such_tool","arguments":{}}}'
   

   Pass: a JSON-RPC error with a code and message, or a result with isError: true and a message that names the problem. Fail: a normal-looking result with no isError, which means the server accepts anything and your pass in step 3 proves nothing. Also request the same path with GET. Verified on 8 October 2026: on the Allbirds host, GET /api/mcp returned a 404 HTML page, which is a fair answer for a POST-only endpoint.

If a server insists on a session, the specification's lifecycle starts with an initialize request followed by a notifications/initialized message. Both servers I tested answered tools/list without it, so try the short form first and add the handshake only if you get a session error.

Path 2: a script that returns pass or fail

The by-hand test is good once. A script is better when you change the server often. This one uses only the Python standard library, accepts both JSON and event-stream replies, accepts both error shapes, and exits non-zero on the first failure.

1. Save this as mcp_task_test.py.

   import json, sys, urllib.request
   
   EP, TOOL, ARGS, NEED = sys.argv[1], sys.argv[2], json.loads(sys.argv[3]), sys.argv[4]
   
   def fail(msg):
       print("FAIL", msg); sys.exit(1)
   
   def rpc(i, method, params):
       body = json.dumps({"jsonrpc": "2.0", "id": i, "method": method, "params": params}).encode()
       req = urllib.request.Request(EP, body, {"Content-Type": "application/json",
             "Accept": "application/json, text/event-stream", "User-Agent": "Mozilla/5.0"})
       try:
           raw = urllib.request.urlopen(req, timeout=60).read().decode()
       except Exception as e:
           fail("request failed: %s" % e)
       if "data:" in raw:
           raw = raw.split("data:", 1)[1].strip()
       return json.loads(raw)
   
   tools = rpc(1, "tools/list", {}).get("result", {}).get("tools") or fail("no tools")
   if TOOL not in [t["name"] for t in tools]: fail("tool missing: " + TOOL)
   out = rpc(2, "tools/call", {"name": TOOL, "arguments": ARGS})
   if "error" in out or out.get("result", {}).get("isError"): fail(out)
   text = json.dumps(out["result"].get("content", []))
   if NEED.lower() not in text.lower(): fail("answer lacks: " + NEED)
   bad = rpc(3, "tools/call", {"name": "no_such_tool", "arguments": {}})
   if "error" not in bad and not bad.get("result", {}).get("isError"): fail("control call succeeded")
   print("PASS", TOOL)
   

2. Run it with a tool name, its arguments and a word the answer must contain.

   python3 mcp_task_test.py https://example.com/api/mcp search_shop_policies_and_faqs \
     '{"query":"What is your return policy?","policy_types":["refund"]}' "return"
   

3. Read the exit code. 0 prints PASS. Anything else prints one FAIL line naming the step: no tools, tool missing, an error object, an answer without the required word, or a control call that did not report an error.

4. Run it where regressions land. Put the command in a scheduled job or a deploy step. The CI gate guide shows how to fail a build on a regression, and the same pattern works here with this script as the step.

5. Choose the required word with care. Use a word that only a correct answer contains, such as a number of days or a product name. A word like "the" passes on an error message.

Which result means what

Who can skip this

If your site has no MCP endpoint and sells nothing through one, the test does not apply and P2 is optional. According to the P2 guidance, a content site or marketing page can skip the server card. Do not build an endpoint to pass a test. If you run a Shopify store, the /api/mcp endpoint comes from the platform, so your work is the content behind it. StoreSEO generates llms.txt from live products, collections and pages and has an agents.md editor, which are the files an agent reads before it decides to call a tool. For the wider picture, see what the commerce protocols do and whether you need WebMCP yet.

Keep it passing

Re-run the test after every deploy that touches routing, a CDN rule or an authentication layer. Servers change under you: earlier posts on this blog counted ten tools on some Shopify storefronts, and on 8 October 2026 www.allbirds.com listed one. A test that names a tool will tell you the day it disappears.

Next step

Scan your site at aiscan.site and read the P2 row, then run the five steps above against your endpoint. Browse the rest of the AIScan guides for the checks that sit around it, and read the capabilities checks for what each row grades.