---
title: "How to Test Whether an AI Agent Can Complete a Task on Your Site"
slug: test-agent-can-complete-task
published: 2026-10-08T08:17:50.382632+00:00
updated: 2026-10-08T08:17:50.382632+00:00
author: "AIScan Content Team"
author_url: https://aiscan.site/about
category: "AI Readiness"
tags: check:P2, platform:shopify, MCP, agent testing, tools/call, AI readiness
description: "Test whether an AI agent can finish a task on your MCP server: list tools, call one, check the answer, run a control call. Includes a pass/fail script."
url: https://aiscan.site/blog/test-agent-can-complete-task
---

Finding an MCP endpoint is the easy half. The half that matters is whether an agent can finish a job with it: ask a question, get an answer it can parse, and act on it. A server can answer `tools/list` with a clean 200 and still fail every real task. This guide gives you a five-step test that takes about ten minutes and ends in a plain pass or fail, using `curl` and a short script, plus the AIScan scan that covers the discovery half.

On 8 October 2026 I ran this test against two live servers, `www.allbirds.com` and `aiscan.site`. Both passed, but they failed a bad call in two different ways, and a script written for one failed the other until it learned to read both. That is the kind of detail only a task test finds.

## Quick summary

| If you want to... | Do this | What passing looks like | Time |
|---|---|---|---|
| Know whether agents can discover your server | Scan with AIScan (`check:P2`, MCP Server Card) | P2 passes, card parses | 1 min |
| Know whether the endpoint lists tools | `POST` a `tools/list` body | JSON-RPC `result.tools` is a non-empty array | 2 min |
| Know whether a tool does its job | `POST` a `tools/call` for one real task | `result.content` holds the answer you expect | 3 min |
| Know whether failures are honest | Call a tool that does not exist | A JSON-RPC `error`, or a result with `isError: true`, not a fake success | 1 min |
| Keep it working | Re-run the script on a schedule or in CI | Exit code 0 | 5 min once |

**Short answer:** discovery, listing, calling and failing correctly are four separate tests. AIScan's `P2` check grades the first. The other three need a request that completes a task, and a pass means the response body parses and holds the key you asked for. A status code alone proves nothing.

## Why a green status code is not a passing test

An MCP server speaks JSON-RPC 2.0 over HTTP. According to the [Model Context Protocol specification](https://modelcontextprotocol.io/specification/2025-06-18/server/tools) (version 2025-06-18), two requests matter here. It defines `tools/list`, which returns the tools a server offers, and `tools/call`, which runs one of them with arguments.

Think of a restaurant. `tools/list` is the menu. `tools/call` is ordering a dish. A menu on the door tells you nothing about whether the kitchen can cook. Most checks, including most scanners, read the menu. A task test orders the dish and checks what arrives.

Two details of the protocol make status codes misleading:

- **Errors travel in the body, in two shapes.** On 8 October 2026 I called a tool that does not exist. `www.allbirds.com/api/mcp` replied HTTP 200 with a JSON-RPC `error` object (`"Tool not found: no_such_tool"`, code -32602). `aiscan.site/api/mcp` also replied HTTP 200, but with a normal `result` carrying `"isError": true` and the text `MCP error -32602: Tool no_such_tool not found`. Both are valid. A script that checks only the status code calls both a pass.
- **The body may not be plain JSON.** The same request to `aiscan.site/api/mcp` returned `content-type: text/event-stream`, with the JSON-RPC message inside a `data:` line. A parser that expects raw JSON fails on a healthy server.

Here is what the same three requests returned on 8 October 2026, fetched from each endpoint with a desktop browser user agent:

| Request | `www.allbirds.com/api/mcp` | `aiscan.site/api/mcp` |
|---|---|---|
| `tools/list` | JSON, 1 tool (`search_shop_policies_and_faqs`) | `text/event-stream`, several tools including `scan_website` and `get_grade` |
| Real `tools/call` | HTTP 200, answer text in `result.content` | HTTP 200, score JSON in `result.content` |
| Unknown tool | JSON-RPC `error`, code -32602 | `result` with `isError: true` |

According to the [specification's tools page](https://modelcontextprotocol.io/specification/2025-06-18/server/tools), tool execution errors are reported inside the result with `isError`, while protocol errors such as an unknown tool use the JSON-RPC `error` object. Both servers are inside the spec, and a test has to accept both.

So a task test counts success only when the body parses, the response has a `result` and no `error`, and the key you need is present inside `result`. The [Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports) lets a server choose either shape, which is why the script below handles both.

## Path 1: scan first, then test by hand with curl

[AIScan](https://aiscan.site/) is the quickest way to check the discovery half. Run `npx aiscan-cli example.com` or paste the URL at [aiscan.site](https://aiscan.site/). It is free and needs no account. The relevant row is `P2` (MCP Server Card) in the [capabilities dimension](https://aiscan.site/docs/checks/capabilities). According to the P2 guidance fetched from the AIScan MCP on 8 October 2026, the check covers the MCP Server Card, a file at `/.well-known/mcp/server-card.json`, so it tells you whether an agent can find your server and read its declared tools. If you have not published a card yet, [publish an MCP server card](https://aiscan.site/blog/publish-mcp-server-card) first.

What AIScan does not do is call your tools. It cannot tell you whether `search_catalog` returns products or an empty array. That is what the by-hand test is for. If you would rather check everything yourself, this path is complete without the scan.

1. **Set the endpoint.** Use the URL from your server card's `transport` entry, or your own path. Shopify stores expose theirs at `/api/mcp` on the storefront domain.

   ```bash
   EP=https://example.com/api/mcp
   H='Content-Type: application/json'
   A='Accept: application/json, text/event-stream'
   ```

2. **List the tools.** Send a `tools/list` request and read the names, not just the status.

   ```bash
   curl -sL -X POST "$EP" -H "$H" -H "$A" \
     -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' \
     -w '\n%{http_code} %{content_type}\n'
   ```

   Pass: `result.tools` is an array with at least one entry, and each entry has a `name` and an `inputSchema`. Fail: HTML, an empty array, or an `error` object.

3. **Pick one real task and call the tool.** Choose the job a visitor's agent would do first. For a store, that is a policy question. On `www.allbirds.com` on 8 October 2026 the list held one tool, `search_shop_policies_and_faqs`, which needs a `query`.

   ```bash
   curl -sL -X POST "$EP" -H "$H" -H "$A" \
     -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_shop_policies_and_faqs","arguments":{"query":"What is your return policy?","policy_types":["refund"]}}}'
   ```

   Pass: the reply has `result.content`, and the text answers the question. Fail: an `error`, an empty `content` array, or `isError: true` in the result, which the specification uses for tool-level failures.

4. **Read the answer against your own site.** This is the step people skip. The Allbirds structured answer said returns are accepted "within 31 days days of delivery", while the written refund policy it returned in the same reply says 30 days. A call can succeed and still hand an agent a contradiction. Compare the answer with the page a human would read.

5. **Run a control call.** Ask for a tool that does not exist.

   ```bash
   curl -sL -X POST "$EP" -H "$H" -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"no_such_tool","arguments":{}}}'
   ```

   Pass: a JSON-RPC `error` with a code and message, or a `result` with `isError: true` and a message that names the problem. Fail: a normal-looking `result` with no `isError`, which means the server accepts anything and your pass in step 3 proves nothing. Also request the same path with `GET`. Verified on 8 October 2026: on the Allbirds host, `GET /api/mcp` returned a 404 HTML page, which is a fair answer for a POST-only endpoint.

If a server insists on a session, the specification's [lifecycle](https://modelcontextprotocol.io/specification/2025-06-18/basic/lifecycle) starts with an `initialize` request followed by a `notifications/initialized` message. Both servers I tested answered `tools/list` without it, so try the short form first and add the handshake only if you get a session error.

## Path 2: a script that returns pass or fail

The by-hand test is good once. A script is better when you change the server often. This one uses only the Python standard library, accepts both JSON and event-stream replies, accepts both error shapes, and exits non-zero on the first failure.

1. **Save this as `mcp_task_test.py`.**

   ```python
   import json, sys, urllib.request

   EP, TOOL, ARGS, NEED = sys.argv[1], sys.argv[2], json.loads(sys.argv[3]), sys.argv[4]

   def fail(msg):
       print("FAIL", msg); sys.exit(1)

   def rpc(i, method, params):
       body = json.dumps({"jsonrpc": "2.0", "id": i, "method": method, "params": params}).encode()
       req = urllib.request.Request(EP, body, {"Content-Type": "application/json",
             "Accept": "application/json, text/event-stream", "User-Agent": "Mozilla/5.0"})
       try:
           raw = urllib.request.urlopen(req, timeout=60).read().decode()
       except Exception as e:
           fail("request failed: %s" % e)
       if "data:" in raw:
           raw = raw.split("data:", 1)[1].strip()
       return json.loads(raw)

   tools = rpc(1, "tools/list", {}).get("result", {}).get("tools") or fail("no tools")
   if TOOL not in [t["name"] for t in tools]: fail("tool missing: " + TOOL)
   out = rpc(2, "tools/call", {"name": TOOL, "arguments": ARGS})
   if "error" in out or out.get("result", {}).get("isError"): fail(out)
   text = json.dumps(out["result"].get("content", []))
   if NEED.lower() not in text.lower(): fail("answer lacks: " + NEED)
   bad = rpc(3, "tools/call", {"name": "no_such_tool", "arguments": {}})
   if "error" not in bad and not bad.get("result", {}).get("isError"): fail("control call succeeded")
   print("PASS", TOOL)
   ```

2. **Run it with a tool name, its arguments and a word the answer must contain.**

   ```bash
   python3 mcp_task_test.py https://example.com/api/mcp search_shop_policies_and_faqs \
     '{"query":"What is your return policy?","policy_types":["refund"]}' "return"
   ```

3. **Read the exit code.** `0` prints `PASS`. Anything else prints one `FAIL` line naming the step: no tools, tool missing, an error object, an answer without the required word, or a control call that did not report an error.

4. **Run it where regressions land.** Put the command in a scheduled job or a deploy step. The [CI gate guide](https://aiscan.site/blog/aiscan-ci-gate-github-actions) shows how to fail a build on a regression, and the same pattern works here with this script as the step.

5. **Choose the required word with care.** Use a word that only a correct answer contains, such as a number of days or a product name. A word like "the" passes on an error message.

## Which result means what

| What you see | Meaning | First thing to try |
|---|---|---|
| `result.tools` array, call returns content, control returns `error` | The server works for agents | Schedule the script |
| HTTP 200 but HTML body | A catch-all page is answering, not an MCP server | Check the route and the `POST` handler |
| `data:` lines and `content-type: text/event-stream` | Healthy streaming reply | Parse the `data:` line, as the script does |
| HTTP 404 on `POST` | No endpoint at that path | Read the real path from your server card |
| `error` with code -32602 | The server rejected your arguments | Compare them with the tool's `inputSchema` |
| Control call returns a plain `result` | The server accepts anything | Fix validation before trusting step 3 |

## Who can skip this

If your site has no MCP endpoint and sells nothing through one, the test does not apply and P2 is optional. According to the P2 guidance, a content site or marketing page can skip the server card. Do not build an endpoint to pass a test. If you run a Shopify store, the `/api/mcp` endpoint comes from the platform, so your work is the content behind it. [StoreSEO](https://storeseo.com/) generates `llms.txt` from live products, collections and pages and has an `agents.md` editor, which are the files an agent reads before it decides to call a tool. For the wider picture, see [what the commerce protocols do](https://aiscan.site/blog/agentic-commerce-protocols-explained) and [whether you need WebMCP yet](https://aiscan.site/blog/webmcp-do-you-need-it-yet).

## Keep it passing

Re-run the test after every deploy that touches routing, a CDN rule or an authentication layer. Servers change under you: earlier posts on this blog counted ten tools on some Shopify storefronts, and on 8 October 2026 `www.allbirds.com` listed one. A test that names a tool will tell you the day it disappears.

## Next step

Scan your site at [aiscan.site](https://aiscan.site/) and read the `P2` row, then run the five steps above against your endpoint. Browse the rest of the [AIScan guides](https://aiscan.site/guides) for the checks that sit around it, and read the [capabilities checks](https://aiscan.site/docs/checks/capabilities) for what each row grades.