Know how well your customers' agents get things done with you.

When an agent gets stuck, you don't find out.

An agent that gets stuck doesn't complain. It goes somewhere else.

What it is.

How it works.

Setup takes one to three weeks. Most of it is waiting for access.

1. 1

   Connect

   Give us a staging site or test mode for your website and MCP tools.

   Test accounts and test cards. Nothing real is bought, booked or changed.

2. 2

   Design the evals

   We map your surfaces and draft one eval per job.

   50 to 100 tasks. You review them in one 60-minute call and mark the money tasks.

3. 3

   Run the baseline

   Every task runs with our model of each major agent.

   Money tasks run three times. Read-only checks run on your live site with the real agents.

4. 4

   Act on it

   You get the Scorecard and one list of issues and fixes.

   Fix on staging, press re-test, and see the score before and after.

What's inside.

01 · Eval design

A full set of evals, written for your business.

We list every job a customer can send an agent to do, then write one eval per job with many tasks inside. Each task has a test user, a done check, the policies that apply, and a known good path.

• The same job asked many ways, with hard cases like sold out
• Should-not tasks: your policies hold when an agent pushes
• A coverage map of jobs by surface, with the gaps marked
• A core set that runs often, and a full set each quarter

6 evals · 84 tasks · 24 in the core set

• Find and book a flight20Approved
• Change a booking18Approved
• Add bags and seats14Approved
• Cancel and get a refund16Needs review

Change a booking · task 3 of 18

“Move my Thursday flight to Friday. Don’t pay more than $50.”

Done when Friday is booked, the fee is $50 or less, and the agent asked before paying.

Gap: Cancel and get a refund has no MCP tool.

02 · Agent runs

Every task, run many times with each agent.

Our model of each agent does every task on staging, all the way to the end, while a test user answers its questions. Read-only checks run on your live site, with the models and with the real agents by hand.

• Hundreds of runs per baseline
• Money tasks run three times per agent
• A step-by-step replay of every run
• Every result labeled “Model” or “Real agent”

Task

“Cancel my Friday flight and refund it to my card.”

1. 1Opened Manage booking
2. 2Found the Friday flight
3. 3Opened Cancel
4. 4Opened Fare rules
5. 5A pop-up covered the page
6. 6Told the user it couldn’t find the refund rules

Run 2 of 3 · 632 test runs in this baseline · 20 real-agent live checks

03 · Scorecard

One score, and how much of it is yours to fix.

Every run is graded and rolled into the Agent Experience Score, from 0 to 100, always with a range. Five measures stop the score from hiding problems.

• Finished, Correct, Efficient, Safe and Consistent
• Lost points split by owner: you, the agent or the prompt
• Where each agent drops off on your main journeys
• Your rank in your category

61 / 100

Likely 57 to 65
Last week 63

Finished

64

Correct

56

Efficient

40

Safe

85

Consistent

40

Task	Muse	Dots	Spark	Gemini	Claude
Move my flight	90	85	78	70	88
Add a bag (MCP)	30	55	45	–	–
Cancel and refund	20	35	30	25	50
Get an aisle seat	95	90	85	85	92

Gemini and Claude here are their Chrome agents, which work on websites only.

Points lost: 39. You 24 · The agent 11 · The prompt 4.

04 · Issues and fixes

The step that failed, who should fix it, and the fix.

Failed runs that share a cause become one issue. Each one shows the step, the owner, the runs it hurts and the fix in plain steps. Press re-test to prove the fix worked.

• One ranked list, money tasks first
• Causes always named the same way
• Before-and-after scores on every fix
• Tickets for Linear, Jira or GitHub

1. 1

   Fare rules sit behind a pop-up

   Re-tested: 24 → 71

   Cancel and refund, step 5 · You (website) · 142 runs

2. 2

   add_bag asks for a code agents can’t find

   Sent to Linear

   Add a bag, step 3 · You (MCP tool) · 96 runs

3. 3

   change_booking waives the fee with no proof

   Open

   Your policies didn’t hold · You (MCP tool) · 18 runs

Fix: show the fare rules on the page. Up to +11 points.

05 · Ongoing checks

Checked before each release, and when agents change.

Your core tasks run on each release candidate in about 30 minutes. A drop bigger than the range is run again, and if it holds, the release is flagged with the step that broke.

• A pass or a flag in the pull request and in Slack
• Weekly checks on each real agent and its model
• Daily read-only checks on your live site
• An updates feed of what changed and what moved

Flagged: 1 task got worse

• Change a flight (MCP)84 → 38
• Cancel and refund71 → 70
• Get an aisle seat90 → 91

Step 3: change_flight now needs passenger_id, which agents can’t find. Confirmed by a re-run.

Updates · 9 Oct

Muse now opens pop-ups it used to skip. Your Cancel and refund on Muse: 20 → 45.

06 · Analytics

Real agent sessions, turned into new tasks.

We read the agent sessions already in your logs or MCP tracing. They show jobs your evals miss, agents your own rules turned away, and where our models behave differently from the real agents.

• Suggested tasks, added with one click
• Agent requests your rules turned away
• Real weights for each job in the score
• Personal data removed before logs leave your systems

3,214 agent sessions

Verified 1,120 · Declared 410 · Flagged by your bot tool 1,684

Suggested new tasks

• Add a pet to a booking

  Seen 63 times

  Add to evals

• Find the refund policy

  88 requests to a page that doesn’t exist

  Add to evals

Turned away by your own rules: 212 agent requests on /manage-booking.

How we run each agent.

We run the real agents by hand. For hundreds of runs we use our model of each agent, built on the AI it runs on where we can reach it, and checked against the real agent every week. Every result says which.

We check each model every week on our own reference site, a demo store and a demo airline that never change. Grok Bot, Comet, Copilot, Alexa and Instinct come next.

What it checks.

Every run is graded on five measures, and every failure is tied to a cause.

Start with one question.

Tell us what you want to know about how agents do with you. We test it with the agents your customers use.

For example

• “Can agents book, buy or change things with us?”
• “Which agents get stuck, and where?”
• “How do we score with each agent?”
• “What should we fix first, and who owns it?”
• “Will our next release break anything for agents?”
• “How many agents come to us, and do they finish?”