An AI agent evaluation harness becomes useful when the demo works but ordinary customer conversations don’t. A short conversation with a clear request hides the corrections, missing data and tool failures that happen once real users arrive.
You don’t need a huge benchmark to start. Pick one business task and define what a correct outcome looks like. Then make it possible to replay the task with controlled tool behaviour.
Define success outside the reply text
For an order-status task, success might mean returning the verified status for an order the customer owns. For a cancellation task, it might mean preventing a new purchase or requesting reversal of an existing one.
Don’t grade only whether the answer contains a helpful phrase. An agent can say “cancelled” while creating the order. Record tool calls, state changes and customer-facing claims separately.
Anthropic’s evaluation discussion distinguishes outcomes, transcripts and grading methods. We’ll use that distinction to build a small executable harness. Agent evaluation reference.
Give every case a starting state
A case needs a conversation, relevant business records, expected permitted actions and a set of tool responses. Include the permissions and tenant scope, not just the message text.
Use a fresh state for each run. If one case leaves a cancelled order in shared memory, the next case may pass for the wrong reason. Keep external side effects mocked or isolated so rerunning the suite doesn’t create real orders or messages.
For time-sensitive cases, inject the clock. “Approval expired” should mean the same thing whenever the test runs.
Build an outcome grader
The example evaluates a safe implementation and an intentionally broken one against the same fixtures. It checks both the response status and whether an order was created after cancellation.
import assert from 'node:assert/strict';
const cases = [
{ name:'cancelled task', cancelled:true, expected:'cancelled', maxCreates:0 },
{ name:'active task', cancelled:false, expected:'created', maxCreates:1 },
];
function evaluate(agent) {
return cases.map(c => {
let creates=0;
const result=agent(c,()=>{ creates++; });
return { name:c.name, passed:result===c.expected && creates<=c.maxCreates };
});
}
const safe=(c,create)=>{ if(c.cancelled)return 'cancelled'; create();return 'created'; };
const broken=(c,create)=>{ create();return c.cancelled?'cancelled':'created'; };
assert.equal(evaluate(safe).every(r=>r.passed),true);
assert.equal(evaluate(broken)[0].passed,false);
assert.equal(evaluate(broken)[1].passed,true);
console.log('PASS: outcome harness detects hidden action behind a correct-sounding reply');
The safe implementation passes both cases. The broken code is detected because its side effect disagrees with the cancellation, even though its reply says the right word.
This is a deterministic contract test, not a measurement of a language model’s reasoning. It establishes what the harness records and what the grader rejects before you attach a live agent.
Add tool failure fixtures one at a time
Begin with a missing order, a timeout and a stale proposal. For each, write down the business outcome you expect. A missing order may require clarification; an uncertain refund may require reconciliation rather than another mutation.
Give fake tools realistic structured responses. Don’t use an empty string for every failure if your live service returns pending states, error codes and operation IDs. The agent’s behaviour depends on that distinction.
Add a deliberately adversarial case, such as product text requesting an unauthorised action. Check that the action gateway rejects it even if the agent proposes it.
Evaluate variable model behaviour honestly
When you connect a real model, run important cases more than once. Record model version, prompt version, tool schemas, sampling settings and fixture versions. Otherwise, a change in results can be hard to explain.
Keep deterministic business checks for actions and use semantic or human review where wording genuinely has several acceptable forms. A model-based grader can help, but it can also misjudge. Calibrate it against reviewed examples and preserve its disagreements.
Don’t tune only to the cases you show in a demo. Keep some held-out cases and periodically add real anonymised failure patterns, subject to your data policy.
Make regressions actionable
Store the first failing transition and the tool trace alongside the final score. “Seventy percent passed” doesn’t tell an engineer whether the problem is retrieval, permissions, memory or reply wording.
Separate severe action errors from style issues. A minor wording regression should not hide a new ability to issue an unauthorised refund in an average score. Use explicit release gates for those consequential failures.
Also record latency and cost, but compare them only among runs that satisfy the necessary correctness rules. A faster agent that skips the required check isn’t an improvement.
Keep the suite close to the product
Run a small critical set when tools or prompts change, then a broader set before release. When a case fails, decide whether the expected outcome is still right before changing the implementation to satisfy it.
Avoid assertions that simply repeat the code’s current choices. The expected outcome should come from the business requirement: who owns the order, what was confirmed and what may happen next.
Start with five conversations that would be expensive to mishandle. Give each a fresh state, controlled tools and a clear outcome. Once the harness can detect a deliberately broken implementation, it can start protecting you from less obvious regressions.
Run the example and adapt it
Save the code as 12-agent-regression-evaluations.mjs and run it with Node.js 24 or later:
node 12-agent-regression-evaluations.mjs
The code above includes assertions you can run locally. It tests the local decision logic with controlled inputs; it doesn’t connect to a live provider or establish production performance. Keep that boundary when adapting it to your application.
