Every Agent Did Its Job. The Refund Was Still Wrong.
A three-agent refund workflow approved a refund the policy did not allow, while every agent was faithful to what it was given. A production case: the wrong path, the hand-off that changed a fact, the false alarms that make teams stop reading their evals, and how to check each in code.
A customer asked for a refund on an unopened order bought 45 days ago. The policy allows 30. Version 2 of a three-agent support workflow said yes. Every agent did exactly what it was told, every agent was faithful to what it was given, and the release check flagged the version without being able to say which step broke. This post walks through that production case: the two failures that hide between agents, the false alarms that make teams stop reading their evals, and how each can be checked in code.
The report
On 19 November an engineer at a team evaluating us reported a problem: our release gate had failed his version 2, listed zero regressions, and could not tell him which step broke.
He had pressure-tested us the way you should test any evaluation tool: in production, with a known regression planted, to see whether it would find it. It found that something was wrong. It could not say where. That is not good enough, so we spent the next day building what he needed: a comparison of two versions that walks the workflow step by step. This post is his case, rebuilt with sanitised data.
The workflow is ordinary:
- A retrieval agent finds the refund policy that applies.
- A decision agent reads it and calls one of three tools:
approve_refund,deny_refundorescalate_review. - A response agent writes the reply to the customer.
The customer writes: "Can I get a refund for order R-7421? I bought it 45 days ago and it is unopened." They are a standard customer. The standard policy says refunds are allowed within 30 days.
Version 1 got it right. Retrieval found the Standard Refund Policy, the decision agent called deny_refund, and the reply said no, with the reason.
Version 2 changed the retrieval prompt. Retrieval now returned the VIP refund policy: 60 days, VIP customers only. The decision agent read it carefully and called approve_refund ("unopened item is within 60 days"). The response agent wrote a clear, friendly, well-grounded reply: "Yes. Your refund is approved because the item is unopened and within the 60-day window."
Read each step on its own and nothing is wrong:
| Step | What it did | Judged on its own |
|---|---|---|
| Retrieval | Returned a refund policy | A refund policy, for a refund question |
| Decision | Applied the policy it was given | Faithful to its input |
| Response | Explained the decision | Every sentence supported by the document it was given |
The final answer is grounded. It just rests on the wrong document. Our release gate failed the version, but its report listed zero regressions and named no step: it kept only the final answer of each run, and judged the reply on its own. Anyone using it would have had to open the traces one by one. That is the gap he found, and the one the rest of this post is about.
Two failures that live between agents
A multi-agent run can go wrong while every step looks right. In our work with teams shipping these systems, the failures that cost the most time are between the steps, not inside them.
1. The wrong path
The decision agent called approve_refund where deny_refund (or at most escalate_review) was expected. Nothing in the reply tells you that. The reply is fluent either way. You only see it when you check the path: which tools were called, in what order, and whether any were outside the allowed set.
That check is cheap and exact. You state what the run should do, and code compares:
client.log(decide_prompt, decision, agent_name="refund-decision-agent", session_id=session,
tools_called=[{"name": "approve_refund"}],
expected_tools=[{"name": "deny_refund"}],
allowed_tools=["approve_refund", "deny_refund", "escalate_review"])
Version 2 fails Tool correctness and Trajectory on the decision step, and the comparison between the two versions now names that step as the first one to regress, and retrieval as the first step given a different document.

2. The hand-off that changed a fact
The second failure is quieter. Agents pass work to each other: an order ID, a date, an amount, a diagnosis code. When the decision agent hands the case to the response agent, it can drop a field, change one, or invent one. Every later check then passes on a corrupted request: the response agent is perfectly faithful to the wrong number.
We reproduced it with the same workflow. The decision agent had "Order ID: R-7421" and "Days since purchase: 45" in its input. It handed over {"days_since_purchase": 15}. No order ID, and 45 became 15.
A hand-off contract says what a hand-off to an agent must carry, with types:
[{"to": "response-agent",
"fields": [{"name": "order_id", "type": "identifier", "pattern": "R-[0-9]{4}", "required": true},
{"name": "days_since_purchase", "type": "number", "required": true}]}]
Code then checks each hop: every required field is there, every value is valid for its type, and every value is one the sender actually had in its own inputs.

The path checks all pass: the right tool was called, in order. The hand-off is what broke, and it is shown as two findings with the value the sender had and the value it sent. No model is asked whether 45 and 15 are "close enough".
Across many sessions, the path and the hand-offs are two separate columns. A run can take every expected step and still hand the next agent a wrong value, so both are read on their own:

The third problem: false alarms
There is a failure on the other side too, and it decides whether anyone keeps reading the evaluation: the check that fires on a correct run.
We found three in our own scoring of this exact workflow, on the correct version:
- The retrieval step scored 0.47. Its output was "Retrieved document: Standard Refund Policy 2026". Measured as an answer to the customer's question, it is irrelevant and incomplete. But it is not an answer. It is a pointer. A step that names the document it was given is supported by that document, and the measures of an answer (relevance, completeness, intent) do not apply to it. They are now left out rather than counted as 0, and the same step scores 0.71 on what applies.
- "Final answer uses sub-agents" failed. The final answer did not repeat the retrieval step's words ("Retrieved document: …"), so it was flagged for not using it. It used the document's content, which is what matters. A step now counts as used when the answer restates a figure from its documents or states something they support.
- Zero where nothing was measured. When an answer has no countable claim, attribution and context utilization have nothing to measure. Showing 0 reads as "nothing attributed" and "every document wasted". They now show no value.

None of these is exotic. Each is what happens when a metric built for single answers is applied to a step that is not one. In a list of 200 sessions, three false failures per run is enough for a team to stop looking.
How other tools approach this
The tooling for agent evaluation has improved a lot this year, and much of it is good:
- Trajectory matching is widely available, including in open-source libraries. You compare the tool calls an agent made with a reference trajectory in strict, unordered, subset or superset mode, and tune how strictly arguments must match. It is deterministic and runs in CI.
- Tool-call accuracy (the right tools with the right arguments) can be scored without a judge model.
- Goal completion and trajectory quality are usually scored by an LLM judge against a rubric: flexible, and useful for nuance, but a single verdict over a long trace that does not reproduce exactly from run to run and is hard to localise. Recent research on agent traces makes the same point about judges reading whole trajectories at once.
What we did not find in the public documentation we read is a check on the content of a hand-off: whether the fields one agent passes to the next are present, typed, and the same facts the sender had. Nor did we find a comparison of two versions that names the first step to regress. The common workarounds are to log the payload and read it when something goes wrong, write a schema validator that confirms the field exists but not that its value is right, or ask a judge to read the whole trace. If your stack does this differently, we would like to hear how.
What we would check on any multi-agent workflow
- The path, in code: expected tools, order, allowed set.
- Each hand-off, in code: required fields present, valid for their type, and traceable to what the sender had.
- Each step against what it was given: claims against the documents and tool results that step received, not against the final answer.
- The final answer against the run: does it use what the sub-agents found, or contradict it?
- Not applicable as not applicable: a pointer, a routing step or a tool call is not an answer, and should never score 0 as one.
- Two versions, step by step: when a version gets worse, the first step that changed is the one to read.
All of it decided by code where code can decide, with a reasoning model only for what code cannot read, and people labelling a sample to check that the model agrees with them.
How are you handling this?
We are curious how teams running agents in production deal with this today:
- When a multi-agent version gets worse, how do you find the step that caused it?
- Do you check what one agent hands the next, or only the final answer?
- What do you do about checks that fire on correct runs?
Tell us. If you want to see your own workflow checked step by step, send us one trace and we will show you what we find.
Want this kind of evaluation for your RAG system?
Talk to us