A multi-agent diabetes assistant passed its final-answer faithfulness check at 88% — and still gave a patient the wrong advice. No agent errored. Every component reported success. The problem wasn't a bad agent. It was that the meaning of the task drifted between agents while each one, on its own, looked correct.

Multi-agent systems don't usually fail because an agent is bad. They fail because meaning drifts from one agent to the next while every agent, in isolation, appears to be doing its job. Every additional agent adds another opportunity for meaning to drift. And a score on the final answer — the thing almost everyone measures — cannot tell you which agent failed.

Faithfulness is a local property; correctness is a global one.

That one gap is why final-answer evaluation can't debug an agent pipeline. To actually find the break, you have to stop asking one question about the output and start asking three questions about the trace:

1. Was each agent faithful to what it received?
2. Did meaning survive the handoff between agents?
3. Did the overall workflow solve the user's task?

A multi-agent system doesn't have one correctness problem. It has these three — and they fail independently. An agent can be perfectly faithful (1) while the handoff into it quietly lost the user's intent (2), so the workflow confidently answers the wrong question (3). A single number on the final answer blurs all three into one and tells you none of them.

The rest of this post walks a real trace where exactly that happened, shows which of the three levels broke where, and shows how to check all three deterministically — between the agents, not just at the end.

The system

A patient-facing diabetes assistant, built as four cooperating agents — the pattern you'll recognize from most production agent stacks:

1. Triage agent — classifies the query intent (dosing, monitoring, diet, symptoms) and routes it.
2. Retrieval agent — pulls clinical documents for the routed intent.
3. Reasoning agent — generates the clinical answer from the retrieved documents.
4. Safety / formatter agent — adds disclaimers, converts units, and formats the response for a patient.

The query we'll follow:

"Should I adjust my insulin dose if my morning glucose is consistently above 180?"

The true intent is insulin dosing — a titration question. Keep that in mind.

The final answer looked shippable

We scored the final response the way most teams score a RAG system — faithfulness against the documents that reached the generation step:

• Faithfulness: 88%
• Hallucination Rate: 12%

An 88% faithfulness on a clinical answer is the kind of number that clears a review and ships. The response was fluent, cited its sources, and carried a sensible-sounding disclaimer.

It was also answering the wrong question, and it contained a unit-conversion error that could change a dose.

Neither of those shows up in a final-answer faithfulness score. They show up when you score every hop.

The three levels, measured

Here is the same trace, evaluated agent by agent — one column per level. Local faithfulness is Level 1 ("was this agent faithful to what it received?"). Handoff integrity is Level 2 ("did the meaning survive the handoff into this agent?"). Task relevance is Level 3 ("is this hop's output still serving what the patient actually asked?").

Agent	Local Faithfulness	Handoff Integrity	Task Relevance	Failure reason
Triage / Router	—	—	34%	intent misclassification
Retrieval	—	85%	18%	context mismatch
Reasoning	91%	76%	41%	—
Safety / Formatter	88%	52%	44%	numeric mismatch
System (final answer)	88%	—	31%	—

Read it one level at a time and the failure locates itself:

• Level 1 (faithfulness) never broke. 88–91% at every hop that has a score. Each agent stuck to what it was given. This is the only level a final-answer faithfulness score sees — which is exactly why the 88% looked shippable.
• Level 3 (task relevance) broke first, at hop one. The triage agent scored 34% because it routed an insulin-dosing question into the glucose-monitoring path. From there, every downstream agent was faithful (Level 1) to the wrong task (Level 3). The retrieval agent's 85% handoff integrity means it correctly served the intent it was handed — it understood what triage said, not what the patient meant. Format survived; meaning didn't.
• Level 2 (handoff integrity) broke last, at the final hop. The formatter's integrity dropped to 52% because it introduced a numeric mismatch: it "helpfully" rendered "above 180 mg/dL" as "(18.0 mmol/L)." The correct conversion is 10.0. A number the system invented, entered after every real reasoning step — and nothing about the output looks malformed.

The final answer's 88% is real. It's also the least useful number in the table: it reports Level 1 only, the one level that never failed, and averages away a broken route (Level 3) and a fabricated unit (Level 2). One score, three independent failure surfaces, and it can see exactly one of them.

The same pattern, generalized

One trace shows the shape; these are the recurring failure modes across agent pipelines — and the level that catches each:

• Cascading misroute (Level 3) — one bad hop poisons all downstream work; task relevance drops at the hop it originates while local faithfulness stays high.
• The localization gap — the output is wrong but every agent looks fine in isolation; only a per-agent scorecard, not an aggregate, tells you where.
• Handoff drift (Level 2) — a fact, number, or caveat mutates between agents; caught by handoff integrity plus explicit numeric mismatch / contradiction failure reasons.
• Faithful-to-the-wrong-context (Levels 1 vs 3) — each agent is faithful to what it received; the system still answers the wrong question. The gap between local faithfulness and task relevance is the tell.
• Confidence amplification — uncertainty is stripped at each hop, so the answer sounds more certain than the evidence supports; attribution accuracy falls while fluency rises.
• The lucky pipeline — a broken process that happens to land on a right-looking answer; scoring the trace stops a sound process and a lucky guess from getting the same grade.

Every one is invisible to a final-answer score, and legible in a per-hop trace.

The fix: check between the agents, not just at the end

If the failures happen at the handoffs, that's where the checks belong. Not one evaluation on the final answer — a checkpoint on every handoff that verifies Levels 1 and 2 (was this agent faithful, did the meaning survive), plus a workflow-level check that verifies Level 3 (did we actually solve the task).

Because the checks sit between the agents, a failure is localized to the hop that caused it. The triage misroute lights up at the very first checkpoint — not three steps later where the wrong answer finally becomes visible. That's the difference between "the pipeline is broken" and "the triage agent is the root cause; the retrieval and reasoning agents are false suspects." Without it, teams burn days tuning the two agents that were behaving, because that's where the damage surfaced.

There's one non-negotiable property those checks need — and it's where the obvious tool quietly fails.

The checks have to be deterministic

That property is determinism — the same trace scores the same way every run — and it's the thing an LLM-judge can't give you here.

The tempting move is to point a judge model at each handoff: "read this and tell me if the meaning was preserved." But an LLM-judge introduces another probabilistic layer on top of an already-probabilistic pipeline, which makes the two things you most need from a multi-agent trace harder, not easier:

• Reproducible gating. A non-deterministic judge can score the same handoff differently on rerun, so "step 2 contradicts step 5" isn't a verdict you can gate a deploy on — the contradiction can appear or vanish between runs, and across several scored hops that noise stacks rather than cancels.
• Root-cause localization. A judge that reports "the trace looks off" hands you back the question you started with — which agent, which handoff. You've moved the ambiguity up a level, not resolved it.

Deterministic checks — Natural Language Inference for claim support, embedding similarity for relevance, rule-driven claim decomposition, domain-routed clinical scoring — return the same per-hop scorecard on every run. That's what turns "these two agents disagree" into something you can act on instead of a second opinion that drifts as quietly as the system it's grading.

(This trade-off deserves its own treatment — we'll go deeper in a follow-up, "Why LLM-as-Judge Breaks for Multi-Agent Evaluation.")

How variA/Bly runs the three levels

Three deterministic checks on one trace — one per level:

• Level 1 — per-hop grounding. Instrument each agent's input and output through the SDK; every hop is scored independently on faithfulness, attribution, context utilization, and retrieval relevance, with failure reasons on the handoffs. Session tracing groups the hops so the scorecard above is the default view, not something you assemble by hand.
• Level 2 — cross-step coherence. Does step 2's output contradict step 5's? Per-hop scores can each look healthy while two agents quietly disagree; comparing them against each other is the checkpoint that catches the intent drift individual hops miss.
• Level 3 — aggregation grounding. Does the final output actually reflect the upstream signals it claims to combine, or did it drop, invert, or invent one along the way? This is what separates a genuinely-grounded answer from a confident summary that quietly rewrote its inputs.

Same trace, same verdicts, every run — so each level becomes a signal you can gate a deploy on.

The question to ask about your own agents

The honest test isn't "what's my faithfulness score." It's: when your pipeline gives a wrong answer, can you tell which of the three broke — an unfaithful agent, a lossy handoff, or a workflow that faithfully solved the wrong task — and which hop it happened at?

If the answer is a single number on the final output, you can't — it reports one level and averages away the other two.

Multi-agent systems don't have one correctness problem. They have three.

If you're only measuring the final answer, you're measuring just one of them.

---

This evaluation was run with variA/Bly's claim-level grounding analysis, which scores each agent hop independently — faithfulness, attribution, context utilization, retrieval relevance, and handoff failure reasons — deterministically, so the per-hop scorecard is a regression signal you can gate on. If you're running multi-agent AI in production and want to see where your trace actually breaks, reach out.