You built a multi-agent system, and now you want to check whether the agents stayed coherent across the trace. The obvious tool — an LLM judge — is the one that quietly puts you back where you started.

In the last post, the argument was that a multi-agent system has three correctness problems, not one — was each agent faithful, did meaning survive the handoff, did the workflow solve the task — and a single score on the final answer can't tell you which one broke. The natural next question: so what do you evaluate the trace with?

The instinct is to reach for an LLM judge. It's a model, it understands meaning, and the failure is about meaning — so point it at each handoff and ask "did Agent B preserve Agent A's intent?"

That instinct is the trap.

Determinism isn't the point — correctness is

The usual defense of an LLM judge is "run it at temperature 0 and it's repeatable." True, and beside the point. Repeatable isn't the same as correct. A judge can return the same wrong verdict every single time. Determinism buys reproducibility; it says nothing about whether the verdict agrees with a human.

So the real question isn't "is the judge repeatable." It's "is the judge right — and how would you know?"

A judge on a chain reproduces the failure mode

Silent drift — the thing you're trying to catch — is what happens when a probabilistic component confidently reports success while quietly getting the meaning wrong.

An LLM judge is also a probabilistic component that confidently reports a verdict.

Point it at the handoff and you haven't escaped the failure mode — you've added a second instance of it. Now two things can drift: the chain, and the judge grading the chain. Someone has to evaluate the evaluator, and if the answer is "another LLM," it's turtles all the way down.

Three concrete ways it bites in multi-agent

1. The judge drifts the same way the chain does. Asking "did Agent B preserve the intent" requires inferring intent — a probabilistic judgment layered on a probabilistic pipeline. When the judge is unsure, it does exactly what your agents did: returns a fluent, confident answer that may be quietly wrong.

2. You can't gate on a verdict that changes on rerun. A coherence check is only useful if "does step 2 contradict step 5?" returns the same answer twice. A non-deterministic judge doesn't — run the trace again and a contradiction can appear or vanish, so you can't separate a real regression in one agent from judge noise in another. Across several scored hops, that noise stacks rather than cancels.

3. Cost and latency scale with hop count. A judge you ran once per response now runs once per agent, per handoff, per response. A five-agent pipeline is a 5×+ judge bill and 5× the latency — to buy a signal you still can't gate on.

And the punchline: even then, you still don't get localization. A judge that reports "the trace looks off" hands you back the exact question you started with — which agent, which handoff.

Where an LLM judge is genuinely fine

To be fair, LLM-as-judge isn't useless. For subjective, advisory dimensions — tone, helpfulness, "is this a good answer" — where there's no ground truth and the output informs a human rather than gating a deploy, a judge is a reasonable tool. The trap is specifically using it as a gate on factual questions — grounding, coherence, contradiction — where the verdict has to be trustworthy and reproducible.

The alternative: structure + deterministic meaning checks

The strongest answer splits the problem in two:

• Typed, schema-validated handoffs catch a large class of failures deterministically, with no model at all. If Agent A must emit a known schema and produces something malformed or out-of-range, plain code catches it. Cheap and bulletproof.
• But schemas catch structure, not meaning. A handoff can be perfectly valid JSON and still lose the user's intent — the format survives, the meaning drifts. That semantic residue is what still needs evaluating.
• For the residue, use deterministic semantic checks — NLI entailment, embedding similarity, rule-based contradiction and numeric verification — not an LLM judge. Same trace, same verdict, every run.

Use code for structure, deterministic models for meaning, and keep the probabilistic judge out of the path you gate on.

A simple test: gate or advisory?

For any check you add, ask: would you block a deploy on it? If yes, it has to be deterministic — you can't gate on a verdict that changes on rerun. If it's advisory — a signal a human reads — a probabilistic judge is tolerable. Most multi-agent coherence questions you actually care about are gate questions. That's why they need to be deterministic.

Back to the three levels

The three levels of multi-agent evaluation — was each agent faithful, did meaning survive the handoff, did the workflow solve the task — are all factual, gate-relevant questions. Which is exactly why answering them with another probabilistic judge doesn't hold up. Evaluate the chain with something that doesn't share the chain's failure mode.

---

variA/Bly scores agent traces deterministically — per-hop grounding, cross-step coherence, and aggregated-output verification, with the same verdict on every run. If you're running multi-agent AI and want checks you can actually gate on, reach out.