You Shouldn't Have to Choose Between Reproducible and Accurate Evaluation

For two years the pitch was a trade-off: you could have a grounding score you can reproduce and gate on, or you could have the accuracy of an LLM judge — not both. That trade-off is now gone. Here's the architecture that removed it, and why it matters most in healthcare.

The trade-off everyone accepted

If you evaluate whether an AI answer is actually grounded in its sources, you've faced this fork:

• Deterministic scorers (NLI, rule-based checks) give you the same score every run. You can gate a deploy on them, audit them, and reproduce them a year later. But on the hardest cases — paraphrase, multi-hop reasoning, negation — they concede accuracy to a large language model.
• LLM-as-judge scorers (the default in RAGAS and DeepEval) read more like a human. But they're inconsistent across runs, so you can't gate on them; they bill you for every claim; and they ship your data to a third-party API.

Everyone picked a side. In regulated domains — healthcare especially — most teams picked determinism and quietly accepted a lower ceiling on accuracy, because "we sent patient context to OpenAI to score it" is not a sentence you want in an audit.

You don't have to judge every claim

Here's the observation that dissolves the trade-off: an LLM judge earns its cost on maybe a fifth of claims. On the clear-cut majority — an answer that plainly restates the source, or plainly contradicts it — a deterministic scorer is already right, for free, reproducibly. The judge only adds value on the ambiguous residual: the paraphrases and multi-hop chains where meaning is genuinely hard.

So we stopped judging everything. variA/Bly now runs in two modes:

Deterministic Mode (default). Every claim is scored by the deterministic engine. 100% bit-for-bit reproducible — same input, same score, and you can verify the math from the committed results. No model calls, nothing leaves your environment. This is what you gate on.

Max-Accuracy Mode (opt-in). The deterministic engine still scores everything first, but claims it's unsure about — support sitting near the decision boundary, a numeric or directional rule that fired, or entailment and contradiction signals that disagree — get escalated to a local reasoning verifier. Only that uncertain slice, roughly 15–20% of claims, ever touches the heavier model.

The escalation router is the whole trick: it spends reasoning capacity exactly where the deterministic core is weakest, and nowhere else.

Why "local" is the word that matters

The reasoning verifier runs on your infrastructure. No third-party API. No data egress. For a hospital, a fintech, or anyone under a BAA, that's the difference between "we can use this" and "legal said no."

It also means the accuracy gain doesn't come with the two costs that make LLM-judges painful: you're not paying per-claim API fees on 100% of your traffic (you're touching ~20% locally), and you're not exporting sensitive context to score it.

What it does to the numbers

On clinical hallucination detection — the case that motivated this whole effort — the plain-language result is the one worth remembering:

On healthcare RAG answers, Max-Accuracy Mode catches 84% of hallucinated answers. RAGAS catches 27%. DeepEval catches 22%.

"Catches" means the tool correctly flagged a hallucinated answer as unsupported. Same evaluations, same competitor judge model, matched cost. The deterministic core does the heavy lifting; the local verifier resolves the ambiguous fifth that used to be the ceiling.

On general-domain RAG, the deterministic engine already leads without any judge at all — the architecture adapts: deterministic where it suffices, a local verifier only where it's genuinely needed. (We're finalizing those general-domain figures on the production pipeline before we stamp them.)

The honest reproducibility contract

We're careful about the word "deterministic," because it's load-bearing:

• Deterministic Mode is 100% reproducible. Frozen weights, argmax, verify-the-math. Nothing about it changed.
• Max-Accuracy Mode is not bit-for-bit — a local reasoning model has floating-point behavior that varies across hardware. Instead it's replayable: every verdict is greedy (temperature 0), cached, and audit-logged, so a given claim-and-evidence pair always returns the same stored score and is re-derivable. And every score carries its provenance — you always know whether a claim was scored deterministically or by the local verifier.

We disclose this in the opt-in consent. Max-Accuracy is never called "100% deterministic." The default is; the composite isn't, and we say so.

When to use which

• Regulated, air-gapped, strict-audit, or CI gating → Deterministic Mode. Reproducibility is the product.
• You want to beat the LLM-judges on accuracy, and you can run a model in your own environment → turn on Max-Accuracy. You get judge-grade discrimination on the hard cases with no data egress and a fraction of the cost.

Most teams will gate on the deterministic score and turn on Max-Accuracy for the evaluations where being right on the hard cases matters most. That's the point: you choose per project, and you no longer have to choose between reproducible and accurate.

---

variA/Bly is the deterministic AI evaluation platform for production systems — deterministic by default, with an optional local Max-Accuracy mode when you need to beat the LLM-judges. See the benchmarks or read the methodology.