Claim-level grounding, retrieval relevance, deterministic scoring — what we learn building the evaluation methodology behind variA/Bly.
"Deterministic evals" means two things: deterministic checks on closed-form outputs (a test library in CI) and deterministic verification of open-ended answers against their sources. How to tell which one you need for RAG, chat and AI agents — and why most teams need both.
Read the postFor years, grounding evaluation forced a trade-off: a deterministic score you can gate and reproduce, or the accuracy of an LLM judge — not both. A two-mode architecture removes it. Deterministic by default; an opt-in local reasoning verifier for only the ~20% of ambiguous claims, with no third-party API and no data egress. On clinical hallucination detection, it catches 84% vs RAGAS's 27% and DeepEval's 22%.
Read the postThe obvious way to evaluate a multi-agent trace is an LLM judge — and it's the one tool that reintroduces the failure you're trying to catch. Why determinism isn't correctness, why a judge on a chain drifts the same way the chain does, when a judge is still fine, and what to use instead (typed handoffs + deterministic semantic checks).
Read the postA multi-agent system doesn't have one correctness problem — it has three: was each agent faithful, did meaning survive the handoff, did the workflow solve the user's task? A production-style diabetes assistant passed its 88% final-answer faithfulness check and still gave the wrong advice. Why a single score can't debug an agent pipeline, how to check all three levels deterministically between the agents, and why an LLM-judge makes reproducible gating and root-cause localization harder.
Read the postA practitioner's guide to detecting fluent-but-wrong outputs — and why scores you can't reproduce aren't really scores. Anatomy of a hallucination grader, the Fluency Trap, the six categories of hallucination, determinism, per-claim verdicts, domain routing, public benchmark numbers, and an honest 0-to-1 roadmap.
Read the postLangfuse, LangSmith, and variA/Bly all show up in the same "which LLMOps tool should I use" conversations — but they answer different questions. A practical, honest comparison with feature matrix, methodology breakdown, and a decision tree.
Read the postYour LLM bill shocked you last year. The evaluation bill is next. Why RAGAS and DeepEval — both LLM-as-judge under the hood — can quietly double your AI cost, with worked math at 10K calls/month.
Read the postWhy deterministic LLM evaluation is the foundation regression detection, statistical A/B testing, and compliance audits all build on — and the reproducibility test most eval vendors don't want you to run.
Read the postA diabetes clinical agent scored 96% faithfulness on every standard hallucination metric — and was still completely wrong. The number nobody checks tells the real story.
Read the post