blog

Field notes on production AI evaluation.

Claim-level grounding, retrieval relevance, deterministic scoring — what we learn building the evaluation methodology behind variA/Bly.

September 6, 2026Deterministic evaluationHallucination detectionHealthcare AILLM-as-judgeRAG evaluationGrounding
You Shouldn't Have to Choose Between Reproducible and Accurate Evaluation
For years, grounding evaluation forced a trade-off: a deterministic score you can gate and reproduce, or the accuracy of an LLM judge — not both. A two-mode architecture removes it. Deterministic by default; an opt-in local reasoning verifier for only the ~20% of ambiguous claims, with no third-party API and no data egress. On clinical hallucination detection, it catches 84% vs RAGAS's 27% and DeepEval's 22%.
Read the post
July 17, 2026LLM-as-judgeMulti-agent systemsAgent evaluationDeterministic evaluationAI agents
Why LLM-as-Judge Breaks for Multi-Agent Evaluation
The obvious way to evaluate a multi-agent trace is an LLM judge — and it's the one tool that reintroduces the failure you're trying to catch. Why determinism isn't correctness, why a judge on a chain drifts the same way the chain does, when a judge is still fine, and what to use instead (typed handoffs + deterministic semantic checks).
Read the post
July 16, 2026Multi-agent systemsAI agentsAgent evaluationEvaluation frameworkDeterministic evaluationAgent tracesLLM-as-judge
Why Multi-Agent AI Needs Three Levels of Evaluation
A multi-agent system doesn't have one correctness problem — it has three: was each agent faithful, did meaning survive the handoff, did the workflow solve the user's task? A production-style diabetes assistant passed its 88% final-answer faithfulness check and still gave the wrong advice. Why a single score can't debug an agent pipeline, how to check all three levels deterministically between the agents, and why an LLM-judge makes reproducible gating and root-cause localization harder.
Read the post
June 8, 2026Hallucination evaluationNLIDeterministic evaluationRAGAI agentsFluency Trap
Demystifying Hallucination Evaluation for RAG and AI Agents
A practitioner's guide to detecting fluent-but-wrong outputs — and why scores you can't reproduce aren't really scores. Anatomy of a hallucination grader, the Fluency Trap, the six categories of hallucination, determinism, per-claim verdicts, domain routing, public benchmark numbers, and an honest 0-to-1 roadmap.
Read the post
May 20, 2026LLMOpsLangfuseLangSmithComparisonTool selection
When to Use Langfuse, LangSmith, or variA/Bly: Choosing Your LLMOps Stack in 2026
Langfuse, LangSmith, and variA/Bly all show up in the same "which LLMOps tool should I use" conversations — but they answer different questions. A practical, honest comparison with feature matrix, methodology breakdown, and a decision tree.
Read the post
April 8, 2026CostLLM-as-judgeRAGASDeepEvalLow-cost AI evaluation
The Hidden Bill Shock in LLM-as-Judge Evaluation
Your LLM bill shocked you last year. The evaluation bill is next. Why RAGAS and DeepEval — both LLM-as-judge under the hood — can quietly double your AI cost, with worked math at 10K calls/month.
Read the post
February 11, 2026Deterministic evaluationReproducibilityLLM-as-judgeCompliance
Run It Twice. Did You Get the Same Score?
Why deterministic LLM evaluation is the foundation regression detection, statistical A/B testing, and compliance audits all build on — and the reproducibility test most eval vendors don't want you to run.
Read the post
December 17, 2025RAGEvaluationRetrievalHallucination
Your AI Isn't Hallucinating — Your Retrieval Is
A diabetes clinical agent scored 96% faithfulness on every standard hallucination metric — and was still completely wrong. The number nobody checks tells the real story.
Read the post