On clinical answers, standard LLM-as-judge eval approves 78% of hallucinations as “grounded.”
variA/Bly holds that to 5.5%.
We ran 2,000 clinical question-answer evaluations from PubMedQA through variA/Bly's grounding scorer alongside DeepEval, a widely used LLM-as-judge library. Same data, same labels. On this set variA/Bly leads the metric the AI-judge tools advertise (AUC) and the one they don't publish (false-positive rate). Methodology and reproduction are open below.
What this means
DeepEval approves 78 of every 100 hallucinated answers.
variA/Bly catches 95 of them.
A clinical hallucination is an answer that states something the source study doesn't support — a fabricated statistic, an invented dosage, a claim about a population the study never covered. A grounding scorer's job is to flag it before it reaches a clinician or a patient. That's exactly the failure mode this benchmark measured, on 2,000 clinical cases.
DeepEval
78 of 100 approved
hallucinated, called grounded
Run DeepEval on 1,000 hallucinated answers and it calls ~780 of them grounded. Each is a confidently wrong clinical answer auto-passed downstream.
variA/Bly
~945 of 1,000 caught
hallucinated, correctly flagged
On the same 1,000 hallucinated answers variA/Bly flags ~945 of them — they reach your review queue with a per-claim audit trail, not a patient.
That's a ~14× lower false-approval rate — same 2,000 cases, same data. For clinical, financial, and legal AI, how often a hallucination gets approved is the number that matters most.
A real disagreement from the benchmark
Look at one example side-by-side.
Sample pqa_id=22537902 from the public PubMedQA dataset. This is a real evaluation from the run, not a hand-picked illustration.
Question
Colorectal cancer with synchronous liver metastases: does global management at the same centre improve results?
AI response
Notably, the global-management (GM) group demonstrated a slightly higher overall survival rate at 3 years compared to the separate-management (SM) group, although this was not statistically significant.
Source (the study abstract)
“Overall survival rates were 67% and 51% for SM and GM at 3 years (n.s.)…” — i.e. separate management was higher, the opposite of what the response claims.
How each scorer rated this response:
DeepEval
✓ Grounded
1.00
The LLM judge found no hallucination and approved the response. The verdict is a single aggregate score with no per-claim trace.
variA/Bly
✗ Not grounded
0.00
The survival-rate claim (GM higher) is not supported by the source, which reports the reverse (SM 67% vs GM 51%). The unsupported claim is the failure.
Why variA/Bly said no - the per-claim audit trail
DeepEval gives you a score. variA/Bly gives you a full per-claim trace of why. Every claim variA/Bly flags comes with the source excerpt it was checked against and the specific reason it failed - so a reviewer, or an auditor, can see exactly what the model got wrong.
The number to watch at deploy time
False-positive rate on hallucinated answers
How often each scorer wrongly approves an answer the source doesn't support. Lower is better — every false approval is a confidently wrong answer passed downstream.
AI-judge tools tend to publish AUC. We publish AUC and FPR — ~14× cleaner on the same 2,000 clinical answers.
Reading the numbers honestly
This time we lead on both AUC and FPR.
AUC measures overall ranking quality across every threshold — it's the metric AI-judge tools advertise. On this clinical set variA/Bly scores 0.943 vs DeepEval's 0.895 — the highest of any scorer we tested.
And at the standard 0.5 threshold you actually deploy, the gap on false approvals is decisive: 5.5% vs 78.2% — DeepEval reaches near-ceiling recall precisely because it approves almost everything. variA/Bly doesn't trade accuracy for safety; it leads on both.
variA/Bly leads on
- Overall ranking quality (AUC)0.943vs0.895
- When the scorer says “grounded” - is it actually right?93.2%vs55.7%
- What fraction of hallucinations get approved?5.5%vs78.2%
- Can you show an auditor why a verdict was made?Yes (per-claim trail)
- How much does scoring add to your SDK roundtrip?<1 ms
- Will your scoring bill be the same next month?Yes (no judge-model surcharge)
The metrics clinical and compliance teams audit against — variA/Bly leads every one of them on this benchmark while holding the false-positive rate at 5.5%.
Methodology
Same data into every scorer. Reproducible scripts.
- Dataset: PubMedQA (Jin et al., 2019) — a public, expert-annotated biomedical Q&A dataset with a PubMed abstract as the source for each question. pubmedqa.github.io
- Sample count: 1,000 samples × 2 test cases each = 2000 evaluations per scorer. Each sample produces a positive case (a response faithfully grounded in the abstract) and a negative case (the same response style with a subtle, clinically plausible but unsupported claim).
- Isolating scorer behaviour: the response style is held constant across the pair — only the grounding of the claim changes. This measures scorer behaviour, not response quality, and is deliberately adversarial: the negative case is designed to look grounded.
- Competitors: DeepEval's grounding metrics (Faithfulness and Hallucination) and RAGAS Faithfulness, each with its default LLM judge. The headline “DeepEval” figures use its Hallucination metric — its strongest on this set.
- Threshold: 0.5 for precision / recall / FPR, applied uniformly to every scorer. AUC is threshold-independent.
Get the whitepaper
Three editions, same numbers.
Whitepaper
~5 page whitepaper · headline numbers · methodology · cost analysis. Best place to start.
Technical edition
Methodology · confusion matrices · reproduction scripts. For CTOs / ML platform leads.
Business edition
Plain English · clinical-risk math at scale. For ops directors / compliance leads.
Verify the math (no account, no cost)
Our per-sample scores from this run are committed to the public benchmark repo as JSON. To recompute the headline AUC / precision / recall / FPR straight from those raw scores — no API key, no rescoring needed:
git clone https://github.com/varia-bly/variably-benchmark cd variably-benchmark/pubmedqa python3 compare.py
This reads the per-sample JSON in pubmedqa/results/ and prints the same comparison table at the top of this page.
Re-run the scorers from scratch
The AI-judge sides (DeepEval, RAGAS) are self-contained with an OpenAI key. The variA/Bly side requires a free API key from variably.dev because the scoring runs on variA/Bly's hosted infrastructure. Step-by-step commands: pubmedqa/README.md .