LLM-judge accuracy. Without the LLM judge.
Deterministic and reproducible — at about two-thirds lower cost, with no added latency on your request path.
On 592 RAG answers from the public RGB benchmark, variA/Bly's deterministic grounding scorer matches the accuracy of RAGAS — out of the box — while giving the same verdict on every run, scoring every case and adding no judge tokens. Same cases for every scorer, per-case results public: recompute every number yourself.
What this means
The same accuracy — reached by default, with no judge to tune.
Accuracy (AUC)
0.851 vs 0.852 RAGAS
The same ability to tell supported answers from unsupported ones — with fixed models and code instead of an LLM judge.
Out of the box
97.6% · 27.7%
Correct answers and distractor cases approved at variA/Bly's default. RAGAS reaches the same point (96.6% · 27.0%) only with a threshold tuned on labelled data.
Every case, every run
592/592
A verdict for every case, the same on every rerun. DeepEval's Faithfulness metric returned nothing for 298 of the 592.
| Operating point | Correct approved | Distractors approved | Precision |
|---|---|---|---|
| variA/Bly 4.2.1 — default, no tuning | 97.6% | 27.7% | 77.9% |
| RAGAS — threshold tuned on labelled data (0.6) | 96.6% | 27.0% | 78.1% |
| RAGAS — default threshold (0.5) | 97.0% | 38.2% | 71.8% |
About RGB's labels. A distractor case pairs the same correct answer with passages that are supposed not to contain it — but many of RGB's distractor passages contain the answer anyway. Approving those is correct, yet it counts as a false approval, so every scorer's distractor rate is overstated. Compare scorers on the same cases.
Every scorer, the same 592 cases, each at its default threshold
| Scorer | Cases scored | Correct approved | Distractors approved | Precision | AUC |
|---|---|---|---|---|---|
| variA/Bly 4.2.1 (deterministic) | 592/592 | 97.6% | 27.7% | 77.9% | 0.851 |
| RAGAS faithfulness (gpt-4o-mini judge) | 592/592 | 97.0% | 38.2% | 71.8% | 0.852 |
| DeepEval Hallucination (gpt-4o-mini judge) | 592/592 | 42.9% | 4.4% | 90.7% | 0.852 |
| DeepEval Faithfulness (gpt-4o-mini judge) | 294/592 | 52.4% | 38.5% | 27.0% | 0.571 |
variA/Bly 4.2.1 scored on October 24, 2026 through its full production path. RAGAS and DeepEval scores are from April 2026 on the identical answer-and-passage pairs. DeepEval Faithfulness returned no score for 298 cases (its judge's output overran the model's length limit); its precision is on the 294 it scored. DeepEval Hallucination is the strictest scorer — fewest false approvals, but 57% of correct answers rejected.
What changed since April 2026. The April edition reported a 6.1% false-approval rate from an earlier scorer that was strict by design and rejected 46% of correct answers. Today's scorer approves 97.6% of correct answers at LLM-judge accuracy; this edition uses one run per scorer, joined case by case.
A real disagreement from the benchmark
Look at one example side-by-side.
Distractor case rgb_id=70 from the public RGB dataset, scored by the current scorer — a real evaluation from the run, not a hand-picked illustration.
Query
What was Tesla's revenue in Q1 2022?
AI response
Tesla's revenue in Q1 2022 was $18.76 billion.
References (excerpts from the five distractor passages)
“The Wall Street consensus for this quarter was $17.659 billion in revenue…” “Automotive revenue reached $16.86 billion, up 87%…” “…about $17.5 billion in cash at the end of the last quarter.”
How each scorer rated this response:
RAGAS
✓ Grounded
1.00
The passages are all about Tesla's Q1 2022 results, so the LLM judge found the answer close enough. None of them states total revenue.
variA/Bly
✗ Wrong
0.00
The figure in the claim ($18.76 billion) is in no passage. The passages give a revenue consensus, automotive revenue and cash — different figures. The figure check decides it, not a judgment of topic.
Why variA/Bly said no - the audit trail
RAGAS gives you a score. variA/Bly gives you the claim, the verdict and the passage behind it — the same on every rerun. That's what makes the verdict auditable.
At each tool's default threshold
Distractor cases approved
How often each scorer wrongly approves a passage that doesn't support the AI's claim. Lower is better - every false approval is a wrong response auto-passed to the customer.
At its default, RAGAS approves more distractor cases; tuned to a threshold of 0.6 it approves 27.0% — the same as variA/Bly's default. Both rates are overstated by RGB distractor passages that contain the answer.
Reading the numbers honestly
Equal accuracy. No tuning, no judge.
On overall ranking quality (AUC) the two are equal — 0.851 for variA/Bly and 0.852 for RAGAS — and at a matched operating point they make the same decisions: variA/Bly approves 97.6% of correct answers and 27.7% of distractor cases at its default; RAGAS approves 96.6% and 27.0% with its threshold tuned to 0.6.
The difference is how each gets there. variA/Bly reaches that accuracy out of the box, with fixed models and code — the same verdict on every run, with the claim and passage behind it. RAGAS needs a threshold tuned on labelled data, and its verdicts come from an LLM judge that costs tokens and can change between runs and judge versions.
What variA/Bly adds at the same accuracy
- Same input twice - does the verdict change?No (deterministic)
- Does every case get a verdict?Yes - 592 of 592
- Can you show an auditor why a verdict was made?Yes (claim + passage)
- Are dates and amounts compared by value?Yes
- Is there a threshold to tune to reach this accuracy?No - default
- How much does scoring add to your SDK roundtrip?<1 ms
- Will your scoring bill be the same next month?Yes (no judge tokens)
What compliance teams audit against — at LLM-judge accuracy, with the per-case results public.
How to read this
What the difference is.
variA/Bly - deterministic, claim by claim.
Each claim is checked against the passages with fixed models and code, and figures are compared by value: supported, not in the sources, or wrong. 97.6% of correct answers approved and 27.7% of distractor cases at its default. The same verdict on every run, with the evidence behind it — built for regulated, customer-facing AI.
RAGAS - LLM-as-judge.
An LLM reads the answer and the passages and judges whether the statements can be inferred. The same accuracy once its threshold is tuned (96.6% / 27.0%); at its default, 38.2% of distractor cases approved — passages on the right topic are often judged close enough. Each verdict costs judge tokens and can change between runs.
Methodology
Same data into both scorers. Reproducible scripts.
- Dataset: RGB (Retrieval-augmented Generation Benchmark, Chen et al. 2023) - public, labeled, cited by competitors. github.com/chen700564/RGB
- Sample count: 296 RGB samples × 2 test cases each = 592 evaluations per scorer. Each sample produces a positive case (response paired with reference passages that contain the answer) and a negative case (paired with distractor passages that don't).
- Realistic responses: generated with gpt-4o-mini from the positive chunks, then used unchanged in both positive and negative cases. The same response sees the two reference sets - only the references swap. This isolates scorer behaviour from response quality.
- Scorers: variA/Bly 4.2.1, scored on October 24, 2026 through its full production path; RAGAS faithfulness and DeepEval (Faithfulness and Hallucination) with a gpt-4o-mini judge, scored in April 2026 on the identical answer-and-passage pairs. RAGAS cost across the run: ~$17.76 (~$0.030 per evaluation).
- Threshold: 0.5 for the precision / recall / FPR / FNR numbers. AUC is threshold-independent.
Get the whitepaper
Three editions, same numbers.
Whitepaper
~5 page whitepaper · headline numbers · methodology · cost analysis. Best place to start.
Technical edition
25 pages · methodology · error-mode breakdown · reproduction scripts. For CTOs / ML platform leads.
Business edition
5-7 pages · plain English · compliance-breach math at scale. For ops directors / compliance leads.
Verify the math (no account, no cost)
The per-case scores of every scorer — one row per case — are committed to the public benchmark repo. To recompute the table at the top of this page straight from them, with no API key and no rescoring:
git clone https://github.com/varia-bly/variably-benchmark cd variably-benchmark/rgb python3 verify_2026_10.py
This reads the per-sample JSON in rgb/results/ and prints the same four-scorer table as the top of this page. If hand-calc on the same JSON disagrees with verify_2026_10.py, file an issue - we mean it when we say reproducible.
Re-run the scorers from scratch
Both scorers cost money to run end-to-end and require their own account. The RAGAS side is fully self-contained - just an OpenAI key. The variA/Bly side requires a free API key from variably.dev because the scoring algorithm runs on variA/Bly's hosted infrastructure.
# Re-run RAGAS independently (OpenAI key only): git clone https://github.com/varia-bly/variably-benchmark cd variably-benchmark pip install -r requirements.txt git clone --depth 1 https://github.com/chen700564/RGB rgb/data/RGB export OPENAI_API_KEY=<your key> python3 rgb/run_ragas.py python3 rgb/compare.py
RAGAS run: ~$17.76 (gpt-4o-mini judge × 592 evals), ~28 min wall. variA/Bly rerun + step-by-step commands: rgb/README.md .