Deterministic AI evaluation tool for LLMs, RAG and agents
Deterministic AI eval means the same answer, checked against the same sources, gets the same verdict every time — because no LLM grades it. variA/Bly is a deterministic AI evaluation platform: it checks every claim against your documents, retrieved passages and agent tool calls, and shows the evidence behind each verdict.
Deterministic evals vs LLM-as-judge: which AI evals answer which question
| Deterministic evaluation | LLM-as-judge | |
|---|---|---|
| Same answer, same verdict | Yes — fixed models and code | No — the verdict can change between runs and judge versions |
| Shows why | The claim, the passage and the figure that decided it | A rationale the judge writes |
| Cost per evaluation | Predictable — no judge tokens on your bill | Judge tokens on every verdict |
| Regression testing | A score drop is a real change | A drop may be judge noise |
| Open-ended qualities (tone, style) | Not its job | What a judge is for |
An eval tool for everything that can be checked against evidence
No LLM in the grading path: claims are read by fixed models, and code decides every verdict.
Claims against sources
Every claim gets one of three verdicts — supported, not in the sources, or wrong — with the passage that decides it.
Figures by value
Dates, amounts and counts are compared by value, so “¥1,000” against a source saying “¥1,200” is wrong, however it is written.
Agent runs
Tool correctness, wasted and failed calls, loops, allowed tools, and whether the final answer uses what the sub-agents found.
Safety and quality
The same risk limits and quality floor on every answer, with the step to fix when a check fails.
More: RAG evaluation · all features · public benchmarks · docs
AI evals and deterministic evaluation, answered
What are AI evals?
AI evals (LLM evals) are tests that score what an AI system produces — whether its answers are grounded in their sources, correct, safe and on task, and for agents whether they used their tools well. Run on every change and on production traffic, they tell you whether a new prompt, model or retriever made answers better or worse.
What are deterministic evals?
Deterministic evals give the same verdict for the same input every time. They check an answer against evidence with fixed models and code rather than asking an LLM judge, so a change in score always comes from a change in the answer, not from the evaluator.
What is a deterministic AI evaluation tool?
A deterministic AI evaluation tool scores an AI answer with fixed models and code instead of asking another LLM for its opinion, so the same answer checked against the same sources gets the same verdict on every run. variA/Bly breaks each answer into claims and checks every claim against the documents, retrieved passages or tool results it was given: supported, not in the sources, or wrong.
Can you evaluate LLM outputs without an LLM judge?
Yes, for anything that can be checked against evidence: whether a claim is supported by its sources, whether a figure or date matches, whether a calculation holds, whether an agent called the right tools, wasted calls or looped, and whether a safety limit was crossed. variA/Bly does all of these without an LLM in the grading path.
What is the difference between deterministic evals and LLM-as-judge?
An LLM judge reads the answer and gives an opinion; the verdict can change between runs and with the judge model, and each verdict costs judge tokens. A deterministic eval gives a repeatable verdict with the evidence behind it — the claim, the passage and the figure that decided it — so a drop in score is a real regression, not judge noise.
When should I still use an LLM judge?
For open-ended qualities with no evidence to check against, such as tone or style, and for claims that need reasoning a small model cannot do. variA/Bly’s Max-Accuracy mode sends only those uncertain claims to a local judge read at temperature 0 from token probabilities, so even escalated verdicts stay repeatable.
Does a deterministic eval tool work for RAG and AI agents?
Yes. For RAG, every claim is checked against the retrieved passages, with figures compared by value. For agents, runs are checked for tool correctness, step efficiency (failed, repeated and unnecessary calls), loops, allowed tools, and agreement between agents in a multi-agent session.
How do I know a score is reproducible?
Every verdict records the scorer release that produced it. The dashboard says “Repeatable” with that release, and warns when a report or a comparison mixes releases — then a change may come from the scorer, not your prompt.
Try a deterministic eval on your own traffic
One SDK call per answer. Verdicts with their evidence, the same on every run.