Deterministic AI Evals: When a Test Library Is Enough — and When It Isn't

"Deterministic evals" means two different things, and most lists of deterministic eval tools only cover one of them. Here is how to tell which one you need — and why most teams shipping RAG, chat or agents end up needing both.

Ask for a deterministic AI evaluation tool and you will usually get a list of test libraries: run an assertion in CI, get the same pass or fail every time. That is real, useful, and exactly right for a large class of checks. It is also not the problem most teams are trying to solve when their answers are open-ended.

Two kinds of deterministic evals

1. Deterministic checks on closed-form outputs. The answer can be compared to a rule in code: an exact value, a label from a fixed set, a JSON schema, a regex, a tool call with the expected arguments, a test that passes. A test library run in CI is the right tool here. No model is needed, the verdict is obviously repeatable, and it costs nothing to run.

2. Deterministic verification of open-ended answers. The answer is prose — a RAG response, a support reply, an agent's summary — and the question is whether what it says is supported by what it was given. "Is every claim in this answer backed by the retrieved passages?" cannot be written as a regex. The usual way to answer it is to ask another LLM, which gives an opinion that can change between runs and judge versions. The deterministic way is to break the answer into claims and check each one against the evidence with fixed models and code, so the same answer gets the same verdict every time.

Most "deterministic eval tool" lists are lists of the first kind. If your outputs are open-ended, you need the second.

When a test library is enough

• Your outputs are structured: classifications, extractions into a schema, routing decisions, tool-call payloads.
• You have a golden dataset with exact expected answers.
• You want a fast gate in CI on every commit.

Keep those checks. They are cheap, fast and unambiguous.

Where it stops

• Open-ended answers. A support answer can be fluent, on topic and wrong about one price. Only a check against the source catches it: "Premium costs ¥1,000 per month" against a page that says ¥1,200 is wrong, however it is phrased.
• Production traffic, not just goldens. Real questions do not come with expected answers. What you can check is whether the answer is supported by the passages, documents or tool results the system actually used.
• Agents. Whether an agent's final message reads well says little. What matters is whether it called the right tools, wasted calls on repeats and failures, looped, or ignored what a sub-agent found — checks on the trace, not the text.
• Explaining a failure. A failed assertion tells you something broke. A claim-level verdict tells you which sentence, which passage and which step to fix.

What a deterministic evaluation platform adds

• A verdict on every claim — supported, not in the sources, or wrong — with the passage that decides it.
• Figures compared by value. Dates, amounts and counts match however they are written.
• Agent checks on tool correctness, wasted calls, loops and allowed tools.
• Repeatability you can see. Every verdict records the scorer release that produced it, so a drop in score is a real regression, not evaluator noise.
• The same checks on goldens and on production, plus experiments to compare a prompt or model change.

Use both

Run your test library in CI for the closed-form checks. Run claim-level, deterministic verification on your goldens and your production traffic for everything open-ended. One catches the broken JSON; the other catches the wrong price.

Adding the second takes three lines:

# pip install variably-sdk
from variably import observe

observe(
    prompt=user_query,
    response=llm_answer,
    reference_materials=[{"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"}],
)

variA/Bly is a deterministic AI evaluation platform for LLMs, RAG and agents. Read what a deterministic AI evaluation tool checks, or start with the docs.