All docs
Evaluation policyData definitionsExpected answersdashboard

Expected answers

Expected answers power "Matches the expected answer" and "Answers what was asked": values compared in code, with insurance and data-validation examples and how dates, rounding and currencies are handled.

What you'll have

Every answer in a dataset run is compared with the expected one value by value: a wrong benefit %, a date a day off or a count that differs fails.

What it is and what it powers

There is nothing to set in the policy. You send expected_answer= (and optionally expected_answer_kind=) with an answer, usually from a dataset item:

  • Matches the expected answer checks two things.

    • What it says: each sentence of the expected answer is compared with the answer by a model that reads meaning: does the answer say the same, the opposite, or neither? "Approved" against an expected "denied", "yes" against "no", or the right values attached to the wrong facts is a contradiction. The model reads; the verdict rules are fixed, and the same input always gets the same verdict.
    • The values: amounts with their currency, plain numbers with their unit ("60 days" and "60-day" are the same), percentages and dates are read in code and compared at the precision they are written in.

    Fail when a sentence is contradicted or a value differs, pass when every sentence is said and no value differs, warning when part of it is not stated and nothing contradicts it, N/A when the answer says none of it. Not stated never passes.

  • Answers what was asked checks the answer gives the kind of value the question wants: amount, quantity, percent, date, duration, count, yes_no, list, procedure, explanation or other.

A mismatch with the expected answer is reported apart from grounding: an answer can be faithful to its documents and still differ from what you expected.

When it's not set. N/A: "No expected answer was sent (expected_answer=), or it and the answer state no comparable value."

Use cases

Insurance: benefit %, waiting period and effective date

client.log(
    prompt="What does STD pay, after what waiting period, and from when?",
    response="STD pays 60% of weekly earnings, starting after 7 days, from January 1, 2026.",
    expected_answer="66.67% of weekly earnings, starting after 14 days, effective 2026-01-01.",
    expected_answer_kind="percent",
)

Fails: 60% differs from 66.67%, and 7 days from 14 days. The date matches: January 1, 2026 and 2026-01-01 are the same day.

Structured outputs: counts, dates, percentages

expected_answer="412 open claims, 23.7% approved, as of 2026-03-31" against "There are 412 open claims; 23.74% were approved as of March 31, 2026" passes: 23.74% agrees with 23.7% to the coarser precision.

Data-model validation summaries

An agent that validates a data model against its specification writes a summary: how many tables were checked, how many columns failed, which date the snapshot is from. What you can check today:

  • The figures in the summary: send the expected summary as expected_answer ("48 tables validated, 3 type mismatches, snapshot 2026-03-31"). Each count and date is compared in code.
  • What the summary must mention: a requirement template such as "names every column with a type mismatch" or "states the row count of each failed table".

variA/Bly does not compare two data models with each other or prove them equivalent; it checks the summary's values and coverage.

Edge cases

SituationWhat happens
Dates in different formatsThe same day matches (2026-01-01, January 1, 2026). All-numeric dates such as 01/04/2026 are read only with the policy's date_order (MDY or DMY); without it they are not read, never guessed.
RoundingTwo values agree to the coarser of their written precisions: "$875K" is 875,000 ± 500; 23.7% matches 23.74%.
Different currenciesAn amount in € is never paired with one in $: no conversion.
Different kindsA percentage is never paired with an amount; "14 days" pairs with a number followed by "day".
Several values of one kindWhen each expected value has exactly one candidate, code pairs them; otherwise, with Max-Accuracy model reads on, the reasoning model says which answer value corresponds to which (a pointer, never a verdict) and code decides. Without it, N/A.
An expected value the answer does not stateWarning, not a failure.
Over 20,000 charactersRefused when sent: "expected_answer: at most 20000 characters".

How to add it

Data definitions: Expected answersData definitions: Expected answers

  1. Store the expected answer on each dataset item, or have it in your test code.
  2. Send it with the answer: client.log(..., expected_answer="...", expected_answer_kind="amount").
  3. For all-numeric dates, set date_order in the policy YAML (Data definitions).
  4. The verdict appears under Matches the expected answer on the answer and in the dataset run.

What you'll see

Expected answerThe answer saysMatches the expected answer
"The waiting period is 14 days.""Benefits start after a 14-day waiting period."Pass
"The waiting period is 14 days.""Benefits start after a 7-day waiting period."Fail: "expected 14 days, the answer says 7 days"
"60% of salary, from 2026-04-01.""You'll receive 60% of your salary."Warning: "the answer does not state 2026-04-01"

The verdict is its own check: it never changes faithfulness or the overall score.

Common mistakes

MistakeWhat happensFix
An expected answer with several possible values ("$1,500 or $3,000")Neither can be matched cleanlyOne clear value per fact
Dates written 04/03/2026 with no date orderThey are not readWrite 2026-04-03, or set the policy's date_order
Sending the expected answer on live traffic you don't know the answer toNothing to compareSend it on dataset runs and test traffic

Format reference

ParameterTypeRequiredAllowed valuesMeaning
expected_answertextnoup to 20,000 charactersThe answer you expected; its values are compared
expected_answer_kindtextnoamount, quantity, percent, date, duration, count, yes_no, list, procedure, explanation, otherThe kind of value the question wants

A wrong kind is refused when sent: expected_answer_kind "money": one of amount, quantity, percent, date, duration, count, yes_no, list, procedure, explanation, other.


Concepts on this page: Data definitions