Expected answers
Expected answers power "Matches the expected answer" and "Answers what was asked": values compared in code, with insurance and data-validation examples and how dates, rounding and currencies are handled.
Every answer in a dataset run is compared with the expected one value by value: a wrong benefit %, a date a day off or a count that differs fails.
What it is and what it powers
There is nothing to set in the policy. You send expected_answer= (and optionally expected_answer_kind=) with an answer, usually from a dataset item:
-
Matches the expected answer checks two things.
- What it says: each sentence of the expected answer is compared with the answer by a model that reads meaning: does the answer say the same, the opposite, or neither? "Approved" against an expected "denied", "yes" against "no", or the right values attached to the wrong facts is a contradiction. The model reads; the verdict rules are fixed, and the same input always gets the same verdict.
- The values: amounts with their currency, plain numbers with their unit ("60 days" and "60-day" are the same), percentages and dates are read in code and compared at the precision they are written in.
Fail when a sentence is contradicted or a value differs, pass when every sentence is said and no value differs, warning when part of it is not stated and nothing contradicts it, N/A when the answer says none of it. Not stated never passes.
-
Answers what was asked checks the answer gives the kind of value the question wants:
amount,quantity,percent,date,duration,count,yes_no,list,procedure,explanationorother.
A mismatch with the expected answer is reported apart from grounding: an answer can be faithful to its documents and still differ from what you expected.
When it's not set. N/A: "No expected answer was sent (expected_answer=), or it and the answer state no comparable value."
Use cases
Insurance: benefit %, waiting period and effective date
client.log(
prompt="What does STD pay, after what waiting period, and from when?",
response="STD pays 60% of weekly earnings, starting after 7 days, from January 1, 2026.",
expected_answer="66.67% of weekly earnings, starting after 14 days, effective 2026-01-01.",
expected_answer_kind="percent",
)
Fails: 60% differs from 66.67%, and 7 days from 14 days. The date matches: January 1, 2026 and 2026-01-01 are the same day.
Structured outputs: counts, dates, percentages
expected_answer="412 open claims, 23.7% approved, as of 2026-03-31" against "There are 412 open claims; 23.74% were approved as of March 31, 2026" passes: 23.74% agrees with 23.7% to the coarser precision.
Data-model validation summaries
An agent that validates a data model against its specification writes a summary: how many tables were checked, how many columns failed, which date the snapshot is from. What you can check today:
- The figures in the summary: send the expected summary as
expected_answer("48 tables validated, 3 type mismatches, snapshot 2026-03-31"). Each count and date is compared in code. - What the summary must mention: a requirement template such as "names every column with a type mismatch" or "states the row count of each failed table".
variA/Bly does not compare two data models with each other or prove them equivalent; it checks the summary's values and coverage.
Edge cases
| Situation | What happens |
|---|---|
| Dates in different formats | The same day matches (2026-01-01, January 1, 2026). All-numeric dates such as 01/04/2026 are read only with the policy's date_order (MDY or DMY); without it they are not read, never guessed. |
| Rounding | Two values agree to the coarser of their written precisions: "$875K" is 875,000 ± 500; 23.7% matches 23.74%. |
| Different currencies | An amount in € is never paired with one in $: no conversion. |
| Different kinds | A percentage is never paired with an amount; "14 days" pairs with a number followed by "day". |
| Several values of one kind | When each expected value has exactly one candidate, code pairs them; otherwise, with Max-Accuracy model reads on, the reasoning model says which answer value corresponds to which (a pointer, never a verdict) and code decides. Without it, N/A. |
| An expected value the answer does not state | Warning, not a failure. |
| Over 20,000 characters | Refused when sent: "expected_answer: at most 20000 characters". |
How to add it

- Store the expected answer on each dataset item, or have it in your test code.
- Send it with the answer:
client.log(..., expected_answer="...", expected_answer_kind="amount"). - For all-numeric dates, set
date_orderin the policy YAML (Data definitions). - The verdict appears under Matches the expected answer on the answer and in the dataset run.
What you'll see
| Expected answer | The answer says | Matches the expected answer |
|---|---|---|
| "The waiting period is 14 days." | "Benefits start after a 14-day waiting period." | Pass |
| "The waiting period is 14 days." | "Benefits start after a 7-day waiting period." | Fail: "expected 14 days, the answer says 7 days" |
| "60% of salary, from 2026-04-01." | "You'll receive 60% of your salary." | Warning: "the answer does not state 2026-04-01" |
The verdict is its own check: it never changes faithfulness or the overall score.
Common mistakes
| Mistake | What happens | Fix |
|---|---|---|
| An expected answer with several possible values ("$1,500 or $3,000") | Neither can be matched cleanly | One clear value per fact |
| Dates written 04/03/2026 with no date order | They are not read | Write 2026-04-03, or set the policy's date_order |
| Sending the expected answer on live traffic you don't know the answer to | Nothing to compare | Send it on dataset runs and test traffic |
Format reference
| Parameter | Type | Required | Allowed values | Meaning |
|---|---|---|---|---|
expected_answer | text | no | up to 20,000 characters | The answer you expected; its values are compared |
expected_answer_kind | text | no | amount, quantity, percent, date, duration, count, yes_no, list, procedure, explanation, other | The kind of value the question wants |
A wrong kind is refused when sent: expected_answer_kind "money": one of amount, quantity, percent, date, duration, count, yes_no, list, procedure, explanation, other.
Concepts on this page: Data definitions