Read a claim's verdict
What each claim and figure shows — the explanation, the checks and where to correct — with worked examples, including SQL totals, averages, counts and shares checked against the returned rows.
You can read any failed claim and know what to fix: the two values that disagree, what the sources are missing, or the computation that checked a figure.
- Answers sent with their sources (reference_materials) or query rows
How it works
- The answer
- Its sources, with readable titles
- Query rows for SQL answers
- Read with its document title
- Figures compared by value
- SQL figures computed from rows
- A plain explanation
- Checks that failed
- Where to correct
Every claim in an answer gets a verdict, and every claim that did not pass gets one plain sentence saying why. This page shows exactly what you will see — the explanation, the checks and where to correct — for the cases that come up most, including answers built from SQL results.
What changed (16 Nov 2026):
- Safety reports harm only. A wrong fact is reported by Grounding and No contradictions, not Safety.
- Plain explanations. A contradicted claim shows both values; an unsupported claim says what is missing and is never called wrong.
- Where to correct. When a source you supplied contradicts the answer, the answer step is named, not retrieval.
- Documents read with their title. A passage is read with its document's title or source as context.
- SQL answers. Totals, breakdowns, averages, counts and shares are computed from the returned rows, and the computation is shown.
Past answers keep their verdicts; re-score them to apply the new behaviour.
How a claim is checked
- The answer is split into claims — one fact each.
- Each claim is compared with the sources sent with the answer (
reference_materials, tool results and query rows). Each passage is read together with its document's title orsource. - Figures — amounts, dates, percentages, counts — are compared by value: "$6,000", "USD 6,000" and "6000 dollars" are one amount.
- Each claim gets a verdict and, when it did not pass, one sentence explaining why.
- The checks follow from the claims, and where to correct names the step to fix.
Facts the user stated
An answer often repeats what the user said about their own case: "you bought it 45 days ago", "order R-7421". When a claim is not supported by the sources but repeats a value from the question that no source holds, it is read again with the question as evidence, marked from what the user stated:
- a claim of fact that the sources and the question together support passes ("Order R-7421 was bought 45 days ago, past the 30-day refund window");
- a conclusion resting on those facts ("The refund is denied because the purchase was 45 days ago and the window is 30 days") is not counted by grounding, as any conclusion: the decision itself is checked by Matches the expected answer and your business rules;
- the question never overrides the sources: a claim the documents contradict still fails ("refunds are allowed for 90 days" against a 30-day policy), and a value nobody stated still fails.

In the event details, such a claim shows Follows from verified findings and From what the user stated, with the document text that verified its facts; it has no score of its own.
Such a conclusion still counts as using its sources: Context utilization credits the documents that verified its facts. When no claim of an answer is counted, Attribution accuracy and Context utilization have nothing to measure and are not shown, rather than shown as 0.
A step that names its documents
A retrieval step often answers with the document it found: "Retrieved document: Standard Refund Policy 2026". When a claim only names a document the step was given, by its title or id, it is supported by that document and marked names a document it was given. The sentence decides it: the document's name must not be what the sentence says something about, and nothing else may be claimed. "The Standard Refund Policy 2026 allows refunds for 90 days" uses the name to state a fact, and is checked against the document like any claim.

A step whose output only names its documents is no answer to the user's question, so relevance, completeness, intent alignment and similarity to the question do not apply to it: they are not shown and do not lower its score. A correct retrieval step scores on what applies. In a multi-agent run, the final answer counts as using such a step when it draws on the step's documents.
The examples were sent like this
import os
from variably import VariablyClient
client = VariablyClient({"api_key": os.environ["VARIABLY_API_KEY"], # the project's key
"base_url": os.environ["VARIABLY_BASE_URL"]}) # https://api.variably.dev
sources = [{
"id": "sob-2026-p2",
"source": "Summary of Benefits 2026, p.2", # named in explanations: make it readable
"content": "Gold PPO 2026. In-network out-of-pocket maximum: $6,000 individual / $12,000 family. "
"Deductible: $1,500 individual.",
}]
result = client.log(
"What is the in-network out-of-pocket maximum for an individual on Gold PPO in 2026?",
"For 2026 the Gold PPO in-network out-of-pocket maximum is $6,500 for an individual.",
reference_materials=sources,
agent_name="benefits-agent",
session_id="chat-4821",
)
print(result.observation_id, result.status) # queued; scored a few seconds later
observe() takes the same fields.
Example 1: a contradicted figure
The answer says $6,500; the source says $6,000.
| Where | What you see |
|---|---|
| The claim | Wrong (contradicted) — "The answer says $6,500; Summary of Benefits 2026, p.2 says $6,000." |
| Checks | Grounding: fail · No contradictions: fail · Safety: pass |
| Where to correct | benefits-agent, writing the answer — "the right reference was retrieved (relevance 0.82) but the answer states different numbers" |
The source holds the right figure, so retrieval worked and the fix is in writing the answer: the
prompt, the model, or a step that rewrites the figure. That holds even when the sources' overall
relevance is low; the reason then reads "a retrieved reference states the fact (overall relevance
0.41) but the answer states different numbers". The name in the sentence is the document's source
(else its id), so give documents readable names.
Example 2: an unsupported figure
Same source. The answer adds "The plan's lifetime maximum is $900,000." Nothing in the sources gives a lifetime maximum.
| Where | What you see |
|---|---|
| The claim | Not in the sources — "The sources don't give this figure ($900,000)." |
| Checks | Grounding: fail under Strict, a warning under Document review · No contradictions: pass · Safety: pass |
| Where to correct, sources relevant to the question | benefits-agent, writing the answer — "the right reference was retrieved (relevance 0.78) but the answer says things it doesn't contain" |
| Where to correct, sources off-topic | retrieval for benefits-agent — "the retrieved references don't match the question (relevance 0.31), so the answer had little to stand on" |
An unsupported claim without a figure reads "The sources don't state this." It is never called wrong: it may be true, but nothing you sent shows it. Fix it by retrieving the document that holds the fact, or by telling the model to stay within the sources.
Example 3: a correct answer read with the document's title
client.log(
"When do Acme's short-term disability benefits start?",
"Acme's short-term disability benefits start after a 7-day elimination period and pay 60% of weekly earnings.",
reference_materials=[{
"id": "std-2026-s3",
"source": "Acme Short-Term Disability (STD) Plan 2026, section 3",
"content": "Benefits begin after a 7-day elimination period. The weekly benefit is 60% of weekly earnings.",
}],
agent_name="benefits-agent")
Both claims are Supported, and Grounding and No contradictions pass. The passage never says "short-term disability" or "Acme" — the title does — and the passage is read with its title as context. The figures (7 days, 60%) must still be written in the passage itself: a title never supplies a figure. Before 16 Nov 2026 this answer could show Not in the sources.
Safety reports harm only
Safety covers toxicity, bias, harmful content, privacy, inappropriate language, violence and adult content. A wrong fact is a grounding problem:
| The answer | Grounding | No contradictions | Safety |
|---|---|---|---|
| States $6,500 where the source says $6,000 | fail | fail | not affected |
| Adds a figure the sources don't give | fail (Strict) or warning | pass | not affected |
| Contains an insult or personal data | as its claims | as its claims | fail when over the limit |
Misinformation still counts toward Safety for an answer sent without any sources, where there is nothing to ground it against.
Where to correct, at a glance
| What the claims show | Where to correct names | Why |
|---|---|---|
| A supplied source contradicts a figure or fact | The answer step (agent, writing the answer) | The right fact was retrieved |
| Claims not in the sources, sources relevant to the question | The answer step | The answer added things it was not given |
| Claims not in the sources, sources not relevant (relevance under 0.5) | Retrieval (retrieval for agent) | The fact was not retrieved |
| A wrong figure against query rows or the answer's own tool results | The answer step | The answer misread its own data |
| A figure from the wrong document, a wrong calculation | The answer step | The answer's own work is wrong |
SQL answers: figures checked against the returned rows
Send the rows
Send the rows your query returned as a reference with rows (a list of flat records), together with
the SQL. Database values — decimals, dates, driver row objects — are converted for you; a row without
column names is refused at the call. Up to 10,000 rows are read, all of them.
sql = """SELECT region, SUM(paid_amount) AS paid, COUNT(*) AS claims
FROM claims WHERE year = 2026 GROUP BY region"""
rows = cursor.execute(sql).fetchall() # e.g. [{"region": "North", "paid": 425000.50, "claims": 40}, ...]
client.log(
"How much did we pay in claims in 2026, by region?",
answer_text,
sql=sql,
reference_materials=[{"id": "claims_by_region", "source": "claims_by_region query", "rows": rows}],
agent_name="sql-analyst")
The rows in the examples:
| region | paid | claims |
|---|---|---|
| North | 425000.50 | 40 |
| South | 300000 | 35 |
| West | 150000 | 25 |
What each kind of figure shows
Each figure passes only when exactly one computation over the rows gives it, to the precision it is written in. The computation is shown with the claim.
| The answer says | Verdict | Shown with the claim |
|---|---|---|
| Total paid was $875,000.50. | Supported | SUM(paid) over 3 rows = 875,000.50 (the only computation over the rows that gives it) |
| North paid $425,000.50 and South paid $300,000. | Supported (breakdown) | North's paid = 425,000.50 (the only computation over the rows that gives it); South's paid = 300,000 (the only computation over the rows that gives it) |
| Average paid was $291,666.83. | Supported | AVG(paid) over 3 rows = 291,666.83 (the only computation over the rows that gives it) |
| Claims came from 3 regions. | Supported | COUNT(*) over 3 rows = 3 (the only computation over the rows that gives it) |
| Total claims were 100. | Supported | SUM(claims) over 3 rows = 100 (the only computation over the rows that gives it) |
| North accounted for 48.57% of paid. | Supported | share of paid: 425,000.50 of 875,000.50 = 48.57 (the only computation over the rows that gives it) |
| North paid $300,000. | Wrong | $300,000 is South's paid; North's paid is 425,000.50, not $300,000 — explained as "The answer says $300,000; claims_by_region gives North's paid as 425,000.50." |
| Total paid was $900,000. | Not in the sources | No computation over the rows gives it — "The sources don't give this figure ($900,000)." |
| East paid $200,000. | Not in the sources | The results hold no East row; a figure is never passed on another row's value |
The wording around a figure is read too: "average paid" must be an average, not a total that happens to match. A wrong figure against the rows is the answer step's to correct: the query returned the right data and the answer misread it.
An ambiguous figure is left unverified
With rows North (paid 60, claims 40), South (90, 35) and West (150, 25), the answer "Paid claims came to 100." matches two computations:
100: 2 computations over the rows give 100 (SUM(claims) over 3 rows; AVG(paid) over 3 rows): ambiguous
The rows never pass or fail it on a guess, and the standard checks do not count it as supported. Under Max-Accuracy, the reasoning model reads which column and operation the sentence means, and code then computes and decides. To avoid ambiguity, name the column and the operation in the answer ("Total claims were 100").
Limits
- A percentage is compared only with a percentage: a share, a column whose cells are written with "%", or a column whose unit you declare in the project's SQL checks as a fraction (0.2374 is 23.74%) or a percent (23.74 is 23.74%). Against an undeclared plain number it is left unverified.
- In a sentence with two figures, code decides the ones it can point at; the rest are left to Max-Accuracy.
- A row that totals the others (a ROLLUP row) is left out of computed totals.
- variA/Bly checks the SQL text and rows you send. It never connects to your database.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
| A correct SQL total is Not in the sources | The rows were not sent, or the answer names no column of the rows | Send rows with the answer; name the column ("total paid", not "the total") |
| A figure left unverified as ambiguous | Two computations over the rows give the same number | Name the column and operation, or use Max-Accuracy for that traffic |
| A wrong figure did not lower Safety | Safety reports harm only | Track factual errors with Grounding and No contradictions |
The explanation names a source like doc-17 | The document was sent without a source | Send a readable source (title and page) with each document |
| Where to correct says retrieval, but the document was retrieved | The sources as a whole don't match the question (relevance under 0.5), and no source contradicts the claim | Retrieve fewer, more relevant passages; check the passage with the fact was among them |
FAQ
Do these changes re-score my old answers? No. Past answers keep their verdicts until you re-score them.
Why isn't a claim that is not in the sources called wrong? Nothing you sent proves it wrong. It lowers the grounding score and, under Strict, fails Grounding; it is not misinformation.
Is the SQL check a model? No. Code computes the cell, total, count, average or share from your rows. A model is used only under Max-Accuracy, to say which column and operation an ambiguous sentence means; code still decides.
Concepts on this page: Claims, grounding and where to correct · How answers are judged