All docs
See what's happeningRead a claim's verdictdashboard

Read a claim's verdict

What each claim and figure shows — the explanation, the checks and where to correct — with worked examples, including SQL totals, averages, counts and shares checked against the returned rows.

What you'll have

You can read any failed claim and know what to fix: the two values that disagree, what the sources are missing, or the computation that checked a figure.

You'll need
  • Answers sent with their sources (reference_materials) or query rows

How it works

You send
  1. The answer
  2. Its sources, with readable titles
  3. Query rows for SQL answers
Each claim
  1. Read with its document title
  2. Figures compared by value
  3. SQL figures computed from rows
You see
  1. A plain explanation
  2. Checks that failed
  3. Where to correct

Every claim in an answer gets a verdict, and every claim that did not pass gets one plain sentence saying why. This page shows exactly what you will see — the explanation, the checks and where to correct — for the cases that come up most, including answers built from SQL results.

What changed (16 Nov 2026):

  • Safety reports harm only. A wrong fact is reported by Grounding and No contradictions, not Safety.
  • Plain explanations. A contradicted claim shows both values; an unsupported claim says what is missing and is never called wrong.
  • Where to correct. When a source you supplied contradicts the answer, the answer step is named, not retrieval.
  • Documents read with their title. A passage is read with its document's title or source as context.
  • SQL answers. Totals, breakdowns, averages, counts and shares are computed from the returned rows, and the computation is shown.

Past answers keep their verdicts; re-score them to apply the new behaviour.

How a claim is checked

  1. The answer is split into claims — one fact each.
  2. Each claim is compared with the sources sent with the answer (reference_materials, tool results and query rows). Each passage is read together with its document's title or source.
  3. Figures — amounts, dates, percentages, counts — are compared by value: "$6,000", "USD 6,000" and "6000 dollars" are one amount.
  4. Each claim gets a verdict and, when it did not pass, one sentence explaining why.
  5. The checks follow from the claims, and where to correct names the step to fix.

Facts the user stated

An answer often repeats what the user said about their own case: "you bought it 45 days ago", "order R-7421". When a claim is not supported by the sources but repeats a value from the question that no source holds, it is read again with the question as evidence, marked from what the user stated:

  • a claim of fact that the sources and the question together support passes ("Order R-7421 was bought 45 days ago, past the 30-day refund window");
  • a conclusion resting on those facts ("The refund is denied because the purchase was 45 days ago and the window is 30 days") is not counted by grounding, as any conclusion: the decision itself is checked by Matches the expected answer and your business rules;
  • the question never overrides the sources: a claim the documents contradict still fails ("refunds are allowed for 90 days" against a 30-day policy), and a value nobody stated still fails.

A conclusion that follows from verified findings, using a fact the user statedA conclusion that follows from verified findings, using a fact the user stated

In the event details, such a claim shows Follows from verified findings and From what the user stated, with the document text that verified its facts; it has no score of its own.

Such a conclusion still counts as using its sources: Context utilization credits the documents that verified its facts. When no claim of an answer is counted, Attribution accuracy and Context utilization have nothing to measure and are not shown, rather than shown as 0.

A step that names its documents

A retrieval step often answers with the document it found: "Retrieved document: Standard Refund Policy 2026". When a claim only names a document the step was given, by its title or id, it is supported by that document and marked names a document it was given. The sentence decides it: the document's name must not be what the sentence says something about, and nothing else may be claimed. "The Standard Refund Policy 2026 allows refunds for 90 days" uses the name to state a fact, and is checked against the document like any claim.

A retrieval step's claim, supported because it names a document the step was givenA retrieval step's claim, supported because it names a document the step was given

A step whose output only names its documents is no answer to the user's question, so relevance, completeness, intent alignment and similarity to the question do not apply to it: they are not shown and do not lower its score. A correct retrieval step scores on what applies. In a multi-agent run, the final answer counts as using such a step when it draws on the step's documents.

The examples were sent like this

import os
from variably import VariablyClient

client = VariablyClient({"api_key": os.environ["VARIABLY_API_KEY"],      # the project's key
                         "base_url": os.environ["VARIABLY_BASE_URL"]})    # https://api.variably.dev

sources = [{
    "id": "sob-2026-p2",
    "source": "Summary of Benefits 2026, p.2",      # named in explanations: make it readable
    "content": "Gold PPO 2026. In-network out-of-pocket maximum: $6,000 individual / $12,000 family. "
               "Deductible: $1,500 individual.",
}]

result = client.log(
    "What is the in-network out-of-pocket maximum for an individual on Gold PPO in 2026?",
    "For 2026 the Gold PPO in-network out-of-pocket maximum is $6,500 for an individual.",
    reference_materials=sources,
    agent_name="benefits-agent",
    session_id="chat-4821",
)
print(result.observation_id, result.status)     # queued; scored a few seconds later

observe() takes the same fields.

Example 1: a contradicted figure

The answer says $6,500; the source says $6,000.

WhereWhat you see
The claimWrong (contradicted) — "The answer says $6,500; Summary of Benefits 2026, p.2 says $6,000."
ChecksGrounding: fail · No contradictions: fail · Safety: pass
Where to correctbenefits-agent, writing the answer — "the right reference was retrieved (relevance 0.82) but the answer states different numbers"

The source holds the right figure, so retrieval worked and the fix is in writing the answer: the prompt, the model, or a step that rewrites the figure. That holds even when the sources' overall relevance is low; the reason then reads "a retrieved reference states the fact (overall relevance 0.41) but the answer states different numbers". The name in the sentence is the document's source (else its id), so give documents readable names.

Example 2: an unsupported figure

Same source. The answer adds "The plan's lifetime maximum is $900,000." Nothing in the sources gives a lifetime maximum.

WhereWhat you see
The claimNot in the sources — "The sources don't give this figure ($900,000)."
ChecksGrounding: fail under Strict, a warning under Document review · No contradictions: pass · Safety: pass
Where to correct, sources relevant to the questionbenefits-agent, writing the answer — "the right reference was retrieved (relevance 0.78) but the answer says things it doesn't contain"
Where to correct, sources off-topicretrieval for benefits-agent — "the retrieved references don't match the question (relevance 0.31), so the answer had little to stand on"

An unsupported claim without a figure reads "The sources don't state this." It is never called wrong: it may be true, but nothing you sent shows it. Fix it by retrieving the document that holds the fact, or by telling the model to stay within the sources.

Example 3: a correct answer read with the document's title

client.log(
    "When do Acme's short-term disability benefits start?",
    "Acme's short-term disability benefits start after a 7-day elimination period and pay 60% of weekly earnings.",
    reference_materials=[{
        "id": "std-2026-s3",
        "source": "Acme Short-Term Disability (STD) Plan 2026, section 3",
        "content": "Benefits begin after a 7-day elimination period. The weekly benefit is 60% of weekly earnings.",
    }],
    agent_name="benefits-agent")

Both claims are Supported, and Grounding and No contradictions pass. The passage never says "short-term disability" or "Acme" — the title does — and the passage is read with its title as context. The figures (7 days, 60%) must still be written in the passage itself: a title never supplies a figure. Before 16 Nov 2026 this answer could show Not in the sources.

Safety reports harm only

Safety covers toxicity, bias, harmful content, privacy, inappropriate language, violence and adult content. A wrong fact is a grounding problem:

The answerGroundingNo contradictionsSafety
States $6,500 where the source says $6,000failfailnot affected
Adds a figure the sources don't givefail (Strict) or warningpassnot affected
Contains an insult or personal dataas its claimsas its claimsfail when over the limit

Misinformation still counts toward Safety for an answer sent without any sources, where there is nothing to ground it against.

Where to correct, at a glance

What the claims showWhere to correct namesWhy
A supplied source contradicts a figure or factThe answer step (agent, writing the answer)The right fact was retrieved
Claims not in the sources, sources relevant to the questionThe answer stepThe answer added things it was not given
Claims not in the sources, sources not relevant (relevance under 0.5)Retrieval (retrieval for agent)The fact was not retrieved
A wrong figure against query rows or the answer's own tool resultsThe answer stepThe answer misread its own data
A figure from the wrong document, a wrong calculationThe answer stepThe answer's own work is wrong

SQL answers: figures checked against the returned rows

Send the rows

Send the rows your query returned as a reference with rows (a list of flat records), together with the SQL. Database values — decimals, dates, driver row objects — are converted for you; a row without column names is refused at the call. Up to 10,000 rows are read, all of them.

sql = """SELECT region, SUM(paid_amount) AS paid, COUNT(*) AS claims
         FROM claims WHERE year = 2026 GROUP BY region"""
rows = cursor.execute(sql).fetchall()      # e.g. [{"region": "North", "paid": 425000.50, "claims": 40}, ...]

client.log(
    "How much did we pay in claims in 2026, by region?",
    answer_text,
    sql=sql,
    reference_materials=[{"id": "claims_by_region", "source": "claims_by_region query", "rows": rows}],
    agent_name="sql-analyst")

The rows in the examples:

regionpaidclaims
North425000.5040
South30000035
West15000025

What each kind of figure shows

Each figure passes only when exactly one computation over the rows gives it, to the precision it is written in. The computation is shown with the claim.

The answer saysVerdictShown with the claim
Total paid was $875,000.50.SupportedSUM(paid) over 3 rows = 875,000.50 (the only computation over the rows that gives it)
North paid $425,000.50 and South paid $300,000.Supported (breakdown)North's paid = 425,000.50 (the only computation over the rows that gives it); South's paid = 300,000 (the only computation over the rows that gives it)
Average paid was $291,666.83.SupportedAVG(paid) over 3 rows = 291,666.83 (the only computation over the rows that gives it)
Claims came from 3 regions.SupportedCOUNT(*) over 3 rows = 3 (the only computation over the rows that gives it)
Total claims were 100.SupportedSUM(claims) over 3 rows = 100 (the only computation over the rows that gives it)
North accounted for 48.57% of paid.Supportedshare of paid: 425,000.50 of 875,000.50 = 48.57 (the only computation over the rows that gives it)
North paid $300,000.Wrong$300,000 is South's paid; North's paid is 425,000.50, not $300,000 — explained as "The answer says $300,000; claims_by_region gives North's paid as 425,000.50."
Total paid was $900,000.Not in the sourcesNo computation over the rows gives it — "The sources don't give this figure ($900,000)."
East paid $200,000.Not in the sourcesThe results hold no East row; a figure is never passed on another row's value

The wording around a figure is read too: "average paid" must be an average, not a total that happens to match. A wrong figure against the rows is the answer step's to correct: the query returned the right data and the answer misread it.

An ambiguous figure is left unverified

With rows North (paid 60, claims 40), South (90, 35) and West (150, 25), the answer "Paid claims came to 100." matches two computations:

100: 2 computations over the rows give 100 (SUM(claims) over 3 rows; AVG(paid) over 3 rows): ambiguous

The rows never pass or fail it on a guess, and the standard checks do not count it as supported. Under Max-Accuracy, the reasoning model reads which column and operation the sentence means, and code then computes and decides. To avoid ambiguity, name the column and the operation in the answer ("Total claims were 100").

Limits

  • A percentage is compared only with a percentage: a share, a column whose cells are written with "%", or a column whose unit you declare in the project's SQL checks as a fraction (0.2374 is 23.74%) or a percent (23.74 is 23.74%). Against an undeclared plain number it is left unverified.
  • In a sentence with two figures, code decides the ones it can point at; the rest are left to Max-Accuracy.
  • A row that totals the others (a ROLLUP row) is left out of computed totals.
  • variA/Bly checks the SQL text and rows you send. It never connects to your database.

Troubleshooting

You seeWhyFix
A correct SQL total is Not in the sourcesThe rows were not sent, or the answer names no column of the rowsSend rows with the answer; name the column ("total paid", not "the total")
A figure left unverified as ambiguousTwo computations over the rows give the same numberName the column and operation, or use Max-Accuracy for that traffic
A wrong figure did not lower SafetySafety reports harm onlyTrack factual errors with Grounding and No contradictions
The explanation names a source like doc-17The document was sent without a sourceSend a readable source (title and page) with each document
Where to correct says retrieval, but the document was retrievedThe sources as a whole don't match the question (relevance under 0.5), and no source contradicts the claimRetrieve fewer, more relevant passages; check the passage with the fact was among them

FAQ

Do these changes re-score my old answers? No. Past answers keep their verdicts until you re-score them.

Why isn't a claim that is not in the sources called wrong? Nothing you sent proves it wrong. It lowers the grounding score and, under Strict, fails Grounding; it is not misinformation.

Is the SQL check a model? No. Code computes the cell, total, count, average or share from your rows. A model is used only under Max-Accuracy, to say which column and operation an ambiguous sentence means; code still decides.


Concepts on this page: Claims, grounding and where to correct · How answers are judged