Score a document review

Document reviews ask a different question from RAG. A RAG answer is wrong when the context does not support it. A document review — an invoice checked against a contract, a discharge summary against lab results, a financial statement against the ledger, a filing against a judgment — is wrong when the documents prove it wrong: a figure that is not in them, a figure taken from the wrong document, a calculation that does not hold, a conclusion its own figures contradict.

This page takes you from nothing to a running document-review evaluation, step by step. Steps 1–2 are the same as every guide; if Observe is already running, start at step 3.

1. Install the SDK

pip install variably-sdk

2. Set two environment variables

VARIABLY_API_KEY=vbl_your_key_here
VARIABLY_BASE_URL=https://api.variably.dev

The API key is scoped to one project — everything it sends lands there and nowhere else.

3. Send the documents with every answer

Pass each source document as one entry in reference_materials — the whole document, not chunks — with an id that says what it is (its filename is ideal) and its full text as content. Markdown tables, headings and lists are read as such; so are JSON answers and Mermaid diagrams.

from pathlib import Path
from variably import observe

DOCS = Path("documents")  # the files the review was run on
references = [
    {"id": p.name, "content": p.read_text(), "source": p.name}
    for p in sorted(DOCS.iterdir())
]

observe(
    prompt=task,                    # the review task, verbatim
    response=report,                # the answer to check
    reference_materials=references,
    session_id=f"review-{run_id}",  # one session per review
    agent_name="reviewer",
)

Two things make the check sharper:

• A title that names the document. The first line of each document ("INVOICE INV-2041", "Discharge Summary DS-4471", "Balance Sheet FY2025") or its filename is how a claim's mention — "the invoice", "INV-2041", "the discharge summary" — is matched to it. A claim that names a document is checked against that document only.
• The documents the model actually read. Sending more than it saw makes an answer look better supported than it is; sending less makes correct figures look invented.

Paste this into your coding agent

Add variA/Bly evaluation to this document-review application. variA/Bly checks every claim in
the answer against the source documents. It is one function call after each answer is
generated. Do not change any existing logic.

1. Install the SDK: `pip install variably-sdk`

2. Find where the source documents are loaded and where each answer (or each agent's output)
   is produced. Show me the file paths and line numbers.

3. After each answer, call observe() with:
   - prompt              the review task, verbatim
   - response            the answer text
   - reference_materials every source document the model read, WHOLE, as
                         [{"id": <filename>, "content": <full text>, "source": <filename>}]
   - session_id          one id per review run, shared by every step of that run
   - agent_name          the step's name, if the app has several steps or agents
   - is_aggregator=True  ONLY on the one step that produces the final answer

4. Use only the argument names listed above. observe() rejects unknown keyword arguments.

5. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. Do not hardcode the key.

6. observe() is non-blocking and swallows its own network errors. Do not wrap it in try/except.

When you are done, tell me which lines you instrumented and confirm that reference_materials
holds the same documents the model was given.

4. Multi-agent reviews: mark the final step

When several agents work on one review — a researcher, an analyst, a writer — send every agent's output as its own step of the same session, and mark the step that produces the final report with is_aggregator=True:

for step in run.steps:                    # each agent's output
    observe(prompt=task, response=step.output, reference_materials=references,
            session_id=session, agent_name=step.agent)

observe(prompt=task, response=run.final_report, reference_materials=references,
        session_id=session, agent_name="lead", is_aggregator=True)

The final report's summary lines — "the main issues are an overcharge and a shortened payment term" — rarely appear in any document. They pass when they follow from the earlier agents' findings that were themselves verified against the documents; a summary that only an unverified finding supports does not. Without is_aggregator, those summaries can only be marked not confirmed.

5. Switch the experiment to the Document review gate

The grounding check has two gates. The default, Strict, fails an answer when any claim is not supported — right for RAG, too strict for a review that summarises and concludes.

1. Open LLM Experiments and the experiment your API key writes to. With Observe, that is the project's Observations experiment.
2. On the Overview tab, find Grounding gate at the top.
3. Choose Document review.

Answers sent after the change are scored with the Document review rules (summaries checked against verified findings, framing and conclusions shown but not counted).

How figures are checked under Document review. For a claim with figures, a local model on variA/Bly's own infrastructure (no third-party API) reads which document each figure is attributed to and how any computed figure is derived — without seeing the figures themselves, so it cannot bend its reading to make a number fit. Every figure is then checked exactly against your documents: it must be in the document the claim names, or computed from figures the documents hold. The model never decides a verdict; when its reading cannot be confirmed, the claim is marked for review, never proven wrong. The same check is used for invoices and contracts, clinical records and financial statements — no per-domain setup. Answers already in are re-read under the new gate — nothing is re-scored. Switching back to Strict reads every stored claim strictly again. On an A/B experiment the gate can be changed while it is draft or paused.

What the two gates do with each kind of claim:

Claim	Strict	Document review
Supported by the documents	pass	pass
Computed from figures the claim states ("$14.00 × 40 = $560.00")	pass	pass
Computed from figures in the documents, or checked per document	checked like any claim	pass
Proven wrong — wrong figure, wrong document, wrong calculation, wrong conclusion, invented ID	fail	fail
Not confirmed, or possibly contradicted	fail	review (borderline)
Summary of findings the agents verified	checked like any claim	pass
Framing or conclusion	checked like any claim	not counted

The gate also decides the A/B promotion check — see Checks and gates.

6. Run a review and read the result

Run one review. Open the experiment's Quality tab and the Grounding check: each answer is listed as passing, borderline (review) or failing, and opening one shows every claim with its verdict.

Passed

Label	Meaning
(no pill)	A document supports the claim
Computed	Not written in the documents, but computed exactly from figures that are — $14.00 × 40 = $560.00; 3 March 2026 + 45 days = 17 April 2026; "15 days shorter" (45 − 30)
Figures checked per document	Document review: every figure is in the document the claim attributes it to, or computed exactly from figures the documents hold
Checked per document	A comparison across documents where every figure is in the document it is attributed to
Summarises verified findings	A summary that follows from findings already verified against the documents

Proven wrong — fails under both gates

Label	Meaning
Number not in sources	A figure the documents it draws on do not hold, and that is not computed from them
Figure from a different document	"The invoice states $175.00" when $175.00 is the contract's price
Calculation is wrong	A written calculation that does not hold: 120 × $41.00 = $4,820.00
Conclusion contradicts its figures	"Result: match" while the compared figures differ, or "discrepancy" while they are equal
Contradicts a source	A summary contradicting an earlier agent's verified finding

Review — fails Strict, borderline under Document review

Label	Meaning
Not supported by sources	The documents neither confirm nor contradict it — a paraphrase the checker could not match, or a claim to look at
May contradict a source	Unsupported, and a passage may disagree with it. One model score is not proof, so a person decides

Not counted — shown, never for or against

Label	Meaning
Framing	About the report itself, or a reference-list entry — nothing to check
Conclusion	An inference or an absence finding ("no amendment has been provided", "PO-5512: not mentioned") — no passage can confirm or contradict it

Each failing answer also says where to correct it — the retrieval or the writing step, and in a multi-agent run the agent.

7. Prove it catches errors

Before trusting a green check, send one answer you know is wrong and confirm it fails. Copy a real report, change one figure to another document's value — the contract's price stated as the invoice's, or a lab value from the wrong report — and send it as its own session:

observe(prompt=task, response=report.replace("$189.00", "$175.00"),
        reference_materials=references, session_id="review-planted-error")

It should fail the Grounding check with Figure from a different document or Number not in sources on that line. If it passes, the documents were probably not sent — check that reference_materials is non-empty and holds the full text.

Answers that check well

The checker reads the answer the way a reviewer would. These habits — easy to put in the reviewing prompt — make more of a correct answer verifiable:

• Name the document next to each figure: "Invoice INV-2041: $189.00 per unit", not "the price is $189.00".
• Write calculations out: "40 × ($189.00 − $175.00) = $560.00" is checked exactly; "$560.00 over" is checked only if both operands are in the documents.
• Give dates with their basis: "45 days from the invoice date of 3 March 2026, so 17 April 2026".
• Keep tables and nested bullets — a row label or parent bullet is carried into each line.

What it does not do yet

Stated plainly, so a borderline result is read correctly:

• Negated claims ("the $189.00 rate is not supported by the contract") can be read as asserting the figure.
• Figures derived from three or more numbers in one step (a corrected total across several lines) are marked not confirmed rather than computed.
• Conclusions are not verified — they are shown and not counted. Checking reasoning needs a reasoning check, which is on the roadmap.

For the hardest claims, Max-Accuracy mode adds a local reasoning judge.