Checks and gates

Scores tell you how an answer did on a scale. Checks turn them into decisions: every answer — and, for agents, every session — passes, fails, sits on the borderline, or is not applicable to a check. The experiment's Overview leads with the checks that are failing, and each failing case says where to correct it. Gates are the bars those decisions are made against.

This page lists every check and gate, its rule, its default, and where to change it.

Answer checks — every evaluation

Check	Passes when	Not applicable when	Bar and where to change it
Grounding	Every claim is supported by the documents (Strict), or no claim is proven wrong (Document review)	No reference_materials were sent	The Grounding gate — Overview tab
Hallucination-free	No claim is contradicted or proven wrong by the documents — a mismatched number, a figure from the wrong document, a wrong calculation or comparison conclusion	No reference_materials	Per-claim decision; no setting
Safety	Every safety dimension (toxicity, bias, harmful content, privacy, misinformation, inappropriate language, violence, adult content) is at or under its risk limit	No safety dimensions were scored	Risk limit 0.5 per dimension by default. Stricter toxicity and bias limits, or a minimum safety score, are available on request
Quality floor	The overall score across the six categories is at least the floor	Shown on A/B experiments only — in Observe it is a soft average, not a decision	70 by default; Promotion gates → minimum overall score on the Edit page

A passing answer within 0.05 (5 points) of a bar is marked borderline, so a near-miss is visible before it becomes a failure.

The Grounding gate

	Strict (default)	Document review
For	RAG and question answering — an unsupported answer is the failure	Invoices, contracts, clinical summaries, financial statements, filings — anything checked against documents
Fails on	Any claim not supported by the documents	Only claims the documents prove wrong
A claim the documents neither confirm nor contradict	Fail	Borderline — flagged for review
A possible contradiction (one model's doubt, not proof)	Fail	Borderline — flagged for review
Summaries of verified findings, framing and conclusions	Checked like any claim	Summaries pass on verified findings; framing and conclusions are not counted

Set it on the experiment's Overview tab, under Grounding gate. It works in Observe as well as on A/B experiments (draft or paused), and the promotion decision is recomputed at once. Answers already in are re-read under the new gate (nothing is re-scored); new answers are scored with its rules. How each claim is judged under it is in Score a document review.

Agent checks — every session

These need a session (session_id) and, for most, agent steps (agent_name):

Check	Passes when	Needs
Tool correctness	Every expected tool was called with the expected arguments	expected_tools on agent steps
Trajectory	Every expected step was taken, in the expected order	expected_tools on agent steps
No loops	No repeated tool call or stagnant, near-duplicate reasoning step	A session with at least two agent steps
Agents agree	No two agents' outputs contradict each other	A session with at least two agents
Final answer uses sub-agents	The final answer reflects every upstream agent's output and contradicts none	A final step marked is_aggregator=True
Within policy	No tool call outside the allowed list	allowed_tools

observe(prompt=task, response=out, session_id=session, agent_name="router",
        tools_called=[{"name": "search", "arguments": {"q": query}}],
        expected_tools=[{"name": "search"}],
        allowed_tools=["search", "fetch_invoice"])

The full setup is in Score a multi-agent trace.

Promotion gates — A/B experiments

Before a variant can be promoted, it must clear three gates. They are set on the experiment's Edit page under Promotion gates:

Gate	Default	Means
Minimum overall score	70	The variant's mean overall score across the six categories
Grounded answers	90%	The share of its claim-checked answers that pass the Grounding check — under the experiment's Grounding gate, so a Document review experiment counts proven errors, not unconfirmed claims
Minimum sample size	357	Evaluations needed before any winner is named — 95% confidence, ±5% margin

Clearing the gates makes a variant eligible; whether its difference is real is the next question — see Read the verdict.

What Observe uses

An Observe experiment has no variants to promote, so it shows the answer and agent checks only: the Grounding gate is the one setting it takes, and Safety uses its default limits. Upgrade to an experiment to set promotion gates and compare variants.