All docs
ConceptsImprove, review and releasedashboard

Improve, review and release

Issues, human review and agreement, datasets, collection rules, runs, the release gate, alerts and experiments.

Issues

What it is. Failing checks grouped by cause or by agent, ranked, each with its stage and where to correct.

Why it matters. Fix the cause behind fifty failing answers once, instead of reading them one by one.

How it relates. From an issue: open its failing answers, Create alert, or Collect into dataset. Checks that are Not configured aren't failures, so they don't appear in Issues; an answer's Checks tab says which definition to add.

Where you see it. Evaluation › Issues; Top issues on the Overview. See Issues.

Example. "Hand-off fields unchanged · adjudication · 14 answers · Where to correct: hand-off".

Human review and agreement

What it is. People check scored answers: a verdict per check (Pass, Fail or Not judged), Agree or Disagree per claim, and a verdict on the answer. Agreement measures how often reviewers and the scorer agree, with Cohen's kappa (agreement beyond chance; 1 is perfect, 0 is chance) once there are at least 20 pairs.

Why it matters. Evidence that the scorer matches your experts, and a steady stream of corrected cases.

How it relates. Answers arrive from queue rules (a sample of answers failing a check, by agent, environment or overall score) or from Send to review. A reviewed answer can be added to a dataset with the reviewer's text as the expected answer.

Where you see it. Evaluation › Human review: Queue, Queue rules, Agreement. See Human review.

Example. Queue 10% of prod answers failing Grounding; after 40 reviews, kappa 0.71.

Datasets and collection rules

What it is. A dataset is a fixed set of items (question, optional expected answer, sources, tags). Items come from live answers, sessions, human review, a JSONL upload, or collection rules that copy matching live answers automatically.

Why it matters. Test a release on the same questions every time.

How it relates. Optional: live monitoring needs none. Answers logged for a dataset run are never collected again. A JSONL upload with a bad line stores nothing.

Where you see it. Evaluation › Datasets & runs: Datasets, Collection rules. See Datasets and Collection rules.

Example. "claims-regression": 200 items, plus 10% of prod answers scoring below 70.

Runs and comparing runs

What it is. A run is your app answering a dataset, logged from code with log(..., dataset_item=, run=). Runs are compared item by item.

Why it matters. See what a release improved and what it broke.

How it relates. A run is pinned to the policy version in force when its first answer arrived. If the policy changes, Policy changed shows and further answers are refused: start a new run.

Where you see it. Datasets & runs › Runs & release gate; Compare with another run.

Example. Run release-4.3.0 against release-4.2.0: 6 items newly failing.

Release gate

What it is. A pass or fail for a run. It passes when nothing is still being scored, every item has an answer, the pass rate meets the minimum, and regressions against the baseline run stay within the limit.

Why it matters. A CI job can stop a release that got worse.

How it relates. Ask it from CI with variably gate: exit 0 pass, 1 fail (or still scoring when --wait ran out), 2 could not ask. --junit report.xml writes one test case per dataset item, with failed checks named.

Where you see it. Runs & release gate › Release gate, with Ask the same question from CI. See Runs and the release gate.

Example. variably gate --dataset $DATASET_ID --run "$GITHUB_SHA" --min-pass-rate 0.95 --max-regressions 0 --baseline-run release-4.2.0 --wait 600 --junit report.xml

Alerts

What it is. A rule on a metric (share needing attention, mean overall, mean faithfulness, a check's failure rate, or answer count) over a window, with a minimum number of answers and a quiet period after firing.

Why it matters. You hear about a drop instead of finding it on a dashboard.

How it relates. Notifies the project's webhooks and, if you tick Email me, you by email; once per incident and again when it resolves. Test measures the condition now and sends nothing.

Where you see it. Evaluation › Alerts: Rules, Events, Webhooks; Firing alerts on the Overview. See Alerts.

Example. Check failure rate of Hand-off fields unchanged > 5% over 60 minutes, at least 20 answers, environment prod.

Experiments

What it is. A/B (or more) tests that split live traffic between prompt or model versions and compare each with the control, scored by the project policy.

Why it matters. Decide on evidence whether a new prompt is better and no less safe.

How it relates. Ship readiness checks the statistical winner, no safety regression, no check regressing, and sample size. Declare winner… records the winner, saves it as a new prompt version with the evidence and completes the experiment. It moves no traffic.

Where you see it. Experimentation › Experiments. See Experiments.

Example. Control vs a prompt that cites the plan section: P(best) 97%, then Declare winner.