Policy, versions and re-scoring
One evaluation policy per project; every change is a version you publish with a note, compare, and can re-score with.
Evaluation policy
What it is. The project's rules for scoring: the answer type per scoring rule, Max-Accuracy, category weights, per-dimension bars and your data definitions. One per project.
Why it matters. Every score in the project, live, in a dataset run or in an experiment, means the same thing because one policy produces it.
How it relates. Created with the project as v1 from your answers under What does your AI do?. It contains scoring rules, weights and bars and data definitions.
Where you see it. Evaluation › Evaluation policy (Summary; Advanced settings ▸ for the other tabs); Policy in force in the footer. See How the evaluation policy works.
Example. claims-assistant policy v4: letters scored as Document analysis, production double-checked, a hand-off contract for adjudication.
Versions: draft, published, in force
What it is. Every edit saves to a draft, which scores nothing. Publishing makes it the version in force; the previous one becomes superseded. A draft can be discarded. Version numbers never repeat.
Why it matters. Nobody changes how answers are scored by accident, and every verdict can be traced to the exact rules that made it.
How it relates. Each answer records the version that scored it. A dataset run is pinned to one version.
Where you see it. The policy header (v{n} in force, v{n} draft) and Versions & audit.
Example. v5 draft adds a code list; v4 stays in force until v5 is published.
Publishing with a note
What it is. Making a draft the version in force, with a Change note saying why. It applies to the whole project from that moment.
Why it matters. The audit log records who changed scoring, when and why.
How it relates. Policy admins and owners publish in the app; from code, a key with Change the evaluation policy (variably policy publish, or the SDK settings calls, which publish a new version). Past answers keep their verdicts unless you re-score.
Where you see it. Publish v{n}… in the policy header; Versions & audit › Audit log; Organization › Audit log. See Publish, compare and roll back.
Example. Note: "Letters are analyses: framing sentences shouldn't fail grounding."
Comparing versions and policy as YAML
What it is. A diff of any two versions. The YAML tab shows a version as a file you can copy, edit and Apply as draft; variably policy pull | diff | push | publish does the same from your repository.
Why it matters. Review a change before it scores anything; keep the policy in version control next to your code.
How it relates. Applying YAML replaces the whole draft; publishing stays a separate step. Comments are not kept.
Where you see it. Compare versions in the policy header; Versions & audit; YAML.
Example. variably policy diff variably-policy.yaml, then variably policy push variably-policy.yaml --note "Production double-checked".
Re-score
What it is. Scoring stored answers again with a published version. A new verdict is added; the earlier ones are kept.
Why it matters. See how a policy change judges last week's traffic, or apply it to answers you report on.
How it relates. Up to 1,000 answers per job, one job at a time per project, run behind live scoring. Policy admins and owners start it. Lists then show the latest verdict.
Where you see it. The card after publishing, Versions & audit › Re-score answers…, or Live events › select rows › Re-score…. See Re-score past answers.
Example. After publishing v5, re-score the last 7 days of prod answers.
Verdict history
What it is. Every verdict an answer has had: On arrival, each re-score, and human review verdicts.
Why it matters. You can always see what was decided before, and with which version.
How it relates. Re-scores and reviews add to it; nothing is overwritten.
Where you see it. The History tab of an answer.
Example. v3 on arrival: Contradicted. v5 re-score: Supported, after the document scope was set.