How the evaluation policy works
One policy per project decides how every answer is scored: versions, tabs, the Summary settings and the whole policy as one file.
Every project has one evaluation policy: the rulebook that decides how each answer is scored. It says how claims are judged, how much each category counts, which answers get the extra-careful Max-Accuracy check, and what your data looks like (your codes, documents, limits and rules). Creating a project creates policy v1, so your first answer is scored with nothing to set up. You change the policy when you want scoring to fit your business more closely.
Why versions
Changing how answers are scored changes what the numbers mean. So a change never happens silently:
- Edit. Every edit goes into a draft: a copy of the policy that scores nothing yet.
- Compare. See exactly what the draft changes against the version in force.
- Publish. Publish it with a change note (why you changed it). It applies to the whole project.
- New answers are scored with the new version from that moment.
- Optional: re-score recent answers with it, or put an earlier version back.
- Nothing changes how answers are scored until you publish.
- One version is in force for the whole project. Environments don't have their own versions.
- Past answers keep their verdicts. Each one records the version and scoring rule that scored it.
- Every publish is recorded with who did it and the note, in Versions & audit.
Where things are
Open Evaluation › Evaluation policy. The page opens on Summary; Advanced settings ▸ shows the other tabs. The header shows the version in force (v4 in force), the draft when there is one (v5 draft), Compare versions and Publish v5….
| Tab | Holds | Page |
|---|---|---|
| Summary | How answers are judged, Max-Accuracy, category weights, your data | this page |
| Checks & weights | Every dimension and how it is scored; dimension bars; category weights | Checks & weights |
| Scoring rules | Different judging for different traffic; sampling; Max-Accuracy share and scope | Scoring rules |
| Data definitions | Code lists, document scope, hand-off contracts, SQL checks, posting rules, chart of accounts, business rules, expected answers, requirement templates | Data definitions |
| Versions & audit | History, compare, put back in force, discard, re-score, audit log | Publish, compare and roll back |
| YAML | The whole version as one file: copy it, or edit it and save it as the draft | Publish, compare and roll back |
Who may do what: anyone with the Editor role can edit the draft; publishing and re-scoring need Policy admin. See Members & roles.
Summary: the settings most teams change

How answers are judged
Choose what your AI does. This is the answer type of the default scoring rule, which applies to every answer no other rule matches.
| Choose | When your AI… | What changes |
|---|---|---|
| Answers from sources | answers a question using documents, a knowledge base or query results: Q&A, chat, RAG, text-to-SQL, agents | Every factual claim must be supported by the documents. A claim that isn't in them needs attention |
| Document analysis | reviews documents and writes an analysis, or posts a result: invoice and contract reviews, reconciliations, journals | Facts must still be supported, but introductions aren't counted, conclusions drawn from verified facts pass, and a figure said to come from a named document is checked against that document |
Different agents need different types? links to Scoring rules: for example the Q&A agent as Answers from sources and the letter-writing agent as Document analysis.
Max-Accuracy
The standard checks are deterministic: the same answer always gets the same score. A few claims are too close to call that way. With Max-Accuracy on, the reasoning model double-checks those claims. It is more accurate on hard cases; verdicts can occasionally differ between identical runs on close calls.
- Switch Max-Accuracy on. Turning it on sets the default rule's share to 100%.
- To double-check only some answers, click Set the share per rule and change Max-Accuracy share of matching events in Scoring rules.
- Publish.
The status line under the switch tells you where you stand:
| Status line | Means |
|---|---|
| Off: every event is scored deterministically. | Nothing goes to the reasoning model |
| On, but no scoring rule sets a Max-Accuracy share yet. | Every share is 0%, so nothing is double-checked. Set a share in a rule |
Turning it off sets every rule's share to 0.
Category weights
How the six categories (Grounding, Safety, Semantic, Coherence, Quality, Depth) add up to the overall score. Change the weights opens Checks & weights.
Your data
Which data definitions are set, for example Code lists: not set. Set up opens Data definitions. They are optional: they let checks compare exact values instead of wording.
The whole policy as one file
Every version can be read and written as a YAML file, in the YAML tab or with the variably policy
command. You never need it, but it shows everything in one place. Here is a complete policy for a claims
assistant, explained line by line in the comments:
max_accuracy:
enabled: true # the reasoning model may double-check hard claims...
scope:
kind: environments # ...but only on answers logged with environment="prod"
environments: [prod]
weights: # how the six categories add up to Overall (they must total 100)
grounding: 30
safety: 15
semantic: 15
coherence: 10
quality: 15
depth: 15
dimensions: # optional bars on single dimensions (0 to 1, higher is better)
faithfulness: {needs_attention_below: 0.8, release_minimum: 0.85}
toxicity: {needs_attention_below: 0.95}
rules: # checked top to bottom; the first match scores the answer
- id: letters
name: Claim letters are reviews
applies_to:
- {field: agent, op: is, values: [letter-writer]}
answer_type: document_analysis
sample_pct: 100
max_accuracy_pct: 100
- id: faq-sample
name: Score a tenth of the FAQ bot
applies_to:
- {field: agent, op: is, values: [faq-bot]}
- {field: environment, op: is, values: [prod]}
answer_type: answers_from_sources
sample_pct: 10
max_accuracy_pct: 0
- id: default # the last rule has no conditions: it scores everything else
name: Everything else (default)
applies_to: []
answer_type: answers_from_sources
sample_pct: 100
max_accuracy_pct: 20
evaluation_config: # your data definitions
grounding_gate: strict
date_order: MDY # 04/03/2026 is April 3
code_lists:
jurisdiction:
US-NJ: ["New Jersey", "NJ"]
US-NY: ["New York", "NY"]
leave_type:
STD: ["short-term disability"]
LTD: ["long-term disability"]
scope_defaults:
dimensions:
- {name: plan_year, kind: period, start: "07-01", named_by: start}
- effective
- jurisdiction
handoff_contracts:
- to: adjudication
fields:
- {name: claim_id, type: identifier, required: true, pattern: "^CLM-[0-9]{6}$"}
- {name: onset_date, type: date, required: true}
- {name: leave_type, type: code, code_list: leave_type, required: true}
sql_checks:
dialect: postgres
schema:
tables:
claims: {columns: [claim_id, plan_year, status, amount], primary_key: claim_id}
rules:
- {id: plan-year, kind: required_filter, table: claims, column: plan_year}
requirement_templates:
benefit-answer:
version: 1
requirements:
- {id: plan-year, text: "Names the plan year", critical: true}
- {id: network, text: "Says whether the amount is in-network or out-of-network", critical: true}
business_rules:
facts:
- {name: approved_visits, type: number, labels: ["approve", "visits approved"]}
- {name: prior_auth, type: boolean, labels: ["prior authorization"]}
rules:
- name: Physical therapy visits within the limit
check: {fact: approved_visits, at_most: 12}
unless: {fact: prior_auth, is: true}
message: Beyond 12 visits needs prior authorization.
You don't type all of this. Every part has its own screen; the YAML is the same settings written down. How to edit it safely: Publish, compare and roll back.
Common mistakes
| You see | Why | Fix |
|---|---|---|
| A change made no difference | It is still a draft: the header shows v5 draft | Publish it. To change past answers, re-score them |
| Max-Accuracy is on but nothing was double-checked | Every rule's share is 0%, or the answers are outside the Max-Accuracy scope | Set a share in Scoring rules; widen the scope |
| Publish is greyed out | Your role is Editor: publishing needs Policy admin | Ask a Policy admin to publish, or to change your role |