All docs
Evaluation policyHow the evaluation policy worksdashboard

How the evaluation policy works

One policy per project decides how every answer is scored: versions, tabs, the Summary settings and the whole policy as one file.

Every project has one evaluation policy: the rulebook that decides how each answer is scored. It says how claims are judged, how much each category counts, which answers get the extra-careful Max-Accuracy check, and what your data looks like (your codes, documents, limits and rules). Creating a project creates policy v1, so your first answer is scored with nothing to set up. You change the policy when you want scoring to fit your business more closely.

Why versions

Changing how answers are scored changes what the numbers mean. So a change never happens silently:

  1. Edit. Every edit goes into a draft: a copy of the policy that scores nothing yet.
  2. Compare. See exactly what the draft changes against the version in force.
  3. Publish. Publish it with a change note (why you changed it). It applies to the whole project.
  4. New answers are scored with the new version from that moment.
  5. Optional: re-score recent answers with it, or put an earlier version back.
  • Nothing changes how answers are scored until you publish.
  • One version is in force for the whole project. Environments don't have their own versions.
  • Past answers keep their verdicts. Each one records the version and scoring rule that scored it.
  • Every publish is recorded with who did it and the note, in Versions & audit.

Where things are

Open Evaluation › Evaluation policy. The page opens on Summary; Advanced settings ▸ shows the other tabs. The header shows the version in force (v4 in force), the draft when there is one (v5 draft), Compare versions and Publish v5….

TabHoldsPage
SummaryHow answers are judged, Max-Accuracy, category weights, your datathis page
Checks & weightsEvery dimension and how it is scored; dimension bars; category weightsChecks & weights
Scoring rulesDifferent judging for different traffic; sampling; Max-Accuracy share and scopeScoring rules
Data definitionsCode lists, document scope, hand-off contracts, SQL checks, posting rules, chart of accounts, business rules, expected answers, requirement templatesData definitions
Versions & auditHistory, compare, put back in force, discard, re-score, audit logPublish, compare and roll back
YAMLThe whole version as one file: copy it, or edit it and save it as the draftPublish, compare and roll back

Who may do what: anyone with the Editor role can edit the draft; publishing and re-scoring need Policy admin. See Members & roles.

Summary: the settings most teams change

Evaluation policy: SummaryEvaluation policy: Summary

How answers are judged

Choose what your AI does. This is the answer type of the default scoring rule, which applies to every answer no other rule matches.

ChooseWhen your AI…What changes
Answers from sourcesanswers a question using documents, a knowledge base or query results: Q&A, chat, RAG, text-to-SQL, agentsEvery factual claim must be supported by the documents. A claim that isn't in them needs attention
Document analysisreviews documents and writes an analysis, or posts a result: invoice and contract reviews, reconciliations, journalsFacts must still be supported, but introductions aren't counted, conclusions drawn from verified facts pass, and a figure said to come from a named document is checked against that document

Different agents need different types? links to Scoring rules: for example the Q&A agent as Answers from sources and the letter-writing agent as Document analysis.

Max-Accuracy

The standard checks are deterministic: the same answer always gets the same score. A few claims are too close to call that way. With Max-Accuracy on, the reasoning model double-checks those claims. It is more accurate on hard cases; verdicts can occasionally differ between identical runs on close calls.

  1. Switch Max-Accuracy on. Turning it on sets the default rule's share to 100%.
  2. To double-check only some answers, click Set the share per rule and change Max-Accuracy share of matching events in Scoring rules.
  3. Publish.

The status line under the switch tells you where you stand:

Status lineMeans
Off: every event is scored deterministically.Nothing goes to the reasoning model
On, but no scoring rule sets a Max-Accuracy share yet.Every share is 0%, so nothing is double-checked. Set a share in a rule

Turning it off sets every rule's share to 0.

Category weights

How the six categories (Grounding, Safety, Semantic, Coherence, Quality, Depth) add up to the overall score. Change the weights opens Checks & weights.

Your data

Which data definitions are set, for example Code lists: not set. Set up opens Data definitions. They are optional: they let checks compare exact values instead of wording.

The whole policy as one file

Every version can be read and written as a YAML file, in the YAML tab or with the variably policy command. You never need it, but it shows everything in one place. Here is a complete policy for a claims assistant, explained line by line in the comments:

max_accuracy:
  enabled: true                 # the reasoning model may double-check hard claims...
  scope:
    kind: environments          # ...but only on answers logged with environment="prod"
    environments: [prod]
weights:                        # how the six categories add up to Overall (they must total 100)
  grounding: 30
  safety: 15
  semantic: 15
  coherence: 10
  quality: 15
  depth: 15
dimensions:                     # optional bars on single dimensions (0 to 1, higher is better)
  faithfulness: {needs_attention_below: 0.8, release_minimum: 0.85}
  toxicity: {needs_attention_below: 0.95}
rules:                          # checked top to bottom; the first match scores the answer
  - id: letters
    name: Claim letters are reviews
    applies_to:
      - {field: agent, op: is, values: [letter-writer]}
    answer_type: document_analysis
    sample_pct: 100
    max_accuracy_pct: 100
  - id: faq-sample
    name: Score a tenth of the FAQ bot
    applies_to:
      - {field: agent, op: is, values: [faq-bot]}
      - {field: environment, op: is, values: [prod]}
    answer_type: answers_from_sources
    sample_pct: 10
    max_accuracy_pct: 0
  - id: default                 # the last rule has no conditions: it scores everything else
    name: Everything else (default)
    applies_to: []
    answer_type: answers_from_sources
    sample_pct: 100
    max_accuracy_pct: 20
evaluation_config:              # your data definitions
  grounding_gate: strict
  date_order: MDY               # 04/03/2026 is April 3
  code_lists:
    jurisdiction:
      US-NJ: ["New Jersey", "NJ"]
      US-NY: ["New York", "NY"]
    leave_type:
      STD: ["short-term disability"]
      LTD: ["long-term disability"]
  scope_defaults:
    dimensions:
      - {name: plan_year, kind: period, start: "07-01", named_by: start}
      - effective
      - jurisdiction
  handoff_contracts:
    - to: adjudication
      fields:
        - {name: claim_id, type: identifier, required: true, pattern: "^CLM-[0-9]{6}$"}
        - {name: onset_date, type: date, required: true}
        - {name: leave_type, type: code, code_list: leave_type, required: true}
  sql_checks:
    dialect: postgres
    schema:
      tables:
        claims: {columns: [claim_id, plan_year, status, amount], primary_key: claim_id}
    rules:
      - {id: plan-year, kind: required_filter, table: claims, column: plan_year}
  requirement_templates:
    benefit-answer:
      version: 1
      requirements:
        - {id: plan-year, text: "Names the plan year", critical: true}
        - {id: network, text: "Says whether the amount is in-network or out-of-network", critical: true}
  business_rules:
    facts:
      - {name: approved_visits, type: number, labels: ["approve", "visits approved"]}
      - {name: prior_auth, type: boolean, labels: ["prior authorization"]}
    rules:
      - name: Physical therapy visits within the limit
        check: {fact: approved_visits, at_most: 12}
        unless: {fact: prior_auth, is: true}
        message: Beyond 12 visits needs prior authorization.

You don't type all of this. Every part has its own screen; the YAML is the same settings written down. How to edit it safely: Publish, compare and roll back.

Common mistakes

You seeWhyFix
A change made no differenceIt is still a draft: the header shows v5 draftPublish it. To change past answers, re-score them
Max-Accuracy is on but nothing was double-checkedEvery rule's share is 0%, or the answers are outside the Max-Accuracy scopeSet a share in Scoring rules; widen the scope
Publish is greyed outYour role is Editor: publishing needs Policy adminAsk a Policy admin to publish, or to change your role