How answers are judged
Answer types, scoring rules, sample % and Max-Accuracy: which answers are judged how, and how carefully.
Answer types
What it is. Answers from sources: every factual claim must be supported by the documents or rows given. Document analysis: the same, but introductions and connecting sentences aren't counted, conclusions drawn from verified facts pass, and a claim naming a document is checked against that document.
Why it matters. A claims letter or an audit review isn't a chatbot reply; judging it the same way would fail its framing sentences.
How it relates. Each scoring rule sets one. The default rule's type is How answers are judged on the policy Summary.
Where you see it. Evaluation policy › Summary and Scoring rules.
Example. benefits-agent: Answers from sources. letter-writer: Document analysis.
Scoring rules
What it is. Conditions on Environment, Agent, Tag or Metadata (with a key), using is, is not or is one of, that lead to an answer type, a sample % and a Max-Accuracy share. All of a rule's conditions must hold.
Why it matters. Different traffic needs different judging, without separate projects.
How it relates. Rules are checked top to bottom and the first match wins. The last rule is the default: no conditions, scores everything else, can't be removed. Match preview counts recent answers a rule would have scored.
Where you see it. Evaluation policy › Scoring rules; the rule that scored an answer is on its Summary. See Scoring rules.
Example. 1. Agent is letter-writer → Document analysis. 2. Metadata line_of_business is one of group-life, disability → Max-Accuracy 50%. 3. Default.
Sample %
What it is. The share (1 to 100) of a rule's matching answers that are scored.
Why it matters. Score a busy, low-risk agent at 10% to control usage.
How it relates. Answers outside the sample are stored but not scored, never counted as 0. A whole session is in or out together. Dataset runs are never sampled.
Where you see it. Sample % of matching answers scored in a scoring rule; an answer shows Not scored · outside the sample.
Example. Agent is faq-bot → Sample 10%.
Max-Accuracy
What it is. The reasoning model checks only the claims the standard checks can't settle. Everything else stays deterministic.
Why it matters. More accurate on hard cases; verdicts can occasionally differ between identical runs on close calls.
How it relates. Three settings: the switch on the Summary; each rule's Max-Accuracy share of matching events; and the Max-Accuracy scope: All answers, Environments (exact labels) or Release runs. For SQL figures the model only says which column and operation a sentence means; code computes and decides. See Max-Accuracy.
Where you see it. Evaluation policy › Summary and Scoring rules; the Max-Accuracy card on the Overview.
Example. Scope: Environments = prod; default rule share 100%. Staging stays deterministic.