Scoring rules
Judge different traffic differently: answer type, sample %, Max-Accuracy share and scope, per agent, environment, tag or metadata.
One project often holds different kinds of traffic. A claims assistant has a Q&A agent that answers from documents, a letter-writer that drafts claim letters, and a busy FAQ bot. They shouldn't all be judged the same way. Scoring rules say, for each kind of traffic:
- which answers the rule covers (its conditions: environment, agent, tag or metadata);
- the answer type: Answers from sources or Document analysis (what they mean);
- the sample %: the share of matching answers that are scored, to save cost on busy traffic;
- the Max-Accuracy share: the share of matching answers whose hardest claims go to the reasoning model.
Open Evaluation policy › Advanced settings ▸ › Scoring rules.
How rules are chosen

Rules are checked top to bottom, and the first rule that matches an answer scores it. The last rule has no conditions: it scores everything no other rule matched, and it can't be removed. Each answer records the rule that scored it (Scored by rule in its details).
Add a rule

- Click Add rule.
- Give it a Name that says what it is for: Claim letters are reviews.
- Under Applies to, build the conditions. Choose a Field (Environment, Agent, Tag, or Metadata with a Metadata key), an Operator (is, is not, is one of) and a Value. Values you have recently sent are suggested. + Add condition adds another; all conditions must hold.
- Choose the Answer type.
- Set Sample % of matching answers scored, from 1 to 100.
- If Max-Accuracy is on, set Max-Accuracy share of matching events, from 0 to 100.
- Check Match preview (Last 7 days, Last 30 days or Last 90 days): how many recent answers this rule would have scored. It is counted before you save, from history.
- Click Save to draft v5 (the number of your draft), then publish.
A new rule is added just above the default rule, so the newest rule is checked last of yours. The first match wins: to change the order, move the rules in the YAML tab (how) and save it as the draft.
Sampling
On a busy chatbot you may not need every answer scored. With Sample % at 10, one answer in ten of that rule is scored; the rest are stored with Not scored · outside the sample, and never count as a score of 0.
- The choice is stable: the same answer is always in or out.
- A whole session is in or out together, so conversation checks always see every turn.
- Answers of a dataset run, and answers you re-score, are always scored, whatever the sample.
Max-Accuracy share and scope
Max-Accuracy must first be switched on in Summary (how). Then two settings decide which answers the reasoning model reads:
- Max-Accuracy scope, below the rules, limits it for the whole policy: All answers,
Environments (exact labels such as prod; an answer with no label is out of scope) or Release
runs (only answers logged with
run=). Click Save to draft. - Inside the scope, each rule's share decides how many of its answers are double-checked.
The reasoning model reads only the claims the standard checks can't settle, never the whole answer.
Examples
| Goal | Rule | Then |
|---|---|---|
| Claim letters are reviews, not Q&A | Agent is letter-writer | Document analysis |
| Double-check everything in production | Scope Environments: prod; default rule share 100% | Max-Accuracy on production only |
| Score a tenth of a busy FAQ bot | Agent is faq-bot | Sample % 10 |
| Judge a pilot customer's traffic separately | Metadata tenant is acme-insurance | Its own answer type and share |
| Don't double-check test traffic | Tag is not synthetic | Share 100% |
The same rules in the policy file (where to paste it):
max_accuracy:
enabled: true
scope: {kind: environments, environments: [prod]}
rules:
- id: letters
name: Claim letters are reviews
applies_to:
- {field: agent, op: is, values: [letter-writer]}
answer_type: document_analysis
sample_pct: 100
max_accuracy_pct: 100
- id: faq-sample
name: Score a tenth of the FAQ bot
applies_to:
- {field: agent, op: is, values: [faq-bot]}
answer_type: answers_from_sources
sample_pct: 10
max_accuracy_pct: 0
- id: pilot-tenant
name: Pilot customer
applies_to:
- {field: metadata.tenant, op: is, values: [acme-insurance]}
- {field: tag, op: is_one_of, values: [web, mobile]}
answer_type: answers_from_sources
sample_pct: 100
max_accuracy_pct: 50
- id: default
name: Everything else (default)
applies_to: []
answer_type: answers_from_sources
sample_pct: 100
max_accuracy_pct: 100
In the file, the fields are environment, agent, tag and metadata.<key>; the operators is,
is_not and is_one_of. Each rule has a short lower-case id that never changes, so answers keep
pointing at it after you rename the rule.
What you'll see
An answer from the letter-writer shows Scored by rule Claim letters are reviews in its Summary, and its claims are judged as an analysis (introductions not counted). An answer from the FAQ bot outside the sample shows Not scored: outside its rule's sample.
Environment conditions
An answer carries an environment when your code passes environment= to observe() or log(), or
sets it once with variably.set_context(environment=...), which also covers the Google ADK and
LangGraph integrations and OpenTelemetry set up with instrument(). An app exporting OpenTelemetry
its own way sets the standard resource attribute deployment.environment.name. An answer with no
environment never matches an environment condition and falls through to the next rule.
Common mistakes
| You see | Why | Fix |
|---|---|---|
| A value shows "not seen yet" | No recent answer had that value | Check spelling and case: prod is not Prod. Keep it if the traffic hasn't started yet |
| Match preview says 0 | The conditions match nothing in the window | Widen to Last 90 days; check each condition |
| A narrow rule never scores anything | A broader rule above it matches first | Move the narrow rule up in the YAML tab |
| "Max-Accuracy is off for this policy" when saving | A share above 0 while Max-Accuracy is off | Switch Max-Accuracy on in Summary first |
| "the last rule must apply to all events" (YAML) | The last rule has conditions | Keep a last rule with applies_to: [] |