All docs
Evaluation policyScoring rulesdashboard

Scoring rules

Judge different traffic differently: answer type, sample %, Max-Accuracy share and scope, per agent, environment, tag or metadata.

One project often holds different kinds of traffic. A claims assistant has a Q&A agent that answers from documents, a letter-writer that drafts claim letters, and a busy FAQ bot. They shouldn't all be judged the same way. Scoring rules say, for each kind of traffic:

  • which answers the rule covers (its conditions: environment, agent, tag or metadata);
  • the answer type: Answers from sources or Document analysis (what they mean);
  • the sample %: the share of matching answers that are scored, to save cost on busy traffic;
  • the Max-Accuracy share: the share of matching answers whose hardest claims go to the reasoning model.

Open Evaluation policy › Advanced settings ▸ › Scoring rules.

How rules are chosen

Scoring rulesScoring rules

Rules are checked top to bottom, and the first rule that matches an answer scores it. The last rule has no conditions: it scores everything no other rule matched, and it can't be removed. Each answer records the rule that scored it (Scored by rule in its details).

Add a rule

A scoring ruleA scoring rule

  1. Click Add rule.
  2. Give it a Name that says what it is for: Claim letters are reviews.
  3. Under Applies to, build the conditions. Choose a Field (Environment, Agent, Tag, or Metadata with a Metadata key), an Operator (is, is not, is one of) and a Value. Values you have recently sent are suggested. + Add condition adds another; all conditions must hold.
  4. Choose the Answer type.
  5. Set Sample % of matching answers scored, from 1 to 100.
  6. If Max-Accuracy is on, set Max-Accuracy share of matching events, from 0 to 100.
  7. Check Match preview (Last 7 days, Last 30 days or Last 90 days): how many recent answers this rule would have scored. It is counted before you save, from history.
  8. Click Save to draft v5 (the number of your draft), then publish.

A new rule is added just above the default rule, so the newest rule is checked last of yours. The first match wins: to change the order, move the rules in the YAML tab (how) and save it as the draft.

Sampling

On a busy chatbot you may not need every answer scored. With Sample % at 10, one answer in ten of that rule is scored; the rest are stored with Not scored · outside the sample, and never count as a score of 0.

  • The choice is stable: the same answer is always in or out.
  • A whole session is in or out together, so conversation checks always see every turn.
  • Answers of a dataset run, and answers you re-score, are always scored, whatever the sample.

Max-Accuracy share and scope

Max-Accuracy must first be switched on in Summary (how). Then two settings decide which answers the reasoning model reads:

  • Max-Accuracy scope, below the rules, limits it for the whole policy: All answers, Environments (exact labels such as prod; an answer with no label is out of scope) or Release runs (only answers logged with run=). Click Save to draft.
  • Inside the scope, each rule's share decides how many of its answers are double-checked.

The reasoning model reads only the claims the standard checks can't settle, never the whole answer.

Examples

GoalRuleThen
Claim letters are reviews, not Q&AAgent is letter-writerDocument analysis
Double-check everything in productionScope Environments: prod; default rule share 100%Max-Accuracy on production only
Score a tenth of a busy FAQ botAgent is faq-botSample % 10
Judge a pilot customer's traffic separatelyMetadata tenant is acme-insuranceIts own answer type and share
Don't double-check test trafficTag is not syntheticShare 100%

The same rules in the policy file (where to paste it):

max_accuracy:
  enabled: true
  scope: {kind: environments, environments: [prod]}
rules:
  - id: letters
    name: Claim letters are reviews
    applies_to:
      - {field: agent, op: is, values: [letter-writer]}
    answer_type: document_analysis
    sample_pct: 100
    max_accuracy_pct: 100
  - id: faq-sample
    name: Score a tenth of the FAQ bot
    applies_to:
      - {field: agent, op: is, values: [faq-bot]}
    answer_type: answers_from_sources
    sample_pct: 10
    max_accuracy_pct: 0
  - id: pilot-tenant
    name: Pilot customer
    applies_to:
      - {field: metadata.tenant, op: is, values: [acme-insurance]}
      - {field: tag, op: is_one_of, values: [web, mobile]}
    answer_type: answers_from_sources
    sample_pct: 100
    max_accuracy_pct: 50
  - id: default
    name: Everything else (default)
    applies_to: []
    answer_type: answers_from_sources
    sample_pct: 100
    max_accuracy_pct: 100

In the file, the fields are environment, agent, tag and metadata.<key>; the operators is, is_not and is_one_of. Each rule has a short lower-case id that never changes, so answers keep pointing at it after you rename the rule.

What you'll see

An answer from the letter-writer shows Scored by rule Claim letters are reviews in its Summary, and its claims are judged as an analysis (introductions not counted). An answer from the FAQ bot outside the sample shows Not scored: outside its rule's sample.

Environment conditions

An answer carries an environment when your code passes environment= to observe() or log(), or sets it once with variably.set_context(environment=...), which also covers the Google ADK and LangGraph integrations and OpenTelemetry set up with instrument(). An app exporting OpenTelemetry its own way sets the standard resource attribute deployment.environment.name. An answer with no environment never matches an environment condition and falls through to the next rule.

Common mistakes

You seeWhyFix
A value shows "not seen yet"No recent answer had that valueCheck spelling and case: prod is not Prod. Keep it if the traffic hasn't started yet
Match preview says 0The conditions match nothing in the windowWiden to Last 90 days; check each condition
A narrow rule never scores anythingA broader rule above it matches firstMove the narrow rule up in the YAML tab
"Max-Accuracy is off for this policy" when savingA share above 0 while Max-Accuracy is offSwitch Max-Accuracy on in Summary first
"the last rule must apply to all events" (YAML)The last rule has conditionsKeep a last rule with applies_to: []