Human review
Have people check answers, set queue rules, and measure how often they agree with the scorer.
Automatic scores are only worth trusting if they agree with the people who know your business. Human review puts answers in front of a colleague (a claims adjuster, a support lead, an accountant), records their verdict, and measures how often they agree with the scorer. Disagreements tell you where the scoring needs your data definitions or rules, and where your AI is wrong in ways no check catches yet.
Open Evaluation › Human review. Its tabs are Queue (To review, Reviewed, Skipped), Queue rules and Agreement.
How answers get into the queue

- By hand: Send to review in Live events (tick rows), Sessions (every scored step of a session) or an answer's details.
- Automatically: a queue rule sends a sample of newly scored answers.
An answer is queued at most once.
Review an answer

- Open an item in To review and click Assign to me, so two people don't review the same answer.
- For each check, give Your verdict: Not judged, Pass or Fail. Judge only the checks you can: Only checks you judge count toward per-check agreement.
- For each claim, Agree or Disagree with its category, with an optional Note.
- Give Your verdict on the answer: Pass or Fail, with a Note (optional) saying why.
- Click Record verdict, or Skip (skipped items take no verdict).
- After Verdict recorded, Add to dataset with this as the expected answer keeps the case for release tests, with the corrected answer.
The reviewer's verdict never changes the score. It is shown in the answer's History tab under Human verdicts, next to the scorer's.
Example. The scorer passed an answer saying "your claim will be paid within 10 days". The adjuster knows the service level is 10 business days and marks the answer Fail with that note. That disagreement is the cue to add a business rule (see Business rules).
Queue rules

A queue rule sends a sample of newly scored answers to review, so a reviewer always has a fair mix.
- Open Queue rules and click Add rule (New queue rule).
- Fill in a Name, and optionally: Fails this check (or Only answers failing a check, any check), Agent (exact), Environment, Overall below (0–100, optional).
- Set Sample (%), a whole number from 1 to 100.
- Tick Rule on and click Save.
Rules are tried oldest first, and an answer is queued at most once.
| Goal | Rule |
|---|---|
| A reviewer sees 2% of all production answers | Environment prod · Sample 2 |
| Every answer the scorer says took the wrong document | Fails this check Right version of the source · Sample 100 |
| Low scorers from the letter-writer | Agent (exact) letter-writer · Overall below 60 · Sample 25 |
Agreement

How often reviewers and the scorer reach the same verdict, over the last 30, 90 or 365 days:
- Answer verdicts: overall pass or fail, with the matrix of Reviewer pass / Reviewer fail against Scorer pass / Scorer fail.
- Claims: how often reviewers agreed with each claim's category.
- By check: agreement per check, counting only checks reviewers judged.
Each shows Pairs compared, the agreement rate and Cohen's kappa: agreement corrected for the agreement two raters would reach by chance. 1 is perfect agreement, 0 is no better than chance. It is shown only once there are enough pairs.
Reading it. 92% agreement with kappa 0.30 usually means almost every answer passes and both sides say pass: the agreement is mostly luck. Look at the disagreements in the matrix; they tell you what to fix.
Common mistakes
| Mistake | What happens | Fix |
|---|---|---|
| Reviewing only failing answers | Agreement looks worse than it is: the easy passes are missing | Add a small sample of all answers too |
| Marking every check Pass to finish faster | Per-check agreement is meaningless | Leave checks you can't judge as Not judged |
| Two people reviewing the same item | Wasted time | Assign to me first |