All docs
Test before releaseCollection rulesdashboard

Collection rules

Copy live answers into a dataset automatically, for example 10% of production answers scoring below 70.

A collection rule copies live answers into a dataset automatically, after they are scored. It is how a dataset keeps growing with the cases that matter, without anyone copying them by hand.

Example. "Copy 10% of production answers scoring below 70 into regressions." A month later the dataset holds a few hundred real, hard cases, and every release is tested on them.

Add a rule

Collection rulesCollection rules

A collection ruleA collection rule

You need a dataset first: the rule fills one.

  1. Open Datasets & runs › Collection rules and click Add rule.
  2. Give it a Name and choose the dataset in Into dataset.
  3. Under Collect answers where, add conditions: the same fields and operators as scoring rules (environment, agent, tag, metadata). No condition means every live answer.
  4. Optionally choose Only answers failing this check, for example Right version of the source. It uses the checks as Live events shows them once the answer is scored. Any or none means no check filter.
  5. Optionally fill Only when overall is below (0–100, optional), for example 70.
  6. Set Sample (% of matching answers): 10 copies one in ten.
  7. Choose Duplicates: One item per question (the same question is collected once) or Keep every answer.
  8. Tick Rule on and click Save.

From an issue, Issues › Collect into dataset opens this form with the check already filled in.

Examples

GoalCollect answers whereCheckOverall belowSampleDuplicates
Keep production failures for regression testsEnvironment is prodAny or none7010%One item per question
Every wrong-document answer of the benefits agentAgent is benefits-agentRight version of the source100%One item per question
A sample of normal traffic, as a baselineTag is webAny or none1%Keep every answer

What you'll see

Each rule's row shows Collects, Into, Sample and how many it has Collected. In the dataset, collected items show where they came from. Answers logged for a dataset run are never collected, so a release test can't feed itself.

Remove rule stops collecting; the items it already collected stay in the dataset.

Common mistakes

You seeWhyFix
Add rule is disabled: Create a dataset firstThere is no dataset to fillCreate one in Datasets
The dataset fills up with the same questionDuplicates is Keep every answerChoose One item per question
Nothing is collectedThe conditions match no new answers, or the rule is offCheck Rule on, and the conditions' spelling and case
Items collected without an expected answerLive answers have no "right answer"Add expected answers by reviewing them in Human review