Collection rules
Copy live answers into a dataset automatically, for example 10% of production answers scoring below 70.
A collection rule copies live answers into a dataset automatically, after they are scored. It is how a dataset keeps growing with the cases that matter, without anyone copying them by hand.
Example. "Copy 10% of production answers scoring below 70 into regressions." A month later the dataset holds a few hundred real, hard cases, and every release is tested on them.
Add a rule


You need a dataset first: the rule fills one.
- Open Datasets & runs › Collection rules and click Add rule.
- Give it a Name and choose the dataset in Into dataset.
- Under Collect answers where, add conditions: the same fields and operators as scoring rules (environment, agent, tag, metadata). No condition means every live answer.
- Optionally choose Only answers failing this check, for example Right version of the source. It uses the checks as Live events shows them once the answer is scored. Any or none means no check filter.
- Optionally fill Only when overall is below (0–100, optional), for example 70.
- Set Sample (% of matching answers): 10 copies one in ten.
- Choose Duplicates: One item per question (the same question is collected once) or Keep every answer.
- Tick Rule on and click Save.
From an issue, Issues › Collect into dataset opens this form with the check already filled in.
Examples
| Goal | Collect answers where | Check | Overall below | Sample | Duplicates |
|---|---|---|---|---|---|
| Keep production failures for regression tests | Environment is prod | Any or none | 70 | 10% | One item per question |
| Every wrong-document answer of the benefits agent | Agent is benefits-agent | Right version of the source | 100% | One item per question | |
| A sample of normal traffic, as a baseline | Tag is web | Any or none | 1% | Keep every answer |
What you'll see
Each rule's row shows Collects, Into, Sample and how many it has Collected. In the dataset, collected items show where they came from. Answers logged for a dataset run are never collected, so a release test can't feed itself.
Remove rule stops collecting; the items it already collected stay in the dataset.
Common mistakes
| You see | Why | Fix |
|---|---|---|
| Add rule is disabled: Create a dataset first | There is no dataset to fill | Create one in Datasets |
| The dataset fills up with the same question | Duplicates is Keep every answer | Choose One item per question |
| Nothing is collected | The conditions match no new answers, or the rule is off | Check Rule on, and the conditions' spelling and case |
| Items collected without an expected answer | Live answers have no "right answer" | Add expected answers by reviewing them in Human review |