Requirement templates
Requirement templates power "Covers what it must": what a support, claims or clinical answer must cover, critical items, versions you can pin, with examples, edge cases and the format.
An answer that leaves out a critical item, or states it without support in the sources, fails, and you see which item.
What it is and what it powers
Every grounding check judges what an answer says; none can see what it should have said and did not. A requirement is an explicit "the answer must address this". It powers Covers what it must: "Every critical requirement given for this answer (inline or by template) is addressed, and the statement that addresses it is grounded in the documents."
- Addressed: when a requirement names a value (an amount, quantity, percent, date, duration), the answer must state that value, decided in code ("2,000 mg" and "two thousand mg" are the same; "1,000 mg" is not). Otherwise the answer must contain a statement that supports it.
- Grounded: the statement that addresses it is supported by the sources.
- Action requirements: the run made that tool call successfully (
tools_called).
Pass when every critical requirement is addressed and grounded. Non-critical misses are reported, never failing. Fail when a critical one is missing, or addressed by a statement the sources do not support. Warning when a critical one is addressed but its grounding could not be checked.
When it's not set. N/A: "No requirements were given for this answer (required_checks or requirements_ref)." A template (or version) the answer names that the project does not have is Not configured, with what to add; one that could not be read is N/A with the reason. The answer is never judged against a guess.
Use cases
Customer support: what a refund answer must cover
evaluation_config:
requirement_templates:
refund-policy:
version: 2
description: Every refund answer
requirements:
- {id: window, text: "States the 30-day refund window", critical: true}
- {id: method, text: "Says the refund goes to the original payment method", critical: true}
- {id: timing, text: "Says refunds take 5 to 10 business days"}
- {id: ticket, text: create_refund_ticket, kind: action}
client.log(prompt=q, response=answer, reference_materials=docs, requirements_ref="refund-policy")
Insurance: a benefits answer
requirement_templates:
std-benefit:
version: 1
requirements:
- {id: pct, text: "States the benefit as 66.67% of weekly earnings", critical: true}
- {id: wait, text: "States the 14-day waiting period", critical: true}
- {id: year, text: "Names the plan year the answer applies to", critical: true}
"66.67%" and "14-day" are values, so they are checked in code against the answer.
Data-model validation summaries
requirement_templates:
model-validation-summary:
version: 1
requirements:
- {id: tables, text: "States how many tables were validated", critical: true}
- {id: mismatches, text: "Names every column with a type mismatch", critical: true}
- {id: snapshot, text: "States the snapshot date", critical: true}
This checks that the summary covers each item and that each statement is supported by what the agent had. It does not compare the data models themselves.
Inline requirements for one answer
client.log(prompt=q, response=answer, reference_materials=docs,
required_checks=[{"id": "renal", "text": "Checks kidney function before starting", "critical": True}])
Give required_checks or requirements_ref, not both.
Versions
requirements_ref="refund-policy" uses the current version; "refund-policy@v2" pins exactly version 2. A version's meaning must never change: when you change a template's requirements or description, raise its version. The settings history keeps every earlier version, so a pinned reference still resolves after you change or remove the template. Where the saved version can be compared, a change that keeps the version is refused: "Requirement template "refund-policy" changed but kept version 2; raise the version so a pinned refund-policy@v2 keeps its meaning".
Edge cases
| Situation | What happens |
|---|---|
| A non-critical requirement is missed | Reported on a passing answer (borderline), never a failure. |
| Addressed, but no sources were sent | Warning: grounding could not be checked; never counted as met. |
| A template name that does not exist | N/A for that answer, with the reason. A hand-off contract naming one is refused when saved. |
| A pin to a version above the current one | Refused for a contract: Hand-off contract "refunds" pins refund-policy@v3; the template is at v2. |
| A value written another way | Compared as typed values: "two thousand mg (2000 mg)" addresses "2,000 mg". |
How to add it

- Open Evaluation policy › YAML (the Requirement templates sub-tab lists templates; it does not edit them).
- Add the template under
evaluation_config.requirement_templates, withversion: 1. - Compare to check it, Apply as draft, then Publish vN… › Publish.
- Send
requirements_ref="name"(or"name@v1") with each answer. For a hand-off, putrequirements_refon the contract.
Paste this into your coding agent
Paste this into Claude Code, Cursor, or whichever agent has your repository open. It writes templates from the checklists already in your prompts, runbooks and tests.
Write variA/Bly requirement templates for this project: named, versioned lists of what an
answer must address (a protocol, a runbook, a disclosure checklist), so an answer that leaves out
something it must say is caught.
Look first:
1. Find the checklists the app's answers must follow: system prompts that list what to
include, compliance or disclosure text, runbooks, protocol documents, tests that assert an
answer mentions something. Show me each with its file path, and ask me which become
templates, before you write anything.
Then write them under evaluation_config.requirement_templates:
evaluation_config:
requirement_templates:
refund-policy:
version: 1
description: Every refund answer
requirements:
- {id: window, text: "States the 30-day refund window", critical: true}
- {id: method, text: "Says the refund goes to the original payment method", critical: true}
- {id: timing, text: "Says refunds take 5 to 10 business days"}
- critical: true for what must never be missing; the others are reported, never failing.
- A requirement that names a value (an amount, a dose, a date) is met only when the answer
states that value.
- A template's version never changes in place: to change a template that is already
published, raise its version.
- The format is described at https://www.variably.dev/docs/data-requirement-templates.
How to write it, as policy YAML:
1. The variably command comes with the SDK (pip install variably-sdk). It reads
VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment; never hard-code the key or write it into a file.
2. If variably-policy.yaml is not in the repository yet, run `variably policy pull`: it writes
the version in force to variably-policy.yaml. Change only evaluation_config.requirement_templates in that
file and leave every other key exactly as it is.
3. Verify: run `variably policy validate variably-policy.yaml`. It checks the file as the server would
and saves nothing; each problem is printed as "line N: path: message". Fix every problem and
run it again until it exits 0. Do not guess a key the error does not name: unknown keys are
refused.
4. Run `variably policy diff variably-policy.yaml` and show me what would change.
5. Do not push or publish. Tell me the two ways to save it as a draft: paste the file into
Evaluation policy › YAML and click Apply as draft, or run
`variably policy push variably-policy.yaml --note "<why>"` with a key allowed to change the
evaluation policy. A draft scores nothing until it is published.
6. Name the template on each answer it applies to: the observe() or client.log() call passes
requirements_ref="refund-policy" (or "refund-policy@v2" for one version). Add it where the
app already knows which kind of answer it is producing.
When you are done, tell me each template, the file it came from, and which answers now name it.
Verify: after publishing, an answer that names the template shows "Covers what it must" in its
Checks tab in Live events.What you'll see
An answer sent with requirements_ref="benefit-answer" that gives the amount and the plan year but not the
network tier shows Covers what it must failed, with a detail starting "Critical: 1 of 2 met" and the missing
requirement named. An answer that names a template the project doesn't have shows Not configured.
Common mistakes
| Mistake | What happens | Fix |
|---|---|---|
Naming a template in requirements_ref that isn't published | Not configured | Add the template and publish, or fix the name |
| Every requirement marked critical | Answers fail on details that don't matter | Mark critical only what must never be missing |
Changing a template's requirements but keeping its version | Refused where the saved version can be compared: a pinned name@v1 must keep its meaning | Raise version, and send name@v2 when you mean it |
Format reference
requirement_templates is an object: template name → template.
| Field | Type | Required | Allowed values | Default | Meaning |
|---|---|---|---|---|---|
| name (key) | text | yes | lower-case letters, digits and _.-, up to 64 | none | What requirements_ref names |
version | whole number | yes | 1 to 999999 | none | Raise it on every change |
description | text | no | up to 1,000 characters | none | Your note; shown as Applies to |
requirements | list | yes | 1 to 100 | none | What the answer must address |
A requirement (also the shape of required_checks):
| Field | Type | Required | Allowed values | Default | Meaning |
|---|---|---|---|---|---|
id | text | yes | 1–64 letters, digits or _.:-; unique | none | How findings name it |
text | text | yes | up to 500 characters | none | What must be addressed; for action, the tool's name |
critical | true / false | no | false | A miss fails the check | |
kind | text | no | content, action | content | action: the tool call must succeed |
Up to 200 templates. Minimal:
requirement_templates:
refund-policy:
version: 1
requirements:
- {id: window, text: "States the 30-day refund window", critical: true}
Validation messages
| Problem | Message |
|---|---|
| Bad name | Requirement template "Refund Policy": names are lower-case letters, digits and _.- (at most 64) |
| Version | Requirement template "refund-policy": version is a whole number from 1 |
| Empty | Requirement template "refund-policy": requirements must be a non-empty list |
| Bad id | Requirement template "refund-policy": required_checks[0]: id must be 1-64 letters, digits or _.:- |
| Duplicate id | required_checks: id "window" declared twice |
| Kind | required_checks[window]: kind is content or action |
| Both inline and template on an answer | required_checks and requirements_ref: give one — the requirements inline, or the template that holds them |
| Ref format | requirements_ref "Refund@2": a template name, optionally @v<version> (e.g. metformin-initiation@v3) |
Concepts on this page: Data definitions