All docs
Evaluation policyData definitionsRequirement templatesdashboard

Requirement templates

Requirement templates power "Covers what it must": what a support, claims or clinical answer must cover, critical items, versions you can pin, with examples, edge cases and the format.

What you'll have

An answer that leaves out a critical item, or states it without support in the sources, fails, and you see which item.

What it is and what it powers

Every grounding check judges what an answer says; none can see what it should have said and did not. A requirement is an explicit "the answer must address this". It powers Covers what it must: "Every critical requirement given for this answer (inline or by template) is addressed, and the statement that addresses it is grounded in the documents."

  • Addressed: when a requirement names a value (an amount, quantity, percent, date, duration), the answer must state that value, decided in code ("2,000 mg" and "two thousand mg" are the same; "1,000 mg" is not). Otherwise the answer must contain a statement that supports it.
  • Grounded: the statement that addresses it is supported by the sources.
  • Action requirements: the run made that tool call successfully (tools_called).

Pass when every critical requirement is addressed and grounded. Non-critical misses are reported, never failing. Fail when a critical one is missing, or addressed by a statement the sources do not support. Warning when a critical one is addressed but its grounding could not be checked.

When it's not set. N/A: "No requirements were given for this answer (required_checks or requirements_ref)." A template (or version) the answer names that the project does not have is Not configured, with what to add; one that could not be read is N/A with the reason. The answer is never judged against a guess.

Use cases

Customer support: what a refund answer must cover

evaluation_config:
  requirement_templates:
    refund-policy:
      version: 2
      description: Every refund answer
      requirements:
        - {id: window, text: "States the 30-day refund window", critical: true}
        - {id: method, text: "Says the refund goes to the original payment method", critical: true}
        - {id: timing, text: "Says refunds take 5 to 10 business days"}
        - {id: ticket, text: create_refund_ticket, kind: action}
client.log(prompt=q, response=answer, reference_materials=docs, requirements_ref="refund-policy")

Insurance: a benefits answer

requirement_templates:
  std-benefit:
    version: 1
    requirements:
      - {id: pct, text: "States the benefit as 66.67% of weekly earnings", critical: true}
      - {id: wait, text: "States the 14-day waiting period", critical: true}
      - {id: year, text: "Names the plan year the answer applies to", critical: true}

"66.67%" and "14-day" are values, so they are checked in code against the answer.

Data-model validation summaries

requirement_templates:
  model-validation-summary:
    version: 1
    requirements:
      - {id: tables, text: "States how many tables were validated", critical: true}
      - {id: mismatches, text: "Names every column with a type mismatch", critical: true}
      - {id: snapshot, text: "States the snapshot date", critical: true}

This checks that the summary covers each item and that each statement is supported by what the agent had. It does not compare the data models themselves.

Inline requirements for one answer

client.log(prompt=q, response=answer, reference_materials=docs,
           required_checks=[{"id": "renal", "text": "Checks kidney function before starting", "critical": True}])

Give required_checks or requirements_ref, not both.

Versions

requirements_ref="refund-policy" uses the current version; "refund-policy@v2" pins exactly version 2. A version's meaning must never change: when you change a template's requirements or description, raise its version. The settings history keeps every earlier version, so a pinned reference still resolves after you change or remove the template. Where the saved version can be compared, a change that keeps the version is refused: "Requirement template "refund-policy" changed but kept version 2; raise the version so a pinned refund-policy@v2 keeps its meaning".

Edge cases

SituationWhat happens
A non-critical requirement is missedReported on a passing answer (borderline), never a failure.
Addressed, but no sources were sentWarning: grounding could not be checked; never counted as met.
A template name that does not existN/A for that answer, with the reason. A hand-off contract naming one is refused when saved.
A pin to a version above the current oneRefused for a contract: Hand-off contract "refunds" pins refund-policy@v3; the template is at v2.
A value written another wayCompared as typed values: "two thousand mg (2000 mg)" addresses "2,000 mg".

How to add it

Data definitions: Requirement templatesData definitions: Requirement templates

  1. Open Evaluation policy › YAML (the Requirement templates sub-tab lists templates; it does not edit them).
  2. Add the template under evaluation_config.requirement_templates, with version: 1.
  3. Compare to check it, Apply as draft, then Publish vN… › Publish.
  4. Send requirements_ref="name" (or "name@v1") with each answer. For a hand-off, put requirements_ref on the contract.

Paste this into your coding agent

Paste this into Claude Code, Cursor, or whichever agent has your repository open. It writes templates from the checklists already in your prompts, runbooks and tests.

Prompt for your coding agent
Write variA/Bly requirement templates for this project: named, versioned lists of what an
answer must address (a protocol, a runbook, a disclosure checklist), so an answer that leaves out
something it must say is caught.

Look first:
1. Find the checklists the app's answers must follow: system prompts that list what to
   include, compliance or disclosure text, runbooks, protocol documents, tests that assert an
   answer mentions something. Show me each with its file path, and ask me which become
   templates, before you write anything.

Then write them under evaluation_config.requirement_templates:

evaluation_config:
  requirement_templates:
    refund-policy:
      version: 1
      description: Every refund answer
      requirements:
        - {id: window, text: "States the 30-day refund window", critical: true}
        - {id: method, text: "Says the refund goes to the original payment method", critical: true}
        - {id: timing, text: "Says refunds take 5 to 10 business days"}

- critical: true for what must never be missing; the others are reported, never failing.
- A requirement that names a value (an amount, a dose, a date) is met only when the answer
  states that value.
- A template's version never changes in place: to change a template that is already
  published, raise its version.
- The format is described at https://www.variably.dev/docs/data-requirement-templates.

How to write it, as policy YAML:
1. The variably command comes with the SDK (pip install variably-sdk). It reads
   VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment; never hard-code the key or write it into a file.
2. If variably-policy.yaml is not in the repository yet, run `variably policy pull`: it writes
   the version in force to variably-policy.yaml. Change only evaluation_config.requirement_templates in that
   file and leave every other key exactly as it is.
3. Verify: run `variably policy validate variably-policy.yaml`. It checks the file as the server would
   and saves nothing; each problem is printed as "line N: path: message". Fix every problem and
   run it again until it exits 0. Do not guess a key the error does not name: unknown keys are
   refused.
4. Run `variably policy diff variably-policy.yaml` and show me what would change.
5. Do not push or publish. Tell me the two ways to save it as a draft: paste the file into
   Evaluation policy › YAML and click Apply as draft, or run
   `variably policy push variably-policy.yaml --note "<why>"` with a key allowed to change the
   evaluation policy. A draft scores nothing until it is published.

6. Name the template on each answer it applies to: the observe() or client.log() call passes
   requirements_ref="refund-policy" (or "refund-policy@v2" for one version). Add it where the
   app already knows which kind of answer it is producing.

When you are done, tell me each template, the file it came from, and which answers now name it.
Verify: after publishing, an answer that names the template shows "Covers what it must" in its
Checks tab in Live events.

What you'll see

An answer sent with requirements_ref="benefit-answer" that gives the amount and the plan year but not the network tier shows Covers what it must failed, with a detail starting "Critical: 1 of 2 met" and the missing requirement named. An answer that names a template the project doesn't have shows Not configured.

Common mistakes

MistakeWhat happensFix
Naming a template in requirements_ref that isn't publishedNot configuredAdd the template and publish, or fix the name
Every requirement marked criticalAnswers fail on details that don't matterMark critical only what must never be missing
Changing a template's requirements but keeping its versionRefused where the saved version can be compared: a pinned name@v1 must keep its meaningRaise version, and send name@v2 when you mean it

Format reference

requirement_templates is an object: template name → template.

FieldTypeRequiredAllowed valuesDefaultMeaning
name (key)textyeslower-case letters, digits and _.-, up to 64noneWhat requirements_ref names
versionwhole numberyes1 to 999999noneRaise it on every change
descriptiontextnoup to 1,000 charactersnoneYour note; shown as Applies to
requirementslistyes1 to 100noneWhat the answer must address

A requirement (also the shape of required_checks):

FieldTypeRequiredAllowed valuesDefaultMeaning
idtextyes1–64 letters, digits or _.:-; uniquenoneHow findings name it
texttextyesup to 500 charactersnoneWhat must be addressed; for action, the tool's name
criticaltrue / falsenofalseA miss fails the check
kindtextnocontent, actioncontentaction: the tool call must succeed

Up to 200 templates. Minimal:

requirement_templates:
  refund-policy:
    version: 1
    requirements:
      - {id: window, text: "States the 30-day refund window", critical: true}

Validation messages

ProblemMessage
Bad nameRequirement template "Refund Policy": names are lower-case letters, digits and _.- (at most 64)
VersionRequirement template "refund-policy": version is a whole number from 1
EmptyRequirement template "refund-policy": requirements must be a non-empty list
Bad idRequirement template "refund-policy": required_checks[0]: id must be 1-64 letters, digits or _.:-
Duplicate idrequired_checks: id "window" declared twice
Kindrequired_checks[window]: kind is content or action
Both inline and template on an answerrequired_checks and requirements_ref: give one — the requirements inline, or the template that holds them
Ref formatrequirements_ref "Refund@2": a template name, optionally @v<version> (e.g. metformin-initiation@v3)

Concepts on this page: Data definitions