All docs
Evaluation policyData definitionsData definitionsdashboard

Data definitions

What each data definition is, the check it turns on, and the three ways to add one: the app, the policy YAML and the SDK.

What you'll have

You know which definition each of your checks needs, where to set it, and how to verify it before it scores real answers.

Data definitions are part of the project's evaluation policy, under evaluation_config. Like every policy change, an edit goes into the draft and scores answers once you publish it. Every definition is validated when you save it, with the same rules the scorer uses, so a definition that was saved always works when an answer is scored.

In plain words, before you start

Out of the box, variA/Bly checks answers against the documents they were given. It can't know your business's own facts: that "NJ" and "New Jersey" are the same state, that the 2025 benefits guide no longer applies, that physical therapy is capped at 12 visits, or that your claims table must always be filtered by plan year. Data definitions are where you tell it. Each one turns on a check that compares exact values, so a failure is a fact ("the answer approves 16 visits; the rule allows at most 12"), not an opinion.

You don't need to write code. Each definition has its own screen under Evaluation policy › Advanced settings ▸ › Data definitions. Some screens are forms (Add list, Add rule); others take a block of text in JSON or YAML, two common ways of writing settings as text:

  • JSON uses braces and quotes: {"name": "plan_year", "kind": "period"}. The Data definitions editors take JSON.
  • YAML uses indentation: name: plan_year on its own line. The policy's YAML tab and the variably policy command take YAML.

The examples on these pages are mostly YAML, because it is easier to read; the editors' Insert example and Start from a template buttons give you the same in JSON. Either way you never have to get it right first time: Check reads what you typed exactly as saving would and says which line or key is wrong, and nothing is stored until it is valid. Each screen also has a prompt for your coding agent (Show prompt, Copy prompt) that writes the definition from your own documents.

Then publish: a definition scores answers only after you publish the draft.

The definitions and the checks they power

DefinitionWhat it holdsCheck it powersPage
Code listsYour vocabulary: each code and its synonyms (states, leave types, drug or procedure codes)Used by hand-off contracts (code fields) and by document scope (jurisdiction, product, your own text dimensions)Code lists
Document scopeWhat makes a document apply: dates in force, plan or fiscal year, jurisdiction, product, versionRight version of the sourceDocument scope
Hand-off contractsPer receiving agent, the typed fields a hand-off must carryHand-offs carry the factsHand-off contracts
SQL checksYour database dialect, schema (names only), query rules and the unit of percentage columnsSQL follows the rules, and figures checked against the rowsSQL checks
Posting rulesWhich documents must be posted, with which debit and credit accounts, and which must never beJournal findings on the answer's claims (reason "Journal entry breaks a rule")Posting rules
Chart of accountsThe ledger accounts your company books toThe same journal findings: an entry to an account not on the chart is proven wrongChart of accounts
Business rulesFacts read from each answer and the rules they must obey, in any domainFollows business rulesBusiness rules
Expected answersSent with each answer, not stored in the policyMatches the expected answer, Answers what was askedExpected answers
Requirement templatesNamed, versioned lists of what an answer must addressCovers what it mustRequirement templates

What a check shows without its definition

A check that has nothing to compare never passes by default.

CheckWithout its definition
SQL follows the rulesNot configured on every answer that sent SQL: "SQL was sent, but no SQL schema and rules are set for this project. Add them in Evaluation policy › Data definitions › SQL checks."
Right version of the sourceNeeds no definition (the presets apply). N/A: "No reference carried scope metadata (effective dates, plan or fiscal year, jurisdiction, product, version), or the question states no scope."
Hand-offs carry the factsNot configured on a session whose hand-offs carried fields or a message to an agent with no contract, when nothing else about them could be checked: "A hand-off was sent to <agent>, but no hand-off contract is set for it in this project. Add one in Evaluation policy › Data definitions › Hand-off contracts." A session with no hand-off, or one that carried nothing (the receiver reads the shared history), is N/A. A hand-off written as text is still grounded in what the sender had without a contract.
Matches the expected answerNeeds no definition. N/A: "No expected answer was sent (expected_answer=), or the answer does not say what it says."
Covers what it mustNot configured when the answer names a template (requirements_ref) the project does not have: "Requirements were named (requirements_ref "<name>"), but this project has no requirement template "<name>". Add it in Evaluation policy › YAML under requirement_templates; it is then listed in Data definitions › Requirement templates." No requirements sent is N/A: "No requirements were given for this answer (required_checks or requirements_ref)."
Follows business rulesNot configured: "No business rules are set for this project. Add them in Evaluation policy › Data definitions › Business rules."
Posting rules and chart of accountsNo finding is added. The bookkeeping checks that need no definition (unbalanced, outside the stated period, posted twice, amount not in the documents) still run.

N/A and Not configured are never counted as a pass.

Three ways to add a definition

DefinitionIn the appPolicy YAMLSDK settings call
Code listsAdd list, Import CSVcode_listsset_code_lists
Document scopeAdd dimension; When the question names no datescope_defaultsset_scope_defaults
Hand-off contractsAdd contract (JSON, with Insert example)handoff_contractsset_handoff_contracts
SQL checksJSON editor with Insert example and Checksql_checksset_sql_checks
Posting rulesAdd rule, Try itposting_rulesset_posting_rules
Chart of accountsImport CSV (Check, then import)chart_of_accountsset_chart_of_accounts, variably policy chart import
Business rulesJSON editor, Start from a template, Check, Try itbusiness_rulesset_business_rules
Requirement templatesAdd template (JSON)requirement_templatesset_requirement_templates
Date order (dates written 04/03/2026)none: in the YAMLdate_ordernone
Expected answersnothing to setnothing to setlog(expected_answer=...) per answer

Each item the app lists has Edit and Remove. Anything else, edit in the YAML.

In the app

  1. Open Evaluation policy › Advanced settings ▸ › Data definitions and pick the sub-tab.
  2. Use the tab's button (Add list, Add contract, Insert example, Add rule). Replace the example's names with yours.
  3. Save it: Add, Import or Save to draft. The app confirms "Saved to draft vN". A refusal shows Not saved. with the message, and nothing is stored.
  4. Publish: Publish vN… at the top of the policy, then Publish. Answers are scored with the definition from then on; earlier answers keep their verdicts unless you re-score.

In the policy YAML

  1. Open Evaluation policy › YAML, or run variably policy pull to write the version in force to variably-policy.yaml.
  2. Add or edit the definition under evaluation_config (the shapes are on each page and in the format reference).
  3. Check it without saving: Check in the YAML tab (and Compare with to see what it changes), or variably policy validate variably-policy.yaml and variably policy diff variably-policy.yaml. The file is validated exactly as a save would be; an invalid file is refused with its line and changes nothing.
  4. Save it as the draft: Save as draft (with a Change note), or variably policy push variably-policy.yaml --note "why". Saving replaces the whole draft with the file.
  5. Publish: Publish vN…, or variably policy publish --note "why".
evaluation_config:
  code_lists: {}            # Code lists
  scope_defaults: {}        # Document scope
  handoff_contracts: []     # Hand-off contracts
  sql_checks: {}            # SQL checks
  posting_rules: []         # Posting rules
  # chart_of_accounts: a list of {code, name, type}; leave the key out for none ([] is refused)
  requirement_templates: {} # Requirement templates
  business_rules: {}        # Business rules
  date_order: MDY           # How all-numeric dates are written: MDY or DMY

Leave out a key you do not use. Comments are not kept: the server stores the content, not the text.

From the SDK

The settings calls validate and save one definition, and each publishes a new policy version with your note. Their first argument is the id shown under Project configuration › Integration & keys as Project id for SDK settings (the SDK names it experiment_id). The API key needs the Change the evaluation policy permission.

import os
from variably import VariablyClient

client = VariablyClient({"api_key": os.environ["VARIABLY_API_KEY"],
                         "base_url": os.environ["VARIABLY_BASE_URL"]})   # https://api.variably.dev
pid = "your-project-id-for-sdk-settings"

# Lists first: a contract's code field must name a list that exists.
client.set_code_lists(pid, "settings/code_lists.json", note="NJ and NY leave types")
client.set_handoff_contracts(pid, "settings/handoff_contracts.json", note="intake to adjudication")
client.set_scope_defaults(pid, "settings/scope_defaults.yaml")
client.set_sql_checks(pid, "settings/sql_checks.json")
client.set_posting_rules(pid, "settings/posting_rules.yaml", note="supplier invoices")
client.set_chart_of_accounts(pid, "settings/chart_of_accounts.csv", note="FY2026 chart")
client.set_business_rules(pid, "settings/business_rules.yaml", note="visit limit 12")
client.set_requirement_templates(pid, "settings/requirement_templates.yaml", note="benefit answers")

Each call takes a dict, a list or the path of a JSON or YAML file (YAML needs PyYAML). An empty value removes the definition: {} for code lists, scope and SQL checks, [] for contracts. From CI:

python -m variably.settings push --experiment-id "$PID" \
  --code-lists settings/code_lists.json \
  --handoff-contracts settings/handoff_contracts.json \
  --scope-defaults settings/scope_defaults.json \
  --sql-checks settings/sql_checks.json \
  --business-rules settings/business_rules.yaml \
  --note "$GIT_COMMIT_MESSAGE"

Keep the policy in your repository: paste this into your coding agent

Paste this into Claude Code, Cursor, or whichever agent has your repository open. It adds the policy file to your repository and a CI job that validates it on every change.

Prompt for your coding agent
Keep this project's variA/Bly evaluation policy in this repository and check it in CI, so a
change to how answers are scored is reviewed like code.

Look first:
1. Find the CI system this repository uses (GitHub Actions, GitLab CI, CircleCI, another) and
   whether a variably-policy.yaml already exists. Tell me before you change anything.

Then:
2. The variably command comes with the SDK (pip install variably-sdk). It reads
   VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. In CI, read both from the CI
   system's secrets. Never hard-code the key or write it into a file.
3. If the file is missing, run `variably policy pull` to write the version in force to
   variably-policy.yaml, and add it to the repository. Comments in the file are not kept by
   the server, so put explanations in commit messages, not in the file.
4. Add a CI job on every change to variably-policy.yaml that runs:
     variably policy validate variably-policy.yaml   (exit 0 valid, 1 invalid, 2 could not ask)
     variably policy diff variably-policy.yaml       (exit 0 no changes, 1 changes, 2 error)
   validate failing must fail the job. diff exiting 1 only means the file changes the policy:
   print its output, and fail the job only on exit 2.
5. Add a job on the main branch that saves the file as the draft:
     variably policy push variably-policy.yaml --note "<commit sha and subject>"
   That job needs a separate key allowed to change the evaluation policy; read it from its own
   CI secret into VARIABLY_API_KEY for that step only.
6. Do not publish from CI unless I ask for it. A draft scores nothing until a policy admin
   publishes it, in the app or with `variably policy publish --note "<why>"`.

When you are done, tell me the files you added and the secrets I need to create. Verify:
`variably policy validate variably-policy.yaml` exits 0 locally, and the CI job passes on a
branch that changes nothing in the file.

What is checked when you save

  • The shape. Unknown keys, wrong types, empty names and over-long lists are refused, naming the item: Rule 'Supplier invoice': action must be 'post' or 'do_not_post'.
  • The definitions against each other. A code field must name a list that exists (code list "leave_type" is not declared in code_lists.). Removing a list that a saved contract or scope dimension uses is refused: "the saved hand-off contracts would no longer be valid: …". A contract that names a requirement template must name one that exists.
  • Nothing is half-saved. A refused change stores nothing. A YAML file is refused with the key path and line: evaluation_config.posting_rules (line 41): ….
  • Settings only a policy holds. date_order and chart_of_accounts belong to the policy; an experiment's own configuration that sets them is refused: "evaluation_config.date_order is a setting of the project's evaluation policy: set it in a policy draft and publish it".

Date order

All-numeric dates such as 04/03/2026 are read only when you say how your project writes them. Without date_order they are never guessed: such a date in an expected answer is not read, and in a hand-off it is "untraced" (neither a pass nor a failure).

Value04/03/2026 means
MDYApril 3, 2026
DMY4 March 2026

Year-first dates (2026-04-03) are always year, month, day, so there is no YMD. Setting YMD is refused: ""YMD" is not a setting: a year-first date (2026-03-04) is always read year, month, day. Use "DMY" (04/03/2026 is 4 March) or "MDY" (04/03/2026 is April 3) for how day and month are written". A hand-off date field can override it with its own date_order.


Concepts on this page: Data definitions