Data definitions
What each data definition is, the check it turns on, and the three ways to add one: the app, the policy YAML and the SDK.
You know which definition each of your checks needs, where to set it, and how to verify it before it scores real answers.
Data definitions are part of the project's evaluation policy, under evaluation_config. Like every policy change, an edit goes into the draft and scores answers once you publish it. Every definition is validated when you save it, with the same rules the scorer uses, so a definition that was saved always works when an answer is scored.
In plain words, before you start
Out of the box, variA/Bly checks answers against the documents they were given. It can't know your business's own facts: that "NJ" and "New Jersey" are the same state, that the 2025 benefits guide no longer applies, that physical therapy is capped at 12 visits, or that your claims table must always be filtered by plan year. Data definitions are where you tell it. Each one turns on a check that compares exact values, so a failure is a fact ("the answer approves 16 visits; the rule allows at most 12"), not an opinion.
You don't need to write code. Each definition has its own screen under Evaluation policy › Advanced settings ▸ › Data definitions. Some screens are forms (Add list, Add rule); others take a block of text in JSON or YAML, two common ways of writing settings as text:
- JSON uses braces and quotes:
{"name": "plan_year", "kind": "period"}. The Data definitions editors take JSON. - YAML uses indentation:
name: plan_yearon its own line. The policy's YAML tab and thevariably policycommand take YAML.
The examples on these pages are mostly YAML, because it is easier to read; the editors' Insert example and Start from a template buttons give you the same in JSON. Either way you never have to get it right first time: Check reads what you typed exactly as saving would and says which line or key is wrong, and nothing is stored until it is valid. Each screen also has a prompt for your coding agent (Show prompt, Copy prompt) that writes the definition from your own documents.
Then publish: a definition scores answers only after you publish the draft.
The definitions and the checks they power
| Definition | What it holds | Check it powers | Page |
|---|---|---|---|
| Code lists | Your vocabulary: each code and its synonyms (states, leave types, drug or procedure codes) | Used by hand-off contracts (code fields) and by document scope (jurisdiction, product, your own text dimensions) | Code lists |
| Document scope | What makes a document apply: dates in force, plan or fiscal year, jurisdiction, product, version | Right version of the source | Document scope |
| Hand-off contracts | Per receiving agent, the typed fields a hand-off must carry | Hand-offs carry the facts | Hand-off contracts |
| SQL checks | Your database dialect, schema (names only), query rules and the unit of percentage columns | SQL follows the rules, and figures checked against the rows | SQL checks |
| Posting rules | Which documents must be posted, with which debit and credit accounts, and which must never be | Journal findings on the answer's claims (reason "Journal entry breaks a rule") | Posting rules |
| Chart of accounts | The ledger accounts your company books to | The same journal findings: an entry to an account not on the chart is proven wrong | Chart of accounts |
| Business rules | Facts read from each answer and the rules they must obey, in any domain | Follows business rules | Business rules |
| Expected answers | Sent with each answer, not stored in the policy | Matches the expected answer, Answers what was asked | Expected answers |
| Requirement templates | Named, versioned lists of what an answer must address | Covers what it must | Requirement templates |
What a check shows without its definition
A check that has nothing to compare never passes by default.
| Check | Without its definition |
|---|---|
| SQL follows the rules | Not configured on every answer that sent SQL: "SQL was sent, but no SQL schema and rules are set for this project. Add them in Evaluation policy › Data definitions › SQL checks." |
| Right version of the source | Needs no definition (the presets apply). N/A: "No reference carried scope metadata (effective dates, plan or fiscal year, jurisdiction, product, version), or the question states no scope." |
| Hand-offs carry the facts | Not configured on a session whose hand-offs carried fields or a message to an agent with no contract, when nothing else about them could be checked: "A hand-off was sent to <agent>, but no hand-off contract is set for it in this project. Add one in Evaluation policy › Data definitions › Hand-off contracts." A session with no hand-off, or one that carried nothing (the receiver reads the shared history), is N/A. A hand-off written as text is still grounded in what the sender had without a contract. |
| Matches the expected answer | Needs no definition. N/A: "No expected answer was sent (expected_answer=), or the answer does not say what it says." |
| Covers what it must | Not configured when the answer names a template (requirements_ref) the project does not have: "Requirements were named (requirements_ref "<name>"), but this project has no requirement template "<name>". Add it in Evaluation policy › YAML under requirement_templates; it is then listed in Data definitions › Requirement templates." No requirements sent is N/A: "No requirements were given for this answer (required_checks or requirements_ref)." |
| Follows business rules | Not configured: "No business rules are set for this project. Add them in Evaluation policy › Data definitions › Business rules." |
| Posting rules and chart of accounts | No finding is added. The bookkeeping checks that need no definition (unbalanced, outside the stated period, posted twice, amount not in the documents) still run. |
N/A and Not configured are never counted as a pass.
Three ways to add a definition
| Definition | In the app | Policy YAML | SDK settings call |
|---|---|---|---|
| Code lists | Add list, Import CSV | code_lists | set_code_lists |
| Document scope | Add dimension; When the question names no date | scope_defaults | set_scope_defaults |
| Hand-off contracts | Add contract (JSON, with Insert example) | handoff_contracts | set_handoff_contracts |
| SQL checks | JSON editor with Insert example and Check | sql_checks | set_sql_checks |
| Posting rules | Add rule, Try it | posting_rules | set_posting_rules |
| Chart of accounts | Import CSV (Check, then import) | chart_of_accounts | set_chart_of_accounts, variably policy chart import |
| Business rules | JSON editor, Start from a template, Check, Try it | business_rules | set_business_rules |
| Requirement templates | Add template (JSON) | requirement_templates | set_requirement_templates |
| Date order (dates written 04/03/2026) | none: in the YAML | date_order | none |
| Expected answers | nothing to set | nothing to set | log(expected_answer=...) per answer |
Each item the app lists has Edit and Remove. Anything else, edit in the YAML.
In the app
- Open Evaluation policy › Advanced settings ▸ › Data definitions and pick the sub-tab.
- Use the tab's button (Add list, Add contract, Insert example, Add rule). Replace the example's names with yours.
- Save it: Add, Import or Save to draft. The app confirms "Saved to draft vN". A refusal shows Not saved. with the message, and nothing is stored.
- Publish: Publish vN… at the top of the policy, then Publish. Answers are scored with the definition from then on; earlier answers keep their verdicts unless you re-score.
In the policy YAML
- Open Evaluation policy › YAML, or run
variably policy pullto write the version in force tovariably-policy.yaml. - Add or edit the definition under
evaluation_config(the shapes are on each page and in the format reference). - Check it without saving: Check in the YAML tab (and Compare with to see what it changes), or
variably policy validate variably-policy.yamlandvariably policy diff variably-policy.yaml. The file is validated exactly as a save would be; an invalid file is refused with its line and changes nothing. - Save it as the draft: Save as draft (with a Change note), or
variably policy push variably-policy.yaml --note "why". Saving replaces the whole draft with the file. - Publish: Publish vN…, or
variably policy publish --note "why".
evaluation_config:
code_lists: {} # Code lists
scope_defaults: {} # Document scope
handoff_contracts: [] # Hand-off contracts
sql_checks: {} # SQL checks
posting_rules: [] # Posting rules
# chart_of_accounts: a list of {code, name, type}; leave the key out for none ([] is refused)
requirement_templates: {} # Requirement templates
business_rules: {} # Business rules
date_order: MDY # How all-numeric dates are written: MDY or DMY
Leave out a key you do not use. Comments are not kept: the server stores the content, not the text.
From the SDK
The settings calls validate and save one definition, and each publishes a new policy version with your note. Their first argument is the id shown under Project configuration › Integration & keys as Project id for SDK settings (the SDK names it experiment_id). The API key needs the Change the evaluation policy permission.
import os
from variably import VariablyClient
client = VariablyClient({"api_key": os.environ["VARIABLY_API_KEY"],
"base_url": os.environ["VARIABLY_BASE_URL"]}) # https://api.variably.dev
pid = "your-project-id-for-sdk-settings"
# Lists first: a contract's code field must name a list that exists.
client.set_code_lists(pid, "settings/code_lists.json", note="NJ and NY leave types")
client.set_handoff_contracts(pid, "settings/handoff_contracts.json", note="intake to adjudication")
client.set_scope_defaults(pid, "settings/scope_defaults.yaml")
client.set_sql_checks(pid, "settings/sql_checks.json")
client.set_posting_rules(pid, "settings/posting_rules.yaml", note="supplier invoices")
client.set_chart_of_accounts(pid, "settings/chart_of_accounts.csv", note="FY2026 chart")
client.set_business_rules(pid, "settings/business_rules.yaml", note="visit limit 12")
client.set_requirement_templates(pid, "settings/requirement_templates.yaml", note="benefit answers")
Each call takes a dict, a list or the path of a JSON or YAML file (YAML needs PyYAML). An empty value removes the definition: {} for code lists, scope and SQL checks, [] for contracts. From CI:
python -m variably.settings push --experiment-id "$PID" \
--code-lists settings/code_lists.json \
--handoff-contracts settings/handoff_contracts.json \
--scope-defaults settings/scope_defaults.json \
--sql-checks settings/sql_checks.json \
--business-rules settings/business_rules.yaml \
--note "$GIT_COMMIT_MESSAGE"
Keep the policy in your repository: paste this into your coding agent
Paste this into Claude Code, Cursor, or whichever agent has your repository open. It adds the policy file to your repository and a CI job that validates it on every change.
Keep this project's variA/Bly evaluation policy in this repository and check it in CI, so a
change to how answers are scored is reviewed like code.
Look first:
1. Find the CI system this repository uses (GitHub Actions, GitLab CI, CircleCI, another) and
whether a variably-policy.yaml already exists. Tell me before you change anything.
Then:
2. The variably command comes with the SDK (pip install variably-sdk). It reads
VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. In CI, read both from the CI
system's secrets. Never hard-code the key or write it into a file.
3. If the file is missing, run `variably policy pull` to write the version in force to
variably-policy.yaml, and add it to the repository. Comments in the file are not kept by
the server, so put explanations in commit messages, not in the file.
4. Add a CI job on every change to variably-policy.yaml that runs:
variably policy validate variably-policy.yaml (exit 0 valid, 1 invalid, 2 could not ask)
variably policy diff variably-policy.yaml (exit 0 no changes, 1 changes, 2 error)
validate failing must fail the job. diff exiting 1 only means the file changes the policy:
print its output, and fail the job only on exit 2.
5. Add a job on the main branch that saves the file as the draft:
variably policy push variably-policy.yaml --note "<commit sha and subject>"
That job needs a separate key allowed to change the evaluation policy; read it from its own
CI secret into VARIABLY_API_KEY for that step only.
6. Do not publish from CI unless I ask for it. A draft scores nothing until a policy admin
publishes it, in the app or with `variably policy publish --note "<why>"`.
When you are done, tell me the files you added and the secrets I need to create. Verify:
`variably policy validate variably-policy.yaml` exits 0 locally, and the CI job passes on a
branch that changes nothing in the file.What is checked when you save
- The shape. Unknown keys, wrong types, empty names and over-long lists are refused, naming the item:
Rule 'Supplier invoice': action must be 'post' or 'do_not_post'. - The definitions against each other. A code field must name a list that exists (
code list "leave_type" is not declared in code_lists.). Removing a list that a saved contract or scope dimension uses is refused: "the saved hand-off contracts would no longer be valid: …". A contract that names a requirement template must name one that exists. - Nothing is half-saved. A refused change stores nothing. A YAML file is refused with the key path and line:
evaluation_config.posting_rules (line 41): …. - Settings only a policy holds.
date_orderandchart_of_accountsbelong to the policy; an experiment's own configuration that sets them is refused: "evaluation_config.date_order is a setting of the project's evaluation policy: set it in a policy draft and publish it".
Date order
All-numeric dates such as 04/03/2026 are read only when you say how your project writes them. Without date_order they are never guessed: such a date in an expected answer is not read, and in a hand-off it is "untraced" (neither a pass nor a failure).
| Value | 04/03/2026 means |
|---|---|
MDY | April 3, 2026 |
DMY | 4 March 2026 |
Year-first dates (2026-04-03) are always year, month, day, so there is no YMD. Setting YMD is refused: ""YMD" is not a setting: a year-first date (2026-03-04) is always read year, month, day. Use "DMY" (04/03/2026 is 4 March) or "MDY" (04/03/2026 is April 3) for how day and month are written". A hand-off date field can override it with its own date_order.
Concepts on this page: Data definitions