All docs
Evaluation policyData definitionsSQL checksdashboard

SQL checks

SQL checks power "SQL follows the rules": required filters such as plan year, forbidden columns such as SSN, distinct counts, join fan-out, and percentage columns written as fractions, with examples, edge cases and the full format.

What you'll have

A query that forgets the plan-year filter, reads SSNs or double-counts claims fails, and every count, date and percentage in the answer is checked against the returned rows.

What it is and what it powers

SQL checks power SQL follows the rules on every answer that sends the SQL it ran (log(sql=...)):

  • Valid: the query parses in your dialect, as one statement.
  • Read-only: any write or DDL anywhere in the query fails (INSERT, UPDATE, DELETE, MERGE, DROP, CREATE, ALTER, TRUNCATE, GRANT, COPY, SELECT INTO). "DROP" inside a comment or a string is not one.
  • Schema: with a schema, every table exists and every column resolves to exactly one table. Unknown and ambiguous columns are all reported.
  • Your rules: forbidden tables and columns, required filters, distinct counts, and a warning for joins that can count rows twice.

When you also send the rows the query returned, each figure in the answer is checked against them: a cell, a total, a count, an average or a share (examples). column_units tells the check that a column holds fractions (0.2374 is 23.74%) or percentages.

When it's not set. An answer that sent SQL shows Not configured: "SQL was sent, but no SQL schema and rules are set for this project. Add them in Evaluation policy › Data definitions › SQL checks." An answer that sent no SQL is N/A: "The answer sent no SQL (sql=)."

Use cases

Insurance: claims counts, dates and percentages

{
  "dialect": "postgres",
  "schema": {
    "tables": {
      "claims": { "columns": { "claim_id": "text", "policy_id": "text", "plan_year": "int", "status": "text", "onset_date": "date", "amount": "numeric" }, "primary_key": "claim_id" },
      "policies": { "columns": ["policy_id", "holder_name", "state", "ssn"], "primary_key": "policy_id" }
    }
  },
  "rules": [
    { "id": "plan-year", "kind": "required_filter", "table": "claims", "column": "plan_year", "description": "Every claims query names its plan year." },
    { "id": "no-ssn", "kind": "forbidden_column", "table": "policies", "column": "ssn", "description": "Never read social security numbers." },
    { "id": "distinct-claims", "kind": "count_distinct", "table": "claims", "column": "claim_id" },
    { "id": "fanout", "kind": "join_fanout" }
  ],
  "column_units": { "approval_rate": "fraction" }
}
The agent's SQLResult
SELECT COUNT(*) FROM claims WHERE status = 'open'fail: rule plan-year, no plan-year filter
… WHERE plan_year = 2026 OR status = 'open'fail: an OR satisfies a required filter only when every branch constrains the column
SELECT p.* FROM policies p WHERE p.state = 'NJ'fail: SELECT * returns ssn
SELECT COUNT(claim_id) FROM claims c JOIN policies p USING (policy_id) WHERE plan_year = 2026fail: distinct-claims wants COUNT(DISTINCT claim_id)
DELETE FROM claims WHERE plan_year = 2024fail: not read-only

Send the query and its rows with the answer:

client.log(
    prompt="What share of 2026 NJ claims were approved?",
    response="23.7% of 2026 New Jersey claims were approved.",
    sql="SELECT AVG((status = 'approved')::int) AS approval_rate FROM claims c "
        "JOIN policies p USING (policy_id) WHERE c.plan_year = 2026 AND p.state = 'NJ'",
    reference_materials=[{"id": "q1", "content": "approval_rate query result",
                          "rows": [{"approval_rate": 0.2374}]}],
)

With column_units: {"approval_rate": "fraction"}, 0.2374 is 23.74%, and "23.7%" matches to the precision it is written in. Without it, the figure is unverified: the column's unit is not known, and nothing is guessed from the value.

HR and workforce analytics: headcount and attrition

evaluation_config:
  sql_checks:
    dialect: snowflake
    schema:
      tables:
        employees: {columns: [employee_id, snapshot_date, department, status, salary, ssn], primary_key: [employee_id, snapshot_date]}
        terminations: {columns: [employee_id, termination_date, reason], primary_key: employee_id}
    rules:
      - {id: snapshot, kind: required_filter, table: employees, column: snapshot_date, description: "Headcount is as of one date."}
      - {id: no-salary, kind: forbidden_column, table: employees, column: salary, where: select, description: "Salary may filter, never be returned."}
      - {id: no-ssn, kind: forbidden_column, column: ssn}
      - {id: heads, kind: count_distinct, table: employees, column: employee_id}
    column_units: {attrition_rate: percent}

where: select forbids returning the column but allows it in a filter; the default any forbids reading it at all.

Finance: revenue by region

A required_filter on ledger.period, forbidden_table: payroll_raw, and join_fanout to warn when a join to invoice_lines repeats each invoice before SUM(amount).

Edge cases

SituationWhat happens
Fraction vs percent columnConverted only for a declared unit: fraction reads 0.2374 as 23.74%, percent reads 23.74 as 23.74%. Undeclared: unverified. A cell written with "%" is a percentage.
RoundingA figure matches when it equals the value to the precision it is written in: "$875K" is 875,000 ± 500; "23.7%" matches 23.74%.
The answer names a row the results do not holdUnverified: "names a row the query result does not hold".
One row's figure stated for another rowWrong: "figure belongs to a different row of the query result".
A rule names a column but there is no schemaRefused when saved (below).
A join that may count rows twiceA warning, not a failure: a later DISTINCT or GROUP BY can make it harmless. Needs keys in the schema.
Several queries for one answerSend a list: up to 20 queries, each up to 100,000 characters.
A column that is not in the schemaThe query fails as an unknown column; rules naming it are refused when saved.
Quoted or differently cased namesCompared without quotes or case: "Claims" is claims.

How to add it

In the app

Data definitions: SQL checksData definitions: SQL checks

  1. Open Evaluation policy › Advanced settings ▸ › Data definitions › SQL checks.
  2. In Schema & rules, click Insert example. Replace the tables, columns and rules with yours. The JSON, key by key and What is checked sit beside the editor.
  3. The status reads Invalid JSON, Not saved, Valid or Not configured. Revert discards unsaved edits.
  4. Click Save to draft. Saving an empty editor removes the SQL checks.
  5. Publish: Publish vN… › Publish.

In YAML

Any of the examples above, under evaluation_config.sql_checks.

From the SDK

client.set_sql_checks(pid, "settings/sql_checks.json", note="plan-year filter on claims")
client.log(prompt=q, response=answer, sql=[query1, query2])

{} removes the setting.

Paste this into your coding agent

Paste this into Claude Code, Cursor, or whichever agent has your repository open. It writes the schema and rules from your migrations or ORM models; no database connection is needed.

Prompt for your coding agent
Write variA/Bly SQL checks for this project from the database schema in this repository. The
server checks the text of each query the app sends against this schema and these rules; it never
connects to the database and never needs rows or credentials.

Look first:
1. Find the schema: migrations, ORM models (SQLAlchemy, Django, Prisma, ActiveRecord), dbt
   models or a schema dump. Find the database the queries run on. Find the rules the app's
   queries must follow: tenant or account filters, a required plan-year or snapshot filter,
   columns that must never be selected (personal data), entities counted with DISTINCT.
   Show me what you found, with file paths, before you write anything.

Then write it under evaluation_config.sql_checks:

evaluation_config:
  sql_checks:
    dialect: postgres
    schema:
      tables:
        claims: {columns: [claim_id, plan_year, status, amount], primary_key: claim_id}
    rules:
      - {id: plan-year, kind: required_filter, table: claims, column: plan_year}
      - {id: no-ssn, kind: forbidden_column, table: policies, column: ssn}
    column_units: {approval_rate: fraction}

- dialect: bigquery, clickhouse, databricks, duckdb, mysql, oracle, postgres, presto, redshift,
  snowflake, spark, sqlite, trino or tsql.
- schema: table and column names (types optional), with primary_key or unique keys. Never rows
  of data.
- rules: each has an id and a kind: forbidden_table, forbidden_column, required_filter,
  count_distinct or join_fanout. Write only rules the code or its documentation states; list
  the ones you think are missing and ask me.
- column_units: for result columns holding ratios, "fraction" (0.2374 is 23.74%) or "percent".
- The format is described at https://www.variably.dev/docs/data-sql-checks.

How to write it, as policy YAML:
1. The variably command comes with the SDK (pip install variably-sdk). It reads
   VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment; never hard-code the key or write it into a file.
2. If variably-policy.yaml is not in the repository yet, run `variably policy pull`: it writes
   the version in force to variably-policy.yaml. Change only evaluation_config.sql_checks in that
   file and leave every other key exactly as it is.
3. Verify: run `variably policy validate variably-policy.yaml`. It checks the file as the server would
   and saves nothing; each problem is printed as "line N: path: message". Fix every problem and
   run it again until it exits 0. Do not guess a key the error does not name: unknown keys are
   refused.
4. Run `variably policy diff variably-policy.yaml` and show me what would change.
5. Do not push or publish. Tell me the two ways to save it as a draft: paste the file into
   Evaluation policy › YAML and click Apply as draft, or run
   `variably policy push variably-policy.yaml --note "<why>"` with a key allowed to change the
   evaluation policy. A draft scores nothing until it is published.

6. Check that the app sends each query: the observe() or client.log() call passes sql=<the SQL
   text it ran>, and the rows as reference_materials=[{"id": ..., "source": ..., "rows": rows}].
   If it does not, add that.

When you are done, tell me the tables and rules you wrote and which file each came from.

Common mistakes

MistakeWhat happensFix
Sending SQL with no schema setSQL follows the rules shows Not configuredAdd the schema and rules, and publish
A schema that misses tables the agent usesEvery query using them fails as an unknown table or columnList every table and column the agent may use (names only, never data)
Rates stored as fractions (0.2374), unit not declaredThe answer's "23.7%" is unverifiedDeclare column_units: {approval_rate: fraction}
Expecting variA/Bly to run the queryIt never connects to your databaseSend the rows the query returned with the answer

Format reference

FieldTypeRequiredAllowed valuesDefaultMeaning
dialecttextyesbigquery, clickhouse, databricks, duckdb, mysql, oracle, postgres, presto, redshift, snowflake, spark, sqlite, trino, tsqlnoneHow the SQL is parsed
schemaobjectneeded by every rule but forbidden_table{"tables": {table: spec}} or {table: spec}noneNames only, never rows
table speclist or objecta list of column names; an object column → type; or {"columns": …, "primary_key": …, "unique": [...]}A table's columns and keys
ruleslistnoup to 200 rulesnoneYour query rules
column_unitsobjectnoresult column → fraction or percent; up to 200noneThe unit of a column of the query results (aliases included)

A rule:

kindRequiredOptionalChecks
forbidden_tableid, tabledescriptionThe table is never read
forbidden_columnid, columntable, where (any default, or select), descriptionThe column is never read (any) or never returned (select); SELECT * counts
required_filterid, table, columndescriptionEvery query block reading the table constrains the column in an AND-ed WHERE, JOIN ON or HAVING
count_distinctid, columntable, descriptionCOUNT over the column uses DISTINCT
join_fanoutiddescriptionWarning: a join on no key, or one that repeats a keyed side's rows under SUM, AVG or COUNT (needs primary_key or unique)

Minimal:

sql_checks:
  dialect: postgres

The full example is the insurance one under Use cases, above.

Validation messages

ProblemMessage
Unknown keysql_checks: unknown key "tables"; expected dialect, schema, rules, column_units.
Dialectsql_checks.dialect must be one of bigquery, clickhouse, …, tsql.
Rows in the schemasql_checks.schema: table "claims": unknown field "rows" (a schema holds names and types, never rows).
A rule without a schemasql_checks.rules: rule "plan-year" (required_filter) names columns and needs sql_checks.schema to place them; declare the schema (table and column names) or remove it.
Not in the schemasql_checks.rules: rule "no-ssn": column "ssn" is not in the schema.
Fan-out without keyssql_checks.rules: rule "fanout": join_fanout needs keys (primary_key / unique) in the schema.
Duplicate idsql_checks.rules: rule 3: duplicate id "no-ssn".
Unitsql_checks.column_units: "approval_rate" must be fraction or percent.

Concepts on this page: Data definitions