All docs
Evaluation policyPublish, compare and roll backdashboard

Publish, compare and roll back

Publish a draft with a note, compare versions, put an earlier one back, edit the policy as YAML, and keep it in Git with variably policy.

Every change to the evaluation policy becomes a numbered version. You publish it with a note, can compare any two versions, and can put an earlier one back in one click. This page covers publishing, the history, the YAML tab, and keeping the policy in your own Git repository with the variably policy command.

Publish a draft

Every edit you save goes into the draft. Nothing changes how answers are scored until you publish.

  1. Click Publish v5… in the policy header (the number is your draft's).
  2. Read the summary: every change from the version in force. It applies to the whole project.
  3. Write a Change note: why, not what. "Letters are reviews: stop failing their introductions" helps the colleague reading it in six months; "edited rules" doesn't.
  4. Click Publish. New answers are scored with it from now.
  5. A card offers v5 is in force. Re-score recent answers? Choose Re-score recent answers… to apply it to answers already in (Re-score), or Not now.

Publishing needs the Policy admin role (Members & roles).

Versions & audit

Versions & auditVersions & audit

Open Advanced settings ▸ › Versions & audit.

  • Version history lists every version with its State (Draft, In force, Superseded, Discarded), Change note and date. Click a row to compare it with the version in force (Compare v3 → v5); Compare versions in the header does the same.
  • Put v3 back in force… rolls back to an earlier version, after you write a change note. It becomes the version in force; nothing is deleted.
  • Discard draft… throws the draft away after asking: "Discard draft v5 and its 4 changes? This can't be undone."
  • Re-score answers… starts a re-score.
  • Audit log lists every create, edit, discard, publish, rollback and re-score, with who did it and the note, including changes made from the SDK and the command line.

Example. On Tuesday a colleague publishes v6, which lowers the faithfulness bar. On Wednesday Needs attention halves and nobody can say why. Recent policy changes on the Overview shows v6; Compare v5 → v6 shows the bar change; Put v5 back in force… undoes it while you discuss.

The YAML tab

The YAML tabThe YAML tab

Advanced settings ▸ › YAML shows one version as a file: every setting of the policy written as text. Choose the Version to show, and Copy it. You can also change it here:

  1. Edit the text in the box. The tab says "Edited, not saved".
  2. Click Check: the text is read exactly as saving would read it, and nothing is saved. A problem is shown with its line, for example Line 14 · weights.depth. missing.
  3. Compare with shows what the text would change against a version: every setting that changes, with its value before and after.
  4. Type a Change note and click Save as draft. It checks first and saves only when nothing is wrong. It replaces the whole draft with this file.
  5. Publish as usual. Revert throws away your edits in the box.

YAML is a plain-text format: name: value on each line, with indentation (spaces) showing what belongs to what. You don't need to know more than that: the Check button tells you exactly which line is wrong. Comments (text after #) are allowed but not kept: the server stores the settings, not your text.

Common YAML mistakes

MistakeMessage
A tab character for indentationThe line is refused: indent with spaces
A misspelled key, such as wieghtsThe key is refused, with its path
Text that looks like a number or a date, such as a code 007Put it in quotes: "007"
The same key twiceRefused: a key written twice
Weights that don't add up to 100"weights: must add up to 100 (they add up to 99)"

Keep the policy in your repository

Teams that review code changes can review scoring changes the same way. The variably command comes with the SDK:

pip install variably-sdk
export VARIABLY_API_KEY=vbl_your_key_here
export VARIABLY_BASE_URL=https://api.variably.dev

variably policy pull -o variably-policy.yaml          # the version in force, as a file
variably policy validate variably-policy.yaml         # check it, saving nothing
variably policy diff variably-policy.yaml             # what it would change
variably policy push variably-policy.yaml --note "Letters are reviews"     # save it as the draft
variably policy publish --note "Letters are reviews"  # put the draft in force
variably policy publish --version 4 --note "Back to v4 while we investigate"   # roll back
CommandExit codes
pull, push, publish0 done · 1 refused (an invalid file, no draft, an unknown version) · 2 could not ask
diff0 no changes · 1 changes · 2 error, including an invalid file
validate0 valid · 1 invalid · 2 could not ask

push and publish need a key with Change the evaluation policy; pull, validate and diff work with any key of the project. Keep that key for the one CI job that pushes: a key handed to an app to log answers should not be able to change how they are scored. Publishing from the command line has the same checks and the same audit entry as publishing in the app.

A refused file prints the path and line:

error: weights line 4: must add up to 100 (they add up to 95)

A CI job that checks every change

# .github/workflows/variably-policy.yml
name: variably policy
on:
  pull_request:
    paths: [variably-policy.yaml]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install variably-sdk
      - run: variably policy validate variably-policy.yaml
        env:
          VARIABLY_API_KEY: ${{ secrets.VARIABLY_API_KEY }}
          VARIABLY_BASE_URL: https://api.variably.dev
      - run: variably policy diff variably-policy.yaml || [ $? -eq 1 ]
        env:
          VARIABLY_API_KEY: ${{ secrets.VARIABLY_API_KEY }}
          VARIABLY_BASE_URL: https://api.variably.dev

diff exits 1 when the file changes the policy, which is the point of the pull request, so the job fails only on exit 2.

Paste this into your coding agent

Prompt for your coding agent
Keep this project's variA/Bly evaluation policy in this repository and check it in CI, so a
change to how answers are scored is reviewed like code.

Look first:
1. Find the CI system this repository uses (GitHub Actions, GitLab CI, CircleCI, another) and
   whether a variably-policy.yaml already exists. Tell me before you change anything.

Then:
2. The variably command comes with the SDK (pip install variably-sdk). It reads
   VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. In CI, read both from the CI
   system's secrets. Never hard-code the key or write it into a file.
3. If the file is missing, run `variably policy pull` to write the version in force to
   variably-policy.yaml, and add it to the repository. Comments in the file are not kept by
   the server, so put explanations in commit messages, not in the file.
4. Add a CI job on every change to variably-policy.yaml that runs:
     variably policy validate variably-policy.yaml   (exit 0 valid, 1 invalid, 2 could not ask)
     variably policy diff variably-policy.yaml       (exit 0 no changes, 1 changes, 2 error)
   validate failing must fail the job. diff exiting 1 only means the file changes the policy:
   print its output, and fail the job only on exit 2.
5. Add a job on the main branch that saves the file as the draft:
     variably policy push variably-policy.yaml --note "<commit sha and subject>"
   That job needs a separate key allowed to change the evaluation policy; read it from its own
   CI secret into VARIABLY_API_KEY for that step only.
6. Do not publish from CI unless I ask for it. A draft scores nothing until a policy admin
   publishes it, in the app or with `variably policy publish --note "<why>"`.

When you are done, tell me the files you added and the secrets I need to create. Verify:
`variably policy validate variably-policy.yaml` exits 0 locally, and the CI job passes on a
branch that changes nothing in the file.

Common mistakes

You seeWhyFix
My change made no differenceIt is still a draftPublish it
push exits 2 with a scope errorThe key can't change the policyUse a key with Change the evaluation policy
publish exits 1: no draftNothing was pushed or edited since the last publishPush the file first
Someone's edits disappearedSave as draft in the YAML tab replaces the whole draftPull or copy the current draft before editing
My comments are gone after pullComments aren't keptExplain changes in the change note and your commit message