Publish, compare and roll back
Publish a draft with a note, compare versions, put an earlier one back, edit the policy as YAML, and keep it in Git with variably policy.
Every change to the evaluation policy becomes a numbered version. You
publish it with a note, can compare any two versions, and can put an earlier one back in one click. This
page covers publishing, the history, the YAML tab, and keeping the policy in your own Git repository
with the variably policy command.
Publish a draft
Every edit you save goes into the draft. Nothing changes how answers are scored until you publish.
- Click Publish v5… in the policy header (the number is your draft's).
- Read the summary: every change from the version in force. It applies to the whole project.
- Write a Change note: why, not what. "Letters are reviews: stop failing their introductions" helps the colleague reading it in six months; "edited rules" doesn't.
- Click Publish. New answers are scored with it from now.
- A card offers v5 is in force. Re-score recent answers? Choose Re-score recent answers… to apply it to answers already in (Re-score), or Not now.
Publishing needs the Policy admin role (Members & roles).
Versions & audit

Open Advanced settings ▸ › Versions & audit.
- Version history lists every version with its State (Draft, In force, Superseded, Discarded), Change note and date. Click a row to compare it with the version in force (Compare v3 → v5); Compare versions in the header does the same.
- Put v3 back in force… rolls back to an earlier version, after you write a change note. It becomes the version in force; nothing is deleted.
- Discard draft… throws the draft away after asking: "Discard draft v5 and its 4 changes? This can't be undone."
- Re-score answers… starts a re-score.
- Audit log lists every create, edit, discard, publish, rollback and re-score, with who did it and the note, including changes made from the SDK and the command line.
Example. On Tuesday a colleague publishes v6, which lowers the faithfulness bar. On Wednesday Needs attention halves and nobody can say why. Recent policy changes on the Overview shows v6; Compare v5 → v6 shows the bar change; Put v5 back in force… undoes it while you discuss.
The YAML tab

Advanced settings ▸ › YAML shows one version as a file: every setting of the policy written as text. Choose the Version to show, and Copy it. You can also change it here:
- Edit the text in the box. The tab says "Edited, not saved".
- Click Check: the text is read exactly as saving would read it, and nothing is saved. A problem is shown with its line, for example Line 14 · weights.depth. missing.
- Compare with shows what the text would change against a version: every setting that changes, with its value before and after.
- Type a Change note and click Save as draft. It checks first and saves only when nothing is wrong. It replaces the whole draft with this file.
- Publish as usual. Revert throws away your edits in the box.
YAML is a plain-text format: name: value on each line, with indentation (spaces) showing what belongs
to what. You don't need to know more than that: the Check button tells you exactly which line is wrong.
Comments (text after #) are allowed but not kept: the server stores the settings, not your text.
Common YAML mistakes
| Mistake | Message |
|---|---|
| A tab character for indentation | The line is refused: indent with spaces |
A misspelled key, such as wieghts | The key is refused, with its path |
Text that looks like a number or a date, such as a code 007 | Put it in quotes: "007" |
| The same key twice | Refused: a key written twice |
| Weights that don't add up to 100 | "weights: must add up to 100 (they add up to 99)" |
Keep the policy in your repository
Teams that review code changes can review scoring changes the same way. The variably command comes
with the SDK:
pip install variably-sdk
export VARIABLY_API_KEY=vbl_your_key_here
export VARIABLY_BASE_URL=https://api.variably.dev
variably policy pull -o variably-policy.yaml # the version in force, as a file
variably policy validate variably-policy.yaml # check it, saving nothing
variably policy diff variably-policy.yaml # what it would change
variably policy push variably-policy.yaml --note "Letters are reviews" # save it as the draft
variably policy publish --note "Letters are reviews" # put the draft in force
variably policy publish --version 4 --note "Back to v4 while we investigate" # roll back
| Command | Exit codes |
|---|---|
pull, push, publish | 0 done · 1 refused (an invalid file, no draft, an unknown version) · 2 could not ask |
diff | 0 no changes · 1 changes · 2 error, including an invalid file |
validate | 0 valid · 1 invalid · 2 could not ask |
push and publish need a key with Change the evaluation policy; pull, validate and diff work
with any key of the project. Keep that key for the one CI job that pushes: a key handed to an app to log
answers should not be able to change how they are scored. Publishing from the command line has the same
checks and the same audit entry as publishing in the app.
A refused file prints the path and line:
error: weights line 4: must add up to 100 (they add up to 95)
A CI job that checks every change
# .github/workflows/variably-policy.yml
name: variably policy
on:
pull_request:
paths: [variably-policy.yaml]
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install variably-sdk
- run: variably policy validate variably-policy.yaml
env:
VARIABLY_API_KEY: ${{ secrets.VARIABLY_API_KEY }}
VARIABLY_BASE_URL: https://api.variably.dev
- run: variably policy diff variably-policy.yaml || [ $? -eq 1 ]
env:
VARIABLY_API_KEY: ${{ secrets.VARIABLY_API_KEY }}
VARIABLY_BASE_URL: https://api.variably.dev
diff exits 1 when the file changes the policy, which is the point of the pull request, so the job
fails only on exit 2.
Paste this into your coding agent
Keep this project's variA/Bly evaluation policy in this repository and check it in CI, so a
change to how answers are scored is reviewed like code.
Look first:
1. Find the CI system this repository uses (GitHub Actions, GitLab CI, CircleCI, another) and
whether a variably-policy.yaml already exists. Tell me before you change anything.
Then:
2. The variably command comes with the SDK (pip install variably-sdk). It reads
VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. In CI, read both from the CI
system's secrets. Never hard-code the key or write it into a file.
3. If the file is missing, run `variably policy pull` to write the version in force to
variably-policy.yaml, and add it to the repository. Comments in the file are not kept by
the server, so put explanations in commit messages, not in the file.
4. Add a CI job on every change to variably-policy.yaml that runs:
variably policy validate variably-policy.yaml (exit 0 valid, 1 invalid, 2 could not ask)
variably policy diff variably-policy.yaml (exit 0 no changes, 1 changes, 2 error)
validate failing must fail the job. diff exiting 1 only means the file changes the policy:
print its output, and fail the job only on exit 2.
5. Add a job on the main branch that saves the file as the draft:
variably policy push variably-policy.yaml --note "<commit sha and subject>"
That job needs a separate key allowed to change the evaluation policy; read it from its own
CI secret into VARIABLY_API_KEY for that step only.
6. Do not publish from CI unless I ask for it. A draft scores nothing until a policy admin
publishes it, in the app or with `variably policy publish --note "<why>"`.
When you are done, tell me the files you added and the secrets I need to create. Verify:
`variably policy validate variably-policy.yaml` exits 0 locally, and the CI job passes on a
branch that changes nothing in the file.Common mistakes
| You see | Why | Fix |
|---|---|---|
| My change made no difference | It is still a draft | Publish it |
push exits 2 with a scope error | The key can't change the policy | Use a key with Change the evaluation policy |
publish exits 1: no draft | Nothing was pushed or edited since the last publish | Push the file first |
| Someone's edits disappeared | Save as draft in the YAML tab replaces the whole draft | Pull or copy the current draft before editing |
My comments are gone after pull | Comments aren't kept | Explain changes in the change note and your commit message |