Connect your app: start here
Create an API key, install the SDK, and pick the way of sending answers that fits your app.
Your app sends its answers, and they appear in Live events scored by the project policy.
- A project
- Python 3.9 or later where your app runs
Before variA/Bly can score anything, your application has to send it each answer it gives. This page gets you there in three steps: create an API key (a password your app uses to prove it belongs to your project), install the SDK (a small Python library), and pick the way of sending that fits your app. Whatever you pick, answers appear in Live events and are scored by the project's evaluation policy. There is nothing to configure first.
If a developer does this part for you, send them this page. The rest of the docs is about what you do with the scores.
1. Create an API key
- Open Project configuration › Integration & keys. (In a new project, the Connect step of Create a project has a Create an API key button that does the same.)
- Click Create key. In Create API key, type a Key name: the app or CI job that will use it, for example claims-backend.
- Under What the key may do, keep Log answers and Read experiments and results ticked. They are ticked for you, and they are all an app needs.
- Click Create key, then Copy. The key is shown only once: paste it into your password manager or your app's secret store now. Then click Done.
| Permission | Lets the key | Give it to |
|---|---|---|
| Log answers | Send answers to be scored | Every app (ticked by default) |
| Read experiments and results | Read scores, dataset runs and release-gate results | Every app and CI job (ticked by default) |
| Change experiments | Create, start and stop experiments from code | A job that manages experiments |
| Read analytics | Read project analytics and usage from code | A reporting job |
| Read feature flags · Change feature flags · Read feature gates · Change feature gates | Feature management from code | Apps that read flags |
| Change the evaluation policy | Push and publish the policy from code, and the SDK settings calls | Only the CI job that keeps your policy in Git |
| Create API keys | Create further keys | Only a trusted provisioning job |
Why so careful? A key that can change the evaluation policy can change how every answer is scored. Give each key only what its job needs.
A key belongs to one project: answers sent with it land in that project and nowhere else. Give each application its own project and key, or its numbers mix with another app's.
2. Install the SDK and set two variables
pip install variably-sdk
export VARIABLY_API_KEY=vbl_your_key_here # the key you copied
export VARIABLY_BASE_URL=https://api.variably.dev # always this address
Set the two variables where your app actually runs (its container, server or cloud function), not only in your own terminal. A key in your shell is not a key in your container: this is the most common reason nothing arrives.
3. Choose how to send answers
| Your situation | Use | Page |
|---|---|---|
| A Python app where you call the model yourself (a RAG service, custom agents, a SQL analyst) | observe(): the quickest start | below |
| The same, in a test script or CI job, where a mistake should stop the run | client.log() | below |
| A LangGraph app | instrument() | LangGraph |
| Google ADK, including Vertex AI Agent Engine | VariablyPlugin | Google ADK |
| You already trace with OpenTelemetry | instrument() | OpenTelemetry |
| You trace with MLflow (for example on Databricks) | log_traces() or instrument() | MLflow traces |
| Microsoft Agent Framework | instrument() | Every combination |
| Azure AI Foundry Agent Service | content recording on, and instrument() | Every combination |
| You want to test a release before it ships | variably.context(run=, dataset_item=) around your app, then variably gate | Runs and the release gate |
Two ways to send from your own code
observe() and client.log() send the same answer, take the same fields and get the same checks.
They differ in setup and in what happens when something is wrong:
observe() | client.log() | |
|---|---|---|
| Setup | None: it reads the two variables from step 2 | You create a client with the key and address |
| When something is wrong (no key, a wrong argument) | Never raises: it prints the reason to your app's error output and sends nothing | Raises an error at the line that is wrong |
| Model and tokens | Read from your model's response object, provider_response= | You pass them |
| Use it for | Your live app: one line, and it can never break a user's request | Test scripts, dataset runs and CI, where a mistake should stop the run |
Start with observe(). It is the fastest way to your first score. Use client.log() where a
silent failure would cost you: a dataset run that sends nothing should fail the CI job, not pass it.
You can use both in one app.
observe(): the quickest start
A benefits assistant looks up the member's plan documents and answers a question. After the answer is written, one call sends it with the documents it was based on:
from variably import observe
question = "What is my in-network deductible on the Gold PPO plan for 2026?"
docs = [{"id": "sob-gold-2026-p2",
"source": "Summary of Benefits, Gold PPO 2026, p.2",
"content": "In-network deductible: $1,500 individual / $3,000 family."}]
answer = "Your 2026 in-network deductible is $1,500 for an individual."
observe(
prompt=question, response=answer,
reference_materials=docs, # the documents the answer was based on: turns on grounding
environment="prod", # a label: filters and scoring rules can use it
session_id="chat-4821", # the conversation this turn belongs to
agent_name="benefits-agent", # which part of your app answered
user_id="member-20931",
tags=["web"],
metadata={"plan": "gold", "plan_year": "2026"},
provider_response=completion, # your model's response object: model, tokens and cost are read from it
)
observe() returns at once; scoring happens in the background, so your user never waits for it.
Every argument after the answer is optional: send what your app already has. Each one turns on more
checks, listed in What to send with each answer. If nothing arrives, read your
app's error output: observe() prints why it sent nothing.
What you'll see. Within a few seconds the answer is a row in Live events. Its claim "$1,500 for an individual" is Supported by the document. Had the answer said $2,500, the claim would be Contradicted, with the explanation "The answer says $2,500; Summary of Benefits, Gold PPO 2026, p.2 says $1,500." See Event details.
client.log(): when a mistake should stop the run
The same answer, sent from a client you create. It takes the same fields; the answer and the documents are its first two arguments:
import os
from variably import VariablyClient
client = VariablyClient({"api_key": os.environ["VARIABLY_API_KEY"],
"base_url": os.environ["VARIABLY_BASE_URL"]})
client.log(question, answer, reference_materials=docs, environment="ci")
A wrong argument or a missing key raises an error here instead of being printed, so a test script stops at the line that is wrong.
Set labels and runs once: variably.context()
Values that are the same for many answers (where the app runs, the run of a dataset, a tag) can be
set once instead of on every call. observe(), client.log(), the
Google ADK plugin and the LangGraph integration all read them:
import variably
variably.set_context(environment="staging") # for the whole process, at start-up
with variably.context(tags=["batch-7"]): # for one block: a request, a job
answer_question(question)
An argument you pass on the call wins. Tags are added together; metadata keys are merged.
run, dataset_item and expected_answer go only on the answer the user gets (an answer
sent without agent_name, or the step marked as the final answer), never on a router or query
step of the same run. Runs and the release gate shows them in use.
Environment labels
Answers sent with observe(), client.log(), the Google ADK plugin or the LangGraph integration
carry the environment you pass or set with variably.context(). So do answers sent over
OpenTelemetry with instrument() (including live MLflow traces, Microsoft Agent Framework and Azure AI
Foundry); an app exporting its own way sets the standard resource attribute
deployment.environment.name. For recorded MLflow traces, log_traces(environment=...) sets it.
Conversations: end them when they are over
If your app holds conversations (a chat, a call), send every turn with the same session_id. When the
conversation is over (the customer hangs up, the chat is closed), tell variA/Bly:
client.end_session("chat-4821")
That runs the checks that look at the whole conversation, such as
rules over a whole conversation. Calling it twice is safe. Marking the last
turn with is_aggregator=True (with its agent_name) does the same. Nothing is guessed from a pause in
the conversation. See Sessions.
Paste a prompt into your coding agent

If a coding agent (Claude Code, Cursor or another) has your repository open, it can do the wiring for you. Project configuration › Integration & keys and every integration page here have a prompt for it: Show prompt reads it, Copy prompt copies it. Each prompt names only SDK functions and arguments that exist.
For any Python app, paste this one. It adds observe() after your main model call and passes everything
the app already has:
Add variA/Bly evaluation to this application. variA/Bly scores LLM output on grounding,
hallucination, coherence and ~49 other dimensions. Integration is one function call after an
LLM response. Do not change any existing logic.
1. Install the SDK: `pip install variably-sdk`
2. Find the LLM call sites: the model provider's SDK call (a chat-completions or messages
call), LangChain / LlamaIndex .invoke() or .run(), or direct HTTP calls to a provider.
3. Every answer this key sends lands in the project's Live events and is scored by the
project's evaluation policy. Start with ONE call site: the primary user-facing response.
If there are several, list them with file paths and line numbers, ask me which one, and
leave the rest alone.
4. Add observe() after that call and pass everything the application already has in scope.
Each argument turns on a different set of scores, so do not drop one for looking optional:
- prompt the user's input, verbatim
- response the model's text output
- provider_response the raw provider object (model, tokens and cost are read from it)
- reference_materials if there is a retrieval step, the retrieved chunks, as
[{"id": ..., "content": ..., "source": ...}]. Without this,
grounding, faithfulness, hallucination rate and attribution
cannot be computed at all.
- retrieval_query if the pipeline rewrites or expands the query before searching,
the rewritten query — not the raw prompt
- session_id the conversation or thread ID, if this is multi-turn. Without it
every turn is scored as an isolated exchange and nothing
cross-turn is measured.
- kind "generative" (default) for user-facing prose. Use "classifier",
"router", "tool_use" or "transformer" for turns that emit JSON or
control structure, so grounding is reported as N/A rather than
scored against something that is not prose.
- user_id, tags, metadata if the application has them
5. If this is a multi-agent system (LangGraph, CrewAI, a supervisor loop, or hand-rolled),
instrument every agent step rather than only the final answer, give them all the same
session_id, and additionally pass:
- agent_name which agent produced this step
- tools_called [{"name": ..., "arguments": {...}, "output": ...}], with
"error": "<short message>" on a call that failed, plus
expected_tools and allowed_tools where you know them
- is_aggregator=True on the single final step that merges the others into the answer.
This is what triggers cross-step coherence and trajectory scoring.
6. Use only the argument names listed above. observe() rejects unknown keyword arguments,
so a guessed name such as retrieved_context or context raises a TypeError in my request
path.
7. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. Do not hardcode the API key.
8. observe() is non-blocking and swallows its own network errors, so it cannot fail my
request. Do not wrap it in try/except and do not touch my error handling.
When you are done, tell me which file and line you instrumented, which arguments you filled
and which you left out because the data was not in scope, and which other LLM call sites you
deliberately left alone.Revoke a key
Revoke next to a key asks you to confirm. Apps using it stop working at once, and it can't be undone. Create the replacement key first, move your apps to it, then revoke the old one.
Common mistakes
| You see | Why | Fix |
|---|---|---|
| Nothing in Live events | The key belongs to another project, or VARIABLY_BASE_URL isn't set where the app runs | Check the key is listed under Integration & keys of the project you are looking at; set both variables in the app's environment |
AuthenticationError: Unauthorized | The key was revoked, or its project was deleted | Create a key in this project and update VARIABLY_API_KEY |
| Answers arrive, but grounding isn't scored | No reference_materials were sent, or each item's text isn't under the key content | Send the documents the answer used, each as {"id", "content", "source"} |
| A scoring rule on Environment matches nothing | The answers were sent without an environment | Pass environment=, or set it once with variably.set_context(environment=...); over OpenTelemetry without the SDK, set deployment.environment.name |
| A settings call from the SDK is refused | The key lacks Change the evaluation policy | Create a separate key with that permission for the job |