All docs
Connect your appConnect your app: start herecode

Connect your app: start here

Create an API key, install the SDK, and pick the way of sending answers that fits your app.

What you'll have

Your app sends its answers, and they appear in Live events scored by the project policy.

You'll need
  • A project
  • Python 3.9 or later where your app runs

Before variA/Bly can score anything, your application has to send it each answer it gives. This page gets you there in three steps: create an API key (a password your app uses to prove it belongs to your project), install the SDK (a small Python library), and pick the way of sending that fits your app. Whatever you pick, answers appear in Live events and are scored by the project's evaluation policy. There is nothing to configure first.

If a developer does this part for you, send them this page. The rest of the docs is about what you do with the scores.

1. Create an API key

  1. Open Project configuration › Integration & keys. (In a new project, the Connect step of Create a project has a Create an API key button that does the same.)
  2. Click Create key. In Create API key, type a Key name: the app or CI job that will use it, for example claims-backend.
  3. Under What the key may do, keep Log answers and Read experiments and results ticked. They are ticked for you, and they are all an app needs.
  4. Click Create key, then Copy. The key is shown only once: paste it into your password manager or your app's secret store now. Then click Done.
PermissionLets the keyGive it to
Log answersSend answers to be scoredEvery app (ticked by default)
Read experiments and resultsRead scores, dataset runs and release-gate resultsEvery app and CI job (ticked by default)
Change experimentsCreate, start and stop experiments from codeA job that manages experiments
Read analyticsRead project analytics and usage from codeA reporting job
Read feature flags · Change feature flags · Read feature gates · Change feature gatesFeature management from codeApps that read flags
Change the evaluation policyPush and publish the policy from code, and the SDK settings callsOnly the CI job that keeps your policy in Git
Create API keysCreate further keysOnly a trusted provisioning job

Why so careful? A key that can change the evaluation policy can change how every answer is scored. Give each key only what its job needs.

A key belongs to one project: answers sent with it land in that project and nowhere else. Give each application its own project and key, or its numbers mix with another app's.

2. Install the SDK and set two variables

pip install variably-sdk

export VARIABLY_API_KEY=vbl_your_key_here          # the key you copied
export VARIABLY_BASE_URL=https://api.variably.dev  # always this address

Set the two variables where your app actually runs (its container, server or cloud function), not only in your own terminal. A key in your shell is not a key in your container: this is the most common reason nothing arrives.

3. Choose how to send answers

Your situationUsePage
A Python app where you call the model yourself (a RAG service, custom agents, a SQL analyst)observe(): the quickest startbelow
The same, in a test script or CI job, where a mistake should stop the runclient.log()below
A LangGraph appinstrument()LangGraph
Google ADK, including Vertex AI Agent EngineVariablyPluginGoogle ADK
You already trace with OpenTelemetryinstrument()OpenTelemetry
You trace with MLflow (for example on Databricks)log_traces() or instrument()MLflow traces
Microsoft Agent Frameworkinstrument()Every combination
Azure AI Foundry Agent Servicecontent recording on, and instrument()Every combination
You want to test a release before it shipsvariably.context(run=, dataset_item=) around your app, then variably gateRuns and the release gate

Two ways to send from your own code

observe() and client.log() send the same answer, take the same fields and get the same checks. They differ in setup and in what happens when something is wrong:

observe()client.log()
SetupNone: it reads the two variables from step 2You create a client with the key and address
When something is wrong (no key, a wrong argument)Never raises: it prints the reason to your app's error output and sends nothingRaises an error at the line that is wrong
Model and tokensRead from your model's response object, provider_response=You pass them
Use it forYour live app: one line, and it can never break a user's requestTest scripts, dataset runs and CI, where a mistake should stop the run

Start with observe(). It is the fastest way to your first score. Use client.log() where a silent failure would cost you: a dataset run that sends nothing should fail the CI job, not pass it. You can use both in one app.

observe(): the quickest start

A benefits assistant looks up the member's plan documents and answers a question. After the answer is written, one call sends it with the documents it was based on:

from variably import observe

question = "What is my in-network deductible on the Gold PPO plan for 2026?"
docs = [{"id": "sob-gold-2026-p2",
         "source": "Summary of Benefits, Gold PPO 2026, p.2",
         "content": "In-network deductible: $1,500 individual / $3,000 family."}]
answer = "Your 2026 in-network deductible is $1,500 for an individual."

observe(
    prompt=question, response=answer,
    reference_materials=docs,        # the documents the answer was based on: turns on grounding
    environment="prod",              # a label: filters and scoring rules can use it
    session_id="chat-4821",          # the conversation this turn belongs to
    agent_name="benefits-agent",     # which part of your app answered
    user_id="member-20931",
    tags=["web"],
    metadata={"plan": "gold", "plan_year": "2026"},
    provider_response=completion,    # your model's response object: model, tokens and cost are read from it
)

observe() returns at once; scoring happens in the background, so your user never waits for it. Every argument after the answer is optional: send what your app already has. Each one turns on more checks, listed in What to send with each answer. If nothing arrives, read your app's error output: observe() prints why it sent nothing.

What you'll see. Within a few seconds the answer is a row in Live events. Its claim "$1,500 for an individual" is Supported by the document. Had the answer said $2,500, the claim would be Contradicted, with the explanation "The answer says $2,500; Summary of Benefits, Gold PPO 2026, p.2 says $1,500." See Event details.

client.log(): when a mistake should stop the run

The same answer, sent from a client you create. It takes the same fields; the answer and the documents are its first two arguments:

import os
from variably import VariablyClient

client = VariablyClient({"api_key": os.environ["VARIABLY_API_KEY"],
                         "base_url": os.environ["VARIABLY_BASE_URL"]})

client.log(question, answer, reference_materials=docs, environment="ci")

A wrong argument or a missing key raises an error here instead of being printed, so a test script stops at the line that is wrong.

Set labels and runs once: variably.context()

Values that are the same for many answers (where the app runs, the run of a dataset, a tag) can be set once instead of on every call. observe(), client.log(), the Google ADK plugin and the LangGraph integration all read them:

import variably

variably.set_context(environment="staging")          # for the whole process, at start-up

with variably.context(tags=["batch-7"]):             # for one block: a request, a job
    answer_question(question)

An argument you pass on the call wins. Tags are added together; metadata keys are merged. run, dataset_item and expected_answer go only on the answer the user gets (an answer sent without agent_name, or the step marked as the final answer), never on a router or query step of the same run. Runs and the release gate shows them in use.

Environment labels

Answers sent with observe(), client.log(), the Google ADK plugin or the LangGraph integration carry the environment you pass or set with variably.context(). So do answers sent over OpenTelemetry with instrument() (including live MLflow traces, Microsoft Agent Framework and Azure AI Foundry); an app exporting its own way sets the standard resource attribute deployment.environment.name. For recorded MLflow traces, log_traces(environment=...) sets it.

Conversations: end them when they are over

If your app holds conversations (a chat, a call), send every turn with the same session_id. When the conversation is over (the customer hangs up, the chat is closed), tell variA/Bly:

client.end_session("chat-4821")

That runs the checks that look at the whole conversation, such as rules over a whole conversation. Calling it twice is safe. Marking the last turn with is_aggregator=True (with its agent_name) does the same. Nothing is guessed from a pause in the conversation. See Sessions.

Paste a prompt into your coding agent

Show prompt, then Copy promptShow prompt, then Copy prompt

If a coding agent (Claude Code, Cursor or another) has your repository open, it can do the wiring for you. Project configuration › Integration & keys and every integration page here have a prompt for it: Show prompt reads it, Copy prompt copies it. Each prompt names only SDK functions and arguments that exist.

For any Python app, paste this one. It adds observe() after your main model call and passes everything the app already has:

Prompt for your coding agent
Add variA/Bly evaluation to this application. variA/Bly scores LLM output on grounding,
hallucination, coherence and ~49 other dimensions. Integration is one function call after an
LLM response. Do not change any existing logic.

1. Install the SDK: `pip install variably-sdk`

2. Find the LLM call sites: the model provider's SDK call (a chat-completions or messages
   call), LangChain / LlamaIndex .invoke() or .run(), or direct HTTP calls to a provider.

3. Every answer this key sends lands in the project's Live events and is scored by the
   project's evaluation policy. Start with ONE call site: the primary user-facing response.
   If there are several, list them with file paths and line numbers, ask me which one, and
   leave the rest alone.

4. Add observe() after that call and pass everything the application already has in scope.
   Each argument turns on a different set of scores, so do not drop one for looking optional:

   - prompt              the user's input, verbatim
   - response            the model's text output
   - provider_response   the raw provider object (model, tokens and cost are read from it)
   - reference_materials if there is a retrieval step, the retrieved chunks, as
                         [{"id": ..., "content": ..., "source": ...}]. Without this,
                         grounding, faithfulness, hallucination rate and attribution
                         cannot be computed at all.
   - retrieval_query     if the pipeline rewrites or expands the query before searching,
                         the rewritten query — not the raw prompt
   - session_id          the conversation or thread ID, if this is multi-turn. Without it
                         every turn is scored as an isolated exchange and nothing
                         cross-turn is measured.
   - kind                "generative" (default) for user-facing prose. Use "classifier",
                         "router", "tool_use" or "transformer" for turns that emit JSON or
                         control structure, so grounding is reported as N/A rather than
                         scored against something that is not prose.
   - user_id, tags, metadata   if the application has them

5. If this is a multi-agent system (LangGraph, CrewAI, a supervisor loop, or hand-rolled),
   instrument every agent step rather than only the final answer, give them all the same
   session_id, and additionally pass:

   - agent_name      which agent produced this step
   - tools_called    [{"name": ..., "arguments": {...}, "output": ...}], with
                     "error": "<short message>" on a call that failed, plus
                     expected_tools and allowed_tools where you know them
   - is_aggregator=True on the single final step that merges the others into the answer.
     This is what triggers cross-step coherence and trajectory scoring.

6. Use only the argument names listed above. observe() rejects unknown keyword arguments,
   so a guessed name such as retrieved_context or context raises a TypeError in my request
   path.

7. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. Do not hardcode the API key.

8. observe() is non-blocking and swallows its own network errors, so it cannot fail my
   request. Do not wrap it in try/except and do not touch my error handling.

When you are done, tell me which file and line you instrumented, which arguments you filled
and which you left out because the data was not in scope, and which other LLM call sites you
deliberately left alone.

Revoke a key

Revoke next to a key asks you to confirm. Apps using it stop working at once, and it can't be undone. Create the replacement key first, move your apps to it, then revoke the old one.

Common mistakes

You seeWhyFix
Nothing in Live eventsThe key belongs to another project, or VARIABLY_BASE_URL isn't set where the app runsCheck the key is listed under Integration & keys of the project you are looking at; set both variables in the app's environment
AuthenticationError: UnauthorizedThe key was revoked, or its project was deletedCreate a key in this project and update VARIABLY_API_KEY
Answers arrive, but grounding isn't scoredNo reference_materials were sent, or each item's text isn't under the key contentSend the documents the answer used, each as {"id", "content", "source"}
A scoring rule on Environment matches nothingThe answers were sent without an environmentPass environment=, or set it once with variably.set_context(environment=...); over OpenTelemetry without the SDK, set deployment.environment.name
A settings call from the SDK is refusedThe key lacks Change the evaluation policyCreate a separate key with that permission for the job