Multi-agent systems rarely fail because one agent is bad. They fail because meaning drifts between agents while each one, in isolation, looks correct — so a score on the final answer tells you the run was wrong without telling you where.

This page is a complete setup. If you already have Observe running, steps 1 and 2 are done.

1. Install the SDK

pip install variably-sdk

2. Set two environment variables

VARIABLY_API_KEY=vbl_your_key_here
VARIABLY_BASE_URL=https://api.variably.dev

The API key is scoped to one project — everything it sends lands there and nowhere else.

3. Hand it every step

Three arguments turn a set of separate observations into one scored trace: session_id is the run, agent_name labels each step, and is_aggregator=True marks the one that produces the final answer. Add tools_called and you get trajectory and tool correctness on top.

Your orchestrator already knows all of this at the moment each step runs. Paste this into Claude Code, Cursor, or whichever agent has your repository open — it can see your graph, your node names and your tool wiring, which is the part that would otherwise take you an afternoon.

Add variA/Bly multi-agent evaluation to this application. variA/Bly scores an agentic run as
one trace: trajectory, tool correctness, cross-step coherence and aggregation grounding. It is
one function call per agent step. Do not change any existing logic.

1. Install the SDK: `pip install variably-sdk`

2. Map the system first. List every agent / node / step, what triggers each one, which tools
   each can call, and which single step produces the answer the user finally sees. Show me
   that list with file paths and line numbers before you edit anything.

3. Generate one trace id per run — the thread id or run id the orchestrator already has is
   ideal. Every step of the same run must pass the identical value as session_id. This is
   what makes them one trace instead of unrelated single turns.

4. Call observe() at the end of EVERY step, not only the final one, passing:

   - prompt              what this step was asked to do
   - response            what this step produced
   - provider_response   the raw LLM object, if the step called a model
   - session_id          the run id — identical across every step of the run
   - agent_name          this step's name, e.g. "planner", "retriever", "writer"
   - is_aggregator       True on the ONE step that produces the final user-facing answer,
                         and False/omitted everywhere else. This is what triggers cross-step
                         coherence and aggregation grounding.
   - tools_called        [{"name": ..., "arguments": {...}, "output": ...}] for tools this
                         step actually invoked. Turns on trajectory precision/recall and
                         tool-call correctness.
   - expected_tools      the tools this step should have called, where you know them
   - allowed_tools       names this step is permitted to call. Anything outside it is
                         flagged as an unauthorized action.
   - reference_materials for any step that retrieves, as
                         [{"id": ..., "content": ..., "source": ...}] — and also on the
                         aggregator step, so the final answer is checked against what the
                         steps actually produced
   - kind                "tool_use" for steps returning documents, "router" for steps that
                         only decide where to go next, "classifier" for labelling steps.
                         Leave the default "generative" on steps that write prose. This
                         stops a routing decision being scored as if it were an answer.

5. If the app is built on LangGraph, do NOT hand-instrument. Use:
   from variably.integrations.langgraph import instrument
   instrument(graph, session_id_key="session_id")   # on the uncompiled StateGraph
   and supply context_extractor and tools_extractor. Tell me and stop there.

6. Use only the argument names listed above. observe() rejects unknown keyword arguments, so
   a guessed name such as steps, agents or retrieved_context raises a TypeError in my
   request path.

7. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. Do not hardcode the API key.

8. observe() is non-blocking and swallows its own network errors, so it cannot fail my run.
   Do not wrap it in try/except and do not touch my error handling.

When you are done, tell me: every step you instrumented and its agent_name, which step you
marked as the aggregator and why, and which arguments you could not fill because the data was
not in scope.

What each argument unlocks — and what you miss without it

You pass	You unlock	Without it
session_id	The steps become one trace	Every step scored as an unrelated single turn
agent_name	Per-step scores you can attribute	You know the run failed, not which step
is_aggregator	Cross-step coherence, aggregation grounding	No scoring across steps at all
tools_called	Trajectory precision and recall, tool-call correctness	The path the run took is not measured
expected_tools	Recall against what should have been called	Only precision, not what was missed
allowed_tools	Unauthorized-action flagging	A step calling something it should not is invisible
reference_materials	Grounding on retrieval steps and on the final synthesis	The grounding category reports N/A
kind	Routers and classifiers marked N/A instead of scored	A routing decision scored as if it were prose

A worked example

from variably import observe

run_id = trace_id   # whatever your orchestrator already uses

# --- a retrieval step ---
observe(
    prompt=sub_question,
    response=summarise(chunks),
    session_id=run_id,
    agent_name="retriever",
    kind="tool_use",
    reference_materials=[
        {"id": c.id, "content": c.text, "source": c.source} for c in chunks
    ],
    tools_called=[
        {"name": "vector_search", "arguments": {"q": sub_question}, "output": str(len(chunks))},
    ],
)

# --- the step that produces the answer ---
observe(
    prompt=user_query,
    response=final_answer,
    provider_response=completion,
    session_id=run_id,              # same id as every other step
    agent_name="writer",
    is_aggregator=True,             # the step that produces the final answer
    reference_materials=all_chunks, # what the steps actually produced
    allowed_tools=["vector_search", "calculator"],
)

What gets measured across steps

• Trajectory precision and recall — did the run take the steps it needed, and only those?
• Tool-call correctness — was each tool called with the right arguments?
• Cross-step coherence — did meaning survive each handoff?
• Aggregation grounding — is the final synthesis supported by what the steps actually produced?

All deterministic, all reproducible: the same trace scores the same way every time, so a change in the number is a change in your system rather than a change in a judge's mood.

On LangGraph

One call wraps every node in the graph, and keeps working when you add one:

from variably.integrations.langgraph import instrument

instrument(graph, session_id_key="session_id")   # on the uncompiled StateGraph

See LangGraph for the two extractors that turn on grounding and tool scoring.

Verify it

VARIABLY_LOG_LEVEL=DEBUG python your_app.py

Run once, then open Evaluations. One run should appear as a single trace with a row per step, each carrying its own scores, and trace-level scores across all of them. If you see one row per step but no trace-level scores, no step was marked is_aggregator=True.