A single score on the final answer tells you the answer was wrong. It does not tell you which part of your pipeline made it wrong — and a retriever that missed and a model that ignored good context are different fixes that look identical from the answer alone.

This page is a complete setup. If you already have Observe running, steps 1 and 2 are done.

1. Install the SDK

pip install variably-sdk

2. Set two environment variables

VARIABLY_API_KEY=vbl_your_key_here
VARIABLY_BASE_URL=https://api.variably.dev

The API key is scoped to one project — everything it sends lands there and nowhere else.

3. Report each step

Three arguments do the work. session_id is the thread: every step sharing one id is one trace. agent_name labels which step this is. is_aggregator=True marks the single step that produces the final answer, and that is what triggers scoring across steps rather than within each one.

Paste this into Claude Code, Cursor, or whichever agent has your repository open — it can see where your pipeline splits, which is the part you would otherwise have to trace by hand.

Add variA/Bly per-step evaluation to this application. Today it scores only the final answer;
I want each step scored on its own terms so a bad answer points at the step that caused it.
It is one function call per step. Do not change any existing logic.

1. Install the SDK: `pip install variably-sdk`

2. Find the pipeline steps between the user's question and the final answer — typically a
   retrieval step and a generation step, sometimes a rewrite or a rerank as well. List them
   with file paths and line numbers before editing.

3. Generate one trace id per request — a request id or thread id the app already has is ideal.
   Every step of the same request must pass the identical value as session_id.

4. Call observe() at the end of each step, passing:

   - prompt              what this step was asked to do
   - response            what this step produced
   - session_id          the trace id — identical across every step of the request
   - agent_name          this step's name, e.g. "retriever", "reranker", "generator"
   - is_aggregator       True on the ONE step producing the final answer, omitted elsewhere
   - kind                "tool_use" on steps that return documents, "classifier" or "router"
                         on steps that only label or route. Leave the default "generative"
                         on steps that write prose. This stops a list of chunks being scored
                         as if it were an answer.
   - reference_materials on the retrieval step, the chunks it returned; on the generation
                         step, the chunks that actually reached the prompt. Same shape:
                         [{"id": ..., "content": ..., "source": ...}]
   - retrieval_query     on the retrieval step, the query actually searched with
   - provider_response   on any step that called a model

5. Use only the argument names listed above. observe() rejects unknown keyword arguments, so a
   guessed name such as steps, stage or retrieved_context raises a TypeError in my request path.

6. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. Do not hardcode the API key.

7. observe() is non-blocking and swallows its own network errors. Do not wrap it in try/except
   and do not touch my error handling.

When you are done, tell me each step you instrumented and its agent_name, and which one you
marked as the aggregator.

The retrieval step

from variably import observe

chunks = retriever.search(expanded_query)

observe(
    prompt=user_query,
    response=summarise(chunks),
    reference_materials=[
        {"id": c.id, "content": c.text, "source": c.source}
        for c in chunks
    ],
    retrieval_query=expanded_query,
    session_id=trace_id,
    agent_name="retriever",
    kind="tool_use",
)

kind="tool_use" tells the scorer this step returns documents rather than prose. Retrieval relevance is scored against retrieval_query — the query you actually searched with, which matters if your pipeline rewrites or expands it. The prose dimensions are marked N/A, so a list of chunks never drags down a category it was never meant to be measured on.

The generation step

completion = client.chat.completions.create(...)
answer = completion.choices[0].message.content

observe(
    prompt=user_query,
    response=answer,
    provider_response=completion,
    reference_materials=same_chunks,     # what the model was actually given
    session_id=trace_id,                 # same id as the retrieval step
    agent_name="generator",
    is_aggregator=True,                  # this step produces the final answer
)

Hand the generator the same chunks you handed the model. That is what makes faithfulness a measurement rather than an opinion: every claim in the answer is checked against the context it was given, not against the world.

What you get that a single score cannot give you

Signal	What it tells you
Retrieval relevance on the retrieval step	The retriever found the wrong documents
Faithfulness on the generation step	The retriever was right and the model ignored it
Cross-step coherence	Two steps contradicted each other
Trajectory precision and recall	The pipeline took a path it should not have

The first two are the common case, and they are the two that look identical from the final answer.

Verify it

Run one request, then open Evaluations. You should see one trace with a row per step, each carrying its own scores. Rows but no trace-level scores means no step was marked is_aggregator=True.

Scaling past two steps

The same three arguments work for any number of steps. Add tools_called to record what each step invoked and you get tool-correctness and trajectory scores as well — see Score a multi-agent trace.