Get a score out of variA/Bly on your own traffic. About five minutes, assuming you have already set up and installed.

This is Observe mode — a call after your existing LLM call, into which you hand everything your application already knows about that response. Your prompts, your model and your control flow are untouched. When you want to start changing things and know whether the change helped, move to BYOR — but do this first.

Hand it everything you already have

observe() scores what you give it. Two arguments are required — the prompt and the response — and everything after that is what turns on a whole further category of scoring. Your retrieved chunks turn on grounding. Your thread id turns on cross-turn coherence. Your agent names turn on trajectory and tool correctness.

None of it is extra work: your application already has all of it in scope at the moment it answers. It is a question of passing it along, and the difference between passing it and not is the difference between a quality score and a diagnosis.

So rather than write the call yourself, paste this into Claude Code, Cursor, or whichever agent has your repository open. It can see where your LLM calls are and what your retrieval and conversation state are named — which is the part that would otherwise take you an afternoon.

Add variA/Bly evaluation to this application. variA/Bly scores LLM output on grounding,
hallucination, coherence and ~49 other dimensions. Integration is one function call after an
LLM response. Do not change any existing logic.

1. Install the SDK: `pip install variably-sdk`

2. Find the LLM call sites — openai.chat.completions.create, anthropic.messages.create,
   LangChain / LlamaIndex .invoke() or .run(), or direct HTTP calls to a provider API.

3. This project is in Observe mode, which is tied to ONE experiment. Instrument exactly ONE
   call site: the primary user-facing response. If there are several, list them with file
   paths and line numbers, ask me which one, and leave the rest alone.

4. Add observe() after that call and pass everything the application already has in scope.
   Each argument turns on a different set of scores, so do not drop one for looking optional:

   - prompt              the user's input, verbatim
   - response            the model's text output
   - provider_response   the raw provider object (model, tokens and cost are read from it)
   - reference_materials if there is a retrieval step, the retrieved chunks, as
                         [{"id": ..., "content": ..., "source": ...}]. Without this,
                         grounding, faithfulness, hallucination rate and attribution
                         cannot be computed at all.
   - retrieval_query     if the pipeline rewrites or expands the query before searching,
                         the rewritten query — not the raw prompt
   - session_id          the conversation or thread ID, if this is multi-turn. Without it
                         every turn is scored as an isolated exchange and nothing
                         cross-turn is measured.
   - kind                "generative" (default) for user-facing prose. Use "classifier",
                         "router", "tool_use" or "transformer" for turns that emit JSON or
                         control structure, so grounding is reported as N/A rather than
                         scored against something that is not prose.
   - user_id, tags, metadata   if the application has them

5. If this is a multi-agent system (LangGraph, CrewAI, a supervisor loop, or hand-rolled),
   instrument every agent step rather than only the final answer, give them all the same
   session_id, and additionally pass:

   - agent_name      which agent produced this step
   - tools_called    [{"name": ..., "arguments": {...}, "output": ...}], plus
                     expected_tools and allowed_tools where you know them
   - is_aggregator=True on the single final step that merges the others into the answer.
     This is what triggers cross-step coherence and trajectory scoring.

6. Use only the argument names listed above. observe() rejects unknown keyword arguments,
   so a guessed name such as retrieved_context or context raises a TypeError in my request
   path.

7. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. Do not hardcode the API key.

8. observe() is non-blocking and swallows its own network errors, so it cannot fail my
   request. Do not wrap it in try/except and do not touch my error handling.

When you are done, tell me which file and line you instrumented, which arguments you filled
and which you left out because the data was not in scope, and which other LLM call sites you
deliberately left alone.

What each argument unlocks — and what you miss without it

You pass	You unlock	Without it
prompt + response	49-dimension scoring across quality, safety, semantic and depth	Required — nothing is scored
provider_response	Model, token counts, cost and provider, read from the raw object	Scores with no cost or efficiency view beside them
reference_materials	Faithfulness, hallucination rate, attribution accuracy, context utilisation	The entire grounding category reports N/A
retrieval_query	Retrieval relevance against the query you actually searched with	Relevance is judged against the raw prompt, understating a pipeline that expands queries
session_id	Session tracing, cross-turn coherence, quality trend, degradation detection	Every turn scored as an isolated exchange
prior_turns	Coherence against history you hold outside our reach	Not needed if you pass session_id — history is built from earlier turns
user_id	Quality per user, and which interaction patterns go wrong	No per-user view
tags + metadata	Slicing by environment, version, customer tier or anything you log	One undifferentiated pool of scores
kind	Grounding correctly marked N/A on routers and classifiers	A JSON control turn scored as if it were prose, dragging the average down
agent_name + is_aggregator	Trajectory precision and recall, tool correctness, cross-step coherence	A multi-agent system scored as unrelated single turns

The right-hand column is the honest reason to fill each one in. A grounding category of N/A is not the platform having nothing to say about your system — it is the scorer declining to invent a number for a question you did not give it the evidence to answer.

The full call

Everything observe() accepts, in one place. Aim for this call, not a shorter one — each block is labelled with what it switches on, so you can see what a line is buying before you decide to leave it out.

observe(
    # --- Required ---
    prompt=user_message,
    response=ai_response,

    # --- Auto-extract model, tokens and cost from the raw LLM response object ---
    provider_response=llm_raw_response,      # OpenAI, Anthropic, Google — auto-detected

    # --- Session tracking: groups multi-turn conversations ---
    session_id="conversation-abc-123",
    user_id="user-456",

    # --- RAG grounding: turns on faithfulness and hallucination scoring ---
    reference_materials=[
        {"id": "chunk-1", "content": "Your retrieved doc text...", "source": "policy.pdf"},
    ],
    retrieval_query="expanded search query",  # if you rewrite queries before search

    # --- Conversation history: turns on coherence scoring.
    #     Not needed if you pass session_id — history is built from earlier turns. ---
    prior_turns=[
        {"role": "user", "content": "previous question"},
        {"role": "assistant", "content": "previous answer"},
    ],

    # --- Manual overrides, if you are not passing provider_response ---
    model="gpt-4o",
    provider="openai",
    latency_ms=450,
    prompt_tokens=120,
    completion_tokens=340,
    cost=0.004,

    # --- Tags: short labels for slicing scores by cohort ---
    tags=["production", "rag", "v2"],

    # --- Metadata: per-call signals, with two jobs.
    #     1. Segmentation — any key and value you like
    #     2. Diagnostics — the canonical key names below link a logged-but-never-varied
    #        lever to a failing dimension. See "Metadata keys that earn you more". ---
    metadata={
        "customer_tier": "enterprise",        # segmentation
        "region": "us-east",                  # segmentation
        "retrieved_chunks_count": 5,          # diagnostic -> Retrieval Relevance
        "cited_sources_count": 2,             # diagnostic -> Attribution Accuracy
        "temperature": 0.2,                   # diagnostic -> Faithfulness
    },

    # --- Turn type: skips scorers that do not apply (default: "generative") ---
    kind="generative",                        # or "classifier", "router", "tool_use", "transformer"
)

Turn types

A router that emits {"intent": "refund"} is not a bad answer — it is not an answer at all. Mark what a turn is and the prose dimensions are reported as N/A rather than scored against a JSON object.

kind	Use it for
generative	User-facing prose. The default.
classifier	Turns that emit a label or category
router	Turns that decide where the request goes next
tool_use	Turns that call a tool or return documents
transformer	Turns that reshape data rather than answer

Metadata keys that earn you more

Anything in metadata is logged and available for segmentation. A handful of canonical names do something extra: when a dimension is failing, they let the dashboard connect it to a lever you have been logging but never varied. Custom keys still work for slicing — they just do not earn the link.

Dimension	Keys	What they say
Retrieval relevance	retrieved_chunks_count, top_k, rerank_count, embedding_model	How many chunks your retriever pulled, and with what
Context utilisation	has_context, context_size, context_window	Whether context was passed, and how much room it took
Faithfulness	model, temperature, top_p	The generation knobs that shape how literally the model follows its source
Attribution accuracy	cited_sources_count, citation_count	How many distinct sources the answer cites
Context retention	history_turns, memory_size	How much conversation memory went into the prompt
Workflow consistency	workflow_step, agent_name, tool_count	Which step and which agent produced this call

Multi-agent systems

If your app runs several agents, instrument every step rather than only the final answer: same session_id on each, agent_name to label them, tools_called for what each invoked, and is_aggregator=True on the one that produces the answer. That is what turns separate observations into one scored trace — see Score a multi-agent trace. On LangGraph, instrument() does all of it in one line.

Run your application once. Scores appear under Evaluations within a few seconds.

Scoring happens asynchronously on our infrastructure, so none of this sits in your request path.

What this gives you

How your system is performing right now, on real traffic, per dimension — where you stand, and which answers are failing.

Once you have that baseline, the next question is usually whether a change made things better. Scores move for all sorts of reasons — the traffic changed, the retriever changed, the provider shipped a model update — so to attribute a movement to your change, run both versions against the same traffic at the same time. That is BYOR, and it is the natural step after this one.

Building RAG?

Add one argument and you get faithfulness, hallucination rate, attribution accuracy and context utilisation — every claim in the answer checked against the chunks the model was given. See Score a RAG answer.