Grounding asks one question: is every claim in the answer supported by the context you retrieved? Answering it needs that context, so this page is about getting your chunks to the scorer along with the answer they produced.

This page is a complete setup. If you already have Observe running, steps 1 and 2 are done.

1. Install the SDK

pip install variably-sdk

2. Set two environment variables

VARIABLY_API_KEY=vbl_your_key_here
VARIABLY_BASE_URL=https://api.variably.dev

The API key is scoped to one project — everything it sends lands there and nowhere else.

3. Hand it your chunks

Your retrieval step already has everything the scorer needs — the chunks it returned, the query it searched with, and the answer the model built from them. Paste this into Claude Code, Cursor, or whichever agent has your repository open; it can see where your retriever lives and what its results are called.

Add variA/Bly grounding evaluation to this RAG application. variA/Bly checks every claim in an
answer against the chunks the model was actually given. It is one function call after the
answer is generated. Do not change any existing logic.

1. Install the SDK: `pip install variably-sdk`

2. Find the retrieval-then-generate path: where the retriever is called, what it returns, and
   where those results are passed into the prompt. Show me the file paths and line numbers.

3. Add observe() after the answer is generated, passing:

   - prompt              the user's question, verbatim
   - response            the generated answer
   - provider_response   the raw LLM response object (model, tokens and cost are read from it)
   - reference_materials the chunks that ACTUALLY reached the generation step, as
                         [{"id": ..., "content": ..., "source": ...}]. Not everything the
                         retriever returned, and not the whole corpus — only what went into
                         the prompt. Sending more than the model saw makes the answer look
                         better grounded than it is.
   - retrieval_query     the string actually sent to the retriever. If the pipeline expands,
                         rewrites or translates the question before searching, send the
                         expanded form, not the raw prompt.
   - user_id, tags, metadata   if the application has them

4. If retrieval happens in a separate function or step from generation, do not flatten it into
   one call — tell me, and we will instrument the steps separately so retrieval and generation
   get their own scores.

5. Useful metadata keys for this app, if the values are in scope: retrieved_chunks_count,
   top_k, rerank_count, embedding_model, has_context, context_size, cited_sources_count,
   temperature. These let the dashboard link a failing grounding dimension to the retrieval
   setting behind it.

6. Use only the argument names listed above. observe() rejects unknown keyword arguments, so a
   guessed name such as retrieved_context, context or documents raises a TypeError in my
   request path.

7. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. Do not hardcode the API key.

8. observe() is non-blocking and swallows its own network errors, so it cannot fail my request.
   Do not wrap it in try/except and do not touch my error handling.

When you are done, tell me which file and line you instrumented, and confirm that the chunks you
pass are exactly the ones interpolated into the prompt.

The call

from variably import observe

observe(
    prompt=user_query,
    response=llm_answer,
    provider_response=completion,
    reference_materials=[
        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
    ],
    retrieval_query=expanded_query,
    metadata={
        "retrieved_chunks_count": 5,   # -> Retrieval Relevance
        "cited_sources_count": 2,      # -> Attribution Accuracy
        "temperature": 0.2,            # -> Faithfulness
    },
)

What each argument unlocks — and what you miss without it

You pass	You unlock	Without it
reference_materials	Faithfulness, hallucination rate, attribution accuracy, context utilisation	The entire grounding category reports N/A
retrieval_query	Retrieval relevance against the query you really searched with	Relevance is judged against the raw prompt, understating a pipeline that expands queries
provider_response	Model, tokens and cost beside the grounding scores	Grounding with no cost view next to it
metadata keys	A failing dimension linked to the retrieval setting behind it	Scores without the lever that would fix them

Two details worth getting right, because both quietly distort the result:

reference_materials is the set of chunks that actually reached your generation step — not everything your retriever returned, and not your whole corpus. Sending more than the model saw makes the answer look better grounded than it is.

retrieval_query is the query you sent to your retriever. If you expand or rewrite the user's question before retrieval, send the expanded form — that is what was actually searched. Passing the raw question instead leaves the scores valid but the retrieval diagnostics describing a query you never ran.

What you get

Claims in the answer are checked against those chunks individually. An answer can be fluent, confident, well-structured, and still contain a claim your sources contradict. That is the failure an LLM judge most reliably misses, because the answer reads correct.

Numeric and temporal contradictions are treated as contradictions rather than paraphrases — "within 2–3 business days" against a source saying 5–7 is a failure, not a rewording.

Verify it

Run one query, then open Evaluations. The answer should carry a faithfulness score and a list of claims with what supports each. If the grounding category shows N/A, the chunks did not arrive — check that reference_materials is a list of dicts with id and content keys, and that it is non-empty at the moment you call observe().

If retrieval and generation are separate steps in your pipeline, score them separately so a bad answer tells you which one caused it — see Score every step.