A chatbot that answers every message well can still be a bad chatbot. It contradicts what it said three turns ago, forgets a constraint the user gave it, or re-answers a question that was already settled. None of that is visible one message at a time.

This page is a complete setup. If you already have Observe running, steps 1 and 2 are done.

1. Install the SDK

pip install variably-sdk

2. Set two environment variables

VARIABLY_API_KEY=vbl_your_key_here
VARIABLY_BASE_URL=https://api.variably.dev

The API key is scoped to one project — everything it sends lands there and nowhere else.

3. Pass the conversation id

Call observe() once per turn with the same session_id each time and the turns are assembled into a session for you. You do not need to send the history — it is built from the turns you have already sent.

Paste this into Claude Code, Cursor, or whichever agent has your repository open.

Add variA/Bly session evaluation to this conversational application. Today each message would
be scored in isolation; I want turns linked into sessions so contradiction and drift across a
conversation are measured. It is one function call per turn. Do not change any existing logic.

1. Install the SDK: `pip install variably-sdk`

2. Find where an assistant turn is produced and returned to the user, and find the identifier
   that is stable for the length of one conversation — a thread id, conversation id or chat id
   the app already persists. Show me both with file paths and line numbers.

3. Add observe() after each assistant turn, passing:

   - prompt              the user's message for this turn
   - response            the assistant's reply for this turn
   - provider_response   the raw LLM response object
   - session_id          the conversation id — the SAME value on every turn of a conversation.
                         Do not generate a new id per message; that is the one mistake that
                         silently disables everything on this page.
   - user_id             the end user, if the app knows them
   - tags, metadata      if the application has them. history_turns and memory_size are
                         useful here — they link a coherence drop to how much context was
                         actually passed.

4. Call it per turn, not once per conversation at the end.

5. Only pass prior_turns if the app rebuilds history from a store and turns may be missing
   from what was already sent. Normally session_id alone is enough.

6. Use only the argument names listed above. observe() rejects unknown keyword arguments, so a
   guessed name such as conversation_id, thread_id or messages raises a TypeError in my
   request path.

7. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment. Do not hardcode the API key.

8. observe() is non-blocking and swallows its own network errors. Do not wrap it in try/except
   and do not touch my error handling.

When you are done, tell me where the session id comes from and confirm it is stable across
turns of one conversation.

The call

from variably import observe

observe(
    prompt=user_message,
    response=assistant_reply,
    provider_response=completion,
    session_id=conversation_id,   # the same value on every turn
    user_id=current_user.id,
    metadata={"history_turns": len(history)},
)

What each argument unlocks — and what you miss without it

You pass	You unlock	Without it
session_id	Cross-turn coherence, quality trend, session-level aggregates	Every turn scored as an isolated exchange
user_id	Which users are having bad conversations	Session scores with no one attached
history_turns metadata	A coherence drop linked to how much context was passed	The score without the lever behind it
prior_turns	Coherence against history you hold outside our reach	Not needed if you pass session_id

What gets scored across turns

Each turn keeps its own scores. On top of those you get signals that only exist because the turns are linked:

• Cross-turn coherence — whether this turn contradicts an earlier one
• Quality trend — whether the conversation improves or degrades as it goes
• Session-level aggregates — so a session is good or bad as a whole, not only per message

If you keep history outside our reach

Some apps rebuild context from a store we never see, or start a session mid-way. Pass the prior turns explicitly and coherence is scored against exactly what you supply:

observe(
    prompt=user_message,
    response=assistant_reply,
    session_id=conversation_id,
    prior_turns=[
        {"role": "user", "content": "I need a refund for order 91."},
        {"role": "assistant", "content": "I can help with that."},
    ],
)

Verify it

Send two turns of one conversation, then open Evaluations. Both turns should appear under a single session with a coherence score between them. Two separate sessions means the id changed between turns.

Sessions and traces are the same mechanism

session_id groups turns of a conversation and it groups steps of an agentic run — the difference is only whether you also pass agent_name and is_aggregator. If your chatbot is itself multi-agent, one id covers both.