Observe mode answers how good is my system right now. BYOR answers did my change make it better — a different question, and the one you reach within a week.

BYOR is Bring Your Own Response. You keep your LLM, your provider, your infrastructure. variA/Bly manages the prompt variants, splits traffic between them, and attributes every score to the variant that produced it.

Why not just change the prompt and watch the score?

Because you cannot attribute the result. Scores move for reasons that have nothing to do with you: your traffic mix shifts over a weekend, your retriever indexes new documents, your provider quietly updates the model behind an endpoint. Compare Tuesday against Thursday and you are measuring all of that at once.

Running both versions at the same time on the same traffic removes every one of those explanations. What is left is your change.

Observe measures. BYOR compares. That is the whole reason to move.

Run Observe first anyway

An experiment needs something to beat. A few days of Observe gives you:

• a baseline per dimension, so you know which one you are actually trying to move
• the answers that are failing, so your variant addresses a real weakness rather than a guess
• confidence the integration works, before you add variant routing to the request path

Teams that skip this tend to run their first experiment against a prompt they assume is weak, and learn nothing when the result comes back flat.

What changes in your code

Two calls instead of one: ask which variant to use, then submit what happened.

from variably import VariablyClient

client = VariablyClient({
    "api_key": "vbl_your_key_here",
    "base_url": "https://api.variably.dev",
})

# Before your LLM call — variA/Bly picks the variant and splits traffic
variant = client.get_variant(
    experiment_key="checkout-support-v2",
    user_context={"user_id": "user-123"},
    input_variables={"query": user_message, "context": rag_context},
)

# Your LLM call, using the variant's prompt. Still your model, still your code.
ai_response = your_llm.invoke(variant.prompt_template + rag_context)

# After — submit what happened, attributed to that variant
client.submit_response(
    experiment_key="checkout-support-v2",
    variant_key=variant.variant_key,
    executed_prompt=user_message,
    response=ai_response,
    user_context={"user_id": "user-123", "session_id": conversation_id},
    provider="anthropic",
    model="claude-sonnet-4-20250514",
    evaluation_context={
        "reference_materials": [
            {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
        ],
        "retrieval_query": user_message,
    },
)

user_context matters more than it looks. Assignment is sticky per user, so nobody flips between variants mid-conversation — which would make the result meaningless.

evaluation_context behaves exactly as in Observe mode: without reference_materials, grounding is reported as not-applicable. See Score a RAG application.

What you get back

Both variants scored across the same 49 dimensions, with a statistical verdict rather than a difference of averages. A prompt that raises overall score while lowering grounding is a trade, not a win — and the per-dimension view is where you notice.

See Compare two prompt versions.

When to stay on Observe

If you are not changing prompts yet, BYOR adds a call and buys nothing. Observe every production call, and move when you have a specific change you want to prove.