OpenTelemetry

Already tracing your application with OpenTelemetry? One line at start-up sends the same traces to variA/Bly, and every agent run in them is evaluated — trajectory, tool use, loops and, when the content is recorded, grounding. Your existing tracing backend keeps receiving everything.

1. Install

pip install "variably-sdk[otel]"

This adds the OpenTelemetry SDK and its standard OTLP exporter.

2. Set two environment variables

VARIABLY_API_KEY=vbl_your_key_here
VARIABLY_BASE_URL=https://api.variably.dev

The API key is scoped to one project — everything it sends lands there and nowhere else.

3. Add one line at start-up

from variably.integrations.otel import instrument

instrument()

Call it once at start-up, before your application handles requests. It adds a batching processor to the global tracer provider — or creates one if none is set — that exports spans in the background. If you set up your own provider, do that first and pass it in (below): OpenTelemetry keeps the first global provider and ignores a later one. Calling it twice is harmless.

Already have a tracer provider? Pass it, and your other processors and exporters are untouched:

from opentelemetry.sdk.trace import TracerProvider
from variably.integrations.otel import instrument

provider = TracerProvider(resource=resource)
provider.add_span_processor(your_existing_processor)
instrument(provider)

Want to wire the exporter yourself? It is a standard OTLP span exporter:

from opentelemetry.sdk.trace.export import BatchSpanProcessor
from variably.integrations.otel import span_exporter

provider.add_span_processor(BatchSpanProcessor(span_exporter(), max_queue_size=4096))

If the API key or base URL is missing, instrument() raises at start-up rather than failing silently later. In a short script, call .force_flush() on the processor it returns before exiting.

4. Make sure your spans carry what is evaluated

variA/Bly reads spans that follow the OpenTelemetry GenAI semantic conventions and Google ADK's native spans; everything else in your traces is ignored.

Google ADK and Vertex AI Agent Engine

ADK emits these spans itself, with message content, so nothing else is needed:

from variably.integrations.otel import instrument
instrument()                     # before the Runner is created

runner = Runner(agent=root_agent, app_name="claims-assistant", session_service=session_service)

Other frameworks and model SDKs

Use an instrumentation that emits GenAI spans — for example the OpenAI one — and switch on content capture, which is off by default for privacy:

pip install opentelemetry-instrumentation-openai-v2
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor
from variably.integrations.otel import instrument

instrument()
OpenAIInstrumentor().instrument()

Without content capture, structure is still evaluated — trajectory, tool use and loops — but there is no answer text, so grounding is not.

Your own agents

If your agents are hand-written, wrap each agent run and each tool call in a span with these attributes — this is exactly what is read:

import json
from opentelemetry import trace
from variably.integrations.otel import instrument

instrument()
tracer = trace.get_tracer("my-agents")

def run_researcher(question, session_id):
    with tracer.start_as_current_span("invoke_agent researcher") as agent:
        agent.set_attribute("gen_ai.operation.name", "invoke_agent")
        agent.set_attribute("gen_ai.agent.name", "researcher")
        agent.set_attribute("gen_ai.conversation.id", session_id)   # groups runs into one session

        with tracer.start_as_current_span("execute_tool search_policy") as tool:
            tool.set_attribute("gen_ai.operation.name", "execute_tool")
            tool.set_attribute("gen_ai.tool.name", "search_policy")
            tool.set_attribute("gen_ai.tool.call.arguments", json.dumps({"query": question}))
            result = search_policy(question)
            tool.set_attribute("gen_ai.tool.call.result", json.dumps(result))

        with tracer.start_as_current_span("chat") as llm:
            llm.set_attribute("gen_ai.operation.name", "chat")
            llm.set_attribute("gen_ai.request.model", "gpt-4o")
            answer = write_answer(question, result)
            llm.set_attribute("gen_ai.output.messages", json.dumps(
                [{"role": "assistant", "parts": [{"type": "text", "content": answer}]}]))
        return answer

What gets evaluated

From your trace	Becomes
An invoke_agent span	One step, named by gen_ai.agent.name
Its model calls	The step's answer text, model and token counts
Its execute_tool spans	Tool calls with arguments, result or error — trajectory, tool correctness and loop detection
A transfer_to_agent call with no answer text	A routing step — grounding is not applicable rather than zero
An agent run inside another agent's tool (ADK AgentTool)	A sub-agent step attached to the same session
The top-level agent with the latest answer	The final step (aggregator) — its summary is checked against the others
gen_ai.conversation.id, or ADK's session id	The session; without one, each trace is its own session
Vertex AI Search, RAG Engine and Google Search grounding on the model response	The evidence grounding is checked against
A tool result with a list under documents, results, chunks, passages, contexts or hits, each with content, text, snippet, page_content, chunk or passage	Evidence too — your own retriever, recognised automatically
A trace with no agent spans	One step from its model and tool spans

Every step is tagged otlp and lands in the same dashboards, checks and sessions as SDK traffic.

When scores appear

A trace is evaluated about ten seconds after its last span ends, so late spans from the same run are included; a span arriving up to ten minutes late is still added to its session. Scores then follow within seconds, or longer when Max-Accuracy escalates a claim.

Never on your request path

Spans leave through OpenTelemetry's batch processor on a background thread. If variA/Bly is slow or unreachable your requests are unaffected: the exporter retries and backs off, or drops in the background. Under load we ask the exporter to retry later — spans are never accepted and then silently discarded.

What OpenTelemetry cannot carry

Traces describe what happened, not what should have happened. These need the ADK plugin or observe():

• Expected tools — trajectory and tool correctness against the path you expect, rather than structure alone;
• Allowed tools — flagging any call outside an allow-list;
• Custom evidence — retriever results in a shape not listed above;
• Reference documents for a document review.

Verify it

Run one request, then open the project's Observations experiment and its Sessions tab. Each agent run should appear as a step with its tools. If nothing arrives:

• check that instrument() ran before your framework created its tracer, and that the process lived long enough to flush;
• turn on Python logging at DEBUG for the opentelemetry logger to see export errors;
• a step with no answer text means content capture is off.

Using an OpenTelemetry Collector rather than in-process exporting? We will set up the collector configuration with you — get in touch.