All docs
Connect your appConnect your app: every combinationcode

Connect your app: every combination

The exact setup for what your AI does, one agent or several, and what it is built with, and what is checked for each use case.

What you'll have

Your app sends every input its checks need, for your combination, and nothing shows Couldn't check for want of an input.

You'll need
  • A project and its API key
  • Your answers to the three questions on Create a project

Which checks run on your answers depends on three things: what your AI does, whether it is one agent or several, and what it is built with. Create a project asks all three, and its Connect step gives one setup for exactly your combination. This page has the same setup for every combination, so you can see what your app will be asked to send and why.

For a framework, setup is one line. LangGraph, Google ADK, Microsoft Agent Framework, Azure AI Foundry, MLflow and OpenTelemetry already hold the SQL your agent ran, the rows it got back, the documents it retrieved and the files it read. variA/Bly reads them from the agent's tool calls, so you write no observe() call. With your own code, one call per answer (or per step) sends them.

What gets checked for each use case

Tick every use case that applies: a chat assistant with several agents that queries a database is chat + database + several agents, and gets every check below whose inputs arrive.

What your AI doesWhat must arrive with each answerWhat is checkedAnswers are judged as
Answers questions from documents (RAG, Q&A)the question, the answer, and the passages the model was giveneach claim against the passages, whether the right passages were found, answer quality, your business rulesAnswers from sources
Chat assistant (several turns)the same for each turn, plus one conversation id; the conversation is ended after its last turnthe above, plus whether the turns agree with each other and your rules over the whole conversationAnswers from sources
Queries a database (text-to-SQL)the question, the answer, the SQL that ran, and the rows each query returnedthe SQL against the schema and query rules you declare; every figure in the answer against the rowsAnswers from sources
Reviews documents (invoices, contracts, claims, reconciliations)the task, the review, and every document read, wholeclaims and figures against the documents; that the right documents were used; arithmetic; your business rulesDocument analysis
Reviews code (a code review agent)the task, the review, the diff and every file the agent read, with pathsclaims about the code against the diff and the files; that the review covered what it was asked toDocument analysis
Takes actions with tools (support, operations, booking agents)the task, the answer, the tools called with their arguments and results, the tools allowedthat the right tools were called, and only allowed ones, in a sensible order and number of steps; claims against tool resultsAnswers from sources

If everything you tick is a review, answers are judged as Document analysis; otherwise as Answers from sources. Ticking both kinds is fine: add a scoring rule for each kind of step later in the evaluation policy.

Code review, stated plainly. variA/Bly checks that what the review says about the code is true of the code it was shown ("function X has no null check" must be true of the code), and that it addressed what it was asked to. It does not run the code, and it does not judge whether a suggested fix is the best fix.

Text-to-SQL never touches your database. Your app sends the SQL text and the rows it already has. You declare the schema once, under Evaluation policy › Data definitions › SQL checks; never send it, or any credentials, with an answer.

One agent or several

One agentSeveral agents
What one event isthe answereach agent's step; one run is one session
What else is sentnothingthe same session id on every step, each step's agent name, which step writes the final answer, hand-offs, the tools each step called
The final answer is checked againstits own inputseverything the run gathered: the rows, documents and files every earlier step saw, plus what the earlier steps wrote
A chat with several agentsone session per conversation; each turn is a run inside it; the conversation is ended once

Your own code (Python)

Install the SDK and set VARIABLY_API_KEY and VARIABLY_BASE_URL where your app runs, as in Connect your app. Then, after each answer:

Answers from documents, one agent:

from variably import observe

observe(prompt=question, response=answer, reference_materials=chunks)   # the chunks the model was given

Text-to-SQL, one agent: the SQL that ran, and the rows as a reference with column names:

observe(prompt=question, response=answer, sql=sql,
        reference_materials=[{"id": "q1", "source": "q1", "rows": rows}])

Chat assistant (with any of the above on each turn): the same conversation id on every turn, and the conversation ended once, after its last turn:

import os
from variably import VariablyClient, observe

client = VariablyClient({"api_key": os.environ["VARIABLY_API_KEY"],
                         "base_url": os.environ["VARIABLY_BASE_URL"]})

observe(prompt=message, response=reply, session_id=conversation_id)   # every turn
client.end_session(conversation_id)                                  # once, when it is over

Several agents (any use case): one call per step with the same session id, and the step that writes the final answer marked. evidence_scope="run" checks the final answer against everything the run gathered, so the analyst's figures below are checked against the rows the SQL step fetched:

observe(prompt=step_input, response=sql_summary, session_id=run_id, agent_name="sql_writer",
        sql=sql, reference_materials=[{"id": "q1", "source": "q1", "rows": rows}])

observe(prompt=question, response=final_answer, session_id=run_id, agent_name="analyst",
        is_aggregator=True, evidence_scope="run")

Add handoff={"to": "<next agent>", "fields": {...}} on a step that passes work on, and tools_called= on a step that called tools. See Score a multi-agent trace.

Reviews documents: every document the model read, whole, each with its file name:

observe(prompt=task, response=review,
        reference_materials=[{"id": "INV-2041.pdf", "source": "INV-2041.pdf", "content": invoice_text},
                             {"id": "PO-7731.pdf", "source": "PO-7731.pdf", "content": order_text}])

Reviews code: the diff and every file the agent read, with their paths:

observe(prompt=task, response=review,
        reference_materials=[{"id": "diff", "source": "PR 42", "content": diff_text},
                             {"id": "src/pay.py", "source": "src/pay.py", "content": file_text}])

Takes actions with tools: the tools called, with arguments and results, and the tools allowed:

observe(prompt=request, response=answer,
        tools_called=[{"name": "refund_order", "arguments": {"order_id": "A-1042"},
                       "output": {"status": "refunded"}}],
        allowed_tools=["lookup_order", "refund_order"])

LangGraph

pip install "variably-sdk[langgraph]"
from variably.integrations.langgraph import instrument

instrument(graph, session_id_key="thread_id")   # before graph.compile()

Every node becomes a step of one run, and the last node is the final step, checked against everything the run gathered. SQL, rows, documents and files are read from tool messages. Add only what your app needs:

  • A query tool whose SQL argument has an unusual name: sql_tools={"bq_execute": "statement"}. Arguments named sql, query, statement, sql_query or query_text are read without it.
  • Documents kept in the graph's state rather than in tool results: context_extractor=.
  • A chat: session_id_key names the thread id, the same on every turn; end the conversation with client.end_session(thread_id) when it is over.

See LangGraph.

Google ADK and Vertex AI Agent Engine

pip install "variably-sdk[adk]"
from variably.integrations.adk import VariablyPlugin

runner = Runner(agent=root_agent, app_name="analytics", session_service=session_service,
                plugins=[VariablyPlugin()])

Captured with no other code: each agent's step, its tools, hand-offs, the final step, the SQL from the query tool's arguments, the rows from its result, documents and files. Sub-agents and agents called as tools join the parent run. Only for an unusual tool shape:

VariablyPlugin(sql_tools={"bq_execute": "statement"},
               evidence_extractors={"fetch_contract": lambda r: [{"id": r["no"], "content": r["body"]}]})

For a chat, run each conversation in one ADK session and end it with client.end_session(...) when it is over. See Google ADK & Vertex AI.

Microsoft Agent Framework

pip install "variably-sdk[agent-framework]"
from variably.integrations.agent_framework import instrument

instrument()   # once at start-up: the framework's tracing, with message content, sent to variA/Bly

The framework records each agent run, model call and tool call. instrument() turns that on with message content recorded (the framework calls it sensitive data) and sends it to variA/Bly; it warns at start-up when content recording is off, because the answers would arrive empty. With several agents, each agent run is a step and the agent that answers the user is the final step.

SQL is read from a tool argument named sql, query, statement, sql_query or query_text, and from database spans; the rows from that tool's result. For a query tool whose SQL argument has another name:

instrument(sql_tools={"bq_execute": "statement"})

.NET. Use the framework's own OpenTelemetry setup with an OTLP exporter, and enable sensitive data so answers arrive with their text. Ask us for the exporter settings for your project.

Azure AI Foundry Agent Service

Azure records no message content unless you turn it on. Set it where your app runs, then add the variA/Bly exporter and Azure's own tracing instrumentor for the Azure SDK you use:

import os
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"

from variably.integrations.otel import instrument
instrument()   # adds the variA/Bly exporter; keep any exporter you already have
# then Azure's tracing instrumentor for the Azure SDK you use, as Azure documents it

For a query tool whose SQL argument has an unusual name, name it: instrument(sql_tools={"bq_execute": "statement"}).

Connected agents and multi-agent threads arrive as steps of one run. Results of hosted tools (file search, code interpreter, Azure AI Search) are read when they appear in the trace; when they do not, the event says which input is missing.

MLflow traces

Import traces MLflow already holds, or send live ones:

from variably.integrations.mlflow import log_traces

report = log_traces(traces, dry_run=True)   # read the report, then run it without dry_run

SQL and rows are read from tool spans and database spans. For a query tool whose SQL argument has an unusual name, pass sql_tools={"bq_execute": "statement"} to log_traces(), or to instrument() for live traces. See MLflow traces.

Anything else on OpenTelemetry

CrewAI, LlamaIndex, the OpenAI Agents SDK, Semantic Kernel, TypeScript apps and others:

from variably.integrations.otel import instrument

instrument()   # at start-up, after your own tracer provider is set up

Turn on message content capture in the instrumentation that produces the spans; it is off by default. SQL and rows are read from tool spans and database spans. For a query tool whose SQL argument has an unusual name: instrument(sql_tools={"bq_execute": "statement"}). See OpenTelemetry.

A query that returned no rows counts as received, with every framework: the rows arrived, there were none.

Paste this into your coding agent

The Connect step writes this prompt for your combination. Here it is for several agents that query a database, built with Google ADK:

Prompt for your coding agent
Connect this application to variA/Bly with the SDK's Google ADK / Vertex AI Agent Engine setup, so every answer is scored with no observe() calls in my code. It queries a database (text-to-SQL), with several agents that hand work to each other. What gets checked: the SQL is checked against the schema and query rules I declare in variA/Bly, and every figure in the answer against the rows the query returned (variA/Bly never connects to my database).

Look before you change anything:
1. Find where the Runner is created (or the AdkApp, on Vertex AI Agent Engine), the root agent, each sub-agent and the tools each can call. Show me the file paths and line numbers before you change anything.

Then change only what this needs:
2. Add the dependency the way the project manages them: pip install "variably-sdk[adk]" (Python 3.10 or later, google-adk 2.x).
3. Register the plugin where the Runner or AdkApp is created:

   from variably.integrations.adk import VariablyPlugin

   plugins=[VariablyPlugin()]

   In a script or job that exits after a run, call plugin.flush() before it exits; a server does
   not need it.

   The integration reads what the checks need from the run itself: SQL from the query tool's
   arguments, rows from its result, documents from retrieval tools, and files and diffs from code
   tools. Do not add observe() calls.
4. Only if the code needs them:
   - VariablyPlugin(sql_tools={"<query tool>": "<argument holding the SQL>"}) only for a query
     tool whose SQL argument is not named sql, query, statement, sql_query or query_text.
   - Rows are read when the query tool returns a list of records, {"rows": [...]}, {"columns": [...], "rows": [[...]]} or {"data": [...]}.
     A query that returned no rows counts as received: return the empty result, do not drop it.
     Do not send the database schema or any credentials: the schema is declared once in variA/Bly
     (Evaluation policy › Data definitions › SQL checks).
   - Several agents: sub-agents and agents called as tools join the parent run; the agent that
     answers the user is the final step, checked against everything the run gathered.
5. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment, or from the secrets or
   settings loader the app already uses. Never hard-code the API key and never commit it.
6. Use only the arguments named above; do not invent others.

When you are done, tell me where the setup is and which options you set and why, and the name of the query tool and its SQL argument.

Verify: run one request; the run appears under Sessions in variA/Bly with each agent as a step, and its steps appear in Live events.

The First event step in variA/Bly then checks that the event arrived with: the question and the answer text; the SQL that ran; the rows each query returned; one session id per run or conversation; which step writes the final answer. Anything missing is listed there with the reason and the fix, and its checks show "Couldn't check" until it arrives.

Missing inputs show Couldn't check

Each answer lists the inputs its checks need and which ones arrived. A check whose input never arrived is never left out and never shown as a pass: it shows Couldn't check with the reason, for example "no SQL was sent with this answer".

  • Create a project › First event lists the inputs for your latest event: each one ✓ Received, or Missing with the reason and the fix for what your app is built with.
  • In Live events, an answer with a missing input shows 1 input missing in its header. Click it, or open Checks, to see which input, why, and how to send it.
  • An answer where no check failed but a check could not run for a missing input has the status Couldn't check, in the list and in its header, never Healthy. A failing check still makes it Needs attention.
  • When your project declares SQL checks but none of its recent answers carried SQL, First event and API keys say so, with the fix for what your app is built with.
  • With several agents, the final answer's Details tab lists the evidence the run's other steps added, each piece with the step it came from, and the SQL those steps ran (checked on the step that ran it, listed here for context).

The usual causes:

MissingUsual causeFix
The SQL that ranown code: sql= not passed; a framework: the query tool's SQL argument has an unusual namepass sql=; or name the tool with sql_tools= (LangGraph and OpenTelemetry-based setups on instrument(), ADK on VariablyPlugin(), MLflow imports on log_traces())
The rows each query returnedown code: rows not sent; a framework: the tool returns rows in a shape not recognisedsend them as {"id", "source", "rows"}; or return a list of records. A query that returned no rows counts as received: own code names it in metadata={"empty_results": ["q1"]}; the integrations do this for you
The documents the answer was based onthe passages, documents or files were not sent; Azure or Agent Framework without content recordingpass reference_materials=; or turn content recording on
The question and the answer textcontent recording is off in Azure, Agent Framework or OpenTelemetryturn content recording on
Which step writes the final answerseveral agents, and no step markedown code: is_aggregator=True on that step