Connect your app: every combination
The exact setup for what your AI does, one agent or several, and what it is built with, and what is checked for each use case.
Your app sends every input its checks need, for your combination, and nothing shows Couldn't check for want of an input.
- A project and its API key
- Your answers to the three questions on Create a project
Which checks run on your answers depends on three things: what your AI does, whether it is one agent or several, and what it is built with. Create a project asks all three, and its Connect step gives one setup for exactly your combination. This page has the same setup for every combination, so you can see what your app will be asked to send and why.
For a framework, setup is one line. LangGraph, Google ADK, Microsoft Agent Framework, Azure AI Foundry,
MLflow and OpenTelemetry already hold the SQL your agent ran, the rows it got back, the documents it
retrieved and the files it read. variA/Bly reads them from the agent's tool calls, so you write no
observe() call. With your own code, one call per answer (or per step) sends them.
What gets checked for each use case
Tick every use case that applies: a chat assistant with several agents that queries a database is chat + database + several agents, and gets every check below whose inputs arrive.
| What your AI does | What must arrive with each answer | What is checked | Answers are judged as |
|---|---|---|---|
| Answers questions from documents (RAG, Q&A) | the question, the answer, and the passages the model was given | each claim against the passages, whether the right passages were found, answer quality, your business rules | Answers from sources |
| Chat assistant (several turns) | the same for each turn, plus one conversation id; the conversation is ended after its last turn | the above, plus whether the turns agree with each other and your rules over the whole conversation | Answers from sources |
| Queries a database (text-to-SQL) | the question, the answer, the SQL that ran, and the rows each query returned | the SQL against the schema and query rules you declare; every figure in the answer against the rows | Answers from sources |
| Reviews documents (invoices, contracts, claims, reconciliations) | the task, the review, and every document read, whole | claims and figures against the documents; that the right documents were used; arithmetic; your business rules | Document analysis |
| Reviews code (a code review agent) | the task, the review, the diff and every file the agent read, with paths | claims about the code against the diff and the files; that the review covered what it was asked to | Document analysis |
| Takes actions with tools (support, operations, booking agents) | the task, the answer, the tools called with their arguments and results, the tools allowed | that the right tools were called, and only allowed ones, in a sensible order and number of steps; claims against tool results | Answers from sources |
If everything you tick is a review, answers are judged as Document analysis; otherwise as Answers from sources. Ticking both kinds is fine: add a scoring rule for each kind of step later in the evaluation policy.
Code review, stated plainly. variA/Bly checks that what the review says about the code is true of the code it was shown ("function X has no null check" must be true of the code), and that it addressed what it was asked to. It does not run the code, and it does not judge whether a suggested fix is the best fix.
Text-to-SQL never touches your database. Your app sends the SQL text and the rows it already has. You declare the schema once, under Evaluation policy › Data definitions › SQL checks; never send it, or any credentials, with an answer.
One agent or several
| One agent | Several agents | |
|---|---|---|
| What one event is | the answer | each agent's step; one run is one session |
| What else is sent | nothing | the same session id on every step, each step's agent name, which step writes the final answer, hand-offs, the tools each step called |
| The final answer is checked against | its own inputs | everything the run gathered: the rows, documents and files every earlier step saw, plus what the earlier steps wrote |
| A chat with several agents | one session per conversation; each turn is a run inside it; the conversation is ended once |
Your own code (Python)
Install the SDK and set VARIABLY_API_KEY and VARIABLY_BASE_URL where your app runs, as in
Connect your app. Then, after each answer:
Answers from documents, one agent:
from variably import observe
observe(prompt=question, response=answer, reference_materials=chunks) # the chunks the model was given
Text-to-SQL, one agent: the SQL that ran, and the rows as a reference with column names:
observe(prompt=question, response=answer, sql=sql,
reference_materials=[{"id": "q1", "source": "q1", "rows": rows}])
Chat assistant (with any of the above on each turn): the same conversation id on every turn, and the conversation ended once, after its last turn:
import os
from variably import VariablyClient, observe
client = VariablyClient({"api_key": os.environ["VARIABLY_API_KEY"],
"base_url": os.environ["VARIABLY_BASE_URL"]})
observe(prompt=message, response=reply, session_id=conversation_id) # every turn
client.end_session(conversation_id) # once, when it is over
Several agents (any use case): one call per step with the same session id, and the step that writes the
final answer marked. evidence_scope="run" checks the final answer against everything the run gathered,
so the analyst's figures below are checked against the rows the SQL step fetched:
observe(prompt=step_input, response=sql_summary, session_id=run_id, agent_name="sql_writer",
sql=sql, reference_materials=[{"id": "q1", "source": "q1", "rows": rows}])
observe(prompt=question, response=final_answer, session_id=run_id, agent_name="analyst",
is_aggregator=True, evidence_scope="run")
Add handoff={"to": "<next agent>", "fields": {...}} on a step that passes work on, and
tools_called= on a step that called tools. See Score a multi-agent trace.
Reviews documents: every document the model read, whole, each with its file name:
observe(prompt=task, response=review,
reference_materials=[{"id": "INV-2041.pdf", "source": "INV-2041.pdf", "content": invoice_text},
{"id": "PO-7731.pdf", "source": "PO-7731.pdf", "content": order_text}])
Reviews code: the diff and every file the agent read, with their paths:
observe(prompt=task, response=review,
reference_materials=[{"id": "diff", "source": "PR 42", "content": diff_text},
{"id": "src/pay.py", "source": "src/pay.py", "content": file_text}])
Takes actions with tools: the tools called, with arguments and results, and the tools allowed:
observe(prompt=request, response=answer,
tools_called=[{"name": "refund_order", "arguments": {"order_id": "A-1042"},
"output": {"status": "refunded"}}],
allowed_tools=["lookup_order", "refund_order"])
LangGraph
pip install "variably-sdk[langgraph]"
from variably.integrations.langgraph import instrument
instrument(graph, session_id_key="thread_id") # before graph.compile()
Every node becomes a step of one run, and the last node is the final step, checked against everything the run gathered. SQL, rows, documents and files are read from tool messages. Add only what your app needs:
- A query tool whose SQL argument has an unusual name:
sql_tools={"bq_execute": "statement"}. Arguments namedsql,query,statement,sql_queryorquery_textare read without it. - Documents kept in the graph's state rather than in tool results:
context_extractor=. - A chat:
session_id_keynames the thread id, the same on every turn; end the conversation withclient.end_session(thread_id)when it is over.
See LangGraph.
Google ADK and Vertex AI Agent Engine
pip install "variably-sdk[adk]"
from variably.integrations.adk import VariablyPlugin
runner = Runner(agent=root_agent, app_name="analytics", session_service=session_service,
plugins=[VariablyPlugin()])
Captured with no other code: each agent's step, its tools, hand-offs, the final step, the SQL from the query tool's arguments, the rows from its result, documents and files. Sub-agents and agents called as tools join the parent run. Only for an unusual tool shape:
VariablyPlugin(sql_tools={"bq_execute": "statement"},
evidence_extractors={"fetch_contract": lambda r: [{"id": r["no"], "content": r["body"]}]})
For a chat, run each conversation in one ADK session and end it with client.end_session(...) when it is
over. See Google ADK & Vertex AI.
Microsoft Agent Framework
pip install "variably-sdk[agent-framework]"
from variably.integrations.agent_framework import instrument
instrument() # once at start-up: the framework's tracing, with message content, sent to variA/Bly
The framework records each agent run, model call and tool call. instrument() turns that on with message
content recorded (the framework calls it sensitive data) and sends it to variA/Bly; it warns at start-up
when content recording is off, because the answers would arrive empty. With several agents, each agent run
is a step and the agent that answers the user is the final step.
SQL is read from a tool argument named sql, query, statement, sql_query or query_text, and from
database spans; the rows from that tool's result. For a query tool whose SQL argument has another name:
instrument(sql_tools={"bq_execute": "statement"})
.NET. Use the framework's own OpenTelemetry setup with an OTLP exporter, and enable sensitive data so answers arrive with their text. Ask us for the exporter settings for your project.
Azure AI Foundry Agent Service
Azure records no message content unless you turn it on. Set it where your app runs, then add the variA/Bly exporter and Azure's own tracing instrumentor for the Azure SDK you use:
import os
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"
from variably.integrations.otel import instrument
instrument() # adds the variA/Bly exporter; keep any exporter you already have
# then Azure's tracing instrumentor for the Azure SDK you use, as Azure documents it
For a query tool whose SQL argument has an unusual name, name it: instrument(sql_tools={"bq_execute": "statement"}).
Connected agents and multi-agent threads arrive as steps of one run. Results of hosted tools (file search, code interpreter, Azure AI Search) are read when they appear in the trace; when they do not, the event says which input is missing.
MLflow traces
Import traces MLflow already holds, or send live ones:
from variably.integrations.mlflow import log_traces
report = log_traces(traces, dry_run=True) # read the report, then run it without dry_run
SQL and rows are read from tool spans and database spans. For a query tool whose SQL argument has an unusual
name, pass sql_tools={"bq_execute": "statement"} to log_traces(), or to instrument() for live traces.
See MLflow traces.
Anything else on OpenTelemetry
CrewAI, LlamaIndex, the OpenAI Agents SDK, Semantic Kernel, TypeScript apps and others:
from variably.integrations.otel import instrument
instrument() # at start-up, after your own tracer provider is set up
Turn on message content capture in the instrumentation that produces the spans; it is off by default. SQL
and rows are read from tool spans and database spans. For a query tool whose SQL argument has an unusual
name: instrument(sql_tools={"bq_execute": "statement"}). See OpenTelemetry.
A query that returned no rows counts as received, with every framework: the rows arrived, there were none.
Paste this into your coding agent
The Connect step writes this prompt for your combination. Here it is for several agents that query a database, built with Google ADK:
Connect this application to variA/Bly with the SDK's Google ADK / Vertex AI Agent Engine setup, so every answer is scored with no observe() calls in my code. It queries a database (text-to-SQL), with several agents that hand work to each other. What gets checked: the SQL is checked against the schema and query rules I declare in variA/Bly, and every figure in the answer against the rows the query returned (variA/Bly never connects to my database).
Look before you change anything:
1. Find where the Runner is created (or the AdkApp, on Vertex AI Agent Engine), the root agent, each sub-agent and the tools each can call. Show me the file paths and line numbers before you change anything.
Then change only what this needs:
2. Add the dependency the way the project manages them: pip install "variably-sdk[adk]" (Python 3.10 or later, google-adk 2.x).
3. Register the plugin where the Runner or AdkApp is created:
from variably.integrations.adk import VariablyPlugin
plugins=[VariablyPlugin()]
In a script or job that exits after a run, call plugin.flush() before it exits; a server does
not need it.
The integration reads what the checks need from the run itself: SQL from the query tool's
arguments, rows from its result, documents from retrieval tools, and files and diffs from code
tools. Do not add observe() calls.
4. Only if the code needs them:
- VariablyPlugin(sql_tools={"<query tool>": "<argument holding the SQL>"}) only for a query
tool whose SQL argument is not named sql, query, statement, sql_query or query_text.
- Rows are read when the query tool returns a list of records, {"rows": [...]}, {"columns": [...], "rows": [[...]]} or {"data": [...]}.
A query that returned no rows counts as received: return the empty result, do not drop it.
Do not send the database schema or any credentials: the schema is declared once in variA/Bly
(Evaluation policy › Data definitions › SQL checks).
- Several agents: sub-agents and agents called as tools join the parent run; the agent that
answers the user is the final step, checked against everything the run gathered.
5. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment, or from the secrets or
settings loader the app already uses. Never hard-code the API key and never commit it.
6. Use only the arguments named above; do not invent others.
When you are done, tell me where the setup is and which options you set and why, and the name of the query tool and its SQL argument.
Verify: run one request; the run appears under Sessions in variA/Bly with each agent as a step, and its steps appear in Live events.
The First event step in variA/Bly then checks that the event arrived with: the question and the answer text; the SQL that ran; the rows each query returned; one session id per run or conversation; which step writes the final answer. Anything missing is listed there with the reason and the fix, and its checks show "Couldn't check" until it arrives.Missing inputs show Couldn't check
Each answer lists the inputs its checks need and which ones arrived. A check whose input never arrived is never left out and never shown as a pass: it shows Couldn't check with the reason, for example "no SQL was sent with this answer".
- Create a project › First event lists the inputs for your latest event: each one ✓ Received, or Missing with the reason and the fix for what your app is built with.
- In Live events, an answer with a missing input shows 1 input missing in its header. Click it, or open Checks, to see which input, why, and how to send it.
- An answer where no check failed but a check could not run for a missing input has the status Couldn't check, in the list and in its header, never Healthy. A failing check still makes it Needs attention.
- When your project declares SQL checks but none of its recent answers carried SQL, First event and API keys say so, with the fix for what your app is built with.
- With several agents, the final answer's Details tab lists the evidence the run's other steps added, each piece with the step it came from, and the SQL those steps ran (checked on the step that ran it, listed here for context).
The usual causes:
| Missing | Usual cause | Fix |
|---|---|---|
| The SQL that ran | own code: sql= not passed; a framework: the query tool's SQL argument has an unusual name | pass sql=; or name the tool with sql_tools= (LangGraph and OpenTelemetry-based setups on instrument(), ADK on VariablyPlugin(), MLflow imports on log_traces()) |
| The rows each query returned | own code: rows not sent; a framework: the tool returns rows in a shape not recognised | send them as {"id", "source", "rows"}; or return a list of records. A query that returned no rows counts as received: own code names it in metadata={"empty_results": ["q1"]}; the integrations do this for you |
| The documents the answer was based on | the passages, documents or files were not sent; Azure or Agent Framework without content recording | pass reference_materials=; or turn content recording on |
| The question and the answer text | content recording is off in Azure, Agent Framework or OpenTelemetry | turn content recording on |
| Which step writes the final answer | several agents, and no step marked | own code: is_aggregator=True on that step |