MLflow traces
Evaluate the agents you already trace with MLflow — import recorded traces with the Python SDK, or send them as they happen over OpenTelemetry.
Your MLflow traces evaluated — each agent span scored as one answer, with its retrieved documents, tool calls, SQL and hand-offs checked.
- A project and its API key
- pip install variably-sdk (add [otel] for live traces)
- Traces recorded with MLflow 3, or an MLflow trace export
How it works
- Records traces as it already does
- search_traces or an export file
- Or live spans over OpenTelemetry
- log_traces maps each AGENT span
- dry_run=True previews the mapping
- Reports every skip with its reason
- One answer per agent
- Documents, tools, SQL and hand-offs checked
- Same sessions and dashboards
Agents traced with MLflow — in a notebook or job on Databricks, or anywhere else — are evaluated from the traces they already record. Import the traces MLflow has stored, or send them as they happen.
What changed (16 Nov 2026): MLflow traces can now be evaluated directly — import recorded traces with
log_traces(), or send live traces throughinstrument().
Two ways
| Import recorded traces | Live traces over OpenTelemetry | |
|---|---|---|
| Use it for | Traces already in MLflow: a notebook, a scheduled job, CI, a backfill | Every new trace, as your app runs |
| You call | log_traces(traces) | instrument() once at start-up |
| Preview before sending | Yes, dry_run=True | No |
| Environment, tags, metadata, dataset runs | Yes | Not yet |
| Session | session_key= tag, else the MLflow session, else the trace id | The session MLflow puts on the exported root span, else the trace's request id |
| Undeclared document metadata | That answer is refused and listed under failed | Dropped from the answer, with the reason on the answer |
Both read the spans the same way. Use one path per trace: a trace imported and also sent live is evaluated twice, in two different sessions.
Before you start
pip install variably-sdk # import only
pip install "variably-sdk[otel]" # live traces too
export VARIABLY_API_KEY=vbl_your_key_here # a key of the project the answers should land in
export VARIABLY_BASE_URL=https://api.variably.dev
Paste this into your coding agent
Paste this into Claude Code, Cursor, or whichever agent has your repository open. It finds how your app traces with MLflow and sets up the import or live traces.
Evaluate this application's MLflow traces in variA/Bly: either import traces MLflow already holds, or send live traces as they happen. Ask me which, if it is not clear from the code. Look first: 1. Find where the app uses MLflow tracing (mlflow.trace, autolog, or a framework's MLflow integration), and whether traces are exported or searched in a job or notebook. Show me the file paths before you change anything. To import traces (a job or notebook): 2. pip install variably-sdk, then: from variably.integrations.mlflow import log_traces report = log_traces(traces, dry_run=True) print(report) traces is what the app's MLflow search returns, or a path to an exported JSONL file. Read the dry-run report (what would be sent, skipped and why) and show it to me before running it again without dry_run. Optional arguments: environment="prod", tags=[...], session_key=<trace tag holding the conversation id>, and reference_metadata_keys=[...] with the project's document-scope keys when documents carry metadata. To send live traces: 2. pip install "variably-sdk[otel]", set MLFLOW_USE_DEFAULT_TRACER_PROVIDER=false in the app's environment, and at start-up, before the first trace: from variably.integrations.otel import instrument processor = instrument() In a script that exits quickly, call processor.force_flush() before it exits. 3. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment, or from the secrets or settings loader the app already uses. Never hard-code the API key and never commit it. 4. Use only the arguments named above; do not invent others. When you are done, tell me which path you set up and where. Verify: the imported or live answers appear in variA/Bly under Live events, and runs with several agents appear under Sessions.
1. Import recorded traces
- Fetch the traces as a list with
mlflow.search_traces(..., return_type="list"), or point at an export file. - Run
log_traces(traces, dry_run=True)and read the report: how many answers would be sent, what was skipped and why. - Look at one payload (
report.payloads[0]): the question, answer, documents and session. - Run it again without
dry_run. The answers arrive taggedmlflowand are scored like any other.
import mlflow
from variably.integrations.mlflow import log_traces
mlflow.set_tracking_uri("databricks") # or your tracking server
traces = mlflow.search_traces(
experiment_ids=["1234"],
filter_string="attributes.timestamp_ms > 1763251200000", # traces since 16 Nov 2026
max_results=500,
return_type="list",
)
# 1. Preview: nothing is sent
report = log_traces(traces, dry_run=True, reference_metadata_keys=["plan_year", "state"])
print(report)
print(report.payloads[0])
# 2. Send
report = log_traces(traces, environment="prod", tags=["nightly-import"],
reference_metadata_keys=["plan_year", "state"])
print(report)
for r in report.results[:3]:
print(r["trace_id"], r["agent_name"], r["observation_id"], r["status"])
What the report looks like
A dry run of one benefits trace:
MLflow import: 1 trace(s); would send 1, skipped 0, failed 0
note tr-7f3a9c2e5b1d4e8f benefits_agent: document metadata not sent as scope metadata: page (pass reference_metadata_keys= with your project's scope dimensions)
A real import of 120 traces, where two had no usable answer and one carried an undeclared metadata key:
MLflow import: 120 trace(s); sent 131, skipped 2, failed 1
skipped tr-0b81... router_agent: no answer text in the span's outputs (dict(keys=[route]))
skipped tr-55c2... : trace has no spans (export it with its span data)
failed tr-9e04... benefits_agent: … reference_materials[0] (sob-2026-p2-c4).metadata: unknown key "region"; the project's scope dimensions take plan_year, state
The counts are answers, not traces: a trace with three agent spans sends three. The report also holds
results (trace id, agent, the answer's id and status), skipped_reasons, failures, notes and
payloads.
One dry-run payload
report.payloads[0] is exactly what would be sent for one agent span:
{
"prompt": "What is the in-network out-of-pocket maximum for an individual on the Gold PPO plan in 2026?",
"response": "For 2026 the Gold PPO in-network out-of-pocket maximum is $6,500 for an individual.",
"session_id": "chat-4821",
"metadata": {"source": "mlflow", "mlflow_trace_id": "tr-7f3a9c2e5b1d4e8f", "mlflow_span_id": "a1b2c3d4e5f60001"},
"tags": ["mlflow"],
"agent_name": "benefits_agent",
"is_aggregator": true,
"reference_materials": [{
"id": "sob-2026-p2-c4",
"content": "Gold PPO 2026. In-network out-of-pocket maximum: $6,000 individual / $12,000 family.",
"source": "Summary of Benefits 2026, p.2",
"metadata": {"plan_year": 2026, "state": "CA"}
}],
"retrieval_query": "Gold PPO 2026 out-of-pocket maximum",
"model": "gpt-4o-mini",
"prompt_tokens": 812,
"completion_tokens": 41,
"latency_ms": 2400
}
Once sent, this answer's claim is Contradicted with the explanation "The answer says $6,500; Summary of Benefits 2026, p.2 says $6,000." — see Read a claim's verdict.
Importing an export file
log_traces also takes a Trace from mlflow.get_trace(...), the output of .to_dict() or
.to_json(), a list of any of these, or the path of a JSON or JSONL file (one trace per line). MLflow
does not need to be installed to import a file. One JSONL line, trimmed to the fields that are read:
{"info": {"trace_id": "tr-7f3a9c2e5b1d4e8f",
"trace_metadata": {"mlflow.trace.session": "chat-4821"},
"tags": {"app": "benefits-assistant"}},
"data": {"spans": [
{"name": "benefits_agent", "span_id": "a1b2c3d4e5f60001", "parent_span_id": null,
"start_time_unix_nano": 1763290000000000000, "end_time_unix_nano": 1763290002400000000,
"attributes": {"mlflow.spanType": "\"AGENT\"",
"mlflow.spanInputs": "{\"question\": \"What is the in-network out-of-pocket maximum ...?\"}",
"mlflow.spanOutputs": "{\"answer\": \"For 2026 the Gold PPO ... is $6,500 for an individual.\"}"}},
{"name": "search_plan_docs", "span_id": "a1b2c3d4e5f60002", "parent_span_id": "a1b2c3d4e5f60001",
"attributes": {"mlflow.spanType": "\"RETRIEVER\"",
"mlflow.spanOutputs": "[{\"page_content\": \"Gold PPO 2026. In-network out-of-pocket maximum: $6,000 ...\", \"metadata\": {\"doc_uri\": \"Summary of Benefits 2026, p.2\", \"chunk_id\": \"sob-2026-p2-c4\", \"plan_year\": 2026, \"state\": \"CA\"}}]"}},
{"name": "ChatOpenAI", "span_id": "a1b2c3d4e5f60003", "parent_span_id": "a1b2c3d4e5f60001",
"attributes": {"mlflow.spanType": "\"CHAT_MODEL\"", "mlflow.llm.model": "\"gpt-4o-mini\"",
"mlflow.chat.tokenUsage": "{\"input_tokens\": 812, \"output_tokens\": 41}"}}
]}}
from variably.integrations.mlflow import log_traces
print(log_traces("exports/traces-2026-11-16.jsonl", dry_run=True))
print(log_traces("exports/traces-2026-11-16.jsonl", environment="staging"))
Options
| Option | Default | What it does |
|---|---|---|
dry_run=True | False | Maps the traces and fills report.payloads. Nothing is sent and no API key is needed |
environment= | none | The answer's environment, for filters and scoring rules |
tags= | ["mlflow"] | Added to mlflow, which every imported answer has |
metadata= | trace and span ids | Added to every answer; the MLflow trace and span ids are always kept |
session_key= | the MLflow session, else the trace id | A trace tag or metadata key holding your own session id. A trace without it uses the default, with a note |
handoff_key= | "handoff" | The key, in an agent's outputs, of the object it hands to the next agent; None turns hand-off checks off |
reference_metadata_keys= | every key shaped like a scope value | Which document metadata keys to send — your project's document scope dimensions |
run= + dataset_item_key= | none | Files each answer under a dataset run; dataset_item_key is the trace tag holding the dataset item id. Each needs the other; a trace without the tag is skipped |
client= | built from the two variables | Pass your own VariablyClient |
# A dataset run from MLflow traces: each trace is tagged with its dataset item id
traces = mlflow.search_traces(experiment_ids=["1234"],
filter_string="tags.release = '4.3'", return_type="list")
print(log_traces(traces, run="release-4.3", dataset_item_key="dataset_item"))
2. Send traces as they happen
- Set MLflow's own setting so it writes spans to the standard OpenTelemetry tracer provider.
- Call
instrument()once at start-up, before the first trace. - Run one request; it is evaluated about ten seconds after its last span ends.
pip install "variably-sdk[otel]" "mlflow>=3"
export MLFLOW_USE_DEFAULT_TRACER_PROVIDER=false # MLflow setting: use the OpenTelemetry provider
export VARIABLY_API_KEY=vbl_your_key_here
export VARIABLY_BASE_URL=https://api.variably.dev
import mlflow
from variably.integrations.otel import instrument
processor = instrument() # before the first trace; raises at start-up if a variable is missing
mlflow.set_experiment("benefits-assistant")
mlflow.openai.autolog() # or @mlflow.trace / mlflow.start_span, unchanged
@mlflow.trace(span_type="AGENT", name="benefits_agent")
def answer(question: str) -> dict:
docs = search_plan_docs(question) # a RETRIEVER span
text = write_answer(question, docs) # a CHAT_MODEL span from autolog
return {"answer": text}
answer("What is the in-network out-of-pocket maximum for Gold PPO in 2026?")
processor.force_flush() # in a short script, before it exits
MLflow keeps recording to your tracking server as before. With MLflow's option to emit the
OpenTelemetry GenAI conventions switched on, spans arrive in GenAI form and are read the same way.
Without MLFLOW_USE_DEFAULT_TRACER_PROVIDER=false, MLflow keeps its spans to itself and nothing
arrives — no error is shown. More on the exporter in OpenTelemetry.
How a trace becomes answers
| In your MLflow trace | Becomes |
|---|---|
Each AGENT span | One answer, named after the span |
A trace with no AGENT span | One answer from the root span |
The agent's inputs: question, query, input, prompt, user_input, request or the last user message | The question (else the root span's, else the first model call's user message) |
The agent's outputs: a string, output, answer, response, content, result, text, final_answer, the last assistant message, or an OpenAI or Anthropic response | The answer; when the agent recorded no outputs, its last model call's answer |
RETRIEVER spans under the agent — MLflow Documents, records with page_content, content or text, or strings | The documents the answer is checked against; doc_uri (or source, uri, url) is the source named in explanations, chunk_id (or id, doc_id) the id; the retriever's query is the retrieval query |
Document metadata such as plan_year, state, effective_from | The document's scope, for "Right version of the source" — text, a year (1900–2100) or a list of text |
TOOL spans under the agent | Tool calls; a list of flat records (or {"rows": [...]}) is sent as query result rows, so figures are checked against them; SQL text in the inputs (sql, query, statement) is checked as the answer's SQL |
CHAT_MODEL / LLM spans | Model name, provider and token counts, added up per agent |
| An agent inside another agent | Its own answer in the same session, with its parent recorded |
An agent output handoff holding an object | A hand-off to the next top-level agent, checked against your hand-off contracts; to, fields and message inside it are used as given, any other object is the fields |
| An agent with no answer text but a hand-off or tool calls | A routing step — grounding is not applicable rather than zero |
| The top-level agent that ends last | The final step of the session |
# A hand-off from intake to adjudication, recorded in the intake agent's outputs
@mlflow.trace(span_type="AGENT", name="intake")
def intake(claim_text):
summary = summarise(claim_text)
return {"answer": summary,
"handoff": {"to": "adjudication",
"fields": {"onset_date": "2026-03-01", "leave_type": "STD"}}}
Edge cases and limits
Document metadata keys must be declared. Each metadata key sent with a document must be a scope
dimension your project declares. By default log_traces sends every key that looks like a scope
value, so an undeclared key such as region fails that answer:
failed tr-9e04... benefits_agent: … reference_materials[0] (sob-2026-p2-c4).metadata: unknown key "region"; the project's scope dimensions take plan_year, state
Fix it either way: name the declared keys with reference_metadata_keys=["plan_year", "state"] (the
others are left out and listed in a note), or declare the new dimension in the project's document
scope and publish. Live traces drop undeclared metadata from the answer, record why, and still score it.
Spans outside agents are ignored. When a trace has any AGENT span, a retriever, tool or model
span that is not under an agent belongs to no answer, and its documents are not used. Call the
retriever from inside the agent function, or nest it with mlflow.start_span. A trace with no agent
span is one answer from its root, with every span under it.
Sessions differ between the two paths. An import uses session_key=, else MLflow's session
(mlflow.trace.session), else the trace id. Live traces use the session MLflow puts on the exported
root span — which MLflow does only in some set-ups, such as Unity Catalog on Databricks — and
otherwise the trace's request id, so each live trace can be its own session. To group conversations
reliably, import with session_key= pointing at your own conversation tag.
Also:
- Nothing is guessed: a trace or agent whose shape is not recognised is skipped, with the reason in the report (or on the answer, for live traces).
- Tool output text is kept up to 4,000 characters per call; rows are kept in full.
- Live traces carry the environment, tags and dataset run you set with
variably.context()(OpenTelemetry). An import files only each trace's final answer under its dataset item; the other steps are logged without the run. - An authentication error (a wrong or revoked key) stops the import at once; any other error is counted per answer under failed, and the import continues.
Troubleshooting
| You see | Why | Fix |
|---|---|---|
skipped … no answer text in the span's outputs | The agent's outputs use keys that are not read | Return the answer under answer, output or response, or as a string |
skipped … trace has no spans | The export has the trace's info only | Export traces with their span data |
failed … unknown key "region" | A document carries metadata the project doesn't declare | reference_metadata_keys= with the declared keys, or declare the dimension |
| Answers arrive but grounding is not applicable | The retriever span is outside the agent, or its outputs are not a list of documents | Nest the retriever under the agent; return documents with text |
| Live traces never arrive | MLflow still uses its own tracer provider, instrument() ran after the first trace, or a script exited before flushing | MLFLOW_USE_DEFAULT_TRACER_PROVIDER=false, instrument() at start-up, force_flush() before exit |
| Every live trace is its own session | MLflow did not put a session on the exported root span | Import with session_key=, or use observe() with your own session_id |
AuthenticationError: Unauthorized | The key is wrong, revoked, or its project was deleted | Create a key in the target project and update VARIABLY_API_KEY |
FAQ
Does MLflow need to be installed to import? Only to fetch traces with mlflow.search_traces.
Importing an export file needs only the variA/Bly SDK.
Can I pass the DataFrame search_traces returns by default? Yes, if it has its trace column;
return_type="list" is simpler.
If I import the same trace twice, do I get duplicates? Yes — each import sends new answers. Import each trace once, for example by filtering on time.
Are imported answers scored differently? No. They are scored like any other answer and tagged mlflow.
Concepts on this page: Answers, sessions and agents · Integrations, data and usage