All docs
Connect your appFrameworks and tracesMLflow tracescode

MLflow traces

Evaluate the agents you already trace with MLflow — import recorded traces with the Python SDK, or send them as they happen over OpenTelemetry.

What you'll have

Your MLflow traces evaluated — each agent span scored as one answer, with its retrieved documents, tool calls, SQL and hand-offs checked.

You'll need
  • A project and its API key
  • pip install variably-sdk (add [otel] for live traces)
  • Traces recorded with MLflow 3, or an MLflow trace export

How it works

MLflow
  1. Records traces as it already does
  2. search_traces or an export file
  3. Or live spans over OpenTelemetry
The SDK
  1. log_traces maps each AGENT span
  2. dry_run=True previews the mapping
  3. Reports every skip with its reason
variA/Bly
  1. One answer per agent
  2. Documents, tools, SQL and hand-offs checked
  3. Same sessions and dashboards

Agents traced with MLflow — in a notebook or job on Databricks, or anywhere else — are evaluated from the traces they already record. Import the traces MLflow has stored, or send them as they happen.

What changed (16 Nov 2026): MLflow traces can now be evaluated directly — import recorded traces with log_traces(), or send live traces through instrument().

Two ways

Import recorded tracesLive traces over OpenTelemetry
Use it forTraces already in MLflow: a notebook, a scheduled job, CI, a backfillEvery new trace, as your app runs
You calllog_traces(traces)instrument() once at start-up
Preview before sendingYes, dry_run=TrueNo
Environment, tags, metadata, dataset runsYesNot yet
Sessionsession_key= tag, else the MLflow session, else the trace idThe session MLflow puts on the exported root span, else the trace's request id
Undeclared document metadataThat answer is refused and listed under failedDropped from the answer, with the reason on the answer

Both read the spans the same way. Use one path per trace: a trace imported and also sent live is evaluated twice, in two different sessions.

Before you start

pip install variably-sdk              # import only
pip install "variably-sdk[otel]"      # live traces too

export VARIABLY_API_KEY=vbl_your_key_here      # a key of the project the answers should land in
export VARIABLY_BASE_URL=https://api.variably.dev

Paste this into your coding agent

Paste this into Claude Code, Cursor, or whichever agent has your repository open. It finds how your app traces with MLflow and sets up the import or live traces.

Prompt for your coding agent
Evaluate this application's MLflow traces in variA/Bly: either import traces MLflow already
holds, or send live traces as they happen. Ask me which, if it is not clear from the code.

Look first:
1. Find where the app uses MLflow tracing (mlflow.trace, autolog, or a framework's MLflow
   integration), and whether traces are exported or searched in a job or notebook. Show me the
   file paths before you change anything.

To import traces (a job or notebook):
2. pip install variably-sdk, then:

   from variably.integrations.mlflow import log_traces

   report = log_traces(traces, dry_run=True)
   print(report)

   traces is what the app's MLflow search returns, or a path to an exported JSONL file. Read
   the dry-run report (what would be sent, skipped and why) and show it to me before running
   it again without dry_run. Optional arguments: environment="prod", tags=[...],
   session_key=<trace tag holding the conversation id>, and reference_metadata_keys=[...] with
   the project's document-scope keys when documents carry metadata.

To send live traces:
2. pip install "variably-sdk[otel]", set MLFLOW_USE_DEFAULT_TRACER_PROVIDER=false in the app's
   environment, and at start-up, before the first trace:

   from variably.integrations.otel import instrument
   processor = instrument()

   In a script that exits quickly, call processor.force_flush() before it exits.

3. Read VARIABLY_API_KEY and VARIABLY_BASE_URL from the environment, or from the secrets or
   settings loader the app already uses. Never hard-code the API key and never commit it.
4. Use only the arguments named above; do not invent others.

When you are done, tell me which path you set up and where. Verify: the imported or live answers
appear in variA/Bly under Live events, and runs with several agents appear under Sessions.

1. Import recorded traces

  1. Fetch the traces as a list with mlflow.search_traces(..., return_type="list"), or point at an export file.
  2. Run log_traces(traces, dry_run=True) and read the report: how many answers would be sent, what was skipped and why.
  3. Look at one payload (report.payloads[0]): the question, answer, documents and session.
  4. Run it again without dry_run. The answers arrive tagged mlflow and are scored like any other.
import mlflow
from variably.integrations.mlflow import log_traces

mlflow.set_tracking_uri("databricks")          # or your tracking server
traces = mlflow.search_traces(
    experiment_ids=["1234"],
    filter_string="attributes.timestamp_ms > 1763251200000",   # traces since 16 Nov 2026
    max_results=500,
    return_type="list",
)

# 1. Preview: nothing is sent
report = log_traces(traces, dry_run=True, reference_metadata_keys=["plan_year", "state"])
print(report)
print(report.payloads[0])

# 2. Send
report = log_traces(traces, environment="prod", tags=["nightly-import"],
                    reference_metadata_keys=["plan_year", "state"])
print(report)
for r in report.results[:3]:
    print(r["trace_id"], r["agent_name"], r["observation_id"], r["status"])

What the report looks like

A dry run of one benefits trace:

MLflow import: 1 trace(s); would send 1, skipped 0, failed 0
  note tr-7f3a9c2e5b1d4e8f benefits_agent: document metadata not sent as scope metadata: page (pass reference_metadata_keys= with your project's scope dimensions)

A real import of 120 traces, where two had no usable answer and one carried an undeclared metadata key:

MLflow import: 120 trace(s); sent 131, skipped 2, failed 1
  skipped tr-0b81... router_agent: no answer text in the span's outputs (dict(keys=[route]))
  skipped tr-55c2... : trace has no spans (export it with its span data)
  failed tr-9e04... benefits_agent: … reference_materials[0] (sob-2026-p2-c4).metadata: unknown key "region"; the project's scope dimensions take plan_year, state

The counts are answers, not traces: a trace with three agent spans sends three. The report also holds results (trace id, agent, the answer's id and status), skipped_reasons, failures, notes and payloads.

One dry-run payload

report.payloads[0] is exactly what would be sent for one agent span:

{
  "prompt": "What is the in-network out-of-pocket maximum for an individual on the Gold PPO plan in 2026?",
  "response": "For 2026 the Gold PPO in-network out-of-pocket maximum is $6,500 for an individual.",
  "session_id": "chat-4821",
  "metadata": {"source": "mlflow", "mlflow_trace_id": "tr-7f3a9c2e5b1d4e8f", "mlflow_span_id": "a1b2c3d4e5f60001"},
  "tags": ["mlflow"],
  "agent_name": "benefits_agent",
  "is_aggregator": true,
  "reference_materials": [{
    "id": "sob-2026-p2-c4",
    "content": "Gold PPO 2026. In-network out-of-pocket maximum: $6,000 individual / $12,000 family.",
    "source": "Summary of Benefits 2026, p.2",
    "metadata": {"plan_year": 2026, "state": "CA"}
  }],
  "retrieval_query": "Gold PPO 2026 out-of-pocket maximum",
  "model": "gpt-4o-mini",
  "prompt_tokens": 812,
  "completion_tokens": 41,
  "latency_ms": 2400
}

Once sent, this answer's claim is Contradicted with the explanation "The answer says $6,500; Summary of Benefits 2026, p.2 says $6,000." — see Read a claim's verdict.

Importing an export file

log_traces also takes a Trace from mlflow.get_trace(...), the output of .to_dict() or .to_json(), a list of any of these, or the path of a JSON or JSONL file (one trace per line). MLflow does not need to be installed to import a file. One JSONL line, trimmed to the fields that are read:

{"info": {"trace_id": "tr-7f3a9c2e5b1d4e8f",
          "trace_metadata": {"mlflow.trace.session": "chat-4821"},
          "tags": {"app": "benefits-assistant"}},
 "data": {"spans": [
   {"name": "benefits_agent", "span_id": "a1b2c3d4e5f60001", "parent_span_id": null,
    "start_time_unix_nano": 1763290000000000000, "end_time_unix_nano": 1763290002400000000,
    "attributes": {"mlflow.spanType": "\"AGENT\"",
                   "mlflow.spanInputs": "{\"question\": \"What is the in-network out-of-pocket maximum ...?\"}",
                   "mlflow.spanOutputs": "{\"answer\": \"For 2026 the Gold PPO ... is $6,500 for an individual.\"}"}},
   {"name": "search_plan_docs", "span_id": "a1b2c3d4e5f60002", "parent_span_id": "a1b2c3d4e5f60001",
    "attributes": {"mlflow.spanType": "\"RETRIEVER\"",
                   "mlflow.spanOutputs": "[{\"page_content\": \"Gold PPO 2026. In-network out-of-pocket maximum: $6,000 ...\", \"metadata\": {\"doc_uri\": \"Summary of Benefits 2026, p.2\", \"chunk_id\": \"sob-2026-p2-c4\", \"plan_year\": 2026, \"state\": \"CA\"}}]"}},
   {"name": "ChatOpenAI", "span_id": "a1b2c3d4e5f60003", "parent_span_id": "a1b2c3d4e5f60001",
    "attributes": {"mlflow.spanType": "\"CHAT_MODEL\"", "mlflow.llm.model": "\"gpt-4o-mini\"",
                   "mlflow.chat.tokenUsage": "{\"input_tokens\": 812, \"output_tokens\": 41}"}}
 ]}}
from variably.integrations.mlflow import log_traces
print(log_traces("exports/traces-2026-11-16.jsonl", dry_run=True))
print(log_traces("exports/traces-2026-11-16.jsonl", environment="staging"))

Options

OptionDefaultWhat it does
dry_run=TrueFalseMaps the traces and fills report.payloads. Nothing is sent and no API key is needed
environment=noneThe answer's environment, for filters and scoring rules
tags=["mlflow"]Added to mlflow, which every imported answer has
metadata=trace and span idsAdded to every answer; the MLflow trace and span ids are always kept
session_key=the MLflow session, else the trace idA trace tag or metadata key holding your own session id. A trace without it uses the default, with a note
handoff_key="handoff"The key, in an agent's outputs, of the object it hands to the next agent; None turns hand-off checks off
reference_metadata_keys=every key shaped like a scope valueWhich document metadata keys to send — your project's document scope dimensions
run= + dataset_item_key=noneFiles each answer under a dataset run; dataset_item_key is the trace tag holding the dataset item id. Each needs the other; a trace without the tag is skipped
client=built from the two variablesPass your own VariablyClient
# A dataset run from MLflow traces: each trace is tagged with its dataset item id
traces = mlflow.search_traces(experiment_ids=["1234"],
                              filter_string="tags.release = '4.3'", return_type="list")
print(log_traces(traces, run="release-4.3", dataset_item_key="dataset_item"))

2. Send traces as they happen

  1. Set MLflow's own setting so it writes spans to the standard OpenTelemetry tracer provider.
  2. Call instrument() once at start-up, before the first trace.
  3. Run one request; it is evaluated about ten seconds after its last span ends.
pip install "variably-sdk[otel]" "mlflow>=3"
export MLFLOW_USE_DEFAULT_TRACER_PROVIDER=false      # MLflow setting: use the OpenTelemetry provider
export VARIABLY_API_KEY=vbl_your_key_here
export VARIABLY_BASE_URL=https://api.variably.dev
import mlflow
from variably.integrations.otel import instrument

processor = instrument()          # before the first trace; raises at start-up if a variable is missing

mlflow.set_experiment("benefits-assistant")
mlflow.openai.autolog()           # or @mlflow.trace / mlflow.start_span, unchanged

@mlflow.trace(span_type="AGENT", name="benefits_agent")
def answer(question: str) -> dict:
    docs = search_plan_docs(question)          # a RETRIEVER span
    text = write_answer(question, docs)        # a CHAT_MODEL span from autolog
    return {"answer": text}

answer("What is the in-network out-of-pocket maximum for Gold PPO in 2026?")
processor.force_flush()           # in a short script, before it exits

MLflow keeps recording to your tracking server as before. With MLflow's option to emit the OpenTelemetry GenAI conventions switched on, spans arrive in GenAI form and are read the same way. Without MLFLOW_USE_DEFAULT_TRACER_PROVIDER=false, MLflow keeps its spans to itself and nothing arrives — no error is shown. More on the exporter in OpenTelemetry.

How a trace becomes answers

In your MLflow traceBecomes
Each AGENT spanOne answer, named after the span
A trace with no AGENT spanOne answer from the root span
The agent's inputs: question, query, input, prompt, user_input, request or the last user messageThe question (else the root span's, else the first model call's user message)
The agent's outputs: a string, output, answer, response, content, result, text, final_answer, the last assistant message, or an OpenAI or Anthropic responseThe answer; when the agent recorded no outputs, its last model call's answer
RETRIEVER spans under the agent — MLflow Documents, records with page_content, content or text, or stringsThe documents the answer is checked against; doc_uri (or source, uri, url) is the source named in explanations, chunk_id (or id, doc_id) the id; the retriever's query is the retrieval query
Document metadata such as plan_year, state, effective_fromThe document's scope, for "Right version of the source" — text, a year (1900–2100) or a list of text
TOOL spans under the agentTool calls; a list of flat records (or {"rows": [...]}) is sent as query result rows, so figures are checked against them; SQL text in the inputs (sql, query, statement) is checked as the answer's SQL
CHAT_MODEL / LLM spansModel name, provider and token counts, added up per agent
An agent inside another agentIts own answer in the same session, with its parent recorded
An agent output handoff holding an objectA hand-off to the next top-level agent, checked against your hand-off contracts; to, fields and message inside it are used as given, any other object is the fields
An agent with no answer text but a hand-off or tool callsA routing step — grounding is not applicable rather than zero
The top-level agent that ends lastThe final step of the session
# A hand-off from intake to adjudication, recorded in the intake agent's outputs
@mlflow.trace(span_type="AGENT", name="intake")
def intake(claim_text):
    summary = summarise(claim_text)
    return {"answer": summary,
            "handoff": {"to": "adjudication",
                        "fields": {"onset_date": "2026-03-01", "leave_type": "STD"}}}

Edge cases and limits

Document metadata keys must be declared. Each metadata key sent with a document must be a scope dimension your project declares. By default log_traces sends every key that looks like a scope value, so an undeclared key such as region fails that answer:

failed tr-9e04... benefits_agent: … reference_materials[0] (sob-2026-p2-c4).metadata: unknown key "region"; the project's scope dimensions take plan_year, state

Fix it either way: name the declared keys with reference_metadata_keys=["plan_year", "state"] (the others are left out and listed in a note), or declare the new dimension in the project's document scope and publish. Live traces drop undeclared metadata from the answer, record why, and still score it.

Spans outside agents are ignored. When a trace has any AGENT span, a retriever, tool or model span that is not under an agent belongs to no answer, and its documents are not used. Call the retriever from inside the agent function, or nest it with mlflow.start_span. A trace with no agent span is one answer from its root, with every span under it.

Sessions differ between the two paths. An import uses session_key=, else MLflow's session (mlflow.trace.session), else the trace id. Live traces use the session MLflow puts on the exported root span — which MLflow does only in some set-ups, such as Unity Catalog on Databricks — and otherwise the trace's request id, so each live trace can be its own session. To group conversations reliably, import with session_key= pointing at your own conversation tag.

Also:

  • Nothing is guessed: a trace or agent whose shape is not recognised is skipped, with the reason in the report (or on the answer, for live traces).
  • Tool output text is kept up to 4,000 characters per call; rows are kept in full.
  • Live traces carry the environment, tags and dataset run you set with variably.context() (OpenTelemetry). An import files only each trace's final answer under its dataset item; the other steps are logged without the run.
  • An authentication error (a wrong or revoked key) stops the import at once; any other error is counted per answer under failed, and the import continues.

Troubleshooting

You seeWhyFix
skipped … no answer text in the span's outputsThe agent's outputs use keys that are not readReturn the answer under answer, output or response, or as a string
skipped … trace has no spansThe export has the trace's info onlyExport traces with their span data
failed … unknown key "region"A document carries metadata the project doesn't declarereference_metadata_keys= with the declared keys, or declare the dimension
Answers arrive but grounding is not applicableThe retriever span is outside the agent, or its outputs are not a list of documentsNest the retriever under the agent; return documents with text
Live traces never arriveMLflow still uses its own tracer provider, instrument() ran after the first trace, or a script exited before flushingMLFLOW_USE_DEFAULT_TRACER_PROVIDER=false, instrument() at start-up, force_flush() before exit
Every live trace is its own sessionMLflow did not put a session on the exported root spanImport with session_key=, or use observe() with your own session_id
AuthenticationError: UnauthorizedThe key is wrong, revoked, or its project was deletedCreate a key in the target project and update VARIABLY_API_KEY

FAQ

Does MLflow need to be installed to import? Only to fetch traces with mlflow.search_traces. Importing an export file needs only the variA/Bly SDK.

Can I pass the DataFrame search_traces returns by default? Yes, if it has its trace column; return_type="list" is simpler.

If I import the same trace twice, do I get duplicates? Yes — each import sends new answers. Import each trace once, for example by filtering on time.

Are imported answers scored differently? No. They are scored like any other answer and tagged mlflow.


Concepts on this page: Answers, sessions and agents · Integrations, data and usage