Instrumenting a LangGraph app by hand means finding every node and adding a call to each. instrument() wraps them all in one line, and keeps working when you add a node.

1. Install and instrument

pip install "variably-sdk[langgraph]"

from langgraph.graph import StateGraph
from variably.integrations.langgraph import instrument

graph = StateGraph(AgentState)
graph.add_node("planner", planner)
graph.add_node("retriever", retriever)
graph.add_node("writer", writer)

instrument(graph, session_id_key="session_id")   # before .compile()

app = graph.compile()

Instrument the graph while it is still a StateGraph. Every node is wrapped in place, each run becomes one trace, and terminal nodes are detected as the aggregator automatically. Async nodes are wrapped too.

Your state needs to carry a session id under the key you name. Anything stable per run works — the thread id you already have is the usual choice.

2. Tell it where your retrieved context lives

Node state is yours, so the SDK cannot guess which key holds retrieved documents. Without this, the non-grounding dimensions are scored and grounding is reported as N/A.

def context_extractor(state, update):
    docs = (update or {}).get("documents")
    if not docs:
        return None
    return [{"id": d.id, "content": d.page_content} for d in docs]

instrument(graph, session_id_key="session_id", context_extractor=context_extractor)

Return None for nodes that retrieve nothing — the extractor is called for every node.

3. Tell it which tools ran

The same applies to tool calls. Supply them and you get trajectory precision and recall, tool correctness, and order adherence.

def tools_extractor(state, update):
    calls = (update or {}).get("tool_calls")
    if not calls:
        return None
    return {
        "tools_called": [
            {"name": c.name, "arguments": c.args, "output": c.output} for c in calls
        ]
    }

instrument(
    graph,
    session_id_key="session_id",
    context_extractor=context_extractor,
    tools_extractor=tools_extractor,
)

When to override the aggregator

Terminal nodes are detected automatically, which is right for most graphs. A graph that ends by looping back, or one with several terminal nodes where only one produces the user-facing answer, should say so explicitly:

instrument(graph, session_id_key="session_id", aggregator_nodes={"writer"})

Other frameworks

instrument() is LangGraph-specific because it relies on the node structure. For anything else — CrewAI, a supervisor loop, or a hand-rolled pipeline — call observe() per step with a shared session_id, which is what instrument() does for you. See Score a multi-agent trace.

Debugging an instrumentation

VARIABLY_LOG_LEVEL=DEBUG python your_app.py