Anything you put in metadata is logged and available for segmentation. A specific set of names does something extra: when a dimension is failing, the Untested Levers panel can link it to a setting you have been logging but never varied.

Custom names still work for slicing. They just do not earn the diagnostic link.

The link is a starting point, not a diagnosis. A matched key sitting constant does not mean varying it will fix the dimension — root causes are often elsewhere entirely, in chunking strategy, embeddings or reranking. Read the callout as worth investigating, not fix this first.

Retrieval relevance

How many and which chunks your retrieval layer pulled per query — usually the lever most likely to move retrieval scores.

Key	Type	What it represents
retrieved_chunks_count	int	Number of chunks returned by your retriever (top-k)
top_k	int	The same concept, common alias
rerank_count	int	How many chunks survived post-rerank
embedding_model	string	Which embedding model produced the retrieval

Context utilisation

Whether context was passed and how much room it took — this affects how well the model uses what you gave it.

Key	Type	What it represents
has_context	bool	Whether retrieval returned anything to ground on
context_size	int	Approximate tokens of context passed to the model
context_window	int	Model context window used, e.g. 8000, 200000

Faithfulness

Generation-time knobs that shape how literally the model follows its source.

Key	Type	What it represents
model	string	Generator model, e.g. claude-haiku-4-5, gpt-4o
temperature	float	Sampling temperature — 0 deterministic, 1 creative
top_p	float	Nucleus sampling probability mass

Attribution accuracy

How sources are cited back to the user, which drives attribution recall and precision directly.

Key	Type	What it represents
cited_sources_count	int	How many distinct sources the response cites
citation_count	int	The same concept, common alias

Context retention

Memory and multi-turn signals that affect coherence across a session.

Key	Type	What it represents
history_turns	int	How many prior turns were passed into the prompt
memory_size	int	Bytes or tokens of long-term memory included

Workflow consistency

Agent, tool and step signals — for multi-step pipelines and agentic flows.

Key	Type	What it represents
workflow_step	string	Which step in the pipeline this call belongs to
agent_name	string	Which agent emitted this call
tool_count	int	How many tools were available at decision time

Logging hygiene

Metadata is for per-call signals — things that differ between this call and the next: chunk count, temperature, history depth. Application-level constants like app_name or tenant_id are noise here; log those at experiment creation rather than on every call.

Anything you log but never vary shows up in the Untested Levers panel as a hypothesis backlog, which is the point: it is a list of the things you could have tested and have not.

Putting them in

observe(
    prompt=user_query,
    response=answer,
    reference_materials=chunks,
    metadata={
        "retrieved_chunks_count": len(chunks),
        "embedding_model": "text-embedding-3-large",
        "temperature": 0.2,
        "cited_sources_count": answer.count("["),
        "customer_tier": "enterprise",   # segmentation only, and that is fine
    },
)