Connect your appReferenceWhat to send with each answercode
What to send with each answer
Each optional field of log() and observe(), and the check it turns on.
Every answer you send is scored on what it contains. Each optional field of log() (and observe())
turns on more checks, because it gives variA/Bly something exact to compare against. Send what your app
already has; you never need all of them.
The fields and what they turn on
| You send | It turns on | Read more |
|---|---|---|
reference_materials=[{"id", "content", "source"}] | Grounding: each claim checked against the documents | Claims and figures |
environment="prod" (log() only) | The Environment filter, and scoring rules on Environment | Scoring rules |
session_id, agent_name, is_aggregator=True | Sessions, the agent path, per-agent scoring rules | Sessions |
handoff={"to", "fields", "message"} | Hand-offs carry the facts | Hand-off contracts |
sql="SELECT ..." | SQL follows the rules | SQL checks |
The rows a query returned (rows on a reference or a tool call) | Each figure in the answer checked against the rows | Claims and figures |
question_scope={"as_of": ..., "plan_year": ...}, and metadata on each document | Right version of the source | Document scope |
expected_answer="..." | Matches the expected answer | Expected answers |
expected_answer_kind="amount" | Answers what was asked | Expected answers |
required_checks=[...] or requirements_ref="benefit-answer@v1" | Covers what it must | Requirement templates |
metadata={"weeks_paid": 4} used by a business rule | Follows business rules with your own system's values | Business rules |
tools_called, expected_tools, allowed_tools, max_steps | Tool correctness, Trajectory, Step efficiency, Within policy | Score a multi-agent trace |
tags, metadata | Filters; scoring-rule conditions on tags and metadata keys | Scoring rules |
dataset_item, run (log() only) | Dataset runs and the release gate | Runs and the release gate |
provider, model, latency_ms, prompt_tokens, completion_tokens, cost | The Latency and LLM cost columns | Live events |
One answer with everything
An insurance assistant answers a short-term-disability question. It retrieved two documents, ran one SQL query, and you know what the right answer is because this question is in a test set:
client.log(
"How many weeks of short-term disability can I still claim in 2026?",
"You can claim 9 more weeks of short-term disability in 2026.",
reference_materials=[{
"id": "std-policy-2026", "source": "STD policy, plan year 2026",
"content": "Short-term disability pays up to 13 weeks per plan year.",
"metadata": {"plan_year": 2026, "jurisdiction": "US-NJ"},
}],
question_scope={"plan_year": 2026, "jurisdiction": "US-NJ"},
sql="SELECT weeks_paid FROM claims WHERE member_id = 'M-20931' AND plan_year = 2026",
expected_answer="9 weeks",
expected_answer_kind="duration",
requirements_ref="benefit-answer",
metadata={"weeks_paid": 4},
environment="prod",
session_id="chat-4821",
agent_name="benefits-agent",
tags=["std"],
model="your-model-name", latency_ms=1840, prompt_tokens=912, completion_tokens=38,
)
Rules worth knowing
- A document's text goes under
content. Nottext, notbody. A document with nocontentgives grounding nothing to check against. question_scopekeys are checked. A key the project doesn't declare is refused, so a typo can't silently skip the document-version check.- Document
metadatakeys are checked too. Every key must be a dimension of the project's document scope. is_aggregator=Trueneedsagent_nameandsession_id. Without them the SDK raises before sending, so you find out in development.dataset_itemandrungo together. One without the other is refused before sending. Set once withvariably.context(), they go only on the final answer (noagent_name, or the step markedis_aggregator=True).
Common mistakes
| Mistake | What happens | Fix |
|---|---|---|
Sending the whole knowledge base as reference_materials | Every claim looks supported by something; real mistakes hide | Send only what the answer was given |
Sending the question as expected_answer | Matches the expected answer fails on every value | Send the right answer, with each value written once |
A new session_id per message | Every turn is its own session; conversation checks never see the whole conversation | Use the conversation's id on every turn |
Putting secrets or personal data in metadata | They are stored with the answer and shown to project members | Send only what a check or a filter needs |