Reading your scores

The number is not the point. What the number is made of is the point — and the dimensions in an evaluation are not all made the same way. Some are checked against your retrieved context by a model; some are arithmetic over the text. Knowing which is which is the difference between acting on a score and guessing at it.

How a score is built

Three levels, and you can open any of them:

• Overall — a weighted roll-up of the categories
• Category — a weighted roll-up of its dimensions
• Dimension — where the diagnosis actually lives

An overall score is only useful for spotting that something moved. It never tells you what to fix, so treat it as an index, not a verdict.

The six categories

Category	Dimensions	Asks
Grounding	5	Is every claim supported by the context you retrieved?
Quality	8	Is the answer relevant, complete, accurate and well-formed?
Safety	8	Does it contain toxic, harmful, biased or unsafe content?
Semantic	8	Does it mean what it should mean, given the question?
Coherence	3	Does it hold together across turns of a session?
Depth	17	Structural signals — reasoning markers, formatting, vocabulary range

Agent runs are scored differently

A coding agent or a tool-using workflow with no retrieved context is not judged by how its final message reads. Its overall is built from what it did:

Category	Weight	From
Execution	30%	Expected tools called, with the expected arguments when you give them, in the expected order — and only tools on your allowed list
Answer	20%	Not measured yet: whether the answer meets the task's expectations. Text-quality scores cannot judge it — a correct answer to "What is this repository about?" reads as off-topic to them — so the overall uses the other three until expectations can be sent and checked
Safety	20%	The same risk limits as every answer
Efficiency	30%	Expected calls ÷ all calls — extra steps lower it

A run is scored this way when you call observe() with kind="tool_use", or when the final step of a session (is_aggregator=True) sends tools_called and no reference_materials. Execution and Efficiency need expected_tools — list the minimal tools the task needs; an expected tool without arguments matches any arguments. Without it, the overall uses Safety only. Depth and the formatting-based semantic dimensions are still shown, but they do not count towards an agent run's overall. Open the evaluation to see the breakdown under Scored as an agent run.

What each category is actually measured with

This is the part worth reading before you set a threshold on anything.

Grounding — checked against your context

Your answer is split into individual claims, and each claim is checked against the chunks you passed in reference_materials. A claim is supported only if the context entails it.

• Faithfulness — proportion of claims your context supports
• Hallucination rate — the complement: claims it does not
• Attribution accuracy — whether the sources cited are the ones that support the claim
• Context utilisation — how much of what you retrieved was actually used
• Retrieval relevance — how well your retrieved chunks match the query you searched with

This is the category with the strongest method behind it, and the one that is worthless without reference_materials. Numbers, dates and identifiers are checked exactly, not as paraphrases — "within 2–3 business days" against a source saying 5–7 is a number not in sources, a failure, not a rewording. A figure the answer computes (a total, a difference, a date plus a period) passes when it follows from figures that are in the sources.

Every claim carries a verdict. A claim a single model reading suggests a source disagrees with is a possible contradiction: it fails the default gate, but is never treated as proof. A sentence the sources don't confirm is read by a language model for what kind of sentence it is: framing (an introduction or a remark about the report itself) is shown but not counted under both gates, and under the Document review gate so are conclusions (an inference or an absence finding — "no amendment has been provided"). A sentence with a figure or an identifier is always checked. Only a claim proven wrong — a wrong figure, a figure from another document, a wrong calculation or conclusion — counts as misinformation. The full list of verdicts is in Score a document review; how they decide pass or fail is in Checks and gates.

Safety — a multi-label classifier

Toxicity, bias, harmful content, violence and the rest come from a multi-label classifier over the response text. These are probabilities of harm, so low is good — the opposite direction to every other category. On benign text they sit near zero and stay there; a value that is merely small is not a signal.

Semantic — embeddings

Semantic similarity is the cosine between the embedded prompt and the embedded response. Intent alignment, context awareness and factual consistency build on the same representation.

Cosine is bounded at −1, not 0, so a response with nothing to do with its prompt can score slightly below zero. Read anything at or near zero as "unrelated", not as a scale position.

Quality — part model, part arithmetic

Relevance, completeness, accuracy and coherence use the semantic engine. The four writing dimensions are deterministic text metrics with no model involved:

Dimension	How it is computed
Clarity	Normalised Flesch Reading Ease — sentence length and syllables per word
Conciseness	1 − filler and redundancy density
Structure	Formatting cues and sentence-length balance
Grammar	A lightweight proxy: sentence casing, terminal punctuation, doubled words

Two consequences worth knowing. Clarity never reads your question or your context — a correct answer in long sentences scores lower than a wrong answer in short ones. And grammar is a proxy, not a grammar checker; it catches shape, not agreement or tense.

Depth — structural signals, not judgements

The 17 dimensions in this category are computed from surface features of the response: vocabulary range, reasoning markers, code blocks, list items, paragraph count, sentiment. Creativity, for instance, is vocabulary richness plus a bonus for code blocks and lists, over a fixed base.

They are genuinely useful for comparing two versions of the same prompt on the same traffic, where the structural difference between them is the thing you changed. They are not a measurement of whether an answer is creative, persuasive or maintainable in the sense those words normally carry, and a single absolute value should not be read that way. If you are setting a release gate, set it on grounding, quality or safety.

Reading a low grounding score

Work down, not up. The first answer you find is the fix:

1. Is grounding N/A rather than low? Then no context arrived. Check reference_materials — see Score a RAG answer.
2. Is retrieval relevance low? Your retriever returned the wrong chunks. The generator never had a chance, and no prompt change will fix it.
3. Is retrieval relevance fine but faithfulness low? The retriever was right and the model drifted from what it was given. Now a prompt change is the right move.
4. Both fine but attribution low? The claims are supported; the citations point at the wrong sources.
5. Still unclear? Score the retriever and the generator separately — see Score every step.

N/A is not zero

Zero means measured and bad. N/A means not measured, because the inputs were not there.

A router emitting {"next": "billing"} makes no factual claims, so it cannot be grounded. Scoring that zero would be inventing a measurement. Mark the turn and the dimensions that do not apply are excluded from the aggregate rather than counted as failures:

observe(prompt=q, response=r, kind="router")

See Turn types for all five values.

Coherence needs more than one turn

The three coherence dimensions — workflow consistency, context retention, progressive depth — compare a turn against the turns before it. A session with one turn has nothing to compare, so they carry no information there, and a single-turn application will see nothing useful in this category however good its answers are.

Give conversation turns a shared session_id and the comparison becomes possible — see Score a conversation.

Scores are deterministic

The same input produces the same score on every run. No judge model, no sampling temperature. A score that moved means your system changed, not that the evaluator felt differently today.

That is what makes the comparison in Read the verdict meaningful, and a regression gate possible at all.

Weights and thresholds are yours

The per-category and per-dimension weights above are defaults, not fixed. Every dimension has a weight and a threshold you set when you create an experiment, and turning off a category you have no use for removes it from the roll-up entirely — see Choose which dimensions score.

Set thresholds where your current production traffic actually sits, then raise them. A threshold nothing has ever passed gets ignored within a week.

What a score cannot tell you

Scores describe the answer you produced against the context you supplied. They do not know whether you retrieved the right documents in the first place: a perfectly grounded answer built on the wrong chunk scores well and is still wrong. That is why retrieval diagnostics sit beside the score rather than inside it.

They also cannot tell you whether a change is an improvement. For that you need both versions on the same traffic at the same time — Move to BYOR.