Choose which dimensions score
Score the things your app is actually for, and weight the ones that decide whether you ship.
A scoring profile that matches your application: the categories that matter switched on, each dimension weighted, and a threshold on the ones you would hold a release for.
- A project — create one from the sidebar
- Two minutes on the Create experiment form
How it works
- Switch off categories that don't apply
- Weight what decides a release
- Save the experiment
- Scores only the enabled dimensions
- Rolls them up using your weights
- Applies your thresholds
- No noise from what you do not care about
- One number that means what you decided
- Comparable across versions
Six categories ship enabled. That is the right default before we know anything about your app, and the wrong one once you do: a classifier that emits JSON has no prose to be coherent about, and a chatbot with no retrieval step has nothing to be grounded against.
Scoring is configured per experiment, on the Create experiment form. It is worth a minute of thought at that point, because it decides what every evaluation in that experiment measures.
Turn off what doesn't apply
Each of the six categories has a master switch. Switching one off stops its dimensions contributing to the overall score, and the category is shown as disabled rather than as zero.
A category you leave on with nothing to measure reports N/A rather than zero, so it will not quietly drag your average down either way. Turning it off is about signal, not arithmetic — a column of N/A is noise on every evaluation you read afterwards.
Weight what decides a release
Inside each category, every dimension has:
- Enabled — whether it is scored at all
- Weight — how much it counts toward the category score
- Threshold — the line you would not ship below
Weight is the interesting one. Faithfulness and tone are both real signals, but on a medical summariser they are not equally important, and an unweighted average says they are.
A worked example
A support assistant over a help centre, with no generated code and no long-form prose:
| Category | Setting | Why |
|---|---|---|
| Grounding | On, heavily weighted | Answers must come from the help centre, not the model |
| Quality | On | Relevance and completeness decide whether the user is helped |
| Safety | On | Customer-facing |
| Coherence | On | Multi-turn — see Score a conversation |
| Semantic | On, low weight | Useful signal, rarely the reason to hold a release |
| Advanced analysis | Off | Nothing in this app needs it |
What this changes downstream
Everything read afterwards is read through this profile: the overall score, the category breakdown, alert thresholds, and any comparison between two versions. Two experiments with different scoring profiles are not comparable, which is why the setting lives on the experiment rather than on the project.