An overall score mixes everything together, including the things you decided did not matter. A success metric is the number you actually ship on: your own named score, built from the dimensions you choose and the weights you give them.

What a success metric is

A name, and a set of weighted dimensions. "Support answer quality" might be faithfulness at 0.5, answer relevance at 0.3, and tone at 0.2. That number goes at the top of the experiment, and it is the one to compare versions on.

You can define more than one. A single app often has two audiences for its scores — the engineer who cares about grounding and the product owner who cares about whether the answer helped — and one number cannot serve both without lying to one of them.

Building one

On the Create experiment form, under success metrics:

1. Name it after the decision it informs, not after the dimensions it contains. "Ship gate" and "Support answer quality" are useful names; "Weighted score 2" is not.
2. Add the dimensions that belong to that decision.
3. Give each a weight. They are relative, so 0.5/0.3/0.2 and 5/3/2 behave identically.

Business outcomes

Automatic metrics measure the answer. Business metrics measure what happened next — a resolved ticket, a completed checkout, a thumbs-up — and those come from your side, because only your app knows them:

variably.trackMetric("ticket_resolved", value=1, experiment_key="support-v3")

This is the pairing worth having. Quality that goes up while resolution goes down is the single most useful thing an evaluation platform can tell you, and it is invisible if you only measure one of them.

Choosing thresholds honestly

A threshold is a promise to yourself about what you will not ship. Set it where your current production traffic actually sits, then raise it — a threshold nothing has ever passed gets ignored within a week, and an alert everybody ignores is worse than no alert.