Two averages and an eyeball is not a decision. A version can look better on the mean and be worse on the traffic you care about, and both can be noise.

Is the difference real

The experiment page reports statistical significance for the difference between variants, not just the difference. The number to look at is whether the interval crosses zero: if it does, you have no evidence yet, regardless of how far apart the averages look.

Two things make a real difference look unreal: not enough traffic, and a metric with high variance. Both are fixed by waiting rather than by re-reading.

Is the difference worth having

Significance says an effect is unlikely to be chance. It says nothing about whether the effect matters. A statistically certain improvement of 0.4 points on a 100-point scale is real and irrelevant, and shipping it costs a deploy and a rollback risk.

Ask both questions in order: is it real, then is it big enough to be worth the change.

Where the difference lives

An average hides its own shape. Two things are worth opening before believing a headline number:

• Segments — a variant that wins overall can lose badly on one segment. Segment performance is sorted worst-first for exactly this reason.
• Side by side — the same input, both variants' answers next to each other. It is the fastest way to find out whether a score movement means what you assumed it means.

Has it drifted since

A verdict has a shelf life. The drift view compares a recent window against the baseline for each variant and category, so a variant that won last month and has quietly degraded shows up as a change rather than as a still-good average.

Scores move for reasons that are not your change: traffic shifts, your retriever's corpus grows, a provider ships a model update. That is what makes drift worth watching separately from the original result — and why attributing a change to your change needs both versions running at once, which is BYOR.