Imagine two runs of the same AI feature. One starts quickly; another pauses long enough that the interface begins to feel uncertain. Repeat the same task and the answer may shift too, perhaps in wording, perhaps in substance.
Those are different kinds of variation. They also share one product problem: an average can make both of them look smaller than they are.
A user does not receive the mean latency or the mean evaluation score. They receive one run.
That is where variance stops being an implementation detail and becomes part of the product behavior a team has to understand.
The average can hide the run
Averages are useful because they compress a lot of activity into something readable. They are also very good at hiding tails.
Dean and Barroso's The Tail at Scale made that systems problem explicit for large online services. Temporary high-latency episodes that are minor in smaller systems can dominate overall performance at scale, which is why the paper focuses on creating a predictably responsive whole from less predictable parts.
That is systems evidence, not AI-specific user-experience evidence. I would not use it to claim that a particular delay causes a particular reaction in an AI product.
The product implication is narrower and still useful: improving the mean does not establish that the experience became predictable. A latency distribution can improve in the center while leaving a wide tail. The dashboard looks healthier. Some runs can still feel materially different.
A healthy average can coexist with an uneven individual experience.
The same caution applies to quality metrics. An aggregate score can be useful for comparing versions while saying very little about how much one repeated task moves from run to run.
The compression is useful. It just cannot be the whole description.
Latency is part of the interface
The strongest reason not to turn this into a simple "faster is better" rule comes from human-computer interaction research.
Tan and colleagues' CHI 2026 study varied time to first token at 2, 9, and 20 seconds across Creation and Advice tasks with 240 participants. Participants in the shortest-latency condition rated responses as less thoughtful, and some usefulness comparisons also favored longer waits. Task type mattered as well.
The useful qualification is just as important: the study found no significant main or interaction effect of latency on trustworthiness ratings.
So I would not translate this into "slow responses create trust" or "users prefer waiting." The study does not support either claim.
It supports something more awkward, and more useful for product design: timing can change how an output is perceived, and the effect depends on what the user is doing. A delay is not only a backend number once it becomes visible inside an interaction.
That raises a design question the mean cannot answer. What does the interface communicate during the wait, and is that communication true?
A real progress state can help a user understand what is happening. A fabricated explanation of invisible "thinking" would be a different product choice entirely, and a riskier one.
The interface should not invent system state just because the pause needs a story.
Output variance is a different problem
Latency variation and output variation belong in the same article, but they are not the same phenomenon.
Generative systems can return different outputs under repeated runs. That variation may be desirable, irrelevant, or dangerous depending on the task.
Brainstorming can benefit from diversity. A factual extraction workflow may need much tighter consistency. Decision support becomes a different product problem if small changes in generation lead to materially different recommendations.
The important distinction is that consistency is not correctness.
Shyr and colleagues' biomedical preprint on LLM repeatability and reproducibility makes that separation concrete. The work measures repeatability under identical conditions and reproducibility under predefined changes in conditions, using semantic and internal measures of variability. In the medical-reasoning tasks they studied, repeatability and reproducibility did not correlate with diagnostic accuracy.
That result should stay inside its boundary. It is a biomedical preprint using specific diagnostic tasks, not a universal benchmark for every LLM product.
But the distinction travels well: a model can be consistent and wrong, or variable while still producing useful answers. Repeatability measures one property. Correctness measures another. User-perceived quality is another again.
A related Dream Atlas analysis, The Evaluator Became a Production Dependency, looks at the adjacent problem from the measurement side: noisy evaluation results need enough repeatability context for operators to know what one result actually means.
For product work, that means "make it deterministic" is usually too crude a requirement. The better question is which dimensions of variation are acceptable for this task.
The product contract lives in the envelope
This is the part I find more useful than arguing about whether variance is good or bad.
A product already has an implicit envelope of behavior. Some runs are fast, some slower. Some outputs stay close together, some spread out. Some tasks tolerate that movement. Others do not.
The product contract becomes fragile when the interface or the team behaves as though only the average case exists.
A distribution-level review asks at least four things:
- What varies? Measure latency and output behavior as distributions rather than only as central averages.
- Which variation matters for the task? A creative task and a factual extraction task do not need the same envelope.
- What can the interface truthfully communicate? Status, progress, confidence, and waiting states should map to real system state.
- What remains unestablished? A perception effect in one study is not evidence of a universal trust effect, retention effect, or abandonment threshold.
This is also where I would separate measurement from interpretation.
Latency tails can be measured. Output consistency can be measured. Task-specific perceptions can be studied. Whether those properties change long-term trust, retention, or satisfaction in a particular product is a separate empirical question.
The temptation is to compress all of that into one word such as reliability. I think that makes the product harder to reason about because the underlying properties can move independently.
Review the distribution, not just the score
A practical review does not need one grand variance metric.
For latency, inspect the shape of the experience:
- central latency;
- slower percentiles and tail behavior;
- task classes or conditions associated with the tail;
- time to first token separately from total completion time when that distinction matters to the interface.
For output behavior, repeat important tasks under controlled conditions and ask:
- Does the semantic answer stay stable?
- Which facts, recommendations, or constraints change?
- Is the variability useful diversity or product instability for this task?
- Is repeatability being confused with correctness?
Then inspect the interface around those distributions.
If the product exposes a status cue, does that cue correspond to a real state? If it implies certainty, is that certainty measured? If a wait is framed as progress, is progress actually observable?
Finally, test the actual task classes that matter. The CHI study is a useful warning against treating one latency rule as universal because Creation and Advice did not behave identically. The biomedical repeatability work is a useful warning against treating one consistency result as universal because its findings come from a particular medical setting.
Different tasks can have different tolerances. That is not a reason to avoid measurement. It is the reason to measure the right thing.
Predictability is not sameness
The target does not have to be zero variance.
Some variation is unavoidable. Some is useful. Some may even change how an interaction is perceived in ways that are not captured by a simple "faster is better" objective.
The product still needs a boundary it can defend.
For me, predictability means knowing enough about the envelope to avoid promising the average case as though it were every case. It means separating diversity from instability, repeatability from correctness, and measured perception effects from claims the evidence has not established.
That is less tidy than one latency target or one quality score. It is also more honest.
The average still belongs on the dashboard. It just should not be mistaken for the product.
// End of transmission. Keep the envelope visible — ZYANE