The same prompt produces different output each time it is run. Ignoring that variation is the most common source of false conclusions about what works.
Why identical prompts give different results
Generation selects each token from a probability distribution rather than always taking the most likely option, which introduces deliberate randomness.
A temperature setting controls how much, and even at low settings the output is not guaranteed identical across runs.
Providers also update models and routing without announcement, so a prompt that behaved one way last month may behave differently now. A prompt that stopped working without being edited has usually met one of these changes.
What the variation tells you
Running the same prompt five times shows the range the prompt actually produces, which is more useful than any single sample.
A prompt whose five outputs are broadly similar is stable, and a prompt whose outputs diverge widely is underconstrained regardless of whether you liked one of them.
Divergence points at the specific decisions the prompt left open, which is a direct list of what to specify. Reading five outputs side by side makes those open decisions obvious in a way a single output never does.
How to test a change honestly
Comparing one output from the old prompt against one from the new prompt is a coin flip dressed as an experiment.
Three to five runs of each, judged against a criterion written before looking, is the minimum that separates a real improvement from noise.
Keeping the old prompt and its samples makes the comparison possible later, and it is the step almost everyone skips. Without it, a team cannot tell whether six months of prompt edits improved anything at all.
Why one bad output proves little
A single failure may be the tail of an otherwise acceptable distribution rather than evidence of a broken prompt.
Reacting to it with an added instruction produces prompts stuffed with rules addressing failures that occurred once.
The threshold worth acting on is a failure that recurs across runs, which is the only kind a prompt change can reliably remove.
Where rewording is genuinely needed
Consistent failure across every run means the prompt is the problem, and rerunning will not help.
The same applies when the output is wrong in the same specific way each time, which indicates an instruction the model is interpreting differently from you.
Distinguishing these two cases takes a few extra runs and determines whether the next hour is spent productively or spent editing wording that was never at fault.