Temperature does one thing, and it is not creativity

Temperature gets described as a creativity dial, which is the wrong mental model and leads to the wrong settings. The model does not become more imaginative at 1.2 and more rigorous at 0.2. Its knowledge does not change. Its ranking of the next tokentoken The unit a language model actually reads. Usually a few characters, so one word can be several tokens. does not even change.

What changes is how much of its own ranking it is willing to ignore.

What the number actually does

Fig 01

A pipeline, not a chain of attempts. Temperature touches exactly one step.

The model's final layer emits logitslogits The raw scores a model produces for every possible next token, before they are turned into probabilities., a raw score for every token it knows. Before those become a probability distributionprobability distribution A set of options with a likelihood attached to each, where the likelihoods add up to one., every one of them is divided by the temperature. Then softmaxsoftmax The function that turns raw scores into probabilities that add up to one. runs as normal.

That division is the whole mechanism. Because softmax exponentiates, dividing by a small number stretches the gaps between scores, and the leader pulls further ahead. Dividing by a large number squashes the gaps, and everything moves closer to a tie. At a temperature approaching zero the top token takes essentially all the probability, which is greedy decodinggreedy decoding Always taking the highest scoring next token instead of drawing one at random.. At a high temperature the distribution flattens towards a coin toss across the vocabulary.

Note what is not in that pipeline. Temperature does not change the model's weights, its ranking, or how much it knows. It runs after the thinking is over.

The same scores, reshaped

Fig 02

This is the shape worth carrying in your head. The ranking is identical in every row. The only thing moving is how much of the distribution is left for everything that is not the model's first choice.

Which explains both failure modes. Push it down and the model repeats itself, because at every step it re-picks the same safe continuation and can fall into loops. Push it up and you are asking it to select tokens it scored poorly, which is how you get text that reads fluently and is quietly wrong.

Neither is creativity. One is stubbornness and the other is noise.

Zero is not determinism

The most expensive misunderstanding about temperature is that 0 makes a system reproducible. It does not.

Temperature 0 means one thing: given a set of probabilities, take the biggest one. It says nothing about where those probabilities came from. The model still has to compute them, every single time, and that computation is not guaranteed to land on exactly the same numbers.

When two tokens are nearly tied

Most of the time none of this matters. The model is 99% sure the next word is Paris, a difference in the ninth decimal place changes nothing, and you get Paris.

Then you hit a step where two tokens are almost level.

Fig 03

Look at what did not change: the rule. It took the biggest number both times.

Nothing random happened there. Temperature 0 did exactly what it promises, twice. It took the biggest number. The biggest number moved.

It moved because the model adds up millions of numbers to produce each score, and computers are not perfectly exact with decimals. Add the same set of numbers in a different order and you can land a hair away from where you landed last time. In school, a + b + c equals a + (b + c). On a GPU, those two can differ in the eighth decimal place.

The order changes because your request does not get the machine to itself. It is processed alongside whatever other traffic is in flight, and how many requests are in that group changes how the arithmetic is divided up. The group depends on load. Load depends on strangers.

One token, two answers

Ask a model the same question twice, at temperature 0, and you can get this.

Fig 04

Illustrative. The runs agree word for word, then one token lands differently.

Both runs open identically. Then at one position the model was nearly tied between sunlight and light, the arithmetic landed a hair differently, and the two runs picked different words.

Neither answer is wrong. That is part of what makes this hard to notice.

Why it does not stop there

Here is the part that surprises people. The model writes one token at a time and feeds its own output back to itself as input. So the moment one word differs, the next prediction is being made from a different sentence.

Fig 05

Nothing random happened. A rounding difference got amplified.

After sunlight, the natural continuation is is scattered by. After light, it is from the sun. Neither run is drifting. Each is making a perfectly reasonable prediction from the text it can see, and the text it can see is no longer the same.

Usually the two answers converge back to roughly the same meaning, which is why this can sit unnoticed in a system for a long time. Sometimes they do not, and that is the day you find out that temperature 0 was never a guarantee.

Thinking Machines Lab put numbers on exactly this in September 2025. They ran the same prompt 1,000 times at temperature 0 and got 80 unique completions. Then they rewrote three of the routines that do the adding up so their results no longer depend on how many requests are grouped together. After that, all 1,000 runs came back identical.

Which is the proof of the point. The randomness was never in the sampler. It was in the arithmetic underneath.

Unless you control the serving stack, you do not get to apply that fix. So treat temperature 0 as a much narrower range of outputs rather than a guarantee, and keep schema validation and a retry path on anything a parser touches.

The part you do not control

The fix from that study is real. It is also in a layer you probably do not own.

When you call a hosted model, a lot happens between your request and the answer. All of it together is the serving stackserving stack Everything between your API call and the model: load balancing, batching, scheduling, GPU allocation, kernels, and the runtime., and almost none of it is yours.

Fig 06

You set two things: what you ask, and how the winner is picked. Everything in between belongs to whoever runs the servers. Which GPU you land on, who you are batched with, how big that batch is, which kernel version is deployed this week. You will not be told when any of it changes.

So temperature: 0 is a narrow instruction. It says "once you have the scores, do not roll a dice." It does not say "compute the scores the same way you did last time," because that is not yours to ask.

What determinism actually needs

This is where I would correct the way this usually gets stated, including earlier in this post. Temperature 0 is not the opposite of determinism. It is one half of it.

Fig 07

Both halves, or neither. One on its own guarantees nothing.

The second column is the one people skip. Identical scores means pinning the model version, the runtime, the precision, the hardware path, and the batching behaviour. On someone else's API you can pin roughly one of those.

The tie-break is smaller but real. If two tokens come back at exactly the same probability, "take the biggest" has not decided anything, and the implementation falls back to whatever its own ordering happens to be. A deterministic system needs a stated rule, usually the lowest token id.

Run the model yourself and both halves are available to you. That is what the 1,000 identical runs required: not a temperature setting, but ownership of the arithmetic. Rent the model and you get the first half only.

So the honest version is this. Temperature 0 gives you deterministic sampling. Deterministic sampling plus a deterministic serving stack gives you a deterministic generation. On a hosted API you have the first and not the second, which is a much narrower spread of outputs rather than a guarantee.

Choosing a temperature

Fig 08

The question I ask at each call site is who reads this. Not what the feature is, who reads it.

If a parser reads it, go as low as the provider allows. Extraction, structured output, classification, routing, tool selection. There is one right answer and you want the model's top choice with as little interference as possible. Pair it with a schema and a retry, because low temperature reduces variance rather than eliminating it.

If a person reads it, go up. Explanations, summaries, anything conversational. Somewhere near the provider's default is a reasonable starting point, and the only way to tune it is to read the output.

Two things that are easy to miss. top-ptop-p Keep only the most likely tokens whose probabilities add up to p, then draw from those. Also called nucleus sampling. and top-ktop-k Keep only the k highest scoring tokens, then draw from those. truncate the distribution after temperature has reshaped it, so turning both knobs at once makes the effect of either hard to reason about. Change one. And when a generation has to be reproducible for a test or an audit, temperature is not the lever, for the reasons above. Cache the output, or run the model yourself.

What to do with this

• Set temperature per call site, low where a parser consumes the output and higher where a person does, and write the reason next to the setting.
• Keep schema validation and a retry path even at temperature 0, because on a hosted API the scores shift underneath you with someone else's traffic.
• When output is wrong rather than repetitive, leave temperature alone. It is a symptom of the prompt, the retrieval, or the model, and turning the dial only changes how confidently you are told the wrong thing.