What Temperature Actually Does to the Output
It divides the model's scores by a single number just before they turn into probabilities, and that is the whole mechanism. The model has already finished computing those scores by the time temperature touches them, which is why it can never change the token that came out on top, only how often something else gets drawn instead. The newest models are removing the parameter altogether.
This one assumes some background. You’ll want a working understanding of probability distributions, a general picture of how a transformer is put together, and in particular what happens at the end of one, where the final layer turns the model’s internal state into a prediction for the next token. Everything below sits on top of that.
Temperature gets talked about as if it were a general control on how a model behaves, and it isn’t. It’s one step, right at the end. By the time it does anything the model has already run and already produced a score for every token in its vocabulary, and all that’s left is the arithmetic that turns those scores into probabilities and picks one of them. There’s nothing here about attention, or how the model came to know anything, or top-k and top-p, or what number you should use for your own task. They’re all worth understanding and none of them are this. Most of the confusion about temperature comes from not knowing where it sits, so here it is.
Where temperature sits
The full path from your prompt to the next token. This article covers the last four boxes only.
Everything greyed out has finished running before temperature exists. The prompt has been tokenised, the vectors have been through every layer, attention has done whatever attention does, and the output head has turned the result into one score per token. That’s the model’s contribution and it’s over. The four highlighted boxes are what’s left, and they run again for every single token the model writes.
It only decides how much probability the top token keeps
Every time a model extends a piece of text it doesn’t just choose the next token. It works out a probability for every token that could come next, something like 67% on the highest-probability one, 14% on the one behind it and steadily smaller numbers after that, and then it draws from that spread. Temperature changes the shape of the spread before the draw happens. Turn it down and probability piles onto the top token, so 67% becomes 94% and everything else is squeezed towards nothing, which is why low temperature gives you much the same answer run after run. Turn it up and the probability spreads out, the top token drops to 41%, and tokens that were long shots become real possibilities, which buys you variety and stranger phrasing along with a rising chance of something that doesn’t hold together. Push it high enough and the numbers are nearly level and the output stops making sense. What temperature can’t do is change which token has the highest probability, or make the model more accurate or better informed. It only decides how much of the probability that token gets to keep.
The whole thing is one division and a softmax
What the model actually hands over at the end is a list of raw scores called logits, one for every token in its vocabulary. A logit isn’t a probability. It sits on no fixed scale, it can be negative, and on its own it means nothing. Only the gaps between logits carry information.
Temperature divides each logit by T. Softmax takes that divided list and turns it into probabilities: everything comes out positive, the ordering is preserved, and the whole set sums to 1.
p = softmax(logit / T)Take the sentence The cat sat on the ___ and say the model produced these seven logits. Real models produce tens of thousands of them, and current ones well over a hundred thousand, but seven is enough to watch what happens. At T = 1 the division changes nothing and you get the model’s own probabilities:
| token | logit | logit / T | softmax(logit / T) |
|---|---|---|---|
| mat | 6.0 | 6.00 | 67.3% |
| floor | 4.4 | 4.40 | 13.6% |
| couch | 4.0 | 4.00 | 9.1% |
| sofa | 3.6 | 3.60 | 6.1% |
| bed | 2.8 | 2.80 | 2.7% |
| roof | 1.9 | 1.90 | 1.1% |
| zebra | -2.0 | -2.00 | 0.02% |
| 100% |
At T = 2 every logit is halved before softmax sees it, and the picture changes:
| token | logit | logit / T | softmax(logit / T) |
|---|---|---|---|
| mat | 6.0 | 3.00 | 40.5% |
| floor | 4.4 | 2.20 | 18.2% |
| couch | 4.0 | 2.00 | 14.9% |
| sofa | 3.6 | 1.80 | 12.2% |
| bed | 2.8 | 1.40 | 8.2% |
| roof | 1.9 | 0.95 | 5.2% |
| zebra | -2.0 | -1.00 | 0.7% |
| 100% |
The logit column is identical in both tables. That’s the column the model produced, and it’s the same at every temperature. Everything that moved, moved in the two columns after it.
Picking the most likely token every time makes the writing worse
Having probabilities isn’t the same as having a token. Something still has to choose one, and the model doesn’t take the highest. It draws at random, weighted by those numbers. On the T = 1 table above, mat comes up around 67 times in 100, floor about 14, and zebra roughly twice in every ten thousand runs. Ask again and you get a different draw.
Always taking the top token is a real option, called greedy decoding, and it’s what most runtimes give you at temperature 0. They arrive there by different routes, because the formula itself breaks at zero: vLLM computes the argmax separately, llama.cpp masks every other logit to negative infinity, and Hugging Face refuses the setting outright and tells you to turn sampling off instead. Whether greedy is a good idea depends on the task, and less obviously than it used to. For long open-ended text it isn’t, because always taking the most likely token gives writing that repeats itself and eventually loops, which is the problem nucleus sampling was invented to solve. Human writing isn’t the most likely sequence of words, and a model that walks to the top of the list every time walks straight past it. On short verifiable work the picture reverses: greedy measurably beats sampling on maths and code benchmarks, which is why it’s the default in Hugging Face transformers and the norm in evaluation harnesses, even though every hosted API ships with sampling on.
So the last step is a weighted draw, and it runs once per token rather than once per answer. A hundred-token reply is a hundred separate draws, every one of them from a distribution temperature has already reshaped. That’s why one setting does so much more damage to a long answer than a short one.
Whatever the top token loses, the tail gains
Drag the temperature slider and watch the shape of the distribution change. Below T = 1 the logits are divided by a number smaller than one, which widens the gaps between them, and the distribution becomes more peaked: probability mass concentrates onto the top-ranked token and the tail collapses towards zero. Above T = 1 the gaps narrow, the distribution flattens, mass migrates out of the top token and into the tail, and the entropy of the distribution rises. At exactly T = 1 the distribution is the model’s own, with nothing done to it. Two things hold at every setting, and both are worth watching for. The rank order of the tokens never changes, because dividing every logit by the same positive number cannot reorder them. And the probabilities always sum to 1, because softmax renormalises after each division, so any mass the top token loses has to go somewhere.
The cat sat on the ___
67% on the top token, 33% spread across everything else. Roughly the model's own view of the sentence. This is what temperature 1 means: the scores pass through untouched. The order never changes. Every bar keeps its rank at every temperature, because dividing all the scores by the same number cannot reorder them.
And then it rolls a weighted die
Models don't take the top token every time. They draw one at random, weighted by these probabilities. Greedy decoding (always taking the top token) is what you get at temperature 0, and it makes text repetitive, not better.
| token | logit | logit / T | probability |
|---|---|---|---|
| mat | 6.0 | 6.00 | 67.31% |
| floor | 4.4 | 4.40 | 13.59% |
| couch | 4.0 | 4.00 | 9.11% |
| sofa | 3.6 | 3.60 | 6.11% |
| bed | 2.8 | 2.80 | 2.74% |
| roof | 1.9 | 1.90 | 1.12% |
| zebra | -2.0 | -2.00 | 0.02% |
On the newest models the knob is being taken away
Worth knowing, because it’s already happening. Claude Opus 4.7 and everything after it, including Sonnet 5 and Opus 5, return a 400 error if temperature, top_p or top_k is set to anything other than the default. OpenAI’s reasoning models don’t accept the parameter either. The reason is that these models are trained with reinforcement learning against a particular sampling distribution, which makes that distribution part of what was optimised rather than a setting layered on top of it. Moving off it doesn’t give you a more careful model, it gives you a model running outside the conditions it was tuned under. The same logic shows up in the open weights, where Qwen3’s model card says in capitals not to use greedy decoding because it causes degradation and endless repetition, and Google recommend leaving Gemini 3 at 1.0. Anthropic’s migration guidance is to drop the parameter and shape behaviour with the prompt instead.