The randomness dial
Everything so far has been about the words you send. But models also have settings: knobs you (or the app you're using) can turn. The star knob is temperature, and it controls one thing: how much randomness goes into picking each next token.
Remember: a model generates text by predicting the next token. At each step it doesn't have one candidate; it has a whole ranked list with probabilities. Temperature decides how that list gets sampled:
- Temperature 0 (or near it): always take the top-ranked token. Ask twice, get (almost exactly) the same answer twice. Predictable, focused, repeatable.
- Medium (around 0.7-1.0): mostly favor likely tokens, but let runners-up win sometimes. Natural variety, the usual default for chat.
- High (1.5 and up): give long-shot tokens a real chance. Surprising, creative, and eventually unhinged.
Think of it as a jukebox setting. Temp 0: it plays the number-one hit, every time, forever. Medium: a sensible shuffle of the top 40. High: suddenly it's 3am and the jukebox is playing Mongolian throat metal and you don't remember asking for this.
See it with your own eyes
Same prompt, three temperatures. Drag through the samples and watch determinism melt into chaos:
๐ก๏ธ Temperature demo
Prompt
Suggest a name for a cozy neighborhood coffee shop.
Temperature: 0
The Morning Cup
Low temperature โ predictable, focused output.
Notice the shape of the change: at 0 you get the safest possible answer. At 0.7 you get plausible variety: any of those could be a real shop. At 1.5 you get ideas no focus group would survive... one of which might be brilliant.
Matching the knob to the job
| Task | Temperature | Why |
|---|---|---|
| Reviewing a legal contract clause | 0 | You want the careful, repeatable reading, not a creative interpretation of an indemnity clause |
| Extracting data, math, classification | 0 to 0.3 | Consistency is the whole point; randomness only adds errors |
| Everyday chat and writing help | around 0.7 | Natural-sounding variety without wandering off |
| Brainstorming 20 product names | 1.0 and up | You want the weird long shots; you can throw away 18 of them |
The rule of thumb: when there's one right answer, turn it down. When you want many possible answers, turn it up.
Temperature changes how tokens are sampled, not how much the model knows. Temp 0 does not mean "always correct"; a model can be confidently wrong at any temperature. It just means it will be wrong the same way every time, which is at least easier to debug.
The other knob worth knowing: max tokens
Max tokens caps how long the response can be. It's a hard cutoff, not a suggestion: set it too low and the answer just stops mid-sente
Sorry. Like that. If outputs keep ending abruptly, your max tokens setting is too tight. And note it's a ceiling, not a target: if you want concise, say so in the prompt ("under 100 words"). The knob limits; the prompt instructs.
Where do you set these? In API calls and developer playgrounds they're explicit fields. Consumer chat apps usually choose them for you, which is why the same model can feel different in different products.
Checkpoint
Temperature scales randomness in next-token sampling (low for one-right-answer tasks, high for brainstorming), and max tokens is a hard length ceiling, not a style request.
๐ Quiz
Question 1 of 4What does temperature actually control?