LLM Temperature Explained: What It Does and What to Set
Written By:
Matt Tanner
Published on

Alt text (deployment only): Illustrative three-token distributions for build, tests, and network at temperatures 0.2 and 1.5: the probability of build falls from 99.3% to 56.3%, while the other tokens gain probability.
Most advice on LLM temperature fits on a sticky note: use 0 when you want consistent answers and something around 0.7 when you want variety. That note has aged badly. Send a non-default temperature to a Claude 4.7 or later model, and the request fails with a 400 error, and Gemini 3.6 and later models ignore the field altogether. And even a perfectly repeatable classifier can route the same ticket to the wrong team every time.
This guide covers what temperature does, which values to start from, what each major API accepts today, and how to pick a value with a test instead of a guess.
What is LLM temperature?
LLM temperature is a sampling parameter that rescales the model's logits, its raw scores for every candidate next token, before the softmax function turns them into probabilities. Values below 1 sharpen the probability distribution toward the most likely token. Values above 1 flatten it, so less likely tokens get picked more often.
The temperature parameter changes how the next token is drawn, and nothing else: the weights, the prompt, and the context are identical at every setting. So it can make answers more or less varied, but it can't give the model knowledge it doesn't have or turn a hallucination into a fact.
OpenAI's Chat Completions API and Anthropic's API both default temperature to 1. Open models vary: Qwen2.5-0.5B-Instruct, a small open model, ships with 0.7 in its generation config.
How temperature works
At each step, the model outputs one logit per token in its vocabulary. The softmax function exponentiates each one and divides by the sum, turning scores into probabilities that add up to 1. Temperature T divides every logit first:
Take three candidate tokens after "The deploy failed because the", with made-up logits of 2.0 for "build", 1.0 for "tests" and 0.0 for "network". At T = 0.2:
Divide by 0.2: the logits become 10, 5, and 0.
Exponentiate:
e^10โ 22,026.5,e^5โ 148.4,e^0= 1.Sum: about 22,175.9.
Divide: "build" gets 99.3%, "tests" 0.7%, and "network" well under 0.1%.
Run the same steps at other temperatures, and the gap closes:
Temperature | "build" | "tests" | "network" |
|---|---|---|---|
0.2 | 99.3% | 0.7% | <0.1% |
1.0 | 66.5% | 24.5% | 9.0% |
1.5 | 56.3% | 28.9% | 14.8% |
Here's the calculation in Python if you want to try your own logits:
The ranking never changes. "build" keeps the highest probability at every temperature; only the size of its lead moves. At T = 1, the logits pass through untouched, so you sample from the model's own distribution. As T approaches 0, nearly all the probability lands on the top token, so temperature 0 amounts to greedy decoding: always take the most likely next word.
You can't literally divide by zero, so Hugging Face Transformers requires a strictly positive temperature and expects you to switch sampling off (do_sample=False) for greedy decoding instead.

Illustrative probabilities for the same token scores: higher temperature flattens the distribution without changing token rank.
Alt text (deployment only): Illustrative logits 2.0, 1.0, and 0.0 for build, tests, and network produce probabilities of 99.3%, 0.7%, and less than 0.1% at temperature 0.2; 66.5%, 24.5%, and 9.0% at 1.0; and 56.3%, 28.9%, and 14.8% at 1.5.
Low vs. high temperature
Consider this prompt: "Suggest a name for a command-line tool that deletes stale Git branches. Reply with the name only." The constructed examples below illustrate repetition, useful variety, and drift; they are not measured Qwen outputs or predictions for a particular temperature. To compare your own model, hold top-p at 1.0 and disable top-k where your server supports it.
Temperature | Five illustrative responses |
|---|---|
0 (greedy) | GitCleanBranches, GitCleanBranches, GitCleanBranches, GitCleanBranches, GitCleanBranches |
0.7 | SyncBranchesTool, GitClean, Git History Cleaner, GitFest Delete, GitCleanCLI |
1.2 | SyncStone, eraser-git-stale-bugs, Git History Cleaner, GitBaseCLI, Mateer-cli LongAGO |
Low temperature gives you the model's single best guess, over and over. That's what you want for a field extraction or a label, and useless for brainstorming, where five copies of "GitCleanBranches" is one idea.
The middle row illustrates variety that stays near the task. Some names would still need editing; a temperature value alone cannot guarantee useful ideas.
Higher temperatures sample from the tail. At 1.2, "Mateer-cli LongAGO" shows the cost: once an unlikely token gets picked, every later token is conditioned on it, so the output wanders.
Does temperature 0 make an LLM deterministic?
No. In principle, temperature 0 makes the sampling step deterministic, since it always takes the top token. It can't make the logits themselves identical from run to run, and on a hosted API that's where the variation comes from.
Batching changes the math. Thinking Machines traced most endpoint nondeterminism to batch size, which varies with server load, plus floating-point non-associativity. In their test, 1,000 completions of one prompt at temperature 0 on
Qwen3-235B-A22Bproduced 80 different outputs, all identical for the first 102 tokens. When two tokens nearly tie, a small numerical difference can flip the pick, and later tokens are conditioned on that choice.Seeds are best effort. OpenAI's
seedparameter is in beta, and the reference says determinism is not guaranteed, pointing you tosystem_fingerprintto spot backend changes. Gemini picks a random seed when you don't send one.Anthropic says as much. Its older Python SDK documents that even at a temperature of 0.0, results won't be fully deterministic.
The backend can change. A provider update changes outputs regardless of your settings.
Repeatability isn't correctness either. A classifier that always labels "Search returns a 500 error since this morning" as billing is consistently wrong.
Temperature vs. top-p and top-k
Temperature reshapes the distribution; top-p and top-k restrict which tokens remain eligible. Their order is implementation-specific: Hugging Face Transformers applies temperature before top-k and top-p, so changing temperature can also change which tokens enter the nucleus.
Control | What it does | Example ( | When to use it |
|---|---|---|---|
Temperature | Rescales every logit; sharper below 1, flatter above 1 | 66.5% / 24.5% / 9.0% | The default lever, where accepted |
Top-p (nucleus sampling) | Keeps the smallest set of tokens whose probabilities add up to p |
| Cut the long tail, keep variety among strong candidates |
Top-k | Keeps only the k most likely tokens |
| Mostly self-hosted; OpenAI's Chat Completions API has no |
Nucleus sampling comes from Holtzman et al., who proposed sampling from "the dynamic nucleus of the probability distribution" instead of the full tail. OpenAI documents top_p as an alternative to temperature, where 0.1 means only tokens in the top 10% of probability mass are considered.
Change one at a time. OpenAI recommends altering temperature or top_p, not both, and on Claude 4.7 and later models, sending both returns a 400 even at their defaults. If you self-host, check the model's own defaults too: Qwen2.5-0.5B-Instruct ships top_k 20 and top_p 0.8, so changing only temperature still samples through those filters.
Recommended temperature settings by task
Treat these temperature settings as starting points to test. They follow the providers' own guidance: OpenAI calls 0.2 more focused and 0.8 more random, and Anthropic's SDK suggests values near 0.0 for analytical work and near 1.0 for creative work.
Task | Starting temperature | Why |
|---|---|---|
Extraction and classification | 0 to 0.2 | One right answer; variety only adds wrong ones |
Code generation | 0 to 0.3 | Syntax and APIs leave little room for creative token choices |
Summarization | 0.2 to 0.5 | Some freedom in wording, little in facts |
Chat and assistants | 0.5 to 0.8, or the default | Natural phrasing without wandering |
Brainstorming and creative writing | 0.8 to 1.2 | You want distinct ideas, and you'll filter them |
Research doesn't back a universal best value. A 2025 study of 12 open models across six abilities found no single temperature optimal for every task: higher wasn't always best for creative writing, and zero wasn't always best for following instructions.
Lower temperatures also rarely fix accuracy. If a classifier repeatedly misroutes a 500 error as billing, changing sampling alone is not a reliable fix. Labeled examples via few-shot prompting go after the real problem. And if your model only accepts its default temperature setting, the table doesn't apply at all.
Temperature in the main APIs
Support varies by model and API version, so check the docs for your exact combination. Checked November 13, 2026:
API and model | Range | Default temperature | What to know |
|---|---|---|---|
OpenAI Chat Completions | 0 to 2 | 1 | GPT-6 models accept temperature and |
Anthropic, Claude 4.7 and later models | Default only | 1.0 | Non-default temperature, |
Anthropic, older models such as Claude Sonnet 4.6 on API version 2023-06-01 | 0.0 to 1.0 | 1.0 | Temperature is incompatible with extended thinking; on API version 2026-10-21, sending it at any value returns a 400 on every model |
Gemini 3.6 and later, Gemini 4 | Ignored | Model default | Custom values are ignored with deprecation warning headers; future generations will return a 400 (Gemini docs) |
Earlier Gemini 3 models | 0.0 to 2.0 per the API reference | Varies by model | Google strongly recommends keeping the defaults; its troubleshooting page says 0.0 to 1.0 |
Hugging Face Transformers (self-hosted) | Greater than 0 | 1.0, or the model's | Needs |
Where temperature is still accepted, it's one argument. Here's the OpenAI Python SDK with a GPT-6 model that supports reasoning effort "none":
The newest Claude and Gemini models have removed temperature as a user control. If your product relies on a low temperature setting, that's now a real criterion in model selection, next to quality, cost, and latency.
How to choose a temperature for your app
Temperature is a hyperparameter whose best value depends on your task, prompt, and model, and because outputs vary, one run per setting tells you little. Here's the loop we use:
Fix the eval set and scoring first, including what counts as a format failure. Our guide to evaluation metrics covers the options.
Pick three or four temperature settings across the range your API accepts.
Repeat every input several times at each setting.
Measure quality, consistency, and failure rate: accuracy, inputs whose answers changed across repeats, and outputs that broke your format.
Change nothing else between settings.
Here's that loop as a script for an OpenAI-compatible local server. Install the openai package, start your server separately, and set LOCAL_LLM_BASE_URL to its /v1 endpoint and LOCAL_LLM_MODEL to its served model name. Set LOCAL_LLM_API_KEY if it requires authentication. Configure the server to disable top-k and use greedy decoding for temperature 0. Keep the same model, prompt, and decoding settings across arms. The five-token cap tests a deliberately short label contract; truncation can itself cause format failures.
The illustrative scorecard below shows how to read two repetitions of the whole sweep. These are constructed counts, not measured model results; the scoring logic was checked with synthetic responses.
Temperature | Correct, of 24 (run 1, run 2) | Valid label, of 24 (run 1, run 2) | Tickets with differing answers, of 8 (run 1, run 2) |
|---|---|---|---|
0 | 15, 15 | 24, 24 | 0, 0 |
0.7 | 12, 10 | 23, 20 | 5, 5 |
1.2 | 11, 12 | 21, 22 | 7, 7 |
In this illustrative scorecard, temperature 0 ranks first, but still misses 9 of 24 labels. Run 2 reverses the accuracy ordering of 0.7 and 1.2, showing why one sweep is not enough to select a winner. Outputs such as "Billed", "refund", or "Account campaign" count as format failures under the exact-label contract. Investigate the prompt, model, and token cap before attributing errors to temperature. Eight tickets demonstrate the method, not performance on your workload.
With real eval sets and several models, this gets slow. Our open-source RapidFire engine runs it as one experiment: put the temperature values in a List inside an RFAPIModelConfig's model_config, and the grid search turns each value into its own run. RapidFire schedules all runs on one shard of the eval data at a time and cycles through the shards, so every temperature shows up side by side early.
With online aggregation configured, the dashboard shows running estimates with confidence intervals. That lets you stop a weak run with the Stop IC Op well before the full eval finishes. Converge automates those calls: it stops weak versions, clones strong ones, logs every decision, and stays within the time and spend limits you set.
Common mistakes
Tuning temperature to fix a bad prompt. Wrong answers at temperature 0 are the model's best guess given what it was told. Fix the instructions with prompt optimization, or fix what the model sees with context engineering, then revisit temperature.
Changing temperature and top-p together. You won't know which one moved the results, and newer Claude models reject the combination outright.
Judging from one sample. The illustrative scorecard shows how two sweeps can disagree on which setting is more accurate.
Sweeping a parameter the model ignores. On Gemini 3.6 and later, custom temperature values are ignored, so every arm of a sweep samples the same way and any differences you see are noise. Check the provider docs before you spend the eval budget.
To run a multi-config sweep on your own data, start with the RapidFire RAG walkthrough, which has a Colab notebook and a local install path.
FAQ
What is a good temperature for LLM?
There's no universal best value. Start at 0 to 0.2 for extraction and classification, 0 to 0.3 for code, 0.5 to 0.8 for chat, and 0.8 to 1.2 for brainstorming. Then test a few values on your own eval set with repeated runs. Newer Claude and Gemini models only use their default.
Can you set LLM temperature to 0?
On OpenAI's Chat Completions API, yes: the accepted range is 0 to 2 for models and reasoning settings that take temperature. Hugging Face Transformers requires a positive value, so you switch sampling off instead. Claude 4.7 and later models reject non-default values, and Gemini 3.6 and later ignore them.
What does it mean when the LLM temperature is 1?
At temperature 1, the logits pass through unchanged, so the model samples from its own unmodified probability distribution. It's the default on OpenAI's Chat Completions API and on Anthropic's API. Low-probability tokens still get picked regularly, which is why outputs vary between runs.
What does temperature 0 mean in LLMs?
Temperature 0 means greedy decoding in principle: the model always takes the highest-probability next token instead of sampling. Outputs become far more repeatable, but not guaranteed identical, because server-side batching and floating-point effects can change the logits between runs. Repeatable answers can also be repeatably wrong, so test accuracy separately.