What does the temperature setting do?
LLM and transformer basics · Intermediate 1
Short answer
Temperature controls how random the model's choice of the next token is. Low temperature makes it pick the most likely tokens, so answers are more consistent. Higher temperature spreads the choice, giving more varied output. Use low values for extraction and classification, higher ones for brainstorming.
Deeper answer
The model produces a probability for each possible next token. Temperature reshapes that distribution before sampling: below 1 it sharpens it towards the top choices, above 1 it flattens it. Near zero the output is close to always taking the top token, though many providers do not guarantee identical output even then.
Temperature does not make a model smarter or more truthful. A low temperature makes a wrong answer repeat consistently. For tasks where you need correctness, rely on good prompts, retrieval and checks, and use low temperature mainly for consistency.
What a strong answer covers
- Controls randomness of next-token choice
- Low means consistent, likely tokens
- High means varied output
- Use low for extraction, higher for brainstorming
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- Does temperature zero guarantee the same answer every time?
- Which temperature would you use for a JSON extraction task, and why?
Common mistakes
- Saying higher temperature makes the model more creative and therefore better
- Believing low temperature makes answers factually correct
Learn it
System prompts and prompt design
- At a high level, what does attention do in a transformer?
- What is a context window, and what happens when a conversation gets too long?
- Your chatbot costs about three times more per reply in Tamil than in English. Why, and what would you do?
- What is DPO, and how is it different from RLHF?
- How is a chat model like ChatGPT trained? What is the difference between SFT and RLHF?
- What is test-time compute, and how can you use it to get better answers?
- What is a token, and why do tokens matter when you build with LLMs?
- What is a large language model, and how does it produce an answer?
- What is perplexity, and what are its limits as a metric?
- Why do language models hallucinate, and how do you reduce it in an app?