What is test-time compute, and how can you use it to get better answers?
LLM and transformer basics · Pro 1
Short answer
Test-time compute means spending more computing at answer time instead of training time. You can ask the model to reason step by step, sample several answers and keep the best using a checker or a majority vote, or use a reasoning model that thinks first. Each trades extra cost and waiting time for accuracy.
Deeper answer
Chain-of-thought prompting is the cheapest step and often a big gain on multi-step problems. Best-of-N needs a way to choose: a verifier such as tests for code, a majority vote for exact answers, or a judge for open text. OpenAI found a small model with a good maths verifier matched a model about 30 times larger.
Reasoning models are trained to think at length and expose an effort setting. The production pattern is routing: answer easy questions fast, and spend extra compute only when a check fails or samples disagree.
What a strong answer covers
- Spends more compute at inference, not training
- Step-by-step or chain-of-thought reasoning
- Several samples with a verifier or vote
- Trades cost and latency for accuracy
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- When does a majority vote not work?
- How would you keep reasoning costs under control?
Common mistakes
- Treating it as a training technique
- Ignoring the extra cost and latency
Learn it
Reasoning models and test-time compute: when to let the model think longer
- At a high level, what does attention do in a transformer?
- What is a context window, and what happens when a conversation gets too long?
- Your chatbot costs about three times more per reply in Tamil than in English. Why, and what would you do?
- What is DPO, and how is it different from RLHF?
- How is a chat model like ChatGPT trained? What is the difference between SFT and RLHF?
- What does the temperature setting do?
- What is a token, and why do tokens matter when you build with LLMs?
- What is a large language model, and how does it produce an answer?
- What is perplexity, and what are its limits as a metric?
- Why do language models hallucinate, and how do you reduce it in an app?