What is a context window, and what happens when a conversation gets too long?
LLM and transformer basics · Intermediate 1
Short answer
The context window is the maximum number of tokens the model can consider in one request: system prompt, conversation history, documents and its own reply together. When you go past it, the request fails or something must be cut, so apps trim, summarise or retrieve only the relevant parts.
Deeper answer
Everything the model sees in one call shares the window. In a long chat, each turn resends the earlier turns, so the input grows. Common strategies are to drop the oldest turns, summarise earlier parts, or store information outside the prompt and retrieve only what the current question needs.
A bigger window is not free: more input tokens cost more and take longer, and models can pay less attention to material buried in the middle of a very long input. So even with a large window, sending only what is relevant usually gives better and cheaper answers.
What a strong answer covers
- The maximum tokens a model handles in one request
- Prompt, history, documents and reply all share it
- Too long means failure or cutting
- Fixes: trim, summarise or retrieve
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- If the window is huge, why use RAG at all?
- How would you handle a chat that runs for hundreds of turns?
Common mistakes
- Thinking only the latest message counts towards the limit
- Assuming a larger window always gives better answers
Learn it
- At a high level, what does attention do in a transformer?
- Your chatbot costs about three times more per reply in Tamil than in English. Why, and what would you do?
- What is DPO, and how is it different from RLHF?
- How is a chat model like ChatGPT trained? What is the difference between SFT and RLHF?
- What does the temperature setting do?
- What is test-time compute, and how can you use it to get better answers?
- What is a token, and why do tokens matter when you build with LLMs?
- What is a large language model, and how does it produce an answer?
- What is perplexity, and what are its limits as a metric?
- Why do language models hallucinate, and how do you reduce it in an app?