At a high level, what does attention do in a transformer?
LLM and transformer basics · Intermediate 1
Short answer
Attention lets each token look at the other tokens in the input and decide which ones matter for understanding it. For example, in "the bank of the river", attention helps "bank" draw meaning from "river". Stacking many attention layers lets the model build up context across a whole passage.
Deeper answer
In a transformer each token is turned into a vector. In an attention layer, every token computes how relevant every other token is to it and takes a weighted mix of their information. Multi-head attention does this several ways at once, so different heads can track different relationships, such as grammar or reference.
For generation, models use causal attention: a token can attend only to earlier tokens, which is what makes next-token prediction possible. Attention cost grows with the square of the input length in the basic form, which is one reason long context is expensive.
What a strong answer covers
- Each token looks at other tokens in the input
- Weights which tokens matter
- Gives an example of context changing meaning
- Stacked layers build context across the passage
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- Why can a generating model only attend to earlier tokens?
- Why does long input make attention expensive?
Common mistakes
- Describing attention as the model looking things up on the internet
- Going deep into maths without first giving the intuition
Learn it
What AI, machine learning and a language model are
- What is a context window, and what happens when a conversation gets too long?
- Your chatbot costs about three times more per reply in Tamil than in English. Why, and what would you do?
- What is DPO, and how is it different from RLHF?
- How is a chat model like ChatGPT trained? What is the difference between SFT and RLHF?
- What does the temperature setting do?
- What is test-time compute, and how can you use it to get better answers?
- What is a token, and why do tokens matter when you build with LLMs?
- What is a large language model, and how does it produce an answer?
- What is perplexity, and what are its limits as a metric?
- Why do language models hallucinate, and how do you reduce it in an app?