Skip to main content

At a high level, what does attention do in a transformer?

LLM and transformer basics · Intermediate 1

Short answer

Attention lets each token look at the other tokens in the input and decide which ones matter for understanding it. For example, in "the bank of the river", attention helps "bank" draw meaning from "river". Stacking many attention layers lets the model build up context across a whole passage.

Deeper answer

In a transformer each token is turned into a vector. In an attention layer, every token computes how relevant every other token is to it and takes a weighted mix of their information. Multi-head attention does this several ways at once, so different heads can track different relationships, such as grammar or reference.

For generation, models use causal attention: a token can attend only to earlier tokens, which is what makes next-token prediction possible. Attention cost grows with the square of the input length in the basic form, which is one reason long context is expensive.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

What AI, machine learning and a language model are

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.