What is a token, and why do tokens matter when you build with LLMs?
LLM and transformer basics · Intermediate 1
Short answer
A token is a chunk of text, often a word or part of one, that the model reads and writes. Tokens matter because price, speed and limits are all counted in them: you pay per input and output token, longer outputs take longer, and everything must fit in the context window.
Deeper answer
Tokenizers split text into pieces from a fixed vocabulary. Common English words are often one token, while rare words, code, numbers and many non-English scripts take more tokens per word. That is worth knowing in India: the same sentence in Hindi or Tamil can cost more tokens than in English.
In practice you count tokens to estimate cost (input tokens times input price plus output tokens times output price), to keep prompts inside the context window, and to set a sensible maximum output length. Providers offer token counting tools so you can measure instead of guessing.
What a strong answer covers
- A token is a chunk of text, often part of a word
- Price is counted in tokens
- Speed and output length depend on tokens
- Everything must fit in the context window
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- Why might the same sentence cost more tokens in Hindi than in English?
- How would you estimate the monthly cost of a feature?
Common mistakes
- Assuming one token equals one word
- Forgetting that output tokens are usually priced higher than input tokens
Learn it
- At a high level, what does attention do in a transformer?
- What is a context window, and what happens when a conversation gets too long?
- Your chatbot costs about three times more per reply in Tamil than in English. Why, and what would you do?
- What is DPO, and how is it different from RLHF?
- How is a chat model like ChatGPT trained? What is the difference between SFT and RLHF?
- What does the temperature setting do?
- What is test-time compute, and how can you use it to get better answers?
- What is a large language model, and how does it produce an answer?
- What is perplexity, and what are its limits as a metric?
- Why do language models hallucinate, and how do you reduce it in an app?