Skip to main content

What is a token, and why do tokens matter when you build with LLMs?

LLM and transformer basics · Intermediate 1

Short answer

A token is a chunk of text, often a word or part of one, that the model reads and writes. Tokens matter because price, speed and limits are all counted in them: you pay per input and output token, longer outputs take longer, and everything must fit in the context window.

Deeper answer

Tokenizers split text into pieces from a fixed vocabulary. Common English words are often one token, while rare words, code, numbers and many non-English scripts take more tokens per word. That is worth knowing in India: the same sentence in Hindi or Tamil can cost more tokens than in English.

In practice you count tokens to estimate cost (input tokens times input price plus output tokens times output price), to keep prompts inside the context window, and to set a sensible maximum output length. Providers offer token counting tools so you can measure instead of guessing.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

Tokens and the context window

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.