Your chatbot costs about three times more per reply in Tamil than in English. Why, and what would you do?
LLM and transformer basics · Intermediate 2
Short answer
APIs charge per token, and many tokenizers were built mainly for English, so they split Tamil into several times more tokens for the same meaning. I would measure tokens per reply for each language and model, compare models with more efficient tokenizers or Indian-language models, keep prompts lean, and budget cost per language.
Deeper answer
Higher token counts also mean slower replies and less room in the context window, so the effect is not only cost. Newer tokenizers handle Indian scripts much better, so the gap varies a lot between models.
Options include a model with an efficient tokenizer for Tamil, an Indian-language model, or translating to English and back for some steps, each tested on the same Tamil test set for quality, since saving tokens is no use if answers get worse. Romanised Tamil may also tokenise differently from Tamil script.
What a strong answer covers
- Billing is per token
- Tokenizers split Indian scripts into more tokens
- Measures per language and model
- Compares alternatives while checking quality
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- How would you measure token fertility for a language?
- What are the risks of translating to English and back?
Common mistakes
- Assuming the model is simply more verbose in Tamil
- Optimising tokens without checking quality
Learn it
AI in Indian languages: tokens, quality gaps and what to use
- At a high level, what does attention do in a transformer?
- What is a context window, and what happens when a conversation gets too long?
- What is DPO, and how is it different from RLHF?
- How is a chat model like ChatGPT trained? What is the difference between SFT and RLHF?
- What does the temperature setting do?
- What is test-time compute, and how can you use it to get better answers?
- What is a token, and why do tokens matter when you build with LLMs?
- What is a large language model, and how does it produce an answer?
- What is perplexity, and what are its limits as a metric?
- Why do language models hallucinate, and how do you reduce it in an app?