Skip to main content

Your chatbot costs about three times more per reply in Tamil than in English. Why, and what would you do?

LLM and transformer basics · Intermediate 2

Short answer

APIs charge per token, and many tokenizers were built mainly for English, so they split Tamil into several times more tokens for the same meaning. I would measure tokens per reply for each language and model, compare models with more efficient tokenizers or Indian-language models, keep prompts lean, and budget cost per language.

Deeper answer

Higher token counts also mean slower replies and less room in the context window, so the effect is not only cost. Newer tokenizers handle Indian scripts much better, so the gap varies a lot between models.

Options include a model with an efficient tokenizer for Tamil, an Indian-language model, or translating to English and back for some steps, each tested on the same Tamil test set for quality, since saving tokens is no use if answers get worse. Romanised Tamil may also tokenise differently from Tamil script.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

AI in Indian languages: tokens, quality gaps and what to use

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.