What is prompt caching, and when does it help?
Prompting, RAG or fine-tuning · Pro 2
Short answer
Prompt caching lets the provider reuse the processing of a repeated prompt prefix, such as a long system prompt, documents or tool definitions, so later requests that start with the same prefix are cheaper and faster. It helps when many requests share a large, unchanging beginning.
Deeper answer
Caches match on an exact prefix, so put stable content first and anything that changes, such as the user's question, last. Caches expire after a short time unless refreshed or a longer duration is chosen, so they suit steady traffic and multi-turn conversations.
This is different from caching responses, where you store and reuse whole answers for identical or very similar questions. Response caching saves more but risks stale or wrong answers, so use it only where repeated questions should get the same answer.
What a strong answer covers
- The provider reuses processing of a repeated prefix
- Cheaper and faster later requests
- Examples: long system prompt, documents, tool definitions
- Helps when many requests share an unchanging start
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- Where in the prompt should the changing part go?
- What is the difference between prompt caching and response caching?
Common mistakes
- Putting the user's question before the long fixed instructions
- Confusing prompt caching with storing final answers
Learn it
- How would you build a dataset to fine-tune an LLM for a business task?
- Your fine-tuned model is no better than the base model. How do you debug it?
- Models now have very large context windows. Why not put all the documents in the prompt instead of using RAG?
- You have three LoRA adapters for three tasks on the same base model. Would you merge them, serve them separately, or train one multi-task model?
- How do you decide which model size to use for a feature?
- How would you choose a base model to continue training or fine-tune for a specialised domain or language?
- When would you use prompting, RAG or fine-tuning?
- How would you build an AI assistant that works well in several Indian languages?
- What is model distillation, and when would you use it?
- When is fine-tuning worth the effort?
- A reasoning model is more accurate on your task but five times the cost and much slower. How do you decide whether to use it?
- When would you use synthetic data to train or test a model, and what are the risks?