When would you use a semantic cache for an LLM app, and what can go wrong?
Deployment and LLMOps · Pro 2
Short answer
A semantic cache reuses an earlier answer when a new question means the same thing, matched by embedding similarity above a threshold. It pays off when many questions repeat with different wording and answers do not depend on user-specific details. It goes wrong when similar-looking questions need different answers, such as different order numbers, names or dates.
Deeper answer
It adds an embedding call and a vector search to every request, so the hit rate must be high enough to save more than it costs. The threshold needs testing on real pairs: too loose returns wrong answers, too strict rarely hits.
Safer options come first: stock answers for the top FAQs, an exact cache on a normalised question plus prompt version, model and date, and provider prompt caching. Include anything that changes the answer in the cache key, expire entries when the source data changes, and measure the cache's error rate, not only its hit rate.
What a strong answer covers
- Reuses answers for questions with the same meaning
- Good when questions repeat and are not personal
- Risk: similar questions needing different answers
- Safer alternatives or controls
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- How would you choose the similarity threshold?
- What belongs in a cache key?
Common mistakes
- Caching answers that depend on user details
- Measuring hit rate but not wrong answers
Learn it
Grow an AI app in layers: context, guardrails, routing, caching and agents
- When would you serve predictions in batch, and when online?
- How would you set up CI/CD for an LLM application?
- What is the difference between data parallelism and model parallelism?
- Why does experiment tracking matter, and what do you record?
- Your model scored well offline but performed badly after launch. What could explain it?
- Explain Kubernetes pods, deployments and services, and why you would use Kubernetes for AI apps.
- What is the KV cache, and why can it cause out-of-memory errors when serving an LLM?
- How would you load test an LLM feature and set a latency budget?
- Your model provider has an outage during peak hours. What should already be in place?
- How do you manage API keys and secrets across environments?
- How do you detect and handle model drift in production?
- A bank wants your AI system but says no data may leave its network. What are the options?
- Why does an LLM reply have a wait before the first word and then a different speed after it?
- What is quantisation, and what are the trade-offs?
- How would you serve an LLM feature as an API for a web app?
- How would you serve a model that is too big for one GPU, and how would you scale it to more users?
- How would you add single sign-on to an enterprise AI assistant?
- How do you version prompts and models so you can roll back safely?
- What would you capture in traces for an LLM application?
- Why use Docker for an AI application?
- What does an LLM gateway do, and when do you need one?
- Why use vLLM to serve an open-weight model instead of a simple Hugging Face script?