Skip to main content

When would you use a semantic cache for an LLM app, and what can go wrong?

Deployment and LLMOps · Pro 2

Short answer

A semantic cache reuses an earlier answer when a new question means the same thing, matched by embedding similarity above a threshold. It pays off when many questions repeat with different wording and answers do not depend on user-specific details. It goes wrong when similar-looking questions need different answers, such as different order numbers, names or dates.

Deeper answer

It adds an embedding call and a vector search to every request, so the hit rate must be high enough to save more than it costs. The threshold needs testing on real pairs: too loose returns wrong answers, too strict rarely hits.

Safer options come first: stock answers for the top FAQs, an exact cache on a normalised question plus prompt version, model and date, and provider prompt caching. Include anything that changes the answer in the cache key, expire entries when the source data changes, and measure the cache's error rate, not only its hit rate.

What a strong answer covers

Answer it out loud in the mock interview about 1 min

Follow-ups to expect

Common mistakes

Learn it

Grow an AI app in layers: context, guardrails, routing, caching and agents

Practice Deployment and LLMOps questions

More on this topic

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.