Skip to main content

Pattern · Serving and cost

Prompt and response caching

Reuse work: cache the stable start of prompts, and cache whole answers to repeated questions.

Say it in the interview

I would order every prompt stable-first so the provider can cache the prefix, and add a response cache keyed on the normalised question plus prompt and model versions, with a short expiry for anything time-sensitive.

When to use it

  • Long fixed instructions or documents sent every call
  • Many users ask the same questions

How it works

  1. Request → Stable prefix first (instructions, tools, documents)
  2. Stable prefix first (instructions, tools, documents) → prompt cache hit → Cheaper, faster first token
  3. Request → Response cache (key = normalised question + versions)
  4. Response cache (key = normalised question + versions) → hit → Answer instantly
  5. Response cache (key = normalised question + versions) → miss → Model
  6. Model → Stream to the user

Trade-offs

  • Lower cost and latency
  • A stale cache gives outdated answers

Pitfalls interviewers probe

  • Putting the changing part first, which breaks prompt caching
  • Cache keys without prompt and model versions

Practise it on these questions

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.