Pattern · Serving and cost
Prompt and response caching
Reuse work: cache the stable start of prompts, and cache whole answers to repeated questions.
Say it in the interview
I would order every prompt stable-first so the provider can cache the prefix, and add a response cache keyed on the normalised question plus prompt and model versions, with a short expiry for anything time-sensitive.
When to use it
- Long fixed instructions or documents sent every call
- Many users ask the same questions
How it works
- Request → Stable prefix first (instructions, tools, documents)
- Stable prefix first (instructions, tools, documents) → prompt cache hit → Cheaper, faster first token
- Request → Response cache (key = normalised question + versions)
- Response cache (key = normalised question + versions) → hit → Answer instantly
- Response cache (key = normalised question + versions) → miss → Model
- Model → Stream to the user
Trade-offs
- Lower cost and latency
- A stale cache gives outdated answers
Pitfalls interviewers probe
- Putting the changing part first, which breaks prompt caching
- Cache keys without prompt and model versions