Pattern · Serving and cost
Streaming and background queues
Stream tokens so users see words at once, and move slow work (OCR, indexing, batch jobs) to a queue.
Say it in the interview
I would stream tokens so the first words arrive in under a second, set timeouts above the model's worst case, and move slow work like OCR to a queue with retries, a dead-letter queue and a job id the user can poll.
When to use it
- Model calls take seconds
- Some work does not need to finish before replying
How it works
- User → API
- API → stream tokens → First words in under 1 s
- API → slow work → Queue
- Queue → Workers (OCR, indexing)
- Workers (OCR, indexing) → failures retry with backoff → Dead-letter queue after N tries
- API → job id → User checks status
Trade-offs
- Feels fast even when total time is long
- Queues add moving parts to monitor
Pitfalls interviewers probe
- Timeouts shorter than model calls
- No dead-letter queue, so failed jobs vanish