Skip to main content

Pattern · Serving and cost

Streaming and background queues

Stream tokens so users see words at once, and move slow work (OCR, indexing, batch jobs) to a queue.

Say it in the interview

I would stream tokens so the first words arrive in under a second, set timeouts above the model's worst case, and move slow work like OCR to a queue with retries, a dead-letter queue and a job id the user can poll.

When to use it

  • Model calls take seconds
  • Some work does not need to finish before replying

How it works

  1. User → API
  2. API → stream tokens → First words in under 1 s
  3. API → slow work → Queue
  4. Queue → Workers (OCR, indexing)
  5. Workers (OCR, indexing) → failures retry with backoff → Dead-letter queue after N tries
  6. API → job id → User checks status

Trade-offs

  • Feels fast even when total time is long
  • Queues add moving parts to monitor

Pitfalls interviewers probe

  • Timeouts shorter than model calls
  • No dead-letter queue, so failed jobs vanish

Practise it on these questions

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.