Skip to main content

Pattern · Serving and cost

Model routing and tiering

Send easy requests to a small, cheap model and hard ones to a large model, checked by a validator.

Say it in the interview

I would measure what share of traffic is easy, route it to a small model with a validator that escalates failures to the large model, and push anything that can wait into the discounted batch API.

When to use it

  • High volume with mixed difficulty
  • Cost or latency pressure

How it works

  1. Request → Router
  2. Router → easy → Small model
  3. Small model → Validator passes?
  4. Validator passes? → yes → Answer
  5. Validator passes? → no → Large model
  6. Router → hard → Large model
  7. Large model → Answer
  8. Request → can wait hours → Batch API, discounted

Trade-offs

  • Large savings at similar quality
  • A router that misroutes hurts quality silently

Pitfalls interviewers probe

  • Routing without measuring quality per route
  • Never using the batch API for work that can wait

Practise it on these questions

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.