Pattern · Serving and cost
Model routing and tiering
Send easy requests to a small, cheap model and hard ones to a large model, checked by a validator.
Say it in the interview
I would measure what share of traffic is easy, route it to a small model with a validator that escalates failures to the large model, and push anything that can wait into the discounted batch API.
When to use it
- High volume with mixed difficulty
- Cost or latency pressure
How it works
- Request → Router
- Router → easy → Small model
- Small model → Validator passes?
- Validator passes? → yes → Answer
- Validator passes? → no → Large model
- Router → hard → Large model
- Large model → Answer
- Request → can wait hours → Batch API, discounted
Trade-offs
- Large savings at similar quality
- A router that misroutes hurts quality silently
Pitfalls interviewers probe
- Routing without measuring quality per route
- Never using the batch API for work that can wait