A new model has just topped a public leaderboard. Your manager wants to switch. What do you do?
Evals · Pro 1
Short answer
Treat it as a candidate, not a decision. Check hard requirements first: licence, data location, languages, price and speed. If it passes, run it on our own test set with the same prompt and rubric as the current model, and compare quality, cost and latency together. Switch only if it clearly wins on our task.
Deeper answer
Leaderboards use a few chosen public benchmarks, mostly in English, and small differences in prompts and scoring can move results. Popular benchmarks may also be contaminated, meaning the questions were in the training data.
Our own test set uses real inputs and our quality bar, so it answers the question that matters. If the new model wins, I would roll it out gradually, for example to 10% of traffic, and watch quality and cost before a full switch. Having a test set ready is what makes switching models a routine job.
What a strong answer covers
- Treats the leaderboard as a shortlist signal only
- Checks hard requirements such as licence, data location and price
- Tests on its own data with the same prompt and rubric
- Compares quality, cost and latency together
- Rolls out gradually and monitors
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How big should your test set be for this comparison?
- How would you roll out the switch safely?
- What if the new model is better but twice the price?
Common mistakes
- Switching on the headline score alone
- Comparing on a different prompt from the one in production
- Looking only at quality, not cost and latency
Learn it
Choosing a model: benchmarks, leaderboards and your own tests
- What is benchmark contamination, and how can you detect or avoid it?
- How do you evaluate an agent, beyond checking its final answer?
- How would you evaluate an LLM application?
- How do you evaluate a RAG system?
- How would you build a test set for an LLM feature when you have no labelled data?
- What are the pros and cons of using an LLM as a judge?
- What is the difference between pointwise and pairwise evaluation, and when would you use each?
- How do you stop a prompt or model change from breaking things that used to work?
- How would you evaluate an LLM feature whose answers are open-ended text?