Skip to main content

A new model has just topped a public leaderboard. Your manager wants to switch. What do you do?

Evals · Pro 1

Short answer

Treat it as a candidate, not a decision. Check hard requirements first: licence, data location, languages, price and speed. If it passes, run it on our own test set with the same prompt and rubric as the current model, and compare quality, cost and latency together. Switch only if it clearly wins on our task.

Deeper answer

Leaderboards use a few chosen public benchmarks, mostly in English, and small differences in prompts and scoring can move results. Popular benchmarks may also be contaminated, meaning the questions were in the training data.

Our own test set uses real inputs and our quality bar, so it answers the question that matters. If the new model wins, I would roll it out gradually, for example to 10% of traffic, and watch quality and cost before a full switch. Having a test set ready is what makes switching models a routine job.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Choosing a model: benchmarks, leaderboards and your own tests

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.