Skip to main content

How would you tell whether a new model really performs better in an A/B test?

Machine learning basics · Intermediate 1

Short answer

Decide the metric and the smallest effect worth detecting before starting, work out the sample size needed, randomly split users, and run for full weeks. Then compare the groups with a significance test and a confidence interval, and only call a winner when the interval excludes zero and the effect is large enough to matter.

Deeper answer

A p-value below 0.05 means a difference this large would be unlikely if there were truly no effect; it is not the probability that the new model is better. Peeking daily and stopping when it looks good inflates false positives, so fix the duration or use a sequential method.

Watch for novelty effects, weekday patterns and segments that react differently, and check guardrail metrics such as latency or complaints. For ML, offline gains often shrink online, which is exactly why the test matters.

What a strong answer covers

Answer it out loud in the mock interview about 1.5 min

Follow-ups to expect

Common mistakes

Learn it

Statistics for machine learning

Today's AI turns are used

Lessons, roadmaps, videos, practice cards and the code playground stay free with your account, with no limit.