How would you tell whether a new model really performs better in an A/B test?
Machine learning basics · Intermediate 1
Short answer
Decide the metric and the smallest effect worth detecting before starting, work out the sample size needed, randomly split users, and run for full weeks. Then compare the groups with a significance test and a confidence interval, and only call a winner when the interval excludes zero and the effect is large enough to matter.
Deeper answer
A p-value below 0.05 means a difference this large would be unlikely if there were truly no effect; it is not the probability that the new model is better. Peeking daily and stopping when it looks good inflates false positives, so fix the duration or use a sequential method.
Watch for novelty effects, weekday patterns and segments that react differently, and check guardrail metrics such as latency or complaints. For ML, offline gains often shrink online, which is exactly why the test matters.
What a strong answer covers
- Fixes metric and minimum effect up front
- Computes sample size or duration
- Random assignment of users
- Significance test and confidence interval
- Avoids peeking; checks guardrail metrics
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- What is wrong with stopping the test as soon as it looks significant?
- What does a 95% confidence interval tell you?
Common mistakes
- Misreading the p-value
- Peeking and stopping early
Learn it
Statistics for machine learning
- Explain the bias-variance trade-off.
- What is the difference between a random forest and gradient boosting?
- How does k-means work, and how do you choose k?
- What is data leakage, and how do you prevent it?
- Give an example of feature engineering that made a model better.
- How do you know whether you have enough training data?
- How do you handle a heavily imbalanced classification problem, such as fraud detection?
- What is the difference between linear and logistic regression?
- How do you deal with missing values in a dataset?
- Walk me through an ML project you did, end to end.
- What is overfitting, and how do you spot it?
- Explain precision and recall with an example.
- What do L1 and L2 regularisation do?
- What is the difference between supervised and unsupervised learning?
- Why do we split data into training, validation and test sets?
- What is the difference between training and inference?