Explain the bias-variance trade-off.
Machine learning basics · Intermediate 1
Short answer
Bias is error from a model that is too simple to capture the pattern, so it underfits. Variance is error from a model so flexible that it fits noise, so it overfits. Making a model more complex usually lowers bias and raises variance; you look for the complexity that gives the lowest error on validation data.
Deeper answer
A straight line fitted to curved data has high bias: it is wrong in the same way on every dataset. A very deep decision tree has high variance: train it on a slightly different sample and you get a very different tree. Total error on new data is roughly bias squared plus variance plus noise you cannot remove.
In practice you see it in learning curves. High bias shows as poor scores on both training and validation data; high variance shows as a big gap between them. Remedies differ: for bias, add features or use a more flexible model; for variance, get more data, regularise, simplify, or average many models, which is what random forests do.
What a strong answer covers
- Bias: too simple, underfits
- Variance: too flexible, fits noise, overfits
- More complexity lowers bias and raises variance
- Pick complexity using validation error
- Remedies: regularise, more data, or more features
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How would a learning curve look for a high-variance model?
- Why does bagging reduce variance?
Common mistakes
- Mixing up which of bias and variance means underfitting
- Saying more data fixes high bias
Learn it
- How would you tell whether a new model really performs better in an A/B test?
- What is the difference between a random forest and gradient boosting?
- How does k-means work, and how do you choose k?
- What is data leakage, and how do you prevent it?
- Give an example of feature engineering that made a model better.
- How do you know whether you have enough training data?
- How do you handle a heavily imbalanced classification problem, such as fraud detection?
- What is the difference between linear and logistic regression?
- How do you deal with missing values in a dataset?
- Walk me through an ML project you did, end to end.
- What is overfitting, and how do you spot it?
- Explain precision and recall with an example.
- What do L1 and L2 regularisation do?
- What is the difference between supervised and unsupervised learning?
- Why do we split data into training, validation and test sets?
- What is the difference between training and inference?