What do L1 and L2 regularisation do?
Machine learning basics · Intermediate 1
Short answer
Both add a penalty on the size of the model's weights to the loss, which discourages overfitting. L2 (ridge) penalises squared weights and shrinks them all smoothly towards zero. L1 (lasso) penalises absolute weights and pushes many to exactly zero, so it also selects features. The strength is a hyperparameter you tune on validation data.
Deeper answer
Without a penalty, a model with many features can give large weights to noise. Regularisation trades a little training error for simpler weights that generalise better.
Use L2 when many features each carry a little signal, and L1 when you suspect only a few matter and want a sparse, explainable model. Elastic net mixes both and behaves better when features are correlated, where lasso tends to pick one at random. Scale features first, because the penalty treats all weights alike. In neural networks, L2 appears as weight decay, alongside other regularisers such as dropout.
What a strong answer covers
- Adds a penalty on weight size to reduce overfitting
- L2 shrinks all weights smoothly
- L1 sets some weights to zero, selecting features
- Penalty strength is tuned on validation data
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- Why should you scale features before regularising?
- What happens as the penalty strength grows very large?
Common mistakes
- Saying L2 sets weights to exactly zero
- Not scaling features first
Learn it
- How would you tell whether a new model really performs better in an A/B test?
- Explain the bias-variance trade-off.
- What is the difference between a random forest and gradient boosting?
- How does k-means work, and how do you choose k?
- What is data leakage, and how do you prevent it?
- Give an example of feature engineering that made a model better.
- How do you know whether you have enough training data?
- How do you handle a heavily imbalanced classification problem, such as fraud detection?
- What is the difference between linear and logistic regression?
- How do you deal with missing values in a dataset?
- Walk me through an ML project you did, end to end.
- What is overfitting, and how do you spot it?
- Explain precision and recall with an example.
- What is the difference between supervised and unsupervised learning?
- Why do we split data into training, validation and test sets?
- What is the difference between training and inference?