What is data leakage, and how do you prevent it?
Machine learning basics · Intermediate 2
Short answer
Leakage is when information that would not be available at prediction time gets into training, so validation scores look great but production disappoints. Causes include features computed from the future or the target, and preprocessing fitted on the whole dataset. Prevent it by splitting first, fitting every transform inside the training fold, and splitting by time or group where needed.
Deeper answer
Classic examples: predicting loan default with a feature like "number of collection calls", which only exists after default; scaling or imputing with statistics from the full dataset before splitting; or random splits on time-series data so the model sees the future.
The defence is mostly discipline. Split before any exploration that drives modelling choices. Put preprocessing in a pipeline so cross-validation refits it on each training fold. Use time-based splits for anything that changes over time, and group splits when the same customer or patient appears in many rows. A suspiciously high score is itself a warning sign worth investigating.
What a strong answer covers
- Defines leakage: training sees information unavailable at prediction time
- Validation looks too good, production disappoints
- Example: target or future-derived feature
- Fit preprocessing inside the training fold, use pipelines
- Time or group splits where needed
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- How would you split data for a model predicting next month's sales?
- Why does fitting a scaler before the split leak information?
Common mistakes
- Scaling or imputing on the full dataset before splitting
- Random splits on time-ordered data
Learn it
- How would you tell whether a new model really performs better in an A/B test?
- Explain the bias-variance trade-off.
- What is the difference between a random forest and gradient boosting?
- How does k-means work, and how do you choose k?
- Give an example of feature engineering that made a model better.
- How do you know whether you have enough training data?
- How do you handle a heavily imbalanced classification problem, such as fraud detection?
- What is the difference between linear and logistic regression?
- How do you deal with missing values in a dataset?
- Walk me through an ML project you did, end to end.
- What is overfitting, and how do you spot it?
- Explain precision and recall with an example.
- What do L1 and L2 regularisation do?
- What is the difference between supervised and unsupervised learning?
- Why do we split data into training, validation and test sets?
- What is the difference between training and inference?