Why do we split data into training, validation and test sets?
Machine learning basics · Pro 1
Short answer
To measure how a model or system performs on data it has not seen. You train on the training set, tune choices on the validation set, and report final performance once on the test set. If you tune on the test set, your score becomes optimistic and stops predicting real-world results.
Deeper answer
Each tuning decision leaks a little information about the data you tuned on. Keeping a test set untouched until the end gives an honest estimate. For time-based data, split by time so you do not train on the future.
The same discipline applies to LLM apps: keep a fixed eval set, and if you iterate heavily on it, keep a second held-out set to confirm the improvement is real. Make sure the sets reflect real user inputs, including messy and adversarial ones.
What a strong answer covers
- Measures performance on unseen data
- Train on training, tune on validation
- Report once on the test set
- Tuning on the test set makes scores optimistic
Answer it out loud in the mock interview about 1 min
Follow-ups to expect
- How would you split data that changes over time?
- How does this apply when you only change prompts, not train a model?
Common mistakes
- Tuning repeatedly on the test set
- Letting near-duplicate examples appear in both training and test data
Learn it
- How would you tell whether a new model really performs better in an A/B test?
- Explain the bias-variance trade-off.
- What is the difference between a random forest and gradient boosting?
- How does k-means work, and how do you choose k?
- What is data leakage, and how do you prevent it?
- Give an example of feature engineering that made a model better.
- How do you know whether you have enough training data?
- How do you handle a heavily imbalanced classification problem, such as fraud detection?
- What is the difference between linear and logistic regression?
- How do you deal with missing values in a dataset?
- Walk me through an ML project you did, end to end.
- What is overfitting, and how do you spot it?
- Explain precision and recall with an example.
- What do L1 and L2 regularisation do?
- What is the difference between supervised and unsupervised learning?
- What is the difference between training and inference?