What is the difference between a random forest and gradient boosting?
Machine learning basics · Intermediate 2
Short answer
Both combine many decision trees. A random forest trains deep trees independently on random samples of rows and features and averages them, which mainly cuts variance. Gradient boosting trains shallow trees one after another, each fixing the errors of the ones before, which mainly cuts bias. Boosting is often more accurate on tables but needs more careful tuning.
Deeper answer
Random forests are forgiving: sensible defaults work, they rarely overfit badly as you add trees, and training parallelises. Gradient boosting libraries such as XGBoost and LightGBM usually win on tabular data competitions, but you must tune the learning rate, number of trees, tree depth and regularisation, and use early stopping on a validation set because adding trees can overfit.
Both give feature importance scores, though these can mislead with correlated features; permutation importance or SHAP values are more reliable. For a first model on a table, a random forest is a strong baseline; gradient boosting is usually the one you ship after tuning.
What a strong answer covers
- Both are ensembles of decision trees
- Random forest: independent trees on random samples, averaged
- Boosting: trees in sequence fixing earlier errors
- Forest reduces variance, boosting reduces bias
- Boosting needs tuning and early stopping
Answer it out loud in the mock interview about 1.5 min
Follow-ups to expect
- Why does adding trees to a random forest not cause overfitting the way it can in boosting?
- How does the learning rate interact with the number of trees?
Common mistakes
- Saying boosting trains trees in parallel
- Forgetting early stopping when boosting
Learn it
- How would you tell whether a new model really performs better in an A/B test?
- Explain the bias-variance trade-off.
- How does k-means work, and how do you choose k?
- What is data leakage, and how do you prevent it?
- Give an example of feature engineering that made a model better.
- How do you know whether you have enough training data?
- How do you handle a heavily imbalanced classification problem, such as fraud detection?
- What is the difference between linear and logistic regression?
- How do you deal with missing values in a dataset?
- Walk me through an ML project you did, end to end.
- What is overfitting, and how do you spot it?
- Explain precision and recall with an example.
- What do L1 and L2 regularisation do?
- What is the difference between supervised and unsupervised learning?
- Why do we split data into training, validation and test sets?
- What is the difference between training and inference?