It earns a place when
- Use them as reliable tabular baselines with modest preprocessing and useful feature importance tools.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 2 · Lesson 15
Random forests and extremely randomised trees
Understand bagging and random feature selection as ways to reduce tree variance.
Lesson goal
Understand bagging and random feature selection as ways to reduce tree variance.
The core idea
Instead of trusting one brittle tree, grow many somewhat different trees and average their errors.
Mental model
Picture it this way. Instead of trusting one brittle tree, grow many somewhat different trees and average their errors. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
Bagging trains models on bootstrap samples and averages their predictions. Random forests also sample features at splits. Extra Trees add more randomisation to split thresholds.
Worked example
For tabular churn data, many trees can capture different interactions. Their average is usually more stable than one deep tree.
When to use it
Do not make it the default when
- Do not expect a forest to extrapolate smoothly outside the training range or to solve leakage and target-definition problems.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Feature importance can favour high-cardinality or continuous variables. Large forests increase memory and inference cost.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Compare one tree, RandomForestClassifier and ExtraTreesClassifier. Use permutation importance on a held-out set, not only impurity importance.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What problem does bagging mainly address?
2. What extra randomness does a random forest use?
3. Why prefer permutation importance on held-out data?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
scikit-learn User Guide. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.