It earns a place when
- Use gradient boosting for strong tabular performance, especially when careful validation and feature handling are available.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 2 · Lesson 16
Gradient boosting
Understand boosting as sequential error correction and compare major gradient-boosted tree implementations.
Lesson goal
Understand boosting as sequential error correction and compare major gradient-boosted tree implementations.
The core idea
Each new small tree concentrates on what the current ensemble still gets wrong. The ensemble grows in a direction that reduces the loss.
Mental model
Picture it this way. Each new small tree concentrates on what the current ensemble still gets wrong. The ensemble grows in a direction that reduces the loss. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
At stage t, a weak learner approximates the negative gradient of the loss with respect to current predictions. A learning rate controls how much each learner contributes.
Worked example
A boosted model for late payments starts with a simple prediction, then adds shallow trees that correct residual structure involving account age, payment history and usage.
When to use it
Do not make it the default when
- Do not assume XGBoost, LightGBM or CatBoost wins without data-dependent validation. More trees and depth are not free.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Boosting can overfit noisy labels, become sensitive to leakage and be hard to calibrate. Hyperparameters interact strongly.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Use HistGradientBoostingClassifier as a local baseline. Tune learning rate and tree depth separately before trying a large search.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What does each boosting learner mainly do?
2. What does a learning rate control?
3. Why can boosting overfit?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
scikit-learn User Guide. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.