It earns a place when
- Use a validation split or cross-validation for model selection, and reserve the test set for a final estimate.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 1 · Lesson 4
Train, validation and test sets
Explain why data is split and how validation and test data serve different roles.
Lesson goal
Explain why data is split and how validation and test data serve different roles.
The core idea
The test set is a sealed exam. If you keep looking at it while changing the model, it stops being an honest exam.
Mental model
Picture it this way. The test set is a sealed exam. If you keep looking at it while changing the model, it stops being an honest exam. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
Training minimises an objective on training data. Validation estimates choices during development. The final test estimate is used once, after choices are locked.
Worked example
If five model settings are tried and the best one is selected by test accuracy, the reported accuracy is optimistic because the test set influenced the choice.
When to use it
Do not make it the default when
- Do not repeatedly tune against the test set, and do not split randomly when time or groups create dependence.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Leakage between splits can make a model appear strong. Small datasets can also make one split noisy.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Sketch a split for a customer dataset, then redesign it for monthly forecasting where the future must not enter the past.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What is the main job of validation data?
2. Why keep the test set sealed?
3. When is a random split risky?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
scikit-learn User Guide. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.