It earns a place when
- Use the full workflow whenever a model will influence people, money, operations or product behaviour.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 7 · Lesson 60
Capstone: design, train and defend an ML system
Integrate problem framing, model choice, evaluation, deployment and risk into one defensible ML system.
Lesson goal
Integrate problem framing, model choice, evaluation, deployment and risk into one defensible ML system.
The core idea
The capstone is not a model bake-off. It is an argument that a particular system should exist, with evidence for its usefulness and boundaries for its failure.
Mental model
Picture it this way. The capstone is not a model bake-off. It is an argument that a particular system should exist, with evidence for its usefulness and boundaries for its failure. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
State the estimand or prediction target, split and metric before fitting. Compare a baseline and at least one stronger model, quantify uncertainty, inspect slices and define the serving and monitoring loop.
Worked example
For a cancellation-risk system, define the prediction time, target window and action. Build a leakage-safe pipeline, compare logistic regression with a tree ensemble, calibrate the result and decide whether an intervention is justified.
When to use it
Do not make it the default when
- Do not ship a score without an owner, decision policy, data contract, rollback path and a plan for what happens when the model is wrong.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. The most dangerous failure is not a low benchmark score. It is a system that looks successful while measuring the wrong target or causing unmonitored harm.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Produce a short design pack: problem statement, data dictionary, split, baseline, model comparison, error analysis, fairness and calibration checks, serving contract, monitoring dashboard and go/no-go decision.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What should be fixed before model comparison?
2. What makes a capstone defence strong?
3. What is a valid go-no-go question?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
NIST AI Risk Management Framework. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.