Choose metrics that match the prediction task and the cost of different errors.
This is one focused step in the 60-lesson course. Use the retrieval check before moving on.
Lesson goal
Choose metrics that match the prediction task and the cost of different errors.
The core idea
A model score is not a decision. A threshold turns a probability or ranking score into an action, and the threshold should reflect the cost of mistakes.
Mental model
Picture it this way. A model score is not a decision. A threshold turns a probability or ranking score into an action, and the threshold should reflect the cost of mistakes. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
Accuracy is correct divided by total. Precision is true positives divided by predicted positives. Recall is true positives divided by actual positives. Calibration asks whether predicted probabilities match observed frequencies.
Worked example
For fraud screening, a low threshold catches more fraud but sends more legitimate transactions to review. The right choice depends on review capacity and the cost of missed fraud.
When to use it
It earns a place when
Use multiple views: a primary decision metric, a confusion matrix, threshold curves and calibration when probabilities drive action.
You can evaluate it against a credible baseline.
Its output fits the decision and data constraints.
Do not make it the default when
Do not report accuracy alone on a rare-event problem, and do not tune a threshold on the sealed test set.
A simpler model has not been tested.
The data or target definition is still unclear.
Failure modes
Watch for this. A strong ranking model can have poor calibration. A high AUC can still be useless if the operating threshold is wrong.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Build a confusion matrix by hand for ten predictions. Calculate precision and recall, then explain which one matters more in a chosen use case.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What does recall measure?
2. What does calibration concern?
3. Why can accuracy mislead on rare events?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
scikit-learn User Guide. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.