It earns a place when
- Use it for small datasets where a meaningful distance exists and local smoothness is plausible.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 2 · Lesson 12
k-nearest neighbours
Use similarity-based prediction and understand its dependence on distance, scale and local data density.
Lesson goal
Use similarity-based prediction and understand its dependence on distance, scale and local data density.
The core idea
To predict a new case, ask what happened to the most similar known cases. The model is a memory lookup with a distance rule.
Mental model
Picture it this way. To predict a new case, ask what happened to the most similar known cases. The model is a memory lookup with a distance rule. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
For k neighbours, classification can use a majority vote and regression can average targets. Common distances include Euclidean distance, but the metric is part of the model.
Worked example
Recommend a product to a new user by finding users with similar recent interactions. If one feature is measured in dollars and another in seconds, scaling affects who counts as similar.
When to use it
Do not make it the default when
- Do not use raw Euclidean distance on mixed units, huge datasets or high-dimensional sparse data without thinking through the geometry.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. The curse of dimensionality makes neighbours less distinct. Prediction can be slow because many training rows must be searched.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Standardise two features, vary k, and plot the decision boundary. Explain what changes when one feature is left unscaled.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What does kNN use at prediction time?
2. Why does feature scaling matter?
3. What is a common kNN weakness?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
scikit-learn User Guide. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.