It earns a place when
- Use the ideas to diagnose learning curves, choose learning rates and understand what automatic differentiation is doing.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 4 · Lesson 30
Backpropagation and gradient descent
Trace how a neural network computes gradients and uses them to reduce a loss.
Lesson goal
Trace how a neural network computes gradients and uses them to reduce a loss.
The core idea
Backpropagation is disciplined bookkeeping for the chain rule. Gradient descent takes small steps downhill on the loss surface.
Mental model
Picture it this way. Backpropagation is disciplined bookkeeping for the chain rule. Gradient descent takes small steps downhill on the loss surface. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
For parameters theta, gradient descent updates theta_new = theta - eta times gradient(L). Backpropagation applies the chain rule from output error back through each layer.
Worked example
If a final prediction is too high, the gradient tells each weight how its local contribution affected the error. The learning rate determines the step size.
When to use it
Do not make it the default when
- Do not describe backpropagation as the model. It is a gradient computation used by an optimiser to fit parameters.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Learning can stall with tiny gradients, diverge with steps that are too large or overfit despite a falling training loss.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Python practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Implement gradient descent for y = wx + b using NumPy. Compare your hand-derived gradient with an automatic differentiation result in PyTorch.Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What does backpropagation compute?
2. What does the learning rate control?
3. What can an overly large learning rate do?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
Deep Learning. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.