It earns a place when
- Use these tools when training is unstable, gradients vanish or the network memorises before learning a robust representation.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 4 · Lesson 31
Activations, normalisation and regularisation
Understand the practical tools that keep deep networks expressive and trainable.
Lesson goal
Understand the practical tools that keep deep networks expressive and trainable.
The core idea
Activations bend the function. Normalisation keeps internal scales manageable. Regularisation makes memorisation less attractive.
Mental model
Picture it this way. Activations bend the function. Normalisation keeps internal scales manageable. Regularisation makes memorisation less attractive. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
ReLU is max(0, z); sigmoid and tanh saturate. Batch normalisation standardises intermediate activations using batch statistics. Weight decay adds a parameter penalty; dropout randomly masks units during training.
Worked example
A vision network with ReLU, sensible initialisation, normalisation and weight decay can train much more reliably than the same network with saturated activations and no control of capacity.
When to use it
Do not make it the default when
- Do not add every regulariser at once. Batch statistics can behave differently at inference, and dropout is not a cure for leakage or bad labels.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Train and validation curves can improve for different reasons. Regularisation that reduces training accuracy may still improve test performance.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Train the same small MLP with and without standardisation, weight decay and dropout. Compare curves and explain which change affected optimisation versus generalisation.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What is the main role of a nonlinear activation?
2. What does weight decay discourage?
3. Why can dropout help?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
Deep Learning. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.