It earns a place when
- Use recurrent models when sequence order and state matter and a recurrent inductive bias is useful.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 4 · Lesson 33
RNNs, LSTMs and GRUs
Understand recurrent state and why gated variants help with long dependencies.
Lesson goal
Understand recurrent state and why gated variants help with long dependencies.
The core idea
A recurrent model carries a compact state forward through a sequence. Gating decides what to retain, update and expose.
Mental model
Picture it this way. A recurrent model carries a compact state forward through a sequence. Gating decides what to retain, update and expose. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
An RNN updates h_t = g(W_x x_t + W_h h_{t-1} + b). LSTMs add gates and a cell state to control information flow; GRUs use a simpler gated design.
Worked example
Predict the next character or classify a sensor sequence. A basic RNN may forget an early event; an LSTM can carry that event longer if training supports it.
When to use it
Do not make it the default when
- Do not assume RNNs are the default for every sequence. Transformers often parallelise training and capture long context more directly.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Vanishing and exploding gradients, padding mistakes and teacher-forcing mismatch can hurt sequence performance.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Train a tiny character-level RNN or use a library sequence classifier. Plot gradient norms or compare performance as sequence length increases.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What does recurrent state carry?
2. Why were LSTMs introduced?
3. What is teacher forcing?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
Deep Learning. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.