It earns a place when
- Use transformers when long-range relationships, parallel training or reusable pretrained representations matter.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 5 · Lesson 37
Attention and transformers
Understand attention as content-dependent information routing and transformers as parallel sequence models.
Lesson goal
Understand attention as content-dependent information routing and transformers as parallel sequence models.
The core idea
For each token, attention asks which other tokens matter right now, then forms a weighted mixture of their values.
Mental model
Picture it this way. For each token, attention asks which other tokens matter right now, then forms a weighted mixture of their values. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
Scaled dot-product attention is softmax(Q K^T / sqrt(d_k)) V. Multi-head attention learns several relational views in parallel.
Worked example
In 'The animal did not cross the road because it was tired', attention can connect 'it' with relevant context. Positional information is added because attention alone has no order.
When to use it
Do not make it the default when
- Do not imagine attention as a guaranteed explanation of human reasoning. It is a learned routing mechanism, and its weights do not prove causality.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Quadratic attention cost can be expensive for long contexts. Position handling, data quality and evaluation remain central.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Calculate a tiny one-head attention matrix by hand, then inspect an existing transformer block and identify Q, K, V and positional information.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What does attention weight?
2. Why add positional information?
3. What is a practical transformer cost?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
Attention Is All You Need. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.