It earns a place when
- Use bandits when actions have relatively immediate feedback and the long-term state does not dominate.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 6 · Lesson 48
Contextual bandits
Understand exploration and exploitation when actions produce immediate or short-term feedback.
Lesson goal
Understand exploration and exploitation when actions produce immediate or short-term feedback.
The core idea
A bandit chooses an action, observes a reward, and learns while operating. It must balance trying uncertain options with using the current best option.
Mental model
Picture it this way. A bandit chooses an action, observes a reward, and learns while operating. It must balance trying uncertain options with using the current best option. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
A contextual policy maps context x to an action. Methods estimate expected reward and uncertainty, then choose optimistically or sample from a posterior.
Worked example
A news app chooses which headline to show. A contextual bandit can test alternatives for different readers, but exploration itself affects users and needs guardrails.
When to use it
Do not make it the default when
- Do not use a bandit when delayed consequences, safety constraints or state transitions make a full reinforcement-learning or causal design necessary.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Naive exploration can harm users. Logged feedback is policy-dependent, so off-policy evaluation is difficult.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Simulate epsilon-greedy and upper-confidence-bound policies on a few arms. Plot cumulative reward and regret, then add a constraint that blocks unsafe arms.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What is the exploration-exploitation trade-off?
2. What makes a bandit contextual?
3. Why can logged bandit data be biased?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
Reinforcement Learning: An Introduction. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.