It earns a place when
- Use RL when actions change future states and a reward or utility can be defined through interaction or a simulator.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 6 · Lesson 49
Reinforcement learning and Markov decision processes
Represent sequential decisions with states, actions, transitions, rewards and value.
Lesson goal
Represent sequential decisions with states, actions, transitions, rewards and value.
The core idea
An agent acts in an environment. The quality of an action includes not only its immediate reward but also what it leads to later.
Mental model
Picture it this way. An agent acts in an environment. The quality of an action includes not only its immediate reward but also what it leads to later. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
An MDP contains states S, actions A, transition probabilities P, reward R and discount gamma. A policy pi chooses actions; a value function sums expected discounted future rewards.
Worked example
A warehouse robot choosing a route trades a small delay now against a blocked aisle later. The state must include the information needed for the decision.
When to use it
Do not make it the default when
- Do not use RL to replace a supervised target when the correct action is already labelled, and do not deploy online exploration without safety constraints.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Reward misspecification, partial observability, sparse feedback and distribution shift between simulator and world are serious risks.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Define an MDP for a simple grid world. Write its state, action, transition, reward and terminal condition, then identify one reward loophole.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What is an MDP state meant to capture?
2. Why discount future rewards?
3. What is reward misspecification?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
Reinforcement Learning: An Introduction. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.