It earns a place when
- Use Q methods for manageable discrete action spaces and policy or actor-critic methods for large or continuous spaces.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 6 · Lesson 50
Q-learning, policy gradients and actor-critic methods
Compare value-based and policy-based reinforcement-learning methods.
Lesson goal
Compare value-based and policy-based reinforcement-learning methods.
The core idea
Q-learning learns how good each state-action pair is. Policy gradients adjust the policy directly. Actor-critic methods use a value estimate to guide policy updates.
Mental model
Picture it this way. Q-learning learns how good each state-action pair is. Policy gradients adjust the policy directly. Actor-critic methods use a value estimate to guide policy updates. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
The Q-learning target is r + gamma max_a' Q(s', a'). Policy gradients use the gradient of expected return. An actor chooses; a critic estimates value or advantage.
Worked example
For a discrete game, Q-learning can store a table. For continuous control, a policy network may output actions directly, with a critic helping reduce variance.
When to use it
Do not make it the default when
- Do not compare algorithms without matching exploration, reward scale, environment and evaluation protocol.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Bootstrapping targets can be unstable. Policy-gradient estimates are noisy, and function approximation can amplify errors.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Implement tabular Q-learning in the grid world, then sketch the actor and critic inputs and outputs for a continuous-control task.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What does Q(s,a) estimate?
2. What is the actor in actor-critic?
3. Why use a critic?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
Reinforcement Learning: An Introduction. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.