Machine learning · extension
Reinforcement learning
How can a useful move be learned when the reward arrives much later? Follow a delivery route, then repeat it to see earlier moves change in value.
Reinforcement learning updates action estimates using rewards from interaction. Here, moving and delivering produce points; repeated experience changes the estimates. No coding is needed.
The reward arrives at the end
live Computed updates · guided route through an invented world
A delivery earns +1 at the goal. Each earlier move costs 0.1. Predict: can an earlier move become more valuable even though it still costs the same?
Trial 1 · move 0 of 6
Use “Next move” six times to reach the goal. Repeat the route to see its reward affect an earlier move’s estimate.
live The first trial starts with all action estimates at zero.
The route here is chosen for the demonstration; it is not discovered by these clicks. The full lab below also chooses actions and explores other routes. Estimates are predictions of future rewards, not promises. A reward for the wrong outcome can teach the wrong behaviour.
Explain it back: a sorting machine
A sorting machine earns a point only when a parcel reaches the correct bin, several moves after a conveyor switch. Explain how the switch’s action estimate could improve over repeated attempts, and what a successful attempt does not prove.
Try saying it in your own words before opening the explanation. No calculation is needed.
Compare with one explanation
An action leading to a position with a better estimated future reward can receive a better estimate too. Repeated updates can carry the final reward’s influence back towards the earlier switch. One successful parcel does not prove the route will always succeed, or that the reward measures everything we care about.
Explore action choice, training and the update calculation
Rewards evaluate outcomes; labels specify targets
In supervised learning, an example supplies a target output, such as the correct image category. Here, the environment returns the next state and a numerical reward after an action. It does not provide the correct action at every position. The action changes which experience arrives next.
State → action
Position 14 → move right
Environment response
Goal 15, reward +1, episode ends
Learning update
Change Q(14, right) towards +1
illustrative A terminal transition in the grid below. Rewards and walls are fixed; movement is deterministic.
Inspect one update, then train a policy
Q(state, action) estimates discounted future return after that action. A policy is the rule for choosing actions. This demo trains with epsilon-greedy sampling and evaluates with the largest Q value only. The heatmap shows max Q at each state; the arrows show the greedy policy derived from it.
Try one learning step, then reset and train 100 episodes. Select a square to compare its four action values. Set ε to 1 to sample every training action randomly; evaluation still uses the greedy arrows. Set γ to 0 to remove the future-value term and compare how the policy changes.
live Computed in this browser · synthetic gridworld · seed 42
Changing any setting clears Q and restarts seed 42. α controls update size; γ weights future reward; ε is the probability of sampling a random action, which may also be the greedy action.
Policy arrows and value heatmap live
Wall
Wall
● current position. Green is positive max Q; orange is negative. Arrows select the largest action value. Ties use up, right, down, left, in that order. The goal has no next action.
State 0 action values live
| Action | Q estimate |
|---|---|
| Up | 0.000 |
| Right | 0.000 |
| Down | 0.000 |
| Left | 0.000 |
State 0 is the start. Every move costs −0.1, except entering goal 15 gives +1 and ends the episode. A boundary or wall leaves the position unchanged and still costs −0.1.
live Current episode: 0 steps · undiscounted return 0.000 · completed episodes 0
Each episode stops at the goal or 80 steps. A time limit truncates the episode; it does not turn the last state into a terminal goal. A batch starts fresh episodes and retains Q. The next learning step after an episode ends starts at state 0.
Ready. All Q values are zero; seed 42.
One transition, one update live
Take one step to expose the state, sampled action and numerical update.
Training and evaluation measure different behaviour
Run a batch to see its training success count and mean return.
| Q table | Goal | Steps | Return |
|---|---|---|---|
| Untrained | No | 80 | -8.000 |
| Current | No | 80 | -8.000 |
Return here is the sum of rewards without discounting; Q estimates discounted future return. This deterministic world has one start, so repeating the same greedy evaluation adds no evidence. Finite training can leave a looping or incomplete policy.
Delayed credit moves through estimates
On the first visit to the goal, the action just before it receives a positive update. An earlier action can then receive a better target when its next-state estimate has improved. Repeated visits can carry the influence of a distant reward back towards the start.
The update is Q ← Q + α × (reward + γ × max next Q − Q). α changes how far one update moves. γ discounts future reward. The max uses an estimated value, so an update is not proof that the estimate is correct.
Exploration changes the experience
With probability ε, training samples a random action. Otherwise it uses a greedy action. Q-learning still builds its target from the largest next-state Q value, even when the sampled action came from exploration. This difference makes it an off-policy method.
A fixed learning rate, a finite number of episodes and limited exploration do not guarantee convergence. This small deterministic world leaves out uncertain observations, changing dynamics, continuous actions and the cost of gathering real experience.
Check the mechanism
Reward is part of the specification
This grid rewards reaching one marked goal. In a larger system, a high measured return only shows success against the reward that was defined. The next lesson changes the reward and examines how a policy can score well while missing the intended outcome.
Reference: Sutton and Barto, Reinforcement Learning: An Introduction, second edition, chapters 3 and 6; Q-learning update in section 6.5.