shortcut-rl

Tabular temporal-difference control on a gridworld with a cliff and a learnable shortcut. Implements and compares Q-learning, SARSA, Expected SARSA, and n-step SARSA.

Environment

A 12×12 grid (ShortcutEnvironment). The agent starts from one of two positions (chosen at random on each reset) and must reach the goal G. Cells marked C are cliffs.

• Actions: 0 up, 1 down, 2 left, 3 right
• Rewards: -1 per step; -100 for stepping onto a cliff, which sends the agent back to its start; reaching the goal ends the episode
• Shortcut: the two start positions make one route shorter but riskier (it runs alongside the cliff), so the agents differ in whether they learn to take it

WindyShortcutEnvironment adds a 50% chance of being pushed one cell down after every action, which makes the cliff-side shortcut considerably more dangerous.

Agents

All agents are ε-greedy and tabular (Q of shape [n_states, n_actions]).

Agent	Update
QLearningAgent	off-policy, bootstraps on max_a Q(s',a)
SARSAAgent	on-policy, bootstraps on the actually taken Q(s',a')
ExpectedSARSAAgent	on-policy, bootstraps on the ε-greedy expectation over Q(s',·)
nStepSARSAAgent	on-policy n-step return

Files

• ShortCutEnvironment.py — environment definitions and greedy-policy rendering
• ShortCutAgents.py — the four agents
• ShortCutExperiment.py — experiment runners and plotting

Running

pip install numpy matplotlib
python ShortCutExperiment.py

Plots are written to results/.

Experiments

ShortCutExperiment.py reproduces:

• Single long run (10 000 episodes): greedy policy rendered in the grid plus a smoothed reward curve
• Repetitions (100 runs × 1000 episodes): averaged learning curves
• α sweep over {0.01, 0.1, 0.5, 0.9} for each agent
• Windy comparison of Q-learning vs SARSA
• n-step SARSA for n ∈ {1, 2, 5, 10, 25}
• Final comparison of all four agents on the same axes