Choose what to replay

Loading reproducible training traces…

Select one or more algorithms
Environment 5 × 5 Gridworld Episodes — Seed — Discount —

Episodic return

Episode —
Episodic return and its moving average during training.

Thin: each episode · thick: 20-episode moving average

REINFORCE + baseline

Step —

Simulation + estimated V(s)

A Gridworld showing the REINFORCE trajectory over its estimated state values.
lowhigh V(s) Blue dot: agent · orange: path · number: V(s)

One-step actor–critic

Step —

Simulation + estimated V(s)

A Gridworld showing the actor-critic trajectory over its estimated state values.
lowhigh V(s) Blue dot: agent · orange: path · number: V(s)

PPO-Clip

Step —

Simulation + estimated V(s)

A Gridworld showing the PPO trajectory over its estimated state values.
lowhigh V(s) Blue dot: agent · orange: path · number: V(s)

The algorithms share an interaction budget but update at different boundaries.