Monte Carlo credit
Experiment controls
Choose what to replay
Loading reproducible training traces…
Training metric
Episodic return
Thin: each episode · thick: 20-episode moving average
One-step TD credit
One-step actor–critic
Simulation + estimated V(s)
Clipped batch updates
PPO-Clip
Simulation + estimated V(s)
The algorithms share an interaction budget but update at different boundaries.