What’s Included

The following algorithms are implemented in the Spinning Up package:

• Vanilla Policy Gradient (VPG)
• Trust Region Policy Optimization (TRPO)
• Proximal Policy Optimization (PPO)
• Deep Deterministic Policy Gradient (DDPG)
• Twin Delayed DDPG (TD3)
• Soft Actor-Critic (SAC)

On-Policy Algorithms

don’t use old data → weaker on sample efficiency.

works out mathematically

tradeoff: sample efficiency vs stability

Off-Policy Algorithms

can use old data → stronger sample efficiency

Bellman’s equations for optimality

DDPG and Q-Learning

Algorithm Function for a PyTorch Implementation