ppo#

Source code: tianshou/algorithm/modelfree/ppo.py

class PPO(*, policy: ProbabilisticActorPolicy, critic: Module | ContinuousCritic | DiscreteCritic, optim: OptimizerFactory, eps_clip: float = 0.2, dual_clip: float | None = None, value_clip: bool = False, advantage_normalization: bool = True, recompute_advantage: bool = False, vf_coef: float = 0.5, ent_coef: float = 0.01, max_grad_norm: float | None = None, gae_lambda: float = 0.95, max_batchsize: int = 256, gamma: float = 0.99, return_scaling: bool = False)[source]#

Bases: A2C

Implementation of Proximal Policy Optimization. arXiv:1707.06347.

Parameters:

  • policy – the policy containing the actor network.

  • critic – the critic network. (s -> V(s))

  • optim – the optimizer factory for the policy’s actor network and the critic networks.

  • eps_clip – determines the range of allowed change in the policy during a policy update: The ratio of action probabilities indicated by the new and old policy is constrained to stay in the interval [1 - eps_clip, 1 + eps_clip]. Small values thus force the new policy to stay close to the old policy. Typical values range between 0.1 and 0.3, the value of 0.2 is recommended in the original PPO paper. The optimal value depends on the environment; more stochastic environments may need larger values.

  • dual_clip – a clipping parameter (denoted as c in the literature) that prevents excessive pessimism in policy updates for negative-advantage actions. Excessive pessimism occurs when the policy update too strongly reduces the probability of selecting actions that led to negative advantages, potentially eliminating useful actions based on limited negative experiences. When enabled (c > 1), the objective for negative advantages becomes: max(min(r(θ)A, clip(r(θ), 1-ε, 1+ε)A), c*A), where min(r(θ)A, clip(r(θ), 1-ε, 1+ε)A) is the original single-clipping objective determined by eps_clip. This creates a floor on negative policy gradients, maintaining some probability of exploring actions despite initial negative outcomes. Larger values (e.g., 2.0 to 5.0) maintain more exploration, while values closer to 1.0 provide less protection against pessimistic updates. Set to None to disable dual clipping.

  • value_clip – flag indicating whether to enable clipping for value function updates. When enabled, restricts how much the value function estimate can change from its previous prediction, using the same clipping range as the policy updates (eps_clip). This stabilizes training by preventing large fluctuations in value estimates, particularly useful in environments with high reward variance. The clipped value loss uses a pessimistic approach, taking the maximum of the original and clipped value errors: max((returns - value)², (returns - v_clipped)²) Setting to True often improves training stability but may slow convergence. Implementation follows the approach mentioned in arXiv:1811.02553v3 Sec. 4.1.

  • advantage_normalization – whether to do per mini-batch advantage normalization.

  • recompute_advantage – whether to recompute advantage every update repeat according to https://arxiv.org/pdf/2006.05990.pdf Sec. 3.5.

  • vf_coef – coefficient that weights the value loss relative to the actor loss in the overall loss function. Higher values prioritize accurate value function estimation over policy improvement. Controls the trade-off between policy optimization and value function fitting. Typically set between 0.5 and 1.0 for most actor-critic implementations.

  • ent_coef – coefficient that weights the entropy bonus relative to the actor loss. Controls the exploration-exploitation trade-off by encouraging policy entropy. Higher values promote more exploration by encouraging a more uniform action distribution. Lower values focus more on exploitation of the current policy’s knowledge. Typically set between 0.01 and 0.05 for most actor-critic implementations.

  • max_grad_norm – the maximum L2 norm threshold for gradient clipping. When not None, gradients will be rescaled using to ensure their L2 norm does not exceed this value. This prevents exploding gradients and stabilizes training by limiting the magnitude of parameter updates. Set to None to disable gradient clipping.

  • gae_lambda – the lambda parameter in [0, 1] for generalized advantage estimation (GAE). Controls the bias-variance tradeoff in advantage estimates, acting as a weighting factor for combining different n-step advantage estimators. Higher values (closer to 1) reduce bias but increase variance by giving more weight to longer trajectories, while lower values (closer to 0) reduce variance but increase bias by relying more on the immediate TD error and value function estimates. At λ=0, GAE becomes equivalent to the one-step TD error (high bias, low variance); at λ=1, it becomes equivalent to Monte Carlo advantage estimation (low bias, high variance). Intermediate values create a weighted average of n-step returns, with exponentially decaying weights for longer-horizon returns. Typically set between 0.9 and 0.99 for most policy gradient methods.

  • max_batchsize – the maximum size of the batch when computing GAE.

  • gamma – the discount factor in [0, 1] for future rewards. This determines how much future rewards are valued compared to immediate ones. Lower values (closer to 0) make the agent focus on immediate rewards, creating “myopic” behavior. Higher values (closer to 1) make the agent value long-term rewards more, potentially improving performance in tasks where delayed rewards are important but increasing training variance by incorporating more environmental stochasticity. Typically set between 0.9 and 0.99 for most reinforcement learning tasks

  • return_scaling – flag indicating whether to enable scaling of estimated returns by dividing them by their running standard deviation without centering the mean. This reduces the magnitude variation of advantages across different episodes while preserving their signs and relative ordering. The use of running statistics (rather than batch-specific scaling) means that early training experiences may be scaled differently than later ones as the statistics evolve. When enabled, this improves training stability in environments with highly variable reward scales and makes the algorithm less sensitive to learning rate settings. However, it may reduce the algorithm’s ability to distinguish between episodes with different absolute return magnitudes. Best used in environments where the relative ordering of actions is more important than the absolute scale of returns.