trpo#

Source code: tianshou/algorithm/modelfree/trpo.py

class TRPOTrainingStats(*, train_time: float = 0.0, smoothed_loss: dict = <factory>, actor_loss: tianshou.data.stats.SequenceSummaryStats, vf_loss: tianshou.data.stats.SequenceSummaryStats, kl: tianshou.data.stats.SequenceSummaryStats, step_size: tianshou.data.stats.SequenceSummaryStats)[source]#

Bases: NPGTrainingStats

step_size: SequenceSummaryStats#
class TRPO(*, policy: ProbabilisticActorPolicy, critic: Module | ContinuousCritic | DiscreteCritic, optim: OptimizerFactory, max_kl: float = 0.01, backtrack_coeff: float = 0.8, max_backtracks: int = 10, optim_critic_iters: int = 5, trust_region_size: float = 0.5, advantage_normalization: bool = True, gae_lambda: float = 0.95, max_batchsize: int = 256, gamma: float = 0.99, return_scaling: bool = False)[source]#

Bases: NPG

Implementation of Trust Region Policy Optimization. arXiv:1502.05477.

Parameters:

  • policy – the policy

  • critic – the critic network. (s -> V(s))

  • optim – the optimizer factory for the critic network.

  • max_kl – max kl-divergence used to constrain each actor network update.

  • backtrack_coeff – Coefficient to be multiplied by step size when constraints are not met.

  • max_backtracks – Max number of backtracking times in linesearch.

  • optim_critic_iters – the number of optimization steps performed on the critic network for each policy (actor) update. Controls the learning rate balance between critic and actor. Higher values prioritize critic accuracy by training the value function more extensively before each policy update, which can improve stability but slow down training. Lower values maintain a more even learning pace between policy and value function but may lead to less reliable advantage estimates. Typically set between 1 and 10, depending on the complexity of the value function.

  • trust_region_size – the parameter delta - a scalar multiplier for policy updates in the natural gradient direction. The mathematical meaning is the trust region size, which is the maximum KL divergence allowed between the old and new policy distributions. Controls how far the policy parameters move in the calculated direction during each update. Higher values allow for faster learning but may cause instability or policy deterioration; lower values provide more stable but slower learning. Unlike regular policy gradients, natural gradients already account for the local geometry of the parameter space, making this step size more robust to different parameterizations. Typically set between 0.1 and 1.0 for most reinforcement learning tasks.

  • advantage_normalization – whether to do per mini-batch advantage normalization.

  • gae_lambda – the lambda parameter in [0, 1] for generalized advantage estimation (GAE). Controls the bias-variance tradeoff in advantage estimates, acting as a weighting factor for combining different n-step advantage estimators. Higher values (closer to 1) reduce bias but increase variance by giving more weight to longer trajectories, while lower values (closer to 0) reduce variance but increase bias by relying more on the immediate TD error and value function estimates. At λ=0, GAE becomes equivalent to the one-step TD error (high bias, low variance); at λ=1, it becomes equivalent to Monte Carlo advantage estimation (low bias, high variance). Intermediate values create a weighted average of n-step returns, with exponentially decaying weights for longer-horizon returns. Typically set between 0.9 and 0.99 for most policy gradient methods.

  • max_batchsize – the maximum number of samples to process at once when computing generalized advantage estimation (GAE) and value function predictions. Controls memory usage by breaking large batches into smaller chunks processed sequentially. Higher values may increase speed but require more GPU/CPU memory; lower values reduce memory requirements but may increase computation time. Should be adjusted based on available hardware resources and total batch size of your training data.

  • gamma – the discount factor in [0, 1] for future rewards. This determines how much future rewards are valued compared to immediate ones. Lower values (closer to 0) make the agent focus on immediate rewards, creating “myopic” behavior. Higher values (closer to 1) make the agent value long-term rewards more, potentially improving performance in tasks where delayed rewards are important but increasing training variance by incorporating more environmental stochasticity. Typically set between 0.9 and 0.99 for most reinforcement learning tasks

  • return_scaling – flag indicating whether to enable scaling of estimated returns by dividing them by their running standard deviation without centering the mean. This reduces the magnitude variation of advantages across different episodes while preserving their signs and relative ordering. The use of running statistics (rather than batch-specific scaling) means that early training experiences may be scaled differently than later ones as the statistics evolve. When enabled, this improves training stability in environments with highly variable reward scales and makes the algorithm less sensitive to learning rate settings. However, it may reduce the algorithm’s ability to distinguish between episodes with different absolute return magnitudes. Best used in environments where the relative ordering of actions is more important than the absolute scale of returns.