experiment#

Source code: tianshou/highlevel/experiment.py

The experiment module provides high-level interfaces for setting up and running reinforcement learning experiments.

The main entry points are:

  • ExperimentConfig: a dataclass for configuring the experiment. The configuration is different from RL specific configuration (such as policy and trainer parameters) and only pertains to configuration that is common to all experiments.

  • Experiment: represents a reinforcement learning experiment. It is composed of configuration and factory objects, is lightweight and serializable. An instance of Experiment is usually saved as a pickle file after an experiment is executed.

  • ExperimentBuilder: a helper class for creating experiments. It contains a lot of defaults and allows for easy customization of the experiment setup.

  • ExperimentCollection: a shallow wrapper around a list of experiments providing a simple interface for running them with a launcher. Useful for running multiple experiments in parallel, in particular, for the important case of running experiments that only differ in their random seeds.

Various implementations of the ExperimentBuilder are provided for each of the algorithms supported by Tianshou.

class ExperimentConfig(seed: int = 42, device: str | device = 'cpu', policy_restore_directory: str | None = None, train: bool = True, watch: bool = True, watch_num_episodes: int = 10, watch_render: float = 0.0, persistence_base_dir: str = 'log', persistence_enabled: bool = True, log_file_enabled: bool = True, policy_persistence_mode: Mode = Mode.POLICY)[source]#

Bases: ToStringMixin, DataclassPPrintMixin

Generic config for setting up the experiment, not RL or training specific.

seed: int = 42#

The random seed with which to initialize random number generators.

device: str | device = 'cpu'#

The torch device to use

policy_restore_directory: str | None = None#

Directory from which to load the policy neural network parameters (persistence directory of a previous run)

train: bool = True#

Whether to perform training

watch: bool = True#

Whether to watch agent performance (after training)

watch_num_episodes: int = 10#

Number of episodes for which to watch performance (if watch is enabled)

watch_render: float = 0.0#

Milliseconds between rendered frames when watching agent performance (if watch is enabled)

persistence_base_dir: str = 'log'#

Base directory in which experiment data is to be stored. Every experiment run will create a subdirectory in this directory based on the run’s experiment name

persistence_enabled: bool = True#

Whether persistence is enabled, allowing files to be stored

log_file_enabled: bool = True#

Whether to write to a log file; has no effect if persistence_enabled is False. Disable this if you have externally configured log file generation.

policy_persistence_mode: Mode = 'policy'#

Controls the way in which the policy is persisted

class ExperimentResult(world: World, trainer_result: InfoStats | None)[source]#

Bases: object

Contains the results of an experiment.

world: World#

The World contains all the essential instances of the experiment. Can also be created via Experiment.create_experiment_world for more custom setups, see docstring there.

Note: it is typically not serializable, so it is not stored in the experiment pickle, and shouldn’t be sent across processes, meaning also that ExperimentResult itself is typically not serializable.

trainer_result: InfoStats | None#

dataclass of results as returned by the trainer (if any)

class Experiment(config: ExperimentConfig, env_factory: EnvFactory, agent_factory: AgentFactory, sampling_config: SamplingConfig, name: str, logger_factory: LoggerFactory | None = None)[source]#

Bases: ToStringMixin, DataclassPPrintMixin

Represents a reinforcement learning experiment.

An experiment is composed only of configuration and factory objects, which themselves should be designed to contain only configuration. Therefore, experiments can easily be stored/pickled and later restored without any problems.

The main entry points are:

  1. run(): runs the experiment and returns the results

  2. create_experiment_world(): creates the world object for the experiment, which contains all relevant instances.

    Useful for setting up the experiment and running it in a more custom way.

The methods save() and from_directory() can be used to store and restore experiments.

LOG_FILENAME = 'log.txt'#
EXPERIMENT_PICKLE_FILENAME = 'experiment.pkl'#
classmethod from_directory(directory: str, restore_policy: bool = True) → Experiment[source]#

Restores an experiment from a previously stored pickle.

Parameters:

  • directory – persistence directory of a previous run, in which a pickled experiment is found

  • restore_policy – whether the experiment shall be configured to restore the policy that was persisted in the given directory

get_seeding_info_as_str() → str[source]#

Returns information on the seeds used in the experiment as a string.

This can be useful for creating unique experiment names based on seeds, e.g. A typical example is to do experiment.name = f”{experiment.name}_{experiment.get_seeding_info_as_str()}”.

save(directory: str) → None[source]#
create_experiment_world(override_experiment_name: str | None = None, logger_run_id: str | None = None, raise_error_on_dirname_collision: bool = True, reset_collectors: bool = True) → World[source]#

Creates the world object for the experiment.

The world object contains all relevant instances for the experiment, such as environments, policy, collectors, etc. This method is the main entrypoint for users who don’t want to use run directly. A common use case is that some configuration or custom logic should happen before the training loop starts, but one still wants to use the convenience of high-level interfaces for setting up the experiment.

Parameters:

  • override_experiment_name – pass to override the experiment name in the resulting World. Affects the name of the persistence directory and logger configuration. If None, the experiment’s name will be used. The name may contain path separators (i.e. os.path.sep, as used by os.path.join), in which case a nested directory structure will be created.

  • logger_run_id – Run identifier to use for logger initialization/resumption.

  • raise_error_on_dirname_collision – whether to raise an error on collisions when creating the persistence directory. Only takes effect if persistence is enabled. Set to False e.g., when continuing a previously executed experiment with the same persistence_base_dir and name.

  • reset_collectors – whether to reset the collectors before training starts. Setting to False can be useful when continuing training from a previous run with restored collectors, or for adding custom logic before training starts.

run(run_name: str | None = None, logger_run_id: str | None = None, raise_error_on_dirname_collision: bool = True, **kwargs: dict[str, Any]) → ExperimentResult[source]#

Run the experiment and return the results.

Parameters:

  • run_name – Defines a name for this run of the experiment, which determines the subdirectory (within the persistence base directory) where all results will be saved. If None, the experiment’s name will be used. The name may contain path separators (i.e. os.path.sep, as used by os.path.join), in which case a nested directory structure will be created.

  • logger_run_id – Run identifier to use for logger initialization/resumption (applies when using wandb, in particular).

  • raise_error_on_dirname_collision – set to False e.g., when continuing a previously executed experiment with the same name.

  • kwargs – for backwards compatibility with old parameter names only

Returns:

class ExperimentCollection(experiments: list[Experiment])[source]#

Bases: object

Shallow wrapper around a list of experiments providing a simple interface for running them with a launcher.

run(launcher: ExpLauncher | RegisteredExpLauncher) → list[InfoStats | None][source]#
class ExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ABC

A helper class (following the builder pattern) for creating experiments.

It contains a lot of defaults for the setup which can be adjusted using the various with_ methods. For example, the default optimizer is Adam, but can be adjusted using with_optim_factory(). Moreover, for simply configuring the default optimizer instead of using a different one, one can use with_optim_factory_default().

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

copy() → Self[source]#
property experiment_config: ExperimentConfig#
property sampling_config: SamplingConfig#
with_logger_factory(logger_factory: LoggerFactory) → Self[source]#

Allows to customize the logger factory to use.

If this method is not called, the default logger factory LoggerFactoryDefault will be used.

Parameters:

logger_factory – the factory to use

Returns:

the builder

with_policy_wrapper_factory(policy_wrapper_factory: PolicyWrapperFactory) → Self[source]#

Allows to define a wrapper around the policy that is created, extending the original policy.

Parameters:

policy_wrapper_factory – the factory for the wrapper

Returns:

the builder

with_optim_factory(optim_factory: OptimizerFactory) → Self[source]#

Allows to customize the gradient-based optimizer to use.

By default, OptimizerFactoryAdam will be used with default parameters.

Parameters:

optim_factory – the optimizer factory

Returns:

the builder

with_optim_factory_default(betas: tuple[float, float] = (0.9, 0.999), eps: float = 1e-08, weight_decay: float = 0) → Self[source]#

Configures the use of the default optimizer, Adam, with the given parameters.

Parameters:

  • betas – coefficients used for computing running averages of gradient and its square

  • eps – term added to the denominator to improve numerical stability

  • weight_decay – weight decay (L2 penalty)

Returns:

the builder

with_epoch_train_callback(callback: EpochTrainCallback) → Self[source]#

Allows to define a callback function which is called at the beginning of every epoch during training.

Parameters:

callback – the callback

Returns:

the builder

with_epoch_test_callback(callback: EpochTestCallback) → Self[source]#

Allows to define a callback function which is called at the beginning of testing in each epoch.

Parameters:

callback – the callback

Returns:

the builder

with_epoch_stop_callback(callback: EpochStopCallback) → Self[source]#

Allows to define a callback that decides whether training shall stop early.

The callback receives the undiscounted returns of the testing result.

Parameters:

callback – the callback

Returns:

the builder

with_name(name: str) → Self[source]#

Sets the name of the experiment.

Parameters:

name – the name to use for this experiment, which, when the experiment is run, will determine the storage sub-folder by default

Returns:

the builder

build() → Experiment[source]#

Creates the experiment based on the options specified via this builder.

Returns:

the experiment

build_seeded_collection(num_experiments: int) → ExperimentCollection[source]#

Creates a collection of experiments with non-overlapping random seeds, starting from the configured seed.

Useful for performing statistically meaningful evaluations of an algorithm’s performance. The rliable recommendation is to use at least 5 experiments for computing quantities such as the interquantile mean and performance profiles. See the usage in example scripts like examples/mujoco/mujoco_ppo_hl_multi.py.

Each experiment in the collection will have a unique name created from the original experiment name and the seeds used.

class RandomActionExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

class PGExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_pg_params(params: PGParams) → Self[source]#
class A2CExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinSingleCriticCanUseActorFactory

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_a2c_params(params: A2CParams) → Self[source]#
class PPOExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinSingleCriticCanUseActorFactory

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_ppo_params(params: PPOParams) → Self[source]#
class NPGExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinSingleCriticCanUseActorFactory

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_npg_params(params: NPGParams) → Self[source]#
class TRPOExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinSingleCriticCanUseActorFactory

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_trpo_params(params: TRPOParams) → Self[source]#
class DQNExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_dqn_params(params: DQNParams) → Self[source]#
with_model_factory(module_factory: IntermediateModuleFactory) → Self[source]#
Parameters:

module_factory – factory for a module which maps environment observations to a vector of Q-values (one for each action)

Returns:

the builder

with_model_factory_default(hidden_sizes: ~collections.abc.Sequence[int], hidden_activation: type[~torch.nn.modules.module.Module] = <class 'torch.nn.modules.activation.ReLU'>) → Self[source]#

Allows to configure the default factory for the model of the Q function, which maps environment observations to a vector of Q-values (one for each action). The default model is a multi-layer perceptron.

Parameters:

  • hidden_sizes – the sequence of dimensions used for hidden layers

  • hidden_activation – the activation function to use for hidden layers (not used for the output layer)

Returns:

the builder

class IQNExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_iqn_params(params: IQNParams) → Self[source]#
with_preprocess_network_factory(module_factory: IntermediateModuleFactory) → Self[source]#
class DDPGExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder, _BuilderMixinActorFactory_ContinuousDeterministic, _BuilderMixinSingleCriticCanUseActorFactory

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_ddpg_params(params: DDPGParams) → Self[source]#
class REDQExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinCriticEnsembleFactory

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_redq_params(params: REDQParams) → Self[source]#
class SACExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinDualCriticFactory

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_sac_params(params: SACParams) → Self[source]#
class DiscreteSACExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder, _BuilderMixinActorFactory, _BuilderMixinDualCriticFactory

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_sac_params(params: DiscreteSACParams) → Self[source]#
class TD3ExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, sampling_config: SamplingConfig | None = None)[source]#

Bases: ExperimentBuilder, _BuilderMixinActorFactory_ContinuousDeterministic, _BuilderMixinDualCriticFactory

Parameters:

  • env_factory – controls how environments are to be created.

  • experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

  • sampling_config – the sampling configuration to use. If None, will use the default values of SamplingConfig.

with_td3_params(params: TD3Params) → Self[source]#