experiment

Source code: tianshou/highlevel/experiment.py

---

The experiment module provides high-level interfaces for setting up and running reinforcement learning experiments.

The main entry points are:

• ExperimentConfig: a dataclass for configuring the experiment. The configuration is different from RL specific configuration (such as policy and trainer parameters) and only pertains to configuration that is common to all experiments.

• Experiment: represents a reinforcement learning experiment. It is composed of configuration and factory objects, is lightweight and serializable. An instance of Experiment is usually saved as a pickle file after an experiment is executed.

• ExperimentBuilder: a helper class for creating experiments. It contains a lot of defaults and allows for easy customization of the experiment setup.

• ExperimentCollection: a shallow wrapper around a list of experiments providing a simple interface for running them with a launcher. Useful for running multiple experiments in parallel, in particular, for the important case of running experiments that only differ in their random seeds.

Various implementations of the ExperimentBuilder are provided for each of the algorithms supported by Tianshou.

class ExperimentConfig(seed: int = 42, device: str | device = 'cpu', policy_restore_directory: str | None = None, train: bool = True, watch: bool = True, watch_num_episodes: int = 10, watch_render: float = 0.0, persistence_base_dir: str = 'log', persistence_enabled: bool = True, log_file_enabled: bool = True, policy_persistence_mode: Mode = Mode.POLICY)

Bases: object

Generic config for setting up the experiment, not RL or training specific.

seed: int = 42

The random seed with which to initialize random number generators.

device: str | device = 'cpu'

The torch device to use

policy_restore_directory: str | None = None

Directory from which to load the policy neural network parameters (persistence directory of a previous run)

train: bool = True

Whether to perform training

watch: bool = True

Whether to watch agent performance (after training)

watch_num_episodes: int = 10

Number of episodes for which to watch performance (if watch is enabled)

watch_render: float = 0.0

Milliseconds between rendered frames when watching agent performance (if watch is enabled)

persistence_base_dir: str = 'log'

Base directory in which experiment data is to be stored. Every experiment run will create a subdirectory in this directory based on the run’s experiment name

persistence_enabled: bool = True

Whether persistence is enabled, allowing files to be stored

log_file_enabled: bool = True

Whether to write to a log file; has no effect if persistence_enabled is False. Disable this if you have externally configured log file generation.

policy_persistence_mode: Mode = 'policy'

Controls the way in which the policy is persisted

class ExperimentResult(world: World, trainer_result: InfoStats | None)

Bases: object

Contains the results of an experiment.

world: World

The World contains all the essential instances of the experiment. Can also be created via Experiment.create_experiment_world for more custom setups, see docstring there.

Note: it is typically not serializable, so it is not stored in the experiment pickle, and shouldn’t be sent across processes, meaning also that ExperimentResult itself is typically not serializable.

trainer_result: InfoStats | None

dataclass of results as returned by the trainer (if any)

class Experiment(config: ExperimentConfig, env_factory: EnvFactory, algorithm_factory: AlgorithmFactory, training_config: TrainingConfig, name: str, logger_factory: LoggerFactory | None = None)

Bases: ToStringMixin

Represents a reinforcement learning experiment.

An experiment is composed only of configuration and factory objects, which themselves should be designed to contain only configuration. Therefore, experiments can easily be stored/pickled and later restored without any problems.

The main entry points are:

1. run(): runs the experiment and returns the results

2. create_experiment_world(): creates the world object for the experiment, which contains all relevant instances.

   Useful for setting up the experiment and running it in a more custom way.

The methods save() and from_directory() can be used to store and restore experiments.

LOG_FILENAME = 'log.txt'

EXPERIMENT_PICKLE_FILENAME = 'experiment.pkl'

classmethod from_directory(directory: str, restore_policy: bool = True) → Experiment

Restores an experiment from a previously stored pickle.

Parameters:

• directory – persistence directory of a previous run, in which a pickled experiment is found

• restore_policy – whether the experiment shall be configured to restore the policy that was persisted in the given directory

static seeding_info_str_static(seed: int) → str

Static method variant of get_seeding_info_as_str, which can be used without an Experiment instance.

get_seeding_info_as_str() → str

Returns information on the seeds used in the experiment as a string.

This can be useful for creating unique experiment names based on seeds, e.g. A typical example is to do experiment.name = f”{experiment.name}_{experiment.get_seeding_info_as_str()}”.

save(directory: str) → None

static persistence_dir_static(persistence_base_dir: str, experiment_name: str, seed: int | None = None) → str

Static method for constructing the persistence directory for an experiment from the base persistence directory and the experiment name. Useful for contexts where one wants access to the persistence directory without having access to the corresponding Experiment instance.

Parameters:

• persistence_base_dir – base persistence directory

• experiment_name – name of the experiment

• seed – optional seed. Experiments are saved within a subdirectory named after the seed, but it is often sufficient to know the base directory without the seed subdirectory in user code (for example, for restoring logs or performing rliable evaluations)

create_experiment_world(override_experiment_name: str | None = None, logger_run_id: str | None = None, raise_error_on_dirname_collision: bool = True, reset_collectors: bool = True) → World

Creates the world object for the experiment.

The world object contains all relevant instances for the experiment, such as environments, policy, collectors, etc. This method is the main entrypoint for users who don’t want to use run directly. A common use case is that some configuration or custom logic should happen before the training loop starts, but one still wants to use the convenience of high-level interfaces for setting up the experiment.

Parameters:

• override_experiment_name – pass to override the experiment name in the resulting World. Affects the name of the persistence directory and logger configuration. If None, the experiment’s name will be used. The name may contain path separators (i.e. os.path.sep, as used by os.path.join), in which case a nested directory structure will be created.

• logger_run_id – Run identifier to use for logger initialization/resumption.

• raise_error_on_dirname_collision – whether to raise an error on collisions when creating the persistence directory. Only takes effect if persistence is enabled. Set to False e.g., when continuing a previously executed experiment with the same persistence_base_dir and name.

• reset_collectors – whether to reset the collectors before training starts. Setting to False can be useful when continuing training from a previous run with restored collectors, or for adding custom logic before training starts.

run(run_name: str | None = None, logger_run_id: str | None = None, raise_error_on_dirname_collision: bool = True) → ExperimentResult

Run the experiment and return the results.

Parameters:

• run_name – Defines a name for this run of the experiment, which determines the subdirectory (within the persistence base directory) where all results will be saved. If None, the experiment’s name will be used. The name may contain path separators (i.e. os.path.sep, as used by os.path.join), in which case a nested directory structure will be created.

• logger_run_id – Run identifier to use for logger initialization/resumption (applies when using wandb, in particular).

• raise_error_on_dirname_collision – set to False e.g., when continuing a previously executed experiment with the same name.

Returns:

class ExperimentCollection(experiments: list[Experiment])

Bases: object

Shallow wrapper around a list of experiments providing a simple interface for running them with a launcher.

run(launcher: ExpLauncher | RegisteredExpLauncher | str = RegisteredExpLauncher.SEQUENTIAL) → list[InfoStats | None]

class ExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: TTrainingConfig | None = None)

Bases: ABC, Generic[TTrainingConfig]

A helper class (following the builder pattern) for creating experiments.

It contains a lot of defaults for the setup which can be adjusted using the various with_ methods. For example, the default optimizer is Adam, but can be adjusted using with_optim_factory(). Moreover, for simply configuring the default optimizer instead of using a different one, one can use with_optim_factory_default().

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

copy() → Self

property experiment_config: ExperimentConfig

property training_config: TrainingConfig

with_logger_factory(logger_factory: LoggerFactory) → Self

Allows to customize the logger factory to use.

If this method is not called, the default logger factory LoggerFactoryDefault will be used.

Parameters:

logger_factory – the factory to use

Returns:

the builder

with_algorithm_wrapper_factory(algorithm_wrapper_factory: AlgorithmWrapperFactory) → Self

Allows to define a wrapper around the algorithm that is created, extending the original algorithm.

Parameters:

algorithm_wrapper_factory – the factory for the wrapper

Returns:

the builder

with_optim_default(optim_factory: OptimizerFactoryFactory) → Self

Allows to customize the default optimizer to use.

The default optimizer applies when optimizer factory factories are set to None in algorithm parameter objects.

By default, OptimizerFactoryFactoryAdam will be used with default parameters.

Parameters:

optim_factory – the optimizer factory

Returns:

the builder

with_epoch_train_callback(callback: EpochTrainCallback) → Self

Allows to define a callback function which is called at the beginning of every epoch during training.

Parameters:

callback – the callback

Returns:

the builder

with_epoch_test_callback(callback: EpochTestCallback) → Self

Allows to define a callback function which is called at the beginning of testing in each epoch.

Parameters:

callback – the callback

Returns:

the builder

with_epoch_stop_callback(callback: EpochStopCallback) → Self

Allows to define a callback that decides whether training shall stop early.

The callback receives the undiscounted returns of the testing result.

Parameters:

callback – the callback

Returns:

the builder

with_name(name: str) → Self

Sets the name of the experiment.

Parameters:

name – the name to use for this experiment, which, when the experiment is run, will determine the storage sub-folder by default

Returns:

the builder

with_collector_factory(collector_factory: CollectorFactory) → Self

Allows customizing the collector factory to use.

Parameters:

collector_factory – the factory to use for the creation of collectors

Returns:

the builder

build() → Experiment

Creates the experiment based on the options specified via this builder.

Returns:

the experiment

build_seeded_collection(num_experiments: int) → ExperimentCollection

Creates a collection of experiments with non-overlapping random seeds, starting from the configured seed.

Useful for performing statistically meaningful evaluations of an algorithm’s performance. The rliable recommendation is to use at least 5 experiments for computing quantities such as the interquantile mean and performance profiles. See the usage in example scripts like examples/mujoco/mujoco_ppo_hl_multi.py.

Each experiment in the collection will have a unique name created from the original experiment name and the seeds used.

build_and_run(num_experiments: int = 1, launcher: ExpLauncher | RegisteredExpLauncher | str = RegisteredExpLauncher.SEQUENTIAL, perform_rliable_analysis: bool = True) → list[InfoStats | None]

Build and run experiments. With multiple experiments, the seeds will be non-overlapping and the parallelism is controlled by the launcher.

Parameters:

• num_experiments – the number of experiments to create and run

• launcher – the launcher (or the corresponding enum value) to use for running the experiments

• perform_rliable_analysis – whether to perform rliable analysis on the results (only applicable if num_experiments > 1). This will show plots and store them in the persistence directory.

Returns:

list of results, one per experiment

class OnPolicyExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OnPolicyTrainingConfig | None = None)

Bases: ExperimentBuilder[OnPolicyTrainingConfig], ABC

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

class OffPolicyExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OffPolicyTrainingConfig | None = None)

Bases: ExperimentBuilder[OffPolicyTrainingConfig], ABC

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

class ReinforceExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OnPolicyTrainingConfig | None = None)

Bases: OnPolicyExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_reinforce_params(params: ReinforceParams) → Self

class A2CExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OnPolicyTrainingConfig | None = None)

Bases: OnPolicyExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinSingleCriticCanUseActorFactory

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_a2c_params(params: A2CParams) → Self

class PPOExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OnPolicyTrainingConfig | None = None)

Bases: OnPolicyExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinSingleCriticCanUseActorFactory

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_ppo_params(params: PPOParams) → Self

class NPGExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OnPolicyTrainingConfig | None = None)

Bases: OnPolicyExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinSingleCriticCanUseActorFactory

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_npg_params(params: NPGParams) → Self

class TRPOExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OnPolicyTrainingConfig | None = None)

Bases: OnPolicyExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinSingleCriticCanUseActorFactory

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_trpo_params(params: TRPOParams) → Self

class DQNExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OffPolicyTrainingConfig | None = None)

Bases: OffPolicyExperimentBuilder

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_dqn_params(params: DQNParams) → Self

with_model_factory(module_factory: IntermediateModuleFactory) → Self

Parameters:

module_factory – factory for a module which maps environment observations to a vector of Q-values (one for each action)

Returns:

the builder

with_model_factory_default(hidden_sizes: ~collections.abc.Sequence[int], hidden_activation: type[~torch.nn.modules.module.Module] = <class 'torch.nn.modules.activation.ReLU'>) → Self

Allows to configure the default factory for the model of the Q function, which maps environment observations to a vector of Q-values (one for each action). The default model is a multi-layer perceptron.

Parameters:

• hidden_sizes – the sequence of dimensions used for hidden layers

• hidden_activation – the activation function to use for hidden layers (not used for the output layer)

Returns:

the builder

class IQNExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OffPolicyTrainingConfig | None = None)

Bases: OffPolicyExperimentBuilder

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_iqn_params(params: IQNParams) → Self

with_preprocess_network_factory(module_factory: IntermediateModuleFactory) → Self

class DDPGExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OffPolicyTrainingConfig | None = None)

Bases: OffPolicyExperimentBuilder, _BuilderMixinActorFactory_ContinuousDeterministic, _BuilderMixinSingleCriticCanUseActorFactory

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_ddpg_params(params: DDPGParams) → Self

class REDQExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OffPolicyTrainingConfig | None = None)

Bases: OffPolicyExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinCriticEnsembleFactory

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_redq_params(params: REDQParams) → Self

class SACExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OffPolicyTrainingConfig | None = None)

Bases: OffPolicyExperimentBuilder, _BuilderMixinActorFactory_ContinuousGaussian, _BuilderMixinDualCriticFactory

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_sac_params(params: SACParams) → Self

class DiscreteSACExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OffPolicyTrainingConfig | None = None)

Bases: OffPolicyExperimentBuilder, _BuilderMixinActorFactory_DiscreteOnly, _BuilderMixinDualCriticFactory

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_sac_params(params: DiscreteSACParams) → Self

class TD3ExperimentBuilder(env_factory: EnvFactory, experiment_config: ExperimentConfig | None = None, training_config: OffPolicyTrainingConfig | None = None)

Bases: OffPolicyExperimentBuilder, _BuilderMixinActorFactory_ContinuousDeterministic, _BuilderMixinDualCriticFactory

Parameters:

• env_factory – controls how environments are to be created.

• experiment_config – the configuration for the experiment. If None, will use the default values of ExperimentConfig.

• training_config – the training configuration to use. If None, use default values (not recommended).

with_td3_params(params: TD3Params) → Self