Dimensionality Reduction and Prioritized Exploration for Policy Search

by Memmel M., Liu P., Tateo D., Peters J.

[Abstract] Black-box policy optimization is a class of reinforcement learning algorithms that explores and updates the policies at the parameter level. This class of algorithms is widely applied in robotics with movement primitives or non-differentiable policies. Furthermore, these approaches are particularly relevant where exploration at the action level could cause actuator damage or other safety issues. However, Black-box optimization does not scale well with the increasing dimensionality of the policy, leading to high demand for samples, which are expensive to obtain in real-world systems. In many practical applications, policy parameters do not contribute equally to the return. Identifying the most relevant parameters allows to narrow down the exploration and speed up the learning. Furthermore, updating only the effective parameters requires fewer samples, improving the scalability of the method. We present a novel method to prioritize the exploration of effective parameters and cope with full covariance matrix updates. Our algorithm learns faster than recent approaches and requires fewer samples to achieve state-of-the-art results. To select the effective parameters, we consider both the Pearson correlation coefficient and the Mutual Information. We showcase the capabilities of our approach on the Relative Entropy Policy Search algorithm in several simulated environments, including robotics simulations.

Getting Started

Code tested with python==3.7. We use MushroomRL-v.1.7.0, Benchmark-v2, and experiment_launcher. Required packages must be installed via pip3 install -r requirements.txt.

An example to setup the repository using Anaconda:

conda create --name drcreps python=3.7
git pull https://github.com/memmelma/DR-CREPS.git
cd DR-CREPS
pip install -r requirements.txt

Reproduce Results and Launch Custom Experiments

To reproduce our results simply execute python run.py which contains all experiments from our paper. Note that if you do not run on a cluster that supports SLURM or Joblib you have to set local=True for successful execution. You can run custom experiments similar to the given examples.

Launch Single Experiment

To run a single (local) experiment directly execute experiment_config.py and specify the parameters via the command line or directly in the file by overriding the default parameter dictionary. The script corresponding to the given algorithm and environment setup will then be called from the experiments/ directory. This mode does not support SLURM or Joblib.

Set Up Your Environment/Experiments

We provide implementations for the following environments:

• lqr
• ship_steering
• air_hockey
• ball_stopping
• optimization_function: e.g. Himmelblau, Rosenbrock

To get an idea of how to set up more detailed experiments please have a look at experiments/ENVIRONMENT/ where you can find scripts for the following algorithms:

• el_es.py: natural evolution strategies (ES, NES)
• el_grad.py: gradient-based methods (REINFORCE, TRPO, PPO)
• el_optim.py: classic optimizers (Nelder-Mead, L-BFGS-B)
• el_bbo.py: policy search algorithms (RWR, REPS, CREPS, CEM, MORE, DR-REPS, DR-CREPS, PRO)

Mutual Information Estimation

We provide an ablation study of different Mutual Information estimators in mutual_information. You can reproduce the experiments on our toy example that we introduce in the appendix. Specify your experiments and plot the results by executing mi.py.

Reuse Our Code!

Our implementation builds on MushroomRL. It also integrates nicely with it, i.e., all of our algorithms and environments either inherit directly from MushroomRL modules or implement the same intuitive interfaces.

Bibtex

@inproceedings{AISTATS_2022_Dimensionality_Reduction
  author    = "Memmel, M. and Liu, P. and Tateo, D. and Peters, J.",
  year      = "2022",
  title     = "Dimensionality Reduction and Prioritized Exploration for Policy Search",
  booktitle = "Proceedings of The International Conference on Artificial Intelligence and Statistics"
}