Revolve RL: Project Overview

This repository contains code for evolving reward functions using genetic programming (GP). Below are the instructions and descriptions for key scripts and directories in this project.

Getting Started

Running the GP Evolution

To execute the GP evolution process, use the following command:

python gprun.py

Specify the environment to run using env (which would be a module from languages/) and env_language (eg: ant.AntLanguage)

Simulating Reward Functions

To run simulations with the best reward function or the default reward function, use:

python simulate_runs.py

Converting TensorBoard Event Files to CSV

To convert TensorBoard event files to CSV format, use:

python tb_to_csv.py

Specify the tensorboard log directory using tb_dir

Directory Structure

wrapper/

Contains custom_monitor.py, which is used to monitor logs during training. Use fitness_keyword to specify which key from info (dict) to monitor when env.step() is called.

envs/

Includes the environment code for the following environments:

• MuJoCo Environments: Ant, Half Cheetah, Hopper
• Classic Control Environments: Bipedal Walker, Lunar Lander
• Atari Environments: Private Eye, Video Pinball
• Toy Environments: Taxi, Black Jack

recorders/

Contains stable-baselines callbacks:

• mujocorecorder.py: Common for Ant, Half Cheetah, and Hopper environments.
• atarirecorder.py: Common for all Atari environments.

job_scripts/

Contains the job scripts submitted to the compute canada cluster

Additional Information

requirements.txt contains the the pacakges built using the wheels provided by Compute Canada, so it's ideal to use it on the cluster. Also some of the commented code is for using ray instead of pathos for multiprocessing. ProcessPool from pathos.pool tackles the pickling errors due non-serializable objects used by deap.

There is also an issue with pip install mujoco in Compute Canada, and it is necessary to set the MUJOCO_PATH and MUJOCO_PLUGINS_PATH (can be empty) environment variables by the mujoco wheels from (https://github.com/google-deepmind/mujoco/releases)

export MUJOCO_PATH=/path/to/mujoco
export MUJOCO_PLUGIN_PATH=~/mujoco_plugins

Also MuJoCo requires libglfw3, otherwise it'll throw ImportError: Failed to load GLFW3 shared library. But since Compute Canada doesn't have the libglw3 module, it has to be built from source (as there's no root access too). Run

wget https://github.com/glfw/glfw/archive/refs/tags/3.3.8.tar.gz
tar -xzvf 3.3.8.tar.gz
cd glfw-3.3.8

Now create the build directory

mkdir build
cd build

Configure the build

cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/local -DBUILD_SHARED_LIBS=ON

Now build and install

make
make install

Set the environment variables LD_LIBRARY_PATH and PYGLFW_LIBRARY

export LD_LIBRARY_PATH=~/path/to/lib64:$LD_LIBRARY_PATH
export PYGLFW_LIBRARY=~/path/to//lib64/libglfw.so

Now do pip install mujoco and you are good to go!!

Link to the tensorboard logs and other run outputs:

https://drive.google.com/drive/folders/1vTD2FTKLd9WT-xS3dqjvHxnlBO5GJpML?usp=sharing