ADMPO: Any-step Dynamics Model for Policy Optimization

This is the code for the paper "Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning" in ICLR 2025.

Requirements

To install all the required dependencies:

1. Install MuJoCo engine, which can be downloaded from here.
2. Install Python packages listed in requirements.txt using pip install -r requirements.txt. You should specify the version of mujoco-py in requirements.txt depending on the version of MuJoCo engine you have installed.
3. Manually download and install d4rl package from here.
4. Manually download and install neorl package from here.

Run an experiment

Online Setting

python main4online.py --env-name [Env name] 

The config files act as defaults for a task. They are all located in config. --env-name refers to the config files in config/ including Hopper-v3, Walker2d-v3, AntTruncatedObs-v3, and HumanoidTruncatedObs-v3. All results will be stored in the result folder.

For example, run ADMPO-ON on Hopper:

python main4online.py --env-name Hopper-v3

Offline Setting

python main4offline.py --env [Env] --env-name [Env name] 

The config files act as defaults for a task. They are all located in config. --env refers to the benchmark, D4RL or NeoRL. --env-name refers to the config files in config/. All results will be stored in the result folder.

For example, run ADMPO-OFF on hopper-medium-v2 dataset of D4RL benchmark:

python main4offline.py --env d4rl --env-name hopper-medium-v2

Citation

If you find this repository useful for your research, please cite:

@inproceedings{
    admpo,
    author       = {Haoxin Lin and
                  Yu{-}Yan Xu and
                  Yihao Sun and
                  Zhilong Zhang and
                  Yi{-}Chen Li and
                  Chengxing Jia and
                  Junyin Ye and
                  Jiaji Zhang and
                  Yang Yu},
    title        = {Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning},
    booktitle    = {The 13th International Conference on Learning Representations (ICLR'25)},
    year         = {2025},
    address      = {Singapore}
}