ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers

Vrushabh Zinage¹ · Narek Harutyunyan² · Eric Verheyden¹ · Fred Hadaegh¹ · Soon-Jo Chung¹

¹California Institute of Technology ²Brown University

Overview

This repository contains the implementation of ContractionPPO, a novel reinforcement learning framework for robust legged robot locomotion that provides certified stability guarantees. The method augments Proximal Policy Optimization (PPO) with state-dependent contraction metric layers to enable robust control of quadruped robots under external perturbations.

Abstract

Legged locomotion in unstructured environments demands not only high-performance control policies but also formal guarantees to ensure robustness under perturbations. ContractionPPO uses a Lipschitz neural network to parameterize the contraction metric and jointly trains the policy and contraction metric to maximize performance and stability.

Key Contributions:

• Enables robust control of quadruped robots under external perturbations
• Provides certifiable incremental exponential stability
• Demonstrates generalization from simulation to real-world deployment
• Outperformed baseline methods in wind speed experiments with zero failure ratio

Installation

• Install Isaac Lab by following the installation guide. We recommend using the conda installation as it simplifies calling Python scripts from the terminal.

• Clone or copy this project/repository separately from the Isaac Lab installation (i.e. outside the IsaacLab directory):

• Using a python interpreter that has Isaac Lab installed, install the ContractionPPO library in editable mode using:

  # use 'PATH_TO_isaaclab.sh|bat -p' instead of 'python' if Isaac Lab is not installed in Python venv or conda
  python -m pip install -e source/ContractionPPO
  

• Verify that the extension is correctly installed by:

  • Listing the available tasks:

    Available ContractionPPO tasks:

    • ContractionPPO-Handstand-v0: Quadruped handstand balancing on boat deck
    • Template-Contractionppo-v0: Template ContractionPPO environment

    # use 'FULL_PATH_TO_isaaclab.sh|bat -p' instead of 'python' if Isaac Lab is not installed in Python venv or conda
    python scripts/list_envs.py

  • Running ContractionPPO training:

    # use 'FULL_PATH_TO_isaaclab.sh|bat -p' instead of 'python' if Isaac Lab is not installed in Python venv or conda
    python scripts/contraction_ppo/train.py --task=ContractionPPO-Handstand-v0

  • Running with RSL-RL (baseline comparison):

    # use 'FULL_PATH_TO_isaaclab.sh|bat -p' instead of 'python' if Isaac Lab is not installed in Python venv or conda
    python scripts/rsl_rl/train.py --task=ContractionPPO-Handstand-v0

  • Running a task with dummy agents:

    These include dummy agents that output zero or random agents. They are useful to ensure that the environments are configured correctly.

    • Zero-action agent

      # use 'FULL_PATH_TO_isaaclab.sh|bat -p' instead of 'python' if Isaac Lab is not installed in Python venv or conda
      python scripts/zero_agent.py --task=ContractionPPO-Handstand-v0

    • Random-action agent

      # use 'FULL_PATH_TO_isaaclab.sh|bat -p' instead of 'python' if Isaac Lab is not installed in Python venv or conda
      python scripts/random_agent.py --task=ContractionPPO-Handstand-v0

  • Playing/Evaluating trained policies:

    # use 'FULL_PATH_TO_isaaclab.sh|bat -p' instead of 'python' if Isaac Lab is not installed in Python venv or conda
    python scripts/contraction_ppo/play.py --task=ContractionPPO-Handstand-Play-v0 --checkpoint=/path/to/checkpoint.pt

Set up IDE (Optional)

To setup the IDE, please follow these instructions:

• Run VSCode Tasks, by pressing Ctrl+Shift+P, selecting Tasks: Run Task and running the setup_python_env in the drop down menu. When running this task, you will be prompted to add the absolute path to your Isaac Sim installation.

If everything executes correctly, it should create a file .python.env in the .vscode directory. The file contains the python paths to all the extensions provided by Isaac Sim and Omniverse. This helps in indexing all the python modules for intelligent suggestions while writing code.

Setup as Omniverse Extension (Optional)

We provide an example UI extension that will load upon enabling your ContractionPPO extension defined in source/ContractionPPO/ContractionPPO/ui_extension_example.py.

To enable your extension, follow these steps:

1. Add the search path of this project/repository to the extension manager:

   • Navigate to the extension manager using Window -> Extensions.
   • Click on the Hamburger Icon, then go to Settings.
   • In the Extension Search Paths, enter the absolute path to the source directory of this project/repository.
   • If not already present, in the Extension Search Paths, enter the path that leads to Isaac Lab's extension directory directory (IsaacLab/source)
   • Click on the Hamburger Icon, then click Refresh.

2. Search and enable your ContractionPPO extension:

   • Find your ContractionPPO extension under the Third Party category.
   • Toggle it to enable your extension.

Code formatting

We have a pre-commit template to automatically format your code. To install pre-commit:

pip install pre-commit

Then you can run pre-commit with:

pre-commit run --all-files

Citation

If you use ContractionPPO in your research, please cite our paper:

@article{zinage2025contractionppo,
  title={ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers},
  author={Zinage, Vrushabh and Harutyunyan, Narek and Verheyden, Eric and Hadaegh, Fred and Chung, Soon-Jo},
  journal={IEEE Robotics and Automation Letters (RAL)},
  year={2025}
}

Paper Website: https://contractionppo.github.io/

Troubleshooting

Pylance Missing Indexing of Extensions

In some VsCode versions, the indexing of part of the extensions is missing. In this case, add the path to your ContractionPPO extension in .vscode/settings.json under the key "python.analysis.extraPaths".

{
    "python.analysis.extraPaths": [
        "<path-to-ContractionPPO-repo>/source/ContractionPPO"
    ]
}

Pylance Crash

If you encounter a crash in pylance, it is probable that too many files are indexed and you run out of memory. A possible solution is to exclude some of omniverse packages that are not used in your project. To do so, modify .vscode/settings.json and comment out packages under the key "python.analysis.extraPaths" Some examples of packages that can likely be excluded are:

"<path-to-isaac-sim>/extscache/omni.anim.*"         // Animation packages
"<path-to-isaac-sim>/extscache/omni.kit.*"          // Kit UI tools
"<path-to-isaac-sim>/extscache/omni.graph.*"        // Graph UI tools
"<path-to-isaac-sim>/extscache/omni.services.*"     // Services tools
...