KinematicRL

A Sim-to-Real Reinforcement Learning Framework for Social Navigation with Kinodynamic Feasibility

KinematicRL is a sim-to-real reinforcement learning framework for socially compliant robot navigation under kinodynamic constraints. The robot perceives nearby pedestrians with a cluster-based tracking pipeline using only 2D LiDAR, enforces kinodynamic feasibility through a higher-order action space, and learns a deployable policy through imitation learning followed by reinforcement fine-tuning. The framework is designed to mitigate the sim-to-real gap in DRL-based social navigation and generates trajectory with smoothness and social awareness in pedestrian-rich environments.

Features

• Modern Gymnasium simulation. Built on the CrowdNav crowd navigation environment and updated to Gymnasium 1.x with vectorized environment support. The simulator provides a simulated LiDAR sensor and randomizes start and goal positions each episode to improve policy generalization.
• Vectorized maximum-entropy iLQR expert. Extends the iLQR solver from PythonLinearNonlinearControl to a maximum-entropy variant with batch processing that solves multiple environments in parallel. This matches Gymnasium vectorized environments and significantly accelerates expert data collection.
• KinematicRL policy. A gated transformer policy that maps past observations to kinodynamically feasible control commands, trained via behavior cloning (with a divergence objective) and DAgger, then refined with PPO.
• Modular design. Simulation, controllers, network architectures, and training pipelines are separated into independent folders so you can swap robot models, cost functions, or policy backbones with minimal changes.

Results

Left: simulation with simulated LiDAR and randomized start/goal positions. Right: real-world deployment on a differential-drive robot.

This repository ships two ready-to-use configurations. Each pairs a simulation YAML with a robot/iLQR YAML that defines control bounds and expert cost weights. Pass the robot/iLQR file explicitly via --robot_ilqr_config in training and search scripts.

Setting	Train script	Simulation config	Robot/iLQR config	Training config	Robot / sensor	Start / goal
Benchmark (thesis reproduction)	scripts/train_benchmark.sh	configs/sim_benchmark.yaml	configs/robot_ilqr_benchmark.yaml	configs/train_benchmark.yaml	Fast TurtleBot-like robot, unlimited LiDAR (scan_range=20)	Fixed start/goal (fixed_goal: true)
Real-world deployment	scripts/train_realworld.sh	configs/sim_realworld.yaml	configs/robot_ilqr_realworld.yaml	configs/train_realworld.yaml	Slower real-robot limits, limited LiDAR (scan_range=6)	Random start/goal (fixed_goal: false)

Set Sim.fixed_goal in your simulation YAML to control robot placement: true fixes the robot at (0, -4) with goal (0, 4) (benchmark reproduction); false samples random start and goal positions within the simulation area (recommended for real-world deployment).

To use a custom robot, create a new YAML following the same structure as the files above (control_upper_bound, control_lower_bound, and an ilqr block), then pass its path explicitly to train.py or the training scripts.

Evaluation

Configuration	SR	CR	μT	DF	Dist.	vof	ωof	Δa/Δt	κ
Benchmark (sim_benchmark)	0.84	0.15	9.76±2.96	0.03±0.06	0.098±0.055	0.00	0.00	0.16±0.44	0.43±0.69
Real-world (sim_realworld)	0.81	0.12	18.42±4.73	0.027±0.046	0.107±0.053	0.00	0.00	0.103±0.422	0.614±1.316

Metric definitions: SR — success rate; CR — collision rate; μT — average navigation time (s); DF — frequency of invading humans' comfort distances; Dist. — average minimum separating distance between robot and humans (m); v_of — oscillating frequency of linear velocity; ω_of — oscillating frequency of angular velocity; Δa/Δt — acceleration change rate (jerk); κ — average trajectory curvature.

Getting Started

Guide	Description
Installation	Environment setup and dependencies
Training	Configure simulation, tune iLQR weights, and train a policy
Evaluation	Evaluate a trained checkpoint
Custom Development	Extend algorithms, networks, and robot configurations

The training pipeline follows a three-stage workflow: iLQR expert tuning → imitation learning (BC + DAgger) → PPO fine-tuning. Quick start after installation:

# 1. Choose a simulation config, matching robot/iLQR YAML, and training YAML
#    scripts/train_benchmark.sh  -> sim_benchmark.yaml + robot_ilqr_benchmark.yaml + train_benchmark.yaml
#    scripts/train_realworld.sh  -> sim_realworld.yaml + robot_ilqr_realworld.yaml + train_realworld.yaml

# 2. (Optional) Search for optimal iLQR cost weights
bash scripts/ilqr_param_search.sh

# 3. Copy best weights into the robot/iLQR YAML (see docs/training.md)

# 4. Train
bash scripts/train_benchmark.sh    # thesis benchmark
bash scripts/train_realworld.sh    # real-world deployment

# Or invoke train.py directly:
python train.py --logs_dir logs/my_run \
    --env_config configs/sim_realworld.yaml \
    --train_config configs/train_realworld.yaml \
    --policy_config configs/policy.yaml \
    --robot_ilqr_config configs/robot_ilqr_realworld.yaml

# Custom robot: copy an existing robot/iLQR YAML, edit bounds and weights,
# then pass your file path via --robot_ilqr_config.

# 5. Evaluate
bash scripts/eval.sh

# 6. Inspect training curves
python -c "from crowd_nav.helper.plotter import Plotter; Plotter.plot_reward('logs/<experiment>/monitor').figure.savefig('reward.png')"

On-Going

Real-world deployment code is being prepared for release. The following repositories are on the way:

• LiDAR pedestrian detection — cluster-based tracking pipeline that detects and tracks nearby pedestrians from 2D LiDAR scans
• KinematicRL real-world deployment — ROS integration for loading a trained policy and running socially compliant navigation on a differential-drive robot

We will add links to these repositories here once they are ready.

Acknowledgements

This project builds upon:

• CrowdNav — crowd navigation simulation environment
• PythonLinearNonlinearControl — iLQR solver reference implementation
• Stable-Baselines3 — vectorized environments and PPO training framework

Citation

If you find this work useful, please cite:

@article{kinematicrl,
  author={Xu, Zhiming and Yang, Haodong and Liu, Chengju and Chen, Qijun and Yao, Chenpeng},
  journal={IEEE Transactions on Automation Science and Engineering}, 
  title={KinematicRL: A Sim-to-Real Reinforcement Learning Framework for Social Navigation With Kinodynamic Feasibility}, 
  year={2026},
  volume={23},
  pages={11824--11839},
  doi={10.1109/TASE.2026.3703136}
}