HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation

CVPR 2025

Project Status

• ✅Release MotionVid dataset.
• ✅Release train and inference code.

---

Environment Setup

To set up the environment, run the following commands:

conda create -n humandreamer python=3.10
conda activate humandreamer
pip install -r requirements.txt

Download Model and Dataset

1. Download the MotionVid dataset from Hugging Face.
2. Download the model checkpoint from Hugging Face.
3. Place the model checkpoint files in your specified exp_root_path.

Configuration Adjustments

Make the following changes to the configuration files:

1. Modify batch_size_per_gpu in giga-train/gt_projects/motiondit/configs/YOUR_CONFIG.py to match your GPU memory.
2. Modify exp_root_path in the same file to your desired path.
3. Modify dataset root path in giga-train/gt_projects/motiondit/configs/dataset_path_list.py to point to your dataset location.
4. Modify tmr_weight_path in giga-train/gt_projects/motiondit/motiondit/dit_trainer.py.
5. Modify clip_path in giga-train/gt_projects/motiondit/motiondit/dit_trainer.py and giga-train/gt_projects/motiondit/motiondit/dit_tester.py.
6. Modify path in giga-train/gt_projects/motiondit/scripts/paths.py to your project path.

Training MotionDiT

To train the model, execute:

cd giga-train/gt_projects/motiondit/scripts
python launch.py --stages train

Inference MotionDiT

To test the model, execute:

python launch.py --stages test

MotionVid Dataset

MotionVid is a dataset containing 1.2M text-pose-video pairs. The videos are sourced from the internet and public datasets, the poses are extracted using DWPose, and the text descriptions are generated using ShareGPT4Video.

Note: The repository does not include video files due to licensing restrictions. If you require the video files, you must download them separately from their respective public sources. Additionally, a small portion of the data (≈9%) cannot be made public due to policies considerations.

We have divided the data from different sources into multiple subsets. Each subset stores basic data information in .pkl format and pose information in .mdb format.

You can download MotionVid dataset here.

Dataset includes the following fields:

• data_index: Index of the sample.
• prompt: Description of the activity performed by the person in the video.
• video_height: Original height of the video.
• video_width: Original width of the video.
• video_length: Length of the video sequence.
• video_path: Name of the video file in the public dataset, which can be used to locate and download the video from its source.
• poses: Human keypoint information.
• poses_scores: Confidence scores for the keypoints.

License

All the data and code within this repo are under CC BY-NC-SA 4.0.

BibTeX

@article{wang2025humandreamer,
  title={HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation}, 
  author={Boyuan Wang and Xiaofeng Wang and Chaojun Ni and Guosheng Zhao and Zhiqin Yang and Zheng Zhu and Muyang Zhang and Yukun Zhou and Xinze Chen and Guan Huang and Lihong Liu and Xingang Wang},
  journal={arXiv preprint arXiv:2503.24026},
  year={2025}
}