Photorealistic Synthetic Crowds Simulation in Indoor environments (PSCS-I): a novel synthetic dataset for realistic simulation of crowd panic and violence behaviors
This repository contains code and pretrained models for the paper entitled "Photorealistic Synthetic Crowds Simulation in Indoor environments (PSCS-I): a novel synthetic dataset for realistic simulation of crowd panic and violence behaviors". PSCS-I is a synthetic datasset generated with the state-of-the-art rendering capabilities of Unreal Engine 5 (UE5) and was specifically designed for crowd behavior analysis tasks. In detail, PSCS-I is the first synthetic dataset to:
- Capture multiple scenes from 26 unique indoor environments with more than 53.8 hours of annotated videos at the frame-level.
- Contain multiple normal (walking and social) and abnormal (mixed running, evacuation, and fighting) crowd behaviors with each accompanied by trigger events (e.g., earthquake).
- Introduce videos captured from Body Worn Cameras (BWC).
- Outperform real-world datasets by mitigating the sim2real gap through the rendering capabilities of UE5 and the utilization of the Enhancing Photorealism Enhancement method.
- Significantly challenge the current state-of-the-art crowd behavior analysis methods in multiclass classification.
The PSCS-I dataset is available for download here. To get the access credentials, fill out the following form, and they will be emailed to you: Request Form
For the creation of PSCS-I, assets were employed from the UE Marketplace/FAB that are depicted in the videos. The dataset was designed for predictive tasks (e.g., image classification and object detection) that solely operate on the original content and produce tags. Based on the FAB EULA, to access the dataset, you have to agree that it will be employed only for such predictive tasks. Below is the related section from the EULA:
"For purposes of this Agreement, "Generative AI Programs" means artificial intelligence, machine learning, deep learning, neural networks, or similar technologies designed to automate the generation of or aid in the creation of new content, including but not limited to audio, visual, or text-based content. Programs do not meet this definition of Generative AI Programs where they, by non-limiting example, (a) solely operate on the original content; (b) generate tags to classify visual input content; or (c) generate instructions to arrange existing content, without creating new content."
Enhancing Photorealism Enhancement was applied only on a subset of the videos included in the PSCS-I dataset that depicted assets that, at the time of creating the dataset, were not tagged as 'noAI'. Additionally, none of the videos were used as training data for Enhancing Photorealism Enhancement since it was pretrained on a dataset from the CARLA simulator. In fact, it was only employed as a post-process filter on the videos.
In this repository, we provide code and pretrained models compatible with the CrimeNet architecture used in the paper. To run the code, first, it is required to download the CrimeNet architecture from the official repository. Specifically, download the ViT.py script and place it inside /CrowdBehavior/.
To train CrimeNet on the PSCS-I dataset, inside the /CrowdBehavior/pscsi.py script, set the following paths on the respective lines:
34: with open("/data/PSCS-I/Annotations/behavior_analysis_annot.json", 'r') as file: #path to the behavior_analysis_annot.json file
35: annotations = json.load(file)
37: with open("/data/PSCS-I/Annotations/environment_annot.json", 'r') as file: #path to the environment_annot.json file
38: env = json.load(file)
42: directories = [
43: '/data/PSCS-I/Videos_sample/',
44: #'/data/CrowdDatasetFinal/Enhanced/KITTI/',
45: #'/data/CrowdDatasetFinal/Enhanced/Cityscapes/'
46: ]
Specifically, for the directory list, more than one folder that contains videos can be defined. This enables the training with the multiple variations of the PSCS-I dataset (e.g., PSCS-I Enhanced and PSCS-I Union used in the paper). After defining the required paths, run the following command to start training:
python train.py
The trained model will be saved in the log directory that will be created in the same location as the /CrowdBehavior/train.py script after the first epoch.
To test and replicate the results of the paper, we provide three pretrained CrimeNet models on PSCS-I Synthetic, PSCS-I Enhanced, and PSCS-I Union variations of the dataset. In detail, the code for testing the models on the Movies Fights dataset is provided. First, download Movies Fights from here and place the normal videos in /CrowdBehavior/dataset/normal and the violent videos in /CrowdBehavior/dataset/abnormal. Then execute the following script:
python evaluate.py
The overall ROC-AUC and PR-AUC metrics will be printed on the CMD after the inference of each batch. The pretrained model used for evaluation can be changed in the following line inside /CrowdBehavior/evaluate.py:
35: adv_model.load_weights('./pretrained/pscsi_union/20250314-202126')
Besides crowd behavior analysis, the PSCS-I dataset can also support other tasks, such as object detection of persons. In detail, the dataset contains a person and a person's head class. These object detection annotations are provided in an (x,y) coordinate format. To generate a YOLO-compatible dataset for training an object detection model, first set the following variables inside /ObjectDetection/crowd2yolo.py:
41: dataset_path = "/data/PSCS-I/" #the path of the PSCS-I dataset
42: out_path = "./CrowdYOLO/" #the path where the YOLO-copatible dataset will be saved
43: sampling_step = 60 #The step to sample frames from the videos (e.g., keep each 60 frames of the videos)
69: video_outputs = ['Videos'] #'Enhanced/Cityscapes','Enhanced/KITTI' can be added to also include photorealism enhanced frames
Then execute the following command to create the YOLO-compatible dataset:
python crowd2yolo.py
To visualize the resulting annotations, the /ObjectDetection/yolo_viz.py script can be utilized. Additionally, to train an object detector, the /ObjectDetection/yolo_train.py is provided using the Ultralytics library.
📝 Note: A YOLOv11 trained model is provided and can be tested on images using the
/ObjectDetection/yolo_predict.pyscript. In detail, the script expects a directory namedimagesinside the same location as the script that contains test images.
Code is also provided to generate a dataset in COCO-compatible format for object detection and semantic/instance segmentation. To create a COCO-compatible dataset, first set the following variables inside /Segmentation/crowd2coco.py:
85: dataset_path = "/data/PSCS-I/" #the path of the PSCS-I dataset
86: out_path = "./CrowdCOCO/" #the path where the YOLO-copatible dataset will be saved
87: sampling_step = 8999 #The step to sample frames from the videos (e.g., keep each 60 frames of the videos)
105: video_outputs = ['Videos'] #'Enhanced/Cityscapes','Enhanced/KITTI' can be added to also include photorealism enhanced frames
Then execute the following command to create the COCO-compatible dataset:
python crowd2coco.py
To visualize the resulting annotations, the /Segmentation/coco_visualization.py script can be utilized.
If you use this dataset, code, or pretrained models in your research, please cite it as follows:
@ARTICLE{11267404,
author={Pasios, Stefanos and Gkountakos, Konstantinos and Ioannidis, Konstantinos and Tsikrika, Theodora and Vrochidis, Stefanos and Kompatsiaris, Ioannis},
journal={IEEE Access},
title={Photorealistic Synthetic Crowds Simulation in Indoor environments (PSCS-I): a novel synthetic dataset for realistic simulation of crowd panic and violence behaviors},
year={2025},
volume={},
number={},
pages={1-1},
keywords={Videos;Synthetic data;Indoor environment;Photorealism;Engines;Annotations;Visualization;Training;Weapons;Lighting;Synthetic Dataset;Crowd Analysis;Abnormal Behavior Detection;Photorealism Enhancement;Unreal Engine},
doi={10.1109/ACCESS.2025.3636727}}