Reverberation

This is the official code of our paper "Reverberation: Learning the Latencies Before Forecasting Trajectories". The full paper is now available at https://arxiv.org/abs/2511.11164. Our model weights are available at here.

Authors' Note

This work is the third part in our Echolocation Trilogy. It describes and predicts how long agents' echoes from the past last in the future.

Here are all the repositories involved in our trilogy:

• Part I, Where do the echoes come from?: SocialCircle (CVPR 2024) and SocialCirclePlus (Journal, Under Review);
• Part II, How echoes interact with each other?: Resonance (ICCV 2025);
• Part III, How long do the echoes last?: Reverberation (Journal, Under Review).

Note that these repositories share the same training engine and the weight files are compatible with each other (in the order in which the repositories were released, the later released repositories are compatible with the weights of the previous releases, the Reverberation repository is recommended as it is compatible with all previous models). You can copy only the core model folders, e.g. SocialCircle, Re, Rev, etc., to a repository's root path (i.e., the folder where qpid is located), and train and test the models via main.py.

Getting Started

You can clone this repository by the following command:

git clone https://github.com/cocoon2wong/Rev.git

Then, run the following command to initialize all submodules (Submodules include our unified training engine (qpid) and dataset processing utilities):

git submodule update --init --recursive

Requirements

The code is developed with Python 3.10. Additional packages used are included in the requirements.txt file.

Warning

We recommend installing all required Python packages in a virtual environment (like the conda environment). Otherwise, there COULD be other problems due to the package version conflicts.

Run the following command to install the required packages in your Python environment:

pip install -r requirements.txt

Preparing Datasets

ETH-UCY, SDD, nuScenes, Human3.6M

Warning

If you want to validate Rev models on these datasets, make sure you are getting this repository via git clone and that all git submodules have been properly initialized via git submodule update --init --recursive.

You can run the following commands to prepare dataset files that have been validated in our paper:

1. Run Python the script inner the dataset_original folder:

   cd dataset_original

   • For ETH-UCY and SDD, run

     python main_ethucysdd.py

     [!NOTE] Our reported results and provided weights on the eth set is actually the 6-frame-interval version. To test the 10-frame-sampled eth, please run python codes/ethucy/create_10sampled_eth_data.py, then use the clip name eth10 to train or test models. For more information and notes our used datasets and splits, please refer to this page.

   • For nuScenes, please download their dataset files, put them into the given path listed within dataset_original/main_nuscenes.py, then run

     python main_nuscenes.py

   • For Human3.6M, please download their dataset files, put them into the given path listed within dataset_original/main_h36m.py, then run

     python main_h36m.py

2. Back to the repo folder and create soft links:

   cd ..
   ln -s dataset_original/dataset_processed ./
   ln -s dataset_original/dataset_configs ./

Note

You can also download our processed dataset files manually from here, and put them into dataset_processed and dataset_configs folders manually to reproduce our results. Due to size limits, processed dataset files of the Human3.6M dataset is available at here.

Click the following buttons to learn how we process these dataset files and the detailed dataset settings.

💡 Dataset Guidelines 💡 Datasets and Splits Information

Training on Your New Datasets

Before training Rev models on your own dataset, you should add your dataset information. See this page for more details.

Model Weights

We have provided our pre-trained model weights to help you quickly evaluate Rev models' performance.

Click the following buttons to download our model weights. We recommend that you download the weights and place them in the weights folder.

⬇️ Download Weights

You can start evaluating these weights by

python main.py --load SOME_MODEL_WEIGHTS

Here, SOME_MODEL_WEIGHTS is the path of the weights folder, for example, ./weights/revzara1.

Training

You can start training a Rev model via the following command:

python main.py --model rev --split DATASET_SPLIT

Here, DATASET_SPLIT is the identifier (i.e., the name of dataset's split files in dataset_configs, for example eth is the identifier of the split list in dataset_configs/ETH-UCY/eth.plist) of the dataset or splits used for training. It accepts:

• ETH-UCY: {eth, hotel, univ13, zara1, zara2};
• SDD: sdd;
• nuScenes: nuScenes_ov_v1.0;
• Human3.6M: h36m.

For example, you can start training the Rev model on the zara1 split by

python main.py --model rev --split zara1

Also, other args may need to be specified, like the learning rate --lr, batch size --batch_size, etc. See detailed args in the Args Used Section.

Reproducing Our Results

The simplest way to reproduce our results is to copy all training args we used in the provided weights. For example, you can start a training of Rev on zara1 using the same args as we did by:

python main.py --model rev --restore_args ${PATH_TO_YOUR_DOWNLOADED_WEIGHTS}/revzara1

Here, ${PATH_TO_YOUR_DOWNLOADED_WEIGHTS} is your path to save our pretrained weights, which usually could be named as Project-Monandaeg-Rev after downloading.

You can open a Tensorboard to see how losses and metrics change during training, by:

tensorboard --logdir ./logs

Visualization & Playground

We have build a simple user interface to validate the qualitative trajectory prediction performance of our proposed Rev models. You can use it to visualize model predictions and learn how the proposed Rev works to handle social interactions in an interactive way by adding any manual neighbors at any positions in the scene.

Warning

Visualizations may need dataset videos. For copyright reasons and size limitations, we do not provide them in our repo. Instead, a static image will be displayed if you have no videos put into the corresponding path.

Visualization Requirements

This playground interface is implemented with PyQt6. Install this package in your python environment to start:

pip install pyqt6

Open a Playground

Run the following command to open a playground:

python playground/main.py

Load Models and Datasets

You can load a supported Rev model or one of its variations by clicking the Load Model button. By clicking the Run button, you can see how the loaded model performs on the given sample. You can also load different datasets (video clips) by clicking the More Settings ... button.

Add Manual Neighbors

You can also directly click the visualized figure to add a new neighbor to the scene. Through this neighbor that wasn't supposed to exist in the prediction scene, you can verify how models handle social interactions qualitatively.

Visualize Reverberation Curves

You can add the arg --draw_kernels to visualize the non-interactive or social reverberation strengths, i.e., our defined non-interactive or social reverberation curves of different prediction samples. For example, you can use the following command to open a playground with our pretrained zara1 weights loaded (supposed to be located at ./weights/revzara1), and visualize reverberation curves by clicking the Run button:

python playground/main.py --load ./weights/revzara1 --draw_kernels

Note that the arg --draw_kernels also accepts an integer arg:

• --draw_kernels 0: Do nothing;
• --draw_kernels 1 (equals to the non-arged --draw_kernels): Visualize only (original) reverberation curves (computed from reverberation kernel $\mathbf{R}$);
• --draw_kernels 2: Visualize both original reverberation curves ($\mathbf{R}$) and their altered curves ($\mathbf{G}$), both non-interactive and social ones.

You can check different agents' reverberation curves by clicking the Random button to randomly select a prediction sample, then click Run to start predict and visualize reverberation curves:

Visualize Social Modifications

You can use the following command to visualize how a manual neighbor modifies the ego-agent's predicted trajectories, especially in the form of our defined social modification:

python playground/main.py --compute_social_mod

Load a model (the Load Model button), add a manual neighbor (click on the right canvas twice), then click the left Compute Social Modifications to start visualize. The printed results are similar to our main paper's last figure. Try to explore how our model handles social interactions in this way!

Contact us

Conghao Wong (@cocoon2wong): conghaowong@icloud.com
Ziqian Zou (@LivepoolQ): ziqianzoulive@icloud.com

---

Args Used

Please specify your customized args when training or testing your model in the following way:

python main.py --ARG_KEY1 ARG_VALUE2 --ARG_KEY2 ARG_VALUE2 -SHORT_ARG_KEY3 ARG_VALUE3 ...

where ARG_KEY is the name of args, and ARG_VALUE is the corresponding value. All args and their usages are listed below.

About the argtype:

• Args with argtype=static can not be changed once after training. When testing the model, the program will not parse these args to overwrite the saved values.
• Args with argtype=dynamic can be changed anytime. The program will try to first parse inputs from the terminal and then try to load from the saved JSON file.
• Args with argtype=temporary will not be saved into JSON files. The program will parse these args from the terminal at each time.

Basic Args

--K

The number of multiple generations when testing. This arg only works for multiple-generation models.

• Type=int, argtype=dynamic;
• The default value is 20.

--K_train

The number of multiple generations when training. This arg only works for multiple-generation models.

• Type=int, argtype=static;
• The default value is 10.

--anntype

Model's predicted annotation type. Can be 'coordinate' or 'boundingbox'.

• Type=str, argtype=static;
• The default value is coordinate.

--auto_clear

Controls whether to clear all other saved weights except for the best one. It performs similarly to running python scripts/clear.py --logs logs.

• Type=int, argtype=temporary;
• The default value is 1.

--batch_size (short for -bs)

Batch size when implementation.

• Type=int, argtype=dynamic;
• The default value is 5000.

--compute_loss

Controls whether to compute losses when testing.

• Type=int, argtype=temporary;
• The default value is 0.

--compute_metrics_with_types

Controls whether to compute metrics separately on different kinds of agents.

• Type=int, argtype=temporary;
• The default value is 0.

--compute_statistical_metrics

(bool) Choose whether to compute metrics (ADE/FDE) as mean $\pm$ std.

• Type=int, argtype=temporary;
• The default value is 0.

--dataset

Name of the video dataset to train or evaluate. For example, 'ETH-UCY' or 'SDD'. NOTE: DO NOT set this argument manually.

• Type=str, argtype=static;
• The default value is Unavailable.

--down_sampling_rate

Selects whether to down-sample from multiple-generated predicted trajectories. This arg only works for multiple-generative models.

• Type=float, argtype=temporary;
• The default value is 1.0.

--draw_results (short for -dr)

Controls whether to draw visualized results on video frames. Accept the name of one video clip. The codes will first try to load the video file according to the path saved in the plist file (saved in dataset_configs folder), and if it loads successfully it will draw the results on that video, otherwise it will draw results on a blank canvas. Note that test_mode will be set to 'one' and force_split will be set to draw_results if draw_results != 'null'.

• Type=str, argtype=temporary;
• The default value is null.

--draw_videos

Controls whether to draw visualized results on video frames and save them as images. Accept the name of one video clip. The codes will first try to load the video according to the path saved in the plist file, and if successful it will draw the visualization on the video, otherwise it will draw on a blank canvas. Note that test_mode will be set to 'one' and force_split will be set to draw_videos if draw_videos != 'null'.

• Type=str, argtype=temporary;
• The default value is null.

--epochs

Maximum training epochs.

• Type=int, argtype=static;
• The default value is 500.

--experimental

NOTE: It is only used for code tests.

• Type=bool, argtype=temporary;
• The default value is False.

--feature_dim

Feature dimensions that are used in most layers.

• Type=int, argtype=static;
• The default value is 128.

--force_anntype

Assign the prediction type. It is now only used for silverballers models that are trained with annotation type coordinate but to be tested on datasets with annotation type boundingbox.

• Type=str, argtype=temporary;
• The default value is null.

--force_clip

Force test video clip (ignore the train/test split). It only works when test_mode has been set to one.

• Type=str, argtype=temporary;
• The default value is null.

--force_dataset

Force test dataset (ignore the train/test split). It only works when test_mode has been set to one.

• Type=str, argtype=temporary;
• The default value is null.

--force_split

Force test dataset (ignore the train/test split). It only works when test_mode has been set to one.

• Type=str, argtype=temporary;
• The default value is null.

--gpu

Speed up training or test if you have at least one NVidia GPU. If you have no GPUs or want to run the code on your CPU, please set it to -1. NOTE: It only supports training or testing on one GPU.

• Type=str, argtype=temporary;
• The default value is 0.

--help (short for -h)

Print help information on the screen.

• Type=str, argtype=temporary;
• The default value is null.

--input_pred_steps

Indices of future time steps that are used as extra model inputs. It accepts a string that contains several integer numbers separated with '_'. For example, '3_6_9'. It will take the corresponding ground truth points as the input when training the model, and take the first output of the former network as this input when testing the model. Set it to 'null' to disable these extra model inputs.

• Type=str, argtype=static;
• The default value is null.

--interval

Time interval of each sampled trajectory point.

• Type=float, argtype=static;
• The default value is 0.4.

--load (short for -l)

Folder to load model weights (to test). If it is set to null, the training manager will start training new models according to other reveived args. NOTE: Leave this arg to null when training new models.

• Type=str, argtype=temporary;
• The default value is null.

--load_epoch

Load model weights that is saved after specific training epochs. It will try to load the weight file in the load dir whose name is end with _epoch${load_epoch}. This arg only works when the auto_clear arg is disabled (by passing --auto_clear 0 when training). Set it to -1 to disable this function.

• Type=int, argtype=temporary;
• The default value is -1.

--load_part

Choose whether to load only a part of the model weights if the state_dict of the saved model and the model in the code do not match. *IMPORTANT NOTE*: This arg is only used for some ablation experiments. It MAY lead to incorrect predictions or metrics.

• Type=int, argtype=temporary;
• The default value is 0.

--log_dir

Folder to save training logs and model weights. Logs will save at ${save_base_dir}/${log_dir}. DO NOT change this arg manually. (You can still change the saving path by passing the save_base_dir arg.).

• Type=str, argtype=static;
• The default value is Unavailable.

--loss_weights

Configure the agent-wise loss weights. It now only supports the dataset-clip-wise re-weight.

• Type=str, argtype=dynamic;
• The default value is {}.

--lr (short for -lr)

Learning rate.

• Type=float, argtype=static;
• The default value is 0.001.

--macos

(Experimental) Choose whether to enable the MPS (Metal Performance Shaders) on Apple platforms (instead of running on CPUs).

• Type=int, argtype=temporary;
• The default value is 0.

--max_agents

Max number of agents to predict per frame. It only works when model_type == 'frame-based'.

• Type=int, argtype=static;
• The default value is 50.

--model

The model type used to train or test.

• Type=str, argtype=static;
• The default value is none.

--model_name

Customized model name.

• Type=str, argtype=static;
• The default value is model.

--model_type

Model type. It can be 'agent-based' or 'frame-based'.

• Type=str, argtype=static;
• The default value is agent-based.

--noise_depth

Depth of the random noise vector.

• Type=int, argtype=static;
• This arg can also be spelled as--depth;
• The default value is 16.

--obs_frames (short for -obs)

Observation frames for prediction.

• Type=int, argtype=static;
• The default value is 8.

--output_pred_steps

Indices of future time steps to be predicted. It accepts a string that contains several integer numbers separated with '_'. For example, '3_6_9'. Set it to 'all' to predict points among all future steps.

• Type=str, argtype=static;
• This arg can also be spelled as--key_points;
• The default value is all.

--pmove

(Pre/post-process Arg) Index of the reference point when moving trajectories.

• Type=int, argtype=static;
• The default value is -1.

--pred_frames (short for -pred)

Prediction frames.

• Type=int, argtype=static;
• The default value is 12.

--preprocess

Controls whether to run any pre-process before the model inference. It accepts a 3-bit-like string value (like '111'): - The first bit: MOVE trajectories to (0, 0); - The second bit: re-SCALE trajectories; - The third bit: ROTATE trajectories.

• Type=str, argtype=static;
• The default value is 100.

--restore

Path to restore the pre-trained weights before training. It will not restore any weights if args.restore == 'null'.

• Type=str, argtype=temporary;
• The default value is null.

--restore_args

Path to restore the reference args before training. It will not restore any args if args.restore_args == 'null'.

• Type=str, argtype=temporary;
• The default value is null.

--save_base_dir

Base folder to save all running logs.

• Type=str, argtype=static;
• The default value is ./logs.

--split (short for -s)

The dataset split that used to train and evaluate.

• Type=str, argtype=static;
• The default value is zara1.

--start_test_percent

Set when (at which epoch) to start validation during training. The range of this arg should be 0 <= x <= 1. Validation may start at epoch args.epochs * args.start_test_percent.

• Type=float, argtype=temporary;
• The default value is 0.0.

--step

Frame interval for sampling training data.

• Type=float, argtype=dynamic;
• The default value is 1.0.

--test_mode

Test settings. It can be 'one', 'all', or 'mix'. When setting it to one, it will test the model on the args.force_split only; When setting it to all, it will test on each of the test datasets in args.split; When setting it to mix, it will test on all test datasets in args.split together.

• Type=str, argtype=temporary;
• The default value is mix.

--test_step

Epoch interval to run validation during training.

• Type=int, argtype=temporary;
• The default value is 1.

--update_saved_args

Choose whether to update (overwrite) the saved arg files or not.

• Type=int, argtype=temporary;
• The default value is 0.

--verbose (short for -v)

Controls whether to print verbose logs and outputs to the terminal.

• Type=int, argtype=temporary;
• The default value is 0.

Reverberation Args

--Kc

The number of generations when making predictions. It is also the channels of the generating kernel in the proposed reverberation transform.

• Type=int, argtype=static;
• This arg can also be spelled as--Kg, --K_g;
• The default value is 20.

--T (short for -T)

Transform type used to compute trajectory spectrums. It could be: - none: no transformations; - haar: haar wavelet transform; - db2: DB2 wavelet transform.

• Type=str, argtype=static;
• The default value is haar.

--compute_linear

(bool) Choose whether to learn to forecast the linear trajectory during training.

• Type=int, argtype=static;
• This arg can also be spelled as--compute_linear_base;
• The default value is 1.

--compute_noninteractive

(bool) Choose whether to learn to forecast the non-interactive trajectory during training.

• Type=int, argtype=static;
• This arg can also be spelled as--learn_self_bias, --compute_self_bias, --compute_non;
• The default value is 1.

--compute_social

(bool) Choose whether to learn to forecast the social trajectory during training.

• Type=int, argtype=static;
• This arg can also be spelled as--learn_re_bias, --compute_re_bias;
• The default value is 1.

--disable_G

(bool) Choose whether to disable the generating kernels when applying the reverberation transform. An MSN-like generating approach will be used if this arg is set to 1.

• Type=int, argtype=static;
• The default value is 0.

--disable_R

(bool) Choose whether to disable the reverberation kernels when applying the reverberation transform. Flatten and fc layers will be used if this arg is set to 1.

• Type=int, argtype=static;
• The default value is 0.

--draw_kernels

Choose whether or in which ways to draw and show visualized kernels when testing. It accepts an int value, including [0, 1, 2, 3]: - 0: Do nothing; - 1: Only visualize the reverberation kernel; - 2: Visualize both reverberation and generating kernels; - 3: Visualize both kernels and their inverse kernels. This arg is typically used in the playground mode.

• Type=int, argtype=temporary;
• The default value is 0.

--encode_agent_types

(bool) Choose whether to encode the type name of each agent. It is mainly used in multi-type-agent prediction scenes, providing a unique type-coding for each type of agents when encoding their trajectories.

• Type=int, argtype=static;
• The default value is 0.

--no_interaction

(bool) Whether to forecast trajectories by considering social interactions. It will compute all social-interaction-related components on the set of empty neighbors if this args is set to 1.

• Type=int, argtype=temporary;
• The default value is 0.

--partitions

The number of partitions when computing the angle-based feature. It is only used when modeling social interactions.

• Type=int, argtype=static;
• The default value is -1.

--select_generating_channel

(Ablation Only) Select one of the generating channel as the direct output of the prediction network. Value range: 0 <= n < K_g. NOTE: This MAY lead to significant performance degradation as only one channel is reserved for prediction. This arg is only used for conducting ablation analyses or discussions.

• Type=int, argtype=temporary;
• The default value is -1.

--select_social_partition

Choose which social partition will be displayed when visualizing social generating kernels. The indices of social partitions start from 1, rather than 0. It only works when the arg draw_kernels is set to 2 or 3. NOTE: This value should be no more than the number of total partitions.

• Type=int, argtype=temporary;
• The default value is 1.

--test_with_linear

(bool) Choose whether to ignore the linear base when forecasting. It only works when testing.

• Type=int, argtype=temporary;
• The default value is 0.

--test_with_noninteractive

(bool) Choose whether to ignore the self-bias when forecasting. It only works when testing.

• Type=int, argtype=temporary;
• This arg can also be spelled as--test_with_non;
• The default value is 0.

--test_with_social

(bool) Choose whether to ignore the resonance-bias when forecasting. It only works when testing.

• Type=int, argtype=temporary;
• This arg can also be spelled as--test_with_soc;
• The default value is 0.

Visualization Args

--distribution_steps

Controls which time step(s) should be considered when visualizing the distribution of forecasted trajectories. It accepts one or more integer numbers (started with 0) split by '_'. For example, '4_8_11'. Set it to 'all' to show the distribution of all predictions.

• Type=str, argtype=temporary;
• The default value is all.

--draw_distribution (short for -dd)

Controls whether to draw distributions of predictions instead of points. If draw_distribution == 0, it will draw results as normal coordinates; If draw_distribution == 1, it will draw all results in the distribution way, and points from different time steps will be drawn with different colors.

• Type=int, argtype=temporary;
• The default value is 0.

--draw_exclude_type

Draw visualized results of agents except for user-assigned types. If the assigned types are "Biker_Cart" and the draw_results or draw_videos is not "null", it will draw results of all types of agents except "Biker" and "Cart". It supports partial match, and it is case-sensitive.

• Type=str, argtype=temporary;
• The default value is null.

--draw_extra_outputs

Choose whether to draw (put text) extra model outputs on the visualized images.

• Type=int, argtype=temporary;
• The default value is 0.

--draw_full_neighbors

Choose whether to draw the full observed trajectories of all neighbor agents or only the last trajectory point at the current observation moment.

• Type=int, argtype=temporary;
• The default value is 0.

--draw_index

Indexes of test agents to visualize. Numbers are split with _. For example, '123_456_789'.

• Type=str, argtype=temporary;
• The default value is all.

--draw_lines

Choose whether to draw lines between each two 2D trajectory points.

• Type=int, argtype=temporary;
• The default value is 0.

--draw_on_empty_canvas

Controls whether to draw visualized results on the empty canvas instead of the actual video.

• Type=int, argtype=temporary;
• The default value is 0.

--draw_with_plt

(bool) Choose whether to use PLT as the preferred method for visualizing trajectories (on the empty canvas). It will try to visualize all points on the scene images if this arg is not enabled.

• Type=int, argtype=temporary;
• The default value is 0.

Playground Args

--clip

The video clip to run this playground.

• Type=str, argtype=temporary;
• The default value is zara1.

--compute_social_mod

(bool) Choose whether to enable the computing of social modifications.

• Type=int, argtype=temporary;
• The default value is 0.

--default_agent

Set the default index of agent to be predicted.

• Type=int, argtype=temporary;
• The default value is 0.

--do_not_draw_neighbors

(bool) Choose whether to draw neighboring-agents' trajectories.

• Type=int, argtype=temporary;
• The default value is 0.

--draw_seg_map

(bool) Choose whether to draw segmentation maps on the canvas.

• Type=int, argtype=temporary;
• The default value is 1.

--lite

(bool) Choose whether to show the lite-version's visualization window.

• Type=int, argtype=temporary;
• The default value is 0.

--physical_manual_neighbor_mode

Mode for the manual neighbor on segmentation maps. - Mode 1: Add obstacles to the given position; - Mode 0: Set areas to be walkable.

• Type=float, argtype=temporary;
• The default value is 1.0.

--points

The number of points to simulate the trajectory of manual neighbor. It only accepts 2 or 3.

• Type=int, argtype=temporary;
• The default value is 2.

--save_full_outputs

(bool) Choose whether to save all outputs as images.

• Type=int, argtype=temporary;
• The default value is 0.

--show_manual_neighbor_boxes

(Working in process).

• Type=int, argtype=temporary;
• The default value is 0.