Issues reproducing GraphBEV performance on nuScenes

[Choi-HanEol]

I’m currently working on reproducing the results of GraphBEV, but I'm struggling to reach the performance metrics reported in the paper.
I have encountered some difficulties in reaching the reported mAP/NDS and would like to ask for your guidance.

My Setup & Steps:

Weights Initialization: I used the recommended pretrained weights: cbgs_transfusion_lidar.pth from OpenPCDet and swint-nuimages-pretrained.pth for the image backbone.

Configuration: I haven't modified the original config file provided in the official repository. (https://github.com/adept-thu/GraphBEV/blob/main/tools/cfgs/nuscenes_models/bevfusion_graph_deformable.yaml)

Hardware & Batch Size:
Paper: 8 GPUs with a total batch size of 24.
My Setup: 4 GPUs with a total batch size of 24 (6 per GPU).

Config Discrepancy: I noticed that some hyperparameters in the provided bevfusion_graph_deformable.yaml seem to differ from the descriptions in the paper. Could you please clarify if this YAML is the exact configuration used for the reported results, or if there is another recommended config for reproduction?

epoch 11
***car error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.17, 0.15, 0.09, 0.26, 0.19 | 77.71, 87.10, 89.69, 91.20 | mean AP: 0.8642638931803254
***truck error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.32, 0.18, 0.10, 0.25, 0.22 | 41.56, 59.25, 67.62, 71.19 | mean AP: 0.5990361180098093
***construction_vehicle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.69, 0.43, 0.84, 0.12, 0.32 | 2.92, 16.09, 30.70, 44.68 | mean AP: 0.2359561996428472
***bus error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.33, 0.19, 0.08, 0.52, 0.25 | 45.60, 72.03, 83.62, 86.77 | mean AP: 0.7200624312826853
***trailer error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.51, 0.21, 0.54, 0.21, 0.17 | 13.73, 37.44, 56.38, 66.38 | mean AP: 0.4348350669932525
***barrier error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.19, 0.28, 0.07, nan, nan | 58.38, 67.99, 71.38, 72.76 | mean AP: 0.6762909928277809
***motorcycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.20, 0.24, 0.23, 0.44, 0.25 | 58.76, 69.40, 71.00, 72.19 | mean AP: 0.6783683518159449
***bicycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.16, 0.25, 0.46, 0.26, 0.01 | 51.31, 53.70, 53.84, 54.46 | mean AP: 0.5332534763089352
***pedestrian error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.14, 0.28, 0.39, 0.22, 0.09 | 84.19, 85.37, 86.59, 87.70 | mean AP: 0.8596213464346448
***traffic_cone error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.13, 0.32, nan, nan, nan | 70.48, 71.61, 73.53, 76.40 | mean AP: 0.7300517216626066
--------------average performance-------------
trans_err: 0.2831
scale_err: 0.2549
orient_err: 0.3097
vel_err: 0.2871
attr_err: 0.1876
mAP: 0.6332
NDS: 0.6844

[taoyuhang97]

Hello, I have been reproducing this project recently, but the current computing resources are limited, and it will take a long time to fully reproduce it. Can I obtain a copy of the weights of the model you trained from you? Thank you very much.

[whuxfx]

我也遇到了同样的问题，无法复现论文中的实验结果2026-05-25 06:08:18,235 INFO The predictions of NuScenes have been saved to /data/xfx/GraphBEV/output/cfgs/nuscenes_models/bevfusion_graph/bevfusion_graph_deformable_result_scenes_K_graph8/eval/eval_with_train/epoch_10/val/final_result/data/results_nusc.json
Initializing nuScenes detection evaluation
Loaded results from /data/xfx/GraphBEV/output/cfgs/nuscenes_models/bevfusion_graph/bevfusion_graph_deformable_result_scenes_K_graph8/eval/eval_with_train/epoch_10/val/final_result/data/results_nusc.json. Found detections for 6019 samples.
Loading annotations for val split from nuScenes version: v1.0-trainval
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6019/6019 [00:07<00:00, 846.94it/s]
Loaded ground truth annotations for 6019 samples.
Filtering predictions
=> Original number of boxes: 1203800
=> After distance based filtering: 772706
=> After LIDAR points based filtering: 772706
=> After bike rack filtering: 772442
Filtering ground truth annotations
=> Original number of boxes: 187528
=> After distance based filtering: 134565
=> After LIDAR points based filtering: 121871
=> After bike rack filtering: 121861
Accumulating metric data...
Calculating metrics...
Saving metrics to: /data/xfx/GraphBEV/output/cfgs/nuscenes_models/bevfusion_graph/bevfusion_graph_deformable_result_scenes_K_graph8/eval/eval_with_train/epoch_10/val/final_result/data
mAP: 0.6131
mATE: 0.3084
mASE: 0.2678
mAOE: 0.4328
mAVE: 0.3380
mAAE: 0.1801
NDS: 0.6538
Eval time: 90.3s

Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.873 0.173 0.157 0.089 0.309 0.198
truck 0.519 0.371 0.214 0.128 0.320 0.225
bus 0.685 0.366 0.197 0.106 0.699 0.235
trailer 0.376 0.535 0.237 0.795 0.308 0.180
construction_vehicle 0.270 0.758 0.448 0.956 0.129 0.290
pedestrian 0.856 0.140 0.282 0.393 0.245 0.093
motorcycle 0.665 0.223 0.260 0.550 0.411 0.201
bicycle 0.421 0.186 0.256 0.782 0.282 0.019
traffic_cone 0.761 0.131 0.337 nan nan nan
barrier 0.704 0.203 0.290 0.097 nan nan
2026-05-25 06:10:57,750 INFO ----------------Nuscene detection_cvpr_2019 results-----------------
***car error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.17, 0.16, 0.09, 0.31, 0.20 | 78.31, 87.97, 90.93, 92.02 | mean AP: 0.8730968659590024
***truck error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.37, 0.21, 0.13, 0.32, 0.23 | 30.99, 51.45, 60.23, 64.76 | mean AP: 0.5185694800605999
***construction_vehicle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.76, 0.45, 0.96, 0.13, 0.29 | 3.39, 18.20, 38.30, 47.96 | mean AP: 0.2696286294122551
***bus error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.37, 0.20, 0.11, 0.70, 0.23 | 43.06, 66.77, 81.22, 83.14 | mean AP: 0.6854768624787597
***trailer error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.53, 0.24, 0.79, 0.31, 0.18 | 9.51, 30.64, 48.98, 61.33 | mean AP: 0.37615084942528126
***barrier error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.20, 0.29, 0.10, nan, nan | 58.93, 70.87, 74.95, 77.00 | mean AP: 0.7043890546822207
***motorcycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.22, 0.26, 0.55, 0.41, 0.20 | 55.18, 68.42, 70.67, 71.93 | mean AP: 0.6654870156103454
***bicycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.19, 0.26, 0.78, 0.28, 0.02 | 39.21, 42.70, 42.96, 43.44 | mean AP: 0.4207957438668043
***pedestrian error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.14, 0.28, 0.39, 0.25, 0.09 | 83.52, 85.01, 86.25, 87.55 | mean AP: 0.8558313292533245
***traffic_cone error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0
0.13, 0.34, nan, nan, nan | 73.33, 74.52, 76.68, 79.91 | mean AP: 0.7611162289489917
--------------average performance-------------
trans_err: 0.3084
scale_err: 0.2678
orient_err: 0.4328
vel_err: 0.3380
attr_err: 0.1801
mAP: 0.6131
NDS: 0.6538