REVIEW 3 major objections 5 minor 23 references
VADet: Multi-frame LiDAR 3D Object Detection using Variable Aggregation
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper claims that per-object, variable numbers of aggregated LiDAR frames—chosen by speed and point density—outperform fixed whole-scene aggregation and can beat far more complex multi-frame detectors.
desk verdict The per-object variable aggregation idea is real and worth a look, but the main empirical claim is undersupported until the extra-epoch confound in Table 3 is controlled. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the aggregation-count function $\eta(b_{\tau-1})$, which maps a previous-frame detection's estimated speed $\|v_{\tau-1}\|$ and point density $\rho_{\tau-1}$ (points per unit surface area of the predicted box) to an integer frame count. The function is learned empirically: after RAT training, the training set is split into speed and density bins, and for each bin the frame count with the highest average precision becomes the lookup value. At inference, Algorithm 1 computes the object's current position with a constant-velocity model, enlarges the box by speed and frame count to capture motion smudges, crops the ego-corrected past frames inside that region, and concatenates the object points while leaving background points at a fixed 3-frame aggregation. The same mechanism also introduces a reweighted subset precision metric so that per-speed and per-density AP numbers are comparable across subsets of different sizes.
What would settle it
Evaluate VADet on Waymo validation separated by whether the object was detected in the previous frame and by the error of the velocity estimate; if objects missed in the previous frame show no AP gain over 3-frame fixed aggregation, the benefit is conditional on prior detections. A companion experiment feeding ground-truth previous boxes instead of detector boxes would show how much of the gain depends on the detector's own upstream recall.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that fixed aggregation has a per-object performance trade-off that can be removed by variable per-object aggregation. Using Random Aggregation Training (RAT), a single detector is trained on scenes with randomly varying numbers of aggregated frames, which lets the authors cheaply measure how average precision varies with frame count for objects binned by speed and point density. The resulting function $\eta$ is a piecewise lookup table that assigns each object a frame count, and the input is built by propagating the object's previous detection forward with a constant-velocity model and cropping the ego-corrected past point clouds inside an enlarged box. Applied to CenterPoint, VoxelNeXt, and DSVT-P, VADet exceeds every fixed frame count for the same architecture, and the best variant, VADet-VoxelNeXt, outperforms the attention-based MPPNet by a large margin while adding about 50 ms of overhead versus MPPNet's 900–2500 ms.
Load-bearing premise
The whole scheme is driven by the previous frame's detection: the region to aggregate is computed from that detection and its velocity, so an object that was missed in the previous frame, or has a wrong velocity estimate, receives no extra frames or is aggregated in the wrong place, and the paper does not analyze how often that happens.
Editorial extensions
If this is right
- For each of the three tested architectures, VADet's L2 vehicle AP exceeds the best fixed aggregation count on the Waymo validation split, so the gain is not tied to one backbone.
- VADet-VoxelNeXt achieves 76.1 L2 APH on validation and 79.4 on the test split, surpassing MPPNet and slightly exceeding LoGoNet despite using LiDAR only and no test-time augmentation or ensemble.
- The overhead of VADet's input aggregation is about 50 ms, roughly 20 to 50 times less than the 900–2500 ms second stage of MPPNet, making the gain available for real-time use.
- RAT lets one detector evaluate and serve many frame counts, so it can be a cheap pretraining strategy and a tool for studying input-level temporal fusion before designing architecture-level attention.
Reading between the lines
- Because $\eta$ is a static lookup table, a learned regressor from raw object point clouds to frame counts could capture interactions between speed and density that the hand-chosen bins miss; the paper names this as a possible extension.
- The background points are kept at fixed 3-frame aggregation, so the variable aggregation principle could be pushed further by making background density adaptive too, which might reduce computation and confusion in dense scenes.
- The reweighted subset precision metric proposed for speed and density breakdowns could be applied to any future per-group evaluation on Waymo, since the official metric's treatment of unmatched false positives makes subset sizes incomparable.
- Per-object variable aggregation should transfer to tracking and motion forecasting pipelines, because those tasks already require per-object temporal association; a testable extension is whether the same $\eta$ improves downstream tracking accuracy on Waymo.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes VADet, an input-level per-object aggregation scheme for multi-frame LiDAR 3D detection. It first trains a detector with randomly varying numbers of aggregated frames (RAT), then constructs a lookup table η that maps an object's estimated speed and point density to the number of past frames to aggregate, and finally aggregates per-object regions centered on previous-frame detections propagated by a constant-velocity model. VADet is applied to CenterPoint, VoxelNeXt, and a pillar variant of DSVT and evaluated on Waymo vehicle detection, reporting consistent gains over fixed-aggregation baselines and competitive or superior results against recent multi-frame detectors.
Significance. If the reported gains are robust, VADet is a valuable result: it shows that input-level adaptive aggregation can capture much of the benefit of more complex feature-level temporal fusion at low latency and without architecture-specific changes. The paper's strengths include the RAT training scheme (which makes broad frame-count studies feasible), the well-structured speed/density breakdown analysis in Section 5.3, and the explicit discussion of subset-evaluation metrics in Section 4.2. The central comparison in Table 3, however, is confounded by an extra fine-tuning epoch, and the policy η is fit to the training split with several unspecified hyperparameters; these issues need to be addressed before the central claim can be accepted.
major comments (3)
- [§4.3.2, Table 3] The VADet-versus-fixed comparison in Table 3 is confounded. VADet models are initialized from the RAT baselines and fine-tuned for an additional epoch with cosine decay and mini-sequence shuffling (Sec. 4.3.2), while the fixed-aggregation columns report the RAT baselines without this extra training. Section 3.1 and Table 1 show that additional training can change AP by amounts comparable to the reported margins (e.g., RAT versus separate training differs by up to 0.7 AP). The paper does not report a control in which a fixed k-frame aggregation model receives the same extra fine-tuning epoch; without such a control, the +0.5, +0.7, and +1.3 AP margins in Table 3 cannot be attributed to variable aggregation. The additional mismatch between cached offline predictions used in training and online VADet predictions at inference (Sec. 4.3.2) further complicates attribution. No error bars are reported anywhere, which is especially important because the headline margins are small.
- [§3.2.1, §4.3.3] The lookup table η is fit to the training split: for each speed-density bin, the frame count maximizing AP on the training split is selected over 3–16 frames. This is a high-dimensional model-selection procedure with no multiplicity correction, no reported stability of the selected frame counts, and no sensitivity analysis with respect to the bin thresholds chosen from training statistics. The validation and test results are therefore not an evaluation of an a priori policy, and the claim that η gives the 'empirically best' number of frames needs support in the form of cross-validated η construction or a report of how much the selected frame counts and final AP change under perturbation of the thresholds or the training split.
- [§3.2.2, Algorithm 1, Eqs. (3)–(8)] The per-object aggregation region is computed from the previous frame's detection b_{τ−1} and its velocity estimate v_{τ−1}; if an object is not detected in the previous frame, no extra frames are aggregated for it, so the claimed benefits for sparse or occluded objects (e.g., the qualitative example in Fig. 2) presuppose a prior detection and an accurate velocity. The paper does not analyze this failure mode, nor the sensitivity of the results to errors in v_{τ−1}. In addition, the enlargement factor σ in Eqs. (5)–(7) is introduced but its value is never specified and no ablation or sensitivity analysis is provided; this is a reproducibility issue for a hyperparameter that directly controls the aggregation region.
minor comments (5)
- [§4.3.3] The speed threshold list contains a likely typo: 81.6 m/s should almost certainly be 8.16 m/s; as printed, the thresholds are not monotonic.
- [Algorithm 1] The set-builder notation in the line defining the filtered box is confusing, as it appears to define a set in terms of itself; please rewrite it to indicate that the box is kept when η(b_{τ−1}) > i.
- [§4.3.1, Table 4] The DSVT-P variant used in the experiments is not the original DSVT-pillar: it uses static voxelization and reduced channel widths. This should be stated where Table 4 is discussed, since it affects the interpretation of the comparison to DSVT-pillar.
- [§6] The statement 'we do not anticipate any obvious limitations' is at odds with the acknowledged failure mode in the same section and with the method's dependence on previous-frame detections; please revise.
- [§3.1, Table 1] The RAT-versus-separate-training comparison is reported only for VoxelNeXt and without variance estimates; please clarify whether the equivalence claim is established for the other two architectures.
Circularity Check
No significant circularity: η is a fitted lookup table, but headline comparisons are on held-out Waymo validation/test splits; the unmatched extra fine-tuning epoch and offline/online prediction mismatch are experimental confounds, not circular steps.
full rationale
The paper constructs η by empirically selecting the frame count with the highest AP per (speed, density) bin on the training split (Sec. 3.2.1, 4.3.3), so per-object frame counts are fitted quantities rather than independently predicted ones. However, the central claims (Tables 3-5) are evaluated on the held-out validation and test splits, with speed/density thresholds chosen from training statistics and no refitting of η on the evaluation splits, so the comparison is externally grounded rather than true by construction. The only self-citation is [7] (SOAP), used for the non-central background statement that stationary object point clouds are aligned after ego-motion-corrected aggregation; this is not load-bearing for the VADet derivation. A genuine experimental concern is that VADet models are initialized from RAT baselines and fine-tuned for an additional epoch with cosine decay while the fixed-aggregation columns in Table 3 report the baseline RAT models without that extra epoch (Sec. 4.3.2), and VADet training uses cached offline 3-frame predictions while inference uses online predictions. This makes the Table 3 margins not cleanly attributable to variable aggregation alone, but it does not make the central claim equivalent to its inputs by construction. No circular step is exhibited.
Assumptions & free parameters
free parameters (5)
- η speed bin thresholds =
[0.00, 0.20, 1.55, 3.63, 5.90, 8.16, 11.34, 17.53] m/s (paper prints 81.6, likely typo)
- η density bin thresholds =
[0.00, 0.68, 1.86, 3.86, 8.02, 18.81, 71.37] pts/m2
- Per-bin selected frame count (η output) =
3 to 16, chosen by highest AP per bin on training split
- Enlargement factor σ =
not specified in paper
- Background fixed frame count =
3
assumptions (6)
- domain assumption Ego-motion-corrected concatenation aligns static object points across frames (Eq. 1 standard practice).
- ad hoc to paper The optimal number of aggregated frames for an object depends only on its speed and point density.
- domain assumption A detector trained with random aggregation (RAT) generalizes to all frame counts and can serve as the evaluation model for constructing η.
- domain assumption Previous-frame detections b_{τ-1} are available and reliable enough to define per-object aggregation regions.
- domain assumption Constant velocity motion model (Eq. 3) adequately predicts object location across frames.
- ad hoc to paper The proposed subset precision metric (Eq. 11) fairly weights FPunknown for comparing subsets.
Cite this review
Pith. "Pith review of VADet: Multi-frame LiDAR 3D Object Detection using Variable Aggregation." pith.science (2026). https://pith.science/paper/NF45VQF4
@misc{pith2026241113186,
author = {Pith},
title = {Pith review of: VADet: Multi-frame LiDAR 3D Object Detection using Variable Aggregation},
year = {2026},
howpublished = {\url{https://pith.science/paper/NF45VQF4}},
note = {Machine review of arXiv:2411.13186}
}
read the original abstract
Input aggregation is a simple technique used by state-of-the-art LiDAR 3D object detectors to improve detection. However, increasing aggregation is known to have diminishing returns and even performance degradation, due to objects responding differently to the number of aggregated frames. To address this limitation, we propose an efficient adaptive method, which we call Variable Aggregation Detection (VADet). Instead of aggregating the entire scene using a fixed number of frames, VADet performs aggregation per object, with the number of frames determined by an object's observed properties, such as speed and point density. VADet thus reduces the inherent trade-offs of fixed aggregation and is not architecture specific. To demonstrate its benefits, we apply VADet to three popular single-stage detectors and achieve state-of-the-art performance on the Waymo dataset.
Figures
Reference graph
Works this paper leans on
-
[1]
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krish- nan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driv- ing. In IEEE/CVF Conf. on Computer Vision and Pat- tern Recognition (CVPR), June 2020. 1, 2, 3, 5
work page 2020
-
[2]
MPPNet: Multi-frame feature intertwining with proxy points for 3D temporal object detection
Xuesong Chen, Shaoshuai Shi, Benjin Zhu, Ka Chun Cheung, Hang Xu, and Hongsheng Li. MPPNet: Multi-frame feature intertwining with proxy points for 3D temporal object detection. In European Conf. on Computer Vision (ECCV) , pages 680–697, Cham,
-
[3]
V oxelNeXt: Fully sparse V oxelNet for 3D object detection and tracking
Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. V oxelNeXt: Fully sparse V oxelNet for 3D object detection and tracking. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 21674–21683, 2023. 1, 3, 4, 5
work page 2023
-
[4]
1st Place Solution for Waymo Open Dataset Challenge -- 3D Detection and Domain Adaptation
Zhuangzhuang Ding, Yihan Hu, Runzhou Ge, Li Huang, Sijia Chen, Yu Wang, and Jie Liao. 1st place solution for waymo open dataset challenge– 3d detection and domain adaptation. arXiv preprint arXiv:2006.15505, 2020. 6
work page Pith review arXiv 2006
-
[5]
Super sparse 3D object detection
Lue Fan, Yuxue Yang, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Super sparse 3D object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12490–12505, 2023. 5, 6
work page 2023
-
[6]
AFDetV2: Rethinking the necessity of the second stage for ob- ject detection from point clouds
Yihan Hu, Zhuangzhuang Ding, Runzhou Ge, Wenxin Shao, Li Huang, Kun Li, and Qiang Liu. AFDetV2: Rethinking the necessity of the second stage for ob- ject detection from point clouds. In Proceedings of the AAAI Conference on Artificial Intelligence , vol- ume 36, pages 969–979, 2022. 6
work page 2022
-
[7]
Chengjie Huang, Vahdat Abdelzad, Sean Sedwards, and Krzysztof Czarnecki. SOAP: Cross-sensor do- main adaptation for 3D object detection using Sta- tionary Object Aggregation Pseudo-labelling. In IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV), 2024. 1, 2, 3
work page 2024
-
[8]
Ross, Thomas Funkhouser, and Alireza Fathi
Rui Huang, Wanyue Zhang, Abhijit Kundu, Caroline Pantofaru, David A. Ross, Thomas Funkhouser, and Alireza Fathi. An LSTM approach to temporal 3D object detection in LiDAR point clouds. In European Conf. on Computer Vision (ECCV) , pages 266–282, Cham, 2020. Springer International Publishing. 2
work page 2020
Show all 23 references
-
[9]
Pil- larNeXt: Rethinking network designs for 3D object detection in LiDAR point clouds
Jinyu Li, Chenxu Luo, and Xiaodong Yang. Pil- larNeXt: Rethinking network designs for 3D object detection in LiDAR point clouds. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 17567–17576, 2023. 6
2023
-
[10]
Logonet: Towards accurate 3d ob- ject detection with local-to-global cross-modal fusion
Xin Li, Tao Ma, Yuenan Hou, Botian Shi, Yuchen Yang, Youquan Liu, Xingjiao Wu, Qin Chen, Yikang Li, Yu Qiao, et al. Logonet: Towards accurate 3d ob- ject detection with local-to-global cross-modal fusion. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pag...
2023
-
[11]
Deep- fusion: Lidar-camera deep fusion for multi-modal 3d object detection
Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V Le, et al. Deep- fusion: Lidar-camera deep fusion for multi-modal 3d object detection. In IEEE/CVF Conf. on Computer Vi- sion and Pattern Recognition (CVPR...
2022
-
[12]
Bevfusion: Multi-task multi-sensor fusion with uni- fied bird’s-eye view representation
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with uni- fied bird’s-eye view representation. In Int. Conf. on Robotics and Automation (ICRA) , pages 2774–2781,
-
[13]
Fast and Furious: Real time end-to-end 3D detection, track- ing and motion forecasting with a single convolutional net
Wenjie Luo, Bin Yang, and Raquel Urtasun. Fast and Furious: Real time end-to-end 3D detection, track- ing and motion forecasting with a single convolutional net. In IEEE/CVF Conf. on Computer Vision and Pat- tern Recognition (CVPR), June 2018. 2
2018
-
[14]
TransPillars: Coarse-to-fine aggregation for multi-frame 3D object detection
Zhipeng Luo, Gongjie Zhang, Changqing Zhou, Tian- rui Liu, Shijian Lu, and Liang Pan. TransPillars: Coarse-to-fine aggregation for multi-frame 3D object detection. In IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV), pages 4230–4239, Jan- uary 2023. 1, 2
2023
-
[15]
PV-RCNN++: Point-voxel feature set abstraction with local vector representation for 3D ob- ject detection
Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN++: Point-voxel feature set abstraction with local vector representation for 3D ob- ject detection. International Journal of Computer Vi- sion, 131(2):531–551, 2023. 6
2023
-
[16]
Scalability in perception for autonomous driving: Waymo Open Dataset
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Au- relien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vi- jay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zh...
2020
-
[17]
SWFormer: Sparse window transformer for 3D object detection in point clouds
Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. SWFormer: Sparse window transformer for 3D object detection in point clouds. In European Conf. on Com- puter Vision (ECCV), pages 426–442. Springer, 2022. 6
2022
-
[18]
DSVT: Dynamic sparse voxel transformer with ro- tated sets
Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele, and Liwei Wang. DSVT: Dynamic sparse voxel transformer with ro- tated sets. In IEEE/CVF Conf. on Computer Vi- sion and Pattern Recognition (CVPR) , pages 13520– 13529, June 2023. 1, 2, 4, 5, 6
2023
-
[19]
3D-MAN: 3D multi-frame attention network for object detection
Zetong Yang, Yin Zhou, Zhifeng Chen, and Jiquan Ngiam. 3D-MAN: 3D multi-frame attention network for object detection. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 1863– 1872, June 2021. 1, 2, 3, 6
2021
-
[20]
LiDAR-based online 3D video object detection with graph-based message passing and spatiotemporal transformer attention
Junbo Yin, Jianbing Shen, Chenye Guan, Dingfu Zhou, and Ruigang Yang. LiDAR-based online 3D video object detection with graph-based message passing and spatiotemporal transformer attention. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), June 2020. 2
2020
-
[21]
Center-based 3D object detection and tracking
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3D object detection and tracking. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 11784–11793, 2021. 4, 5
2021
-
[22]
CenterFormer: Center- based transformer for 3D object detection
Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. CenterFormer: Center- based transformer for 3D object detection. In Euro- pean Conf. on Computer Vision (ECCV) , pages 496– 513, Cham, 2022. Springer, Springer Nature Switzer- land. 6
2022
-
[2022]
1, 2, 3, 6
Springer Nature Switzerland. 1, 2, 3, 6
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.