REVIEW 1 major objections 1 minor 53 references
A hierarchical transformer fuses RGB and event data to reduce fusion noise in object detection.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-06-30 08:05 UTC pith:C4TB66M4
load-bearing objection CMTFormer gives a staged RGB-event fusion design with SAM/CEM/LDFM but the mAP gains do not isolate whether those modules actually cut noise or redundancy. the 1 major comments →
CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
CMTFormer marries a transformer architecture with hierarchical cross-modal information interaction, using SAM to fuse low-level features while mitigating disparities, CEM to reinforce middle-level features with texture and edge info, LDFM to aggregate high-level clues via learnable weights, and a Spatial Prior Module for better localization, resulting in superior performance on RGB-event object detection tasks.
What carries the argument
The shallow-to-deep information interaction scheme with Shallow Alignment Module (SAM), Cross-modal Enhancement Module (CEM), and Learnable Deep Fusion Module (LDFM) that enables efficient and stable multimodal collaboration.
Load-bearing premise
The shallow-to-deep interaction scheme mitigates attribute disparities and prevents noisy or redundant integration between RGB and event data without introducing new fusion artifacts.
What would settle it
An ablation study showing that removing the hierarchical modules (SAM, CEM, LDFM) does not degrade performance on the DSEC-Detection benchmark would falsify the central claim.
If this is right
- Surpasses detection counterparts in both uni-modal and multi-modal settings on DSEC-Detection and PKU-DAVIS-SOD benchmarks.
- Prevents noisy or redundant feature integration during cross-modal fusion.
- The learnable weights in LDFM allow adaptive fusion of RGB and event clues.
- Global spatial information from the Spatial Prior Module enhances localization accuracy.
Where Pith is reading between the lines
- Similar hierarchical fusion could be applied to other sensor combinations like RGB and LiDAR.
- The approach may generalize to video object detection where temporal information is key.
- By avoiding new fusion artifacts, it could enable more reliable deployment in challenging environments like low light or fast motion.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CMTFormer, a transformer architecture for RGB-event object detection that employs a shallow-to-deep hierarchical interaction scheme. This includes the Shallow Alignment Module (SAM) for low-level fusion to reduce attribute disparities, the Cross-modal Enhancement Module (CEM) for mutual reinforcement of middle-level features using texture and edge cues, the Learnable Deep Fusion Module (LDFM) for adaptive high-level aggregation via learnable weights, and an additional Spatial Prior Module for improved localization. The central claim is that this design enables efficient multimodal collaboration without noise amplification or redundancy, leading to consistent outperformance over uni-modal and multi-modal baselines on the DSEC-Detection and PKU-DAVIS-SOD benchmarks.
Significance. If the empirical results hold under rigorous validation, the hierarchical fusion paradigm could advance RGB-event detection by addressing modality heterogeneity more systematically than prior fusion techniques. The learnable adaptive fusion in LDFM and the staged interaction represent a structured approach that may generalize to other event-based vision tasks, with the promise of reproducibility noted via planned code release.
major comments (1)
- [Experiments section] Experiments section (and associated tables/figures): The claim that SAM, CEM, and LDFM specifically mitigate attribute disparities and prevent noisy/redundant integration (as stated in the abstract and §1) rests solely on overall mAP improvements versus baselines. No ablation studies isolating each module's contribution, nor direct quantitative analyses (e.g., feature correlation, noise statistics, or redundancy metrics before/after fusion), are described to confirm the modules achieve the stated mitigation rather than other factors such as transformer capacity or training details.
minor comments (1)
- [Abstract] Abstract: No quantitative metrics, baseline names, or specific mAP values are reported, which weakens the standalone readability of the claims.
Simulated Author's Rebuttal
We thank the referee for the constructive comment on the experimental validation of our proposed modules. We address the point below and will revise the manuscript accordingly to strengthen the evidence.
read point-by-point responses
-
Referee: [Experiments section] Experiments section (and associated tables/figures): The claim that SAM, CEM, and LDFM specifically mitigate attribute disparities and prevent noisy/redundant integration (as stated in the abstract and §1) rests solely on overall mAP improvements versus baselines. No ablation studies isolating each module's contribution, nor direct quantitative analyses (e.g., feature correlation, noise statistics, or redundancy metrics before/after fusion), are described to confirm the modules achieve the stated mitigation rather than other factors such as transformer capacity or training details.
Authors: We agree that the current experiments demonstrate overall mAP gains but do not isolate the specific contributions of SAM, CEM, and LDFM through dedicated ablations or direct metrics on disparity mitigation, noise, or redundancy. While the staged design rationale (detailed in §3) and consistent outperformance over comparable baselines provide supporting context, additional targeted experiments are needed to rule out confounding factors such as model capacity. In the revised manuscript, we will add: (i) module-wise ablation tables reporting performance when each component is removed individually, (ii) capacity-controlled variants that retain transformer blocks but omit the hierarchical cross-modal interaction, and (iii) supplementary quantitative analyses (e.g., feature correlation or similarity metrics across modalities before/after fusion) together with qualitative visualizations. These additions will directly address the concern. revision: yes
Circularity Check
No circularity; empirical architecture validated on external benchmarks
full rationale
The paper proposes CMTFormer with descriptive modules (SAM, CEM, LDFM, Spatial Prior) for RGB-event fusion and supports effectiveness solely via end-to-end mAP gains on DSEC-Detection and PKU-DAVIS-SOD against uni- and multi-modal baselines. No equations, derivations, fitted parameters renamed as predictions, or load-bearing self-citations appear. The central claims reduce to experimental comparison rather than any self-referential construction, making the result self-contained against external data.
Axiom & Free-Parameter Ledger
read the original abstract
Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range, compensating for the deficiencies of the conventional RGB frames. However, previous multi-modal fusion techniques typically fail to handle the inherent heterogeneity between RGB frames and event streams, thus easily leading to noise amplification or redundant feature integration during cross-modal fusion. In this paper, we propose a Cross-Modal information inTeraction transFormer, coined as CMTFormer, which hierarchically integrates RGB and event information to achieve efficient and stable multimodal collaboration. Specifically, we design a shallow-to-deep information interaction scheme. In the shallow stage, we present the Shallow Alignment Module (SAM) to achieve an efficient fusion of RGB and event low-level features, which mitigates attribute disparities and prevents noisy information. In the middle stage, we devise the Cross-modal Enhancement Module (CEM) that utilizes texture and edge information to produce mutually reinforced middle-level features. In the deep stage, we present the Learnable Deep Fusion Module (LDFM) which performs high-level information aggregation through learnable weights, thus enabling the network to adaptively fuse RGB and event clues. A Spatial Prior Module is further designed to utilize global spatial information to enhance localization accuracy. Extensive experiments are conducted on two prevalent event-based object detection benchmarks, i.e., DSEC-Detection and PKU-DAVIS-SOD. Our CMTFormer consistently surpasses the detection counterparts in both uni-modal and multi-modal settings, strongly demonstrating the effectiveness of our paradigm. Codes will be available upon publication.
Figures
Reference graph
Works this paper leans on
-
[1]
Z. Zou, K. Chen, Z. Shi, Y . Guo, J. Ye, Object detection in 20 years: A survey, Proceedings of the IEEE 111 (3) (2023) 257–276
work page 2023
-
[2]
Y . Zhou, A yolo-nl object detector for real-time detection, Expert Systems with Applications 238 (2024) 122256
work page 2024
-
[3]
Y . Hou, Z. Ma, C. Liu, C. C. Loy, Learning lightweight lane detection CNNs by self attention distillation, in: IEEE International Conference on Computer Vision, 2019, pp. 1013–1021
work page 2019
-
[4]
Y . Ma, T. Wang, X. Bai, H. Yang, Y . Hou, Y . Wang, Y . Qiao, R. Yang, X. Zhu, Vision-centric bev perception: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (12) (2024) 10978–10997
work page 2024
-
[5]
W. Li, W. Yang, Y . Hou, L. Liu, Y . Liu, X. Li, Saratr-x: Toward building a foundation model for sar target recognition, IEEE Transactions on Im- age Processing 34 (2025) 869–884
work page 2025
- [6]
-
[7]
Y . Hu, T. Delbruck, S.-C. Liu, Learning to ex- ploit multiple vision modalities by using grafted networks, in: European Conference on Computer Vision, Springer, 2020, pp. 85–101
work page 2020
-
[8]
B. Chakravarthi, A. A. Verma, K. Daniilidis, C. Fermuller, Y . Yang, Recent event camera inno- vations: A survey, in: European Conference on Computer Vision, Springer, 2024, pp. 342–376
work page 2024
-
[9]
D. Liu, Y . Fan, W. Lu, C. Liu, W. Zhang, Spiking depth: Depth estimation from sparse events with spiking neural networks, Expert Systems with Ap- plications (2025) 129977
work page 2025
-
[10]
D. Kang, D. Kang, An adaptive learning frame- work for event-based remote eye tracking, Expert Systems with Applications (2025) 128038
work page 2025
-
[11]
H. Cao, G. Chen, J. Xia, G. Zhuang, A. Knoll, Fusion-based feature attention gate component for vehicle detection based on event camera, IEEE Sensors Journal 21 (21) (2021) 24540–24548
work page 2021
-
[12]
A. Tomy, A. Paigwar, K. S. Mann, A. Renzaglia, C. Laugier, Fusing event-based and rgb camera for robust object detection in adverse conditions, in: 2022 International Conference on Robotics and Automation (ICRA), IEEE, 2022, pp. 933–939
work page 2022
-
[13]
D. Li, Y . Tian, J. Li, Sodformer: Streaming ob- ject detection with transformer using events and frames, IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (11) (2023) 14020– 14037
work page 2023
- [14]
-
[15]
J. Li, S. Dong, Z. Yu, Y . Tian, T. Huang, Event- based vision enhanced: A joint detection frame- work in autonomous driving, in: 2019 ieee interna- tional conference on multimedia and expo (icme), IEEE, 2019, pp. 1396–1401
work page 2019
- [16]
- [17]
- [19]
-
[20]
Y . Peng, Y . Zhang, P. Xiao, X. Sun, F. Wu, Better and faster: Adaptive event conversion for event-based object detection, in: Proceedings of the AAAI Conference on Artificial Intelligence, V ol. 37, 2023, pp. 2056–2064
work page 2023
- [21]
- [22]
-
[23]
S. Schaefer, D. Gehrig, D. Scaramuzza, Aegnn: Asynchronous event-based graph neural networks, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12371–12381
work page 2022
- [24]
-
[25]
S. Tulyakov, D. Gehrig, S. Georgoulis, J. Erbach, M. Gehrig, Y . Li, D. Scaramuzza, Time lens: Event-based video frame interpolation, in: Pro- ceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 2021, pp. 16155–16164
work page 2021
-
[26]
P. Duan, Z. W. Wang, B. Shi, O. Cossairt, T. Huang, A. K. Katsaggelos, Guided event filter- ing: Synergy between intensity images and neu- romorphic events for high performance imaging, IEEE Transactions on Pattern Analysis and Ma- chine Intelligence 44 (11) (2021) 8261–8275
work page 2021
- [27]
- [28]
-
[29]
Y .-F. Zuo, J. Yang, J. Chen, X. Wang, Y . Wang, L. Kneip, Devo: Depth-event camera visual odom- etry in challenging conditions, in: 2022 Interna- tional Conference on Robotics and Automation (ICRA), IEEE, 2022, pp. 2179–2185
work page 2022
-
[30]
L. Gao, Y . Liang, J. Yang, S. Wu, C. Wang, J. Chen, L. Kneip, Vector: A versatile event- centric benchmark for multi-sensor slam, IEEE Robotics and Automation Letters 7 (3) (2022) 8217–8224
work page 2022
-
[31]
M. Liu, N. Qi, Y . Shi, B. Yin, An attention fusion network for event-based vehicle object detection, in: 2021 IEEE International Conference on Image Processing (ICIP), IEEE, 2021, pp. 3363–3367
work page 2021
-
[32]
L. Fan, J. Yang, L. Wang, J. Zhang, X. Lian, H. Shen, Efficient spiking neural network for rgb– event fusion-based object detection, Electronics 14 (6) (2025) 1105
work page 2025
-
[33]
R. L. Graham, Concrete mathematics: a founda- tion for computer science, Pearson Education In- dia, 1994
work page 1994
-
[34]
A. Z. Zhu, L. Yuan, K. Chaney, K. Daniilidis, Un- supervised event-based learning of optical flow, depth, and egomotion, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 989–997
work page 2019
-
[35]
N. F. Chen, Pseudo-labels for supervised learning on dynamic vision sensor data, applied to object detection under ego-motion, in: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018, pp. 644–653
work page 2018
-
[36]
A. I. Maqueda, A. Loquercio, G. Gallego, N. Gar- cía, D. Scaramuzza, Event-based vision meets 14 deep learning on steering prediction for self- driving cars, in: Proceedings of the IEEE confer- ence on computer vision and pattern recognition, 2018, pp. 5419–5427
work page 2018
-
[37]
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pat- tern recognition, 2016, pp. 770–778
work page 2016
-
[38]
X. Zhu, W. Su, L. Lu, B. Li, X. Wang, J. Dai, Deformable detr: Deformable transform- ers for end-to-end object detection, arXiv preprint arXiv:2010.04159 (2020)
work page internal anchor Pith review Pith/arXiv arXiv 2010
- [39]
-
[40]
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Per- ona, D. Ramanan, P. Dollár, C. L. Zitnick, Mi- crosoft coco: Common objects in context, in: Eu- ropean conference on computer vision, Springer, 2014, pp. 740–755
work page 2014
-
[41]
S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, IEEE transactions on pattern analysis and machine intelligence 39 (6) (2016) 1137–1149
work page 2016
-
[42]
T.-Y . Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal loss for dense object detection, in: Proceed- ings of the IEEE international conference on com- puter vision, 2017, pp. 2980–2988
work page 2017
-
[43]
X. Zhou, D. Wang, P. Krähenbühl, Objects as points, arXiv preprint arXiv:1904.07850 (2019)
work page internal anchor Pith review Pith/arXiv arXiv 1904
-
[44]
X. Zhu, S. Lyu, X. Wang, Q. Zhao, Tph-yolov5: Improved yolov5 based on transformer prediction head for object detection on drone-captured sce- narios, in: Proceedings of the IEEE/CVF interna- tional conference on computer vision, 2021, pp. 2778–2788
work page 2021
-
[45]
C.-Y . Wang, A. Bochkovskiy, H.-Y . M. Liao, Yolov7: Trainable bag-of-freebies sets new state- of-the-art for real-time object detectors, in: Pro- ceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 2023, pp. 7464–7475
work page 2023
-
[46]
Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, S. Xie, A convnet for the 2020s, in: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 2022, pp. 11976–11986
work page 2022
-
[47]
Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, B. Guo, Swin transformer: Hierarchical vi- sion transformer using shifted windows, in: Pro- ceedings of the IEEE/CVF international confer- ence on computer vision, 2021, pp. 10012–10022
work page 2021
-
[48]
Q. Su, Y . Chou, Y . Hu, J. Li, S. Mei, Z. Zhang, G. Li, Deep directly-trained spiking neural net- works for object detection, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 6555–6565
work page 2023
-
[49]
Z. Liu, N. Yang, Y . Wang, Y . Li, X. Zhao, F.- Y . Wang, Enhancing traffic object detection in variable illumination with rgb-event fusion, IEEE Transactions on Intelligent Transportation Sys- tems (2024)
work page 2024
-
[50]
H. Cao, Z. Zhang, Y . Xia, X. Li, J. Xia, G. Chen, A. Knoll, Embracing events and frames with hier- archical feature refinement network for object de- tection, in: European Conference on Computer Vi- sion, Springer, 2024, pp. 161–177
work page 2024
- [51]
-
[52]
YOLOv3: An Incremental Improvement
J. Redmon, A. Farhadi, Yolov3: An incremen- tal improvement, arXiv preprint arXiv:1804.02767 (2018)
work page internal anchor Pith review Pith/arXiv arXiv 2018
-
[53]
M. Liu, M. Zhu, Mobile video object detection with temporally-aware feature maps, in: Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5686–5695
work page 2018
-
[54]
J. Li, J. Li, L. Zhu, X. Xiang, T. Huang, Y . Tian, Asynchronous spatio-temporal memory network for continuous event-based object detec- tion, IEEE Transactions on Image Processing 31 (2022) 2975–2987. 15
work page 2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.