REVIEW 3 major objections 5 minor 29 references
MTevent: A Multi-Task Event Camera Dataset for 6D Pose Estimation and Moving Object Detection
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read MTevent is a new event-camera benchmark that provides ground-truth 6D poses and 3D bounding boxes for fast-moving objects, and its RGB-only baseline scores only 0.22 average recall.
desk verdict Useful new dataset for event-based robotics, but the event-camera ground truth rests on an unmeasured temporal transfer that needs to be fixed before the annotations can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the annotation pipeline that converts 200 Hz motion-capture tracks into camera-frame ground truth. Eye-in-hand calibration maps the tracked camera rig to the RGB camera's optical frame, while a per-object calibration, performed with the BOP manual annotation tool, aligns each object's tracked frame with its mesh geometric center. Object 6D poses are then transformed from the world frame into the optical frames of all three cameras, and segmentation masks for rigid objects come from projecting the 3D meshes; for humans and forklifts, masks come from Segment-Anything-2 and 3D boxes from the tracked markers. All annotations are generated at the RGB camera's frame rate, which means the 25 FPS scenes—with 25 ms exposure time—can carry motion blur that the paper acknowledges degrades RGB annotation accuracy, while asserting that the event-camera annotations remain accurate. An event camera is defined by asynchronous per-pixel brightness-change reporting, which is the property the dataset is designed to exploit.
What would settle it
Check the calibration by reprojecting the tracked 6D poses into high-contrast frames and measuring pixel error against manually clicked object corners; if the error exceeds a few pixels, the ground truth is biased. Second, in a fast-motion scene recorded with both 25 FPS and 100 FPS RGB, compare the 25 FPS labels against the 100 FPS labels: systematic disagreement during fast motion would disprove the paper's assertion that annotations generated at the RGB rate remain accurate for the event cameras.
Extended reading notes
Core claim
The paper's discovery is the dataset itself, presented as the missing combination of properties in event-camera benchmarks. MTevent provides 75 scenes averaging 16 seconds each, recorded with two synchronized DVXplorer event cameras (640×480, 10.2 cm baseline) and an RGB camera running at either 25 FPS or 100 FPS. The 16 rigid objects are larger than typical household items—ranging from 16 × 19 × 42 cm to 80 × 120 × 144 cm—and each has a 3D mesh (CAD for the Euro pallet, BundleSDF reconstructions for the rest) plus motion-capture-based 6D pose annotations. For non-rigid moving entities such as humans and a forklift, the dataset supplies 3D bounding boxes derived from marker tracks, and segmentation masks for those entities come from Segment-Anything-2. The authors evaluate 6D pose estimation on the 25 FPS RGB subset using FoundationPose with ground-truth masks and synthetic depth, obtaining an Average Recall of 0.2207 (VSD 0.1874, MSSD 0.1716, MSPD 0.3031), which they interpret as evidence that RGB-only methods are limited in these dynamic conditions.
Load-bearing premise
The accuracy of every pose and bounding-box label rests on the motion-capture-to-camera and object-to-mesh calibration chain, and the paper asserts rather than measures that event annotations stay accurate even when RGB annotations are unreliable.
Editorial extensions
If this is right
- With ground-truth masks and synthetic depth supplied, FoundationPose reaches only 0.22 average recall on MTevent's 25 FPS RGB subset, making the dataset a stress test in which RGB-only pose estimators visibly struggle under motion blur and clutter.
- Each scene bundles synchronized event streams, RGB frames, motion-capture tracks, 6D poses for 16 rigid objects, and 3D bounding boxes for all moving objects, so one dataset supports direct comparison across six perception tasks.
- The 100 FPS RGB scenes provide higher-temporal-resolution annotations, enabling controlled experiments on how frame rate and motion blur affect pose-estimation accuracy and annotation reliability.
- Because synthetic depth is rendered from the object meshes, depth-requiring pose pipelines can be evaluated on MTevent without a physical depth sensor.
- The overlap of the 16 objects with an existing RGB pose-tracking benchmark means models already tested on those objects can be transferred directly to the event-based setting.
Reading between the lines
- A straightforward test of the paper's motivation would be to run FoundationPose on the 100 FPS RGB scenes and compare average recall with the 25 FPS result; a large gap would confirm motion blur, rather than background clutter, as the main reason RGB pose estimation underperforms.
- Because the motion-capture system tracks at 200 Hz but labels are published at RGB frame rate, re-rendering the same annotations at motion-capture frequency would give event-based methods a higher-rate evaluation without any additional hardware.
- The claim that event annotations remain accurate even when RGB annotations lag could be quantified by measuring whether projected mesh edges align with event-image edges in fast-motion slices; the paper shows a visual example but no such metric.
- The same motion-capture-based annotation pipeline could be ported to outdoor high-speed platforms, replacing the indoor tracking system with GPS/RTK, to cover the 5-10 m/s regime the introduction motivates; the authors note the camera rig in MTevent moved only up to 2 m/s.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MTevent, a multi-task event-camera dataset for 6D pose estimation and moving-object detection in dynamic indoor scenes. The recording platform is a stereo pair of DVXplorer event cameras plus an RGB camera (either 25 FPS or 100 FPS), tracked by a Vicon motion-capture system at 200 Hz. The dataset contains 75 scenes with 16 rigid industrial and household objects; annotations include 6D poses and segmentation masks for rigid objects, and 3D bounding boxes for humans and a forklift. All annotations are generated at the RGB camera's frame rate and transferred to the event cameras. The authors report a FoundationPose baseline on RGB images using ground-truth masks and synthetic depth, achieving an Average Recall of 0.2207, and argue this highlights the limitations of RGB-based approaches in high-speed scenarios.
Significance. If the annotation quality is established, MTevent fills a genuine gap: existing event-camera datasets with independently moving objects are mostly limited to small household items or short ranges, while MTevent provides larger objects, distances of 2–7 m, occlusions, varying lighting, and MoCap-based ground truth for multiple tasks. The public release of data and toolkit, the use of standard calibration tools (Kalibr, e2vid), the BOP-compatible evaluation format, and the inclusion of a FoundationPose baseline are concrete strengths. The main weakness is that the accuracy of the event-camera ground truth is asserted rather than measured; because the dataset's core value is reliable event-frame labels, this missing validation is load-bearing.
major comments (3)
- [3.5, Fig. 7] The claim that event-camera annotations remain accurate is not supported by quantitative evidence. The paper states that all annotations are generated at the RGB camera's recording rate, which is 25 FPS in the lower-rate setup, and that event images are accumulated over 10 ms windows. With camera translation speeds up to 2 m/s (Fig. 8), an object or camera can move 4–8 cm between consecutive 25 FPS annotation times, and 1–2 cm within a 10 ms accumulation window. This is not negligible relative to the object sizes reported in Fig. 9 and to the 10 cm tolerance used in the BOP evaluation. The cited evidence, Fig. 7, shows RGB bounding boxes lagging behind the object, but it does not quantitatively demonstrate that the event-frame annotations are correct. Because the MoCap system provides 200 Hz poses, the authors should either (a) report a quantitative temporal-alignment analysis (e.g., reprojection error of meshes at event timestamps versus at RGB frame times, or a comparison against per-event-interval MoCap interpolation), or (b) release annotations interpolated to event timestamps. Without this, the central resource of the dataset is not verified.
- [3.1, 3.3, 3.5] The accuracy of all 6D pose annotations depends on an unvalidated calibration chain: Vicon tracking, eye-in-hand calibration between the MoCap-tracked camera system and the RGB optical frame, and object calibration aligning the MoCap-tracked frame with the mesh geometric center. The paper does not report calibration residuals or reprojection errors for any of these transformations. I request a quantitative validation, for example the mean/median reprojection error of projected object meshes into RGB and event frames on a held-out set of frames, or a comparison of MoCap-derived poses with an independent pose estimator on sampled frames. Without such numbers, the ground-truth poses cannot be assessed.
- [3.5] The 3D bounding box annotations for non-rigid objects (humans and forklift) are computed from individual MoCap marker positions, but the manuscript does not specify the marker placement protocol or the rule by which markers are converted to a 3D bounding box. Since moving-object detection is one of the two headline tasks, the accuracy of these boxes is load-bearing. The authors should provide a precise definition of the bounding box (e.g., axes from markers, extents, and coordinate frame), and report a quantitative evaluation, such as 3D IoU against manual annotations on a sample of frames.
minor comments (5)
- [Table 2] The table header repeats 'AR' and is confusing: 0.2207 is the mean of the three component scores, so the columns should be labeled 'AR (mean)', 'AR-VSD', 'AR-MSSD', and 'AR-MSPD'.
- [2, References] References [7] and [8] appear to be the same paper, and Section 2 cites both; one duplicate should be removed.
- [3.3] The sentence 'We collected the dataset in 2 research hall both are hangar buildings' is grammatically incomplete; it should state the number of halls and the building type more clearly.
- [Table 1] In Table 1, the row for EED lists 'UA Vs' under Environment, which appears to be a typo; also 'MoCAP' is inconsistently capitalized across the table.
- [4] The conclusion that RGB-based methods struggle in this setting is based on a single method, FoundationPose, evaluated with synthetic depth and ground-truth masks; I recommend framing this as a proof-of-concept baseline rather than a general statement about RGB approaches.
Circularity Check
No circularity: MTevent is a dataset/benchmark paper whose annotations come from external MoCap and mesh reconstruction, with no fitted parameter renamed as a prediction.
full rationale
MTevent is a dataset and evaluation paper, not a derivational claim. The 6D poses and 3D bounding boxes are obtained from an external Vicon MoCap system (200 Hz), BundleSDF reconstructions, and eye-in-hand calibration, none of which are defined in terms of the paper's own claims or evaluation metrics. The FoundationPose baseline is an external method evaluated on RGB images using the dataset's ground-truth masks, so there is no fitted input that is then called a prediction. The 'first dataset' statement is a novelty claim rather than a self-referential derivation. The assertion in Section 3.5 that event camera annotations 'remain accurate' is an empirical validity assumption about temporal alignment, but it is not circular: the paper does not define event annotation accuracy in terms of itself, nor does it use the assertion to justify the annotation pipeline by construction. The incidental mention of MR6D objects is not load-bearing and does not form a self-citation chain. Overall, the paper's central value is an independently constructed resource, and no circular step can be exhibited from its text.
Assumptions & free parameters
assumptions (5)
- domain assumption Vicon MoCap system tracks objects and camera at 200 Hz with millimeter-level accuracy
- domain assumption Object meshes reconstructed with BundleSDF are accurate enough for pose projection
- domain assumption Eye-in-hand and object calibrations are correct
- domain assumption Annotations at RGB frame rate are valid for event data
- domain assumption Segment Anything 2 masks for human and forklift are correct
Cite this review
Pith. "Pith review of MTevent: A Multi-Task Event Camera Dataset for 6D Pose Estimation and Moving Object Detection." pith.science (2026). https://pith.science/paper/VYHJUK4D
@misc{pith2026250511282,
author = {Pith},
title = {Pith review of: MTevent: A Multi-Task Event Camera Dataset for 6D Pose Estimation and Moving Object Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/VYHJUK4D}},
note = {Machine review of arXiv:2505.11282}
}
read the original abstract
Mobile robots are reaching unprecedented speeds, with platforms like Unitree B2, and Fraunhofer O3dyn achieving maximum speeds between 5 and 10 m/s. However, effectively utilizing such speeds remains a challenge due to the limitations of RGB cameras, which suffer from motion blur and fail to provide real-time responsiveness. Event cameras, with their asynchronous operation, and low-latency sensing, offer a promising alternative for high-speed robotic perception. In this work, we introduce MTevent, a dataset designed for 6D pose estimation and moving object detection in highly dynamic environments with large detection distances. Our setup consists of a stereo-event camera and an RGB camera, capturing 75 scenes, each on average 16 seconds, and featuring 16 unique objects under challenging conditions such as extreme viewing angles, varying lighting, and occlusions. MTevent is the first dataset to combine high-speed motion, long-range perception, and real-world object interactions, making it a valuable resource for advancing event-based vision in robotics. To establish a baseline, we evaluate the task of 6D pose estimation using NVIDIA's FoundationPose on RGB images, achieving an Average Recall of 0.22 with ground-truth masks, highlighting the limitations of RGB-based approaches in such dynamic settings. With MTevent, we provide a novel resource to improve perception models and foster further research in high-speed robotic vision. The dataset is available for download https://huggingface.co/datasets/anas-gouda/MTevent
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Levi Burner, Anton Mitrokhin, Cornelia Ferm ¨uller, and Yiannis Aloimonos. Evimo2: An event camera dataset for motion segmentation, optical flow, structure from motion, and visual inertial odometry in indoor scenes with monoc- ular or stereo algorithms. arXiv preprint arXiv:2205.03467,
-
[2]
M3ed: Multi-robot, multi-sensor, multi-environment event dataset
Kenneth Chaney, Fernando Cladera, Ziyun Wang, Anthony Bisulco, M Ani Hsieh, Christopher Korpela, Vijay Kumar, Camillo J Taylor, and Kostas Daniilidis. M3ed: Multi-robot, multi-sensor, multi-environment event dataset. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4016–4023, 2023. 2
work page 2023
-
[3]
Event-based dataset for the detection and classification of manufacturing assembly tasks
Laura Duarte and Pedro Neto. Event-based dataset for the detection and classification of manufacturing assembly tasks. Data in Brief, 54:110340, 2024. 2, 3
work page 2024
-
[4]
Real-time 6-dof pose estimation by an event-based camera using active led markers
Gerald Ebmer, Adam Loch, Minh Nhat Vu, Roberto Mecca, Germain Haessig, Christian Hartl-Nesic, Markus Vincze, and Andreas Kugi. Real-time 6-dof pose estimation by an event-based camera using active led markers. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8137–8146, 2024. 3
work page 2024
-
[5]
Guillermo Gallego, Tobi Delbr ¨uck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, J ¨org Conradt, Kostas Daniilidis, et al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154–180, 2020. 1
work page 2020
-
[6]
Vector: A versa- tile event-centric benchmark for multi-sensor slam
Ling Gao, Yuxuan Liang, Jiaqi Yang, Shaoxun Wu, Chenyu Wang, Jiaben Chen, and Laurent Kneip. Vector: A versa- tile event-centric benchmark for multi-sensor slam. IEEE Robotics and Automation Letters, 7(3):8217–8224, 2022. 3
work page 2022
-
[7]
Hypergraph-based multi-view action recognition using event cameras
Yue Gao, Jiaxuan Lu, Siqi Li, Yipeng Li, and Shaoyi Du. Hypergraph-based multi-view action recognition using event cameras. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 46(10):6610–6622, 2024. 3
work page 2024
-
[8]
Hypergraph-based multi-view action recognition using event cameras
Yue Gao, Jiaxuan Lu, Siqi Li, Yipeng Li, and Shaoyi Du. Hypergraph-based multi-view action recognition using event cameras. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 2024. 2, 3
work page 2024
Show all 29 references
-
[9]
Dsec: A stereo event camera dataset for driv- ing scenarios
Mathias Gehrig, Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. Dsec: A stereo event camera dataset for driv- ing scenarios. IEEE Robotics and Automation Letters, 6(3): 4947–4954, 2021. 2
2021
-
[10]
Edopt: Event-camera 6-dof dynamic object pose tracking
Arren Glover, Luna Gava, Zhichao Li, and Chiara Bar- tolozzi. Edopt: Event-camera 6-dof dynamic object pose tracking. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 18200–18206. IEEE, 2024. 3
2024
-
[11]
E-pose: A large scale event camera dataset for object pose estimation
Oussama Abdul Hay, Xiaoqian Huang, Abdulla Ayyad, Es- lam Sherif, Randa Almadhoun, Yusra Abdulrahman, Lak- mal Seneviratne, Abdulqader Abusafieh, and Yahya Zweiri. E-pose: A large scale event camera dataset for object pose estimation. Scientific Data, 12(1):245, 2025. 2, 3
2025
-
[12]
BOP: Benchmark for 6D object pose esti- mation
Tom ´aˇs Hodaˇn, Frank Michel, Eric Brachmann, Wadim Kehl, Anders Glent Buch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Man- hardt, Federico Tombari, Tae-Kyun Kim, Ji ˇr´ı Matas, and Carsten Rother. BOP: Benchmark for 6D object...
-
[13]
BOP challenge 2020 on 6D object localization
Tom ´aˇs Hoda ˇn, Martin Sundermeyer, Bertram Drost, Yann Labb´e, Eric Brachmann, Frank Michel, Carsten Rother, and Jiˇr´ı Matas. BOP challenge 2020 on 6D object localization. European Conference on Computer Vision Workshops (EC- CVW), 2020. 7
2020
-
[14]
Tum-vie: The tum stereo visual-inertial event dataset
Simon Klenk, Jason Chui, Nikolaus Demmel, and Daniel Cremers. Tum-vie: The tum stereo visual-inertial event dataset. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 8601–8608. IEEE, 2021. 2, 3
2021
-
[15]
Toward a large-scale multimodal event-based dataset for neuromorphic deep learning applica- tions
Sarah Leung, E Jared Shamwell, Christopher Maxey, and William D Nothwang. Toward a large-scale multimodal event-based dataset for neuromorphic deep learning applica- tions. In Micro-and Nanotechnology Sensors, Systems, and Applications X, pages 279–288. SPIE, 2018. 3
2018
-
[16]
Hybrid object tracking with events and frames
Zhichao Li, Nicola A Piga, Franco Di Pietro, Massimil- iano Iacono, Arren Glover, Lorenzo Natale, and Chiara Bar- tolozzi. Hybrid object tracking with events and frames. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9057–9064. IEEE, 2023. 3
2023
-
[17]
Line-based 6-dof object pose estimation and tracking with an event camera
Zibin Liu, Banglei Guan, Yang Shang, Qifeng Yu, and Lau- rent Kneip. Line-based 6-dof object pose estimation and tracking with an event camera. IEEE Transactions on Im- age Processing, 2024. 3
2024
-
[18]
Event-based moving object detection and tracking
Anton Mitrokhin, Cornelia Ferm ¨uller, Chethan Paramesh- wara, and Yiannis Aloimonos. Event-based moving object detection and tracking. In2018 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pages 1–9,
-
[19]
Ev-imo: Motion seg- mentation dataset and learning pipeline for event cameras
Anton Mitrokhin, Chengxi Ye, Cornelia Ferm ¨uller, Yian- nis Aloimonos, and Tobi Delbruck. Ev-imo: Motion seg- mentation dataset and learning pipeline for event cameras. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6105–6112, 2019. 2, 3
2019
-
[20]
Ma- vied: A multisensor automotive visual inertial event dataset
Giuseppe Mollica, Simone Felicioni, Marco Legittimo, Leonardo Meli, Gabriele Costante, and Paolo Valigi. Ma- vied: A multisensor automotive visual inertial event dataset. IEEE Transactions on Intelligent Transportation Systems, 25 (1):214–224, 2023. 2
2023
-
[21]
Learning to detect objects with a 1 megapixel event camera
Etienne Perot, Pierre De Tournemire, Davide Nitti, Jonathan Masci, and Amos Sironi. Learning to detect objects with a 1 megapixel event camera. Advances in Neural Information Processing Systems, 33:16639–16652, 2020. 2
2020
-
[22]
Sam 2: Segment anything in images and videos
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...
2024 arXiv
-
[23]
Events-to-video: Bringing modern computer vision to event cameras
Henri Rebecq, Ren ´e Ranftl, Vladlen Koltun, and Davide Scaramuzza. Events-to-video: Bringing modern computer vision to event cameras. IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), 2019. 4
2019
-
[24]
Extending kalibr: Cali- brating the extrinsics of multiple imus and of individual axes
Joern Rehder, Janosch Nikolic, Thomas Schneider, Timo Hinzmann, and Roland Siegwart. Extending kalibr: Cali- brating the extrinsics of multiple imus and of individual axes. In 2016 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 4304–4311. IEEE, 2016. 4
2016
-
[25]
BundleSDF: Neural 6-DoF tracking and 3D reconstruction of unknown objects
Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas M ¨uller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. BundleSDF: Neural 6-DoF tracking and 3D reconstruction of unknown objects. In CVPR, 2023. 4
2023
-
[26]
FoundationPose: Unified 6d pose estimation and tracking of novel objects
Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. FoundationPose: Unified 6d pose estimation and tracking of novel objects. In CVPR, 2024. 6
2024
-
[27]
Simulation modeling of highly dynamic omnidirectional mobile robots based on real-world data
Marvin Wiedemann, Ossama Ahmed, Anna Dieckh ¨ofer, Re- nato Gasoto, and S ¨oren Kerner. Simulation modeling of highly dynamic omnidirectional mobile robots based on real-world data. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 16923–16929,
2024
-
[28]
Event3dgs: Event-based 3d gaussian splatting for high-speed robot egomotion
Tianyi Xiong, Jiayi Wu, Botao He, Cornelia Fermuller, Yian- nis Aloimonos, Heng Huang, and Christopher A Metzler. Event3dgs: Event-based 3d gaussian splatting for high-speed robot egomotion. arXiv preprint arXiv:2406.02972, 2024. 1
2024 arXiv
-
[29]
The multi- vehicle stereo event camera dataset: An event camera dataset for 3d perception
Alex Zihao Zhu, Dinesh Thakur, Tolga ¨Ozaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Daniilidis. The multi- vehicle stereo event camera dataset: An event camera dataset for 3d perception. IEEE Robotics and Automation Letters, 3 (3):2032–2039, 2018. 2
2018
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.