REVIEW 4 major objections 6 minor 1 cited by
How Real is CARLAs Dynamic Vision Sensor? A Study on the Sim-to-Real Gap in Traffic Object Detection
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Models trained solely on CARLA's synthetic DVS event data reach only 4.26 mAP on real traffic streams, while adding real data lifts performance to 15.69 mAP, quantifying a substantial sim-to-real gap.
desk verdict A useful CARLA DVS traffic dataset and training-mix sweep, but the headline sim-to-real claim is untested because the test sets are fixed and fully real, and the split accounting does not add up. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The analysis is carried by seven training datasets built by time-aligning synthetic SeTraM clips (from CARLA's sensor.camera.dvs) with real eTram clips, so that only the real/synthetic mix changes while total duration stays around 2300 seconds. All models use the RVT-Small recurrent vision transformer, which ingests event streams in 1MPX-format .npy/.h5 files, and evaluation uses mAP and AP@50 on fixed validation and test sets. The time-based alignment is the operative design choice that lets the authors swap data sources while keeping duration fixed.
What would settle it
Measure event density, object-size distributions, and class balance in SeTraM versus eTram clips, and check the timestamp ranges of the training and test splits for overlap. If the synthetic clips turn out to have far fewer events per second, smaller objects, or shared sequence IDs with the test set, the gap in Table 4 would reflect data statistics or leakage rather than DVS fidelity.
Extended reading notes
Core claim
The central claim is that models trained exclusively on CARLA DVS synthetic data perform well when tested on synthetic-heavy data but degrade significantly as the test stream becomes more real, while models trained on real-world data generalize more strongly across domains. The evidence is a controlled sweep: seven training sets with identical total duration differ only in the proportion of real eTram data (0%, 14.3%, ..., 85.7%), with fixed validation and test sets. On the mixed real-plus-synthetic test set, mAP rises from 4.26 for the synthetic-only model to 15.69 for the most real-heavy model; the night-only test set shows the same direction, from 4.22 to 12.63. The authors present this as the first quantitative demonstration that CARLA's native DVS module, despite its controllability and photorealistic environment, leaves a substantial domain gap in traffic object detection.
Load-bearing premise
The paper's conclusion rests on the assumption that the synthetic and real clips are interchangeable except for sensor realism, meaning same viewpoint, classes, annotation quality, and difficulty, and that the training, validation, and test clips are disjoint.
Editorial extensions
If this is right
- Synthetic-only CARLA DVS training is insufficient for real-world traffic object detection; the best fully synthetic model reaches only 4.26 mAP on the mixed test set.
- Each increment of real training data yields measurable gains, so even small amounts of annotated real event data are valuable for closing the domain gap.
- Models trained on real-dominant data generalize better across day/night test conditions, suggesting domain adaptation should focus on transferring real-world robustness into synthetic training.
- SeTraM can serve as a controlled benchmark for future sim-to-real studies, but it should not be treated as a drop-in replacement for real event data.
- The near-linear trend on real test data implies a quantitative target: simulator improvements must raise synthetic-only performance toward the 15.69 mAP achieved with 85.7% real data.
Reading between the lines
- Because the paper aligns synthetic and real clips by time rather than by event count or object instances, the measured slope may partly reflect differences in event density and scene complexity, not purely sensor realism; re-running the sweep with event-count-matched subsets would test this.
- If the gap is driven by low-level event statistics such as sparsity, noise, and temporal uniformity rather than semantic content, then post-processing synthetic streams with realistic noise models or event-rate matching could recover much of the lost performance without new real data.
- The validation peak at intermediate real proportions suggests a practical recipe for domain adaptation: a minority of real data can anchor the target domain while synthetic data provides scale and diversity, pointing to a sweet-spot mix rather than a rule that more real data is always better.
- The event quality score mentioned in the paper could be applied to SeTraM clips to predict transferability before training, turning the binary result that a gap exists into a per-clip diagnostic.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SeTraM, a synthetic event-based traffic-monitoring dataset generated with CARLA's DVS, and trains an RVT-small detector on seven training sets that mix SeTraM and real eTram data in increasing real-data proportions (0% to 85.7%). It reports validation and test mAP/AP@50 and argues that the monotonic improvement in test performance as more real training data is added quantifies a sim-to-real gap, concluding that CARLA's synthetic events are not yet a substitute for real events in traffic object detection. The abstract additionally claims that synthetic-only models perform well on synthetic-heavy test sets and degrade as the real proportion in the test stream increases.
Significance. The empirical question—whether CARLA's built-in DVS can produce event streams useful for real traffic-object detection—is practically important, and the paper's controlled comparison across seven training compositions is a useful start. If the results were fully supported, the paper would provide a valuable benchmark for domain adaptation in neuromorphic traffic perception. The main strengths are the construction of a synthetic dataset with fixed overhead intersections, the use of a strong event-based detector (RVT), and the attempt to isolate the effect of real-data proportion by holding training duration constant. However, the headline claim about synthetic-heavy test sets is not measured by the reported experiments, and there is a data-accounting inconsistency that threatens the validity of the quantitative conclusions.
major comments (4)
- [Abstract; Section 3.3; Table 4] The abstract's central claim—that models trained solely on synthetic data 'perform well on synthetic-heavy test sets but suffer significant performance degradation as the proportion of real-world data increases'—is not tested by the reported experiments. Section 3.3 states that the validation and test sets remain fixed across all runs, and Section 3.1 describes the two test variants as subsets of real eTram data; in no experiment is the test-set composition varied. Table 4 therefore shows a training-data effect: a model trained with more real data scores higher on a fixed real test set. This is expected and does not quantify the sim-to-real gap of CARLA's DVS. To support the abstract claim, the authors would need to fix the training data (for example, Dataset #1, synthetic-only) and evaluate that fixed model on test sets with increasing real-data fraction, holding total duration and annotation format constant.
- [Section 3.1; Section 3.3; Table 2] There is a data-accounting inconsistency that risks train/test leakage. Section 3.3 gives seven groups of 333 seconds each, implying a total SeTraM duration of 2331 seconds (~38.9 minutes), which matches the '38 minutes' stated in Section 3.1. However, Dataset #1 in Table 2 uses approximately 2300 seconds of SeTraM for training, while Section 3.3 additionally allocates 320 seconds of SeTraM to validation and 160 seconds to test. The sum of these portions is 2780 seconds, which exceeds the stated total by roughly 450 seconds. Either the total duration, the split durations, or the disjointness of the splits is misreported; if the training and validation/test SeTraM clips overlap, the reported results would be invalid.
- [Section 5; Tables 3–4; Figure 9] The headline quantitative summary, 'an average slope of 0.115 mAP per unit increase in real data proportion,' cannot be reproduced from the reported tables. From Table 4, the mixed-test mAP increases from 4.26 (Dataset #1) to 15.69 (Dataset #7) over a real-proportion change of 0.857, which corresponds to roughly 13.3 mAP per unit (or about 0.133 per percentage point), not 0.115. Table 3's validation mAP is non-monotonic, so the slope cannot refer to that curve either. The authors should state exactly which curve the slope is fitted to and report the fitted value; as written, the central quantitative claim is unsupported.
- [Section 3.2; Section 3.3; Tables 2–4] The comparison does not establish that the observed performance gap is attributable to CARLA's DVS realism rather than to other dataset differences. Section 3.3 aligns SeTraM and eTram by temporal duration only, and no statistics are provided for event rate, event density, object-size distribution, class balance, annotation quality, camera perspective, or task difficulty. Without such characteristics, the monotone improvement in Table 4 could reflect differences in scene content or annotation protocols. This is load-bearing for the paper's claim to be a quantifiable sim-to-real gap rather than a dataset-comparison artifact.
minor comments (6)
- [Abstract; Section 1; Section 4] The abstract and introduction say a model is trained 'exclusively on synthetic data,' but Section 4 trains seven models on mixed datasets; only Dataset #1 is synthetic-only. The wording should be clarified.
- [Figure 9; Section 4.2] Figure 9's caption reads 'Validation mAP across models...' while the text describes it as test performance on fully real test sets; the caption and the referenced table should be made consistent.
- [Section 3.1; Section 3.3] The paper refers inconsistently to 'five daytime and two nighttime sequences' versus 'seven groups' of four intersection instances each; the terminology should be unified because a group is 333 seconds, not 80 seconds.
- [Throughout] The naming 'eTram' versus 'eTraM' is inconsistent (for example, Table 4 header versus Section 3.1), and the text contains typos such as 'CARLAs' in the abstract and '1M PXformat' in Section 3.2.
- [Section 4.1; Tables 3–4] No error bars or multiple seeds are reported; several adjacent configurations in Table 4 differ by less than 1.5 mAP, so the claimed monotonic trend should be supported by repeated runs or an explicit statement that the results are single-seed.
- [Table 3] Dataset #7's validation AP@75 collapses to 4.81 while its mAP is 12.21; the authors should discuss whether this is a metric artifact or a real failure mode of the heavily real-data model.
Circularity Check
No circular derivation; the claimed sim-to-real gap is an empirical measurement, though the abstract's 'synthetic-heavy test set' claim is not supported by the fixed-test design.
full rationale
This is an empirical benchmark study, not a derivation chain. The monotone rise in Table 4 mAP from 4.26 to 15.69 is a measured training-data effect on fixed real eTram test sets, and the validation slope of 0.115 is a descriptive summary, not a parameter used to generate predictions. No equation defines a target quantity in terms of its own inputs, and no fitted value is renamed as a prediction. The eTram, SeTraM, and EQS references include same-group work, but eTram is an independently captured published dataset and EQS is only mentioned as a future metric, so none of these citations is load-bearing. The abstract's statement that models were 'tested on varying combinations of synthetic and real-world event streams' is contradicted by Section 3.3, where validation and test sets are fixed; this is a correctness/validity concern about the central claim, but it is not circularity under the specified patterns.
Assumptions & free parameters
assumptions (4)
- domain assumption eTraM is a representative real-world event camera benchmark for traffic monitoring.
- domain assumption Time-based alignment makes SeTraM and eTraM domain-comparable while controlling for event density.
- domain assumption Training, validation, and test subsets are disjoint.
- domain assumption Single-run RVT training is stable enough that mAP differences are meaningful.
invented entities (1)
-
SeTraM dataset
Cite this review
Pith. "Pith review of How Real is CARLAs Dynamic Vision Sensor? A Study on the Sim-to-Real Gap in Traffic Object Detection." pith.science (2026). https://pith.science/paper/VG4EALOE
@misc{pith2026250613722,
author = {Pith},
title = {Pith review of: How Real is CARLAs Dynamic Vision Sensor? A Study on the Sim-to-Real Gap in Traffic Object Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/VG4EALOE}},
note = {Machine review of arXiv:2506.13722}
}
read the original abstract
Event cameras are gaining traction in traffic monitoring applications due to their low latency, high temporal resolution, and energy efficiency, which makes them well-suited for real-time object detection at traffic intersections. However, the development of robust event-based detection models is hindered by the limited availability of annotated real-world datasets. To address this, several simulation tools have been developed to generate synthetic event data. Among these, the CARLA driving simulator includes a built-in dynamic vision sensor (DVS) module that emulates event camera output. Despite its potential, the sim-to-real gap for event-based object detection remains insufficiently studied. In this work, we present a systematic evaluation of this gap by training a recurrent vision transformer model exclusively on synthetic data generated using CARLAs DVS and testing it on varying combinations of synthetic and real-world event streams. Our experiments show that models trained solely on synthetic data perform well on synthetic-heavy test sets but suffer significant performance degradation as the proportion of real-world data increases. In contrast, models trained on real-world data demonstrate stronger generalization across domains. This study offers the first quantifiable analysis of the sim-to-real gap in event-based object detection using CARLAs DVS. Our findings highlight limitations in current DVS simulation fidelity and underscore the need for improved domain adaptation techniques in neuromorphic vision for traffic monitoring.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 1 Pith paper
-
Event-RGB Adaptive Tracking for Nighttime Highway Perception
JEAT jointly associates RGB and event detections with NIS-adapted measurement noise, raising MOTA on unlit nighttime highways from 46% (RGB) / 69% (event) to 77% on a new CARLA dataset.
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:2404.10540 (2024)
Aliminati, M.R., Chakravarthi, B., Verma, A.A., Vaghela, A., Wei, H., Zhou, X., Yang, Y.: Sevd: Synthetic event-based vision dataset for ego and fixed traffic per- ception. arXiv preprint arXiv:2404.10540 (2024)
arXiv 2024
-
[2]
In: 2017 IEEE International Conference on Image Processing (ICIP)
Bi, Y., Andreopoulos, Y.: Pix2nvs: Parameterized conversion of pixel-domain video frames to neuromorphic vision streams. In: 2017 IEEE International Conference on Image Processing (ICIP). pp. 1990–1994. IEEE (2017)
work page 2017
-
[3]
Chakravarthi, B., Manoj Kumar, M., Pavan Kumar, B.: Event-based sensing for improved traffic detection and tracking in intelligent transport systems toward sustainable mobility. In: International Conference on Interdisciplinary Approaches in Civil Engineering for Sustainable Development. pp. 83–95. Springer (2023)
work page 2023
-
[4]
arXiv preprint arXiv:2408.13627 (2024)
Chakravarthi, B., Verma, A.A., Daniilidis, K., Fermuller, C., Yang, Y.: Recent event camera innovations: A survey. arXiv preprint arXiv:2408.13627 (2024)
arXiv 2024
-
[5]
In: Proceedings of the Computer Vision and Pattern Recognition Conference
Chanda, K., Verma, A., Vaghela, A., Yang, Y., Chakravarthi, B.: Event quality score (eqs): Assessing the realism of simulated event camera streams via distance in latent space. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 5105–5113 (2025)
work page 2025
-
[6]
In: Conference on robot learning
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: Carla: An open urban driving simulator. In: Conference on robot learning. pp. 1–16. PMLR (2017) This paper has been accepted at the 10th International Conference on Computer Vision & Image Processing (CVIP 2025) How Real is CARLA’s Dynamic Vision Sensor? 15
work page 2017
-
[7]
IEEE transactions on pattern analysis and machine intelligence 44(1), 154–180 (2020)
Gallego, G., Delbrück, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A.J., Conradt, J., Daniilidis, K., et al.: Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence 44(1), 154–180 (2020)
2020
-
[8]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Gehrig, M., Scaramuzza, D.: Recurrent vision transformers for object detection with event cameras. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13884–13893 (2023)
2023
Show all 25 references
-
[9]
In: European Conference on Computer Vision
Han, H., Lyu, J., Li, J., Wei, H., Li, C., Wei, Y., Chen, S., Ji, X.: Physical-based event camera simulator. In: European Conference on Computer Vision. pp. 19–35. Springer (2024)
2024
-
[10]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition
Hu, Y., Liu, S.C., Delbruck, T.: v2e: From video frames to realistic dvs events. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition. pp. 1312–1321 (2021)
2021
-
[11]
https://inivation.com/buy/, accessed: 2025-04-16
iniVation AG: Buy – inivation. https://inivation.com/buy/, accessed: 2025-04-16
2025
-
[12]
https://inivation.com/solutions/, accessed: 2025-04-16
iniVation AG: Solutions – inivation. https://inivation.com/solutions/, accessed: 2025-04-16
2025
-
[13]
arXiv preprint arXiv:2411.12250 (2024)
Jiang, X., Zhou, F., Lin, J.: Adv2e: Bridging the gap between analogue circuit and discrete frames in the video-to-events simulator. arXiv preprint arXiv:2411.12250 (2024)
2024 arXiv
-
[14]
Frontiers in Neuroscience 15, 702765 (2021)
Joubert, D., Marcireau, A., Ralph, N., Jolley, A., Van Schaik, A., Cohen, G.: Event camera simulator improvements via characterized parameters. Frontiers in Neuroscience 15, 702765 (2021)
2021
-
[15]
In: European Conference on Computer Vi- sion
Lin, S., Ma, Y., Guo, Z., Wen, B.: Dvs-voltmeter: Stochastic process-based event simulator for dynamic vision sensors. In: European Conference on Computer Vi- sion. pp. 578–593. Springer (2022)
2022
-
[16]
The International journal of robotics research36(2), 142–149 (2017)
Mueggler, E., Rebecq, H., Gallego, G., Delbruck, T., Scaramuzza, D.: The event- camera dataset and simulator: Event-based data for pose estimation, visual odome- try, and slam. The International journal of robotics research36(2), 142–149 (2017)
2017
-
[17]
Advances in Neural Information Processing Systems 33, 16639–16652 (2020)
Perot, E., De Tournemire, P., Nitti, D., Masci, J., Sironi, A.: Learning to detect ob- jects with a 1 megapixel event camera. Advances in Neural Information Processing Systems 33, 16639–16652 (2020)
2020
-
[18]
https://www.prophesee.ai/event- based-evaluation-kits/, accessed: 2025-04-16
Prophesee: Event-based vision evaluation kits. https://www.prophesee.ai/event- based-evaluation-kits/, accessed: 2025-04-16
2025
-
[19]
https://www.prophesee.ai/event-based-vision-more-applications/ (2024), ac- cessed: 2024-04-14
Prophesee: Event-based vision applications | prophesee. https://www.prophesee.ai/event-based-vision-more-applications/ (2024), ac- cessed: 2024-04-14
2024
-
[20]
https://docs.prophesee.ai/stable/ appli- cations.html (2024), accessed: 2025-04-16
Prophesee: Video to event gpu simulator. https://docs.prophesee.ai/stable/ appli- cations.html (2024), accessed: 2025-04-16
2024
-
[21]
In: Conference on robot learning
Rebecq, H., Gehrig, D., Scaramuzza, D.: Esim: an open event camera simulator. In: Conference on robot learning. pp. 969–982. PMLR (2018)
2018
-
[22]
IEEE Access (2024)
Shariff, W., Dilmaghani, M.S., Kielty, P., Moustafa, M., Lemley, J., Corcoran, P.: Event cameras in automotive sensing: A review. IEEE Access (2024)
2024
-
[23]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Verma, A.A., Chakravarthi, B., Vaghela, A., Wei, H., Yang, Y.: etram: Event- based traffic monitoring dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22637–22646 (2024)
2024
-
[24]
In: 2024 IEEE International Conference on Robotics and Automation (ICRA)
Zhang, Z., Cui, S., Chai, K., Yu, H., Dasgupta, S., Mahbub, U., Rahman, T.: V2ce: Video to continuous events simulator. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). pp. 12455–12461. IEEE (2024)
2024
-
[25]
In: 2021 IEEE international conference on com- putational photography (ICCP)
Zhu, A.Z., Wang, Z., Khant, K., Daniilidis, K.: Eventgan: Leveraging large scale image datasets for event cameras. In: 2021 IEEE international conference on com- putational photography (ICCP). pp. 1–11. IEEE (2021)
2021
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.