REVIEW 2 major objections 30 references
Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation
T0 review · 2 major / 0 minor · reviewed 2026-07-11 · grok-4.5
Pith's one-line read Cam2Sim turns real driving camera recordings into closed-loop CARLA scenarios whose Gaussian-Splatting views make an end-to-end driver behave like the real runs.
desk verdict Useful end-to-end tool paper with a real closed-loop win on one urban log; the GS-overfitting stress test is fair but does not erase the contribution. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Cam2Sim pipeline: camera/pose extraction, OpenStreetMap road reconstruction, parked-vehicle placement, local Gaussian Splatting training, and closed-loop CARLA execution that feeds the GS-rendered camera stream to the system under test.
What would settle it
Repeat the identical closed-loop DAVE-2 protocol on a second, independently collected urban route whose visual appearance or geometry differs from the original street; if the GS-augmented scenario no longer completes while the real vehicle does, or if raw CARLA suddenly matches real behavior, the central claim fails.
Extended reading notes
Core claim
On a real urban recording, Cam2Sim-reconstructed CARLA scenarios rendered with Gaussian Splatting reduce the visual discrepancy relative to standard simulator images and produce closed-loop DAVE-2 behavior that matches real-world reference runs (3/3 completions), while the identical reconstructed scenario under raw CARLA rendering fails every run (3/3).
Load-bearing premise
That rebuilding one static urban street and training Gaussian Splatting on that same route, then testing one fixed-speed camera-only driver three times, is enough to show the method improves closed-loop testing realism in general.
Editorial extensions
If this is right
- Camera-based ADS that currently fail under synthetic rendering can be re-tested inside reconstructed real routes without new physical drives.
- Trajectory-replay and closed-loop logs become comparable across real, sim-only, and GS-rendered domains for the same geometry.
- Parked-vehicle and map assets extracted from ordinary camera logs become reusable CARLA scenarios for regression testing.
- Local Gaussian Splatting models can be swapped or retrained when a route's appearance changes without rebuilding the entire map.
Reading between the lines
- If the same pipeline works for multi-camera or LiDAR-only stacks, the bottleneck for trustworthy sim testing may shift from geometry fidelity to dynamic-object modeling.
- The 3/3 versus 0/3 completion gap suggests visual domain gap, not control or map error, was the dominant failure mode for this particular end-to-end model.
- Extending the method to moving traffic and weather would turn one-off route reconstructions into reusable digital twins for continuous integration of perception models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Cam2Sim is a tool paper that reconstructs playable CARLA scenarios from real-world front-camera images and poses, then augments them with local Gaussian Splatting so a camera-based ADS can be tested closed-loop on observations that resemble the original recording. The pipeline covers ROS extraction, parked-vehicle detection (FCOS3D or PointPillars plus manual refinement), OpenStreetMap/OpenDRIVE map generation, trajectory and asset export, COLMAP+Nerfstudio GS training with sky masks and coordinate alignment, and both trajectory-replay and closed-loop execution with a DAVE-2 model. On one 450 m Munich residential street the authors report geometric reconstruction mIoU of 0.774 over 3144 frames (road 0.847, cars 0.577), and closed-loop results in which GS-augmented simulation completes 3/3 runs with trajectories closer to real-world references while the simulation-only CARLA baseline fails 3/3 (Table 1). Image-generation quality metrics are left as unfinished TODOs.
Significance. If the central claim holds, Cam2Sim would be a practically useful bridge between real recordings and closed-loop CARLA testing for camera-based ADS, addressing a well-known sim-to-real limitation that pure scenario reconstruction or pure neural rendering alone do not solve. Strengths that should be credited include a complete, publicly released artifact (Apache-2.0 GitHub with ROS bag, ADS, scripts, and dependency documentation), a clear five-stage pipeline, quantified geometric fidelity on thousands of frames, and a stark closed-loop contrast (GS 3/3 complete vs sim-only 3/3 fail) against independent real-world reference runs. For a tool paper these engineering and reproducibility contributions are material even if the empirical scope remains preliminary.
major comments (2)
- Section 3.2 (Image Generation Quality) is unfinished: three full-reference metrics and their results table are still TODO placeholders, and the abstract/intro claim that GS 'reduces the visual gap' is therefore unsupported by reported numbers. Completing this subsection with the promised metrics (and preferably a modest off-trajectory or lighting-variation check) is load-bearing for attributing the closed-loop gains to improved visual fidelity rather than route-specific appearance memorization.
- Section 3 / Collection Scenario and Behavior Fidelity (Table 1): the closed-loop claim rests on a single 450 m residential route, one fixed-speed (20 km/h PID) camera-only DAVE-2 model, and n=3 runs per domain. Because GS is trained on the same route's frames/poses (Step 4) that later supply closed-loop viewpoints, the 3/3 vs 0/3 completion gap may partly reflect static-scene overfitting rather than generalizable sim-to-real reduction. At minimum the manuscript should acknowledge this scope limit and, if possible, add a small held-out viewpoint or second-route check; without that, the generality claim for 'closed-loop ADS testing' should be narrowed to the demonstrated setting.
Circularity Check
No circular derivation: Cam2Sim’s behavioral claims are empirical comparisons against independent real-world and simulation-only baselines, not quantities forced by GS training or self-citation.
full rationale
Cam2Sim is an empirical tool paper, not a first-principles derivation. The load-bearing claim—that GS-rendered observations improve closed-loop DAVE-2 fidelity relative to stock CARLA—is evaluated by completion rate, Fréchet distance, corridor violations, lateral excess, and steering jitter against real-world reference runs and a simulation-only baseline that never sees the GS model (Section 3.3, Table 1). Those external references and failure modes are not defined in terms of the GS fit, nor recovered by renaming a fitted parameter. Geometric mIoU (Section 3.1) likewise compares reconstructed semantic maps to real-world semantic observations from the same viewpoints. Self-citations (e.g., prior sim-to-real and ADS-testing work by overlapping authors) appear only as motivation and related work; none supply a uniqueness theorem or ansatz that forces the completion-rate result. Route-specific GS overfitting is a generalization/correctness concern, not circularity by construction. Image-quality metrics are unfinished (TO DO placeholders) but that incompleteness does not create a circular step. Score 0; steps empty.
Assumptions & free parameters
free parameters (4)
- GS training configuration (splatfacto / splatfacto-big, chunking, sky masks)
- Closed-loop target speed (20 km/h PID)
- Parked-vehicle detection and clustering thresholds / manual refinement
- Trajectory export convention (center vs rear-axle) and coordinate alignment
assumptions (5)
- domain assumption OpenStreetMap geometry retrieved for the trajectory or address is accurate enough to serve as the CARLA OpenDRIVE map for the target street.
- domain assumption The scene can be treated as static aside from the ego vehicle (parked cars only; no dynamic traffic or pedestrians that must be reconstructed).
- domain assumption A single camera-based DAVE-2 model with independent PID speed control is a sufficient system under test to assess closed-loop fidelity of the reconstructed observations.
- domain assumption Semantic IoU of road/car/background under trajectory replay is a meaningful proxy for geometric reconstruction quality usable for ADS testing.
- standard math Standard computer-vision and SfM tools (COLMAP, SegFormer sky masks, Nerfstudio) produce usable GS models when poses and frames are supplied.
Cite this review
Pith. "Pith review of Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation." pith.science (2026). https://pith.science/paper/FG73K7LT
@misc{pith2026260704770,
author = {Pith},
title = {Pith review of: Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation},
year = {2026},
howpublished = {\url{https://pith.science/paper/FG73K7LT}},
note = {Machine review of arXiv:2607.04770}
}
read the original abstract
Simulation-based testing enables safe and repeatable evaluation of autonomous driving systems, but its effectiveness is limited by the gap between synthetic simulator outputs and real-world camera observations. To address this problem, we present Cam2Sim, a tool that transforms real-world driving recordings into playable CARLA simulation scenarios. Starting from camera images and poses, Cam2Sim reconstructs road geometry, ego trajectories, parked vehicles, and simulation assets, and augments the reconstructed environment with Gaussian Splatting to render camera observations that resemble the original recording. The framework supports ROS-based data extraction, parked-vehicle detection, OpenStreetMap-based map generation, CARLA scenario construction, Gaussian Splatting training, trajectory replay, and closed-loop execution with a system under test. We validate Cam2Sim on a real-world urban-driving scenario with a camera-based end-to-end driving model, comparing reconstruction quality, image-generation quality, and closed-loop behavior against both a simulation-only baseline and the real-world target. Results show that Gaussian-Splatting-based rendering reduces the visual gap with respect to standard simulator rendering and improves behavioral similarity to the real-world reference runs. The artifact is publicly available at https: //github.com/ast-fortiss-tum/cam2sim, and a screencast showing the tool is available at https://youtu.be/KmZ74l1__lI
Figures
Reference graph
Works this paper leans on
-
[1]
Luciano Baresi, Davide Yi Xian Hu, Andrea Stocco, and Paolo Tonella. 2025. Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Models. InProceedings of 47th International Conference on Software Engineering (ICSE ’25). IEEE
2025
-
[2]
Matteo Biagiola, Andrea Stocco, Vincenzo Riccio, and Paolo Tonella. 2024. Two is better than one: digital siblings to improve autonomous driving testing.Empirical Cam2Sim : Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation ASE ’26, October 12–16, 2026, Munich, Germany Softw. Engg.29, 4 (May 2024), 33 pages. doi:10.1007/s10664-02...
-
[3]
Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. 2016. End to End Learning for Self-Driving Cars. arXiv:1604.07316 [cs.CV]
arXiv 2016
-
[4]
Martin Buechel, Malte Schellmann, Holger Rosier, Tobias Kessler, and Alois Knoll
-
[5]
Fortuna: Presenting the 5G-connected automated vehicle prototype of the project PROVIDENTIA.Researchgate. https://doi. org/10.13140/RG2, 24402.91842 (2019). doi:10.13140/rg.2.2.24402.91842
-
[6]
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele
-
[7]
In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
The Cityscapes Dataset for Semantic Urban Scene Understanding. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3213–3223. doi:10.1109/CVPR.2016.350
-
[8]
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. InProceedings of the 1st Annual Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 78), Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (Eds.). PMLR, 1–16. https://proceedings.mlr.press/v78/dosovit...
2017
Show all 30 references
-
[9]
Overpass-API Drolbr. 2024. Drolbr/overpass-API: A database engine to query the OpenStreetMap data
2024
-
[10]
Alessio Gambi, Tri Huynh, and Gordon Fraser. 2019. Automatically Recon- structing Car Crashes from Police Reports for Testing Self-Driving Cars. In2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Pro- ceedings (ICSE-Companion). 290–291. doi:10.110...
2019 doi
-
[11]
Yuan Gao, Mattia Piccinini, Yuchen Zhang, Dingrui Wang, Korbinian Moller, Roberto Brusnicki, Baha Zarrouki, Alessio Gambi, Jan Frederik Totz, Kai Storms, Steven Peters, Andrea Stocco, Bassam Alrifaee, Marco Pavone, and Johannes Betz. 2026. Foundation Models in Autonomous Drivi...
2026 doi
-
[12]
Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, An- drea Stocco, and Paolo Tonella. 2020. Taxonomy of real faults in deep learning systems. InProceedings of the ACM/IEEE 42nd international conference on software engineering. doi:10.1145/3377811.3380395
2020 doi
-
[13]
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis
-
[14]
Graph.42, 4, Article 139 (July 2023), 14 pages
3D Gaussian Splatting for Real-Time Radiance Field Rendering.ACM Trans. Graph.42, 4, Article 139 (July 2023), 14 pages. doi:10.1145/3592433
2023 doi
-
[15]
Stefano Carlo Lambertenghi, Mirena Flores Valdez, and Andrea Stocco. 2025. A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 2808–2820. doi:10.1109/ASE63991.2025.00230
2025 doi
-
[16]
Stefano Carlo Lambertenghi and Andrea Stocco. 2024. Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE Computer Society, Los Alamitos, CA, USA, 1...
2024 doi
-
[17]
Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom
Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. 2019. PointPillars: Fast Encoders for Object Detection From Point Clouds. In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12689–12697. doi:10.1109/CVPR.2019.01298
2019 doi
-
[18]
Hao LU, Tianshuo Xu, Wenzhao Zheng, Yunpeng Zhang, Wei Zhan, Dalong Du, Masayoshi Tomizuka, Kurt Keutzer, and Ying-Cong Chen. 2026. DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving. InThe Thirty- ninth Annual Conference on Neural Information Processi...
2026
-
[19]
Replication package
replication-package 2026. Replication package. https://github.com/ast-fortiss- tum/cam2sim
2026
-
[20]
Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing Machine Learning based Systems: A Systematic Mapping.Empirical Software Engineering(2020). doi:10.1007/s10664- 020-09881-0
2020 doi
-
[21]
Schönberger and Jan-Michael Frahm
Johannes L. Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4104–4113. doi:10.1109/CVPR.2016.445
2016 doi
-
[22]
Lev Sorokin, Matteo Biagiola, and Andrea Stocco. 2026. Simulator ensembles for trustworthy autonomous driving systems testing.Empirical Softw. Engg.31, 4 (Feb. 2026), 39 pages. doi:10.1007/s10664-026-10821-7
2026 doi
-
[23]
Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2023. Mind the Gap! A Study on the Transferability of Virtual Versus Physical-World Testing of Autonomous Driving Systems.IEEE Transactions on Software Engineering49, 04 (apr 2023). doi:10.1109/TSE.2022.3202311
2023 doi
-
[24]
Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2023. Model vs system level test- ing of autonomous driving systems: a replication and extension study.Empirical Software Engineering28, 3 (May 2023), 26 pages. doi:10.1007/s10664-023-10306-x
2023 doi
-
[25]
Andrea Stocco and Paolo Tonella. 2022. Confidence-driven weighted retraining for predicting safety-critical failures in autonomous driving systems.Journal of Software: Evolution and Process34, 10 (2022), e2386. doi:10.1002/smr.2386
2022 doi
-
[26]
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David Mcallis- ter, Justin Kerr, and Angjoo Kanazawa. 2023. Nerfstudio: A Modular Framework for Neural Radiance Field Development. InA...
2023 doi
-
[27]
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. 2021. FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection. In2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). 913–922. doi:10. 1109/ICCVW54120.2021.00107
2021
-
[28]
Alvarez, and Ping Luo
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. 2021. SegFormer: simple and efficient design for semantic segmentation with transformers. InProceedings of the 35th International Conference on Neural Information Processing Systems (NIPS ’21)....
2021
-
[29]
Ziyuan Zhong, Yun Tang, Yuan Zhou, Vania de Oliveira Neves, Yang Liu, and Baishakhi Ray. 2021. A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation. arXiv:2112.00964 [cs.SE]
2021 arXiv
-
[30]
Huixin Zhu, Zhili Zhang, Junyang Zhao, Hui Duan, Yao Ding, Xiongwu Xiao, and Junsong Yuan. 2024. Scene reconstruction techniques for autonomous driving: a review of 3D Gaussian splatting.Artificial Intelligence Review58 (2024). doi:10. 1007/s10462-024-10955-4
2024
Reviewed July 11, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.