Pith. sign in

REVIEW 2 major objections 30 references

Cam2Sim turns real driving camera recordings into closed-loop CARLA scenarios whose Gaussian-Splatting views make an end-to-end driver behave like the real runs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 13:46 UTC pith:FG73K7LT

load-bearing objection Useful end-to-end tool paper with a real closed-loop win on one urban log; the GS-overfitting stress test is fair but does not erase the contribution. the 2 major comments →

arxiv 2607.04770 v1 pith:FG73K7LT submitted 2026-07-06 cs.SE

Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation

Davide Jannussi , Stefano Carlo Lambertenghi , Constantin Carste , Andrea Stocco This is my paper
classification cs.SE
keywords autonomous drivingsim-to-real gapGaussian Splattingscenario reconstructionCARLAclosed-loop testingneural renderingend-to-end driving
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Simulation is the only practical way to stress-test autonomous drivers repeatedly, yet camera-based models often fail in the simulator because synthetic images look nothing like the road. Cam2Sim takes ordinary front-camera frames and poses from a real drive, rebuilds the road map, path, and parked cars inside CARLA, then trains local Gaussian Splatting models so the simulator can re-render those same viewpoints in a photorealistic style. In a closed-loop test on a 450 m residential street, a DAVE-2 steering network that always left the road under raw CARLA images completed every run when it saw the Gaussian-Splatting images and produced trajectories close to the real-world reference. The tool therefore claims that geometry reconstruction plus neural rendering is enough to shrink the sim-to-real gap for camera-driven systems and make simulation results trustworthy for testing.

Core claim

On a real urban recording, Cam2Sim-reconstructed CARLA scenarios rendered with Gaussian Splatting reduce the visual discrepancy relative to standard simulator images and produce closed-loop DAVE-2 behavior that matches real-world reference runs (3/3 completions), while the identical reconstructed scenario under raw CARLA rendering fails every run (3/3).

What carries the argument

Cam2Sim pipeline: camera/pose extraction, OpenStreetMap road reconstruction, parked-vehicle placement, local Gaussian Splatting training, and closed-loop CARLA execution that feeds the GS-rendered camera stream to the system under test.

Load-bearing premise

That rebuilding one static urban street and training Gaussian Splatting on that same route, then testing one fixed-speed camera-only driver three times, is enough to show the method improves closed-loop testing realism in general.

What would settle it

Repeat the identical closed-loop DAVE-2 protocol on a second, independently collected urban route whose visual appearance or geometry differs from the original street; if the GS-augmented scenario no longer completes while the real vehicle does, or if raw CARLA suddenly matches real behavior, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Camera-based ADS that currently fail under synthetic rendering can be re-tested inside reconstructed real routes without new physical drives.
  • Trajectory-replay and closed-loop logs become comparable across real, sim-only, and GS-rendered domains for the same geometry.
  • Parked-vehicle and map assets extracted from ordinary camera logs become reusable CARLA scenarios for regression testing.
  • Local Gaussian Splatting models can be swapped or retrained when a route's appearance changes without rebuilding the entire map.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same pipeline works for multi-camera or LiDAR-only stacks, the bottleneck for trustworthy sim testing may shift from geometry fidelity to dynamic-object modeling.
  • The 3/3 versus 0/3 completion gap suggests visual domain gap, not control or map error, was the dominant failure mode for this particular end-to-end model.
  • Extending the method to moving traffic and weather would turn one-off route reconstructions into reusable digital twins for continuous integration of perception models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. Cam2Sim is a tool paper that reconstructs playable CARLA scenarios from real-world front-camera images and poses, then augments them with local Gaussian Splatting so a camera-based ADS can be tested closed-loop on observations that resemble the original recording. The pipeline covers ROS extraction, parked-vehicle detection (FCOS3D or PointPillars plus manual refinement), OpenStreetMap/OpenDRIVE map generation, trajectory and asset export, COLMAP+Nerfstudio GS training with sky masks and coordinate alignment, and both trajectory-replay and closed-loop execution with a DAVE-2 model. On one 450 m Munich residential street the authors report geometric reconstruction mIoU of 0.774 over 3144 frames (road 0.847, cars 0.577), and closed-loop results in which GS-augmented simulation completes 3/3 runs with trajectories closer to real-world references while the simulation-only CARLA baseline fails 3/3 (Table 1). Image-generation quality metrics are left as unfinished TODOs.

Significance. If the central claim holds, Cam2Sim would be a practically useful bridge between real recordings and closed-loop CARLA testing for camera-based ADS, addressing a well-known sim-to-real limitation that pure scenario reconstruction or pure neural rendering alone do not solve. Strengths that should be credited include a complete, publicly released artifact (Apache-2.0 GitHub with ROS bag, ADS, scripts, and dependency documentation), a clear five-stage pipeline, quantified geometric fidelity on thousands of frames, and a stark closed-loop contrast (GS 3/3 complete vs sim-only 3/3 fail) against independent real-world reference runs. For a tool paper these engineering and reproducibility contributions are material even if the empirical scope remains preliminary.

major comments (2)
  1. Section 3.2 (Image Generation Quality) is unfinished: three full-reference metrics and their results table are still TODO placeholders, and the abstract/intro claim that GS 'reduces the visual gap' is therefore unsupported by reported numbers. Completing this subsection with the promised metrics (and preferably a modest off-trajectory or lighting-variation check) is load-bearing for attributing the closed-loop gains to improved visual fidelity rather than route-specific appearance memorization.
  2. Section 3 / Collection Scenario and Behavior Fidelity (Table 1): the closed-loop claim rests on a single 450 m residential route, one fixed-speed (20 km/h PID) camera-only DAVE-2 model, and n=3 runs per domain. Because GS is trained on the same route's frames/poses (Step 4) that later supply closed-loop viewpoints, the 3/3 vs 0/3 completion gap may partly reflect static-scene overfitting rather than generalizable sim-to-real reduction. At minimum the manuscript should acknowledge this scope limit and, if possible, add a small held-out viewpoint or second-route check; without that, the generality claim for 'closed-loop ADS testing' should be narrowed to the demonstrated setting.

Circularity Check

0 steps flagged

No circular derivation: Cam2Sim’s behavioral claims are empirical comparisons against independent real-world and simulation-only baselines, not quantities forced by GS training or self-citation.

full rationale

Cam2Sim is an empirical tool paper, not a first-principles derivation. The load-bearing claim—that GS-rendered observations improve closed-loop DAVE-2 fidelity relative to stock CARLA—is evaluated by completion rate, Fréchet distance, corridor violations, lateral excess, and steering jitter against real-world reference runs and a simulation-only baseline that never sees the GS model (Section 3.3, Table 1). Those external references and failure modes are not defined in terms of the GS fit, nor recovered by renaming a fitted parameter. Geometric mIoU (Section 3.1) likewise compares reconstructed semantic maps to real-world semantic observations from the same viewpoints. Self-citations (e.g., prior sim-to-real and ADS-testing work by overlapping authors) appear only as motivation and related work; none supply a uniqueness theorem or ansatz that forces the completion-rate result. Route-specific GS overfitting is a generalization/correctness concern, not circularity by construction. Image-quality metrics are unfinished (TO DO placeholders) but that incompleteness does not create a circular step. Score 0; steps empty.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central behavioral claim rests on standard ADS-testing assumptions (static parked traffic, OSM geometry, CARLA as execution substrate, DAVE-2 as representative camera E2E model) plus engineering choices in detection, map export, and local GS training. No new physical entities are postulated; free parameters are ordinary pipeline knobs rather than fitted universal constants.

free parameters (4)
  • GS training configuration (splatfacto / splatfacto-big, chunking, sky masks)
    Local Nerfstudio training choices and frame-interval selection control visual fidelity and are not derived from first principles; they are set for the target route.
  • Closed-loop target speed (20 km/h PID)
    Longitudinal control is fixed by hand for all domains; behavioral comparison depends on this operating point.
  • Parked-vehicle detection and clustering thresholds / manual refinement
    FCOS3D or PointPillars detections plus confidence-weighted averaging and optional GUI edits determine spawn poses that affect the reconstructed scene.
  • Trajectory export convention (center vs rear-axle) and coordinate alignment
    Hand-chosen mapping between recorded poses and CARLA vehicle transforms affects replay and closed-loop start state.
axioms (5)
  • domain assumption OpenStreetMap geometry retrieved for the trajectory or address is accurate enough to serve as the CARLA OpenDRIVE map for the target street.
    Map generation step (Section 2.2–2.3) treats OSM/Overpass output as ground truth for road layout.
  • domain assumption The scene can be treated as static aside from the ego vehicle (parked cars only; no dynamic traffic or pedestrians that must be reconstructed).
    Pipeline reconstructs parked vehicles and static GS content; future-work section acknowledges missing dynamic-object reconstruction.
  • domain assumption A single camera-based DAVE-2 model with independent PID speed control is a sufficient system under test to assess closed-loop fidelity of the reconstructed observations.
    All behavior results (Section 3.3, Table 1) use this one architecture and fixed speed.
  • domain assumption Semantic IoU of road/car/background under trajectory replay is a meaningful proxy for geometric reconstruction quality usable for ADS testing.
    Section 3.1 reports mIoU as the primary reconstruction metric.
  • standard math Standard computer-vision and SfM tools (COLMAP, SegFormer sky masks, Nerfstudio) produce usable GS models when poses and frames are supplied.
    GS preparation (Section 2.4) relies on these established pipelines without re-deriving them.

pith-pipeline@v1.1.0-grok45 · 13867 in / 3273 out tokens · 34745 ms · 2026-07-11T13:46:37.441798+00:00 · methodology

0 comments
read the original abstract

Simulation-based testing enables safe and repeatable evaluation of autonomous driving systems, but its effectiveness is limited by the gap between synthetic simulator outputs and real-world camera observations. To address this problem, we present Cam2Sim, a tool that transforms real-world driving recordings into playable CARLA simulation scenarios. Starting from camera images and poses, Cam2Sim reconstructs road geometry, ego trajectories, parked vehicles, and simulation assets, and augments the reconstructed environment with Gaussian Splatting to render camera observations that resemble the original recording. The framework supports ROS-based data extraction, parked-vehicle detection, OpenStreetMap-based map generation, CARLA scenario construction, Gaussian Splatting training, trajectory replay, and closed-loop execution with a system under test. We validate Cam2Sim on a real-world urban-driving scenario with a camera-based end-to-end driving model, comparing reconstruction quality, image-generation quality, and closed-loop behavior against both a simulation-only baseline and the real-world target. Results show that Gaussian-Splatting-based rendering reduces the visual gap with respect to standard simulator rendering and improves behavioral similarity to the real-world reference runs. The artifact is publicly available at https: //github.com/ast-fortiss-tum/cam2sim, and a screencast showing the tool is available at https://youtu.be/KmZ74l1__lI

Figures

Figures reproduced from arXiv: 2607.04770 by Andrea Stocco, Constantin Carste, Davide Jannussi, Stefano Carlo Lambertenghi.

Figure 1
Figure 1. Figure 1: Overview of Cam2Sim: (1) Data Extraction, (2) Dataset Processing, (3) Simulation Data Generation, (4) Gaussian [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Validation setup and results: (A) the ADS, (B) the autonomous-driving platform used to collect the target recording, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 2 canonical work pages

  1. [1]

    Luciano Baresi, Davide Yi Xian Hu, Andrea Stocco, and Paolo Tonella. 2025. Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Models. InProceedings of 47th International Conference on Software Engineering (ICSE ’25). IEEE

    2025

  2. [2]

    Matteo Biagiola, Andrea Stocco, Vincenzo Riccio, and Paolo Tonella. 2024. Two is better than one: digital siblings to improve autonomous driving testing.Empirical Cam2Sim : Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation ASE ’26, October 12–16, 2026, Munich, Germany Softw. Engg.29, 4 (May 2024), 33 pages. doi:10.1007/s10664-02...

    doi:10.1007/s10664-024-10458-4 2024

  3. [3]

    Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba

    Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. 2016. End to End Learning for Self-Driving Cars. arXiv:1604.07316 [cs.CV]

    Pith/arXiv arXiv 2016

  4. [4]

    Martin Buechel, Malte Schellmann, Holger Rosier, Tobias Kessler, and Alois Knoll

  5. [5]

    https://doi

    Fortuna: Presenting the 5G-connected automated vehicle prototype of the project PROVIDENTIA.Researchgate. https://doi. org/10.13140/RG2, 24402.91842 (2019). doi:10.13140/rg.2.2.24402.91842

    work page doi:10.13140/rg2 2019

  6. [6]

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele

  7. [7]

    In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    The Cityscapes Dataset for Semantic Urban Scene Understanding. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3213–3223. doi:10.1109/CVPR.2016.350

    doi:10.1109/cvpr.2016.350 2016

  8. [8]

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. InProceedings of the 1st Annual Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 78), Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (Eds.). PMLR, 1–16. https://proceedings.mlr.press/v78/dosovit...

    2017

  9. [9]

    Overpass-API Drolbr. 2024. Drolbr/overpass-API: A database engine to query the OpenStreetMap data

    2024

  10. [10]

    Alessio Gambi, Tri Huynh, and Gordon Fraser. 2019. Automatically Recon- structing Car Crashes from Police Reports for Testing Self-Driving Cars. In2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Pro- ceedings (ICSE-Companion). 290–291. doi:10.1109/ICSE-Companion.2019.00119

    doi:10.1109/icse-companion.2019.00119 2019

  11. [11]

    Yuan Gao, Mattia Piccinini, Yuchen Zhang, Dingrui Wang, Korbinian Moller, Roberto Brusnicki, Baha Zarrouki, Alessio Gambi, Jan Frederik Totz, Kai Storms, Steven Peters, Andrea Stocco, Bassam Alrifaee, Marco Pavone, and Johannes Betz. 2026. Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis.IEEE Open Journal of I...

    doi:10.1109/ojits.2026.3660686 2026

  12. [12]

    Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, An- drea Stocco, and Paolo Tonella. 2020. Taxonomy of real faults in deep learning systems. InProceedings of the ACM/IEEE 42nd international conference on software engineering. doi:10.1145/3377811.3380395

    doi:10.1145/3377811.3380395 2020

  13. [13]

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis

  14. [14]

    Graph.42, 4, Article 139 (July 2023), 14 pages

    3D Gaussian Splatting for Real-Time Radiance Field Rendering.ACM Trans. Graph.42, 4, Article 139 (July 2023), 14 pages. doi:10.1145/3592433

    doi:10.1145/3592433 2023

  15. [15]

    Stefano Carlo Lambertenghi, Mirena Flores Valdez, and Andrea Stocco. 2025. A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). 2808–2820. doi:10.1109/ASE63991.2025.00230

    doi:10.1109/ase63991.2025.00230 2025

  16. [16]

    Stefano Carlo Lambertenghi and Andrea Stocco. 2024. Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE Computer Society, Los Alamitos, CA, USA, 173–184. doi:10.1109/ICST60714.2024. 00024

    doi:10.1109/icst60714.2024 2024

  17. [17]

    Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom

    Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. 2019. PointPillars: Fast Encoders for Object Detection From Point Clouds. In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12689–12697. doi:10.1109/CVPR.2019.01298

    doi:10.1109/cvpr.2019.01298 2019

  18. [18]

    Hao LU, Tianshuo Xu, Wenzhao Zheng, Yunpeng Zhang, Wei Zhan, Dalong Du, Masayoshi Tomizuka, Kurt Keutzer, and Ying-Cong Chen. 2026. DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving. InThe Thirty- ninth Annual Conference on Neural Information Processing Systems. https:// openreview.net/forum?id=HF5A73jmxq

    2026

  19. [19]

    Replication package

    replication-package 2026. Replication package. https://github.com/ast-fortiss- tum/cam2sim

    2026

  20. [20]

    Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing Machine Learning based Systems: A Systematic Mapping.Empirical Software Engineering(2020). doi:10.1007/s10664- 020-09881-0

    doi:10.1007/s10664- 2020

  21. [21]

    Schönberger and Jan-Michael Frahm

    Johannes L. Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4104–4113. doi:10.1109/CVPR.2016.445

    doi:10.1109/cvpr.2016.445 2016

  22. [22]

    Lev Sorokin, Matteo Biagiola, and Andrea Stocco. 2026. Simulator ensembles for trustworthy autonomous driving systems testing.Empirical Softw. Engg.31, 4 (Feb. 2026), 39 pages. doi:10.1007/s10664-026-10821-7

    doi:10.1007/s10664-026-10821-7 2026

  23. [23]

    Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2023. Mind the Gap! A Study on the Transferability of Virtual Versus Physical-World Testing of Autonomous Driving Systems.IEEE Transactions on Software Engineering49, 04 (apr 2023). doi:10.1109/TSE.2022.3202311

    doi:10.1109/tse.2022.3202311 2023

  24. [24]

    Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2023. Model vs system level test- ing of autonomous driving systems: a replication and extension study.Empirical Software Engineering28, 3 (May 2023), 26 pages. doi:10.1007/s10664-023-10306-x

    doi:10.1007/s10664-023-10306-x 2023

  25. [25]

    Andrea Stocco and Paolo Tonella. 2022. Confidence-driven weighted retraining for predicting safety-critical failures in autonomous driving systems.Journal of Software: Evolution and Process34, 10 (2022), e2386. doi:10.1002/smr.2386

    work page doi:10.1002/smr.2386 2022

  26. [26]

    Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David Mcallis- ter, Justin Kerr, and Angjoo Kanazawa. 2023. Nerfstudio: A Modular Framework for Neural Radiance Field Development. InACM SIGGRAPH 2023 Conference Proceedings(Los Angeles, CA, USA)(SIGGRAPH ’23). Ass...

    doi:10.1145/3588432.3591516 2023

  27. [27]

    Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. 2021. FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection. In2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). 913–922. doi:10. 1109/ICCVW54120.2021.00107

    arXiv 2021

  28. [28]

    Alvarez, and Ping Luo

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. 2021. SegFormer: simple and efficient design for semantic segmentation with transformers. InProceedings of the 35th International Conference on Neural Information Processing Systems (NIPS ’21). Curran Associates Inc., Red Hook, NY, USA, Article 924, 14 pages

    2021

  29. [29]

    Ziyuan Zhong, Yun Tang, Yuan Zhou, Vania de Oliveira Neves, Yang Liu, and Baishakhi Ray. 2021. A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation. arXiv:2112.00964 [cs.SE]

    Pith/arXiv arXiv 2021

  30. [30]

    Huixin Zhu, Zhili Zhang, Junyang Zhao, Hui Duan, Yao Ding, Xiongwu Xiao, and Junsong Yuan. 2024. Scene reconstruction techniques for autonomous driving: a review of 3D Gaussian splatting.Artificial Intelligence Review58 (2024). doi:10. 1007/s10462-024-10955-4

    2024