Pith. sign in

REVIEW 2 major objections 6 minor 73 references

Offline robustness scores do not reliably flag the sensor corruptions that make autonomous cars fail on the road.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 10:59 UTC pith:5GICPC7Y

load-bearing objection Solid empirical demonstration that offline perturbation rankings only moderately predict closed-loop failures on a real vehicle; the gap is real on this platform even if generalizability is limited. the 2 major comments →

arxiv 2607.04953 v1 pith:5GICPC7Y submitted 2026-07-06 cs.SE cs.RO

Real-World Perturbation Testing of Autonomous Driving Systems

Stefano Carlo Lambertenghi , Matthias Weil , Andrea Stocco This is my paper
classification cs.SE cs.RO
keywords autonomous driving systemsperturbation testingrobustness evaluationcamera and LiDARhardware-in-the-loopvehicle-in-the-loopclosed-loop testingreal-world evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the usual way of stress-testing autonomous driving systems—corrupt camera images or LiDAR point clouds offline and measure model error—does not tell you which corruptions will actually produce bad driving. The authors run 72 camera and LiDAR perturbations through three stages: static model evaluation, hardware-in-the-loop replay on vehicle computers, and live closed-loop injection on a full-size car, covering both an end-to-end vision driver and a modular LiDAR perception-planning stack. They find a clear gap: camera corruptions with mild offline error can still cause unstable steering, understeer, oversteer, or collisions, while LiDAR perception loss is more consistent offline but only weakly predicts system-level failure. Real-time latency further filters which tests can even run on the vehicle. A reader who cares about safety should treat dataset-level robustness numbers as incomplete without complementary closed-loop, on-vehicle evaluation.

Core claim

Across both camera and LiDAR modalities, model-level metrics alone are insufficient to identify the most harmful perturbations. Robustness observations from recorded data do not consistently transfer to closed-loop behavior on a physical vehicle: camera perturbations with limited offline impact can still induce unstable control and failures in real driving, while LiDAR degradation is more consistent at the perception level but weakly predictive of system-level failures.

What carries the argument

A unified library of 72 camera and LiDAR perturbations executed at three nested levels—offline model evaluation, hardware-in-the-loop ROS replay, and live vehicle-in-the-loop injection—so the same corruptions can be compared from prediction error through real driving outcomes.

Load-bearing premise

The broad claim that offline metrics are systematically insufficient rests on one short urban route, one vehicle platform, fixed low speed, and two specific driving stacks.

What would settle it

Repeat the same multi-level protocol on several different routes, speeds, and ADS architectures; if offline rankings of harmful perturbations then reliably match vehicle-in-the-loop failure rankings (strong correlation and high top-k overlap), the transfer-gap claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Offline metrics such as steering MSE or detection retention will mis-rank which perturbations are safety-critical in closed-loop driving.
  • Real-time latency budgets must be part of any real-world perturbation suite; several published LiDAR weather models are too slow to run live.
  • Simple training on perturbed data can remove failures under rain, glare, mud, and wet roads while preserving nominal performance.
  • Vision and LiDAR stacks fail differently: camera systems show diverse control instability; LiDAR systems fail more often by collision after unstable detections.
  • Claiming robustness for ADS requires complementary physical closed-loop evaluation, not dataset scores alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Certification processes that rely mainly on static corruption benchmarks may systematically under-detect closed-loop failure modes.
  • Even high-fidelity simulators will leave the same gap unless they reproduce vehicle dynamics, sensor timing, and feedback tightly enough to match live ViL rankings.
  • End-to-end stacks need control-stability metrics under perturbation; modular stacks need temporal detection stability, not only frame-wise recall.
  • Extending this multi-level protocol to multi-route, multi-vehicle fleets is the natural test of whether the offline-to-online gap is structural.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper presents a large-scale empirical study of 72 camera and LiDAR perturbations applied to two ADS stacks (an end-to-end vision model and a modular LiDAR perception–planning stack) across three testing modalities: offline model-level evaluation, hardware-in-the-loop on vehicle compute, and closed-loop vehicle-in-the-loop driving on a full-scale Passat. Using MSE/max deviation for camera steering, retention/ATE for LiDAR detections, latency budgets, completion rates, failure-type taxonomies, CTE, jitter, and near-field coverage, the authors report that offline model-level metrics transfer only moderately (Spearman ρ ≈ 0.40–0.56) and with weak top-k agreement to closed-loop failures, that real-time feasibility filters many published LiDAR weather models, and that perturbation-based fine-tuning of the vision model improves robustness under real weather while preserving nominal performance. The central claim is that model-level metrics alone are insufficient to identify the most harmful perturbations and that recorded-data robustness does not consistently transfer to physical closed-loop behavior.

Significance. If the reported transfer gap holds under the studied conditions, the work is a substantial contribution to software engineering for ADS: it is among the first systematic cross-modality, cross-level robustness studies that include real full-scale closed-loop driving rather than only datasets or simulation. Strengths include a unified ROS perturbation library spanning 72 techniques, transparent real-time cut-offs (33 ms camera / 200 ms LiDAR), quantified transfer analysis with bootstrap CIs and BH correction, failure-hotspot clustering, and a practical demonstration that simple synthetic fine-tuning can reduce real-weather failures. These results give concrete evidence that offline robustness rankings can mislead system-level risk assessment and that runtime feasibility is a first-class constraint for on-vehicle testing—findings that are actionable for both researchers and practitioners building ADS test pipelines.

major comments (2)
  1. [Abstract; §4.2; §5 Transfer analysis; §8 Conclusions] The abstract, introduction, and conclusions state quite generally that “model-level metrics alone are insufficient” and that offline observations “do not consistently transfer” to closed-loop physical behavior. The supporting evidence (Table 1, RQ3 failure profiles, Section 5 Spearman/top-k analysis) is convincing for the studied platform, but all ViL results come from a single 450 m urban route without intersections, fixed 20 km/h, one vehicle, and two specific stacks (Section 4.2; Threats to Validity). This design is already acknowledged, yet the breadth of the claim still exceeds the sampling frame. Please either (a) rephrase the central claims to the scope of the evidence (these systems, this route class, these severity schedules) or (b) add at least one additional route/environment or speed regime that shows the same qualitative mismatch, so the transfer-gap conclusion is not carrie
  2. [§4.4.3 RQ3; Table 1; §5 Transfer analysis] ViL coverage is 119 runs across 72 perturbations and multiple severities (Section 4.4.5). For many camera/LiDAR entries in Table 1 the outcome vector is a single severity ladder rather than repeated trials, so failure-type labels and completion rates have no reported run-to-run variance. The transfer analysis therefore pools perturbation–intensity pairs that may be noisy. Please report the number of independent ViL repetitions per (perturbation, severity) cell used in the Spearman and top-k calculations, or add a short sensitivity check (e.g., bootstrap over runs where multiples exist) so that the moderate ρ and low top-5 overlap cannot be attributed to single-shot outcomes.
minor comments (6)
  1. [Table 1] Table 1 is extremely dense (camera + LiDAR, many columns, mixed units). Consider splitting into two tables or moving latency/completion columns to an appendix figure so that the offline-vs-ViL mismatch is easier to read at a glance.
  2. [Figure 4; §5 Failure hotspots] Figure 4 hotspots are informative but the legend and semi-transparent noise points are hard to parse in grayscale; a small inset table of cluster centroids and dominant categories would help.
  3. [§4.2 Driving Scenario] Camera and LiDAR ViL use different sub-routes (B vs C). A one-sentence justification that the shared geometry still allows qualitative cross-modality comparison would reduce a possible confound.
  4. [§4.5.4 RQ4] RQ4 fine-tuning is only for the vision E2E model. Briefly note why an analogous LiDAR augmentation experiment was out of scope, or flag it as future work more explicitly.
  5. [§2; Table 1] A few identifiers are inconsistent in the text (e.g., C-C-XII vs C-CXII style). Normalize the compact IDs throughout.
  6. [§3.3 Perturbation Library] The Beer–Lambert survival formula is given without a numbered equation; numbering it would make the lightweight LiDAR approximations easier to cite.

Circularity Check

0 steps flagged

No circularity: independent offline vs. closed-loop measurements; transfer is empirical correlation, not definitional.

full rationale

This is a purely empirical SE study. Model-level metrics (MSE, max steering deviation, retention, ATE) are computed offline on recorded data; HiL latency and ViL outcomes (completion, failure type, CTE, jitter, coverage) are measured separately on hardware and on a physical vehicle. The central claim—that offline metrics do not reliably identify the most harmful perturbations—is supported by comparing these independently obtained quantities (Table 1; Section 5 Spearman ρ and top-k overlap), not by fitting a parameter and renaming it as a prediction. Fine-tuning (RQ4) trains on synthetic perturbations and evaluates on real weather, so the robustness gain is not forced by construction. Self-citations (e.g., PerturbationDrive) supply the library and prior methodology; they do not define or force the transfer gap. No equation, uniqueness theorem, or ansatz reduces a claimed result to its inputs. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

Empirical systems paper; load-bearing content is experimental design choices rather than mathematical axioms. Free parameters are thresholds and severity schedules chosen by the authors; domain assumptions are standard ADS sensing and control facts; the only invented entities are the lightweight real-time LiDAR weather approximations.

free parameters (5)
  • camera real-time latency budget = 33 ms
    33 ms hard cut-off used to exclude perturbations from ViL (Section 4.4.2); chosen to match typical camera frame rates rather than derived.
  • LiDAR real-time latency budget = 200 ms
    200 ms hard cut-off for ViL inclusion; engineering choice matching Velodyne rates.
  • CTE pre-failure truncation distance d = 5 m
    Last 5 m before failure discarded when computing CTE (Section 4.4.3); ad-hoc to avoid terminal instability.
  • severity level schedules = 5 / 3 levels
    Five levels for camera (from PerturbationDrive), three for LiDAR (unified by authors); discrete ordinals that determine which intensities are tested.
  • near-field detection coverage radius = 5 m
    5 m radius used for LiDAR coverage metric; slightly larger than vehicle length, chosen by authors.
axioms (4)
  • domain assumption Controlled synthetic perturbations of camera images and LiDAR point clouds are useful proxies for rare real-world adverse conditions.
    Stated in Introduction and Section 2; underpins the entire experimental program.
  • ad hoc to paper A single 450 m urban route without intersections, driven at fixed 20 km/h, is sufficiently representative to expose general transfer gaps between testing levels.
    Driving Scenario (Section 4.2) and Threats to Validity; the general claims rest on this limited setting.
  • domain assumption DAVE-2-style end-to-end vision and Autoware Mini modular LiDAR stacks are representative of the two major ADS architectural families.
    Objects of Study (Section 4.3); used to generalize findings across modalities.
  • ad hoc to paper Beer–Lambert-style stochastic attenuation plus near-field back-scatter is a faithful enough phenomenological model of weather effects for real-time LiDAR testing.
    Section 3.3; introduced to replace physics-based models that violate latency budgets.
invented entities (2)
  • Lightweight LiDAR weather approximations (L-AIV, L-AVI, L-AIX) no independent evidence
    purpose: Enable real-time injection of snow/rain/fog effects that full physics models cannot achieve under the 200 ms budget.
    Explicitly introduced by the authors in Section 2.2 and 3.3 as phenomenological substitutes; no independent external validation beyond the paper’s own latency and retention numbers.
  • ROS-based unified perturbation framework spanning model/HiL/ViL no independent evidence
    purpose: Provide a single execution interface for the 72 corruptions on the physical vehicle.
    Contribution claim in Introduction; engineering artifact whose independent utility is not demonstrated outside this study.

pith-pipeline@v1.1.0-grok45 · 28729 in / 3124 out tokens · 27834 ms · 2026-07-11T10:59:18.168129+00:00 · methodology

0 comments
read the original abstract

Autonomous Driving Systems (ADS) must operate reliably under diverse conditions, yet representative data for rare or adverse scenarios is difficult to obtain. Perturbation-based testing is widely used to assess robustness, but most studies focus on offline datasets or simulation, leaving open questions about how such results translate to real-world driving. We present a large-scale study of 72 camera and LiDAR perturbations, evaluated across three testing modalities: offline model-level analysis, hardware-in-the-loop execution, and closed-loop system-level testing on a full-scale autonomous vehicle. The study covers both an end-to-end vision-based driving model and a modular LiDAR-based perception and planning stack. Our results reveal a clear gap between testing levels. For camera-based systems, perturbations with limited offline impact can still induce unstable control and failures in real-world driving. For LiDAR-based systems, degradation is more consistent at the perception level but weakly predictive of system-level failures. Across both modalities, model-level metrics alone are insufficient to identify the most harmful perturbations. We further show that real-time feasibility is a key constraint in real-world testing, and that robustness observations obtained from recorded data do not consistently transfer to closed-loop behavior on a physical vehicle, highlighting the importance of complementary real-world, system-level evaluation.

Figures

Figures reproduced from arXiv: 2607.04953 by Andrea Stocco, Matthias Weil, Stefano Carlo Lambertenghi.

Figure 1
Figure 1. Figure 1: Examples of camera (top) and LiDAR (bottom) per [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Autonomous driving platform used in this work: [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Real-world testing scenario. 4.2 Driving Scenario Experiments are conducted in anonymized, a public urban street (450 m, two-way, 30 km/h, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Testing trajectories for camera (left) and LiDAR (right). [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 11 canonical work pages · 1 internal anchor

  1. [1]

    PerturbationDrive

    2023. PerturbationDrive. https://github.com/HannesLeonhard/ PerturbationDrive

    2023

  2. [2]

    Nourdine Aliane. 2025. A Survey of Open-Source Autonomous Driving Systems and Their Impact on Research.Information16, 4 (2025). https://www.mdpi.com/ 2078-2489/16/4/317

    2025

  3. [3]

    Anonymous authors. 2019. Anonymized work

    2019

  4. [4]

    Autonomous Driving Lab, University of Tartu. 2023. Vehicle. https://adl.cs.ut.ee/ lab/vehicle. Accessed: 2026-03-06

    2023

  5. [5]

    Jon Ayerdi, Asier Iriarte, Pablo Valle, Ibai Roman, Miren Illarramendi, and Aitor Arrieta. 2024. MarMot: Metamorphic Runtime Monitoring of Autonomous Driv- ing Systems.ACM Trans. Softw. Eng. Methodol.34, 1, Article 18 (Dec. 2024), 35 pages. doi:10.1145/3678171

    work page doi:10.1145/3678171 2024

  6. [6]

    Luciano Baresi, Davide Yi Xian Hu, Andrea Stocco, and Paolo Tonella. 2025. Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Models. InProceedings of 47th International Conference on Software Engineering (ICSE ’25). IEEE

    2025

  7. [7]

    Till Beemelmanns, Quan Zhang, Christian Geller, and Lutz Eckstein. 2024. Multi- Corrupt: A Multi-Modal Robustness Dataset and Benchmark of LiDAR-Camera Fusion for 3D Object Detection. In2024 IEEE Intelligent Vehicles Symposium (IV). 3255–3261. doi:10.1109/IV55156.2024.10588664

    doi:10.1109/iv55156.2024.10588664 2024

  8. [8]

    Matteo Biagiola, Andrea Stocco, Vincenzo Riccio, and Paolo Tonella. 2024. Two is better than one: digital siblings to improve autonomous driving testing.Empirical Softw. Engg.29, 4 (May 2024), 33 pages. doi:10.1007/s10664-024-10458-4

    doi:10.1007/s10664-024-10458-4 2024

  9. [9]

    Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba

    Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. 2016. End to End Learning for Self-Driving Cars. arXiv:1604.07316 [cs.CV]

    Pith/arXiv arXiv 2016

  10. [10]

    Boudette

    Neal E. Boudette. 2017. Tesla’s Self-Driving System Cleared in Deadly Crash. https://www.nytimes.com/2017/01/19/business/tesla-model-s-autopilot- fatal-crash.html

    2017

  11. [11]

    Caesar, V

    H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. 2020. nuScenes: A Multimodal Dataset for Autonomous Driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11621–11631

    2020

  12. [12]

    ROS Community. 2021. velodyne - ROS Wiki. https://wiki.ros.org/velodyne. Accessed: 2025-05-22

    2021

  13. [13]

    Victor Crespo-Rodriguez, Neelofar, and Aldeida Aleti. 2024. PAFOT: A Position- Based Approach for Finding Optimal Tests of Autonomous Vehicles. InProceed- ings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)(Lisbon, Portugal)(AST ’24). ACM, 159–170. doi:10.1145/3644032. 3644457

    work page doi:10.1145/3644032 2024

  14. [14]

    Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. 2020. RandAug- ment: Practical Automated Data Augmentation with a Reduced Search Space. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc

    2020

  15. [15]

    Sha, Victor Chung Sing Lee, and Sang Hyuk Son

    Penglin Dai, Kai Liu, Qingfeng Zhuge, Edwin H.-M. Sha, Victor Chung Sing Lee, and Sang Hyuk Son. 2016. Quality-of-Experience-Oriented Autonomous Intersection Control in Vehicular Networks.IEEE Transactions on Intelligent Transportation Systems17, 7 (2016), 1956–1967. doi:10.1109/TITS.2016.2514271

    work page doi:10.1109/tits.2016.2514271 2016

  16. [16]

    Samuel Dodge and Lina Karam. 2016. Understanding how image quality affects deep neural networks. In2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX). 1–6. doi:10.1109/QoMEX.2016.7498955

    work page doi:10.1109/qomex.2016.7498955 2016

  17. [18]

    Yinpeng Dong, Caixin Kang, Jinlai Zhang, Zijian Zhu, Yikai Wang, Xiao Yang, Hang Su, Xingxing Wei, and Jun Zhu. 2023. Benchmarking Robust- ness of 3D Object Detection to Common Corruptions in Autonomous Driving. arXiv:2303.11040 [cs.CV] https://arxiv.org/abs/2303.11040 nuscenes-C + pertur- bations

    Pith/arXiv arXiv 2023

  18. [19]

    dSPACE dSPACE. 2026. dSPACE Microautobox. https://www.dspace.com/de/ gmb/home.cfm

    2026

  19. [20]

    Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density- based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining(Portland, Oregon)(KDD’96). AAAI Press, 226–231

    1996

  20. [21]

    2022–2025

    Autoware Foundation. 2022–2025. Autoware Core/Universe. https://github.com/ autowarefoundation/autoware

    2022

  21. [22]

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. 2013. Vision meets Robotics: The KITTI Dataset.International Journal of Robotics Research (IJRR)(2013)

    2013

  22. [23]

    Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. 2018. Generalisation in humans and deep neural networks. InAdvances in Neural Information Processing Systems, Vol. 31. Curran Associates, Inc

    2018

  23. [24]

    Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. 2020. A survey of deep learning techniques for autonomous driving.Journal of Field Robotics37, 3 (2020), 362–386

    2020

  24. [25]

    David Grossman. 2018. Uber Self-Driving Car Kills Pedestrian in Ari- zona. https://www.popularmechanics.com/technology/infrastructure/a19482100/ uber-self-driving-car-kills-pedestrian-in-arizona/

    2018

  25. [26]

    Martin Hahner, Christos Sakaridis, Mario Bijelic, Felix Heide, Fisher Yu, Dengxin Dai, and Luc Van Gool. 2022. LiDAR Snowfall Simulation for Robust 3D Object Detection. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  26. [27]

    Fitash Ul Haq, Donghwan Shin, Shiva Nejati, and Lionel Briand. 2020. Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. InProceedings of 13th IEEE International Conference on Software Testing, Verification and Validation (ICST ’20). IEEE

    2020

  27. [28]

    Fitash Ul Haq, Donghwan Shin, Shiva Nejati, and Lionel Briand. 2021. Can offline testing of deep neural networks replace their online testing? a case study of automated driving systems.Empirical Software Engineering26, 5 (2021), 90

    2021

  28. [29]

    Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. arXiv:1903.12261 [cs.LG] https://arxiv.org/abs/1903.12261

    Pith/arXiv arXiv 2019

  29. [30]

    Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking Neural Network Ro- bustness to Common Corruptions and Perturbations. InInternational Conference on Learning Representations (ICLR)

    2019

  30. [31]

    Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan

    Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. 2020. AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty.Proceedings of the International Conference on Learning Representations (ICLR)(2020)

    2020

  31. [32]

    Hoffmann, Claire J

    Gabriel M. Hoffmann, Claire J. Tomlin, Michael Montemerlo, and Sebastian Thrun. 2007. Autonomous Automobile Trajectory Tracking for Off-Road Driving: Controller Design, Experimental Validation and Racing. In2007 American Control Conference. 2296–2301. doi:10.1109/ACC.2007.4282788

    doi:10.1109/acc.2007.4282788 2007

  32. [33]

    Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, An- drea Stocco, and Paolo Tonella. 2020. Taxonomy of real faults in deep learning systems. InProceedings of the ACM/IEEE 42nd international conference on software engineering. 1110–1121. ASE ’26, October 12–16, 2026, Munich, Germany Stefano Carlo Lambertenghi, Matthias Weil, and Andrea Stocco

    2020

  33. [34]

    2025.Tesla found partly to blame for fatal Autopilot crash

    Lily Jamali. 2025.Tesla found partly to blame for fatal Autopilot crash. BBC News. https://www.bbc.com/news/articles/c93dqpkwx4xo Accessed: 2026-02-03

    2025

  34. [35]

    Chiranjeevi Karri, José Machado da Silva, and Miguel Velhote Correia. 2023. Key Indicators to Assess the Performance of LiDAR-Based Perception Algorithms: A Literature Review.IEEE Access11 (2023), 109142–109168. doi:10.1109/ACCESS. 2023.3321912

    doi:10.1109/access 2023

  35. [36]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Opti- mization. arXiv:1412.6980 [cs.LG] https://arxiv.org/abs/1412.6980

    Pith/arXiv arXiv 2017

  36. [37]

    Philip Koopman and Michael Wagner. 2017. Autonomous Vehicle Safety: An Interdisciplinary Challenge.IEEE Intelligent Transportation Systems Magazine9, 1 (2017), 90–96. doi:10.1109/MITS.2016.2583491

    doi:10.1109/mits.2016.2583491 2017

  37. [38]

    Jan Laermann, Wojciech Samek, and Nils Strodthoff. 2019. Achieving Gener- alizable Robustness of Deep Neural Networks by Stability Training. InPattern Recognition. Springer, 360–373

    2019

  38. [39]

    Stefano Carlo Lambertenghi, Hannes Leonhard, and Andrea Stocco. 2025. Bench- marking Image Perturbations for Testing Automated Driving Assistance Systems. InProceedings of the 18th IEEE International Conference on Software Testing, Veri- fication and Validation (ICST ’25). IEEE, 12 pages

    2025

  39. [40]

    Stefano Carlo Lambertenghi and Andrea Stocco. 2024. Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing. In Proceedings of 17th IEEE International Conference on Software Testing, Verification and Validation (ICST ’24). IEEE, 12 pages

    2024

  40. [41]

    Stefano Carlo Lambertenghi, Mirena Flores Valdez, and Andrea Stocco. 2025. A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems. In40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 2808–2820. doi:10. 1109/ASE63991.2025.00230

    arXiv 2025

  41. [42]

    Damon Lavrinc. 2018. This Is How Bad Self-Driving Cars Suck In The Rain. https://jalopnik.com/this-is-how-bad-self-driving-cars-suck-in-the-rain- 1666268433

    2018

  42. [43]

    Hannes Leonhard, Stefano Carlo Lambertenghi, and Andrea Stocco. 2026. PerturbationDrive: A Framework for Perturbation-Based Testing of ADAS. arXiv:2603.23661 [cs.SE] https://arxiv.org/abs/2603.23661

    arXiv 2026

  43. [44]

    Yunge Li and Lanyu Xu. 2024. Panoptic Perception for Autonomous Driving: A Survey. arXiv:2408.15388 [cs.RO] https://arxiv.org/abs/2408.15388

    Pith/arXiv arXiv 2024

  44. [45]

    Dongjie Liu, Jin Zhao, Axin Xi, Xinnian Huang Chao Wang, Kuncheng Lai, and Chang Liu. 2020. Data Augmentation Technology Driven By Image Style Transfer in Self-Driving Car Based on End-to-End Learning.Computer Modeling in Engineering & Sciences122, 2 (2020), 593–617

    2020

  45. [46]

    Chengjie Lu, Shaukat Ali, and Tao Yue. 2024. EpiTESTER: Testing Autonomous Vehicles with Epigenetic Algorithm and Attention Mechanism.IEEE Transactions on Software Engineering(2024), 1–19. doi:10.1109/TSE.2024.3449429

    work page doi:10.1109/tse.2024.3449429 2024

  46. [47]

    Chengjie Lu, Tao Yue, Man Zhang, and Shaukat Ali. 2023. DeepQTest: Testing Autonomous Driving Systems with Reinforcement Learning and Real-world Weather Data. arXiv:2310.05170 [cs.SE] https://arxiv.org/abs/2310.05170

    Pith/arXiv arXiv 2023

  47. [48]

    Siyu Luan, Zonghua Gu, and Shaohua Wan. 2023. Efficient Performance Predic- tion of End-to-End Autonomous Driving Under Continuous Distribution Shifts Based on Anomaly Detection.Journal of Signal Processing Systems95, 12 (12 2023), 1455–1468. doi:10.1007/s11265-023-01893-5

    work page doi:10.1007/s11265-023-01893-5 2023

  48. [49]

    Ecker, Matthias Bethge, and Wieland Brendel

    Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, and Wieland Brendel. 2020. Benchmarking Robustness in Object Detection: Autonomous Driving when Win- ter is Coming. arXiv:1907.07484 [cs.CV] https://arxiv.org/abs/1907.07484

    Pith/arXiv arXiv 2020

  49. [50]

    Agnieszka Mikołajczyk and Michał Grochowski. 2018. Data augmentation for improving deep learning in image classification problem. In2018 International Interdisciplinary PhD Workshop (IIPhDW). 117–122. doi:10.1109/IIPHDW.2018. 8388338

    work page doi:10.1109/iiphdw.2018 2018

  50. [51]

    Norman Mu and Justin Gilmer. 2019. MNIST-C: A Robustness Benchmark for Computer Vision.CoRRabs/1906.02337 (2019). arXiv:1906.02337 https://arxiv. org/abs/1906.02337

    Pith/arXiv arXiv 2019

  51. [52]

    Neelofar Neelofar and Aldeida Aleti. 2024. Identifying and Explaining Safety- critical Scenarios for Autonomous Vehicles via Key Features.ACM Trans. Softw. Eng. Methodol.33, 4, Article 94 (April 2024), 32 pages. doi:10.1145/3640335

    work page doi:10.1145/3640335 2024

  52. [53]

    Neelofar Neelofar and Aldeida Aleti. 2024. Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE)(Lisbon, Portugal). ACM, Article 68, 12 pages. doi:10.1145/3597503.3623314

    doi:10.1145/3597503.3623314 2024

  53. [54]

    Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepXplore: Auto- mated Whitebox Testing of Deep Learning Systems. InProceedings of the 26th Symposium on Operating Systems Principles(Shanghai, China)(SOSP ’17). ACM, 1–18. doi:10.1145/3132747.3132785

    doi:10.1145/3132747.3132785 2017

  54. [55]

    Fabian Poggenhans, Jan-Hendrik Pauls, Johannes Janosovits, Stefan Orf, Max- imilian Naumann, Florian Kuhnt, and Matthias Mayr. 2018. Lanelet2: A High- Definition Map Framework for the Future of Automated Driving. InProc. IEEE Intell. Trans. Syst. Conf.Hawaii, USA. http://www.mrt.kit.edu/z/publ/download/ 2018/Poggenhans2018Lanelet2.pdf

    2018

  55. [56]

    2026.US opens probe after Waymo self-driving vehicle strikes child near school

    Reuters. 2026.US opens probe after Waymo self-driving vehicle strikes child near school. Reuters. https://www.reuters.com/world/us/us-opens-probe-after- waymo-self-driving-vehicle-strikes-child-near-school-2026-01-29/ Accessed: 2026-02-03

    2026

  56. [57]

    Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing machine learning based systems: a systematic mapping.Empirical Software Engineering25 (2020), 5193–5254

    2020

  57. [58]

    Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel

    Evgenia Rusak, Lukas Schott, Roland S. Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. 2020. A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions. InComputer Vision – ECCV 2020. Springer, 53–69

    2020

  58. [59]

    Lev Sorokin, Matteo Biagiola, and Andrea Stocco. 2025. Simulator Ensembles for Trustworthy Autonomous Driving Testing. arXiv:2503.08936 [cs.SE] https: //arxiv.org/abs/2503.08936

    Pith/arXiv arXiv 2025

  59. [60]

    2025.Tesla sued over Model S crash that killed three in New Jersey

    Jonathan Stempel. 2025.Tesla sued over Model S crash that killed three in New Jersey. Reuters. https://www.reuters.com/legal/litigation/tesla-sued-over-new- jersey-crash-model-s-that-killed-three-2025-06-23/ Accessed: 2026-02-03

    2025

  60. [61]

    Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2023. Mind the Gap! A Study on the Transferability of Virtual Versus Physical-World Testing of Autonomous Driving Systems.IEEE Transactions on Software Engineering49, 04 (apr 2023), 1928–1940. doi:10.1109/TSE.2022.3202311

    work page doi:10.1109/tse.2022.3202311 2023

  61. [62]

    Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2023. Model vs system level test- ing of autonomous driving systems: a replication and extension study.Empirical Software Engineering28, 3 (May 2023), 73. doi:10.1007/s10664-023-10306-x

    work page doi:10.1007/s10664-023-10306-x 2023

  62. [63]

    DF Swinehart. 1962. The beer-lambert law.Journal of chemical education39, 7 (1962), 333

    1962

  63. [64]

    Shuncheng Tang, Zhenya Zhang, Yi Zhang, Jixiang Zhou, Yan Guo, Shuang Liu, Shengjian Guo, Yan-Fu Li, Lei Ma, Yinxing Xue, and Yang Liu. 2023. A Survey on Automated Driving System Testing: Landscapes and Trends.ACM Trans. Softw. Eng. Methodol.32, 5, Article 124 (July 2023), 62 pages. doi:10.1145/3579642

    doi:10.1145/3579642 2023

  64. [65]

    Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: automated testing of deep-neural-network-driven autonomous cars. InProceedings of the 40th International Conference on Software Engineering(Gothenburg, Sweden) (ICSE ’18). ACM, 303–314. doi:10.1145/3180155.3180220

    doi:10.1145/3180155.3180220 2018

  65. [66]

    Autonomous Driving Lab University of Tartu. 2026. Autoware Mini. https:// github.com/UT-ADL/autoware_mini/tree/release/nodes/detection/lidar/cluster

    2026

  66. [67]

    Meriel von Stein, David Shriver, and Sebastian Elbaum. 2023. DeepManeuver: Adversarial Test Generation for Trajectory Manipulation of Autonomous Vehicles. IEEE Transactions on Software Engineering49, 10 (2023), 4496–4509. doi:10.1109/ TSE.2023.3301443

    arXiv 2023

  67. [68]

    Shuai Wang and Zhendong Su. 2019. Metamorphic Testing for Object Detection Systems.CoRRabs/1912.12162 (2019). arXiv:1912.12162 http://arxiv.org/abs/ 1912.12162

    Pith/arXiv arXiv 2019

  68. [69]

    Shuai Wang and Zhendong Su. 2021. Metamorphic object insertion for test- ing object detection systems. InProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering(Virtual Event, Australia)(ASE ’20). Association for Computing Machinery, New York, NY, USA, 1053–1065. doi:10.1145/3324884.3416584

    doi:10.1145/3324884.3416584 2021

  69. [70]

    Hyung-Jin Yoon, Hamidreza Jafarnejadsani, and Petros Voulgaris. 2023. Learning When to Use Adaptive Adversarial Image Perturbations Against Autonomous Vehicles.IEEE Robotics and Automation Letters8, 7 (2023), 4179–4186. doi:10. 1109/LRA.2023.3280813

    arXiv 2023

  70. [71]

    Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. 2020. A survey of autonomous driving: Common practices and emerging technologies. IEEE access8 (2020), 58443–58469

    2020

  71. [72]

    Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khur- shid. 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. InProceedings of the 33rd ACM/IEEE ASE (ASE ’18). ACM, 132–142. doi:10.1145/3238147.3238187

    doi:10.1145/3238147.3238187 2018

  72. [73]

    Xingyu Zhao, Javier Salido, Simos Gerasimou, and Radu Calinescu. 2025. On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles.arXiv preprint arXiv:2505.02274(2025). https://arxiv.org/abs/2505.02274 Supported by UK EPSRC New Investigator Award

    Pith/arXiv arXiv 2025

  73. [74]

    Husheng Zhou, Wei Li, Yuankun Zhu, Yuqun Zhang, Bei Yu, Lingming Zhang, and Cong Liu. 2018. DeepBillboard: Systematic Physical-World Testing of Au- tonomous Driving Systems. doi:10.48550/ARXIV.1812.10812

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1812.10812 2018