Benchmarking Image Perturbations for Testing Automated Driving Assistance Systems

Andrea Stocco; Hannes Leonhard; Stefano Carlo Lambertenghi

REVIEW 3 major objections 6 minor 72 references

Benchmarking Image Perturbations for Testing Automated Driving Assistance Systems

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Common synthetic image distortions expose ADAS perception failures and, after one epoch of fine-tuning, improve performance on unseen simulated weather.

desk verdict Useful ADAS perturbation benchmark and library, but the RQ2 generalization claim is confounded by mixing perturbations with new road content; a control condition is needed. read the letter →

arxiv 2501.12269 v1 pith:SBHI6DGG submitted 2025-01-21 cs.SE cs.CV

Stefano Carlo Lambertenghi , Hannes Leonhard , Andrea Stocco This is my paper · ORCID

classification cs.SEcs.CV

keywords imageperturbationsADASrobustnesstestingsemanticsegmentationend-to-enddrivingmodelsdataaugmentationdomaingeneralizationcontinuouslearningsimulation-based

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the same cheap synthetic visual distortions used to test neural-network robustness can also be used to retrain automated driving assistance systems (ADAS) for new operating conditions. The authors catalogued 38 perturbation categories from the literature, kept 32 that preserve driving-scene semantics, and evaluated them on a pixel-level scene-understanding model and an end-to-end lane-keeping and cruise-control model, both on individual images and inside two driving simulators. Every perturbation type exposed failures at some intensity, but the more consequential result is the retraining one: after one epoch of fine-tuning on perturbed images, scene-understanding accuracy in simulated fog and rain improved by roughly 40% to 70%, and driving success rates in several weather conditions rose by tens of percentage points, including from 5% to 85% in dark conditions. If these results hold, ADAS developers could adapt perception models to new environments with a few distorted frames rather than large collections of new real-world data.

What carries the argument

The mechanism is a catalogue of synthetic image perturbations—controlled distortions such as noise, blur, fog, snow, brightness and saturation changes, and cutout patches—applied at five intensity levels. For the driving model, the perturbations are injected into live simulator frames; for the segmentation model, they are applied to stored images. The perturbations are implemented in a modular library that works both offline and inside driving simulators, with a latency check that excludes any perturbation too slow for the simulator's frame rate. The generalization experiments then use a deliberately lightweight retraining step: one epoch of fine-tuning on perturbed images collected from newly generated roads, while an expert controller silently records ground-truth commands. The catalogue plus the one-epoch fine-tuning step together carry the paper's claim that cheap synthetic distortions can substitute for collecting new data.

What would settle it

Retrain the same baseline models on the same fine-tuning roads and newly collected driving data with the perturbations turned off, then compare success rates and segmentation overlap on the unseen-weather test sets; if the no-perturbation version shows the same gains, the claim that perturbations cause the improvement is falsified.

Watch

Extended reading notes

Core claim

The central discovery the paper argues for is that perturbation-based augmentation and continuous learning improve ADAS effectiveness in novel, unseen environments. The authors report that a single epoch of fine-tuning on images distorted by common perturbations raised mean segmentation overlap on all tested weather variants of a rendered driving dataset, with the largest gains in fog (from 0.365 to 0.520) and rain (from 0.385 to 0.650), while nominal-condition performance also improved (from 0.663 to 0.718). For the end-to-end driving model, fine-tuning with perturbed frames from newly generated roads raised success rates on unseen test roads under every tested weather configuration, for example from 5% to 85% in dark/overcast conditions and from 14% to 57% in fog, while reducing out-of-road failures. The authors interpret this as evidence that common, arguably non-naturalistic perturbations generalize to naturalistic environmental changes.

Load-bearing premise

The study attributes the generalization gains to the distortions, but the retraining step also exposed the model to new road layouts, and no comparison run used the same new roads without distortions.

Editorial extensions

If this is right

If the claim holds, a one-epoch augmentation step could replace large-scale adverse-weather data collection when adapting ADAS perception to a new operating domain.
The results imply that robustness test suites should report intensity-resolved results, since many perturbations only cause failures at higher intensities.
The results imply that no single perturbation ranking transfers across models or simulators, so system-level ADAS testing should use multiple platforms and metrics.
The observed trade-off between higher success rates and higher driving jitter implies that robustness retraining should be tuned with both functional and ride-quality metrics in mind.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same catalogue and latency filter could be applied to other perception tasks such as object detection or multi-task driving stacks, where the failure patterns may differ from segmentation and end-to-end control.
Editorial inference: a control condition that fine-tunes on the same newly generated roads without any perturbations would isolate whether the observed gains come from the distortions or simply from exposure to additional road layouts; the reported design does not include this condition.
Editorial inference: if non-naturalistic corruptions transfer to weather effects, then a small, deliberately diverse set of cheap corruptions may be a lower-cost alternative to generative data augmentation for domain shift.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the benchmark and library are real contributions; the generalization story is not. The RQ2 design changes two variables at once—perturbations and road/scenario content—so the abstract's causal claim that augmentation and continuous learning improve ADAS performance in unseen environments is not supported by the experiments as run.

The paper's strength is its RQ1 benchmark. It compares 32 perturbation types across semantic segmentation and end-to-end LK/ACC, at model and system level, with five intensity levels, a real-time latency check against simulator frame rates, and two simulators. The PerturbationDrive library and replication package are concrete, and Table II is a useful reference for which distortions disturb which kind of model. That alone is worth citing.

Soft spots, in order of severity. First, the RQ2 confound. For SegFormer, fine-tuning uses perturbed images from vKITTI Scenario 18, a scenario absent from the training split, tested on Scenario 20. For DAVE-2, fine-tuning uses new randomly generated roads with different curvature and length. There is no control condition fine-tuned on the same new data without perturbations. The nominal-condition gains (64% to 78% success; IoU 0.663 to 0.718) show the model learns something about new road content, so the gains cannot be causally assigned to perturbations. This is load-bearing because the abstract's main claim rests on it.

Second, the intro says 'all image perturbations expose ADAS failures,' which Table II directly contradicts: ZigZag, brightness, white balance, greyscale, and several others show no OR/OT failures and no meaningful metric movement. The abstract's 'all categories' is defensible only if 'category' means the eight perturbation groups, but the intro overclaims.

Third, minor: the abstract says 38 categories, then 32 retained; and the RQ2 fine-tuning is one epoch at maximum intensity, a detail that deserves a limitation statement.

Who should read it: anyone working on robustness testing or data augmentation for driving perception. The RQ1 benchmark is citable as-is. The RQ2 claims need a control condition and a rewrite. The paper is worth engaging with, but the current version should not be accepted without that fix.

Recommendation: send to peer review. The empirical scope and the library justify referee time. My own verdict on the current claims is skeptical, but the fix is clear: add an unperturbed fine-tuning control and adjust the claims. That is a substantial revision, not a desk reject.

Referee Report

3 major / 6 minor

Summary. The paper surveys the image-perturbation literature (38 perturbation types), implements 32 of them in a library called PerturbationDrive, and benchmarks them on two ADAS perception tasks: SegFormer semantic segmentation on vKITTI and a DAVE-2 LK/ACC controller in the Udacity and Donkey Car simulators. RQ1 measures how each perturbation at five intensity levels degrades IoU or driving success/failure metrics. RQ2 fine-tunes SegFormer and DAVE-2 on perturbation-augmented data and reports improved performance on nominal and weather-affected test sets. The abstract claims that all perturbation categories expose robustness failures and that perturbation-based augmentation and continuous learning significantly improve ADAS performance in novel, unseen environments.

Significance. The RQ1 benchmark is a potentially useful empirical resource: it covers more perturbation types than prior ADAS testing studies, runs both component-level and system-level evaluations, and ships a replication package (PerturbationDrive) plus experimental logs. The differential results across simulators and tasks are valuable for future benchmark design. However, the paper's second contribution, the generalization claim, is the main novelty in the abstract and is currently not supported by the experimental design: the fine-tuning intervention is confounded with exposure to new road/scenario data, and some test conditions overlap with the perturbations used for fine-tuning. If RQ2 is repaired with proper control conditions, the paper could make a meaningful contribution; as it stands, the causal statement about perturbations improving generalization should not be accepted.

major comments (3)

[Section III-D2, Table III] The RQ2 design changes two variables simultaneously, so the causal role of perturbations is not established. For LK/ACC, DAVE-2 (FT) is fine-tuned on 'additional perturbed data collected from the randomly generated roads' (Section III-D2), where the Fine-tuning roads are explicitly chosen to differ from both Training roads and Testing roads RQ2 (Section III-C2); DAVE-2 (N) is not fine-tuned on any data from those roads. For semantic segmentation, the extended SegFormer is fine-tuned on perturbed Scenario 18 while the original model is trained only on Scenarios 01/02/06 (Section III-C1). Because there is no control condition that fine-tunes on the same new roads or the same new scenario without perturbations, the Table III gains (for example, nominal LK/ACC success from 64% to 78% and SegFormer IoU from 0.663 to 0.718) are equally explainable as ordinary domain adaptation to new road geometry and expert demonstrations. The abstract's statement that 'the use of dataset augmentation and continuous learning significantly improves ADAS performance' cannot be attributed to perturbations on the basis of the reported experiments.
[Section III-C1, Section III-D2, Table III] The 'unseen environments' claim is weakened by overlap between the fine-tuning perturbation set and the test weather set. In Section III-D2, the SegFormer extended model is fine-tuned using 'the perturbation types identified in RQ1 at maximum intensity,' which includes Fog (C-III); Table III then reports the fog-row improvement from 0.365 to 0.520 as evidence of generalization to an unseen weather domain, although fog was part of the augmentation distribution. For LK/ACC, the fine-tuning data include Snow (C-II) and Brightness (C-IV), while the test rows include snow and dark/overcast, a brightness-like condition. The paper should either remove overlapping conditions from the 'unseen' evaluation or report them separately; otherwise the generalization result is partially an in-distribution robustness result.
[Section III-E, Table III] No statistical support is provided for the word 'significantly' in the abstract and in the RQ2 answer. Table III reports success rates over 15 roads; the nominal improvement from 64% to 78% corresponds to about two additional successful scenarios, yet no confidence intervals, repeated runs, or significance tests are reported. The IoU comparisons in the same table are also presented as point estimates without variance. The conclusions in Section III-E and Section IV-B should be tempered or supplemented with appropriate statistical analysis (e.g., bootstrap confidence intervals or paired tests over roads/weather frames).

minor comments (6)

[Table I and Section II-B] Table I's column headers label the categories as 'Focus Perturbations (C)' and 'Weather Perturbations (D)', but Section II-B describes Weather as category C and Distortion as category D; the table's C-I..C-V entries are weather perturbations. Please align the labels.
[Section III-E] The RQ1 results text mentions 'G-VIII' for Udacity jitter, but Table II has no G-VIII; this appears to be a typo for G-VII or G-VIb.
[Section III-D2 and Table III] Section III-D2 states that DAVE-2 (N) succeeds in 10 of 15 nominal scenarios, while Table III reports a 64% success rate (about 9.6 of 15); please clarify how the percentage is computed or round consistently.
[Section III-D2] The label 'Online Continuous-learning' is misleading because the procedure collects data offline with a shadow expert and then performs a single epoch of fine-tuning; there is no online or continuous update during deployment.
[Section V-B] Section V-B attributes a real-time adversarial attack to 'Wu et al. [35]', but reference [35] is Liu et al., 'Data augmentation technology driven by image style transfer'; the citation appears to be mismatched.
[Abstract and Section II-B] The abstract uses '38 categories of perturbations' while the body says '38 perturbations' and later '32 types'; please clarify the relationship between categories, perturbation types, and the 32 retained types.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports direct empirical measurements, and the RQ2 confounds are validity threats rather than circular reductions.

full rationale

The paper's derivation chain is empirical rather than formal: RQ1 effectiveness is measured by comparing perturbed and nominal outputs on fixed test sets (Table II), and RQ2 generalization is measured by comparing original and fine-tuned models on independent test roads and weather domains (Table III). No parameter is fitted to the target outcome and then renamed as a prediction, and no target result is assumed inside an input definition. The reader-identified weaknesses are real experimental confounds—RQ2 fine-tuning simultaneously exposes the model to new road layouts, and some fine-tuning perturbation categories (fog, snow, brightness) resemble the simulator weather test conditions—but a confound is an alternative explanation for an observed effect, not a circular derivation of the effect. Self-citations appear (e.g., [12], [19], [55], [57], [58], [71]) but none is load-bearing: they support background claims about ADAS testing practice and are not invoked to define or force the RQ1/RQ2 results. The replication package is offered as open evidence. Therefore, no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on simulation as a proxy for reality, on a subjective validity filter for perturbations, and on the assumption that fine-tuning improvements are caused by perturbations. The RQ2 claim is especially sensitive to the missing control condition.

free parameters (3)

Perturbation intensity levels
Five intensity levels per perturbation calibrated by subjective visual assessment of when the scene becomes unintelligible; this manual threshold affects every effectiveness measurement.
Real-time latency cutoff = 33.3 ms per frame
Perturbations slower than one frame at 30 fps (33.3 ms) are considered infeasible; Zoom blur is excluded. This cutoff is a design choice, not derived from system requirements.
Fine-tuning hyperparameters = one epoch; learning rate 6e-5 for SegFormer
Chosen without ablation; affects RQ2 improvement magnitudes.

assumptions (5)

domain assumption Simulated environments (vKITTI, Udacity, Donkey Car) are valid proxies for real-world ADAS operation.
All conclusions about robustness and generalization are drawn from simulation; threats to validity acknowledge real-world differences.
ad hoc to paper The selected perturbations preserve semantic content and produce valid driving images.
Validity filter was performed manually on a per-perturbation basis using visual inspection (Section II-C); no objective or inter-rater check is reported.
domain assumption IoU and success rate are appropriate oracles for ADAS perception and control quality.
The paper selects IoU for segmentation and success rate or completion for LK/ACC without systematic justification of oracle adequacy.
domain assumption DAVE-2 and SegFormer are representative ADAS perception systems.
The study uses one segmentation model and one imitation-learning controller; results may not generalize to other architectures or higher autonomy levels.
domain assumption The pure-pursuit and PID expert driver provides correct ground-truth driving behavior for imitation learning.
Used to collect training data for DAVE-2 in RQ2; any bias in the expert affects the learned policy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Benchmarking Image Perturbations for Testing Automated Driving Assistance Systems." pith.science (2026). https://pith.science/paper/SBHI6DGG

@misc{pith2026250112269,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Image Perturbations for Testing Automated Driving Assistance Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SBHI6DGG}},
  note         = {Machine review of arXiv:2501.12269}
}

read the original abstract

Advanced Driver Assistance Systems (ADAS) based on deep neural networks (DNNs) are widely used in autonomous vehicles for critical perception tasks such as object detection, semantic segmentation, and lane recognition. However, these systems are highly sensitive to input variations, such as noise and changes in lighting, which can compromise their effectiveness and potentially lead to safety-critical failures. This study offers a comprehensive empirical evaluation of image perturbations, techniques commonly used to assess the robustness of DNNs, to validate and improve the robustness and generalization of ADAS perception systems. We first conducted a systematic review of the literature, identifying 38 categories of perturbations. Next, we evaluated their effectiveness in revealing failures in two different ADAS, both at the component and at the system level. Finally, we explored the use of perturbation-based data augmentation and continuous learning strategies to improve ADAS adaptation to new operational design domains. Our results demonstrate that all categories of image perturbations successfully expose robustness issues in ADAS and that the use of dataset augmentation and continuous learning significantly improves ADAS performance in novel, unseen environments.

Figures

Figures reproduced from arXiv: 2501.12269 by the authors.

**Figure 1.** Valid and invalid perturbation types. with larger angles (e.g., θ = 180◦ ) inverting the image, while reflections (E-V) duplicate content, both of which may confuse DNNs and mislead the ADAS. Similarly, generative-based perturbations (H) were excluded. CycleGAN (H-I) relies on the input domain matching the source domain of its training dataset, making it unsuitable for general driving scenarios, while styletransfer… view at source ↗

**Figure 2.** Benchmarking perturbations. 1) RQ1: Semantic Segmentation. We first fine-tune a pretrained SegFormer model [64] for 10 epochs using the training set split of the vKITTI dataset (see Section III-C1) and the Adam optimizer with a learning rate of 6e −5 . Regarding the evaluation phase, we instructed PerturbationDrive to introduce controlled perturbations in the images of the Testing set (N). Each image was perturbed … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 61 canonical work pages

[1]

A Survey on Automated Driving System Testing: Landscapes and Trends,

S. Tang, Z. Zhang, Y . Zhang, J. Zhou, Y . Guo, S. Liu, S. Guo, Y .-F. Li, L. Ma, Y . Xue, and Y . Liu, “A Survey on Automated Driving System Testing: Landscapes and Trends,” ACM Trans. Softw. Eng. Methodol. , vol. 32, no. 5, Jul. 2023

work page 2023
[2]

A survey of autonomous driving: Common practices and emerging technologies,

E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” IEEE access, vol. 8, pp. 58 443–58 469, 2020

2020
[3]

Panoptic perception for autonomous driving: A survey,

Y . Li and L. Xu, “Panoptic perception for autonomous driving: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2408.15388

arXiv 2024
[4]

A survey of deep learning techniques for autonomous driving,

S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” Journal of Field Robotics , vol. 37, no. 3, pp. 362–386, 2020

work page 2020
[5]

Understanding how image quality affects deep neural networks,

S. Dodge and L. Karam, “Understanding how image quality affects deep neural networks,” in 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX) , 2016, pp. 1–6

work page 2016
[6]

Generalisation in humans and deep neural networks,

R. Geirhos, C. R. M. Temme, J. Rauber, H. H. Sch ¨utt, M. Bethge, and F. A. Wichmann, “Generalisation in humans and deep neural networks,” in Advances in Neural Information Processing Systems , vol. 31. Curran Associates, Inc., 2018

work page 2018
[7]

Benchmarking neural network ro- bustness to common corruptions and perturbations,

D. Hendrycks and T. Dietterich, “Benchmarking neural network ro- bustness to common corruptions and perturbations,” in International Conference on Learning Representations (ICLR) , 2019

2019
[8]

AugMix: A simple data processing method to improve robustness and uncertainty,

D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Laksh- minarayanan, “AugMix: A simple data processing method to improve robustness and uncertainty,” Proceedings of the International Conference on Learning Representations (ICLR) , 2020

work page 2020

Show all 72 references

[9]

A simple way to make neural networks robust against diverse image corruptions,

E. Rusak, L. Schott, R. S. Zimmermann, J. Bitterwolf, O. Bringmann, M. Bethge, and W. Brendel, “A simple way to make neural networks robust against diverse image corruptions,” in Computer Vision – ECCV
[10]

Achieving generalizable robustness of deep neural networks by stability training,

J. Laermann, W. Samek, and N. Strodthoff, “Achieving generalizable robustness of deep neural networks by stability training,” in Pattern Recognition. Springer, 2019, pp. 360–373

2019
[11]

Data augmentation for improving deep learning in image classification problem,

A. Mikołajczyk and M. Grochowski, “Data augmentation for improving deep learning in image classification problem,” in 2018 International Interdisciplinary PhD Workshop (IIPhDW) , 2018, pp. 117–122

2018
[12]

Mind the Gap! A Study on the Transferability of Virtual Versus Physical-World Testing of Autonomous Driving Systems,

A. Stocco, B. Pulfer, and P. Tonella, “Mind the Gap! A Study on the Transferability of Virtual Versus Physical-World Testing of Autonomous Driving Systems,” IEEE Transactions on Software Engineering , vol. 49, no. 04, pp. 1928–1940, apr 2023

1928
[13]

Marmot: Metamorphic runtime monitoring of autonomous driving systems,

J. Ayerdi, A. Iriarte, P. Valle, I. Roman, M. Illarramendi, and A. Arrieta, “Marmot: Metamorphic runtime monitoring of autonomous driving systems,” ACM Trans. Softw. Eng. Methodol. , vol. 34, no. 1, Dec. 2024

2024
[14]

Deepxplore: Automated whitebox testing of deep learning systems,

K. Pei, Y . Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” in Proceedings of the 26th Symposium on Operating Systems Principles , ser. SOSP ’17. ACM, 2017, p. 1–18

2017
[15]

Deeptest: automated testing of deep-neural-network-driven autonomous cars,

Y . Tian, K. Pei, S. Jana, and B. Ray, “Deeptest: automated testing of deep-neural-network-driven autonomous cars,” in Proceedings of the 40th International Conference on Software Engineering , ser. ICSE ’18. ACM, 2018, p. 303–314

2018
[16]

Deepbillboard: Systematic physical-world testing of autonomous driving systems,

H. Zhou, W. Li, Y . Zhu, Y . Zhang, B. Yu, L. Zhang, and C. Liu, “Deepbillboard: Systematic physical-world testing of autonomous driving systems,” 2018

2018
[17]

Comparing offline and online testing of deep neural networks: An autonomous car case study,

F. U. Haq, D. Shin, S. Nejati, and L. Briand, “Comparing offline and online testing of deep neural networks: An autonomous car case study,” in Proceedings of 13th IEEE International Conference on Software Testing, Verification and Validation, ser. ICST ’20. IEEE, 2020

2020
[18]

Can offline testing of deep neural networks replace their online testing? a case study of automated driving systems,

——, “Can offline testing of deep neural networks replace their online testing? a case study of automated driving systems,” Empirical Software Engineering, vol. 26, no. 5, p. 90, 2021

2021
[19]

Model vs system level testing of autonomous driving systems: a replication and extension study,

A. Stocco, B. Pulfer, and P. Tonella, “Model vs system level testing of autonomous driving systems: a replication and extension study,”Empirical Software Engineering, vol. 28, no. 3, p. 73, May 2023

2023
[20]

Identifying and explaining safety-critical scenarios for autonomous vehicles via key features,

N. Neelofar and A. Aleti, “Identifying and explaining safety-critical scenarios for autonomous vehicles via key features,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 4, Apr. 2024

2024
[21]

Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles,

——, “Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE). ACM, 2024

2024
[22]

PAFOT: A Position- Based Approach for Finding Optimal Tests of Autonomous Vehicles,

V . Crespo-Rodriguez, Neelofar, and A. Aleti, “PAFOT: A Position- Based Approach for Finding Optimal Tests of Autonomous Vehicles,” in Proceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024) , ser. AST ’24. ACM, 2024, p. 159–170

2024
[23]

Epitester: Testing autonomous vehicles with epigenetic algorithm and attention mechanism,

C. Lu, S. Ali, and T. Yue, “Epitester: Testing autonomous vehicles with epigenetic algorithm and attention mechanism,” IEEE Transactions on Software Engineering, pp. 1–19, 2024

2024
[24]

DeepQTest: Testing Autonomous Driving Systems with Reinforcement Learning and Real-world Weather Data,

C. Lu, T. Yue, M. Zhang, and S. Ali, “DeepQTest: Testing Autonomous Driving Systems with Reinforcement Learning and Real-world Weather Data,” 2023. [Online]. Available: https://arxiv.org/abs/2310.05170

2023 arXiv
[25]

Safety assessment of vehicle characteristics variations in autonomous driving systems,

Q. Pan, T. Wang, P. Arcaini, T. Yue, and S. Ali, “Safety assessment of vehicle characteristics variations in autonomous driving systems,” 2023. [Online]. Available: https://arxiv.org/abs/2311.14461

2023 arXiv
[26]

An empirical comparison of combinatorial testing and search-based testing in the context of automated and autonomous driving systems,

F. Kl¨uck, Y . Li, J. Tao, and F. Wotawa, “An empirical comparison of combinatorial testing and search-based testing in the context of automated and autonomous driving systems,” Information and Software Technology, vol. 160, p. 107225, 2023

2023
[27]

Utilizing genetic algorithms for generating critical scenarios for testing autonomous driving functions,

F. Kl¨uck, D. Sumann, and F. Wotawa, “Utilizing genetic algorithms for generating critical scenarios for testing autonomous driving functions,” in 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), 2024, pp. 73–80

2024
[28]

Ambiegen: A search-based framework for autonomous systems testingimage 1,

D. Humeniuk, F. Khomh, and G. Antoniol, “Ambiegen: A search-based framework for autonomous systems testingimage 1,” Science of Computer Programming, vol. 230, p. 102990, 2023

2023
[29]

Reality bites: Assessing the realism of driving scenarios with large language models,

J. Wu, C. Lu, A. Arrieta, T. Yue, and S. Ali, “Reality bites: Assessing the realism of driving scenarios with large language models,” in Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering (FORGE) . ACM, 2024, p. 40–51

2024
[30]

Crag – a combinatorial testing-based generator of road geometries for ads testing,

P. Arcaini and A. Cetinkaya, “Crag – a combinatorial testing-based generator of road geometries for ads testing,” Science of Computer Programming, vol. 238, p. 103171, 2024

2024
[31]

Parameter coverage for testing of autonomous driving systems under uncertainty,

T. Laurent, S. Klikovits, P. Arcaini, F. Ishikawa, and A. Ventresque, “Parameter coverage for testing of autonomous driving systems under uncertainty,” ACM Trans. Softw. Eng. Methodol. , vol. 32, no. 3, Apr. 2023

2023
[32]

Simulation-based safety testing of automated driving systems,

F. Khan, H. Anwar, and D. Pfahl, “Simulation-based safety testing of automated driving systems,” in Product-Focused Software Process Improvement. Springer, 2024, pp. 133–138

2024
[33]

A process for scenario prioritization and selection in simulation- based safety testing of automated driving systems,

——, “A process for scenario prioritization and selection in simulation- based safety testing of automated driving systems,” in Product-Focused Software Process Improvement. Springer, 2024, pp. 89–99

2024
[34]

Efficient domain augmentation for autonomous driving testing using diffusion models,

L. Baresi, D. Y . X. Hu, A. Stocco, and P. Tonella, “Efficient domain augmentation for autonomous driving testing using diffusion models,” in Proceedings of 47th International Conference on Software Engineering , ser. ICSE ’25. IEEE, 2025

2025
[35]

Data augmentation technology driven by image style transfer in self-driving car based on end-to-end learning,

D. Liu, J. Zhao, A. Xi, X. H. Chao Wang, K. Lai, and C. Liu, “Data augmentation technology driven by image style transfer in self-driving car based on end-to-end learning,” Computer Modeling in Engineering & Sciences, vol. 122, no. 2, pp. 593–617, 2020

2020
[36]

Learning when to use adaptive adversarial image perturbations against autonomous vehicles,

H.-J. Yoon, H. Jafarnejadsani, and P. V oulgaris, “Learning when to use adaptive adversarial image perturbations against autonomous vehicles,” IEEE Robotics and Automation Letters , vol. 8, no. 7, pp. 4179–4186, 2023

2023
[37]

DeepManeuver: Adversarial Test Generation for Trajectory Manipulation of Autonomous Vehicles,

M. von Stein, D. Shriver, and S. Elbaum, “DeepManeuver: Adversarial Test Generation for Trajectory Manipulation of Autonomous Vehicles,” IEEE Transactions on Software Engineering , vol. 49, no. 10, pp. 4496– 4509, 2023

2023
[38]

Efficient performance prediction of end- to-end autonomous driving under continuous distribution shifts based on anomaly detection,

S. Luan, Z. Gu, and S. Wan, “Efficient performance prediction of end- to-end autonomous driving under continuous distribution shifts based on anomaly detection,” Journal of Signal Processing Systems , vol. 95, no. 12, pp. 1455–1468, 12 2023

2023
[39]

Replication package,

“Replication package,” https://github.com/ast-fortiss-tum/ perturbation-drive/tree/Replication, 2025

2025
[40]

Benchmarking robustness in object detection: Autonomous driving when winter is coming,

C. Michaelis, B. Mitzkus, R. Geirhos, E. Rusak, O. Bringmann, A. S. Ecker, M. Bethge, and W. Brendel, “Benchmarking robustness in object detection: Autonomous driving when winter is coming,” 2020. [Online]. Available: https://arxiv.org/abs/1907.07484

2020 arXiv
[41]

MNIST-C: A robustness benchmark for computer vision,

N. Mu and J. Gilmer, “MNIST-C: A robustness benchmark for computer vision,” CoRR, vol. abs/1906.02337, 2019. [Online]. Available: https://arxiv.org/abs/1906.02337

1906 arXiv
[42]

Autoaugment: Learning augmentation strategies from data,

E. D. Cubuk, B. Zoph, D. Man ´e, V . Vasudevan, and Q. V . Le, “Autoaugment: Learning augmentation strategies from data,” in 2019 IEEE/CVF CVPR, 2019, pp. 113–123

2019
[43]

Randaugment: Practical automated data augmentation with a reduced search space,

E. D. Cubuk, B. Zoph, J. Shlens, and Q. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” in Advances in Neural Information Processing Systems , vol. 33. Curran Associates, Inc., 2020

2020
[44]

Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, “Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.” in International Conference on Learning Representations (ICLR) , 2019

2019
[45]

Adversarial Self-Defense for Cycle-Consistent GANs,

D. Bashkirova, B. Usman, and K. Saenko, “Adversarial Self-Defense for Cycle-Consistent GANs,” in Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc., 2019

2019
[46]

Generating Adversarial Examples in One Shot With Image- to-Image Translation GAN,

W. Zhang, “Generating Adversarial Examples in One Shot With Image- to-Image Translation GAN,” IEEE Access, vol. 7, pp. 151 103–151 119, 2019

2019
[47]

DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems,

M. Zhang, Y . Zhang, L. Zhang, C. Liu, and S. Khurshid, “DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems,” in Proceedings of the 33rd ACM/IEEE ASE, ser. ASE ’18. ACM, 2018, p. 132–142

2018
[48]

Udacity self-driving car simulator,

Udacity, “Udacity self-driving car simulator,” https://github.com/udacity/ self-driving-car-sim, 2021, accessed: [2024-01-15]

2021
[49]

Sdsandbox,

T. Kramer, “Sdsandbox,” https://github.com/tawnkramer/sdsandbox, 2022

2022
[50]

OpenCat: Improving Interoperability of ADS Testing,

Q. Ali, A. Stocco, L. Mariani, and O. Riganelli, “OpenCat: Improving Interoperability of ADS Testing,” in Proceedings of 47th International Conference on Software Engineering Workshops, ser. ICSEW ’24. IEEE, 2025, p. 10 pages

2025
[51]

A framework for automated driving system testable cases and scenarios,

U. D. of Transportation, “A framework for automated driving system testable cases and scenarios,” https://rosap.ntl.bts.gov/view/dot/38824/ dot 38824 DS1.pdf, 2018

2018
[52]

Standing general order on crash reporting for level 2 advanced driver assistance systems,

——, “Standing general order on crash reporting for level 2 advanced driver assistance systems,” https://www.nhtsa.gov/sites/nhtsa.gov/files/ 2022-06/ADAS-L2-SGO-Report-June-2022.pdf, 2022

2022
[53]

Segformer: simple and efficient design for semantic segmentation with transformers,

E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: simple and efficient design for semantic segmentation with transformers,” in Proceedings of the 35th International Conference on Neural Information Processing Systems , ser. NIPS ’21. Red Hook, NY , ...

2024
[54]

The Cityscapes Dataset for Semantic Urban Scene Understanding,

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The Cityscapes Dataset for Semantic Urban Scene Understanding,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 3213–3223

2016
[55]

Assessing quality metrics for neural reality gap input mitigation in autonomous driving testing,

S. C. Lambertenghi and A. Stocco, “Assessing quality metrics for neural reality gap input mitigation in autonomous driving testing,” in Proceedings of 17th IEEE International Conference on Software Testing, Verification and Validation, ser. ICST ’24. IEEE, 2024, p. 12 pages

2024
[56]

End to end learning for self-driving cars

M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba, “End to end learning for self-driving cars.” CoRR, vol. abs/1604.07316, 2016

2016 arXiv
[57]

Two is better than one: digital siblings to improve autonomous driving testing,

M. Biagiola, A. Stocco, V . Riccio, and P. Tonella, “Two is better than one: digital siblings to improve autonomous driving testing,” Empirical Softw. Engg., vol. 29, no. 4, May 2024

2024
[58]

Quality metrics and oracles for autonomous vehicles testing,

G. Jahangirova, A. Stocco, and P. Tonella, “Quality metrics and oracles for autonomous vehicles testing,” in Proceedings of 14th IEEE International Conference on Software Testing, Verification and Validation , ser. ICST ’21. IEEE, 2021

2021
[59]

Boundary state generation for testing and improvement of autonomous driving systems,

M. Biagiola and P. Tonella, “Boundary state generation for testing and improvement of autonomous driving systems,” IEEE Trans. Softw. Eng., vol. 50, no. 8, p. 2040–2053, Jul. 2024. [Online]. Available: https://doi.org/10.1109/TSE.2024.3420816

2024
[60]

Virtualworlds as proxy for multi-object tracking analysis,

A. Gaidon, Q. Wang, Y . Cabon, and E. Vig, “Virtualworlds as proxy for multi-object tracking analysis,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016, pp. 4340–4349

2016
[61]

“Unity,” https://unity.com/, 2024, accessed: 11-01-2024

2024
[62]

Nvidia PhysX,

“Nvidia PhysX,” https://developer.nvidia.com/physx-sdk, 2022

2022
[63]

Python Software Foundation, “pyperf,” https://github.com/psf/pyperf, 2024, accessed: 2024-01-22

2024
[64]

nvidia/segformer-b0-finetuned-cityscapes-640-1280 · hugging face,

“nvidia/segformer-b0-finetuned-cityscapes-640-1280 · hugging face,” Huggingface.co, 2017. [Online]. Available: https://huggingface.co/nvidia/ segformer-b0-finetuned-cityscapes-640-1280

2017
[65]

Papers with code, cityscapes segmentation bench- marks

“Papers with code, cityscapes segmentation bench- marks.” [Online]. Available: https://paperswithcode.com/sota/ semantic-segmentation-on-cityscapes
[66]

Augmented reality meets computer vision: Efficient data generation for urban driving scenes,

Alhaija, Hassan, Mustikovela, Siva, Mescheder, Lars, Geiger, Andreas, Rother, and Carsten, “Augmented reality meets computer vision: Efficient data generation for urban driving scenes,” IJCV, 2018

2018
[67]

Evaluating the impact of flaky simulators on testing autonomous driving systems,

M. H. Amini, S. Naseri, and S. Nejati, “Evaluating the impact of flaky simulators on testing autonomous driving systems,” Empirical Softw. Engg., vol. 29, no. 2, feb 2024

2024
[68]

Digital twins are not monozygotic–cross-replicating adas testing in two industry-grade automotive simulators,

M. Borg, R. B. Abdessalem, S. Nejati, F.-X. Jegeden, and D. Shin, “Digital twins are not monozygotic–cross-replicating adas testing in two industry-grade automotive simulators,” in ICST ’21. IEEE, 2021

2021
[69]

Choose your simulator wisely: A review on open-source simulators for autonomous driving,

Y . Li, W. Yuan, S. Zhang, W. Yan, Q. Shen, C. Wang, and M. Yang, “Choose your simulator wisely: A review on open-source simulators for autonomous driving,” IEEE Transactions on Intelligent Vehicles , vol. 9, no. 5, p. 4861–4876, May 2024

2024
[70]

Towards a review on simulated adas/ad testing,

Y . Koroglu and F. Wotawa, “Towards a review on simulated adas/ad testing,” in 2023 IEEE/ACM International Conference on Automation of Software Test (AST), 2023, pp. 112–122

2023
[71]

Benchmarking Generative AI Models for Deep Learning Test Input Generation,

Maryam, M. Biagiola, A. Stocco, and V . Riccio, “Benchmarking Generative AI Models for Deep Learning Test Input Generation,” in Proceedings of 18th IEEE International Conference on Software Testing, Verification and Validation, ser. ICST ’25. IEEE, 2025, p. 12 pages

2025
[2020]

Springer, 2020, pp. 53–69

2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

A Survey on Automated Driving System Testing: Landscapes and Trends,

S. Tang, Z. Zhang, Y . Zhang, J. Zhou, Y . Guo, S. Liu, S. Guo, Y .-F. Li, L. Ma, Y . Xue, and Y . Liu, “A Survey on Automated Driving System Testing: Landscapes and Trends,” ACM Trans. Softw. Eng. Methodol. , vol. 32, no. 5, Jul. 2023

work page 2023

[2] [2]

A survey of autonomous driving: Common practices and emerging technologies,

E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” IEEE access, vol. 8, pp. 58 443–58 469, 2020

2020

[3] [3]

Panoptic perception for autonomous driving: A survey,

Y . Li and L. Xu, “Panoptic perception for autonomous driving: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2408.15388

arXiv 2024

[4] [4]

A survey of deep learning techniques for autonomous driving,

S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” Journal of Field Robotics , vol. 37, no. 3, pp. 362–386, 2020

work page 2020

[5] [5]

Understanding how image quality affects deep neural networks,

S. Dodge and L. Karam, “Understanding how image quality affects deep neural networks,” in 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX) , 2016, pp. 1–6

work page 2016

[6] [6]

Generalisation in humans and deep neural networks,

R. Geirhos, C. R. M. Temme, J. Rauber, H. H. Sch ¨utt, M. Bethge, and F. A. Wichmann, “Generalisation in humans and deep neural networks,” in Advances in Neural Information Processing Systems , vol. 31. Curran Associates, Inc., 2018

work page 2018

[7] [7]

Benchmarking neural network ro- bustness to common corruptions and perturbations,

D. Hendrycks and T. Dietterich, “Benchmarking neural network ro- bustness to common corruptions and perturbations,” in International Conference on Learning Representations (ICLR) , 2019

2019

[8] [8]

AugMix: A simple data processing method to improve robustness and uncertainty,

D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Laksh- minarayanan, “AugMix: A simple data processing method to improve robustness and uncertainty,” Proceedings of the International Conference on Learning Representations (ICLR) , 2020

work page 2020