Pith. sign in

REVIEW 3 major objections 5 minor 52 references

Testing Autonomous Driving Systems -- What Really Matters and What Doesn't

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Passing a critical test does not imply passing easier ones for most current autopilots, so strong safety guarantees stay out of reach unless autopilots are designed to be rational and determinate.

desk verdict A useful conceptual framework and an honest but under-calibrated empirical claim: the determinacy results are robust, but the rationality frequencies and the impossibility conclusion lean on unquantified A/D estimates. read the letter →

arxiv 2507.13661 v2 pith:KW2LJYJF submitted 2025-07-18 cs.SE

Changwen Li , Joseph Sifakis , Rongjie Yan , Jian Zhang This is my paper
classification cs.SE
keywords autonomousdrivingsystemssafetytestingcriticalscenariogenerationfalserejectionautopilotrationalitydeterminacyacceleration/decelerationfunctionstestcoverage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current methods for testing autonomous driving systems cannot deliver the safety guarantees they promise, because they rest on two assumptions that real autopilots violate and because many critical tests place vehicles in situations that are physically impossible to handle. It proposes a framework that separates test methods by effectiveness and validity, and defines an optimal criticality order for elementary adverse scenarios—single conflicts between the ego vehicle and one arriving or front vehicle—using acceleration and deceleration functions. Applying the framework to eight open autopilots, the paper reports that most fail to be rational (passing a critical test does not imply passing less critical tests) and that all fail to be determinate (behavior from a visited state need not match behavior from the initial state). It concludes that, as things stand, the required guarantees cannot be obtained unless autopilots are designed from the outset to be rational and determinate.

What carries the argument

The load-bearing objects are the A/D functions: the braking function $B(v)$ giving the distance needed to stop from speed $v$, the acceleration time function $T_A(x,v)$ giving the time to cover distance $x$ from speed $v$, and the acceleration speed function $V_A(x,v)$ giving the speed reached over distance $x$. From these, the paper computes the most critical safe test case for an elementary adverse scenario as $\hat{x}_a = T_A(x_e,v_e)\cdot v_l$ and $\hat{x}_f = B(V_A(x_e,v_e))$, where $x_e,v_e$ are the ego vehicle's distance and speed, $v_l$ is the speed limit of the arriving vehicle, and $\hat{x}_a,\hat{x}_f$ are the critical distances of the arriving and front vehicles. This critical boundary defines an optimal criticality order: any test case with larger $x_a,x_f$ has a superset of safe policies. Rationality and determinacy are then defined against this boundary—rationality as consistency in passing ordered test cases, determinacy as consistency of the policy when restarted from intermediate states—and the paper shows how monotonicity of the A/D functions permits partitioning the speed domain to reduce test counts.

What would settle it

Measure the actual A/D functions of a given autopilot with high precision, then run the maximally critical test case $(\hat{x}_a,\hat{x}_f)$ and every less critical test case in the ordered region; if any autopilot passes the critical case and all less critical cases without exception, or if restarting it from intermediate states of a braking or acceleration policy always reproduces the original trajectory, the paper's claim that these properties are systematically absent would be contradicted. A second check: if critical test methods already avoided non-ego-attributable accidents at negligible rates, the false-rejection claim would fail.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that test effectiveness and validity for autonomous driving systems are not just properties of the test method; they depend on design properties of the autopilot under test. Most existing methods implicitly assume rationality—that success on a maximally critical test case implies success on all less critical ones—and determinacy—that success from an initial state implies success from every state visited along the policy. Experiments on eight open autopilots, covering four scenario types (merge with yield, lane change, intersection with yield, intersection with traffic lights), show widespread violations: transition safety failures at rates up to 71.5% in one scenario type, irrational overcaution, and nondeterminate braking for seven of eight autopilots and nondeterminate acceleration for all eight. The paper therefore maintains that critical-scenario testing as currently practiced produces both false rejections (scenarios no vehicle could handle under nominal capabilities) and false acceptances, and that strong safety guarantees are impossible under the current state of the art.

Load-bearing premise

The framework's computed critical boundaries and all failure classifications rest on estimates of the autopilot's acceleration and deceleration functions, which the paper admits are not known exactly, so inaccurate estimates could move the boundaries and change the measured rationality and determinacy violations.

Editorial extensions

If this is right

  • If autopilots are rational and determinate, a small number of maximally critical test cases can replace a very large test space; the paper provides a concrete partitioning methodology and a coverage-rate estimate.
  • Most current critical-scenario methods generate a large fraction of accidents attributable to non-ego vehicles, so reported failure rates overstate autopilot risk and induce false rejections.
  • End-to-end AI autopilots show more irrational safety failures than modular ones, suggesting that the rationality assumption is especially unsafe for learning-based designs.
  • Without determinacy, even checking a simple braking-to-a-stop property may require infinitely many tests, since success from the initial state does not transfer to states on the braking curve.
  • A practical design recommendation follows: autopilots should be built so that control policies are generated by iterating a transition function, which yields determinate policies.
  • The framework's requirement for monotonic A/D functions implies that test-space partitioning and the resulting coverage estimates hold only when the autopilot's acceleration and deceleration behavior is monotonic in speed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable implication for regulators is that acceptance criteria for autonomous driving systems should include demonstrations of rationality and determinacy on benchmark scenarios, not just accident counts.
  • The framework's reliance on A/D function estimation means the measured violation rates are conditional on those estimates; if better measurement shifts the critical boundary, some transition failures may be reclassified as nominal or vice versa.
  • The rationality requirement connects to adversarial robustness: an autopilot that is rational in this ordered sense has decisions that change monotonically with scenario difficulty, which could be checked cheaply in simulation before road testing.
  • For end-to-end neural autopilots, the paper's argument implies that training objectives should include consistency across scenario criticality and state restart, not just collision avoidance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a framework for assessing the effectiveness and validity of autonomous driving system (ADS) test methods, distinguishing false acceptance and false rejection. It derives, for elementary adverse intersection/merge scenarios, an optimal criticality order based on inclusion of safe control policies, and identifies the most critical test cases in terms of acceleration/deceleration (A/D) functions. The authors then report experiments on eight open autopilots, claiming that most lack rationality (passing a critical test does not imply passing less critical tests) and determinacy (behavior from visited states is inconsistent with behavior from the initial state), and conclude that current ADS testing cannot provide required safety guarantees unless autopilots are designed to be rational and determinate.

Significance. If the central claims hold, the paper makes a substantial contribution by providing a principled, partly formal basis for comparing ADS test methods and by identifying two concrete autopilot design properties that would make testing tractable. The kinematic derivation of most-critical test cases in Section 2.3.3 is clean under the stated assumptions, and the experimental study across eight open autopilots is a useful concrete illustration. The paper also offers an actionable test methodology (Section 3.4) that combines rationality, determinacy, and monotonicity to reduce the number of required test cases. However, the empirical pillar of the paper—especially the reported frequencies of irrationality and the five-of-eight claim—rests on estimated A/D functions whose precision is admitted to be limited, and no sensitivity analysis is provided. The monotonicity assumptions in Section 3.3 are also not validated for the tested autopilots. These issues must be addressed before the main conclusions can be regarded as quantitatively grounded.

major comments (3)
  1. [§3.1, Table 2] The manuscript states in Section 3.1 that 'it is hard to estimate precise values of x̂_a and x̂_f because the A/D functions are not known exactly or due to other measurement errors,' yet Table 2 reports specific irrational-safety (IS) frequencies (e.g., 1.7%–7.3%) and the conclusion states that five of eight autopilots lack rationality. These classifications depend on which test cases lie inside or outside the computed safe-progress region, which in turn depends on the estimated A/D functions B(v), T_A(x,v), and V_A(x,v). If the estimates are biased, test cases labeled IS could instead be transition failures or irrelevant cases, and the number of autopilots classified as irrational could change. Please provide a sensitivity analysis that perturbs x̂_a and x̂_f within plausible error bounds and re-derives the classifications, or report confidence intervals for the IS frequencies. Without this, the central empirical claim is not robust to the acknowledged measurement error.
  2. [§3.3, Figures 13–15] The test-case partitioning technique and the coverage ratio calculation assume monotonicity of the A/D functions (V_B monotonic in v, T_A and V_A monotonic in the appropriate directions). This assumption is stated but not established for the eight tested autopilots. In fact, the non-determinacy results in Figures 10–12 show that braking and acceleration behavior can depend on the state in ways that violate simple composability, and it is not shown whether monotonicity holds in the tested data. Please validate the monotonicity assumptions on the tested autopilots, or clearly restrict the proposed methodology to autopilots for which monotonicity has been verified. As written, the coverage improvement claim in Section 3.4 is conditional on an unverified property.
  3. [§2.3.3, Eq. for x̂_a and x̂_f] The derivation of the most critical test case assumes that the ego vehicle's progress policy is safe exactly when T_A(x_e,v_e) ≤ x_a/v_l and B(V_A(x_e,v_e)) ≤ x_f, with equality defining the critical boundary. This is a simplified model that neglects, for example, the finite length of the critical zone and the time spent within it, as acknowledged by the use of a distance d in the merge example in Section 2.3.1. The paper should state clearly that the derived critical values are approximations under idealized point-vehicle kinematics, and should discuss how the finite size of the conflict zone affects the boundary. This is not a fatal flaw, but it strengthens the need for the sensitivity analysis requested above.
minor comments (5)
  1. [§3.1, Figure 6] In the text accompanying Figure 6, the condition for no safe progress policy is written as 'x_a ≤ x̂_a or x̂_f ≤ x_f'; this should be 'x_f ≤ x̂_f' for consistency with the earlier definitions.
  2. [§3.3, paragraph before Figure 15] The inequality 'T_A(v_i,x_e) < V_A(v_{i+1},x_e)' appears to be a typo; it should compare travel times, e.g., 'T_A(v_i,x_e) < T_A(v_{i+1},x_e)'.
  3. [Table 2] The frequencies in Table 2 are reported as percentages with no denominators or confidence intervals. Please clarify how IS (and TF) percentages are computed, the number of test cases in each scenario type, and whether the percentages are over all explored (x_a,x_f) pairs or only over the safe-progress region. This would make the table interpretable and reproducible.
  4. [§3.2, Conclusions] The claim that 'all eight tested autopilots exhibit non-determinacy for basic acceleration policies' is supported only by the illustrative example in Figure 12 and summary statements. Please report quantitative measures of non-determinacy (e.g., differences in achieved speed or braking distance across state-restart experiments) for each autopilot, so that the claim is verifiable.
  5. [General] The A/D functions are said to be 'either mathematically defined by autopilots developers or can be estimated experimentally [20]' in Section 2.3.3. Since reference [20] is the authors' own prior work, an independent description of the estimation procedure (e.g., number of trials, simulator settings, error metrics) would strengthen reproducibility and help readers judge the reliability of the critical boundaries.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the irrationality and non-determinacy results are observed behavior of eight external autopilots, not by-construction outputs of the framework; self-cited A/D estimation is a limitation rather than a circular reduction.

full rationale

The paper's central claims — that most of the eight tested autopilots are not rational and not determinate — are empirical observations obtained by running the autopilots on generated scenarios, not conclusions derived from the framework's definitions. Rationality is operationalized as passing a more critical test and failing a less critical one; such violations are directly observable once a dominance order on test cases is fixed. Determinacy is measured by comparing behavior from an initial state with behavior from intermediate states on the same trajectory (Figures 10-12), a comparison that does not depend on A/D function estimates. The paper explicitly acknowledges the imprecision of x-hat_a and x-hat_f in Section 3.1 ('it is hard to estimate precise values of x-hat_a and x-hat_f because the A/D functions are not known exactly or due to other measurement errors') and states that the exploration method works 'even without knowing these values,' so the IS/OF classifications are not forced by construction from fitted parameters. The A/D functions B(v), T_A(x,v), and V_A(x,v) are inputs to the framework; their estimation is cited to the authors' own [20], but this is a methodology citation and a validity limitation (no sensitivity analysis), not a case of a fitted parameter being renamed as a prediction. The 'impossibility' conclusion is an inductive generalization from the observed failures, not a tautology. Hence no circular step is exhibited; the minor self-citation and the lack of independent A/D calibration keep the score at 2 rather than 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on estimated A/D functions, hand-selected test-space ranges, and the domain assumptions of adverse scenarios and nominal conditions. No new physical entities are introduced. The framework's quantitative outputs are only as good as the estimated functions and the chosen exploration ranges.

free parameters (4)
  • A/D functions B(v), TA(x,v), VA(x,v) = Not disclosed; estimated experimentally or taken from [20]
    Used to compute x̂a and x̂f, the critical test-case boundaries that define rationality and determinacy verdicts.
  • Test-space exploration ranges and grid density for x_e, v_e, x_a, x_f = Not specified in detail
    Table 2 frequencies depend on which initial states and parameter ranges were searched.
  • Speed limit v_l used for arriving vehicle = Not stated per scenario
    Maximal criticality test cases assume the arriving vehicle travels at v_l, and x̂a = TA*v_l depends on it.
  • Critical zone distance d in merge example = 5 m in the example
    Used to illustrate false acceptance, not central to main experimental claims.
assumptions (5)
  • domain assumption Vehicles obey a kinematic model with bounded speeds, and the arriving and front vehicles follow constant or zero speed trajectories in maximal criticality test cases.
    Section 2.3.3 builds safe-progress conditions on these simplified trajectories.
  • domain assumption In adverse scenarios the ego vehicle always has lower priority and surrounding vehicles only need to follow traffic rules.
    Section 2.1.1 uses this to justify generating NPC trajectories independently of the ego vehicle.
  • domain assumption Nominal operating conditions exclude equipment failures, hostile actors, and rule violations by other vehicles.
    Section 1.1 defines nominal conditions this way; tests violating them are classified as false rejections.
  • ad hoc to paper Monotonicity of A/D functions holds, enabling test-case partitioning by speed intervals.
    Section 3.3 assumes VB, TA, and VA satisfy monotonicity; not verified for the eight autopilots.
  • ad hoc to paper Rationality and determinacy are necessary for the validity of worst-case and coverage-based testing.
    This is the paper's conceptual premise, argued rather than derived from external standards.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Testing Autonomous Driving Systems -- What Really Matters and What Doesn't." pith.science (2026). https://pith.science/paper/KW2LJYJF

@misc{pith2026250713661,
  author       = {Pith},
  title        = {Pith review of: Testing Autonomous Driving Systems -- What Really Matters and What Doesn't},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KW2LJYJF}},
  note         = {Machine review of arXiv:2507.13661}
}
read the original abstract

Despite extensive research, the testing of autonomous driving systems (ADS) landscape remains fragmented, and there is currently no basis for an informed technical assessment of the importance and contribution of the current state of the art. This paper attempts to address this problem by exploring two complementary aspects. First, it proposes a framework for comparing existing test methods in terms of their intrinsic effectiveness and validity. It shows that many methods do not meet both of these requirements. Either because they are based on criteria that do not allow for rapid, inexpensive, and comprehensive detection of failures, or because the degree of validity of the properties tested cannot be accurately estimated. In particular, it is shown that most critical test methods do not take into account the nominal operational capabilities of autopilots and generate scenarios that are impossible for the tested vehicles to handle, resulting in unjustified rejections. Secondly, the paper shows that test effectiveness and validity are highly dependent on how autopilots are designed: how they choose between different control policies to perform maneuvers, as well as on the reproducibility of the results. In fact, most test methods take for granted two principles underlying traditional methods, but do not generally apply to ADS. We maintain that the absence of rationality and determinacy significantly impairs the effectiveness and validity of test methods, and provide test results on eight open autopilots, in which most do not satisfy these properties, thereby illustrating this fact. We conclude that under the current state of the art, it is impossible to obtain strong enough guarantees for essential autopilot properties and recommend that autopilots be developed with a view to both rationality and determinacy.

Figures

Figures reproduced from arXiv: 2507.13661 by the authors.

Figure 1
Figure 1. State evolution in a scenario A test case is a stimulus for the tested autopilot. It is denoted by (𝑞0, 𝑠[𝑛]), specifying the initial state of the ego vehicle and a sequence of 𝑛 states𝑠[𝑛] of its environment. In response to this stimulus, the ego vehicle applies a control policy function 𝑝𝑜𝑙(𝑞0, 𝑠[𝑛]) = 𝑞[𝑛] that generates a corresponding sequence of states 𝑞[𝑛] that is the policy applied by the autopilot. So, the … view at source ↗
Figure 2
Figure 2. Possible relations between 𝑁𝑆, 𝐺𝑆 and 𝐶𝑆 [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Merge of the ego vehicle into a main road The initial state of the ego vehicle is characterized by its distance 𝑥𝑒 from the merge point and its speed 𝑣𝑒 , while the arriving vehicle is initially at distance 𝑥𝑎 and keeps moving at constant speed 𝑣𝑎. Assume that the ego vehicle applies a policy that keeps its speed constant at 𝑣𝑒 . The generated scenario may fail if a collision occurs. This can happen if the two vehic… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Separation between safe and unsafe policies using the optimal criticality order This definition of criticality, based on the inclusion of safety policies, can be used to better understand and analyze the behavior of autopilots. An autopilot faced with an unfavorable co…
Figure 5
Figure 5. Figure 5: The two elementary adverse scenarios We consider two types of elementary adverse scenarios, whose static parts are shown in [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Theoretical decomposition of the test space defined by a most critical test case Irrelevant scenarios Safe cautious scenarios for Safe progress scenarios [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Decomposition obtained in practice Irrelevant scenarios Safe cautious scenarios for Safe progress scenarios Unsafe policies Overcautious irrational policies Unsafe irrational policies [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 9
Figure 9. Figure 9: Determinate and non-determinate braking policy Note that the determinacy requirement is trivially satisfied when 𝑉 𝐵 is defined for a constant braking rate 𝑏. We have 𝑉 𝐵(𝑣, 𝑥) = 𝑣 ′ if (𝑣 ′2 − 𝑣 2 )/2𝑏 = 𝑥. Using this definition, we get for 𝑉 𝐵(𝑣0, 𝑥𝑖) = 𝑣𝑖 , the equa…
Figure 10
Figure 10. Figure 10: Braking curves for Apollo and Autoware autopilots The same type of reasoning can be applied to acceleration functions. Consider the scenario of a progress maneuver of the ego vehicle starting with speed 𝑣𝑒 at a distance 𝑥𝑒 from a critical zone where it can have a coll…
Figure 11
Figure 11. Figure 11: Determinate and non-determinate progress policy A check of the acceleration policies for the eight open autopilots shows that they are not determinate [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Non-determinate progress policy for the Carla autopilot at an intersection with yield signs 3.3 Improving coverage by test case partitioning Traditional methods use test case partitioning techniques that allow test cases to be grouped into classes, so that only one co…
Figure 13
Figure 13. Figure 13: Monotonicity of braking functions [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 15
Figure 15. Figure 15: Decomposition of the test space defined by safe policies of a speed partition In [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Test methodology [PITH_FULL_IMAGE:figures/full_fig_p028_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 45 canonical work pages

  1. [20]

    Changwen Li, Joseph Sifakis, Rongjie Yan, and Jian Zhang. 2024. Rigorous Simulation-based Testing for Autonomous Driving Systems–Targeting the Achilles’ Heel of Four Open Autopilots.arXiv preprint arXiv:2405.16914(2024)

    work page Pith review arXiv 2024

  2. [1]

    Matthias Althoff and Sebastian Lutz. 2018. Automatic generation of safety-critical test scenarios for collision avoidance of road vehicles. In2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1326–1333

    work page 2018

  3. [2]

    Christian Amersbach and Hermann Winner. 2019. Functional decomposition—A contribution to overcome the parameter space explosion during validation of highly automated driving.Traffic injury prevention20, sup1 (2019), S52–S57

    work page 2019

  4. [3]

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. 2020. nuscenes: A multimodal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 11621–11631

    2020

  5. [4]

    Mingfei Cheng, Yuan Zhou, and Xiaofei Xie. 2023. Behavexplor: Behavior diversity guided testing for autonomous driving systems. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 488–500

    work page 2023

  6. [5]

    Anthony Corso, Ritchie Lee, and Mykel J Kochenderfer. 2020. Scalable autonomous vehicle safety validation through dynamic programming and scene decomposition. In2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). IEEE, 1–6

    work page 2020

  7. [6]

    Adel Djoudi, Loic Coquelin, and Rémi Regnier. 2020. A simulation-based framework for functional testing of automated driving controllers. InIEEE 23rd International Conference on intelligent transportation systems. IEEE, 1–6

    work page 2020

  8. [7]

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. InConference on robot learning. PMLR, 1–16

    2017

Show all 52 references
  1. [8]

    Darrell Etherington. 2019. Waymo has now driven 10 billion autonomous miles in simulation. InTechcrunch Sessions: Mobility. TechCrunch

    2019

  2. [9]

    Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. 2023. Dense reinforcement learning for safety validation of autonomous vehicles.Nature615, 7953 (2023), 620–627

    2023

  3. [10]

    Christian Gold, Moritz Körber, David Lechner, and Klaus Bengler. 2016. Taking over control from highly automated vehicles in complex traffic situations: The role of traffic density.Human factors58, 4 (2016), 642–652

    2016

  4. [11]

    Kunkun Hao, Wen Cui, Yonggang Luo, Lecheng Xie, Yuqiao Bai, Jucheng Yang, Songyang Yan, Yuxi Pan, and Zijiang Yang. 2023. Adversarial safety-critical scenario generation using naturalistic human driving priors.IEEE Transactions on Intelligent Vehicles(2023)

    2023

  5. [12]

    Yuqi Huai, Sumaya Almanee, Yuntianyi Chen, Xiafa Wu, Qi Alfred Chen, and Joshua Garcia. 2023. scenoRITA: generating diverse, fully mutable, test scenarios for autonomous vehicle planning.IEEE Transactions on Software Engineering49, 10 (2023), 4656–4676

    2023

  6. [13]

    Yuqi Huai, Yuntianyi Chen, Sumaya Almanee, Tuan Ngo, Xiang Liao, Ziwen Wan, Qi Alfred Chen, and Joshua Garcia

  7. [14]

    Nidhi Kalra and Susan M Paddock. 2016. Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?Transportation research part A: policy and practice94 (2016), 182–193

    2016

  8. [15]

    Abolfazl Karimi and Parasara Sridhar Duggirala. 2022. Automatic generation of test-cases of increasing complexity for autonomous vehicles at intersections. In2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS). IEEE, 01–11

    2022

  9. [16]

    Seulbae Kim, Major Liu, Junghwan" John" Rhee, Yuseok Jeon, Yonghwi Kwon, and Chung Hwan Kim. 2022. Drivefuzz: Discovering autonomous driving bugs through driving quality-guided fuzzing. InProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 1753–1767

    2022

  10. [17]

    Carla Autonomous Driving Leaderboard. 2020. Carla Autonomous Driving Leaderboard. https://leaderboard.carla.org/. Accessed: 2020-01-17

    2020

  11. [18]

    Changwen Li, Chih-Hong Cheng, Tiantian Sun, Yuhang Chen, and Rongjie Yan. 2022. ComOpT: Combination and optimization for testing autonomous driving systems. In2022 International Conference on Robotics and Automation (ICRA). IEEE, 7738–7744

    2022

  12. [19]

    Changwen Li, Joseph Sifakis, Qiang Wang, Rongjie Yan, and Jian Zhang. 2023. Simulation-based validation for autonomous driving systems. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 842–853

    2023

  13. [21]

    Changwen Li, Joseph Sifakis, Rongjie Yan, and Jian Zhang. 2025. A Comprehensive Evaluation of Four End-to-End AI Autopilots Using CCTest and the Carla Leaderboard.arXiv preprint arXiv:2501.12090(2025)

    2025 arXiv

  14. [22]

    Guanpeng Li, Yiran Li, Saurabh Jha, Timothy Tsai, Michael Sullivan, Siva Kumar Sastry Hari, Zbigniew Kalbarczyk, and Ravishankar Iyer. 2020. Av-fuzzer: Finding safety violations in autonomous driving systems. In2020 IEEE 31st international symposium on software reliability eng...

    2020

  15. [23]

    Yihao Li, Jianbo Tao, and Franz Wotawa. 2020. Ontology-based test generation for automated and autonomous driving functions.Information and software technology117 (2020), 106200

    2020

  16. [24]

    Shenghao Lin, Fansong Chen, Laile Xi, Gaosheng Wang, Rongrong Xi, Yuyan Sun, and Hongsong Zhu. 2024. TM-fuzzer: fuzzing autonomous driving systems through traffic management.Automated Software Engineering31, 2 (2024), 61

    2024

  17. [25]

    You Lu, Yifan Tian, Yuyang Bi, Bihuan Chen, and Xin Peng. 2024. Diavio: Llm-empowered diagnosis of safety violations in ads simulation testing. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 376–388

    2024

  18. [26]

    Rupak Majumdar, Aman Mathur, Marcus Pirron, Laura Stegner, and Damien Zufferey. 2019. Paracosm: A language and tool for testing autonomous driving systems.arXiv preprint arXiv:1902.01084(2019)

    2019 arXiv

  19. [27]

    W. G. Najm, J. D. Smith, M. Yanagisawa, and J. A. Volpe. 2007.Pre-crash scenario typology for crash avoidance research. Tech. Rep. DOT-VNTSC-NHTSA-06-02. National Highway Traffic Safety Administration, Washington, DC, USA

    2007

  20. [28]

    Allen Newell. 1980. Physical symbol systems.Cognitive science4, 2 (1980), 135–183

    1980

  21. [29]

    Jonas Nilsson, Anders CE Ödblom, and Jonas Fredriksson. 2015. Worst-case analysis of automotive collision avoidance systems.IEEE Transactions on Vehicular Technology65, 4 (2015), 1899–1911

    2015

  22. [30]

    Guodong Rong, Byung Hyun Shin, Hadi Tabatabaee, Qiang Lu, Steve Lemke, M¯artin, š Možeiko, Eric Boise, Geehoon Uhm, Mark Gerow, Shalin Mehta, et al. 2020. Lgsvl simulator: A high fidelity simulator for autonomous driving. In 2020 IEEE 23rd International conference on intellige...

    2020

  23. [31]

    Sebastian Söntges and Matthias Althoff. 2017. Computing the drivable area of autonomous road vehicles in dynamic road scenes.IEEE Transactions on Intelligent Transportation Systems19, 6 (2017), 1855–1866

    2017

  24. [32]

    Daniel Stumper and Klaus Dietmayer. 2018. Towards criticality characterization of situational space. In2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, 3378–3382

    2018

  25. [33]

    Liangliang Sun, Yan-Fu Li, and Enrico Zio. 2022. Comparison of the HAZOP, FMEA, FRAM, and STPA methods for the hazard analysis of automatic emergency brake systems.ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems, Part B: Mechanical Engineering8, 3 (2022), 031104

    2022

  26. [34]

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. 2020. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on compu...

    2020

  27. [35]

    Yang Sun, Christopher M Poskitt, Jun Sun, Yuqi Chen, and Zijiang Yang. 2022. LawBreaker: An approach for specifying traffic laws and fuzzing autonomous vehicles. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–12

    2022

  28. [36]

    Yun Tang, Yuan Zhou, Tianwei Zhang, Fenghua Wu, Yang Liu, and Gang Wang. 2021. Systematic testing of autonomous driving systems using map topology-based scenario classification. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1342–1346

    2021

  29. [37]

    2018.A framework for automated driving system testable cases and scenarios

    Eric Thorn, Shawn C Kimmel, Michelle Chaka, et al. 2018.A framework for automated driving system testable cases and scenarios. Technical Report. United States. Department of Transportation. National Highway Traffic Safety

    2018

  30. [38]

    Haoxiang Tian, Wenqiang Ding, Xingshuo Han, Guoquan Wu, An Guo, Junqi Zhang, Wei Chen, Jun Wei, and Tianwei Zhang. 2025. Testing the Fault-Tolerance of Multi-sensor Fusion Perception in Autonomous Driving Systems. Proceedings of the ACM on Software Engineering2, ISSTA (2025), 777–800

    2025

  31. [39]

    Haoxiang Tian, Yan Jiang, Guoquan Wu, Jiren Yan, Jun Wei, Wei Chen, Shuo Li, and Dan Ye. 2022. MOSAT: finding safety violations of autonomous driving systems using multi-objective genetic algorithm. InProceedings of the 30th ACM Joint European Software Engineering Conference a...

    2022

  32. [40]

    Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. Deeptest: Automated testing of deep-neural-network- driven autonomous cars. InProceedings of the 40th international conference on software engineering. 303–314

    2018

  33. [41]

    Paul FMJ Verschure and Philipp Althaus. 2003. A real-world rational agent: unifying old and new AI.Cognitive science 27, 4 (2003), 561–590

    2003

  34. [42]

    Jiajie Wang, Chi Zhang, Yuehu Liu, and Qilin Zhang. 2018. Traffic sensory data classification by quantifying scenario complexity. In2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1543–1548

    2018

  35. [43]

    Tong Wang, Taotao Gu, Huan Deng, Hu Li, Xiaohui Kuang, and Gang Zhao. 2024. Dance of the ads: Orchestrating failures through historically-informed scenario fuzzing. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1086–1098

    2024

  36. [44]

    Lukas Westhofen, Christian Neurohr, Tjark Koopmann, Martin Butz, Barbara Schütt, Fabian Utesch, Birte Neurohr, Christian Gutenkunst, and Eckard Böde. 2023. Criticality metrics for automated driving: A review and suitability analysis of the state of the art.Archives of Computat...

    2023

  37. [45]

    Qin Xia, Jianli Duan, Feng Gao, Qiuxia Hu, and Yingdong He. 2018. Test scenario design for intelligent driving system ensuring coverage and effectiveness.International Journal of Automotive Technology19 (2018), 751–758. 32 Changwen Li, Joseph Sifakis, Rongjie Yan, and Jian Zhang

    2018

  38. [46]

    Fan Yang, You Lu, Bihuan Chen, Peng Qin, and Xin Peng. 2024. RoadGen: Generating Road Scenarios for Autonomous Vehicle Testing.arXiv preprint arXiv:2411.19577(2024)

    2024 arXiv

  39. [47]

    Yukuan Yang, Xucheng Lu, Zhili Zhang, Zepeng Wu, Guoqi Li, Lingzhong Meng, and Yunzhi Xue. 2025. AuthSim: Towards Authentic and Effective Safety-critical Scenario Generation for Autonomous Driving Tests.arXiv preprint arXiv:2502.21100(2025)

    2025 arXiv

  40. [48]

    Songan Zhang, Huei Peng, Ding Zhao, and H Eric Tseng. 2018. Accelerated evaluation of autonomous vehicles in the lane change scenario based on subset simulation technique. In2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, 3935–3940

    2018

  41. [49]

    Xudong Zhang and Yan Cai. 2023. Building critical testing scenarios for autonomous driving from real accidents. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 462–474

    2023

  42. [50]

    Xinhai Zhang, Jianbo Tao, Kaige Tan, Martin Törngren, José Manuel Gaspar Sánchez, Muhammad Rusyadi Ramli, Xin Tao, Magnus Gyllenhammar, Franz Wotawa, Naveen Mohan, et al. 2021. Finding critical scenarios for automated driving systems: A systematic literature review.arXiv prepr...

    2021 arXiv

  43. [51]

    Ziyuan Zhong, Gail Kaiser, and Baishakhi Ray. 2022. Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles.IEEE Transactions on Software Engineering49, 4 (2022), 1860–1875

    2022

  44. [2023]

    In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

    Doppelgänger test generation for revealing bugs in autonomous driving software. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2591–2603

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.