Pith. sign in

REVIEW 4 major objections 4 minor 57 references

Behavioral Safety Assessment towards Large-scale Deployment of Autonomous Vehicles

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that behavioral safety, not component-level functional safety, is the missing evaluation layer for autonomous vehicles, and it reports a two-test framework that puts the open-source Level 4 stack Autoware.Universe at…

desk verdict The two-part behavioral safety framework is worth engaging, but the headline 1,000x crash-rate claim is not supported by the presented statistics, and the text has a 6-vs-8 DLT pass discrepancy. read the letter →

arxiv 2505.16214 v2 pith:FCAFWQO5 submitted 2025-05-22 cs.RO cs.SE

Henry X. Liu , Xintao Yan , Haowei Sun , Tinghan Wang , Zhijie Qiao , Haojie Zhu , Shengyin Shen , Shuo Feng
show 2 more authors
Greg Stevens Greg McGuire
This is my paper · ORCID
classification cs.ROcs.SE
keywords behavioralsafetyautonomousvehicletestingDriverLicensingTestDrivingIntelligenceimportancesamplingnaturalisticadversarialenvironmentcrashrateestimationAutoware.Universe
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autonomous-vehicle safety is usually checked component by component, but the authors argue that what matters at scale is behavioral safety: how the vehicle reacts to and interacts with the surrounding traffic. They propose a third-party evaluation with two parts, the Driver Licensing Test (controlled reactive scenarios) and the Driving Intelligence Test (naturalistic traffic with rare events amplified for statistical measurement). Applied to the open-source Level 4 stack Autoware.Universe, the paper reports six passed licensing scenarios out of fourteen (the results text also says eight, while the abstract and Table 1 say six) and a crash rate of $3.01\times10^{-3}$ crashes per mile, roughly a thousand times the average human driver's $3.00\times10^{-6}$ per mile. The paper's point is that this kind of black-box behavioral testing can supply a quantitative safety measure before large-scale deployment.

What carries the argument

The load-bearing object is the Naturalistic and Adversarial Driving Environment (NADE), built on top of a naturalistic driving environment whose crash rates and crash-type mix are calibrated against real-world Michigan data. At each critical moment NADE picks the Principal Other Vehicle (POV), the background car whose action most threatens the AV, and amplifies that action's probability by a factor of $\epsilon = 3000$ with location-specific caps (0.1 for intersection collisions, 0.01 for others). Each episode is then reweighted by the likelihood ratio $W_q(\mathbf{x}) = P(\mathbf{x})/q(\mathbf{x})$, the ratio of the naturalistic probability to the amplified probability, so that the crash count can be converted back into a naturalistic rate. The Driver Licensing Test is the scenario-scale half of the machinery: parameterized cases from naturalistic datasets, binned into risk levels by the deceleration required to avoid collision, with a pass/fail verdict per scenario.

What would settle it

Recompute the DIT crash rate two ways: once with the importance-sampling weights applied and once treating each episode equally. If the two estimates differ by a large factor, or if instrumenting the simulator shows the implemented $q(\mathbf{x})$ (with $\epsilon=3000$ and the 0.1/0.01 caps) does not match the $q(\mathbf{x})$ used in $W_q$, then the reported $3.01\times10^{-3}$ crashes per mile is not a naturalistic estimate.

Watch

Extended reading notes

Core claim

The paper's central claim is that behavioral safety is a distinct and necessary evaluation layer, separate from crashworthiness and functional safety, and that it can be measured without any access to the AV's internal design. The Driver Licensing Test checks reactive competence in scenarios drawn from real crash data, and the Driving Intelligence Test estimates the frequency of safety-critical events by placing the AV in an environment where background vehicles' adversarial maneuvers are amplified by importance sampling and then reweighted to remain unbiased. On these tests, the paper reports, Autoware.Universe passed six of the fourteen licensing scenarios and produced an estimated $3.01\times10^{-3}$ crashes per mile, about one thousand times the human benchmark of $3.00\times10^{-6}$. The authors also report discovering previously unknown failure modes, including roundabout replanning errors, rigid right-of-way yielding, and trajectory prediction that ignores traffic-light state, and they demonstrate the same testing approach on a physical track with two different vehicles.

Load-bearing premise

The whole DIT crash-rate estimate is only unbiased if the simulator actually samples from the same distribution that appears in the importance-sampling weights, and if episodes thrown away because the AV got stuck do not bias the remaining data.

Editorial extensions

If this is right

  • If the crash-rate estimate is correct, an open-source Level 4 stack in this urban ODD is roughly three orders of magnitude away from the average human driver's crash rate, so behavioral testing can separate deployment-ready systems from those that are not.
  • The DLT/DIT structure gives regulators a reusable, black-box testing template: the AV under test needs no data interface, only its own sensors and behavior, so the same protocol can compare different vendors.
  • The DIT's discovered failure modes, including roundabout stalls, right-of-way yielding, and trajectory prediction that ignores red lights, show how simulation-based behavioral testing can feed concrete fixes back to developers in a closed loop.
  • The framework converts 'how safe is safe enough' from a slogan into a number: a statistically reweighted crash rate per mile, with crash-type and severity distributions attached.
  • Because the same scenarios and environment are used in simulation and on a physical test track, results from one setting can be cross-checked against the other before large-scale deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A check the paper leaves implicit: if the implemented sampling distribution differs from the one in the importance-sampling weight, for instance if the $\epsilon=3000$ amplification and location caps are not exactly reflected in $q(\mathbf{x})$, the reported $3.01\times10^{-3}$ figure could be an unweighted frequency rather than a naturalistic rate.
  • The same two-test architecture could be extended to measure non-safety behaviors that affect public acceptance, such as blocking intersections or delaying traffic, since the DIT environment already records interactions over full trips.
  • Because the simulated DIT bypasses the perception module, the estimate speaks to decision-making and planning; a version that feeds simulated sensor data could likely reveal additional perception-related failures at scale.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a third-party behavioral safety assessment framework for autonomous vehicles, consisting of a Driver Licensing Test (DLT) for scenario-based competency evaluation and a Driving Intelligence Test (DIT) for statistical crash-rate estimation in a naturalistic adversarial environment. The framework is demonstrated on the open-source Level 4 stack Autoware.Universe in simulation at the Mcity test facility and with limited physical tests using a Tesla FSD vehicle and an Autoware-equipped Lincoln MKZ. The headline results are that Autoware.Universe passed 6 (elsewhere 8) of 14 DLT scenarios and that the DIT estimated a crash rate of 3.01×10^-3 per mile, about 1000 times the human benchmark of 3.00×10^-6 per mile.

Significance. If the central quantitative claims are correct, the paper would provide an important third-party, black-box safety evaluation methodology and an unusually public assessment of a widely used open-source AV stack. The framework is timely and the deployment of both scenario-based and statistical testing is conceptually attractive. The paper's strengths include the use of open-source tools (TeraSim, Autoware.Universe), publication of experimental data and code on Zenodo, and the demonstration that unknown unsafe behaviors can be discovered and fed back to developers. However, the significance is conditional: the statistical unbiasedness of the DIT estimate is the foundation of the headline crash-rate comparison, and that foundation is not established in the manuscript as written.

major comments (4)
  1. [Methods (Driving intelligence test; Simulation settings) and Results (Results of the Driving Intelligence Test)] The reported DIT crash rate of 3.01×10^-3 crashes per mile is not connected to the importance-sampling estimator defined in the Methods. The estimator in the Methods is P(A) = (1/n) Σ P(A|x_i) W_q(x_i) with W_q(x) = ∏ P(u(k)|s(k))/q(u(k)|s(k)), but the Results report only a single point estimate, with no weights, no weight summary, no effective sample size, and no confidence intervals. The NADE implementation described later multiplies POV maneuver probabilities by ε=3000, applies location-specific caps (0.1 for intersection collisions, 0.01 for others), and excludes 'effective episodes' where the AV gets stuck. The q(x) actually implemented therefore differs from the q(x) in the importance-sampling weight unless the caps, the ε multiplier, and the episode exclusions are all incorporated into W_q. The manuscript must show either the weighted estimate with the true implementation-consistent q, or a demonstrated argument that the reported number is unbiased. As written, the reported 3.01e-3 could be a biased raw frequency of the amplified NADE, which would invalidate the central 1000× comparison to the human crash rate.
  2. [Abstract and Introduction vs. Table 1 and Results] The number of DLT scenarios passed by Autoware.Universe is inconsistent across the manuscript. The Abstract states 'passed 6 out of 14 scenarios,' while the Introduction states 'passes eight out of 14 scenarios' (page 4). Table 1 lists P for scenarios b, f, j, k, m, n, which is 6 passes and 8 failures, and the Results section reports 'successfully passed 6 out of 14 scenarios.' This numerical contradiction affects the reported outcome and must be resolved in revision.
  3. [Results (Results of the Driving Intelligence Test) and Methods (Simulation settings, NDE simulator)] The human benchmark used in the headline comparison (3.00×10^-6 crashes per mile, Ref. 38) is the same Michigan crash dataset used to calibrate the human-error model in the NDE simulator, whose validation yields 2.93×10^-6 crashes per mile. This means the 'average human driver' rate is not an independent benchmark relative to the simulation pipeline; it is the target used to tune the background-agent error probabilities. While an external crash statistic is a legitimate reference, the paper should discuss this calibration-dependence and state clearly that the comparison is to an aggregate Michigan crash rate, not to a matched human-driver performance in the same ODD. Without such discussion, the claim of 'about 1,000 times higher' overstates the independence of the comparison.
  4. [Results (Results of the Driving Intelligence Test)] The DIT estimate is reported as a single point value with no measure of statistical uncertainty. The experiment comprised about 3,500 episodes; even if each episode is roughly one mile, a crash rate near 3e-3 would correspond to only about 10 observed crashes, yielding a wide confidence interval. The paper should report a confidence interval or credible interval for the crash rate, and ideally a convergence plot of the weighted estimator, before claiming a precise value such as 3.01×10^-3.
minor comments (4)
  1. [Methods (Simulation settings)] The sentence 'effective episodes exclude cases where the simulation or AV system behaves abnormally, for example, AV getting stuck' needs elaboration: how many episodes were excluded, from what total, and whether the exclusion was determined before or after observing the outcome. This is relevant to the bias concern in the major comments.
  2. [Figure 6c] Figure 6c plots 'Collisions per Mile' against 'Number of Tests' but the curve is not labeled with the estimator used (weighted or unweighted), and no error bars or confidence bands are shown.
  3. [Results (Demonstration of AV testing in physical test tracks)] The physical DIT test bypassed the perception module by feeding ground-truth object states directly into the AV stack. This should be explicitly stated as a limitation in the main text rather than only in the Discussion, since it means the physical DIT did not exercise the full sense-plan-act loop.
  4. [Abstract and Introduction] The phrase 'passes eight out of 14 scenarios' in the Introduction should be corrected to match Table 1 and the Abstract after the inconsistency is resolved.

Circularity Check

3 steps flagged · score 8.0 of 10

Central DIT crash-rate estimate is not shown to come from the unbiased importance-sampling estimator; the reported number is consistent with the amplified NADE frequency, and the human benchmark is the same calibration target.

  1. fitted input called prediction [Results / Development process of the Driving Intelligence Test (page 7-8) and Methods / Simulation settings (page 20)]
    "The ground-truth crash rate is 3.00 × 10−6 crashes per mile, while our simulation yielded 2.93 × 10−6 crashes per mile, confirming the fidelity of the NDE. ... Leveraging 5 years (2016-2021) crash data from Michigan, US38, the human error source probabilities are calibrated, ensuring the consistency of crash rate, distribution of simulated crash types and locations with empirical crash statistics."

    The NDE simulation's crash rate is calibrated to the Michigan crash data (Ref 38), and the same Ref 38 value (3.00e-6 crashes per mile) is then used as the human-driver benchmark in the headline comparison. The 'confirmation' of NDE fidelity is therefore a check that the calibration target was hit, not an independent validation. This makes the baseline of the central '1,000 times higher than human' comparison a fitted input rather than an external benchmark.

  2. fitted input called prediction [Results / Results of the Driving Intelligence Test (page 11, Fig. 6c) and Methods / NADE environment for Driving Intelligence Test (page 21)]
    "The crash rate for Autoware.Universe was estimated at 3.01 × 10−3 crashes per mile (Fig. 6c), about 1,000 times higher than that of an average human driver38 (3.00 × 10−6 crash per mile). ... we multiply the base probability of each predicted collision type by a fixed factor of ε = 3000 ... we use 0.1 for intersection collisions and 0.01 for other types ... effective episodes exclude cases where the simulation or AV system behaves abnormally, for example, AV getting stuck."

    The Methods define an unbiased importance-sampling estimator P(A) ≈ (1/n) Σ P(A|x_i) W_q(x_i), with W_q(x) = P(x)/q(x). The implemented NADE changes q by multiplying POV maneuver probabilities by ε=3000, applying location caps, and excluding 'stuck' episodes. The Results report only the point estimate 3.01e-3 and plot 'Collisions per Mile' with no weights, no weight distribution, no effective sample size, and no reweighting for excluded episodes. If 3.01e-3 is the raw collision frequency in the amplified environment, it is directly determined by the amplification parameters; if it is supposed to be weighted, the paper never shows the connection to the defined estimator.

1 more flagged steps
  1. self citation load bearing [Methods / Driving intelligence test (page 20)]
    "In this study, ε is set to 3000, in line with the configuration in Ref27. Theoretical analysis in Ref27 has also demonstrated that testing AVs using the proposed NADE framework is both efficient and unbiased, offering a solid theoretical foundation for the methodology."

    Ref27 is the authors' own prior work (Feng, Yan, Sun, Feng, Liu, Nature Communications 2021). The unbiasedness of the entire DIT crash-rate estimate is imported from that self-citation rather than demonstrated for the present implementation, which changes the sampling distribution via ε=3000, location caps, and episode exclusions. This self-citation is load-bearing because it is the stated justification that the amplified NADE results remain unbiased; without it, the central 3.01e-3 claim lacks a demonstrated theoretical basis in this paper.

full rationale

The paper proposes a framework and applies it to Autoware.Universe. The DLT results are scenario pass/fail counts and are not circular; they are direct observations in constructed scenarios. The DIT, however, has a central numerical claim that is not connected to the unbiased estimator the paper defines. The importance-sampling weights W_q are never reported, and the implementation details (ε=3000, caps of 0.1/0.01, and exclusion of 'stuck' episodes) change the sampling distribution and the estimator without a demonstrated correction. The reported 3.01e-3 crashes per mile is therefore consistent with being the raw amplified NADE frequency, i.e., a number forced by the amplification settings, rather than an unbiased naturalistic crash-rate estimate. Additionally, the NDE simulator's human-error probabilities are calibrated to the same Michigan crash data that supplies the human benchmark (3.00e-6), so the '1,000 times higher' comparison is not independent of the simulation's calibration target. The unbiasedness of the NADE method is also imported from the authors' own Ref27 self-citation rather than established for this implementation. These issues affect the central quantitative conclusion, so the circularity score is high. The DLT pass-count inconsistency (6 vs 8 in the text) is not itself circularity but further indicates reporting unreliability. Honest non-finding is not appropriate here because the central crash-rate claim does reduce, on the paper's own description, to amplified inputs and a self-cited unbiasedness guarantee.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. NADE and behavioral safety are conceptual constructs carried over from prior work, so the ledger mainly records fitted amplification parameters and assumptions about the unbiasedness and fidelity of the simulation.

free parameters (3)
  • epsilon (NADE amplification) = 3000
    Set in Methods in line with Ref 27; controls how strongly the Principal Other Vehicle's adversarial maneuvers are amplified. The reported crash rate depends on this value if importance-sampling weights are not applied.
  • Location-specific upper bounds for amplified collision probabilities = 0.1 for intersection collisions, 0.01 for other types
    Chosen by authors in Methods to balance crash type diversity; alters the NADE sampling distribution without demonstrating the corresponding change to the likelihood ratio W_q(x).
  • Human error source probabilities = Not given explicitly; calibrated to Michigan crash data
    Assigned to simulated vehicles in Methods and calibrated so the NDE reproduces the ground-truth crash rate and distributions. This makes the human crash-rate baseline an input to the simulation rather than an independent benchmark.
assumptions (5)
  • ad hoc to paper The importance-sampling estimator remains unbiased for the implemented q(x) with epsilon multipliers, location caps, and episode exclusions.
    Methods relies on W_q(x) = P(x)/q(x), but the implemented NADE uses epsilon, caps, and excludes stuck episodes, so the stated estimator does not necessarily correspond to the true sampling distribution.
  • domain assumption The simulated NDE crash rate and composition are a valid benchmark for human driver crash rate.
    The NDE is calibrated to Michigan crash data and then compared with the same Michigan human crash rate; this assumes the simulated background traffic represents real human exposure and risk in the AV's ODD.
  • domain assumption BVs in DLT follow deterministic, non-reactive trajectories and a crash in any of 3 repeated runs constitutes failure.
    This simplifies physical execution but changes the distribution of test difficulty; pass/fail outcome is sensitive to the number of repeated runs and to BV non-reactiveness.
  • domain assumption The 14 selected scenarios cover 71.5% of crash types in the urban ODD.
    The coverage claim is based on Washtenaw County crash statistics (Ref 26), not on a validated completeness analysis of the AV's ODD.
  • standard math BV actions follow a Markovian conditional distribution P(u(k)|s(k)).
    Used in Methods to factor episode probability over timesteps; standard but an idealization of real traffic behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Behavioral Safety Assessment towards Large-scale Deployment of Autonomous Vehicles." pith.science (2026). https://pith.science/paper/FCAFWQO5

@misc{pith2026250516214,
  author       = {Pith},
  title        = {Pith review of: Behavioral Safety Assessment towards Large-scale Deployment of Autonomous Vehicles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FCAFWQO5}},
  note         = {Machine review of arXiv:2505.16214}
}
read the original abstract

Autonomous vehicles (AVs) have significantly advanced in real-world deployment in recent years, yet safety continues to be a critical barrier to widespread adoption. Traditional functional safety approaches, which primarily verify the reliability, robustness, and adequacy of AV hardware and software systems from a vehicle-centric perspective, do not sufficiently address the AV's broader interactions and behavioral impact on the surrounding traffic environment. To overcome this limitation, we propose a paradigm shift toward behavioral safety, a comprehensive approach focused on evaluating AV responses and interactions within traffic environment. To systematically assess behavioral safety, we introduce a third-party AV safety assessment framework comprising two complementary evaluation components: Driver Licensing Test and Driving Intelligence Test. The Driver Licensing Test evaluates AV's reactive behaviors under controlled scenarios, ensuring basic behavioral competency. In contrast, the Driving Intelligence Test assesses AV's interactive behaviors within naturalistic traffic conditions, quantifying the frequency of safety-critical events to deliver statistically meaningful safety metrics before large-scale deployment. We validated our proposed framework using \texttt{Autoware.Universe}, an open-source Level 4 AV, tested both in simulated environments and on the physical test track at the University of Michigan's Mcity Testing Facility. The results indicate that \texttt{Autoware.Universe} passed 6 out of 14 scenarios and exhibited a crash rate of 3.01e-3 crashes per mile, approximately 1,000 times higher than average human driver crash rate. During the tests, we also uncovered several unknown unsafe scenarios for \texttt{Autoware.Universe}. These findings underscore the necessity of behavioral safety evaluations for improving AV safety performance prior to widespread public deployment.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 55 canonical work pages

  1. [1]

    Symon, E. (2024). CPUC Announces Expansion of Paid Robotaxis in Bay Area & LA. California Globe. https://californiaglobe.com/fr/cpuc-announces-expansion-of-paid-robotaxis-in-bay-area-la/

    work page 2024

  2. [2]

    Zhang, Y. (2024). Baidu's robotaxis orders surge, paving way for large-scale autonomous ride-hailing services. Global Times. https://www.globaltimes.cn/page/202407/1315784.shtml

    work page 2024

  3. [3]

    Lu, Y. (2023). San Francisco Balks at Expanding Driverless Car Services on City’s Roads. The New York Times. https://www.nytimes.com/2023/08/09/technology/san-francisco-driverless-cars.html

    work page 2023

  4. [4]

    Thadani, T. (2023). Cruise recalls all its driverless cars after pedestrian hit and dragged. The Washington Post. https://www.washingtonpost.com/technology/2023/11/08/cruise-crash-driverless-recall/

    work page 2023

  5. [5]

    Moye, B. (2025). AAA: Fear in Self-Driving Vehicles Persists. AAA Newsroom. https://newsroom.aaa.com/2025/02/aaa-fear-in-self-driving-vehicles-persists/

    work page 2025

  6. [6]

    Jing, P., Xu, G., Chen, Y., Shi, Y., & Zhan, F. (2020). The determinants behind the acceptance of autonomous vehicles: A systematic review. Sustainability, 12(5), 1719

    work page 2020

  7. [7]

    https://en.wikipedia.org/wiki/New_Car_Assessment_Program

    NCAP. https://en.wikipedia.org/wiki/New_Car_Assessment_Program

    work page

  8. [8]

    https://en.wikipedia.org/wiki/Euro_NCAP

    Euro NCAP. https://en.wikipedia.org/wiki/Euro_NCAP

    work page

Show all 57 references
  1. [9]

    https://en.wikipedia.org/wiki/Insurance_Institute_for_Highway_Safety

    IIHS. https://en.wikipedia.org/wiki/Insurance_Institute_for_Highway_Safety

  2. [10]

    https://en.wikipedia.org/wiki/C-NCAP

    China NCAP. https://en.wikipedia.org/wiki/C-NCAP

  3. [11]

    ISO 26262, Road vehicles — Functional safety

  4. [12]

    ISO 21448, Road vehicles — Safety of the intended functionality

  5. [13]

    ISO/PAS 8800, Road vehicles — Safety and artificial intelligence

  6. [14]

    ISO 34502, Road vehicles — Test scenarios for automated driving systems — Scenario based safety evaluation framework

  7. [15]

    UL 4600, Standard for safety for the evaluation of autonomous products

  8. [16]

    CertiCAV Assurance Paper

    Connected Places Catapult (CPC) and WMG, University of Warwick. CertiCAV Assurance Paper. https://cp-catapult.s3.amazonaws.com/uploads/2021/06/CertiCAV-Assurance-Paper-v1-4.pdf

    2021

  9. [17]

    Favarò, F., Fraade-Blanar, L., Schnelle, S., Victor, T., Peña, M., Engstrom, J., Scanlon, J., Kusano, K., and Smith, D. (2023). Building a credible case for safety: Waymo's approach for the determination of absence of unreasonable risk

    2023

  10. [18]

    Cruise safety report. (2023). https://assets.ctfassets.net/95kuvdv8zn1v/zKJHD7X22fNzpAJztpd5K/ac6cd2419f2665000e4eac3b7d16ad1 c/Cruise_Safety_Report_2022_sm-optimized.pdf

    2023

  11. [19]

    Safety Case Framework development and tailoring

    Aurora. Safety Case Framework development and tailoring. https://blog.aurora.tech/safety/safety-case- framework-development-and-tailoring

  12. [20]

    Abdel-Aty, M., & Ding, S. (2024). A matched case-control analysis of autonomous vs human-driven vehicle accidents. Nature Communications, 15(1), 4931

    2024

  13. [21]

    Yan, X., Zou, Z., Feng, S., Zhu, H., Sun, H., & Liu, H. X. (2023). Learning naturalistic driving environment with statistical realism. Nature Communications, 14(1), 2037. Page 23 of 25

    2023

  14. [22]

    X., & Feng, S

    Liu, H. X., & Feng, S. (2024). Curse of rarity for autonomous vehicles. Nature Communications, 15(1), 4808

    2024

  15. [23]

    Feng, S., Sun, H., Yan, X., Zhu, H., Zou, Z., Shen, S., & Liu, H. X. (2023). Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615(7953), 620-627

    2023

  16. [24]

    https://www.tuvsud.com/en- us/industries/mobility-and-automotive/automotive-and-oem/autonomous-driving/assessment-of-automated- vehicles-with-scenario-based-testing

    TUV SUD, Assessment of automated vehicles with scenario-based testing. https://www.tuvsud.com/en- us/industries/mobility-and-automotive/automotive-and-oem/autonomous-driving/assessment-of-automated- vehicles-with-scenario-based-testing

  17. [25]

    https://www.notateslaapp.com/news/2401/tesla- reveals-how-it-tests-fsd-software-video

    Tesla Reveals How It Tests FSD - A Series of Videos. https://www.notateslaapp.com/news/2401/tesla- reveals-how-it-tests-fsd-software-video

  18. [26]

    Washtenaw Area Transportation Study. (2023). 2021 Traffic Crash Report & 2017-2021 5-Year Crash Trends for Washtenaw County. https://static1.squarespace.com/static/524e0929e4b093015db69c07/t/63dd346d44d5c0070dace95e/1675441 268988/2021+Crash+Report+and+Five-Year+Trends_2_3_23.pdf

    2023

  19. [27]

    Feng, S., Yan, X., Sun, H., Feng, Y., & Liu, H. X. (2021). Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment. Nature Communications, 12(1), 748

    2021

  20. [28]

    National Highway Traffic Safety Administration (NHTSA). (2024). Overview of motor vehicle traffic crashes in 2022. https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/813560

    2024

  21. [29]

    Kalra, N., & Paddock, S. M. (2016). Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice, 94, 182- 193

    2016

  22. [30]

    D., Foderaro, F., Yanagisawa, M., Najm, W

    Swanson, E. D., Foderaro, F., Yanagisawa, M., Najm, W. G., Azeredo, P., & John, A. (2019). Statistics of light-vehicle pre-crash scenarios based on 2011–2015 national crash data (No. DOT HS 812 745). United States. Department of Transportation. National Highway Traffic Safety ...

    2019

  23. [31]

    O., & de Boer, N

    de Gelder, E., den Camp, O. O., & de Boer, N. (2020). Scenario categories for the assessment of automated vehicles. CETRAN, Singapore, Version, 1

    2020

  24. [32]

    C., & Chaka, M

    Thorn, E., Kimmel, S. C., & Chaka, M. (2018). A framework for automated driving system testable cases and scenarios (No. DOT HS 812 623). United States. Department of Transportation. National Highway Traffic Safety Administration

    2018

  25. [33]

    Waymo LLC. (2021). Waymo Safety Report. https://downloads.ctfassets.net/sv23gofxcuiz/4gZ7ZUxd4SRj1D1W6z3rpR/2ea16814cdb42f9e8eb34cae4f3 0b35d/2021-03-waymo-safety-report.pdf

    2021

  26. [34]

    Peng, H. (2019). Mcity ABC test: A concept to assess the safety performance of highly automated vehicles. https://mcity.umich.edu/wp-content/uploads/2019/01/mcity-whitepaper-ABC-test.pdf

    2019

  27. [35]

    Wilson, B. et al. (2021). Argoverse 2: Next generation datasets for self-driving perception and forecasting. Proceedings of the Neural Information Processing Systems (NeurIPS)

    2021

  28. [36]

    Sun, H. et al. (2025). TeraSim: Uncovering Unknown Unsafe Events for Autonomous Vehicles through Generative Simulation. arXiv preprint arXiv:2503.03629

    2025 arXiv

  29. [37]

    Yan, X., Feng, S., Sun, H., & Liu, H. X. (2021). Distributionally consistent simulation of naturalistic driving environment for autonomous vehicle testing. arXiv preprint arXiv:2101.02828

    2021 arXiv

  30. [38]

    Michigan Traffic Crash Facts. (2024). Michigan Traffic Crash Facts Data Query Tool. https://www.michigantrafficcrashfacts.org/pub

    2024

  31. [39]

    Autoware.Universe https://github.com/autowarefoundation/autoware.universe

    Autoware Foundation. Autoware.Universe https://github.com/autowarefoundation/autoware.universe

  32. [40]

    Johansson, G., & Rumar, K. (1971). Drivers' brake reaction times. Human factors, 13(1), 23-27

    1971

  33. [41]

    Xu, S., & Peng, H. (2019). Design, analysis, and experiments of preview path tracking control for autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems, 21(1), 48-58

    2019

  34. [42]

    Yu, W. et al. (2024). Online legal driving behavior monitoring for self-driving vehicles. Nature Communications, 15(1), 408. Page 24 of 25

    2024

  35. [43]

    and Kolodny, L

    Wayland, M. and Kolodny, L. (2022). Self-driving cars from GM’s Cruise block San Francisco intersection in latest problem for autonomous vehicles. CNBC. https://www.cnbc.com/2022/07/01/self-driving-cars- from-gms-cruise-block-san-francisco-streets.html

    2022

  36. [44]

    Hartman, A. (2005). Software and hardware testing using combinatorial covering suites. In Graph Theory, Combinatorics and Algorithms: Interdisciplinary Applications (pp. 237-266). Boston, MA: Springer US

    2005

  37. [45]

    Owen, A. B. Monte Carlo theory, methods and examples. Art Owen https://artowen.su.domains/mc/ (2013)

    2013

  38. [46]

    Lopez, P. A. et al. (2018). Microscopic traffic simulation using SUMO. In 2018 21st international conference on intelligent transportation systems (ITSC) (pp. 2575-2582). IEEE

    2018

  39. [47]

    Treiber, M., Hennecke, A., & Helbing, D. (2000). Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2), 1805

    2000

  40. [48]

    Erdmann, J. (2015). SUMO’s lane-changing model. In Modeling Mobility with Open Data: 2nd SUMO Conference 2014 Berlin, Germany, May 15-16, 2014 (pp. 105-123). Springer International Publishing

    2015

  41. [49]

    Hu, X., Zheng, Z., Chen, D., & Sun, J. (2023). Autonomous vehicle’s impact on traffic: Empirical evidence from waymo open dataset and implications from modelling. IEEE Transactions on Intelligent Transportation Systems, 24(6), 6711-6724

    2023

  42. [50]

    Sangster, J., Rakha, H., & Du, J. (2013). Application of naturalistic driving data to modeling of driver car- following behavior. Transportation research record, 2390(1), 20-33

    2013

  43. [51]

    Ettinger, S. et al. (2021). Large scale interactive motion forecasting for autonomous driving: The Waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9710-9719)

    2021

  44. [52]

    Dingus, T. et al. (2006). The 100-car naturalistic driving study, Phase II-results of the 100-car field experiment (No. DOT-HS-810-593). United States. Department of Transportation. National Highway Traffic Safety Administration

    2006

  45. [53]

    Chang A. X. et al. ShapeNet: an information-rich 3D model repository. Preprint at https:// arxiv.org/abs/1512.03012 (2015)

    2015 arXiv

  46. [54]

    Feng, Y., & Liu, H. X. (2019). Augmented reality for robocars. IEEE Spectrum, 56(12), 22-27

    2019

  47. [55]

    Feng, S., Feng, Y., Yan, X., Shen, S., Xu, S., & Liu, H. X. (2020). Safety assessment of highly automated driving systems in test tracks: A new framework. Accident Analysis & Prevention, 144, 105664

    2020

  48. [56]

    Krajewski, R. et al. (2020). The rounD dataset: A drone dataset of road user trajectories at roundabouts in Germany. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)

    2020

  49. [57]

    Singh, S. (2015). Critical reasons for crashes investigated in the national motor vehicle crash causation survey (No. DOT HS 812 115). Acknowledgments This research was partially funded by the U .S. Department of Transportation (USDOT) Automated Driving System Demonstration Gr...

    2015

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.