{"id":"1fd5dc45-d0d3-4c52-83f9-b06c23333b20","arxiv_id":"2505.16214","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-part behavioral safety evaluation framework for AVs is demonstrated on Autoware.Universe; the reported crash rate is about 3 events per 1,000 miles, roughly 1,000 times the human benchmark.","lead":"This paper proposes a two-part testing program for autonomous vehicles: a scenario-based Driver Licensing Test and a statistical Driving Intelligence Test meant to measure crash rates in simulated traffic. The authors applied it to Autoware.Universe and report a crash rate about 1,000 times higher than human drivers, while flagging several specific failure modes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 3.01e-3 crashes/mile is not shown to come from the importance-sampling estimator that would make it unbiased; the NADE implementation changes q(x) via POV amplification, location caps, and episode exclusions, so the central 1,000x-vs-human claim is unsupported as written.","rationale":"The reader's REJECT verdict is supported by this concern. The central claim is the only quantitative basis for the paper's stated conclusion that Autoware.Universe is about 1,000 times less safe than a human driver, and that claim requires the importance-sampling estimator to be unbiased. The Methods define the estimator, but the Results never demonstrate that the reported 3.01e-3 is the weighted estimate rather than an unweighted frequency from the adversarially amplified NADE. The implementation details—ε=3000, location-specific caps, and exclusion of stuck episodes—change the effective sampling distribution q(x), and the paper neither recomputes W_q with the implemented q nor provides a sensitivity analysis for exclusions. This is not a cosmetic issue: importance sampling is precisely the mechanism invoked to overcome the curse of rarity, and unbiasedness is the paper's own stated guarantee. The same weakness is identified by the reader's weakest_assumption, and I agree with it. Secondary issues, such as the 6/14 versus 8/14 discrepancy in the DLT results, do not change the assessment. The verdict should remain REJECT as submitted, although a corrected statistical analysis with explicit weights, confidence intervals, and exclusion sensitivity could make the central claim verifiable.","tokens_in":21563,"tokens_out":4648,"duration_ms":36466,"concrete_test":"Using the released artifacts (Zenodo records 15446750 for code and 15446739 for data), recompute the DIT crash rate from the logged episodes by the book: for each episode, estimate P(x) from the TeraSim NDE models, q(x) from the actual NADE sampling procedure including ε=3000, location caps, and the treatment of stuck episodes, then compute the weighted estimator with W_q and its confidence interval. Also report the unweighted collision frequency. If the weighted rate differs materially from 3.01e-3 (e.g., outside the Poisson confidence interval around the reported value) or has a confidence interval overlapping the 3.00e-6 human benchmark, the central claim fails. If the released pipeline already applies the weights, this re-run will confirm that; the key is to distinguish explicitly between the weighted and unweighted cases.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim—the DIT crash rate of 3.01×10−3 crashes per mile, roughly 1,000× the human 3.00×10−6—depends entirely on the unbiasedness of importance sampling, but the text never connects the reported number to the estimator it defines. In Methods/DIT, the estimator is P(A)≈(1/n)Σ P(A|x_i)W_q(x_i) with W_q(x)=∏P(u(k)|s(k))/q(u(k)|s(k)). In Methods/Simulation settings/NADE, the implemented q is built by multiplying the POV's maneuver probability by ε=3000, applying location-dependent caps of 0.1 for intersection collisions and 0.01 for others, and then 'effective episodes exclude cases where the simulation or AV system behaves abnormally, for example, AV getting stuck.' The Results report only a single point estimate, 3.01e-3, and Fig. 6c plots 'Collisions per Mile' against episode count, with no weights, no weight distributions, no effective sample size, and no confidence intervals. If the estimate is the raw collision frequency in the amplified NADE, it is biased upward by the amplification. If weights were used, the q in the denominator must be the actually implemented q with caps and exclusions, and the dropped episodes need a reweighting or a demonstrated non-bias argument; neither is given. Compounding this, the NDE human-error model is calibrated to the same Michigan crash data (Ref. 38) used as the human benchmark, so the comparison baseline is not independent of the simulation model. The headline numerical comparison is therefore not supported as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a third-party behavioral safety assessment framework for autonomous vehicles, consisting of a Driver Licensing Test (DLT) for scenario-based competency evaluation and a Driving Intelligence Test (DIT) for statistical crash-rate estimation in a naturalistic adversarial environment. The framework is demonstrated on the open-source Level 4 stack Autoware.Universe in simulation at the Mcity test facility and with limited physical tests using a Tesla FSD vehicle and an Autoware-equipped Lincoln MKZ. The headline results are that Autoware.Universe passed 6 (elsewhere 8) of 14 DLT scenarios and that the DIT estimated a crash rate of 3.01×10^-3 per mile, about 1000 times the human benchmark of 3.00×10^-6 per mile.","tokens_in":21887,"tokens_out":3154,"duration_ms":26804,"significance":"If the central quantitative claims are correct, the paper would provide an important third-party, black-box safety evaluation methodology and an unusually public assessment of a widely used open-source AV stack. The framework is timely and the deployment of both scenario-based and statistical testing is conceptually attractive. The paper's strengths include the use of open-source tools (TeraSim, Autoware.Universe), publication of experimental data and code on Zenodo, and the demonstration that unknown unsafe behaviors can be discovered and fed back to developers. However, the significance is conditional: the statistical unbiasedness of the DIT estimate is the foundation of the headline crash-rate comparison, and that foundation is not established in the manuscript as written.","major_comments":[{"comment":"The reported DIT crash rate of 3.01×10^-3 crashes per mile is not connected to the importance-sampling estimator defined in the Methods. The estimator in the Methods is P(A) = (1/n) Σ P(A|x_i) W_q(x_i) with W_q(x) = ∏ P(u(k)|s(k))/q(u(k)|s(k)), but the Results report only a single point estimate, with no weights, no weight summary, no effective sample size, and no confidence intervals. The NADE implementation described later multiplies POV maneuver probabilities by ε=3000, applies location-specific caps (0.1 for intersection collisions, 0.01 for others), and excludes 'effective episodes' where the AV gets stuck. The q(x) actually implemented therefore differs from the q(x) in the importance-sampling weight unless the caps, the ε multiplier, and the episode exclusions are all incorporated into W_q. The manuscript must show either the weighted estimate with the true implementation-consistent q, or a demonstrated argument that the reported number is unbiased. As written, the reported 3.01e-3 could be a biased raw frequency of the amplified NADE, which would invalidate the central 1000× comparison to the human crash rate.","section":"Methods (Driving intelligence test; Simulation settings) and Results (Results of the Driving Intelligence Test)"},{"comment":"The number of DLT scenarios passed by Autoware.Universe is inconsistent across the manuscript. The Abstract states 'passed 6 out of 14 scenarios,' while the Introduction states 'passes eight out of 14 scenarios' (page 4). Table 1 lists P for scenarios b, f, j, k, m, n, which is 6 passes and 8 failures, and the Results section reports 'successfully passed 6 out of 14 scenarios.' This numerical contradiction affects the reported outcome and must be resolved in revision.","section":"Abstract and Introduction vs. Table 1 and Results"},{"comment":"The human benchmark used in the headline comparison (3.00×10^-6 crashes per mile, Ref. 38) is the same Michigan crash dataset used to calibrate the human-error model in the NDE simulator, whose validation yields 2.93×10^-6 crashes per mile. This means the 'average human driver' rate is not an independent benchmark relative to the simulation pipeline; it is the target used to tune the background-agent error probabilities. While an external crash statistic is a legitimate reference, the paper should discuss this calibration-dependence and state clearly that the comparison is to an aggregate Michigan crash rate, not to a matched human-driver performance in the same ODD. Without such discussion, the claim of 'about 1,000 times higher' overstates the independence of the comparison.","section":"Results (Results of the Driving Intelligence Test) and Methods (Simulation settings, NDE simulator)"},{"comment":"The DIT estimate is reported as a single point value with no measure of statistical uncertainty. The experiment comprised about 3,500 episodes; even if each episode is roughly one mile, a crash rate near 3e-3 would correspond to only about 10 observed crashes, yielding a wide confidence interval. The paper should report a confidence interval or credible interval for the crash rate, and ideally a convergence plot of the weighted estimator, before claiming a precise value such as 3.01×10^-3.","section":"Results (Results of the Driving Intelligence Test)"}],"minor_comments":[{"comment":"The sentence 'effective episodes exclude cases where the simulation or AV system behaves abnormally, for example, AV getting stuck' needs elaboration: how many episodes were excluded, from what total, and whether the exclusion was determined before or after observing the outcome. This is relevant to the bias concern in the major comments.","section":"Methods (Simulation settings)"},{"comment":"Figure 6c plots 'Collisions per Mile' against 'Number of Tests' but the curve is not labeled with the estimator used (weighted or unweighted), and no error bars or confidence bands are shown.","section":"Figure 6c"},{"comment":"The physical DIT test bypassed the perception module by feeding ground-truth object states directly into the AV stack. This should be explicitly stated as a limitation in the main text rather than only in the Discussion, since it means the physical DIT did not exercise the full sense-plan-act loop.","section":"Results (Demonstration of AV testing in physical test tracks)"},{"comment":"The phrase 'passes eight out of 14 scenarios' in the Introduction should be corrected to match Table 1 and the Abstract after the inconsistency is resolved.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The referee's core concern about the importance-sampling weights is valid and must be addressed by the authors. I am recommending major_revision rather than reject because the underlying data and code are public, so the authors have the means to re-analyze or re-present the DIT estimate correctly. I would also note that the relationship to the authors' prior work (Ref. 27) should be clarified; the NADE approach appears to be largely carried over from that paper, and the present contribution should be framed around the novel DLT/DIT integration and the concrete Autoware results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper's conceptual frame—split behavioral safety into a competency test (DLT) and a statistical test (DIT)—is sensible, and the Autoware.Universe evaluation is genuinely useful. Second: the headline DIT crash rate, 3.01e-3 per mile versus human 3.00e-6, is not supported as written. The stress-test note is correct. The Methods define an importance-sampling estimator with weight W_q(x), but the Results report a single point estimate with no weights, no confidence intervals, and no discussion of how the implemented NADE q(x)—POV amplification by epsilon=3000, location caps, and exclusion of 'abnormal' episodes—maps to the weight. If the estimate is the raw collision frequency in the amplified distribution, it's biased up by construction. If weights were used, the denominator q must be the actually implemented q, and the dropped episodes need a non-bias argument. Neither is shown.\n\nSeparately, the abstract says Autoware passed 6 of 14 DLT scenarios, the intro says 8 of 14, and Table 1 shows 6 passes. That is a concrete internal inconsistency. The human benchmark also comes from Michigan crash data (Ref 38), and the NDE's human-error probabilities are calibrated to the same data, so the baseline is not independent of the simulation model. That doesn't kill the comparison, but it deserves discussion.\n\nWhat's actually good: the DLT scenario library is concrete and checkable, the table reports N, pass/fail, and TTC metrics, and the identified Autoware bugs (roundabout stuck, intersection prediction error) are credible and useful. The authors shipped code and data on Zenodo, which is real evidence. The physical test demonstrations, especially the AR platform, are a nice feasibility proof. The black-box third-party evaluation idea is timely and well motivated.\n\nVerdict: this deserves a serious referee, and the overall direction is sound, but the paper as submitted should not be accepted because the central quantitative claim does not follow from the presented analysis. A corrected DIT analysis with proper importance weights, confidence intervals, and honest treatment of exclusions could salvage the headline. I would send it to a venue with a strong statistics reviewer and expect major revision.","headline":"The two-part behavioral safety framework is worth engaging, but the headline 1,000x crash-rate claim is not supported by the presented statistics, and the text has a 6-vs-8 DLT pass discrepancy.","tokens_in":22507,"tokens_out":2022,"would_cite":true,"duration_ms":16970,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that behavioral safety, not component-level functional safety, is the missing evaluation layer for autonomous vehicles, and it reports a two-test framework that puts the open-source Level 4 stack Autoware.Universe at…","keywords":["behavioral safety","autonomous vehicle testing","Driver Licensing Test","Driving Intelligence Test","importance sampling","naturalistic adversarial driving environment","crash rate estimation","Autoware.Universe"],"falsifier":"Recompute the DIT crash rate two ways: once with the importance-sampling weights applied and once treating each episode equally. If the two estimates differ by a large factor, or if instrumenting the simulator shows the implemented $q(\\mathbf{x})$ (with $\\epsilon=3000$ and the 0.1/0.01 caps) does not match the $q(\\mathbf{x})$ used in $W_q$, then the reported $3.01\\times10^{-3}$ crashes per mile is not a naturalistic estimate.","tokens_in":21298,"feed_emoji":"🚗","tokens_out":7441,"duration_ms":58901,"temperature":0.7,"pith_summary":"Autonomous-vehicle safety is usually checked component by component, but the authors argue that what matters at scale is behavioral safety: how the vehicle reacts to and interacts with the surrounding traffic. They propose a third-party evaluation with two parts, the Driver Licensing Test (controlled reactive scenarios) and the Driving Intelligence Test (naturalistic traffic with rare events amplified for statistical measurement). Applied to the open-source Level 4 stack Autoware.Universe, the paper reports six passed licensing scenarios out of fourteen (the results text also says eight, while the abstract and Table 1 say six) and a crash rate of $3.01\\times10^{-3}$ crashes per mile, roughly a thousand times the average human driver's $3.00\\times10^{-6}$ per mile. The paper's point is that this kind of black-box behavioral testing can supply a quantitative safety measure before large-scale deployment.","feed_headline":"Open-source robotaxi stack crashes 1,000x more than human drivers","feed_subtitle":"Two-part safety test clocks Autoware.Universe at 3 crashes per 1,000 miles vs. 3 per million for humans.","key_machinery":"The load-bearing object is the Naturalistic and Adversarial Driving Environment (NADE), built on top of a naturalistic driving environment whose crash rates and crash-type mix are calibrated against real-world Michigan data. At each critical moment NADE picks the Principal Other Vehicle (POV), the background car whose action most threatens the AV, and amplifies that action's probability by a factor of $\\epsilon = 3000$ with location-specific caps (0.1 for intersection collisions, 0.01 for others). Each episode is then reweighted by the likelihood ratio $W_q(\\mathbf{x}) = P(\\mathbf{x})/q(\\mathbf{x})$, the ratio of the naturalistic probability to the amplified probability, so that the crash count can be converted back into a naturalistic rate. The Driver Licensing Test is the scenario-scale half of the machinery: parameterized cases from naturalistic datasets, binned into risk levels by the deceleration required to avoid collision, with a pass/fail verdict per scenario.","core_discovery":"The paper's central claim is that behavioral safety is a distinct and necessary evaluation layer, separate from crashworthiness and functional safety, and that it can be measured without any access to the AV's internal design. The Driver Licensing Test checks reactive competence in scenarios drawn from real crash data, and the Driving Intelligence Test estimates the frequency of safety-critical events by placing the AV in an environment where background vehicles' adversarial maneuvers are amplified by importance sampling and then reweighted to remain unbiased. On these tests, the paper reports, Autoware.Universe passed six of the fourteen licensing scenarios and produced an estimated $3.01\\times10^{-3}$ crashes per mile, about one thousand times the human benchmark of $3.00\\times10^{-6}$. The authors also report discovering previously unknown failure modes, including roundabout replanning errors, rigid right-of-way yielding, and trajectory prediction that ignores traffic-light state, and they demonstrate the same testing approach on a physical track with two different vehicles.","pith_inferences":["A check the paper leaves implicit: if the implemented sampling distribution differs from the one in the importance-sampling weight, for instance if the $\\epsilon=3000$ amplification and location caps are not exactly reflected in $q(\\mathbf{x})$, the reported $3.01\\times10^{-3}$ figure could be an unweighted frequency rather than a naturalistic rate.","The same two-test architecture could be extended to measure non-safety behaviors that affect public acceptance, such as blocking intersections or delaying traffic, since the DIT environment already records interactions over full trips.","Because the simulated DIT bypasses the perception module, the estimate speaks to decision-making and planning; a version that feeds simulated sensor data could likely reveal additional perception-related failures at scale."],"forward_implications":["If the crash-rate estimate is correct, an open-source Level 4 stack in this urban ODD is roughly three orders of magnitude away from the average human driver's crash rate, so behavioral testing can separate deployment-ready systems from those that are not.","The DLT/DIT structure gives regulators a reusable, black-box testing template: the AV under test needs no data interface, only its own sensors and behavior, so the same protocol can compare different vendors.","The DIT's discovered failure modes, including roundabout stalls, right-of-way yielding, and trajectory prediction that ignores red lights, show how simulation-based behavioral testing can feed concrete fixes back to developers in a closed loop.","The framework converts 'how safe is safe enough' from a slogan into a number: a statistically reweighted crash rate per mile, with crash-type and severity distributions attached.","Because the same scenarios and environment are used in simulation and on a physical test track, results from one setting can be cross-checked against the other before large-scale deployment."],"supporting_citations":[{"why":"Supplies the NADE importance-sampling methodology and the epsilon=3000 amplification used in the Driving Intelligence Test.","marker":"Ref27"},{"why":"Establishes the curse of rarity and provides the dense-validation approach and crash-severity classification the DIT builds on.","marker":"Ref23"},{"why":"Defines the naturalistic driving environment with statistical realism that underlies both the DLT calibration and the DIT simulator.","marker":"Ref21"},{"why":"Argoverse 2 dataset from which cut-in segments and other scenario parameters for DLT calibration are extracted.","marker":"Ref35"},{"why":"TeraSim, the open-source simulation platform used to construct the Mcity NDE for the DIT.","marker":"Ref36"},{"why":"Michigan Traffic Crash Facts provides the human benchmark crash rate of 3.00e-6 per mile and the crash-type distributions used for validation.","marker":"Ref38"},{"why":"The Mcity ABC test scenario library that guides selection of the 14 Driver Licensing Test scenarios.","marker":"Ref34"},{"why":"Autoware.Universe, the open-source Level 4 ADS that is the system under test.","marker":"Ref39"},{"why":"Documents the rarity of safety-critical events motivating the NADE acceleration approach in the DIT.","marker":"Ref22"}],"fun_headline_variants":["Open-source AV stack crashes 1,000x more than human drivers in new safety test","Autoware.Universe fails 8 of 14 safety scenarios, 1000x human crash rate","Robotaxi safety test: open-source stack crashes 1000x more than humans","New AV safety test: open-source stack crashes 1000x more than drivers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole DIT crash-rate estimate is only unbiased if the simulator actually samples from the same distribution that appears in the importance-sampling weights, and if episodes thrown away because the AV got stuck do not bias the remaining data.","fun_headline_variants_meta":{"raw":{"variants":["Open-source AV stack crashes 1,000x more than human drivers in new safety test","Autoware.Universe fails 8 of 14 safety scenarios, 1000x human crash rate","Robotaxi safety test: open-source stack crashes 1000x more than humans","New AV safety test: open-source stack crashes 1000x more than drivers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001019,"raw_usage":{"total_tokens":4348,"prompt_tokens":1041,"completion_tokens":3307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":3213}},"tokens_in":657,"tokens_out":3307,"duration_ms":18908,"temperature":1.0,"reasoning_tokens":3213,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:04:49.707728+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the DIT crash rate two ways: once with the importance-sampling weights applied and once treating each episode equally. If the two estimates differ by a large factor, or if instrumenting the simulator shows the implemented $q(\\mathbf{x})$ (with $\\epsilon=3000$ and the 0.1/0.01 caps) does not match the $q(\\mathbf{x})$ used in $W_q$, then the reported $3.01\\times10^{-3}$ crashes per mile is not a naturalistic estimate.","supporting_citations":[],"review_version":1}