{"id":"5af7096b-35ad-46c6-a26e-fc070a204e31","arxiv_id":"2505.02274","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Scenario-based AV testing is modeled statistically, showing neither it nor mile-based testing universally outperforms the other, with a proposed (epsilon, alpha)-REF criterion for simulation fidelity.","lead":"This paper argues that scenario-based testing of autonomous vehicles needs a proper statistical foundation, and it sketches simple probabilistic models to compare scenario-based and mile-based testing. A smart generalist might read it because it connects two mature fields, autonomous vehicle safety assessment and classical software reliability testing, to show that neither testing approach is universally better.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The single-failure-region/perfect-fix model cannot express the paper's own 'rare vs common bug' tradeoff, so the central no-universal-superiority claim is only demonstrated for a restricted bug model; the 'more bugs' insight rests on citation, not on the proof-of-concept.","rationale":"I read the paper as a position/proof-of-concept piece: the goal is to map AV testing to established software-testing results and argue that neither scenario-based nor mile-based testing has universal superiority. The mathematical examples in Section IV.B do support the weak conclusion that, within the stated single-failure-region model, either method can be better depending on parameters. The reader's CONDITIONAL verdict already captures this. My stress-test sharpens the limitation: the missing multi-region structure is not a mere quantitative approximation but a representational gap. The motivating '5,000-year vs 5-year bug' story is precisely about multiple failure regions with different operational probabilities, and Eq. (7) cannot encode it. Thus the paper's own analysis does not demonstrate the 'more bugs' insight; it imports it from [19] and illustrates only a one-bug version. This does not overturn the conclusion 'neither universally outperforms,' which is robust and well-supported by the examples, but it reduces the proof-of-concept's contribution from establishing the central intuition to illustrating a special case. Since the paper explicitly acknowledges [18]'s richer model and calls for future work, the appropriate verdict remains CONDITIONAL; my concern does not move the verdict but explains why the conditionality is substantive.","tokens_in":13282,"tokens_out":19006,"duration_ms":251508,"concrete_test":"Implement the two-failure-region version of Eqs. (5)-(7): let F1 and F2 be disjoint failure regions with OP probabilities q1=1e-5 and q2=1e-9; let scenario-based testing allocate all t=1e5 tests to the partition containing F2 (detection probability per test approximately 1) and mile-based testing draw t i.i.d. scenarios from the OP. Compute expected residual pfs after perfect fixes: E_sce is approximately q1 = 1e-5, while E_mile is approximately q1(1-q1)^t + q2(1-q2)^t, which is about 3.68e-6. If these numbers hold, scenario-based testing detects the rare bug but leaves a higher expected pfs than mile-based testing, reproducing the exact tradeoff the single-region model cannot express; the proof-of-concept would then need the multi-region model to support its motivating claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.B's comparison (Eqs. 5-13) assumes a single failure region and a perfect fix, so any detected failure drives pfs to zero and the residual expected pfs is q times the probability of missing the region. This structure cannot represent the motivating example in Section I(iii), where scenario-based testing finds only '5,000-year bugs' while missing '5-year bugs': with one region there is no tradeoff between finding a rare bug and failing to find a common one. Consequently the paper's model does not itself demonstrate the imported software result that 'more bugs does not necessarily imply less reliable' [19]; that claim is carried by analogy and citation, not by Eq. (7). The paper acknowledges the richer model in [18] (footnote 6) and the i.i.d./constant-pfs caveats in Section V, so the simplification is transparent; but transparency does not make the proof-of-concept adequate for the central interpretive claim. The no-universal-superiority conclusion likely survives a two-region extension, but as written the proof-of-concept is not the source of that insight, weakening the paper's stated contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that scenario-based testing of autonomous vehicles currently lacks a rigorous statistical foundation, and it maps scenario-based and mile-based testing onto partition-based and random software testing, respectively. It defines the probability of failure per randomly selected scenario (pfs), presents proof-of-concept models for comparing the debugging effectiveness of the two testing styles under a single-failure-region, perfect-fix assumption, and concludes that neither approach universally outperforms the other. The paper also introduces an (epsilon, alpha)-Risk Estimation Fidelity criterion for synthetic data, with a stepwise workflow and a numerical illustration, and closes with open problems concerning dynamic operational profiles, asymmetric fidelity, and ultra-high reliability.","tokens_in":13492,"tokens_out":10900,"duration_ms":145242,"significance":"If the mapping to software partition/random testing is accepted, the paper usefully imports a large body of software reliability results into the AV safety discussion, including the non-trivial point that detecting more bugs need not mean a larger reduction in operational risk. The authors are transparent about their proof-of-concept status: the equations in Section IV.B are internally consistent under the stated assumptions, and the REF definition is a formal, falsifiable criterion rather than a fitted model. The survey of statistical inference methods for AV safety (Table I and Remarks 6-9) is a useful contribution in itself. The main weakness is that the formal comparison is restricted to a single failure region, so the motivating '5000-year vs 5-year bug' tradeoff is not actually represented by the model; the paper's central conclusion is therefore illustrated rather than demonstrated at the level of generality implied by the abstract.","major_comments":[{"comment":"The sentence following Eq. (13) states that substituting Eq. (13) 'reverses the inequality, showing that scenario-based testing results in a lower expected pfs than mile-based testing.' This is the opposite of the derivation: when P(D_k) >> 1/n, the expected residual pfs for scenario-based testing is larger than for mile-based testing, so the sentence should say that mile-based testing results in the lower expected pfs. The surrounding heading identifies the case correctly, but the stated conclusion in the text must be corrected.","section":"§IV.B, after Eq. (13)"},{"comment":"The introduction motivates the comparison with the '5000-year bugs vs 5-year bugs' tradeoff, but the model in Section IV.B cannot represent that tradeoff: with a single failure region and a perfect fix, any detected failure drives pfs to zero, so there is no way to model detecting a rare bug while leaving a common bug unfixed. The no-universal-superiority conclusion is therefore only illustrated for a restricted bug model, and the insight 'more bugs does not necessarily imply less reliable' is imported from reference [19] rather than derived from Eqs. (5)-(13). The paper should either add a two-failure-region example, e.g., using the richer model in [18], or explicitly state in the abstract and conclusions that the formal comparison is limited to the single-region case and that the rare/common-bug argument is an external motivation.","section":"§I(iii) and §IV.B"},{"comment":"The text says the two approximations in Eq. (12) require d_k and q to be small. The first approximation is applied to the upper bound q(1 - nP(F))^{t/n}, so the relevant small quantity is nP(F), not d_k; in the 'rare subdomain' regime P(D_k) << 1/n, d_k = P(F)/P(D_k) may actually be large while the bound remains accurate. The stated condition should be corrected, since as written it appears to impose an unnecessary restriction on the derived superiority condition.","section":"§IV.B, Eq. (12)"}],"minor_comments":[{"comment":"Equation (2) renders as 'θ k(1−θ) t−k'; the superscript notation should be θ^k(1−θ)^{t−k}.","section":"Eq. (2)"},{"comment":"The phrase 'classical confidence statement' is imprecise: the probability in Eq. (14) is over repeated realizations of both estimators, not a standard confidence interval for a fixed parameter. Suggest rephrasing as a probabilistic guarantee about the estimator difference.","section":"§IV.C, Eq. (14)"},{"comment":"The CLT approximation for the difference of two binomial proportions is used without stating the usual large-sample conditions near the step; the end of the section acknowledges the issue for rare failures, but a short note at Step 3 would improve readability.","section":"Example 3, Step 3"},{"comment":"The decomposition equation after 'can be decomposed into:' contains a stray vertical bar in the rendered text; it should be typeset as a displayed equation.","section":"§IV.C"},{"comment":"The abbreviations CBI and SRGM appear in Table I before they are defined in the text; define them in the table caption or point forward to Remark 7.","section":"Table I"},{"comment":"Use 'AVs' instead of 'A Vs' for a more standard typographic style.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This is a proof-of-concept/position paper, and the formal apparatus is simple but honestly labeled. The main risk is overclaiming in the abstract, which can be fixed by explicitly qualifying the scope of the formal comparison. The reversed sentence after Eq. (13) is a clear error that must be corrected. I do not see a novelty-disclosure concern; the mapping of AV testing to software testing is the paper's main contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nPunchline: this is a useful position piece that gives AV scenario-based testing a software-reliability vocabulary, and it does the field a service by spelling out the mapping between scenario-based/mile-based testing and partition/random testing. The pfs metric and (epsilon, alpha)-REF definition are clear and reusable. The numerical comparison in Section IV.B is internally consistent for what it is: a proof-of-concept with a very simple bug model.\n\nWhat's actually new: the systematic mapping and the REF definition, plus the observation that the AV testing community can borrow decades of software testing results. The survey in Table I is handy.\n\nSoft spots: the single-failure-region/perfect-fix model is the main one. Equations (5) and (7) only work when there is exactly one failure region and fixing it removes all bugs. That means the model cannot represent the paper's own motivating example of finding a '5,000-year bug' while missing a '5-year bug'—that requires at least two failure regions with different operational probabilities. So the model doesn't itself demonstrate the 'more bugs doesn't imply more reliable' insight; that rests on citation [19]. The paper acknowledges the simplification and points to [18] for the richer model, so it's transparent, but the central comparison is narrower than the rhetoric suggests.\n\nAlso, the REF definition is a confidence statement on the difference between estimates, which is fine, but the workflow in Example 3 uses MLE with normal approximation, which breaks down for rare failures. The authors admit this. Minor.\n\nCitation pattern looks fair, with appropriate self-citation to earlier work. The paper does not overclaim beyond illustrative numbers.\n\nOverall: the conceptual contribution and survey are worth having. The model needs extension to multiple failure regions, or the conclusions should be stated more modestly. For a reader new to AV testing, this is a solid orientation; for a specialist, it's a call to action.\n\nRecommendation: deserves a serious referee. I'd send it to peer review expecting major revision—the framing is valuable and the limitations are honestly stated, but the proof-of-concept needs either a multi-region extension or a rewritten contribution claim.","headline":"Useful mapping of AV scenario-based testing to software partition/random testing, with a clear but limited proof-of-concept that needs a multi-failure-region extension.","tokens_in":14073,"tokens_out":1560,"would_cite":true,"duration_ms":18184,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neither mile-based nor scenario-based AV testing is universally safer, under a new statistical model that also defines a formal simulator-fidelity check.","keywords":["scenario-based testing","autonomous vehicles","probability of failure per scenario","statistical safety assurance","residual risk","operational profile","simulation fidelity","software reliability"],"falsifier":"Take an AV in simulation with one known failure region, run random scenarios until the region is detected, apply the fix, and then re-test across all logical scenarios: if any failures appear in subdomains that were previously clean, the single-failure-region perfect-fix assumption is violated and the equations $E[\\theta]=q(1-q)^t$ and $q\\prod_i(1-d_i)^{t_i}$ no longer describe the post-fix system.","tokens_in":13045,"feed_emoji":"🚗","tokens_out":7144,"duration_ms":74612,"temperature":0.7,"pith_summary":"This paper argues that rigorous safety claims from scenario-based testing of autonomous vehicles need a statistical foundation, and it offers a proof-of-concept model for building one. The core move is to treat scenario-based testing as partition-based software testing and mile-based testing as random testing, then compare both through the probability of failure per randomly selected scenario, $\\theta$ (pfs). Under a single-failure-region model with perfect fixes, the expected post-fix pfs is $q(1-q)^t$ for mile-based testing and $q\\prod_{i=1}^n (1-d_i)^{t_i}$ for scenario-based testing, and the comparison shows that neither method dominates. The paper also defines $(\\epsilon,\\alpha)$-Risk Estimation Fidelity as a confidence-style condition for using simulators to estimate pfs. A sympathetic reader would take the point to be that unresolved questions about stopping rules, residual risk, debug effectiveness, and simulation fidelity can be attacked with existing software reliability mathematics rather than treated as AV-specific mysteries.","feed_headline":"Neither testing strategy always wins for AVs","feed_subtitle":"A pfs model maps scenario testing to partition testing and adds a simulator-fidelity check","key_machinery":"The load-bearing object is the probability of failure per random scenario, $\\theta$, paired with a transplanted software-reliability failure-region repair model: the AV's failures are attributed to a failure region in scenario space, and a detected failure region is removed by a perfect fix that introduces no new failures. That machinery turns testing style into an expected-risk calculation: after $t$ random mile-based scenarios, $E[\\theta]=q(1-q)^t$, and after partition-style scenario testing with $t_i$ tests per logical scenario, $E[\\theta]=q\\prod_i(1-d_i)^{t_i}$. A second mechanism is $(\\epsilon,\\alpha)$-Risk Estimation Fidelity, $\\Pr(|\\hat\\theta_s-\\hat\\theta_r|\\le\\epsilon)\\ge 1-\\alpha$, which gives a formal, testable criterion for when synthetic-data estimates can stand in for real-world ones.","core_discovery":"The paper's central claim is that scenario-based and mile-based testing of AVs can be analysed in one statistical model, with no universal winner. The model defines pfs as $\\theta = \\int_{x\\in D} I\\{x \\text{ causes a failure}\\}(x)\\, Op(x)\\,dx$, the limiting frequency of failures across independently selected concrete scenarios from the operational distribution. Mile-based testing is random testing: over $t$ scenarios it finds and fixes the single failure region with probability $1-(1-q)^t$, leaving expected pfs $E[\\theta]=q(1-q)^t$. Scenario-based testing partitions $D$ into $n$ logical scenarios and spends $t_i$ tests in each: if the failure is found and perfectly fixed the expected pfs is $E[\\theta]=q\\prod_{i=1}^n(1-d_i)^{t_i}$. The comparison yields concrete conditions, e.g. when the failure region is spread uniformly across subdomains scenario-based testing is better only if the detection rate per test $\\bar d$ exceeds $q$; when the failure region sits in one operationally rare subdomain, scenario-based testing wins if $Op(D_k)\\ll 1/n$ and mile-based testing wins if $Op(D_k)\\gg 1/n$. The paper states these results as proof-of-concept, not as a complete theory.","pith_inferences":["We infer that the partition-versus-random testing literature contains many documented cases where directed testing underperforms random testing; if the paper's mapping holds, those cases become concrete AV counterexamples to the implicit assumption that scenario-based testing is always the better investment.","We infer that the model can be turned into an adaptive test-allocation rule: instead of splitting tests equally across $n$ logical scenarios, allocate to minimize $q\\prod_i(1-d_i)^{t_i}$ under an unknown $Op$, which suggests a bandit-style scenario search as a testable extension.","We infer that the REF definition could be sharpened to asymmetric risk, where false negatives (missed real hazards) are penalized more than false positives; the paper lists asymmetric fidelity as future work, and a one-sided version of Eq. (14) would be a direct formalization.","We infer that the single-failure-region restriction is the first thing to drop in a follow-up: with multiple failure regions and partial fixes, the expected-pfs comparison should be re-derived, and the direction of the no-universal-winner conclusion may change in ways the paper does not yet cover."],"forward_implications":["A regulator or developer can choose between mile-based and scenario-based testing only after knowing how the operational distribution concentrates on failure-prone scenario subdomains, not from a blanket preference.","The stopping rule for scenario-based testing can be expressed as a target on residual pfs: under the model, continuing until $\\prod_i(1-d_i)^{t_i}$ is small enough is the formal condition.","In the region where mile-based testing is superior, a scenario-based campaign can still be rescued by allocating tests proportionally to the operational mass of each logical scenario.","The $(\\epsilon,\\alpha)$-REF workflow gives a concrete procedure: certify a simulator on a limited real-world sample, scale up simulation testing only after certification, and monitor the certificate after deployment.","Existing software reliability estimators, such as conservative Bayesian inference and software reliability growth models, transfer directly to estimating pfs from scenario evidence."],"supporting_citations":[{"why":"Supplies the failure-region repair model and the delivered-reliability comparison that the proof-of-concept adapts to AV testing.","marker":"[18]"},{"why":"Establishes that partition testing does not by itself inspire confidence, backing the no-universal-superiority claim.","marker":"[17]"},{"why":"Provides the 'more bugs does not necessarily imply less reliable' observation that motivates comparing debug effectiveness.","marker":"[19]"},{"why":"Documents the massive mileage requirements for mile-based testing that scenario-based testing seeks to avoid.","marker":"[6]"},{"why":"Supplies Bernoulli-process and conservative Bayesian inference machinery for AV road testing that the pfs estimator reuses.","marker":"[7]"},{"why":"Frames the ultra-high-dependability validation problem, the background for residual risk after testing.","marker":"[8]"},{"why":"Gives the random-versus-partition testing relationship imported in Remarks 3 and 4.","marker":"[29]"},{"why":"Exemplifies importance-sampling estimation inside logical scenarios, the statistical tool the partition-based pfs estimator relies on.","marker":"[30]"}],"fun_headline_variants":["No universal winner in AV testing strategies","Scenario vs mile testing: neither wins for AVs","Statistical foundation missing in AV scenario tests","AV safety testing needs statistical rigor","Neither strategy dominates AV safety testing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison relies on the assumption that the AV has one failure region and that the first test to detect it leads to a perfect fix that removes the region and creates no new ones; in real development, fixes are often partial and can introduce fresh failure regions.","fun_headline_variants_meta":{"raw":{"variants":["No universal winner in AV testing strategies","Scenario vs mile testing: neither wins for AVs","Statistical foundation missing in AV scenario tests","AV safety testing needs statistical rigor","Neither strategy dominates AV safety testing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1581,"prompt_tokens":966,"completion_tokens":615,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":553}},"tokens_in":582,"tokens_out":615,"duration_ms":7456,"temperature":1.0,"reasoning_tokens":553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:55:32.277778+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an AV in simulation with one known failure region, run random scenarios until the region is detected, apply the fix, and then re-test across all logical scenarios: if any failures appear in subdomains that were previously clean, the single-failure-region perfect-fix assumption is violated and the equations $E[\\theta]=q(1-q)^t$ and $q\\prod_i(1-d_i)^{t_i}$ no longer describe the post-fix system.","supporting_citations":[{"cited_title":"Evaluating testing methods by delivered reliability [software],","cited_arxiv_id":null,"evidence_quote":"Supplies the failure-region repair model and the delivered-reliability comparison that the proof-of-concept adapts to AV testing."},{"cited_title":"Partition testing does not inspire confidence,","cited_arxiv_id":null,"evidence_quote":"Establishes that partition testing does not by itself inspire confidence, backing the no-universal-superiority claim."},{"cited_title":"Limits to dependability assurance—a controversy revis- ited,","cited_arxiv_id":null,"evidence_quote":"Provides the 'more bugs does not necessarily imply less reliable' observation that motivates comparing debug effectiveness."},{"cited_title":"Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?","cited_arxiv_id":null,"evidence_quote":"Documents the massive mileage requirements for mile-based testing that scenario-based testing seeks to avoid."},{"cited_title":"Assessing the safety and reliability of autonomous vehicles from road testing,","cited_arxiv_id":null,"evidence_quote":"Supplies Bernoulli-process and conservative Bayesian inference machinery for AV road testing that the pfs estimator reuses."},{"cited_title":"Validation of ultra-high dependability for software-based systems,","cited_arxiv_id":null,"evidence_quote":"Frames the ultra-high-dependability validation problem, the background for residual risk after testing."},{"cited_title":"On the relationship between partition and random testing,","cited_arxiv_id":null,"evidence_quote":"Gives the random-versus-partition testing relationship imported in Remarks 3 and 4."},{"cited_title":"Accelerated evaluation of automated vehicles safety in lane- change scenarios based on importance sampling techniques,","cited_arxiv_id":null,"evidence_quote":"Exemplifies importance-sampling estimation inside logical scenarios, the statistical tool the partition-based pfs estimator relies on."}],"review_version":1}