Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Neither mile-based nor scenario-based AV testing is universally safer, under a new statistical model that also defines a formal simulator-fidelity check.

desk verdict Useful mapping of AV scenario-based testing to software partition/random testing, with a clear but limited proof-of-concept that needs a multi-failure-region extension. read the letter →

arxiv 2505.02274 v2 pith:Y62OGPK6 submitted 2025-05-04 cs.SE cs.AIcs.RO

Xingyu Zhao , Robab Aghazadeh-Chakherlou , Chih-Hong Cheng , Peter Popov , Lorenzo Strigini This is my paper · ORCID
classification cs.SEcs.AIcs.RO
keywords scenario-basedtestingautonomousvehiclesprobabilityoffailureperscenariostatisticalsafetyassuranceresidualriskoperationalprofilesimulationfidelitysoftwarereliability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that rigorous safety claims from scenario-based testing of autonomous vehicles need a statistical foundation, and it offers a proof-of-concept model for building one. The core move is to treat scenario-based testing as partition-based software testing and mile-based testing as random testing, then compare both through the probability of failure per randomly selected scenario, $\theta$ (pfs). Under a single-failure-region model with perfect fixes, the expected post-fix pfs is $q(1-q)^t$ for mile-based testing and $q\prod_{i=1}^n (1-d_i)^{t_i}$ for scenario-based testing, and the comparison shows that neither method dominates. The paper also defines $(\epsilon,\alpha)$-Risk Estimation Fidelity as a confidence-style condition for using simulators to estimate pfs. A sympathetic reader would take the point to be that unresolved questions about stopping rules, residual risk, debug effectiveness, and simulation fidelity can be attacked with existing software reliability mathematics rather than treated as AV-specific mysteries.

What carries the argument

The load-bearing object is the probability of failure per random scenario, $\theta$, paired with a transplanted software-reliability failure-region repair model: the AV's failures are attributed to a failure region in scenario space, and a detected failure region is removed by a perfect fix that introduces no new failures. That machinery turns testing style into an expected-risk calculation: after $t$ random mile-based scenarios, $E[\theta]=q(1-q)^t$, and after partition-style scenario testing with $t_i$ tests per logical scenario, $E[\theta]=q\prod_i(1-d_i)^{t_i}$. A second mechanism is $(\epsilon,\alpha)$-Risk Estimation Fidelity, $\Pr(|\hat\theta_s-\hat\theta_r|\le\epsilon)\ge 1-\alpha$, which gives a formal, testable criterion for when synthetic-data estimates can stand in for real-world ones.

What would settle it

Take an AV in simulation with one known failure region, run random scenarios until the region is detected, apply the fix, and then re-test across all logical scenarios: if any failures appear in subdomains that were previously clean, the single-failure-region perfect-fix assumption is violated and the equations $E[\theta]=q(1-q)^t$ and $q\prod_i(1-d_i)^{t_i}$ no longer describe the post-fix system.

Watch

Extended reading notes

Core claim

The paper's central claim is that scenario-based and mile-based testing of AVs can be analysed in one statistical model, with no universal winner. The model defines pfs as $\theta = \int_{x\in D} I\{x \text{ causes a failure}\}(x)\, Op(x)\,dx$, the limiting frequency of failures across independently selected concrete scenarios from the operational distribution. Mile-based testing is random testing: over $t$ scenarios it finds and fixes the single failure region with probability $1-(1-q)^t$, leaving expected pfs $E[\theta]=q(1-q)^t$. Scenario-based testing partitions $D$ into $n$ logical scenarios and spends $t_i$ tests in each: if the failure is found and perfectly fixed the expected pfs is $E[\theta]=q\prod_{i=1}^n(1-d_i)^{t_i}$. The comparison yields concrete conditions, e.g. when the failure region is spread uniformly across subdomains scenario-based testing is better only if the detection rate per test $\bar d$ exceeds $q$; when the failure region sits in one operationally rare subdomain, scenario-based testing wins if $Op(D_k)\ll 1/n$ and mile-based testing wins if $Op(D_k)\gg 1/n$. The paper states these results as proof-of-concept, not as a complete theory.

Load-bearing premise

The comparison relies on the assumption that the AV has one failure region and that the first test to detect it leads to a perfect fix that removes the region and creates no new ones; in real development, fixes are often partial and can introduce fresh failure regions.

Editorial extensions

If this is right

  • A regulator or developer can choose between mile-based and scenario-based testing only after knowing how the operational distribution concentrates on failure-prone scenario subdomains, not from a blanket preference.
  • The stopping rule for scenario-based testing can be expressed as a target on residual pfs: under the model, continuing until $\prod_i(1-d_i)^{t_i}$ is small enough is the formal condition.
  • In the region where mile-based testing is superior, a scenario-based campaign can still be rescued by allocating tests proportionally to the operational mass of each logical scenario.
  • The $(\epsilon,\alpha)$-REF workflow gives a concrete procedure: certify a simulator on a limited real-world sample, scale up simulation testing only after certification, and monitor the certificate after deployment.
  • Existing software reliability estimators, such as conservative Bayesian inference and software reliability growth models, transfer directly to estimating pfs from scenario evidence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the partition-versus-random testing literature contains many documented cases where directed testing underperforms random testing; if the paper's mapping holds, those cases become concrete AV counterexamples to the implicit assumption that scenario-based testing is always the better investment.
  • We infer that the model can be turned into an adaptive test-allocation rule: instead of splitting tests equally across $n$ logical scenarios, allocate to minimize $q\prod_i(1-d_i)^{t_i}$ under an unknown $Op$, which suggests a bandit-style scenario search as a testable extension.
  • We infer that the REF definition could be sharpened to asymmetric risk, where false negatives (missed real hazards) are penalized more than false positives; the paper lists asymmetric fidelity as future work, and a one-sided version of Eq. (14) would be a direct formalization.
  • We infer that the single-failure-region restriction is the first thing to drop in a follow-up: with multiple failure regions and partial fixes, the expected-pfs comparison should be re-derived, and the direction of the no-universal-winner conclusion may change in ways the paper does not yet cover.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that scenario-based testing of autonomous vehicles currently lacks a rigorous statistical foundation, and it maps scenario-based and mile-based testing onto partition-based and random software testing, respectively. It defines the probability of failure per randomly selected scenario (pfs), presents proof-of-concept models for comparing the debugging effectiveness of the two testing styles under a single-failure-region, perfect-fix assumption, and concludes that neither approach universally outperforms the other. The paper also introduces an (epsilon, alpha)-Risk Estimation Fidelity criterion for synthetic data, with a stepwise workflow and a numerical illustration, and closes with open problems concerning dynamic operational profiles, asymmetric fidelity, and ultra-high reliability.

Significance. If the mapping to software partition/random testing is accepted, the paper usefully imports a large body of software reliability results into the AV safety discussion, including the non-trivial point that detecting more bugs need not mean a larger reduction in operational risk. The authors are transparent about their proof-of-concept status: the equations in Section IV.B are internally consistent under the stated assumptions, and the REF definition is a formal, falsifiable criterion rather than a fitted model. The survey of statistical inference methods for AV safety (Table I and Remarks 6-9) is a useful contribution in itself. The main weakness is that the formal comparison is restricted to a single failure region, so the motivating '5000-year vs 5-year bug' tradeoff is not actually represented by the model; the paper's central conclusion is therefore illustrated rather than demonstrated at the level of generality implied by the abstract.

major comments (3)
  1. [§IV.B, after Eq. (13)] The sentence following Eq. (13) states that substituting Eq. (13) 'reverses the inequality, showing that scenario-based testing results in a lower expected pfs than mile-based testing.' This is the opposite of the derivation: when P(D_k) >> 1/n, the expected residual pfs for scenario-based testing is larger than for mile-based testing, so the sentence should say that mile-based testing results in the lower expected pfs. The surrounding heading identifies the case correctly, but the stated conclusion in the text must be corrected.
  2. [§I(iii) and §IV.B] The introduction motivates the comparison with the '5000-year bugs vs 5-year bugs' tradeoff, but the model in Section IV.B cannot represent that tradeoff: with a single failure region and a perfect fix, any detected failure drives pfs to zero, so there is no way to model detecting a rare bug while leaving a common bug unfixed. The no-universal-superiority conclusion is therefore only illustrated for a restricted bug model, and the insight 'more bugs does not necessarily imply less reliable' is imported from reference [19] rather than derived from Eqs. (5)-(13). The paper should either add a two-failure-region example, e.g., using the richer model in [18], or explicitly state in the abstract and conclusions that the formal comparison is limited to the single-region case and that the rare/common-bug argument is an external motivation.
  3. [§IV.B, Eq. (12)] The text says the two approximations in Eq. (12) require d_k and q to be small. The first approximation is applied to the upper bound q(1 - nP(F))^{t/n}, so the relevant small quantity is nP(F), not d_k; in the 'rare subdomain' regime P(D_k) << 1/n, d_k = P(F)/P(D_k) may actually be large while the bound remains accurate. The stated condition should be corrected, since as written it appears to impose an unnecessary restriction on the derived superiority condition.
minor comments (6)
  1. [Eq. (2)] Equation (2) renders as 'θ k(1−θ) t−k'; the superscript notation should be θ^k(1−θ)^{t−k}.
  2. [§IV.C, Eq. (14)] The phrase 'classical confidence statement' is imprecise: the probability in Eq. (14) is over repeated realizations of both estimators, not a standard confidence interval for a fixed parameter. Suggest rephrasing as a probabilistic guarantee about the estimator difference.
  3. [Example 3, Step 3] The CLT approximation for the difference of two binomial proportions is used without stating the usual large-sample conditions near the step; the end of the section acknowledges the issue for rare failures, but a short note at Step 3 would improve readability.
  4. [§IV.C] The decomposition equation after 'can be decomposed into:' contains a stray vertical bar in the rendered text; it should be typeset as a displayed equation.
  5. [Table I] The abbreviations CBI and SRGM appear in Table I before they are defined in the text; define them in the table caption or point forward to Remark 7.
  6. [Throughout] Use 'AVs' instead of 'A Vs' for a more standard typographic style.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the comparative analysis is derived from stated assumptions, not from fitted parameters or self-citations.

full rationale

The paper's pfs definition (Eq. 1) is a standard integral. Equations (5) and (7) follow from the stated single-failure-region/perfect-fix model and the test-count assumptions. Examples 1 and 2 are algebraic comparisons of these expressions, and the no-universal-superiority conclusion is a direct consequence of the fact that the OP mass of a subdomain can be small or large. The REF measure is a definition, and Example 3 is illustrative. The only self-citation is the attribution of the failure-region model to [18], co-authored by L. Strigini; however, [18] is an externally peer-reviewed, independent result, and the paper presents the equations explicitly rather than relying on the citation for their validity. The acknowledged limitations (single failure region, i.i.d. scenarios, perfect fixes) are modeling assumptions, not circularities.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The models introduce no fitted free parameters in the strict sense: all quantities are symbolic or chosen for illustration. The domain assumptions of i.i.d. scenarios and perfect fixes are the main burden, and they are acknowledged as simplifications. The paper is honest that these are proof-of-concept models rather than validated engineering outputs.

free parameters (3)
  • single failure region q = symbolic; not fitted
    The failure probability q is a symbolic quantity in Eqs. (5)-(7), not estimated from data. It is an assumption about the AV under test, not a fitted parameter.
  • detection rate d_i = symbolic; not fitted
    The detection rate for each logical scenario is an assumed input to Eq. (6). In the examples it is set to a constant d bar or to the conditional operational probability, not fitted to data.
  • epsilon and alpha in REF = epsilon = 0.02, alpha = 0.05 in Example 3
    These are chosen by the assessor in the REF definition and example. They are not inferred from data.
assumptions (3)
  • domain assumption Concrete scenarios are i.i.d. samples from the operational distribution Op when using mile-based testing.
    Invoked in Eq. (2) for the Bernoulli likelihood and in Eq. (5) for the expected pfs after t random tests. The paper notes in Remark 7 that such independence and constant probability assumptions may not hold in reality.
  • domain assumption A detected failure region is perfectly fixed, with no residual failures and no new failure regions introduced.
    Used to set theta = 0 after a failure is detected and fixed in Eqs. (5)-(7). The paper cites [18] for validity discussions but does not model partial fixes.
  • domain assumption For Example 3, the difference of sample means is approximately Gaussian.
    The REF example computes Pr(|Delta| <= epsilon) using the normal distribution. The paper explicitly warns in a later paragraph that this fails for rare failures.
invented entities (2)
  • pfs (probability of failure per scenario)
    purpose: A metric for AV safety risk under scenario-based testing, serving as the analogue of pfd in software reliability.
    pfs is defined in Definition 8 and is a re-expression of the standard pfd metric with the demand unit changed to 'scenario'. It provides no new falsifiable handle outside the paper's own analysis.
  • (epsilon, alpha)-REF (Risk Estimation Fidelity)
    purpose: A criterion for certifying that synthetic-data pfs estimates are close to real-data estimates.
    REF is a definitional criterion. It can be checked in future experiments, so it is operationalizable, but the paper itself does not provide independent evidence that any real simulator satisfies it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles." pith.science (2026). https://pith.science/paper/Y62OGPK6

@misc{pith2026250502274,
  author       = {Pith},
  title        = {Pith review of: On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y62OGPK6}},
  note         = {Machine review of arXiv:2505.02274}
}
read the original abstract

Scenario-based testing has emerged as a common method for autonomous vehicles (AVs) safety assessment, offering a more efficient alternative to mile-based testing by focusing on high-risk scenarios. However, fundamental questions persist regarding its stopping rules, residual risk estimation, debug effectiveness, and the impact of simulation fidelity on safety claims. This paper argues that a rigorous statistical foundation is essential to address these challenges and enable rigorous safety assurance. By drawing parallels between AV testing and established software testing methods, we identify shared research gaps and reusable solutions. We propose proof-of-concept models to quantify the probability of failure per scenario (\textit{pfs}) and evaluate testing effectiveness under varying conditions. Our analysis reveals that neither scenario-based nor mile-based testing universally outperforms the other. Furthermore, we give an example of formal reasoning about alignment of synthetic and real-world testing outcomes, a first step towards supporting statistically defensible simulation-based safety claims.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Real-World Perturbation Testing of Autonomous Driving Systems

    cs.SE 2026-07 conditional novelty 7.0 of 10

    Model-level and offline robustness metrics for 72 camera/LiDAR perturbations do not reliably predict closed-loop failures on a full-scale autonomous vehicle.

  2. Bayesian Optimization applied for accelerated Virtual Validation of the Autonomous Driving Function

    cs.RO 2025-07 conditional novelty 4.0 of 10

    A Bayesian optimization framework finds critical scenarios for an MPC motion planner using one to two orders of magnitude fewer simulations than full-factorial testing.

Reference graph

Works this paper leans on

48 extracted references · 43 canonical work pages · cited by 2 Pith papers

  1. [19]

    Limits to dependability assurance—a controversy revis- ited,

    B. Littlewood, “Limits to dependability assurance—a controversy revis- ited,” inKeynote Paper presented at ICSE’07, 2007

    work page 2007

  2. [18]

    Evaluating testing methods by delivered reliability [software],

    P. G. Frankl, R. G. Hamlet, B. Littlewood, and L. Strigini, “Evaluating testing methods by delivered reliability [software],”IEEE Tran. on Software Engineering, vol. 24, no. 8, pp. 586–601, 1998

    work page 1998

  3. [1]

    Autonomous vehicle technology: A guide for policymakers,

    J. M. Anderson, K. Nidhi, K. D. Stanley, P. Sorensen, C. Samaras, and O. A. Oluwatola, “Autonomous vehicle technology: A guide for policymakers,” Rand Corporation, Tech. Rep. RR-443-2-RC, 2016

    2016

  4. [2]

    Preparing a nation for autonomous vehicles: Opportunities, barriers and policy recommendations,

    D. J. Fagnant and K. Kockelman, “Preparing a nation for autonomous vehicles: Opportunities, barriers and policy recommendations,”Transp. Research Part A: Policy and Practice, vol. 77, pp. 167–181, 2015

    work page 2015

  5. [3]

    Autonomous vehicle safety: An interdis- ciplinary challenge,

    P. Koopman and M. Wagner, “Autonomous vehicle safety: An interdis- ciplinary challenge,”IEEE Intelligent Transportation Systems Magazine, vol. 9, no. 1, pp. 90–96, 2017

    work page 2017

  6. [4]

    The social dilemma of autonomous vehicles,

    J.-F. Bonnefon, A. Shariff, and I. Rahwan, “The social dilemma of autonomous vehicles,”Science, vol. 352, no. 6293, pp. 1573–1576, 2016

    2016

  7. [5]

    A Survey on an Emerging Safety Challenge for Autonomous Vehicles: Safety of the Intended Functionality,

    H. Wang, W. Shao, C. Sun, K. Yang, D. Cao, and J. Li, “A Survey on an Emerging Safety Challenge for Autonomous Vehicles: Safety of the Intended Functionality,”Engineering, vol. 33, pp. 17–34, 2024

    work page 2024

  8. [6]

    Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?

    N. Kalra and S. Paddock, “Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?”Transp. Research Part A: Policy and Practice, vol. 94, pp. 182–193, 2016

    work page 2016

Show all 48 references
  1. [7]

    Assessing the safety and reliability of autonomous vehicles from road testing,

    X. Zhao, V . Robu, D. Flynn, K. Salako, and L. Strigini, “Assessing the safety and reliability of autonomous vehicles from road testing,” inInt. Symp. on Software Reliability Engineering (ISSRE), 2019, pp. 13–23

    2019

  2. [8]

    Validation of ultra-high dependability for software-based systems,

    B. Littlewood and L. Strigini, “Validation of ultra-high dependability for software-based systems,”Comm. of the ACM, vol. 36, pp. 69–80, 1993

    1993

  3. [9]

    Defining and substantiating the terms scene, situation, and scenario for automated driving,

    S. Ulbrich, T. Menzel, A. Reschka, F. Schuldt, and M. Maurer, “Defining and substantiating the terms scene, situation, and scenario for automated driving,” inITSC’15. IEEE, 2015, pp. 982–988

    2015

  4. [10]

    Road vehicles - test scenarios for automated driving systems - scenario based safety evaluation framework,

    ISO 34502, “Road vehicles - test scenarios for automated driving systems - scenario based safety evaluation framework,” 2022

    2022

  5. [11]

    Formalised natural language description of scenarios for automated driving systems. Specification,

    BSI Flex 1889, “Formalised natural language description of scenarios for automated driving systems. Specification,” 2023

    2023

  6. [12]

    Survey on scenario-based safety assessment of automated vehicles,

    S. Riedmaier, T. Ponn, D. Ludwig, B. Schick, and F. Diermeyer, “Survey on scenario-based safety assessment of automated vehicles,” IEEE Access, vol. 8, pp. 87 456–87 477, 2020

    2020

  7. [13]

    Defining required and feasible test coverage for scenario-based validation of highly automated vehicles,

    C. Amersbach and H. Winner, “Defining required and feasible test coverage for scenario-based validation of highly automated vehicles,” inITSC’19. IEEE, 2019, pp. 425–430

    2019

  8. [14]

    Automatic virtual test technology for intelligent driving systems considering both coverage and efficiency,

    F. Gao, J. Duan, Z. Han, and Y . He, “Automatic virtual test technology for intelligent driving systems considering both coverage and efficiency,” IEEE Tran. on Vehicular Technology, vol. 69, pp. 14 365–14 376, 2020

    2020

  9. [15]

    Automated scenario generation for regression testing of autonomous vehicles,

    E. Rocklage, H. Kraft, A. Karatas, and J. Seewig, “Automated scenario generation for regression testing of autonomous vehicles,” inInt. Conf. on Intelligent Transportation Systems (ITSC), 2017, pp. 476–483

    2017

  10. [16]

    Coverage based testing for v&v and safety assurance of self-driving autonomous vehicles: A systematic literature review,

    Z. Tahir and R. Alexander, “Coverage based testing for v&v and safety assurance of self-driving autonomous vehicles: A systematic literature review,” inIEEE AITest’20, 2020, pp. 23–30

    2020

  11. [17]

    Partition testing does not inspire confidence,

    D. Hamlet and R. Taylor, “Partition testing does not inspire confidence,” IEEE Tran. on Softw. Engin., vol. 16, no. 12, pp. 1402–1411, 1990

    1990

  12. [20]

    Adams,Minimizing cost impact of software defects

    E. Adams,Minimizing cost impact of software defects. IBM Thomas J. Watson Research Division, 1980

    1980

  13. [21]

    Mind the gap! a study on the transferability of virtual versus physical-world testing of autonomous driving systems,

    A. Stocco, B. Pulfer, and P. Tonella, “Mind the gap! a study on the transferability of virtual versus physical-world testing of autonomous driving systems,”IEEE Tran. on Softw. Engin., vol. 49, no. 4, pp. 1928– 1940, 2022

    1928

  14. [22]

    How simulation helps autonomous driving: A survey of sim2real, digital twins, and parallel intelligence,

    X. Hu, S. Li, T. Huang, B. Tang, R. Huai, and L. Chen, “How simulation helps autonomous driving: A survey of sim2real, digital twins, and parallel intelligence,”IEEE Tran. on Intelligent Vehicles, vol. 9, no. 1, pp. 593–612, 2023

    2023

  15. [23]

    Instance-level safety-aware fidelity of synthetic data and its calibration,

    C.-H. Cheng, P. St ¨ockel, and X. Zhao, “Instance-level safety-aware fidelity of synthetic data and its calibration,” in27th Int. Conf. on Intelligent Transportation Systems. IEEE, 2024, pp. 2354–2361

    2024

  16. [24]

    Learning naturalistic driving environment with statistical realism,

    X. Yan, Z. Zou, S. Feng, H. Zhu, H. Sun, and H. X. Liu, “Learning naturalistic driving environment with statistical realism,”Nature com- munications, vol. 14, no. 1, p. 2037, 2023

    2023

  17. [25]

    Scenarios for development, test and validation of automated vehicles,

    T. Menzel, G. Bagschik, and M. Maurer, “Scenarios for development, test and validation of automated vehicles,” inIV’18, 2018

    2018

  18. [26]

    Relai testing: a technique to assess and improve software reliability,

    D. Cotroneo, R. Pietrantuono, and S. Russo, “Relai testing: a technique to assess and improve software reliability,”IEEE Tran. on Software Engineering, vol. 42, no. 5, pp. 452–475, 2015

    2015

  19. [27]

    Operational profiles in software-reliability engineering,

    J. D. Musa, “Operational profiles in software-reliability engineering,” IEEE Software, vol. 10, no. 02, pp. 14–32, 1993

    1993

  20. [28]

    On random and partition testing,

    S. Ntafos, “On random and partition testing,” inProc. of the ACM SIGSOFT Int. Symp. on Software Testing and Analysis, 1998, pp. 42–48

    1998

  21. [29]

    On the relationship between partition and random testing,

    T. Chen and Y . Yu, “On the relationship between partition and random testing,”IEEE Tran. on Softw. Engi., vol. 20, no. 12, pp. 977–980, 1994

    1994

  22. [30]

    Accelerated evaluation of automated vehicles safety in lane- change scenarios based on importance sampling techniques,

    D. Zhao, H. Lam, H. Peng, S. Bao, D. J. LeBlanc, K. Nobukawa, and C. S. Pan, “Accelerated evaluation of automated vehicles safety in lane- change scenarios based on importance sampling techniques,”IEEE Tran. on Intelligent Transportation Systems, vol. 18, no. 3, pp. 595–607, 2016

    2016

  23. [31]

    Accelerated evaluation of automated vehicles in car-following maneuvers,

    D. Zhao, X. Huang, H. Peng, H. Lam, and D. J. LeBlanc, “Accelerated evaluation of automated vehicles in car-following maneuvers,”IEEE Tran. on Intelligent Transp. Syst., vol. 19, no. 3, pp. 733–744, 2017

    2017

  24. [32]

    Intelligent testing environment generation for autonomous vehicles with implicit distributions of traffic behaviors,

    K. Ren, J. Yang, Q. Lu, Y . Zhang, J. Hu, and S. Feng, “Intelligent testing environment generation for autonomous vehicles with implicit distributions of traffic behaviors,”Transportation Research Part C: Emerging Technologies, vol. 174, p. 105106, 2025

    2025

  25. [33]

    Intelligent driving in- telligence test for autonomous vehicles with naturalistic and adversarial environment,

    S. Feng, X. Yan, H. Sun, Y . Feng, and H. X. Liu, “Intelligent driving in- telligence test for autonomous vehicles with naturalistic and adversarial environment,”Nature communications, vol. 12, no. 1, p. 748, 2021

    2021

  26. [34]

    Learning to collide: An adaptive safety-critical scenarios generating method,

    W. Ding, B. Chen, M. Xu, and D. Zhao, “Learning to collide: An adaptive safety-critical scenarios generating method,” inIEEE/RSJ Int. Conf. on Intelligent Robots and Systems, 2020, pp. 2243–2250

    2020

  27. [35]

    P. D. O’Connor and A. V . Kleyner,Practical reliability engineering. john wiley & sons, 2011

    2011

  28. [36]

    Assessing safety-critical systems from operational testing: A study on autonomous vehicles,

    X. Zhao, K. Salako, L. Strigini, V . Robu, and D. Flynn, “Assessing safety-critical systems from operational testing: A study on autonomous vehicles,”Information and Software Techn., vol. 128, p. 106393, 2020

    2020

  29. [37]

    Reliability analysis of artificial intelligence systems using recurrent events data from au- tonomous vehicles,

    J. Min, Y . Hong, C. B. King, and W. Q. Meeker, “Reliability analysis of artificial intelligence systems using recurrent events data from au- tonomous vehicles,”Journal of the Royal Statistical Society Series C: Applied Statistics, vol. 71, no. 4, pp. 987–1013, 04 2022

    2022

  30. [38]

    Planning reliability assurance tests for autonomous vehicles based on disengagement events data,

    S. Zheng, L. Lu, Y . Hong, and J. Liu, “Planning reliability assurance tests for autonomous vehicles based on disengagement events data,”IISE Transactions, vol. 0, no. ja, pp. 1–25, 2025

    2025

  31. [39]

    On reliability assessment when a software-based system is replaced by a thought-to- be-better one,

    B. Littlewood, K. Salako, L. Strigini, and X. Zhao, “On reliability assessment when a software-based system is replaced by a thought-to- be-better one,”Relia. Engin. & Sys. Safety, vol. 197, p. 106752, 2020

    2020

  32. [40]

    Conservative confidence bounds in safety, from generalised claims of improvement & statistical evidence,

    K. Salako, L. Strigini, and X. Zhao, “Conservative confidence bounds in safety, from generalised claims of improvement & statistical evidence,” inIEEE/IFIP Int. Conf. on Dependable Systems and Networks, 2021

    2021

  33. [41]

    Bootstrapping confidence in future safety from past safe operation,

    P. Bishop, A. Povyakalo, and L. Strigini, “Bootstrapping confidence in future safety from past safe operation,” in33rd Int. Symp. on Software Reliability Engineering (ISSRE). IEEE, 2022, pp. 97–108

    2022

  34. [42]

    Using pre change operational evidence for predicting post change reliability, given prior confidence in fault-freeness,

    R. Aghazadeh Chakherlou, L. Strigini, “Using pre change operational evidence for predicting post change reliability, given prior confidence in fault-freeness,”34th European Safety and Reliability Conf., 2024

    2024

  35. [43]

    Impact of prior beliefs on depend- ability prediction for a changed system using pre-change operational evidence,

    R. A. Chakherlou and L. Strigini, “Impact of prior beliefs on depend- ability prediction for a changed system using pre-change operational evidence,” inQRS’24. IEEE, 2024, pp. 572–583

    2024

  36. [44]

    Modeling the probability of failure on demand (pfd) of a 1-out-of-2 system in which one channel is “quasi-perfect

    X. Zhao, B. Littlewood, A. Povyakalo, L. Strigini, and D. Wright, “Modeling the probability of failure on demand (pfd) of a 1-out-of-2 system in which one channel is “quasi-perfect”,”Reliability Engineering & System Safety, vol. 158, pp. 230–245, 2017

    2017

  37. [45]

    The unnecessity of assuming statistically independent tests in bayesian software reliability assessments,

    K. Salako and X. Zhao, “The unnecessity of assuming statistically independent tests in bayesian software reliability assessments,”IEEE Tran. on Software Engineering, vol. 49, no. 4, pp. 2829–2838, 2023

    2023

  38. [46]

    Total variance approach to software reliability estimation,

    T. Adams, “Total variance approach to software reliability estimation,” IEEE Trans. on Software Engineering, vol. 22, no. 9, pp. 687–688, 1996

    1996

  39. [47]

    Deriving a frequentist conservative confidence bound for probability of failure per demand for systems with different operational and test profiles,

    P. Bishop and A. Povyakalo, “Deriving a frequentist conservative confidence bound for probability of failure per demand for systems with different operational and test profiles,”Reliability Engineering & System Safety, vol. 158, pp. 246–253, 2017

    2017

  40. [48]

    Reliability assessment of service-based software under operational profile uncertainty,

    R. Pietrantuono, P. Popov, and S. Russo, “Reliability assessment of service-based software under operational profile uncertainty,”Reliability Engineering & System Safety, vol. 204, p. 107193, 2020

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.