Pith. sign in

REVIEW 3 major objections 4 minor 65 references

The Impact of Operational-Data Fidelity when Assessing Safety-Critical Autonomous-Vehicle Software

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Coarse operational logs can justify more optimistic safety bounds for AV software than typed failure data, and after a single failure the gap can be infinite.

desk verdict A useful but partially unproved extension of CBI to typed failure data; the headline infinite-gap result rests on Theorem 3, whose zero-infimum proof is omitted. read the letter →

arxiv 2608.10025 v1 pith:SVPQYTVJ submitted 2026-08-09 cs.RO cs.SE

Kizito Salako , Rabiu Tsoho Muhammad This is my paper · ORCID
classification cs.ROcs.SE
keywords conservativeBayesianinferencesoftwarereliabilityassessmentoperationaldatafidelityautonomousvehiclesafetybinaryclassifierfailuremodesfalsepositiveandnegativeprobabilityofperclassificationworst-caseprior
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a safety assessor who only knows how many times an autonomous-vehicle classifier failed, but not whether each failure was a false positive or a false negative, can still make conservative reliability claims. It extends conservative Bayesian inference (CBI), a worst-case Bayesian method that works with a set of priors consistent with stated evidence, to a binary classifier with two failure modes. The central finding is that coarse, untyped operational data can produce posterior confidence in a bound on the probability of failure per classification that is larger than the confidence justified by typed data, and in one scenario the gap is infinite after a single failure. The paper proves four theorems giving worst-case posterior confidence bounds and derives inequality (9), which lets an assessor correct a coarse-data assessment by taking the minimum over all possible typed decompositions of the observed failures. If the paper is right, safety cases for AV software built from aggregate intervention logs can be dangerously optimistic, and logging systems should record failure types or apply the paper's correction.

What carries the argument

The engine of the analysis is a two-parameter categorical model of classifier outcomes: unknown probabilities $P$ for false positives and $Q$ for false negatives, with success probability $\Theta = 1-P-Q$, so the probability of failure per classification is $P+Q$. The assessor specifies only partial prior knowledge, namely PK1, PK2, or PK3, which defines a set $\mathcal{D}$ of admissible priors over the feasible triangle $\Omega$ in the unit square. The four theorems compute the infimum posterior confidence in $P+Q\le b$ by solving a fractional optimization problem whose extremal solution is always a two-point discrete prior: mass $1-a$ placed at the point in the "not good enough" region where the likelihood is largest, and mass $a$ at the point in the "good enough" region where the likelihood is smallest. The untyped likelihood $(p+q)^k(1-p-q)^{n-k}$ is a convex combination of the typed likelihoods $p^{k_1}q^{k_2}(1-p-q)^{n-k_1-k_2}$, which is the mechanism behind inequality (9): the coarse-data confidence is always at least the minimum typed-data confidence over splits of the failure count.

What would settle it

Take any concrete instance of the model, say $n=1000$, $k=1$, $b=10^{-2}$, and fixed PK parameters, and compute the infimum in Theorem 1 together with the minimum over $k_1=0,1$ of the Theorem 3 infima. If the untyped infimum is smaller than that minimum, inequality (9) is contradicted and the paper's central claim fails; a simulation over many parameter values can search for such a counterexample. Alternatively, in field data, find an AV safety monitor with logged failure types and compare the coarse-data confidence curve with the typed-data curve after the first failure: the paper predicts the coarse curve stays above the typed curve, and under PK1/PK2 the typed curve drops to zero.

Watch

Extended reading notes

Core claim

The paper's central claim is that ignoring failure-type labels is not automatically conservative: replacing typed counts $(k_1,k_2)$ of false positives and false negatives with a single total $k$ can raise the assessor's conservative posterior confidence in the statement that the probability of failure per classification $P+Q$ is at most $b$. Under prior knowledge PK1, which says the classifier is imperfect so $P+Q\ge l$, and PK2, which says the assessor has prior confidence $a$ that accuracy exceeds $1-b_1$, Theorem 3 says that any observed false positive or false negative takes the typed-data posterior confidence to zero, so after one failure no finite amount of failure-free operation can restore a 95% confidence target. Theorems 1 and 2, which use only untyped counts, give a positive confidence bound that can be recovered with finitely many further successes. Under the stronger prior PK3, which places positive lower limits on both $P$ and $Q$, Theorem 4 gives a finite recovery requirement that can still be orders of magnitude larger than the untyped requirements. Inequality (9) expresses the general structure of the gap: the untyped-data confidence infimum is bounded below by the minimum, over all ways of splitting $k$ into $k_1$ and $k_2$, of the typed-data confidence infimum. The paper presents this as a demonstration that attempts to use low-fidelity data conservatively can be naive, and as a first conservative estimate of the impact of data fidelity on AV software assessments.

Load-bearing premise

The whole comparison assumes every past outcome can be reliably labeled as a success, a false positive, or a false negative using ground truth and counterfactual reasoning; if those labels are noisy or impossible to obtain, the typed-data benchmark that the paper treats as the conservative gold standard may itself be unavailable.

Editorial extensions

If this is right

  • Safety evidence from AV operational logs that record only aggregate failure counts can overstate confidence in a classifier's probability of failure per classification, compared with the conservative bound the same evidence would justify if failure types were logged.
  • Under prior knowledge PK1 and PK2, a single logged failure of known type collapses the conservative posterior confidence to zero, so no finite amount of subsequent failure-free operation can restore a target confidence level.
  • With the more detailed prior PK3, recovery after one failure is finite but can require orders of magnitude more failure-free classifications than an untyped analysis suggests.
  • When failure types are unknown, inequality (9) provides a way to keep the assessment conservative: take the minimum typed-data confidence over all possible decompositions of the observed failures.
  • During failure-free operation all four theorems coincide, so data fidelity only starts to matter once failures are observed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same convex-mixture argument should extend to any reliability metric that is a convex or linear function of typed failure counts; metrics that weight false positives and false negatives differently could show an even larger coarse-versus-typed gap.
  • Beyond the paper, a practical design consequence is that AV safety logging should record failure-type labels at collection time, since aggregate counters cannot be re-derived once counterfactual ground truth is lost.
  • Beyond the paper, the theorems can be tested on shadow-mode logs where the monitor does not intervene and ground truth is directly observable; the prediction is that coarse summaries will look more optimistic than type-aware summaries as soon as failures appear.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses how coarse operational data, i.e. failure counts without type labels, affects conservative Bayesian reliability claims for safety-critical AV classifier and monitor software. It extends conservative Bayesian inference (CBI) to a trinomial model with FP and FN failure modes, and derives worst-case posterior confidence bounds on the probability of failure per classification under two partial-prior specifications (PK1+PK2 and PK2+PK3), in both untyped and typed versions (Theorems 1-4). The paper applies these bounds to AV safety assessment: after failure-free operation the four theorems give identical evidence requirements, while after a single failure the untyped analyses (Theorems 1-2) permit finite recovery, the typed analysis under PK3 (Theorem 4) requires substantially more evidence, and the typed analysis under PK1 (Theorem 3) is claimed to yield zero posterior confidence so that no finite amount of additional failure-free evidence can restore confidence. Inequality (9) is offered as a formal comparison between untyped and typed infima, and sensitivity analyses examine prior confidence and the i.i.d. assumption.

Significance. If the results are fully substantiated, the paper makes a useful contribution: it gives closed-form conservative posterior bounds for a two-failure-mode CBI problem, identifies a concrete scenario in which coarser data used with conservative intent is optimistic relative to typed data, and connects the analysis to AV safety standards and lifecycle processes. The supplied proof of Theorem 4 in Appendix A and the proof of the infimum interchange in Appendix C are valuable, and the application figures make the practical stakes clear. The variational formulation of the problem and the explicit two-point worst-case priors are also useful for practitioners. However, three of the four theorems are stated without proofs, and the most dramatic conclusion, the infinite recovery gap, depends on the unproved zero-infimum case of Theorem 3; the contribution will be fully credible only when those proofs are supplied or delegated with sufficient detail.

major comments (3)
  1. [Section V, Theorem 3] The sentence "Theorem 4 is proved in Appendix A; the other theorems are proved using analogous steps" is not adequate for the load-bearing zero-infimum claim. For k1>=1 or k2>=1, the zero result is not analogous to Theorem 4: under PK1 the 'good' region S2 touches the coordinate axes, so the typed likelihood p^{k1} q^{k2} (1-p-q)^{n-k1-k2} can vanish on S2, whereas PK3's positive lower bounds l1,l2 prevent this. The manuscript should supply the argument explicitly, including the boundary/limit-point convention and the positivity of the denominator. For example, when k1=1 and k2=0, a prior with mass a at (0,l) in S2 and mass 1-a at a point in S1 with p>0 gives numerator zero and denominator positive, yielding value 0 in the closure; the proof should state this and treat the k1=k2=0 case separately. This is the least secure step in the central argument because the infinite-gap conclusion in Section VI depends directly on it.
  2. [Section V, Theorems 1 and 2] Theorems 1 and 2, which drive the untyped curves in Figures 6 and 7, are also stated without proof. A reader cannot verify the reduction to two-point priors or the claimed extrema L2* and L1* for the untyped likelihood (p+q)^k(1-p-q)^{n-k} from the statement 'proved using analogous steps.' Since these theorems are used in the application section to quantify the evidence needed for 95% confidence after an untyped failure, the paper should include at least a proof sketch or an appendix treatment, or explicitly state where the complete proof appears.
  3. [Section VII-A, inequality (9)] Inequality (9) is formally correct as an interchange of infima over a finite set and D, as shown in Appendix C, but as a 'guarantee of conservatism' it is uninformative for k>=1. Because Theorem 3's infimum is zero whenever k1>=1 or k2>=1, the left-hand minimum over k1 is 0 for every k>=1, so the inequality only says the untyped confidence is at least 0. The text should not suggest that (9) quantitatively bounds the optimism gap; the quantitative content is in the comparison of Theorems 1-2 with Theorems 3-4, not in (9) itself.
minor comments (4)
  1. [Theorems 1-4, notation] The formula "Φ* = ... 11−b1≤b" appears to be a rendering error for the indicator 1_{1-b1 <= b}; it should be typeset as a subscripted indicator throughout Theorems 1-4 to avoid confusion with the number 11.
  2. [Section III-B and Section VII-C] The paper is transparent that typed data requires reliable ground truth and counterfactual reasoning, but given that the headline conclusion is about the danger of untyped data, the conclusion should restate this caveat prominently: the comparison to the 'gold-standard' typed analysis holds only when FP/FN labels are available and reliable.
  3. [Section VII-A, inequality (9)] The sentence "using (9) ensures the CBI-based assessment remains conservative" should be qualified: for k>=1 the bound is vacuous, and the conservatism guarantee that remains meaningful is the definitional one that holds only when the same evidence and likelihood are used in the typed analysis.
  4. [Appendix B] The phrase "The integer asymptotic supremum on n2 this implies is ceil(b1/(1-b1))" is unclear: the limit n2 -> b1/(1-b1) is a real limit, and the ceiling appears without justification; please clarify what exactly is being claimed about the integer-valued n2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's bounds are genuine solutions of its stated CBI optimization problems, and its self-citations are independent technical support rather than restatements of the conclusion.

full rationale

The paper's central claims are obtained by solving well-posed conservative Bayesian inference (CBI) optimization problems, equations (3)-(8), under explicit prior-knowledge constraints PK1-PK3. The theorems report infima of posterior confidence over the admissible prior set; no parameter is fitted to data and then renamed as a prediction, and no quantity is defined in terms of the target quantity it is supposed to establish. The untyped-versus-typed comparison is a comparison of infima under different likelihoods, not an identity imposed by construction. Inequality (9) is derived from the fact that the untyped likelihood is a convex combination of typed likelihoods, and the interchange of infima is proved in Appendix C. The self-citations to prior CBI work ([17], [19]) and to the authors' fixed-point characterization ([59]) are used for general lemmas and for consistency checks; these cited results are parameter-free mathematical or published results that do not themselves assert the paper's conclusions about data fidelity, so they are independent support under the stated rules. The paper does omit proofs for Theorems 1-3 and asserts Theorem 3's zero-infimum case without derivation; if that zero case failed, the infinite-optimism-gap conclusion would weaken. That is a correctness/verification risk, not a circular step, because the theorems do not assume the conclusion they are used to draw. Overall, the derivation chain is self-contained in the sense that the new bounds are genuine consequences of the stated optimization problems, and no circular reduction is exhibited.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The quantitative claims depend on assessor-specified PK parameters (a, b1, b, l, l1, l2) and the i.i.d. assumption; these are not fitted to data but chosen for illustrative scenarios. The central theorems are symbolic and hold for a range of parameter values, but the numerical gap magnitudes and the 'first estimates' claim rest on these choices, so they are listed as free parameters. The other entries are the modeling and mathematical background the derivation pulls from prior work.

free parameters (6)
  • a = 0.9 (Scenario 1; varies in Fig. 10)
    Assessor's prior confidence that accuracy meets the engineering target b1; chosen from elicited belief, not fitted to data.
  • 1-b1 = 1.09e-10 (Scenario 1); 1e-4 (Fig. 12)
    Engineering target pfc; scenario input.
  • b = derived from n1 in Scenario 2; 1e-2 in Fig. 12
    Claimed upper bound on pfc; target of the confidence statement.
  • l = implied by P,Q >= 1e-15 in Scenario 1
    Lower bound on total pfc in PK1; assessor's belief about achievable performance.
  • l1, l2 = 1e-9 each (Fig. 12)
    Lower bounds on individual FP/FN probabilities in PK3; assessor's belief.
  • eta = 0..1 (Fig. 12)
    Confidence weight assigned to the i.i.d. assumption in the Markov sensitivity model; varied to explore sensitivity.
assumptions (6)
  • domain assumption Classifier outcomes are i.i.d. categorical draws with unknown probabilities (P,Q,Theta)
    Invoked in Section IV-A; the paper notes it holds only approximately (footnote 1) and analyzes robustness to it in Section VII.
  • domain assumption Ground truth and counterfactual reasoning reliably label each outcome as success, FP, or FN
    Invoked in Section III-B; needed for Theorems 3-4 to provide the conservative typed-data benchmark.
  • domain assumption Prior knowledge constraints PK1-PK3 define the admissible set D of priors
    Invoked in Section IV-B; the bounds are computed over D.
  • domain assumption Reliability-target ordering 1-b1 <= b holds
    Invoked in Theorems 1-4 and Appendix A; the indicator 1_{1-b1 <= b} in the formulas depends on it.
  • standard math Fixed-point characterization of extremal priors (Theorem 1 of [28] or Proposition 2.2 of [59])
    Used in Appendix A to restrict the optimization to discrete priors; cited background result.
  • standard math Dinkelbach iteration applies to the fractional-programming infimum
    Used in Appendix A to find the infimum; standard technique.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Impact of Operational-Data Fidelity when Assessing Safety-Critical Autonomous-Vehicle Software." pith.science (2026). https://pith.science/paper/SVPQYTVJ

@misc{pith2026260810025,
  author       = {Pith},
  title        = {Pith review of: The Impact of Operational-Data Fidelity when Assessing Safety-Critical Autonomous-Vehicle Software},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SVPQYTVJ}},
  note         = {Machine review of arXiv:2608.10025}
}
read the original abstract

For safety-critical software, data from the software's operational past (e.g. a sequence of success and failure events experienced by the software) can provide strong statistical support for reliability claims about the software. However, such data might not describe past software failure events in sufficient detail, and this might leave a reliability assessment (based on this data) unable to account for important features of past software failures. In this paper, by extending conservative Bayesian inference (CBI) techniques used in reliability assessment, we illustrate a principled statistical approach for checking the robustness of reliability claims derived from insufficiently detailed operational data. We demonstrate the extent to which insufficient detail in operational data can undermine software reliability claims in autonomous vehicle (AV) safety assessment scenarios. Reliability claims derived from insufficiently fine-grained data might be dangerously optimistic, despite a concerted effort by an assessor to use such data conservatively during the assessment. While these findings are consistent with previous work on the impact of statistical model fidelity in Bayesian software reliability assessments, our work clarifies why attempts to use low-fidelity data conservatively can be naive, and we give the first conservative estimates of the impact of data fidelity on assessments.

Figures

Figures reproduced from arXiv: 2608.10025 by the authors.

Figure 1
Figure 1. An AV autonomy stack schematic indicating typical stages in the stack, applications of classifiers (i.e. white boxes) with a focus on the monitoring [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. PK1, PK2, partition the set Ω of feasible (P, Q)-values. p 1 − b1 l2 1 1 − b1 l1 1 0 q S2 S1 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. PK2, PK3, partition the set Ω. claims using statistical/operational testing [58]. Typically, b1 is relatively large, because the development team aims to develop classifiers that exhibit high accuracy (i.e. Θ). Relatedly, a is relatively large because the assessor is very confident that the development team has produced a classifier that meets relia￾bility targets. Since 1 − b is the minimum accuracy required of the… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Example worst-case prior distributions when the assessor has uncertainty about which failure-types have occurred. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Example worst-case prior distributions when the assessor knows which failure-types have occurred. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 2
Figure 2. Figure 2: The infimum Φ ∗ is 0 if k1 ⩾ 1 or k2 ⩾ 1. Otherwise, Φ ∗ = aL2∗ aL2∗ + (1 − a)L ∗ 1 11−b1⩽b, k1=k2=0 That is, objective function (7) attains its infimum, Φ ∗ , with a discrete prior distribution over Ω of the form (e.g. see Fig. 5a): P(P = pi , Q = qi) = ( 1 − a, if i …
Figure 6
Figure 6. Figure 6: The number of failure-free classifications needed for [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The extra failure-free classifications needed to regain lost confidence [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Conservative posterior confidence guaranteed by (9) [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Illustration of where the CBI theorems apply across an ISO 26262-style software (SW) lifecycle (illustrated for the AV stack of Fig. 1). [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: sensitivity of failure-free evidence needed to obtain [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Three-state stationary Markov model for classifier outcomes. The e [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Sensitivity of conservative posterior confidence to [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 50 canonical work pages

  1. [1]

    Keeping safe on the roads: series of standards for vehicle electronics functional safety just updated,

    International Organization for Standardization (ISO), “Keeping safe on the roads: series of standards for vehicle electronics functional safety just updated,” Dec. 2018. [Online]. Available: https://www.iso. org/news/ref2358.html

    work page 2018

  2. [2]

    Overview of the second edition of ISO 26262: Functional safety—road vehicles,

    R. Debouk, “Overview of the second edition of ISO 26262: Functional safety—road vehicles,”Journal of System Safety, vol. 55, no. 1, pp. 13– 21, Mar. 2019. [Online]. Available: https://doi.org/10.56094/jss.v55i1.55

    work page doi:10.56094/jss.v55i1.55 2019

  3. [3]

    ISO 26262-6:2018 road vehicles — functional safety — part 6: Product development at the software level,

    International Organization for Standardization, “ISO 26262-6:2018 road vehicles — functional safety — part 6: Product development at the software level,” Dec. 2018, edition 2. [Online]. Available: https://www.iso.org/standard/68388.html

    work page 2018

  4. [4]

    ISO 26262-8:2018 road vehicles — functional safety — part 8: Supporting processes,

    ——, “ISO 26262-8:2018 road vehicles — functional safety — part 8: Supporting processes,” Dec. 2018, edition 2. [Online]. Available: https://www.iso.org/standard/68390.html 13

    work page 2018

  5. [5]

    The Role of Confirmation Measures in ISO 26262 Functional Safety Automotive Products,

    S. Chonnad, V . Litovtchenko, and V . Sargsyan, “The Role of Confirmation Measures in ISO 26262 Functional Safety Automotive Products,” White paper, Synopsys, Jan. 2023, pub: Jan. 2023. Accessed: 2025-12-29. [Online]. Available: https://data.embeddedcomputing.com/ uploads/articles/whitepapers/16799.pdf [6]ISO 21448:2022 Road vehicles — Safety of the inten...

    work page 2023

  6. [9]

    Assurance 2.0: A manifesto,

    R. Bloomfield and J. Rushby, “Assurance 2.0: A manifesto,” 2020. [Online]. Available: https://arxiv.org/abs/2004.10474

    arXiv 2020

  7. [10]

    Assessing confidence with assurance 2.0,

    ——, “Assessing confidence with assurance 2.0,” 2022. [Online]. Available: https://arxiv.org/abs/2205.04522

    arXiv 2022

  8. [11]

    A scalable framework for safety assurance of self-driving vehicles based on assurance 2.0,

    S. Chen, M. J. Elizebeth, R. Aghazadeh Chakherlou, X. Zhao, E. Barbier, S. Khastgir, and P. Jennings, “A scalable framework for safety assurance of self-driving vehicles based on assurance 2.0,” 2025. [Online]. Available: https://arxiv.org/abs/2510.00092

    arXiv 2025

Show all 65 references
  1. [12]

    Toward a formalism for conservative claims about the dependability of software-based systems,

    P. G. Bishop, R. E. Bloomfield, B. Littlewood, A. A. Povyakalo, and D. R. Wright, “Toward a formalism for conservative claims about the dependability of software-based systems,”IEEE Transactions on Software Engineering, vol. 37, no. 5, pp. 708–717, 2011

    2011

  2. [13]

    Software fault-freeness and reliability predictions,

    L. Strigini and A. A. Povyakalo, “Software fault-freeness and reliability predictions,” inComputer Safety, Reliability, and Security (SAFECOMP 2013), ser. Lecture Notes in Computer Science, vol. 8153. Springer, 2013, pp. 106–117

    2013

  3. [14]

    On reliability assessment when a software-based system is replaced by a thought- to-be-better one,

    B. Littlewood, K. Salako, L. Strigini, and X. Zhao, “On reliability assessment when a software-based system is replaced by a thought- to-be-better one,”Reliability Engineering & System Safety, vol. 197, p. 106752, 2020

    2020

  4. [15]

    Conservative confidence bounds in safety, from generalised claims of improvement & statistical evidence,

    K. Salako, L. Strigini, and X. Zhao, “Conservative confidence bounds in safety, from generalised claims of improvement & statistical evidence,” in51st Annual IEEE/IFIP Int. Conf. on Dependable Systems and Networks, ser. DSN’21. Taipei Taiwan: IEEE/IFIP, 2021, pp. 451–462

    2021

  5. [16]

    Loss-size and reliability trade-offs amongst diverse re- dundant binary classifiers,

    K. Salako, “Loss-size and reliability trade-offs amongst diverse re- dundant binary classifiers,” inQuantitative Evaluation of Systems, M. Gribaudo, D. N. Jansen, and A. Remke, Eds. Springer International Publishing, 2020, pp. 96–114

    2020

  6. [17]

    Assessing the safety and reliability of autonomous vehicles from road testing,

    X. Zhao, V . Robu, D. Flynn, K. Salako, and L. Strigini, “Assessing the safety and reliability of autonomous vehicles from road testing,” inthe 30th Int. Symp. on Software Reliability Engineering. Berlin, Germany: IEEE, 2019, pp. 13–23

    2019

  7. [18]

    Assessing safety-critical systems from operational testing: A study on autonomous vehicles,

    X. Zhao, K. Salako, L. Strigini, V . Robu, and D. Flynn, “Assessing safety-critical systems from operational testing: A study on autonomous vehicles,”Information and Software Technology, vol. 128, p. 106393, 2020

    2020

  8. [19]

    The unnecessity of assuming statistically independent tests in bayesian software reliability assessments,

    K. Salako and X. Zhao, “The unnecessity of assuming statistically independent tests in bayesian software reliability assessments,”IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2829–2838, 2023

    2023

  9. [20]

    Demonstrating software reliability using possibly correlated tests: Insights from a conservative bayesian approach,

    ——, “Demonstrating software reliability using possibly correlated tests: Insights from a conservative bayesian approach,”Quality and Reliability Engineering International, vol. 40, no. 3, pp. 1197–1220, 2024

    2024

  10. [21]

    Bootstrapping confidence in future safety from past safe operation,

    P. Bishop, A. Povyakalo, and L. Strigini, “Bootstrapping confidence in future safety from past safe operation,” in2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE). Charlotte, NC, USA: IEEE, 2022, pp. 97–108

    2022

  11. [22]

    Arguing safety of an improved autonomous vehicle from safe operation before the change: new results,

    R. Aghazadeh Chakherlou, K. Salako, and L. Strigini, “Arguing safety of an improved autonomous vehicle from safe operation before the change: new results,” in2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 2022, pp. 307– 312, pre...

    2022

  12. [23]

    Using pre-change opera- tional evidence for predicting post-change reliability, given prior confi- dence in fault-freeness,

    R. Aghazadeh Chakherlou and L. Strigini, “Using pre-change opera- tional evidence for predicting post-change reliability, given prior confi- dence in fault-freeness,” inAdvances in Reliability, Safety and Security, Part 3: ESREL 2024 Contributions, K. Kolowrocki and E. Dabrows...

    2024

  13. [24]

    Doubt in safety claims is inevitable: What is its impact, what can be done about it?

    P. Bishop, A. Povyakalo, and L. Strigini, “Doubt in safety claims is inevitable: What is its impact, what can be done about it?” inComputer Safety, Reliability, and Security. SAFECOMP 2025 Workshops, ser. Lec- ture Notes in Computer Science, M. T ¨orngren, B. Gallina, E. Schoi...

    2025

  14. [25]

    Robust bayesian analysis: Sensitivity to the prior,

    J. O. Berger, “Robust bayesian analysis: Sensitivity to the prior,”Journal of Statistical Planning and Inference, vol. 25, no. 3, pp. 303–328, 1990

    1990

  15. [26]

    Robust bayes and empirical bayes analysis withϵ-contaminated priors,

    J. Berger and L. M. Berliner, “Robust bayes and empirical bayes analysis withϵ-contaminated priors,”The Annals of Statistics, vol. 14, no. 2, pp. 461–486, 1986. [Online]. Available: https://projecteuclid.org/journals/annals-of-statistics/volume-14/ issue-2/Robust-Bayes-and-Emp...

    1986

  16. [27]

    Walley,Statistical Reasoning with Imprecise Probabilities

    P. Walley,Statistical Reasoning with Imprecise Probabilities. Chapman and Hall, 1991

    1991

  17. [28]

    Robust bayesian analysis withϵ- contaminations partially known,

    E. Moreno and J. A. Cano, “Robust bayesian analysis withϵ- contaminations partially known,”Journal of the Royal Statistical So- ciety: Series B (Methodological), vol. 53, no. 1, pp. 143–155, 1991

    1991

  18. [29]

    An overview of robust Bayesian analysis,

    J. O. Berger, “An overview of robust Bayesian analysis,”Test, vol. 3, no. 1, pp. 5–124, 1994

    1994

  19. [30]

    Conservative software reliability assessments using collections of bayesian inference problems,

    K. Salako and R. T. Muhammad, “Conservative software reliability assessments using collections of bayesian inference problems,” arXiv, 2025, accessed: 2025-12-22. [Online]. Available: https://arxiv.org/abs/ 2511.07038

    2025

  20. [31]

    Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?

    N. Kalra and S. M. Paddock, “Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?” Transportation Research Part A: Policy and Practice, vol. 94, pp. 182– 193, 2016

    2016

  21. [32]

    Dynamic safety assessment of autonomous vehicle based on multivariate bayesian inference (DyA VSA),

    P. Popov, “Dynamic safety assessment of autonomous vehicle based on multivariate bayesian inference (DyA VSA),”Journal of Reliable Intelligent Environments, vol. 11, 2025

    2025

  22. [33]

    Survey on scenario-based safety assessment of automated vehicles,

    S. Riedmaier, T. Ponn, D. Ludwig, B. Schick, and F. Diermeyer, “Survey on scenario-based safety assessment of automated vehicles,” IEEE Access, vol. 8, pp. 87 456–87 477, 2020

    2020

  23. [34]

    On the need for a statistical foundation in scenario-based testing of autonomous vehicles,

    X. Zhao, R. Aghazadeh-Chakherlou, C.-H. Cheng, P. Popov, and L. Strigini, “On the need for a statistical foundation in scenario-based testing of autonomous vehicles,” in2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2025, pp. 3998–4...

    2025

  24. [35]

    Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques,

    D. Zhao, H. Lam, H. Peng, S. Bao, D. J. LeBlanc, K. Nobukawa, and C. S. Pan, “Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques,”IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 3, pp. 595–...

    2017

  25. [36]

    Scal- able end-to-end autonomous vehicle testing via rare-event simulation,

    M. O’Kelly, A. Sinha, H. Namkoong, R. Tedrake, and J. C. Duchi, “Scal- able end-to-end autonomous vehicle testing via rare-event simulation,” inAdvances in Neural Information Processing Systems 31. Curran Associates, Inc., 2018, pp. 9827–9838

    2018

  26. [37]

    Diagnostic tests. 1: Sensitivity and specificity,

    D. G. Altman and J. M. Bland, “Diagnostic tests. 1: Sensitivity and specificity,”BMJ, vol. 308, no. 6943, p. 1552, 1994. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/8019315/

    1994

  27. [38]

    A systematic analysis of performance measures for classification tasks,

    M. Sokolova and G. Lapalme, “A systematic analysis of performance measures for classification tasks,”Information Processing & Manage- ment, vol. 45, no. 4, pp. 427–437, 2009

    2009

  28. [39]

    The relationship between precision-recall and roc curves,

    J. Davis and M. H. Goadrich, “The relationship between precision-recall and roc curves,” inProceedings of the 23rd International Conference on Machine Learning (ICML ’06). ACM, 2006, pp. 233–240

    2006

  29. [40]

    An introduction to ROC analysis,

    T. Fawcett, “An introduction to ROC analysis,”Pattern Recognition Letters, vol. 27, no. 8, pp. 861–874, 2006

    2006

  30. [41]

    The meaning and use of the area under a receiver operating characteristic (ROC) curve,

    J. A. Hanley and B. J. McNeil, “The meaning and use of the area under a receiver operating characteristic (ROC) curve,” Radiology, vol. 143, no. 1, pp. 29–36, 1982. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/7063747/

    1982

  31. [42]

    The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,

    T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets,”PLOS ONE, vol. 10, no. 3, p. e0118432,

  32. [43]

    Bayesian semi- parametric roc analysis,

    A. Erkanli, M. Sung, E. J. Costello, and A. Angold, “Bayesian semi- parametric roc analysis,”Statistics in Medicine, vol. 25, no. 22, pp. 3905–3928, 2006

    2006

  33. [44]

    Bayesian semiparametric roc curve estimation and disease diagnosis,

    A. J. Branscum, W. O. Johnson, T. E. Hanson, and I. A. Gardner, “Bayesian semiparametric roc curve estimation and disease diagnosis,” Statistics in Medicine, vol. 27, no. 13, pp. 2474–2496, 2008

    2008

  34. [45]

    Bayesian bootstrap estimation of roc curve,

    J. Gu, S. Ghosal, and A. Roy, “Bayesian bootstrap estimation of roc curve,”Statistics in Medicine, vol. 27, no. 26, pp. 5407–5420, 2008. 14

    2008

  35. [46]

    Bayesian inferences for receiver operating characteristic curves in the absence of a gold standard,

    Y .-K. Choi, W. O. Johnson, M. T. Collins, and I. A. Gardner, “Bayesian inferences for receiver operating characteristic curves in the absence of a gold standard,”Journal of Agricultural, Biological, and Environmental Statistics, vol. 11, no. 2, pp. 210–229, 2006

    2006

  36. [47]

    Perception, planning, control, and coordination for autonomous vehicles,

    S. D. Pendleton, H. Andersen, X. Du, X. Shen, M. Meghjani, Y . H. Eng, D. Rus, and M. H. Ang, “Perception, planning, control, and coordination for autonomous vehicles,”Machines, vol. 5, no. 1, p. 6,

  37. [48]

    Planning and decision- making for autonomous vehicles,

    W. Schwarting, J. Alonso-Mora, and D. Rus, “Planning and decision- making for autonomous vehicles,”Annual Review of Control, Robotics, and Autonomous Systems, vol. 1, pp. 187–210, 2018

    2018

  38. [49]

    Safe automated driving: Requirements and architectures,

    The Autonomous Working Group (Safety & Architecture), “Safe automated driving: Requirements and architectures,” Full report, Version 1.0 (01.12.2023), Dec. 2023, accessed: 2025-12-29. [Online]. Available: https://www.the-autonomous.com/wp-content/uploads/2023/ 12/wg-safetyarch...

    2023

  39. [50]

    Impact of machine learning on safety monitors,

    F. Terrosi, L. Strigini, and A. Bondavalli, “Impact of machine learning on safety monitors,” inComputer Safety, Reliability, and Security (SAFECOMP 2022), ser. Lecture Notes in Computer Science, vol. 13414. Cham: Springer, 2022, pp. 129–143. [Online]. Available: https://doi.or...

    2022 doi

  40. [51]

    A safety monitoring concept for fully automated driving,

    S. Kojchev, E. Klintberg, and J. Fredriksson, “A safety monitoring concept for fully automated driving,” in2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2020, pp. 1–7. [Online]. Available: https://doi.org/10.1109/ITSC45102.2020. 9294307

    2020

  41. [52]

    Co-design of out-of-distribution detectors for autonomous emergency braking systems,

    M. Yuhas and A. Easwaran, “Co-design of out-of-distribution detectors for autonomous emergency braking systems,” in2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC). Bilbao, Bizkaia, Spain: IEEE, 2023, pp. 1996–2003. [Online]. Available: http...

    2023

  42. [53]

    Certified control: An architecture for verifiable safety of autonomous vehicles,

    D. Jackson, V . Richmond, M. Wang, J. Chow, U. Guajardo, S. Kong, S. Campos, G. Litt, and N. Ar ´echiga, “Certified control: An architecture for verifiable safety of autonomous vehicles,” 2021. [Online]. Available: https://arxiv.org/abs/2104.06178

    2021 arXiv

  43. [54]

    Challenges in autonomous vehicle testing and validation,

    P. Koopman and M. Wagner, “Challenges in autonomous vehicle testing and validation,”SAE International Journal of Transportation Safety, vol. 4, no. 1, pp. 15–24, 2016

    2016

  44. [55]

    Failure correlation in software reliability models,

    K. Goseva-Popstojanova and K. S. Trivedi, “Failure correlation in software reliability models,”IEEE Transactions on Reliability, vol. 49, no. 1, pp. 37–48, 2000

    2000

  45. [56]

    Time-dependent error-detection rate model for software reliability and other performance measures,

    A. L. Goel and K. Okumoto, “Time-dependent error-detection rate model for software reliability and other performance measures,”IEEE Transactions on Reliability, vol. R-28, no. 3, pp. 206–211, 1979

    1979

  46. [57]

    Validation of ultra-high dependability for software-based systems,

    B. Littlewood and L. Strigini, “Validation of ultra-high dependability for software-based systems,”Communications of the ACM, vol. 36, no. 11, pp. 69–80, 1993

    1993

  47. [58]

    Guidelines for statistical testing,

    L. Strigini and B. Littlewood, “Guidelines for statistical testing,” Centre for Software Reliability, City University London, Technical Report, PASCON/WO6-CCN2/TN12, 1997, technical report prepared under ESA/ESTEC project PASCON. [Online]. Available: https://openaccess. city.a...

    1997

  48. [59]

    Fixed-point characterisations of extremal distributions under partial distributional constraints,

    K. Salako and R. T. Muhammad, “Fixed-point characterisations of extremal distributions under partial distributional constraints,” 2026. [Online]. Available: https://arxiv.org/abs/2608.04315

    2026 arXiv

  49. [60]

    Why black-box bayesian safety assessment of autonomous vehicles is problematic and what can be done about it?

    P. Popov, “Why black-box bayesian safety assessment of autonomous vehicles is problematic and what can be done about it?”IEEE Transac- tions on Intelligent Vehicles, vol. 11, no. 2, pp. 311–324, 2026

    2026

  50. [61]

    Assurance cases and prescriptive software safety certification: A comparative study,

    R. Hawkins, I. Habli, T. Kelly, and J. McDermid, “Assurance cases and prescriptive software safety certification: A comparative study,”Safety Science, vol. 59, pp. 55–71, 2013

    2013

  51. [62]

    O’Hagan, C

    A. O’Hagan, C. E. Buck, A. Daneshkhah, J. R. Eiser, P. H. Garthwaite, D. J. Jenkinson, J. E. Oakley, and T. Rakow,Uncertain Judgements: Eliciting Experts’ Probabilities. John Wiley & Sons, 2006

    2006

  52. [63]

    Statistical methods for eliciting probability distributions,

    P. H. Garthwaite, J. B. Kadane, and A. O’Hagan, “Statistical methods for eliciting probability distributions,”Journal of the American Statistical Association, vol. 100, no. 470, pp. 680–701, 2005

    2005

  53. [64]

    Using ranked nodes to model qualitative judgments in bayesian networks,

    N. E. Fenton, M. Neil, and J. G. Caballero, “Using ranked nodes to model qualitative judgments in bayesian networks,”IEEE Transactions on Knowledge and Data Engineering, vol. 19, no. 10, pp. 1420–1432, 2007

    2007

  54. [65]

    When does “Diversity

    K. Salako and L. Strigini, “When does “Diversity” in development reduce common failures? insights from probabilistic modeling,”IEEE Trans. Dependable Secur. Comput., vol. 11, no. 2, pp. 193–206, 2014. [Online]. Available: https://doi.org/10.1109/TDSC.2013.32

    2014 doi

  55. [66]

    The variation of software survival time for different operational input profiles (or why you can wait a long time for a big bug to fail),

    P. G. Bishop, “The variation of software survival time for different operational input profiles (or why you can wait a long time for a big bug to fail),” inFTCS-23: The Twenty-Third International Symposium on Fault-Tolerant Computing. IEEE, 1993, pp. 98–107. APPENDIXA PROOF OF...

    1993

  56. [2015]

    Available: https://journals.plos.org/plosone/article?id= 10.1371/journal.pone.0118432

    [Online]. Available: https://journals.plos.org/plosone/article?id= 10.1371/journal.pone.0118432

    doi

  57. [2017]

    Available: https://www.mdpi.com/2075-1702/5/1/6

    [Online]. Available: https://www.mdpi.com/2075-1702/5/1/6

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.