Pith. sign in

REVIEW 3 major objections 4 minor 18 references

The paper claims that the fraction of defects reported exactly once during beta testing — s/n — is a conservative upper bound on the probability that a released user hits a defect no tester saw, and that this bound needs no operational prof

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 19:08 UTC pith:JHTDC276

load-bearing objection Good-Turing s/n is repackaged as a release-decision upper bound, but the bound only holds in expectation; the paper's own Table 1 shows s/n understates U nearly half the time, so the headline claim does not survive. the 3 major comments →

arxiv 2607.17061 v1 pith:JHTDC276 submitted 2026-07-19 stat.AP

An Upper Bound on the Probability That a User Encounters an Undiscovered Defect

Carlos M. Hern\'andez-Su\'arez , Karla Hern\'andez-Cuevas This is my paper
classification stat.AP
keywords software release decisioncoverage estimationunseen defect probabilitybeta testingcrash telemetrycanonical urnmissing massoperational profile
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a release decision can be made from one number: the fraction s/n of defects in beta or crash telemetry that were reported exactly once. It argues that this number is a conservative upper bound on the probability that the next user encounters a defect not seen in testing — overstating risk rather than understating it, which is the safe direction for a ship decision. The bound is derived by embedding any observed sample into a canonical population with a 'reservoir' of all-different unseen classes, making s/n the exact maximum-likelihood estimate of the unseen mass. Because encounter counts already encode how many users reach each bug, the estimate is claimed to be invariant to whether the code's reachability graph is a tree or a directed acyclic graph and to automatically cover defects hidden behind other defects; no operational profile — no model of how users will exercise the software — is needed. The authors validate the bound on synthetic populations and stress that the user-facing reading depends on encounter-level telemetry, not on public bug-report archives where each defect is typically filed once regardless of how many users hit it.

Core claim

On the paper's own terms, the core discovery is that the classic coverage estimate — the fraction of sampled items seen exactly once — solves the software release question directly when the sample consists of user encounters. The authors prove, through a canonical urn construction, that s/n is the exact maximum-likelihood estimate of the total probability mass of defects never seen in testing, and that because the construction charges every singleton to the unseen reservoir, the estimate dominates the true unseen mass: its expected value is never lower. They then show the bound is architectural: report counts already carry the effect of how users reach defects, so tree-versus-DAG structure a

What carries the argument

The central object is the canonical form: a population whose classes are K−1 ordinary classes, each contributing a single colour that can recur, plus one reservoir of arbitrarily many classes of equal, vanishing probability, so every draw from it yields a previously unseen colour. Any finite sample can be embedded in such a population by assigning every multiply-observed defect to an ordinary class and every singleton to the reservoir; a likelihood comparison (the all-different dominance lemma) shows this assignment is more probable than modelling the singletons as ordinary classes, by a factor of s^s. In this construction the unseen proportion U equals the reservoir mass p_K, so s/n is its

Load-bearing premise

The user-facing reading of s/n holds only if each beta-test report is an independent draw from a single fixed user population with each distinct defect a class; if reporting is filtered (a bug filed once no matter how many users hit it) or the released software changes reachability relative to what was tested, the claimed upper bound need not apply.

What would settle it

Run a beta phase with encounter-level telemetry, compute s/n on that sample, and ship. In the released population, measure the realized fraction of user encounters that land on a defect absent from the beta reports. If that realized fraction exceeds the beta s/n — or exceeds a high-confidence upper bound built from s/n via known concentration inequalities — over a reasonably sized user base, the paper's conservative-bound claim is falsified. This mirrors exactly what the paper's synthetic experiments check against known ground truth, so the same protocol transfers directly to real telemetry.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A developer can translate a risk tolerance into a stopping rule: keep testing until s/n, recomputed on the latest beta/crash sample, falls below the acceptable probability of a user meeting an unseen defect.
  • No operational profile is needed: the estimate is distribution-free over both the user population and the distribution of defect reach probabilities, and it is computed from report counts alone.
  • The bound automatically covers defects hidden behind other defects and is invariant to whether reachability is a tree or a directed acyclic graph, so no architectural model or program-internal knowledge is required.
  • Sampling error can be handled with known concentration inequalities for the missing mass, allowing a high-confidence upper bound built from s/n rather than only a point estimate.
  • Each new testing round yields a fresh valid snapshot for the then-current build; the paper does not claim any relationship between successive estimates as defects are fixed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The per-sample caveat is worth drawing out: the conservative direction is established for expectation, E[s/n] ≥ E[U]; on any single beta sample s/n can sit below the true unseen mass, and the paper's own simulations report Pr(s/n ≥ U) between about 0.5 and 1 depending on sample size.
  • Because fixes change reachability, one natural testable extension is to recompute s/n immediately after a fix on post-fix telemetry; the snapshot language of the paper implies the pre-fix bound does not automatically cover the post-fix build.
  • The same canonical-urn logic could transfer to any domain with per-individual encounter data and hidden classes — field failure reporting for medical devices, unobserved side effects in pharmacovigilance, or content moderation gaps — wherever a single filtered record would otherwise destroy the multiplicity count.
  • A severity-weighted variant is a natural next step, and the paper itself notes it: reweight each defect by expected harm instead of treating a cosmetic glitch and a sign-in crash equally, which would convert the incidence bound into a harm-weighted risk bound.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper claims that the fraction of beta-test reports that are singletons, s/n, is a conservative upper bound on the probability that a released user encounters a defect unseen in testing. The argument proceeds by embedding any population into a 'canonical form' in which all observed singletons are assigned to an all-different reservoir, and showing that s/n is the maximum-likelihood estimate of the reservoir mass, interpreted as the unseen proportion. The paper further argues that the estimate is invariant to software architecture, so that reachability structure is already reflected in report counts, and validates the estimator on synthetic populations.

Significance. If the advertised per-sample upper bound were valid, the result would offer a practically attractive release criterion from beta or crash telemetry, requiring no operational profile. The paper is clearly written and honestly acknowledges some limitations, and the expectation inequality E[s/n] >= E[U] is a known and correct property of the Good–Turing coverage estimator. However, the central claim—that s/n is a conservative upper bound for a realized sample—is not established and is in fact contradicted by the paper's own Table 1. The maximum-likelihood derivation is an artifact of the chosen canonical parametrization, and the estimand is not clearly the user-facing probability stated in the abstract. These issues undermine the main contribution, not merely its presentation.

major comments (3)
  1. [Remark 1 and Table 1] The paper's central claim is that s/n is a 'conservative upper bound' on the unseen proportion U. Remark 1, however, establishes only E[s/n] >= E[U], an inequality in expectation over repeated samples. This does not imply that s/n >= U for the realized sample. Table 1 reports Pr(s/n >= U) = 0.53 at n=25 and 0.57 at n=50, so s/n understates U in roughly 47% and 43% of samples. Under these numbers, s/n is not an upper bound in any decision-relevant sense. A release decision needs a bound that holds for the realized sample, at least with high probability; the paper cites concentration inequalities for the missing mass but never deploys them. The advertised guarantee therefore fails even under the paper's own categorical model.
  2. [Section 2.3, Definition 1 and Proposition 1] Theorem 1's MLE result is forced by the canonical-form construction rather than being an independent derivation of a bound for a real population. Proposition 1 defines the reservoir as the component that supplies singletons and posits no unobserved ordinary classes; under that parametrization the unseen mass is p_K by construction, so maximizing the likelihood cannot help but return s/n. The universality result only shows that any sample is compatible with some canonical population, not that the canonical population is the true population. Consequently Theorem 1 does not provide evidence that s/n upper-bounds U for the actual sampling distribution. The substantive one-sidedness is the expectation inequality, which is a known property of Good–Turing estimation and does not require the canonical form.
  3. [Section 2 (target U) and Abstract] The estimand is not consistently the user-facing quantity stated in the abstract. Section 2 defines U as the probability that the next draw belongs to an unseen class, i.e., a single encounter. But the abstract and release problem concern the probability that a user encounters at least one unseen defect. A real user can have multiple encounters, so the sum of unseen-class probabilities is the expected number of unseen-defect encounters, not the probability of encountering at least one. A union bound would make that sum an upper bound for the probability, but the paper does not make this distinction and instead identifies U directly with the 'fresh user' probability. This conflates 'encounter' and 'user' and leaves the central claim undefined for the stated application.
minor comments (4)
  1. [Definition 1] The reservoir is described as 'arbitrarily many classes of equal, vanishing probability.' This is mathematically imprecise: countably many equal positive probabilities cannot sum to a finite p_K. A limiting or non-atomic formulation should be given.
  2. [Table 1 caption] The caption says 'the bound is one-sided' while the rightmost column shows Pr(s/n >= U) is only 0.53–0.57 at moderate n. The caption is misleading and should be revised to describe the expectation bias, not a per-sample bound.
  3. [Section 4] The sentence 'Since s/n is an accurate but noisy estimate, one may also bound the sampling error, for which distribution-free concentration inequalities for the missing mass are available [11,4]' points exactly at what is missing. The paper should either prove a high-probability bound or state clearly that the current result is only a bias result.
  4. [Section 2.4] The claim that 'tree or DAG, the same reports give the same s/n' is trivially true because s/n is computed from counts only. The substantive claim that hidden defects are 'bounded automatically' is argued informally in the Introduction and is not formalized in Section 2.4.

Circularity Check

2 steps flagged

The exact-MLE and upper-bound claims are engineered by the canonical-form construction: the unseen reservoir is defined as the mass that absorbs all singletons, making s/n the model parameter by construction; the per-sample 'never understating' claim is contradicted by the paper's own Table 1.

specific steps
  1. self definitional [Section 2.2 (Proposition 1), Section 2.3 (Theorem 1, Remark 1)]
    "A class observed exactly once is, from the sample alone, indistinguishable from a fresh draw of the all-different reservoir, and may be assigned to it... Under the canonical form the unseen proportion equals the reservoir mass, which Theorem 1 estimates by s/n. The canonical form reaches this value by charging every singleton to the all-different reservoir."

    The canonical population is not a fixed model; it is fitted to the realized sample by assigning every singleton to the reservoir, and the reservoir is by definition the all-different component that supplies unseen classes. Therefore p_K = s/n in the fitted model by construction, so the 'maximum-likelihood estimate' of the unseen proportion simply recovers the value used to define the model. Proposition 1's universality is a sample-conditional embedding, not an independent constraint on the real population. Theorem 1 is thus a tautology of the parameterization rather than a derived prediction.

  2. fitted input called prediction [Section 2.3, Remark 1; Section 3, Table 1]
    "Attributing every singleton to the reservoir can therefore only inflate the estimate, and s/n overstates the true unseen proportion on average, E[s/n] ≥ E[U]... the estimate errs toward overstating the probability that a user meets an unseen bug, never understating it."

    The one-sided direction is asserted as if the definitional assignment of singletons to the reservoir made s/n an upper bound in the realized sample. But the construction only shifts the expectation; the paper's own Table 1 reports Pr(s/n ≥ U) = 0.53 at n=25 and 0.57 at n=50, i.e. the 'bound' is violated in roughly 47% and 43% of samples. A release-decision bound would need a per-realization or high-probability guarantee, which the canonical-form argument does not supply. The fitted reservoir is being presented as a per-sample prediction it does not provide.

full rationale

The paper's core mathematical result is internally valid but circular in force: the canonical form is an after-the-fact parameterization in which all singletons are assigned to the all-different reservoir, making the unseen proportion p_K equal to s/n by definition. Theorem 1 then 'shows' the MLE is s/n, which is true but is a direct consequence of the construction, not an independent estimate. The extension to a conservative upper bound is not a formal consequence: Remark 1 establishes only E[s/n] ≥ E[U], and Table 1 shows s/n < U with substantial probability, contradicting the 'never understating' language and the abstract's unqualified upper-bound claim. The self-citation of [10] for the canonical urn and the inequality is not independently verified, but because the paper re-proves the core MLE step locally, the main circularity is the definitional one, not a pure citation chain. The synthetic validation and the candid discussion of encounter-level data give the work some empirical content, so the result is not wholly equivalent to its inputs, but the advertised release-decision guarantee is unsupported by the derivation.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

The central claim rests on one domain assumption (one user draw = one defect class) and two ad hoc canonical-form assumptions that are tailor-made to make s/n the MLE of the unseen mass. No free parameters are fitted to data. The invented all-different reservoir has no independent empirical support and exists only to produce the desired estimate.

axioms (3)
  • domain assumption Each beta-test report is an iid draw from a fixed population, and each draw belongs to exactly one defect class.
    Section 2 opening; required for the coverage interpretation. Invalid when a user encounters multiple defects in one session or when reporting filters multiplicity.
  • ad hoc to paper The canonical form's all-different reservoir has classes of vanishing equal probability, making two reservoir draws almost surely distinct.
    Definition 1; this is what makes the reservoir likelihood Q^s and forces the MLE to be s/n.
  • ad hoc to paper For a given sample, the canonical population contains only the observed multiply-seen classes plus the reservoir; no unobserved ordinary classes carry mass.
    Proposition 1; this is how the unseen mass U is made equal to reservoir mass p_K.
invented entities (1)
  • All-different reservoir (canonical reservoir) no independent evidence
    purpose: Models unseen classes as an infinite supply of vanishing-probability classes so that every singleton can be charged to the unseen mass, making s/n the MLE.
    A pure modeling device with no falsifiable content outside the paper; it guarantees the over-attribution direction by construction.

pith-pipeline@v1.3.0-alltime-deepseek · 7271 in / 16511 out tokens · 168209 ms · 2026-08-01T19:08:28.426206+00:00 · methodology

0 comments
read the original abstract

Before releasing software to a general population, a developer must weigh a single question: if we ship now, what fraction of users will still hit a defect? This is not a question about how many defects remain, nor whether any particular defect is present -- the quantities the reliability literature has long estimated -- but about a different and, for a release decision, more consequential one: the probability that a user encounters a defect at all. We give a direct, distribution-free answer. Reading each beta-test report as a draw from the user population and each distinct defect as a class, we show that the fraction of defects reported exactly once, $s/n$, is a conservative upper bound on the probability that a user encounters a defect unseen in testing. This bound is the exact maximum-likelihood estimate of the mass of unseen defects under a general urn construction -- the canonical form -- into which any population of classes embeds; because that construction charges every singleton to the unseen reservoir, $s/n$ overstates the user's risk rather than understating it, the direction a release decision requires. The estimate needs no operational profile, no assumption on the number or frequency of defects, and no model of the program's internal structure -- since a defect's report count already reflects how many users reach it, the estimate is invariant to whether the reachability graph is a tree or a directed acyclic graph, and defects hidden behind other defects are bounded automatically. We validate the estimator against synthetic populations with known ground truth, and discuss the encounter-level data -- beta or crash telemetry -- under which the user-facing reading holds.

Figures

Figures reproduced from arXiv: 2607.17061 by Carlos M. Hern\'andez-Su\'arez, Karla Hern\'andez-Cuevas.

Figure 1
Figure 1. Figure 1: A heavy-tailed population of 20 bugs. The estimate s/n (circles) lies on or above the true missing mass (squares) at every sample size; the shaded region is the over-attribution gap. Both decline as the sample grows, the release story in miniature: continue testing until the bound, not merely the estimate, falls below the tolerated risk. 4 Discussion The release question—what fraction of users will still m… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references

  1. [1]

    A unified maximum likelihood approach for estimating sym- metric properties of discrete distributions

    Jayadev Acharya, Hirakendu Das, Alon Orlitsky, and Ananda Theertha Suresh. A unified maximum likelihood approach for estimating sym- metric properties of discrete distributions. InProceedings of the 34th International Conference on Machine Learning (ICML), volume 70 of PMLR, pages 11–21, 2017

    2017

  2. [2]

    Bishop and Robin E

    Peter G. Bishop and Robin E. Bloomfield. Worst case reliability pre- diction based on a prior estimate of residual defects. InProceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE), pages 295–303. IEEE, 2002

    2002

  3. [3]

    STADS: Software testing as species discovery.ACM Transactions on Software Engineering and Methodology, 27(2):7:1–7:52,

    Marcel Böhme. STADS: Software testing as species discovery.ACM Transactions on Software Engineering and Methodology, 27(2):7:1–7:52,

  4. [4]

    Optimal dis- covery with probabilistic expert advice: Finite time analysis and macro- scopic optimality.Journal of Machine Learning Research, 14(1):601– 623, 2013

    Sébastien Bubeck, Damien Ernst, and Aurélien Garivier. Optimal dis- covery with probabilistic expert advice: Finite time analysis and macro- scopic optimality.Journal of Machine Learning Research, 14(1):601– 623, 2013

    2013

  5. [5]

    Laplante

    Robert Bucholz and Phillip A. Laplante. A dynamic capture–recapture model for software defect prediction.Innovations in Systems and Soft- ware Engineering, 5:265–270, 2009

    2009

  6. [6]

    Estimating the number of classes via sample coverage.Journal of the American Statistical Association, 87 (417):210–217, 1992

    Anne Chao and Shen-Ming Lee. Estimating the number of classes via sample coverage.Journal of the American Statistical Association, 87 (417):210–217, 1992

    1992

  7. [7]

    Warren W. Esty. A normal limit law for a nonparametric estimator of the coverage of a random sample.The Annals of Statistics, 11(3): 905–912, 1983

    1983

  8. [8]

    Goel and Kazuhira Okumoto

    Amrit L. Goel and Kazuhira Okumoto. Time-dependent error-detection rate model for software reliability and other performance measures. IEEE Transactions on Reliability, R-28(3):206–211, 1979. 12

    1979

  9. [9]

    I. J. Good. The population frequencies of species and the estimation of population parameters.Biometrika, 40(3–4):237–264, 1953

    1953

  10. [10]

    Hernández-Suárez

    Carlos M. Hernández-Suárez. Measuring the representativeness of a germplasm collection.Biodiversity and Conservation, 27(6):1471–1486, 2018

    2018

  11. [11]

    Concentration inequalities for the missing mass and for histogram rule error.Journal of Machine Learning Research, 4:895–911, 2003

    David McAllester and Luis Ortiz. Concentration inequalities for the missing mass and for histogram rule error.Journal of Machine Learning Research, 4:895–911, 2003

    2003

  12. [12]

    Ohannessian

    Elchanan Mossel and Mesrob I. Ohannessian. On the impossibility of learning the missing mass.Entropy, 21(1):28, 2019

    2019

  13. [13]

    Competitive distribution estimation: Why is Good–Turing good

    Alon Orlitsky and Ananda Theertha Suresh. Competitive distribution estimation: Why is Good–Turing good. InAdvances in Neural Infor- mation Processing Systems (NIPS), volume 28, pages 2143–2151, 2015

    2015

  14. [14]

    AlwaysGood Turing: Asymptotically optimal probability estimation.Science, 302 (5644):427–431, 2003

    AlonOrlitsky, NarayanaP.Santhanam, andJunanZhang. AlwaysGood Turing: Asymptotically optimal probability estimation.Science, 302 (5644):427–431, 2003

    2003

  15. [15]

    Capture–recapture in software inspections after 10 years research— theory, evaluation and application.Journal of Systems and Software, 72(2):249–264, 2004

    Håkan Petersson, Thomas Thelin, Per Runeson, and Claes Wohlin. Capture–recapture in software inspections after 10 years research— theory, evaluation and application.Journal of Systems and Software, 72(2):249–264, 2004

    2004

  16. [16]

    Conservative software re- liability assessments using collections of Bayesian inference problems

    Kizito Salako and Rabiu Tsoho Muhammad. Conservative software re- liability assessments using collections of Bayesian inference problems. arXiv preprint arXiv:2511.07038, 2025

    arXiv 2025

  17. [17]

    Mean-squared accuracy of Good–Turing estimator

    Maciej Skorski. Mean-squared accuracy of Good–Turing estimator. In 2021 IEEE International Symposium on Information Theory (ISIT), pages 2846–2851. IEEE, 2021. 13

    2021

  18. [2018]

    doi: 10.1145/3210309

    doi:10.1145/3210309