Pith. sign in

REVIEW 3 major objections 4 minor 21 references

This paper derives closed-form approximations for the median discovery significance in Poisson counting experiments with uncertain background, and shows they remain accurate at low event yields where the standard s/√(b+σ_b²) formula fails.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 07:07 UTC pith:TNYWHIM7

load-bearing objection Useful, honest derivation of the on/off median-significance formula with r* corrections; the main caveat is that the uncertain-background MC reference is itself approximate, so the low-yield improvements are not independently grounded. the 3 major comments →

arxiv 2607.29436 v1 pith:TNYWHIM7 submitted 2026-07-31 physics.data-an

Discovery Sensitivity for a Counting Experiment with Background Uncertainty

Enzo Canonero , Glen Cowan This is my paper
classification physics.data-an
keywords discovery significancemedian sensitivityAsimov data setprofile likelihood ratioPoisson counting experimentbackground uncertaintyhigher-order asymptoticsr* statistic
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to give particle physicists a reliable, closed-form way to predict how significant a discovery will be in a simple counting experiment before the experiment is run, even when the expected background is not known exactly. It derives Eq. (20), the full Asimov-data-set expression for the median significance when the background is constrained by a Poisson control measurement, and shows it reduces to the familiar s/√(b+σ_b²) only in the limit of small signal and small background uncertainty. At small event yields, the simple formula overestimates the median sensitivity, while the full expression tracks Monte Carlo results. The paper then applies a higher-order asymptotic correction (the r* statistic) along with, in the known-background case, a continuity correction, extending accuracy to expected counts as low as one. A sympathetic reader would take away that the paper's compact formulae give quantitatively better projections of experimental sensitivity, with errors small enough to matter for cut optimisation in rare-event searches.

Core claim

The paper establishes that the median discovery significance in a Poisson counting experiment with a Poisson-constrained background is, to first asymptotic order, the quantity in Eq. (20), a closed form involving the expected signal s, expected background b, and inferred background uncertainty σ_b. The familiar s/√(b+σ_b²) is only the leading-order limit of this expression, and the paper shows it overestimates sensitivity when s/b or σ_b²/b are not small. It further establishes that applying the r* higher-order correction at the same Asimov point improves both the observed and the median significance at low event yields, and that in the known-background case combining r* with a continuity co

What carries the argument

The central object is the Asimov data set: replacing the actual observed counts n and m by their expectation values s+b and τb inside the profile-likelihood-ratio statistic q0, giving a closed-form estimate of the median significance. The decisive identity is Eq. (20), obtained by eliminating the control-region scale factor τ through σ_b² = b/τ, which reduces to the naive s/√(b+σ_b²) only at leading order. The second mechanism is the r* statistic, a higher-order asymptotic correction to the signed likelihood-ratio root r(0), defined as r*(0) = r(0) + (1/r(0)) ln|u(0)/r(0)|, with u(0) the auxiliary statistic computed from log-likelihood derivatives; it sharpens the Gaussian approximation of t

Load-bearing premise

The load-bearing premise is that the median discovery significance is well approximated by evaluating the test statistic at the single Asimov data point (n=s+b, m=τb); the paper demonstrates this by Monte Carlo agreement (Secs. 4–5) but does not provide an analytic error bound, and Sec. 6.5 acknowledges that the r* correction only improves the significance at that fixed Asimov point, not the Asimov approximation itself. In the uncertain-background case the Monte Carlo referen

What would settle it

Enumerate exactly the joint Poisson distribution over (n,m) for a few small parameter choices (e.g., b=1, τ=1 with s=2 and s=5), compute the true median of the discovery significance using exact background-only p-values, and compare with Eq. (20) and with the r*-corrected Asimov value; a persistent discrepancy larger than the claimed low-yield accuracy would falsify the central claim. The same check should avoid using profile-construction resampling as the reference, since that is itself approximate.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Planned experiments can compute expected discovery sensitivity without toy Monte Carlo, using only s, b, and the control-region scale factor, via a closed formula that is continuous and instantly evaluable.
  • Projections based on s/√(b+σ_b²) overstate the sensitivity of low-statistics searches; cut choices and exposure decisions based on that formula may be suboptimal in exactly the regimes where experiments are background-limited.
  • The r*-corrected Asimov significance interpolates the stepwise Monte Carlo median of the significance, giving an analytic curve that can be used for smooth sensitivity scans.
  • For the known-background case, the continuity-corrected r* significance matches the exact Poisson-tail significance down to an expected background of one event, so it can serve as a fast check of exact calculations.
  • The same formalism applies directly to the astrophysical on/off problem, where the same Poisson control-measurement structure is standard.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's own conclusions flag that no canonical continuity correction exists for the two-count (n,m) discrete problem; a natural extension would be a mid-p or half-bin correction in some transformed coordinate, and the accuracy gain there is untested.
  • The Asimov approximation itself is never bounded analytically; at extremely low counts, the step structure of the median significance suggests that a direct saddlepoint or exact enumeration approach could differ from the Asimov point in ways the r* correction cannot repair.
  • Because Eq. (20) is expressed in terms of σ_b via the control measurement, it could plausibly be adapted to expected exclusion limits (not just discovery) by the same profile-likelihood machinery, although the paper does not pursue that.
  • If the higher-order corrections hold up in the low-yield regime, they could reduce the reliance on toy-Monte-Carlo sensitivity scans in current searches, since the closed forms are effectively instant and differentiable in the physics parameters.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper treats discovery significance in Poisson counting experiments, first for known background and then for an uncertain background constrained by a Poisson control measurement (the on/off problem). Using the profile likelihood ratio test statistic q0 and the Asimov data set, it derives Eq. (20), a compact expression for the median discovery significance in the uncertain-background case, and shows that this reduces to s/sqrt(b+sigma_b^2) in the limit of small s/b and small sigma_b^2/b. The second part of the paper applies the Barndorff-Nielsen r* higher-order correction, defining q0* = [max(0,r*(0))]^2, and applies it to observed and median significances, with a continuity correction in the known-background case. The authors report Monte Carlo comparisons and conclude that the new formulas improve accuracy at low event yields, especially relative to the simple s/sqrt(b+sigma_b^2) formula. The algebraic derivation from the profile likelihood to Eq. (20) is internally consistent, and the paper provides code to reproduce the figures.

Significance. If the numerical claims hold, Eq. (20) supplies a stable, citable reference for a formula already used in the literature, and the q0* correction extends the usable range of asymptotic formulae for discovery significance to very small Poisson counts. Strengths of the paper include the explicit derivation of Eq. (20), reproduction of the known-background limiting cases, and the availability of open-source code for all figures. The main qualification is that the uncertain-background validation is performed against a Monte Carlo reference built from profile construction/hybrid resampling, which is itself approximate; this limits the strength of the low-yield improvement claims for the central on/off case until an exact or better-controlled reference is supplied.

major comments (3)
  1. [Sec. 5, Figs. 2, 4, 6] The MC reference for the uncertain-background model uses the profile-construction/hybrid-resampling procedure: for each simulated (n,m), the background-only p-value is estimated by generating s=0 pseudo-data with b fixed to the profiled value bhat0=(n+m)/(1+tau) of Eq. (16). This plug-in null distribution does not propagate the nuisance-parameter uncertainty and has no guaranteed frequentist coverage, so agreement between Eq. (20)/q0* and these MC points is not an independent validation. Since the on/off model has an exact ancillary reduction under s=0 — conditional on N=n+m, n is Binomial(N, 1/(1+tau)), free of b — exact or near-exact p-values can be computed and used as a benchmark. Please add such a comparison, or at minimum report the difference between the profile-construction reference and the exact conditional-binomial reference for representative (s,b,tau) values, with MC uncerta
  2. [Sec. 6.4, Eq. (35)] The auxiliary statistic u(s) in Eq. (35), and its s=0 specialization Eq. (36), are central to the q0* correction, but they are stated without derivation; the text defers to Refs. [17-19]. Since the r* correction is the second main result of the paper, please provide a derivation in an appendix or online supplement, and state the regularity conditions under which the r* approximation is expected to hold for this discrete exponential-family model. Also clarify the limiting conventions at the boundaries n=0 or m=0: the text assigns u(0)=0 via sqrt(x) ln x -> 0, but the expressions contain ln(n tau / m), and the behavior of r*(0) at those boundary points should be stated more carefully.
  3. [Sec. 6.5, Figs. 5 and 6] The claims that q0* 'improves' agreement with the MC median and that the higher-order corrections provide 'meaningful improvements at small event yields' are supported only by visual inspection of the figures. Please quantify the improvement, e.g., mean and maximum absolute difference in Z units between the Asimov Z_A and the MC median over the plotted b range, before and after the r*/continuity corrections, ideally split into low-yield and large-yield regions. In addition, report the number of MC toys and the fraction of omitted points at the stated 5-sigma toy-resolution ceiling in Figs. 2 and 6; the omission of ceiling-reaching points can distort the apparent median in small-b regions.
minor comments (4)
  1. [Abstract and Sec. 1] The abstract's phrase 'meaningful improvements at small event yields' should be tempered until a quantitative measure is provided; consider saying 'improvements in the specific cases shown' or citing the numerical residuals.
  2. [Sec. 5, Eq. (20)] The provenance of Eq. (20) in the unpublished 2012 SLAC presentation [2] is acknowledged, but since the paper aims to provide a stable reference, it would be useful to state explicitly which of the subsequent applications [3,4] used this exact formula rather than the limiting s/sqrt(b+sigma_b^2) form.
  3. [Sec. 6.2] For the known-background continuity correction, the paper applies the half-unit shift only to q0* and not to q0, to preserve the standard first-order result. This is a reasonable convention, but it should be flagged prominently in the conclusions, where the improved performance of q0* (cc) is partly attributable to the continuity correction rather than to the r* correction alone.
  4. [Figs. 2 and 6] The left and right panels use different parameterizations (tau fixed versus sigma_b/b fixed), which is useful, but the reader must compare across rows and columns carefully; adding a small panel label such as 'tau = 0.5, 2' and 'sigma_b/b = 0.25, 1' inside each panel would improve readability.

Circularity Check

0 steps flagged

No circularity: the central formulas are derived from the Poisson model and benchmarked externally, not fitted or self-referential.

full rationale

The paper's main results are self-contained derivations from the stated statistical model, not re-statements of fitted inputs. Eq. (20) follows from the profile-likelihood-ratio test statistic for the on/off likelihood L(s,b) in Eq. (12), the large-sample identification Z=sqrt(q0), the Asimov substitution n->s+b, m->tau b, and the definitional relation sigma_b^2 = b/tau in Eq. (19). No parameter is fitted to the Monte Carlo values; the MC estimates are external checks. The higher-order q0* correction is built from the Barndorff-Nielsen r* formula in Eq. (25) with explicit auxiliary statistics u(0) in Eqs. (33) and (36), using standard asymptotic theory (refs. 7-11, 17), not the paper's target result. The known-background comparisons use an exact Poisson reference, and the uncertain-background MC reference is the profile-construction/hybrid-resampling method quoted in Sec. 5; that reference is approximate, which is a potential correctness/coverage concern but not a circularity, since the paper's curves are not fitted to those points. The self-citations to Ref. [1] (the q0/Asimov framework), Ref. [2] (where Eq. (20) appeared in an unpublished talk), and Ref. [19] (r* application) are provenance or background; the paper re-derives the needed expressions and does not rely on those citations as the load-bearing derivation. Acknowledged limitations, e.g., the absence of a unique continuity correction for two discrete counts in Secs. 6.5 and 7, weaken the accuracy claims at the lowest yields but do not make the derivation circular.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

No numerical free parameters are fitted to data; the inputs s, b, τ, and σ_b are experimental parameters. The paper's load is carried by standard asymptotic theorems, the Asimov approximation for the median, and cited r* formulas. No new physical entities are introduced.

axioms (6)
  • standard math Wilks' theorem and Wald's theorem: the profile likelihood ratio test statistic is asymptotically chi-square, so Z = sqrt(q0).
    Invoked in Secs. 4 and 5 to convert q0 into a discovery significance.
  • domain assumption The median significance is approximated by evaluating the test statistic at the Asimov data set (n=s+b, m=τb).
    Used throughout Secs. 4, 5, and 6.5; this is a known approximation from the particle-physics literature, not derived exactly in the paper.
  • domain assumption The Barndorff-Nielsen r* formula and the auxiliary statistic u(s) in Eqs. (30), (35), and (36) are correct for these models.
    The r* construction is cited to Refs. [17-19] and specialized to the Poisson counting models; the paper does not derive u(s) from first principles.
  • domain assumption The normal-tail approximation to the Poisson p-value corresponds to the mid-p-value, and the inclusive Poisson tail is obtained by evaluating the statistic at n - 1/2.
    Used in Sec. 6.2 for the known-background continuity correction.
  • domain assumption The control measurement has mean τb with τ known exactly and no signal contamination.
    The likelihood in Eq. (12) and all subsequent formulas assume a clean Poisson control region with known scale factor.
  • domain assumption Profile construction / hybrid resampling provides a reliable reference p-value for the uncertain-background model.
    Used in Sec. 5 and Fig. 4 as the Monte Carlo reference for the uncertain-background case; it is an approximate method, not an exact calculation.

pith-pipeline@v1.3.0-daily-deepseek · 11936 in / 14177 out tokens · 151246 ms · 2026-08-03T07:07:52.026631+00:00 · methodology

0 comments
read the original abstract

In Particle Physics, a search for a new signal process is often based on observing a Poisson-distributed number of events, whose mean contains contributions from background and, if it exists, the hypothesised signal. The discovery significance can be expressed as an equivalent number of standard deviations derived from the $p$-value of the background-only hypothesis. To characterise the experimental sensitivity, one may report the median, assuming a nominal signal strength, of the discovery significance. In this paper, approximate expressions for the median significance are derived both when the expected number of background events is known and when the background rate is uncertain but constrained by a Poisson control measurement. The formulae are based on a test statistic using the profile likelihood ratio, and the median significance is approximated using the Asimov data set. Higher-order asymptotic corrections, based on the Barndorff-Nielsen $r^\ast$ statistic, are incorporated for both the observed and expected discovery significance. The validity of the resulting expressions is compared with Monte Carlo results and with other formulae for expected significance often used in particle physics. The higher-order corrections are found to provide meaningful improvements at small event yields. The results are important for obtaining an accurate assessment of the sensitivity of a planned experiment and for the optimal choice of cuts that determine the expected numbers of signal and background events.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 3 canonical work pages

  1. [1]

    Cowan, G., Cranmer, K., Gross, E., Vitells, O.: Asymptotic formulae for likelihood-based tests of new physics. Eur. Phys. J. C71, 1554 (2011) https: //doi.org/10.1140/epjc/s10052-011-1554-0 arXiv:1007.1727 [physics.data-an]. Erratum: Eur. Phys. J. C 73 (2013) 2501

    Pith/arXiv arXiv 2011

  2. [2]

    Presentation at Progress on Sta- tistical Issues in Searches, SLAC National Accelerator Laboratory, Menlo Park, CA

    Cowan, G.: Two developments in tests for dis- covery: use of weighted Monte Carlo events and an improved measure of experimental sensitivity. Presentation at Progress on Sta- tistical Issues in Searches, SLAC National Accelerator Laboratory, Menlo Park, CA. 4–6 June 2012 (2012)

    2012

  3. [3]

    Kumar, N., Martin, S.P.: Vectorlike leptons at the large hadron collider. Phys. Rev. D 92, 115018 (2015) https://doi.org/10.1103/ PhysRevD.92.115018

    2015

  4. [4]

    The European Physical Journal C 81(2), 123 (2021) https://doi.org/10.1140/ epjc/s10052-020-08819-6

    Bhattiprolu, P.N., Martin, S.P., Wells, J.D.: Criteria for projected discovery and exclusion sensitivities of counting experi- ments. The European Physical Journal C 81(2), 123 (2021) https://doi.org/10.1140/ epjc/s10052-020-08819-6

    2021

  5. [5]

    Wilks, S.S.: The large-sample distribution of the likelihood ratio for testing com- posite hypotheses. Ann. Math. Statist.9, 12 60–62 (1938) https://doi.org/10.1214/aoms/ 1177732360

    doi:10.1214/aoms/ 1938

  6. [6]

    Wald, A.: Tests of statistical hypotheses con- cerning several parameters when the number of observations is large. Trans. Am. Math. Soc.54, 426–482 (1943) https://doi.org/10. 1090/S0002-9947-1943-0012401-3

    1943

  7. [7]

    Biometrika67, 293–310 (1980) https: //doi.org/10.1093/biomet/67.2.293

    Barndorff-Nielsen, O.E.: Conditionality reso- lutions. Biometrika67, 293–310 (1980) https: //doi.org/10.1093/biomet/67.2.293

    work page doi:10.1093/biomet/67.2.293 1980

  8. [8]

    Biometrika70, 343–365 (1983) https: //doi.org/10.1093/biomet/70.2.343

    Barndorff-Nielsen, O.E.: On a formula for the distribution of the maximum likelihood esti- mator. Biometrika70, 343–365 (1983) https: //doi.org/10.1093/biomet/70.2.343

    work page doi:10.1093/biomet/70.2.343 1983

  9. [9]

    Biometrika 73, 307–322 (1986) https://doi.org/10.1093/ biomet/73.2.307

    Barndorff-Nielsen, O.E.: Inference on full or partial parameters based on the standard- ized signed log likelihood ratio. Biometrika 73, 307–322 (1986) https://doi.org/10.1093/ biomet/73.2.307

    1986

  10. [10]

    Barndorff-Nielsen, O.E.: Approximate inter- val probabilities. J. R. Stat. Soc. Ser. B 52, 485–496 (1990) https://doi.org/10.1111/ j.2517-6161.1990.tb01802.x

    arXiv 1990

  11. [11]

    Biometrika78, 557–563 (1991) https://doi.org/10.1093/biomet/78.3

    Barndorff-Nielsen, O.E.: Modified signed log likelihood ratio. Biometrika78, 557–563 (1991) https://doi.org/10.1093/biomet/78.3. 557

    work page doi:10.1093/biomet/78.3 1991

  12. [12]

    Astrophys

    Li, T.-P., Ma, Y.-Q.: Analysis methods for results in gamma-ray astronomy. Astrophys. J.272, 317–324 (1983) https://doi.org/10. 1086/161295

    1983

  13. [13]

    In: Lyons, L., Karag¨ oz ¨Unel, M

    Cranmer, K.: Statistical challenges for searches for new physics at the LHC. In: Lyons, L., Karag¨ oz ¨Unel, M. (eds.) Statis- tical Problems in Particle Physics, Astro- physics and Cosmology: Proceedings of PHY- STAT05, pp. 112–123. Imperial College Press, London (2006). https://doi.org/10. 1142/9781860948985 0026

    2006

  14. [14]

    Cousins, R.D., Linnemann, J.T., Tucker, J.: Evaluation of three methods for cal- culating statistical significance when incor- porating a systematic uncertainty into a test of the background-only hypothe- sis for a Poisson process. Nucl. Instrum. Methods Phys. Res. A595, 480–501 (2008) https://doi.org/10.1016/j.nima.2008. 07.086 arXiv:physics/0702156 [phy...

    Pith/arXiv arXiv 2008

  15. [15]

    Chuang, C.-S., Lai, T.L.: Hybrid resampling methods for confidence intervals. Statist. Sinica10, 1–50 (2000)

    2000

  16. [16]

    Sen, B., Walker, M., Woodroofe, M.: On the unified method with nuisance parameters. Statist. Sinica19, 301–314 (2009)

    2009

  17. [17]

    Cambridge University Press, Cambridge (2007)

    Brazzale, A.R., Davison, A.C., Reid, N.: Applied Asymptotics: Case Studies in Small- Sample Statistics. Cambridge University Press, Cambridge (2007). https://doi.org/10. 1017/CBO9780511611131

    2007

  18. [18]

    Reid, N.: The 2000 Wald memorial lectures: asymptotics and the theory of inference. Ann. Statist.31, 1695–1731 (2003) https://doi. org/10.1214/aos/1074290325

    arXiv 2000

  19. [19]

    Canonero, E., Brazzale, A.R., Cowan, G.: Higher-order asymptotic corrections and their application to the Gamma Variance Model. Eur. Phys. J. C83, 1100 (2023) https: //doi.org/10.1140/epjc/s10052-023-12263-7 arXiv:2304.10574 [physics.data-an]

    Pith/arXiv arXiv 2023

  20. [20]

    Davison, A.C., Fraser, D.A.S., Reid, N.: Improved likelihood inference for discrete data. J. R. Stat. Soc. Ser. B68, 495–508 (2006) https://doi.org/10.1111/j.1467-9868. 2006.00548.x

    Pith/arXiv arXiv 2006

  21. [21]

    Lancaster, H.O.: Significance tests in dis- crete distributions. J. Am. Stat. Assoc. 56, 223–234 (1961) https://doi.org/10.1080/ 01621459.1961.10482105 13

    arXiv 1961