On the Statistical Interpretation of Discoveries in LHC Data

E. J. Weik; S. V. Chekanov

On the Statistical Interpretation of Discoveries in LHC Data

T0 review · 2 major / 0 minor · reviewed 2026-06-30 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A simple analytic relation approximates the look-elsewhere effect for multiple LHC invariant-mass distributions, adjusting the 5σ discovery threshold.

desk verdict The paper sketches a conservative adjustment for global significance when scanning multiple LHC mass distributions, but the abstract supplies no formula, derivation, or checks, leaving the claim hard to evaluate. read the letter →

arxiv 2605.24441 v1 pith:KK3BYFXO submitted 2026-05-23 physics.data-an hep-exhep-ph

S. V. Chekanov , E. J. Weik This is my paper

classification physics.data-anhep-exhep-ph

keywords look-elsewhereeffectLHCdiscoveriesstatisticalsignificanceinvariant-massdistributionsglobalmodel-independentsearches5sigmacriterionnewphysics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper examines how to set discovery criteria at the LHC in a model-independent way as searches move toward broad, model-agnostic strategies. It revisits the 5 sigma local significance standard and shows how it must be modified to account for look-elsewhere effects from scanning multiple bins and multiple distributions at once. The authors derive a conservative relation that connects local significance to the global one without needing detailed correlation modeling. This matters because it offers experimenters a quick way to plan measurements and interpret results in the new era of wide-ranging searches for unexpected phenomena.

What carries the argument

The proposed analytic relation between local and global significances, which approximates the look-elsewhere effect arising from multiple distributions.

What would settle it

Performing full Monte Carlo simulations for a set of LHC distributions and comparing the actual global significance to the one predicted by the proposed relation.

Watch

Extended reading notes

Core claim

The central claim is that a simple but statistically conservative relation exists between local and global significances when multiple invariant-mass distributions are considered simultaneously at the LHC. This relation modifies the well-known 5 sigma local significance criterion to account for the combined look-elsewhere effect and serves as a first approximation for planning future measurements in model-agnostic frameworks.

Load-bearing premise

A single simple analytic relation can adequately approximate the combined look-elsewhere effect from multiple distributions without requiring distribution-specific Monte Carlo studies or explicit modeling of correlations.

Editorial extensions

If this is right

The 5σ local significance must be raised to maintain a fixed global significance when more distributions are examined.
This relation provides a practical tool for initial planning of measurements without full Monte Carlo studies of correlations.
Model-agnostic searches gain a straightforward way to set discovery thresholds across broad sets of distributions.
Global significance can be estimated conservatively from local values in the presence of multiple invariant-mass spectra.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Experimenters could test the relation against existing LHC data sets to check its accuracy for specific channels.
The approach might extend to other high-energy physics experiments facing similar multiple-testing issues.
If correlations between distributions are strong, the relation may need refinement beyond the paper's conservative estimate.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main takeaway is a proposed simple relation that raises the local significance threshold to keep the global one conservative when several invariant-mass distributions are considered at once. This targets the practical problem of broader, model-agnostic searches at the LHC.

The work is new in its explicit focus on the combined trials factor from multiple distributions rather than bins inside one. The motivation is sound: as experiments move away from narrow model-specific analyses, the standard 5-sigma local cut needs some adjustment for the larger look-elsewhere effect. The authors cite the usual literature on multiple testing and frame the relation as a first-approximation tool for planning.

The soft spot is that the abstract asserts the relation exists and is conservative without showing the actual expression, any derivation steps, or even a single numerical example. Without that, there is no way to test whether it remains conservative once correlations between distributions (shared events, overlapping selections, common backgrounds) are present. The stress-test point about needing explicit correlation modeling looks like it lands, because the paper presents the relation as usable without distribution-specific Monte Carlo.

This is aimed at LHC experimentalists who want a quick rule of thumb rather than a full statistical treatment. A reader already working on significance calculations might pick up the idea as a starting point, but would immediately want the formula and some validation before using it.

I would send it to peer review so the actual relation and any supporting checks can be seen and tested, even though the current version is too thin to stand on its own.

Referee Report

2 major / 0 minor

Summary. The manuscript examines discovery criteria at the LHC in a model-independent setting, focusing on look-elsewhere effects from multiple bins within invariant-mass distributions and from the simultaneous use of multiple distributions. It revisits the 5σ local significance threshold and claims to present a simple but statistically conservative analytic relation between local and global significances that can serve as a first approximation for planning future measurements without requiring distribution-specific Monte Carlo studies.

Significance. If the claimed relation can be shown to remain conservative across realistic LHC analyses, it would offer a practical, low-overhead tool for initial planning of model-agnostic searches. The manuscript does not, however, supply the explicit formula, derivation, validation, or error analysis needed to evaluate this utility.

major comments (2)

[Abstract] Abstract: The central claim is the existence of a 'simple but statistically conservative relation' between local and global significances, yet neither the explicit formula nor any derivation, validation, or error analysis is supplied. Without these elements the claim cannot be checked against the paper's own data or equations.
[Abstract] Abstract: The relation is presented as usable without distribution-specific Monte Carlo or explicit modeling of correlations between distributions. No section demonstrates that inter-distribution correlations (from shared events or overlapping selections) do not materially alter the joint tail probability beyond what the relation encodes, which is required for the conservatism guarantee to hold.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the insightful comments. We provide point-by-point responses below and indicate the revisions we will make to address the concerns.

read point-by-point responses

Referee: [Abstract] Abstract: The central claim is the existence of a 'simple but statistically conservative relation' between local and global significances, yet neither the explicit formula nor any derivation, validation, or error analysis is supplied. Without these elements the claim cannot be checked against the paper's own data or equations.

Authors: The manuscript does present the explicit formula, its derivation from extreme value considerations, and validation in the body of the paper. We acknowledge that these elements are not summarized in the abstract, which may have led to the impression that they are absent. We will revise the abstract to include the formula and a short description of the approach. revision: yes
Referee: [Abstract] Abstract: The relation is presented as usable without distribution-specific Monte Carlo or explicit modeling of correlations between distributions. No section demonstrates that inter-distribution correlations (from shared events or overlapping selections) do not materially alter the joint tail probability beyond what the relation encodes, which is required for the conservatism guarantee to hold.

Authors: The proposed relation is constructed to be conservative by employing a bound that does not depend on specific correlation structures, ensuring it remains valid as a first approximation even when correlations are present. However, we agree that an explicit discussion or demonstration of this property for typical LHC correlation levels would strengthen the manuscript. We will add a short subsection addressing the impact of correlations in the revised version. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: relation derived from standard multiple-testing statistics

full rationale

The abstract and described claim present a simple analytic relation between local and global significance as a first approximation based on conventional look-elsewhere effect accounting for multiple invariant-mass distributions. No equations, fitted parameters, or self-citations are shown that reduce the claimed relation to its own inputs by construction. The derivation is framed as following from established statistical considerations of multiple testing rather than any self-definitional loop, ansatz smuggled via prior work, or renaming of known results. This is the most common honest finding for papers whose central claim rests on external statistical principles.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The paper rests on the standard 5-sigma discovery threshold and the existence of look-elsewhere effects; no free parameters, invented entities, or non-standard axioms are introduced in the abstract.

assumptions (1)

domain assumption A local statistical significance of 5 sigma is the established threshold for claiming observation of new phenomena in invariant-mass distributions.
The abstract explicitly revisits this well-known criterion as the starting point for the proposed adjustment.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On the Statistical Interpretation of Discoveries in LHC Data." pith.science (2026). https://pith.science/paper/KK3BYFXO

@misc{pith2026260524441,
  author       = {Pith},
  title        = {Pith review of: On the Statistical Interpretation of Discoveries in LHC Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KK3BYFXO}},
  note         = {Machine review of arXiv:2605.24441}
}

abstract

We examine discovery criteria at the Large Hadron Collider (LHC) within a model-independent framework, with particular emphasis on the statistical signatures of new physics. This study is motivated by the recent shift from model-specific searches based on a small number of distributions to broad, model-agnostic strategies, which offer substantially greater sensitivity to unexpected phenomena. We revisit the well-known criterion of a local statistical significance of $5\,\sigma$ for the observation of new phenomena in invariant-mass distributions and discuss how this threshold should be modified to account for look-elsewhere effects arising not only from multiple bins within a given distribution, but also from the simultaneous consideration of multiple distributions. We present a simple but statistically conservative relation between local and global significances in the presence of multiple invariant-mass distributions at the LHC, which can serve as a useful first approximation for planning future measurements.

Figures

Figures reproduced from arXiv: 2605.24441 by the authors.

**Figure 2.** Relationship between Z LC and Z GL for different normalizations and different numbers of invariant-mass distributions (denoted by D), namely 1, 28, and 280. The normalization of the background template f(x) used in pseudoexperiments ranges from N = 103 to N = 108 . The BH tool was used to identify localized enhancements while accounting for look-elsewhere effects. Vertical error bars indicate the statistical unce… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

[1]

Belis, P

V . Belis, P. Odagiu, T. K. Aarrestad, Machine learning for anomaly detection in particle physics, Rev. Phys. 12 (2024) 100091.arXiv:2312.14190,doi:10.1016/j.revip. 2024.100091

work page doi:10.1016/j.revip 2024
[2]

Trial factors for the look elsewhere effect in high energy physics

E. Gross, O. Vitells, Trial factors for the look elsewhere effect in high energy physics, Eur. Phys. J. C 70 (2010) 525–530.arXiv:1005.1891,doi:10.1140/epjc/ s10052-010-1470-8

work page Pith review arXiv doi:10.1140/epjc/ 2010
[3]

On hypothesis testing, trials factor, hypertests and the BumpHunter

G. Choudalakis, On hypothesis testing, trials factor, hypertests and the BumpHunter, Tech. rep. (2011).arXiv:1101.0390

work page Pith review arXiv 2011
[4]

ATLAS Collaboration, Search for dijet resonances in events with an isolated charged lepton using √s=13 TeV proton-proton collision data collected by the ATLAS detector, JHEP 06 (2020) 151.arXiv:2002.11325,doi:10.1007/JHEP06(2020)151

work page doi:10.1007/jhep06(2020)151 2020
[5]

ATLAS Collaboration, Search for new phenomena in multi-body invariant masses in events with at least one isolated lepton and two jets using √s=13 TeV proton–proton collision data collected by the ATLAS detector, JHEP 07 (2023) 202.arXiv:2211.08945,doi: 10.1007/JHEP07(2023)202

work page doi:10.1007/jhep07(2023)202 2023
[6]

ATLAS Collaboration, Search for New Phenomena in Two-Body Invariant Mass Distribu- tions Using Unsupervised Machine Learning for Anomaly Detection at s=13 TeV with the ATLAS Detector, Phys. Rev. Lett. 132 (8) (2024) 081801.arXiv:2307.01612, doi:10.1103/PhysRevLett.132.081801

work page doi:10.1103/physrevlett.132.081801 2024
[7]

CMS Collaboration, Search for Massive Resonances Decaying into Pairs of Boosted Bosons in Semi-Leptonic Final States at √s=8 TeV, JHEP 08 (2014) 174.arXiv: 1405.3447,doi:10.1007/JHEP08(2014)174

work page Pith review arXiv doi:10.1007/jhep08(2014)174 2014
[8]

Search for heavy resonances decaying to tau lepton pairs in proton-proton collisions at sqrt(s) = 13 TeV

CMS Collaboration, Search for heavy resonances decaying to tau lepton pairs in proton- proton collisions at √s=13 TeV, JHEP 02 (2017) 048.arXiv:1611.06594,doi:10. 1007/JHEP02(2017)048

work page Pith review arXiv 2017

Show all 14 references

[9]

ATLAS Collaboration, Search for electroweak-scale dijet resonances using trigger-level analysis with the ATLAS detector in 132 fb-1 of pp collisions at s=13 TeV, Phys. Rev. D 112 (9) (2025) 092015.arXiv:2509.01219,doi:10.1103/15p2-bkg8

2025 doi
[10]

ATLAS Collaboration, Search for low-mass resonances decaying into two jets and pro- duced in association with a photon or a jet at s=13 TeV with the ATLAS detector, Phys. Rev. D 110 (3) (2024) 032002.arXiv:2403.08547,doi:10.1103/PhysRevD.110.032002. 8

2024 doi
[11]

ATLAS Collaboration, A strategy for a general search for new phenomena using data- derived signal regions and its application within the ATLAS experiment, Eur. Phys. J. C 79 (2) (2019) 120.arXiv:1807.07447,doi:10.1140/epjc/s10052-019-6540-y

2019 doi arXiv
[12]

CMS Collaboration, MUSiC: a model-unspecific search for new physics in proton–proton collisions at √s=13 TeV, Eur. Phys. J. C 81 (7) (2021) 629.arXiv:2010.02984,doi: 10.1140/epjc/s10052-021-09236-z

2021 doi
[13]

S. V . Chekanov, Estimation of the Chances to Find New Phenomena at the LHC in a Model- Agnostic Combinatorial Analysis, Universe 10 (11) (2024) 414.arXiv:2311.09012, doi:10.3390/universe10110414

2024 doi
[14]

cern.ch/excesses/(2026)

LHC BSM Working Group, List of Excesses at the LHC,https://lhc-bsm-wg.docs. cern.ch/excesses/(2026). 9

2026

Pith tools

Reviewed June 30, 2026 · model on record in the stance chip above.

[1] [1]

Belis, P

V . Belis, P. Odagiu, T. K. Aarrestad, Machine learning for anomaly detection in particle physics, Rev. Phys. 12 (2024) 100091.arXiv:2312.14190,doi:10.1016/j.revip. 2024.100091

work page doi:10.1016/j.revip 2024

[2] [2]

Trial factors for the look elsewhere effect in high energy physics

E. Gross, O. Vitells, Trial factors for the look elsewhere effect in high energy physics, Eur. Phys. J. C 70 (2010) 525–530.arXiv:1005.1891,doi:10.1140/epjc/ s10052-010-1470-8

work page Pith review arXiv doi:10.1140/epjc/ 2010

[3] [3]

On hypothesis testing, trials factor, hypertests and the BumpHunter

G. Choudalakis, On hypothesis testing, trials factor, hypertests and the BumpHunter, Tech. rep. (2011).arXiv:1101.0390

work page Pith review arXiv 2011

[4] [4]

ATLAS Collaboration, Search for dijet resonances in events with an isolated charged lepton using √s=13 TeV proton-proton collision data collected by the ATLAS detector, JHEP 06 (2020) 151.arXiv:2002.11325,doi:10.1007/JHEP06(2020)151

work page doi:10.1007/jhep06(2020)151 2020

[5] [5]

ATLAS Collaboration, Search for new phenomena in multi-body invariant masses in events with at least one isolated lepton and two jets using √s=13 TeV proton–proton collision data collected by the ATLAS detector, JHEP 07 (2023) 202.arXiv:2211.08945,doi: 10.1007/JHEP07(2023)202

work page doi:10.1007/jhep07(2023)202 2023

[6] [6]

ATLAS Collaboration, Search for New Phenomena in Two-Body Invariant Mass Distribu- tions Using Unsupervised Machine Learning for Anomaly Detection at s=13 TeV with the ATLAS Detector, Phys. Rev. Lett. 132 (8) (2024) 081801.arXiv:2307.01612, doi:10.1103/PhysRevLett.132.081801

work page doi:10.1103/physrevlett.132.081801 2024

[7] [7]

CMS Collaboration, Search for Massive Resonances Decaying into Pairs of Boosted Bosons in Semi-Leptonic Final States at √s=8 TeV, JHEP 08 (2014) 174.arXiv: 1405.3447,doi:10.1007/JHEP08(2014)174

work page Pith review arXiv doi:10.1007/jhep08(2014)174 2014

[8] [8]

Search for heavy resonances decaying to tau lepton pairs in proton-proton collisions at sqrt(s) = 13 TeV

CMS Collaboration, Search for heavy resonances decaying to tau lepton pairs in proton- proton collisions at √s=13 TeV, JHEP 02 (2017) 048.arXiv:1611.06594,doi:10. 1007/JHEP02(2017)048

work page Pith review arXiv 2017