Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Machine-learning anomaly searches that train and test on the same data produce badly miscalibrated p-values, and k-fold cross-validation is the practical compromise that restores calibration while keeping most of the sensitivity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:21 UTC pith:WP56HGI3

load-bearing objection Solid empirical study of p-value calibration in weakly supervised AD; the k-fold balance claim is real but only measured at k=5 and needs a caveat. the 3 major comments →

arxiv 2512.13787 v2 pith:WP56HGI3 submitted 2025-12-15 hep-ph hep-exphysics.data-an

Look everywhere effects in anomaly detection

Marie Hein , Benjamin Nachman , David Shih This is my paper
classification hep-ph hep-exphysics.data-an
keywords look everywhere effectanomaly detectionp-value calibrationk-fold cross-validationweak supervisionoverfittinglikelihood ratio classifiercollider searches
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that a 'look everywhere effect'—an anomaly detector searching the whole feature space at once—dilutes the statistics of weakly supervised new-physics searches just as the classical look-elsewhere effect dilutes fixed-region searches. Because the detector's choice of where to look is itself adapted to the data, quoting a p-value on the same data used for training is systematically overconfident; the more the detector is allowed to overfit fluctuations, the worse the calibration. Training on one half of the data and testing on the other gives perfectly calibrated p-values but throws away sensitivity, while k-folding (train on k-1 parts, test on the remainder, repeat) lands in between: only mildly miscalibrated and nearly as sensitive as using all data for both. The authors demonstrate this with a 2D Gaussian toy model and a collider benchmark dataset mimicking LHC dijet events, using neural networks and boosted decision trees.

Core claim

The central claim is that in weakly supervised anomaly detection, the standard practice of evaluating the chosen signal region or classifier on the same data that was used to choose it yields p-values that are not what they appear to be: the selection step has already looked everywhere and picked the largest fluctuation, so the reported local p-value is a minimum over many trials. The paper shows that this 'look everywhere effect' is quantitatively equivalent to overfitting: regularization like early stopping reduces the miscalibration, and the effect is stronger for boosted decision trees than for neural networks because the BDT's bin values are derived directly from the training set. The p

What carries the argument

The object doing the work is the anomaly score R(x) = p_SR(x) / p_BT(x), the likelihood ratio between signal-region data and a background template, which is estimated by binning or by a classifier. The paper's statistical engine is the comparison of three data-usage protocols—train/test on the same set, train on half and test on half, and k-fold cross-validation with k=5—together with calibration fits that map calculated p-values to empirical p-values. The crucial mechanism is the 'look everywhere effect': because the region or classifier is selected by maximizing the observed excess, the p-value is conceptually already a minimum over all places searched, and this selection-induced bias is e

Load-bearing premise

The k-fold p-value calculation pools counts across folds as though the folds were independent, yet the paper's own two-fold illustration shows large fluctuations can persist across folds; the claimed balance between calibration and sensitivity depends on this dependence being mild for k=5 and the tested benchmark, and that is not established more generally.

What would settle it

Repeat the k-fold calibration measurement on the Gaussian toy with k=10 and with a much narrower injected signal (so that the same bin is likely to be re-selected in every fold); if the empirical p-value CDF's tail becomes as miscalibrated as the train-on-train curve, the independence assumption underpinning k-fold pooling fails in exactly the regime where the balance claim is needed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reporting a p-value computed on the training data without a trials-factor correction will overstate significance in any look-everywhere search; calibration is not optional if that protocol is used.
  • A search that trains on half the data and tests on the other half is automatically calibrated and requires no trials factor, at the price of reduced sensitivity to small signals.
  • Pooled-count k-fold cross-validation with k=5 is nearly calibrated and retains most of the sensitivity of the full-data approach, making it a reasonable default for experimental searches, with a residual miscalibration that can be corrected.
  • Early stopping regularizes away part of the look-everywhere effect for neural networks, but it can diminish sensitivity because the network may learn the signal only after the early-stopping point; the same fix does not reliably rescue boosted decision trees.
  • Because even small changes in analysis setup (e.g., switching classifier type) can drastically change p-value calibration, any deployment should validate calibration in settings as close as possible to the final analysis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The optimal number of folds is left open by the paper; a natural test is whether increasing k improves tail calibration enough to beat the independent-test-set sensitivity, or whether the growing overlap of training sets makes things worse.
  • The paper's equivalence between overfitting and the look-everywhere effect suggests a broader principle: any intervention that reduces generalization error—architecture changes, pretraining, data denoising—should also improve p-value calibration, which could be verified directly on the same benchmarks.
  • The calibration fits used to rescue the k-fold p-values are fit to the median and extrapolated with a linear tail; in a real search one would want a calibration method with explicit uncertainty bands, since the k-fold residual miscalibration is itself statistics-limited.
  • All numerical examples assume a known background template; with a data-driven background estimate, the ranking of the three protocols could shift, and that is a direct next step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies the statistical effects of 'looking everywhere' in weakly supervised anomaly detection. Using a 2D Gaussian toy and the LHC Olympics R&D dataset, it compares three data-usage strategies: training and evaluating on the same set, training on half and testing on the other half, and k-fold cross-validation (k=5). The authors show that evaluating p-values on the training set leads to severe miscalibration (a 'look everywhere effect' analogous to the look-elsewhere effect), while independent test sets give calibrated p-values but lower sensitivity because less data is used for training. They then examine k-fold as a compromise, finding that it yields relatively limited miscalibration while preserving most of the sensitivity. They also connect the miscalibration to overfitting, showing that early stopping can improve calibration for neural networks but not for BDTs. Finally, they use calibration fits to correct p-values and compare the sensitivity of the three strategies as a function of injected signal.

Significance. If the findings are correct, the paper has immediate practical importance: it warns against quoting p-values computed on the training set without an explicit trials-factor correction, and it identifies k-fold cross-validation as a practical design choice that balances calibration and sensitivity. The empirical characterization is extensive, covering two classifiers (NN and BDT), several background-efficiency thresholds, and a realistic collider benchmark. The authors are transparent about the limitations of their study, provide the code, and explicitly note where their conclusions are extrapolations. The connection between the look-everywhere effect and overfitting is conceptually valuable and may generalize beyond anomaly detection. However, the central 'balance' claim rests on a small set of configurations and on calibration fits that are not rigorously validated.

major comments (3)
  1. [Sec. II, strategy (c); Sec. IV A 1; Fig. 2] The k-fold p-value construction combines counts across the k folds into a single p-value. This is only valid if the fold results are independent, but the manuscript itself demonstrates that this independence is violated: Fig. 2 shows that for large excesses the same bin is selected across folds, so the combined p-value effectively reduces to the training-set p-value. The authors acknowledge this and measure the residual miscalibration at k=5 for the two datasets, but provide no quantitative study of how this residual scales with k, dataset size, feature dimensionality, or classifier. Since the abstract and Sec. IV B claim that k-folding 'strikes an effective balance' between calibration and sensitivity, this is a load-bearing generality. Either a more systematic scan over k and N (or an analytic bound) should be provided, or the conclusion should be explicitly restricted to the tested co
  2. [Appendix B; Fig. 5] The corrected p-values in Fig. 5, which are the quantitative evidence for the sensitivity comparison, are obtained from calibration fits that the authors themselves describe as 'not evaluated rigorously' (App. B). The fit is a spline plus linear tail with an arbitrary switch at p=10^-2 and an extrapolation limit at min(p_empirical)/10; no uncertainties on the calibrated p-values are propagated. This is a load-bearing issue because the conclusion that k-fold with calibration recovers sensitivity depends on these corrected values. I request at least a coverage test on independent background-only samples to validate the calibration, or a clear statement that Fig. 5 is indicative rather than a precise statistical comparison.
  3. [Sec. V] The statement that 'results are expected to carry over to both other background estimation methods and to other anomaly detection methods' is an unvalidated extrapolation. The paper has only demonstrated the effects for the CWoLa-like setup with a known background template. This extrapolation is not inherently wrong, but it should be flagged as a conjecture, not a conclusion, especially given that the BDT behavior already differs qualitatively from the NN behavior within the same framework.
minor comments (4)
  1. [Sec. III B 1] For the calibration tests, the independent-test-set case (b) is implemented by reusing the full 25k-event training and applying to a second independent 25k sample. This is a sensible computational shortcut for the null-hypothesis calibration, but the text could state explicitly that this does not reflect the training-statistics penalty that a fixed total dataset would incur. The signal-sensitivity tests (Sec. III B 2) do use a proper split, so the procedure is internally consistent.
  2. [Sec. IV A 2] The argument that early stopping reduces the look-everywhere effect for NNs but not for BDTs is plausible and consistent with the data, but the explanation is largely qualitative. A quantitative measure of 'overtraining' (e.g., the gap between train and validation losses) would strengthen the connection.
  3. [Fig. 3, App. B] The calibration fits are shown as dashed lines, but the text does not report the fitted parameters or any goodness-of-fit statistic. Since the fits are used later, providing at least the functional form and the number of data points used would improve reproducibility.
  4. [Throughout] Typos and minor wording issues: in Sec. IV B, 'the same trends' should likely be 'the same trend'; in Sec. V, the sentence 'An analysis should always be calibrated' is a bit unclear — the meaning is that calibration should be verified, not that all analyses must use calibrated p-values. Also, the 'Note added' is fine, but it would be useful to cite Ref. [41] as 'in preparation' in the main text if it is mentioned only in the note.

Circularity Check

0 steps flagged

No circularity: the paper's claims are empirical characterizations tested on external toy and collider benchmarks, not derivations that assume their own conclusions.

full rationale

The paper is an empirical study of p-value calibration and sensitivity in anomaly detection. Its central claims—that training and testing on the same data yields miscalibrated p-values, that an independent test set gives calibrated p-values, and that k-folding balances calibration and sensitivity—are direct measurements on simulated Gaussian toy data and the LHCO R&D dataset, not conclusions derived from the assumptions they purport to test. The p-value procedures are defined from binomial and negative-binomial counting in Sec. II and App. A, and the calibration curves in App. B are fitted to background-only calibration data and then applied to signal-injected p-values; this is an external calibration, not a fit of the final prediction. The k-fold balance result is an empirical finding supported by Figs. 1, 3, 4, and 5, with explicit caveats in footnote 4 and App. B that the calibration fits were not rigorously evaluated and the optimal k was not studied. Self-citations such as Refs. [12], [22], and [40] provide datasets, feature choices, and explanatory context about early stopping, but none of these citations carries the paper's central claim or substitutes for the numerical experiments. There is no self-definitional construction, no fitted input relabeled as a prediction, and no load-bearing self-citation chain. The acknowledged limitations are scope and correctness concerns, not circularity.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

The central claims rest on a small number of explicit modeling choices: exact knowledge of the background distribution, the CWoLa mixture form, and the assumption of fold independence in the k-fold p-value combination. No new physical entities are introduced, and no constants are fitted to claim a prediction; the free parameters are analysis/config choices (binning, working points, k, architecture) and calibration-fit policy.

free parameters (7)
  • Number of bins per dimension (5x5 grid on [-2,2]^2 for toy) = 5 bins/dim
    Binning choice for the toy binned analysis; miscalibration magnitude depends on this granularity (Sec. IV A).
  • Background efficiency thresholds ϵ_B = 10^-1, 10^-2 (toy), plus 10^-3 (LHCO)
    Choice of classifier cut working points used for p-values; tighter cuts reduce observed miscalibration (Sec. IV A 1).
  • Number of folds k = 5
    k-fold CV setup; authors note optimal k not studied (footnote 4).
  • NN architecture/hyperparameters = 3x64 MLP, ReLU, Adam lr=1e-3, batch 128, max 50 epochs, early stopping patience 10
    Chosen architecture; early stopping is itself a studied regularizer.
  • Calibration-fit switch/extrapolation parameters = spline→linear switch at 10^-2; extrapolate to min(empirical)/10
    Arbitrary but stated choices in the calibration fits (App. B).
  • Signal efficiency uncertainty cutoff for SIC = 20% relative background error
    Statistics cutoff for sensitivity metric (Sec. III B).
  • LHCO selections = m_JJ in [3.3, 3.7] TeV; pT>1.2 TeV; m_J1, Δm_J, τ21 features
    Reuses standard LHCO R&D selections/features from Refs [22,31]; not fitted here.
axioms (6)
  • domain assumption Background distribution p_B(x) is known exactly; only statistical fluctuations of finite SR/BT samples are considered.
    Stated at end of Sec. I: 'we assume that the background distribution is known, both for choosing the event selection and for determining p-values.' Real searches estimate backgrounds from data/simulation; bias could change calibration/sensitivity tradeoff.
  • domain assumption Signal-region data follows the two-component mixture p_SR = f_S p_S + (1-f_S) p_B (CWoLa).
    Sec. II, Eq. (1); standard weakly-supervised assumption that signal is an additive overdensity.
  • ad hoc to paper The k-fold p-value combining counts across folds assumes fold independence.
    Sec. II(c) states 'all of these assume the k results to be independent'; the paper's own Fig. 2 shows this independence is violated, which is the mechanism of residual miscalibration.
  • domain assumption For the binned toy analysis, the background template has infinite statistics.
    Sec. III B: 'for the binned procedure... we use infinite statistics.'
  • standard math Binomial (binned) and negative-binomial (classifier) p-values are the correct test statistics under the null.
    App. A; Bayesian marginalization over λ with conjugate Gamma prior yields negative binomial; this is a standard statistical model.
  • domain assumption LHCO R&D PYTHIA/Delphes simulation is a representative collider benchmark.
    Benchmark from Ref [31]; external reference, not a physical assumption of the method.

pith-pipeline@v1.3.0-alltime-deepseek · 13694 in / 14573 out tokens · 115138 ms · 2026-08-03T16:21:35.596582+00:00 · methodology

0 comments
read the original abstract

Machine learning-based anomaly detection methods are able to search high-dimensional spaces for hints of new physics with much less theory bias than traditional searches. However, by searching in many directions all at once, the statistical power of these search strategies is diluted by a variant of the look elsewhere effect. We examine this challenge in detail, focusing on weakly supervised methods. We find that training and testing on the same data results in badly miscalibrated $p$-values due to the anomaly detector searching everywhere in the data and overfitting on statistical fluctuations. However, if these $p$-values can be calibrated, they may offer the best sensitivity to anomalies, since this approach uses all of the data. Conversely, training on half of the data and testing on the other half results in perfectly calibrated $p$-values, but at the cost of reduced sensitivity to anomalies. Similarly, regularization methods such as early stopping can help with $p$-value calibration but also possibly at the expense of sensitivity. Finally, we find that k-folding strikes an effective balance between calibration and sensitivity. Our findings are supported by numerical studies with Gaussian random variables as well as from collider physics using the LHC Olympics benchmark anomaly detection dataset.

Figures

Figures reproduced from arXiv: 2512.13787 by Benjamin Nachman, David Shih, Marie Hein.

Figure 1
Figure 1. Figure 1: FIG. 1. Calibration curves showing empirical cumulative probability and calculated [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FIG. 2. Illustration of the miscalibration seen in k-fold cross [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: FIG. 3. Calibration curves and fits showing empirical cumulative probability and calculated [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: FIG. 4. Maximum SIC as a function of the signal injection for classifiers trained on the statistics retained when evaluating on [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: FIG. 5. Uncorrected (dashed lines) and corrected (solid lines) [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Kitchen Sink Anomaly Detection

    hep-ph 2026-04 unverdicted novelty 5.0

    A combined kitchen sink observable set of Energy Flow Polynomials and subjettiness variables outperforms standard baselines in sensitivity to a wide range of resonant signals, with new public benchmarks released and a...

Reference graph

Works this paper leans on

50 extracted references · 30 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Neural Network For the NN classifier, we use a simple three layer multi- layer perceptron (MLP) implemented inTensorflow[28] with 64 nodes each and ReLU activation trained using an Adamoptimizer [29] with a learning rate of 10−3. We train this classifier with a batch size of 128 for a maximum of 50 epochs either without early stopping and a validation set...

  2. [2]

    We use default hyperparameters including early stop- ping with a patience of ten epochs, where we revert back to the best model

    Boosted Decision Tree For the BDT classifier, we use the HistGradientBoostingClassifierfromscikit-learn [30], a gradient boosted decision tree classifier, which histograms the classification features before training. We use default hyperparameters including early stop- ping with a patience of ten epochs, where we revert back to the best model. We ensemble...

  3. [3]

    For both binned and ML methods, we sampled 25,000 events in the SR to serve as the data

    Toy data We use a 2d Gaussian with meanµ= 0 and standard deviationσ= 1 as an easily sampled from toy distribu- tion. For both binned and ML methods, we sampled 25,000 events in the SR to serve as the data. For case (a) in Sec. II (training and evaluation on the same set), the methods were applied directly to this data. For case (b) (training and evaluatio...

  4. [4]

    This data set contains a QCD dijet background andW ′ resonance signal atm W ′ = 3.5 TeV, which decays intoX(→qq) and Y(→qq) with masses mX = 100 GeV andm Y = 500 GeV, respectively

    LHCO data set We use a data set based on the R&D set [31] from the LHC Olympics 2020 [1] as both a more realistic test of the statistical procedures and as a test of the signal sensitivity of the different classifier-based setups and data usage procedures. This data set contains a QCD dijet background andW ′ resonance signal atm W ′ = 3.5 TeV, which decay...

    2020

  5. [5]

    training

    Comparing analysis strategies We start with a brief discussion of the simple binned anomaly detection method on the 2d toy Gaussian data, to gain intuition. We divide the data into five bins per dimension between−2 and 2, which results in a total of 25 bins. We then pick the bin with the highest over- density score on the training set. The results are sho...

  6. [6]

    When optimizing an analy- sis, the ideal bin selection or weight configuration in a NN would rely exclusively on the distributions underlying the data

    Connection between the look-everywhere effect and overfitting While we have seen that neural networks experience a look everywhere effect through their training procedure, we now aim to draw a parallel between the look every- where effect and overfitting. When optimizing an analy- sis, the ideal bin selection or weight configuration in a NN would rely exc...

  7. [7]

    Binned analysis In the binned analysis, the following values charac- terize the result: the total number of signal region eventsN SR, the number of signal region event falling into the selected binN SR,bin and the expected fraction of background events in the selected binp exp,bin. In the background-only case,N SR,bin follows a binomial distri- bution wit...

  8. [8]

    Taken in isolation, bothN ′ BT and N ′ SR follow a Poisson distribution with the common ex- pectation valueλ

    Classifier analysis For the classifier, the p-value depends on the number of signal region events selected by the classifierN ′ SR and the number of background template events selected by the classifierN ′ BT. Taken in isolation, bothN ′ BT and N ′ SR follow a Poisson distribution with the common ex- pectation valueλ. However,λis unknown and can only be e...

  9. [9]

    Kasieczkaet al., Rept.Prog.Phys.84, 124201 (2021), arXiv:2101.08320 [hep-ph]

    G. Kasieczkaet al., Rept.Prog.Phys.84, 124201 (2021), arXiv:2101.08320 [hep-ph]

    Pith/arXiv arXiv 2021

  10. [10]

    Aarrestadet al., SciPost Phys.12, 043 (2021), arXiv:2105.14027 [hep-ph]

    T. Aarrestadet al., SciPost Phys.12, 043 (2021), arXiv:2105.14027 [hep-ph]

    Pith/arXiv arXiv 2021

  11. [11]

    Karagiorgi, G

    G. Karagiorgi, G. Kasieczka, S. Kravitz, B. Nach- man, and D. Shih, Nature Rev. Phys.4, 399 (2022), arXiv:2112.03769 [hep-ph]

    Pith/arXiv arXiv 2022

  12. [12]

    Belis, P

    V. Belis, P. Odagiu, and T. K. ˚Arrestad, Rev.Phys.12, 100091 (2023), arXiv:2312.14190 [physics.data-an]

    Pith/arXiv arXiv 2023

  13. [13]

    Aadet al.(ATLAS), Phys

    G. Aadet al.(ATLAS), Phys. Rev. Lett.125, 131801 (2020), arXiv:2005.02983 [hep-ex]

    Pith/arXiv arXiv 2020

  14. [14]

    CMS Collaboration, Rept.Prog.Phys.88, 067802 (2024), arXiv:2412.03747 [hep-ex]

    Pith/arXiv arXiv 2024

  15. [15]

    Aadet al.(ATLAS), Phys

    G. Aadet al.(ATLAS), Phys. Rev. D112, 072009 (2025), arXiv:2502.09770 [hep-ex]

    arXiv 2025

  16. [16]

    Gambhir, R

    R. Gambhir, R. Mastandrea, B. Nachman, and J. Thaler, Phys.Rev.Lett.135, 021902 (2025), arXiv:2502.14036 [hep-ph]

    Pith/arXiv arXiv 2025

  17. [17]

    Gross and O

    E. Gross and O. Vitells, Eur. Phys. J. C70, 525 (2010), arXiv:1005.1891 [physics.data-an]

    Pith/arXiv arXiv 2010

  18. [18]

    A. M. Sirunyanet al.(CMS), Eur. Phys. J. C81, 629 (2021), arXiv:2010.02984 [hep-ex]

    Pith/arXiv arXiv 2021

  19. [19]

    ATLAS Collaboration, ATLAS-CONF-2014-006 (2014)

    2014

  20. [20]

    Finke, M

    T. Finke, M. Hein, G. Kasieczka, M. Kr¨ amer, A. M¨ uck, P. Prangchaikul, T. Quadfasel, D. Shih, and M. Sommerhalder, Phys.Rev.D109, 034033 (2023), arXiv:2309.13111 [hep-ph]

    Pith/arXiv arXiv 2023

  21. [21]

    Freytsis, M

    M. Freytsis, M. Perelstein, and Y. C. San, JHEP02, 220 (2023), arXiv:2310.13057 [hep-ph]

    Pith/arXiv arXiv 2023

  22. [22]

    Mikuni and B

    V. Mikuni and B. Nachman, Phys.Rev.D111, 054015 (2025), arXiv:2502.14652 [hep-ph]

    arXiv 2025

  23. [23]

    Bhimji, C

    W. Bhimji, C. Harris, V. Mikuni, and B. Nachman, (2025), arXiv:2510.24066 [hep-ph]

    arXiv 2025

  24. [24]

    C. L. Cheng, G. Singh, and B. Nachman, Phys.Rev.Lett. 135, 021801 (2024), arXiv:2405.08889 [hep-ph]

    Pith/arXiv arXiv 2024

  25. [25]

    C. L. Cheng, R. Das, R. Li, R. Mastandrea, V. Mikuni, B. Nachman, D. Shih, and G. Singh, (2025), arXiv:2506.00119 [hep-ph]

    Pith/arXiv arXiv 2025

  26. [26]

    Grosso and M

    G. Grosso and M. Letizia, Eur.Phys.J.C85, 4 (2024), arXiv:2408.12296 [hep-ph]

    Pith/arXiv arXiv 2024

  27. [27]

    Golling, G

    T. Golling, G. Kasieczka, C. Krause, R. Mastandrea, B. Nachman, J. A. Raine, D. Sengupta, D. Shih, and M. Sommerhalder, Eur.Phys.J.C84, 241 (2023), arXiv:2307.11157 [hep-ph]

    Pith/arXiv arXiv 2023

  28. [28]

    J. H. Collins, K. Howe, and B. Nachman, Phys. Rev. Lett.121, 241803 (2018), arXiv:1805.02664 [hep-ph]

    Pith/arXiv arXiv 2018

  29. [29]

    J. H. Collins, K. Howe, and B. Nachman, Phys. Rev. D99, 014038 (2019), arXiv:1902.02634 [hep-ph]

    Pith/arXiv arXiv 2019

  30. [30]

    Hallin, J

    A. Hallin, J. Isaacson, G. Kasieczka, C. Krause, B. Nach- man, T. Quadfasel, M. Schlaffer, D. Shih, and M. Sommerhalder, Phys.Rev.D106, 055006 (2021), arXiv:2109.00546 [hep-ph]

    Pith/arXiv arXiv 2021

  31. [31]

    E. M. Metodiev, B. Nachman, and J. Thaler, JHEP10, 174 (2017), arXiv:1708.02949 [hep-ph]

    Pith/arXiv arXiv 2017

  32. [32]

    J. F. Kamenik and M. Szewc, Phys.Lett.B840, 137836 (2022), arXiv:2210.02226 [hep-ph]

    Pith/arXiv arXiv 2022

  33. [33]

    R. T. D’Agnolo and A. Wulzer, Phys. Rev.D99, 015014 (2019), arXiv:1806.02350 [hep-ph]

    Pith/arXiv arXiv 2019

  34. [34]

    R. T. D’Agnolo, G. Grosso, M. Pierini, A. Wulzer, and M. Zanetti, (2019), 10.1140/epjc/s10052-021-08853-y, arXiv:1912.12155 [hep-ph]

    Pith/arXiv arXiv 2019

  35. [35]

    Neyman and E

    J. Neyman and E. S. Pearson, Phil. Trans. Roy. Soc. Lond. A231, 289 (1933)

    1933

  36. [36]

    TensorFlow: Large-scale machine learn- ing on heterogeneous systems,

    M. Abadiet al., “TensorFlow: Large-scale machine learn- ing on heterogeneous systems,” (2015), software avail- able fromhttps://www.tensorflow.org/

    2015

  37. [37]

    D. P. Kingma and J. Ba, arXiv:1412.6980

    Pith/arXiv arXiv

  38. [38]

    Pedregosaet al., Journal of Machine Learning Re- search12, 2825 (2011)

    F. Pedregosaet al., Journal of Machine Learning Re- search12, 2825 (2011)

    2011

  39. [39]

    R&d dataset for lhc olympics 2020 anomaly detection challenge,

    G. Kasieczka, B. Nachman, and D. Shih, “R&d dataset for lhc olympics 2020 anomaly detection challenge,” https://zenodo.org/record/6466204(2019)

    arXiv 2020

  40. [40]

    Sj¨ ostrand, S

    T. Sj¨ ostrand, S. Mrenna, and P. Z. Skands, JHEP05, 026 (2006), arXiv:hep-ph/0603175 [hep-ph]

    Pith/arXiv arXiv 2006

  41. [41]

    Sj¨ ostrand, S

    T. Sj¨ ostrand, S. Mrenna, and P. Z. Skands, Comput. Phys. Commun.178, 852 (2008), arXiv:0710.3820 [hep- ph]

    Pith/arXiv arXiv 2008

  42. [42]

    de Favereau, C

    J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaitre, A. Mertens, and M. Selvaggi (DELPHES 3), JHEP02, 057 (2014), arXiv:1307.6346 [hep-ex]

    Pith/arXiv arXiv 2014

  43. [43]

    Cacciari and G

    M. Cacciari and G. P. Salam, Phys. Lett.B641, 57 (2006), arXiv:hep-ph/0512210 [hep-ph]

    Pith/arXiv arXiv 2006

  44. [44]

    Cacciari, G

    M. Cacciari, G. P. Salam, and G. Soyez, Eur. Phys. J. C72, 1896 (2012), arXiv:1111.6097 [hep-ph]

    Pith/arXiv arXiv 2012

  45. [45]

    Thaler and K

    J. Thaler and K. Van Tilburg, JHEP03, 015 (2011), arXiv:1011.2268 [hep-ph]

    Pith/arXiv arXiv 2011

  46. [46]

    Thaler and K

    J. Thaler and K. Van Tilburg, JHEP02, 093 (2012), arXiv:1108.2701 [hep-ph]. 12

    Pith/arXiv arXiv 2012

  47. [47]

    Additional qcd background events for lhco2020 r&d (signal region only),

    D. Shih, “Additional qcd background events for lhco2020 r&d (signal region only),”https://zenodo.org/record/ 5759086(2021)

    2021

  48. [48]

    M. Hein, G. Kasieczka, M. Kr¨ amer, L. Moureaux, A. M¨ uck, and D. Shih, (2025), arXiv:2511.14832 [hep- ph]

    arXiv 2025

  49. [49]

    in prepara- tion,

    P. Shyamsundar, N. Smith, and M. Szewc, “in prepara- tion,”

  50. [50]

    Look everywhere effects in anomaly de- tection,

    M. Hein, “Look everywhere effects in anomaly de- tection,”https://github.com/mariehein/look_ everywhere_in_AD(2025)

    2025