REVIEW 3 major objections 7 minor 25 references
Recasting the ATLAS search for displaced hadronic jets in the ATLAS calorimeter with additional jets or leptons using surrogate models
T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Independent validation shows surrogate models reproduce an experimental search for displaced hadronic jets using only truth-level inputs.
desk verdict Useful, honest validation of ATLAS's SuMos, but the 'very good approximation' claim is demonstrated only for the benchmark topologies; the paper's own caveats carry most of that message. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the surrogate model (SuMo): a multi-class random-forest classifier, packaged in a portable interchange format, that acts as a 'forward folding matrix' compressing detector response and event selection into a single truth-level map. For each event it takes a small set of generator-level inputs—decay positions, transverse mass and momentum, pseudorapidity, decay-product PDGIDs, and where relevant vector-boson kinematics—and outputs five probabilities for the mutually exclusive classes 'not selected' and regions A, B, C, and D. The tree structure replaces the fixed bins of earlier efficiency maps with adaptively chosen cuts, and the feature list includes both transverse mass and transverse momentum so mass-difference dependence can be learned. The usage prescription is to evaluate the model on every event and sum the probabilities over the sample, giving approximate efficiencies for all four ABCD regions at once.
What would settle it
Take an LLP model whose decays produce three quarks or light quarks with decay positions inside the stated validity ranges, generate events, apply the surrogate models, and compare with the full selection as implemented in the reference analysis; a discrepancy larger than the factor-of-two bound seen in the worst tested cases would show the truth-feature assumption is incomplete.
Extended reading notes
Core claim
The central claim is that the surrogate models provide a very good approximation of the full analysis efficiency using only truth-level information. Concretely, for each of the six selections (CR+2J, WHS_highET, WHS_lowET, WALP, ZHS_highET, and ZHS_lowET), a multi-class tree-ensemble model maps truth-level inputs—the transverse and longitudinal decay positions, transverse mass, transverse momentum, pseudorapidity, and decay-product PDGIDs of up to two LLPs, plus optional vector-boson kinematics—to the probabilities of the event being not selected or selected in regions A, B, C, or D. Efficiency curves built by summing these probabilities over Monte Carlo event samples reproduce the reference analysis results across the lifetime range where the SuMos are declared valid. The agreement holds for Hidden Sector, dark-photon, and ALP models, with stated exceptions: the WHS_highET selection underestimates the efficiency for a 1 TeV mediator decaying to 475 GeV LLPs that decay to top quarks by about a factor of two, and the training topologies do not cover decays to three quarks or to light quarks.
Load-bearing premise
The load-bearing premise is that the detector and analysis response depends only on the listed truth-level features, so any new model that produces the same decay positions, momenta, and decay-product identities will have the same selection probability; this is what lets the SuMos generalize beyond the benchmark models, and it fails for topologies the training set never saw.
Editorial extensions
If this is right
- The primary consequence is that an external theorist can compute approximate event-selection efficiencies for new LLP models using only truth-level information, without running detector simulation, at a computational cost of evaluating a small tree ensemble per event.
- The advertised and independently checked accuracy is roughly 25% within the stated ranges of LLP decay positions (barrel average transverse decay position between 25 cm and 16 m, endcap longitudinal between 75 cm and 28 m), with at most a factor-of-two error in the most pathological tested case.
- Since the SuMos return probabilities for all four ABCD regions, reinterpretation can account for signal leakage into control regions, not just the signal region, which matters for models that populate several regions.
- Because the models are packaged in a portable interchange format with vectorized evaluation, they can be dropped into existing recasting workflows, and the paper argues this makes the underlying search FAIR-compliant (findable, accessible, interoperable, reusable).
- The success of this validation supports the broader proposal that surrogate models of this kind could preserve other LLP searches and machine-learning-heavy analyses that are otherwise effectively impossible to recast.
Reading between the lines
- Inference: the factor-of-two failure on the high-ET W-channel sample with top-quark decays suggests the SuMos under-sample heavy decay products; a direct extension would be to retrain with top-quark final states among the benchmarks and check whether the discrepancy disappears.
- Inference: the 0.1 probability quantisation from the small number of trees is likely to bias low-probability tails; increasing the number of trees is a cheap fix that could be tested by comparing quantized versus fine-grained probabilities on low-efficiency events.
- Inference: the same truth-feature assumption could be exported to other machine-learning-based searches, but only where detector response is believed to be a function of a small set of generator-level variables; searches whose selections depend on jet substructure or track multiplicity may need additional input features.
- Inference: the published validity ranges suggest a practical guardrail—theorists should check average decay positions before trusting a recast, and this check could be encoded directly into the surrogate-model evaluation code to warn users automatically.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents an independent validation of the machine-learning 'surrogate models' (SuMos) released by ATLAS for the displaced hadronic jet search ATLAS-EXOT-2022-04. The SuMos are boosted decision trees that map truth-level quantities (LLP decay positions, transverse momentum, pseudorapidity, decay-product PDGIDs, and, where relevant, vector-boson kinematics) to probabilities for five analysis regions. The authors generate events with MadGraph+Pythia for the benchmark Hidden Sector, dark-photon, and ALP models, compare SuMo efficiencies against the full ATLAS efficiencies across six channels, and additionally implement the SuMos in the HackAnalysis framework, validating two channels. The agreement is typically within the quoted 25% uncertainty, with one factor-of-two outlier (WHS_highET, mH=1000 GeV, mS=475 GeV). The paper discusses limitations, notably decays to three quarks or to light quarks, and concludes that SuMos are a very good approximation using truth-level information.
Significance. If the SuMos perform as advertised, they constitute an important step in making a highly complex, ML-based LLP search re-interpretable by the theory community, in accordance with the FAIR principles. The paper provides useful, reproducible code (event generation setup, HackAnalysis implementation) and a clear description of the SuMo framework, including its input scaling and output structure. The validation covers six channels and several benchmark models, and the authors are honest about the known failure modes. The main caveat is that the validation is performed on the same benchmark models used to train the SuMos, so the demonstrated accuracy is in-sample and the generalization to genuinely new models remains untested.
major comments (3)
- [Section IV and Section VII] The validation in Figures 1-6 uses the same benchmark models (HSS, ZZd, ALP) that were used to train the SuMos, so the demonstrated agreement is an in-sample test of the surrogate models. The Section VII statement that 'the surrogate models provide a very good approximation of the full analysis efficiency using only truth-level information' should be explicitly scoped to the trained topologies, and the paper should state that no out-of-sample test of generalization is performed. As written, the claim is too strong given the Section VI caveats about three-quark and light-quark decays.
- [Section IV, Figures 1-6] The text repeatedly reports agreement 'within the 25% uncertainty', but none of the figures shows an uncertainty band or error bars on the SuMo predictions. Without such a band, the reader cannot verify the central quantitative claim. Please add a shaded 25% band to the SuMo curves, or provide a table of per-point deviations.
- [Section IV, Figure 3] The HSS point (mH=1000 GeV, mS=475 GeV) in the WHS_highET channel, which decays to top quarks, is underestimated by a factor of about two and therefore violates the quoted 25% accuracy within the stated applicability range. This is not just a minor exception: it shows that the error band can fail for a plausible new-physics signature (boosted top quarks). Please add a visible warning in Section III.D that the 25% accuracy is not guaranteed for decays into top quarks or other high-multiplicity final states, and discuss how users should interpret the SuMo output in such cases.
minor comments (7)
- [Section III.B, footnote 1] The footnote explaining that the variable was originally mislabeled as transverse energy is useful, but the main text still reads 'transverse mass 1' with no definition; please define mT explicitly in the feature list.
- [Section IV] The statement that 'all MadGraph and Pythia settings are chosen at their default values' is vague for a quantitative comparison; please specify the Pythia tune and any non-default settings (e.g., the PDF choice) in the text.
- [Section V] The HackAnalysis validation in Figures 7 and 8 covers only two of the six channels and gives no quantitative measure of agreement; please state the maximum deviation or add a table of deviations for all six channels.
- [Section VI] There is a typo in the FAIR bullet list: 'Findeable' should be 'Findable'.
- [Section III.B] The sentence 'the detector (and more broadly, the analysis) cannot be sensitive the the internal details of the model' contains a duplicated article; please correct 'the the' to 'the'.
- [Section I] In the introduction, 'it’ s' should be 'it's'.
- [Section III.D] The applicability bounds (25 cm to 16 m in the barrel, 75 cm to 28 m in the endcaps) are given in prose; a small table would make them easier to use.
Circularity Check
SuMo validation is in-sample: efficiency predictions are checked against the same ATLAS benchmark models used to train the SuMos, so the 25% accuracy claim is not an independent test of generalization.
-
fitted input called prediction
[Section III D and Section IV (standalone validation); limitations acknowledged in Section VI]
"The helper code which accompanied the HEPData material suggests that the estimated efficiencies are typically accurate within 25%. This claim is borne out by the validation performed in this note. ... we generate samples from the models used in the reference analysis with matching mass parameters, and seek to reproduce the efficiencies which are provided in the Auxiliary Material of the paper and on HEPData."
The SuMos are trained on the reference models of the ATLAS analysis, as confirmed by Section VI: 'all training models assumed decays of the LLPs to pairs of particles' and decays to light quarks 'might obtain predicted efficiencies of zero since none of the reference models contained those decays.' The validation then regenerates those same reference models and compares the SuMo output to the ATLAS efficiencies for those models. Agreement on these points largely demonstrates that the BDTs reproduce their own training targets, not that a new model outside the training topologies will be within 25%. The 25% figure itself is imported from ATLAS's helper code and then said to be 'borne out' by this in-sample check.
full rationale
The paper is an honest validation note: it does not present a derivation, and much of its content (portability, ONNX integration, HackAnalysis workflow) is independent and useful. The central quantitative claim, however, is that the SuMos give 'a very good approximation of the full analysis efficiency using only truth-level information.' The evidence for this claim is mostly in-sample: the same reference benchmark models are generated with 'matching mass parameters' and compared to the ATLAS efficiencies from the Auxiliary Material, while the SuMos were trained on those reference models. This is a functional check of the training fit rather than an external test of generalization to new topologies. The paper partially addresses this by reporting a factor-of-two outlier (WHS_highET with mH=1 TeV, mS=475 GeV) and by listing failure modes in Section VI (three-quark decays, light-quark decays), which shows the authors are not hiding the limitation. There is no load-bearing self-citation chain: citations to prior work by the same authors (Refs [5], [14], [18]) support tooling, not the core efficiency claim. Overall the circularity is moderate and localized to the validation logic, not a fabricated derivation.
Assumptions & free parameters
free parameters (1)
- HackAnalysis LLP definition cuts =
20 cm transverse, 40 cm longitudinal (hand-chosen)
assumptions (4)
- domain assumption Sufficiency of truth-level features
- domain assumption MadGraph/Pythia default settings and NNPDF2.3LO reproduce ATLAS generation sufficiently
- domain assumption The 25% error band from ATLAS helper code is a valid uncertainty for SuMo predictions
- ad hoc to paper HackAnalysis LLP definition approximates SuMo validity region
Cite this review
Pith. "Pith review of Recasting the ATLAS search for displaced hadronic jets in the ATLAS calorimeter with additional jets or leptons using surrogate models." pith.science (2026). https://pith.science/paper/C7XDWTHF
@misc{pith2026250210231,
author = {Pith},
title = {Pith review of: Recasting the ATLAS search for displaced hadronic jets in the ATLAS calorimeter with additional jets or leptons using surrogate models},
year = {2026},
howpublished = {\url{https://pith.science/paper/C7XDWTHF}},
note = {Machine review of arXiv:2502.10231}
}
read the original abstract
This note describes the validation of a new form of re-interpretation material provided by an ATLAS search for hadronically-decaying neutral long-lived particles in association with jets or leptons, using the full Run-2 dataset. This reference ATLAS analysis provided a set of machine-learning-based "surrogate models" which return the probability of an event being selected in a given channel of the analysis, using as input truth-level kinematic information (decay position, transverse momentum and decay products of the long-lived particles). In this document, we describe the surrogate model framework in detail, and how it responds to issues identified in other re-interpretation procedures. We describe independent validations of the surrogate models' performance in reproducing the original analysis results -- first using a standalone framework and then employing the HackAnalysis framework.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Searching for long-lived particles beyond the Standard Model at the Large Hadron Collider
Juliette Alimena et al. “Searching for long-lived particles beyond the Standard Model at the Large Hadron Collider”. In: Journal of Physics G: Nuclear and Particle Physics 47.9 (Sept. 2020), p. 090501. ISSN : 1361-6471. DOI : 10.1088/1361-6471/ab4574. URL : http://dx.doi.org/10.1088/1361- 6471/ ab4574
-
[2]
ATLAS Collaboration. “Search for pair-produced long-lived neutral particles decaying in the ATLAS hadronic calorimeter in p p collisions at ps = 8 TeV”. In: Phys. Lett. B 743 (2015), pp. 15–34. DOI : 10.1016/j.physletb.2015.02.015 . arXiv: 1501. 04020 [hep-ex]
-
[3]
ATLAS Collaboration. “Search for long-lived neutral particles in p p collisions at ps = 13 TeV that decay into displaced hadronic jets in the ATLAS calorimeter”. In: Eur . Phys. J. C 79.6 (2019), p. 481. DOI : 10.1140/epjc/s10052- 019- 6962- 6 . arXiv: 1902.03094 [hep-ex] . 2An implementation in M ADANALYSIS [23–25] should be straightfor- ward once the ON...
arXiv 2019
-
[4]
ATLAS Collaboration. “Search for neutral long-lived particles in pp collisions at ps = 13 TeV that decay into displaced hadronic jets in the ATLAS calorimeter”. In: Journal of High Energy Physics 2022.6 (June 2022). ISSN : 1029-8479. DOI : 10.1007/ jhep06(2022)005. URL : http://dx.doi.org/10. 1007/JHEP06(2022)005
work page 2022
-
[5]
Louie Dartmoor Corpe, Andreas Goudelis, and Thomas Chehab. Notes on recasting the "ATLAS- EXOT -2019-23" search for pairs of displaced hadronic jets in the ATLAS calorimeter . https : / / github . com / ThomasChehab / recastingCodes / blob / master / CalRatioDisplacedJet / Notes _ on _ recasting _ the _ ATLAS _ search _ for _ neutral _ LLPs.pdf. Accessed:...
work page 2019
-
[6]
ATLAS Collaboration. “Search for neutral long-lived particles that decay into displaced jets in the ATLAS calorimeter in association with leptons or jets using pp collisions at ps = 13 TeV”. In: Journal of High Energy Physics 2024.11 (Nov. 2024). ISSN : 1029-8479. DOI : 10 . 1007 / jhep11(2024 ) 036. URL : http : / / dx.doi.org/10.1007/JHEP11(2024)036
-
[7]
How to Find a Hidden World at the Large Hadron Collider
James D. Wells. “How to Find a Hidden World at the Large Hadron Collider”. In: (Mar. 2008). Ed. by Gordon Kane and Aaron Pierce, pp. 283–298. arXiv: 0803.1243 [hep-ph]
arXiv 2008
-
[8]
Exotic decays of the 125 GeV Higgs boson
David Curtin et al. “Exotic decays of the 125 GeV Higgs boson”. In: Phys. Rev. D 90.7 (2014), p. 075004. DOI : 10.1103/PhysRevD.90.075004 . arXiv: 1312. 4992 [hep-ph]
Show all 25 references
-
[9]
ALPs Effective Field Theory and Collider Signatures
I. Brivio et al. “ALPs Effective Field Theory and Collider Signatures”. In: Eur . Phys. J. C 77.8 (2017), p. 572. DOI : 10.1140/epjc/s10052-017-5111-3 . arXiv: 1701.05379 [hep-ph]
2017 arXiv
-
[10]
https : / / www.hepdata.net/record/ins2807458
ATLAS-EXOT -2022-04 HEPdata record . https : / / www.hepdata.net/record/ins2807458. Accessed: 7 Jan 2025
2022
-
[11]
Dataset on Zenodo
Re-interpretation BDTs for ATLAS-EXOT -2022-04 . Dataset on Zenodo. https://doi.org/10.5281/ zenodo.12957030
2022
-
[12]
ONNX Runtime
ONNX Runtime developers. ONNX Runtime. https: //onnxruntime.ai/. 2021
2021
-
[13]
Scikit-learn: Machine Learning in Python
F . Pedregosa et al. “Scikit-learn: Machine Learning in Python”. In: Journal of Machine Learning Research 12 (2011), pp. 2825–2830
2011
-
[14]
HackAnalysis 2: A powerful and hackable recasting tool
Mark D. Goodsell. “HackAnalysis 2: A powerful and hackable recasting tool”. In: (June 2024). arXiv: 2406.10042 [hep-ph]
2024 arXiv
-
[15]
The automated computation of tree- level and next-to-leading order differential cross sections, and their matching to parton shower sim- ulations
J. Alwall et al. “The automated computation of tree- level and next-to-leading order differential cross sections, and their matching to parton shower sim- ulations”. In: JHEP 07 (2014), p. 079. DOI : 10.1007/ JHEP07(2014)079. arXiv: 1405.0301 [hep-ph]
2014 arXiv
-
[16]
A comprehensive guide to the physics and usage of PYTHIA 8.3
Christian Bierlich et al. “A comprehensive guide to the physics and usage of PYTHIA 8.3”. In: (Mar. 2022). DOI : 10.21468/SciPostPhysCodeb.8. arXiv: 2203.11601 [hep-ph]
2022 arXiv
-
[17]
The HepMC3 event record library for Monte Carlo event generators
Andy Buckley et al. “The HepMC3 event record library for Monte Carlo event generators”. In: Com- put. Phys. Commun. 260 (2021), p. 107310. DOI : 10. 1016 / j . cpc . 2020 . 107310. arXiv: 1912 . 08005 [hep-ph]
2021
-
[18]
ATLAS-EXOT -2022-04 Recasting Code
Louie Dartmoor Corpe and Abdelhamid Haddad. ATLAS-EXOT -2022-04 Recasting Code . https : / / github . com / ldcorpe / recastingCodes / tree / exot - 2022 - 04 / CalRatioPlusX - ATLAS - EXOT - 2022-02. Accessed: 2025-02-14. 2024
2022
-
[19]
ATLAS- EXOT -2019-23 Recasting Code
Louie Dartmoor Corpe and Thomas Chehab. ATLAS- EXOT -2019-23 Recasting Code . https : / / github . com / ThomasChehab / recastingCodes / tree / master / CalRatioDisplacedJet. Accessed: 2023- 07-27. 2023
2019
-
[20]
DELPHES 3, A modular frame- work for fast simulation of a generic collider exper- iment
J. de Favereau et al. “DELPHES 3, A modular frame- work for fast simulation of a generic collider exper- iment”. In: JHEP 02 (2014), p. 057. DOI : 10.1007/ JHEP02(2014)057. arXiv: 1307.6346 [hep-ex]
2014 arXiv
-
[21]
Recasting LHC searches for long- lived particles with MadAnalysis 5
Jack Y. Araz et al. “Recasting LHC searches for long- lived particles with MadAnalysis 5”. In: Eur . Phys. J. C 82.7 (2022), p. 597. DOI : 10.1140/epjc/s10052- 022-10511-w. arXiv: 2112.05163 [hep-ph]
2022 arXiv
-
[22]
BSMArt: Simple and fast parameter space scans
Mark D. Goodsell and Ari Joury. “BSMArt: Simple and fast parameter space scans”. In: Comput. Phys. Commun. 297 (2024), p. 109057. DOI : 10.1016/j. cpc.2023.109057. arXiv: 2301.01154 [hep-ph]
2024
-
[23]
MadAnalysis 5, A User-Friendly Framework for Col- lider Phenomenology
Eric Conte, Benjamin Fuks, and Guillaume Serret. “MadAnalysis 5, A User-Friendly Framework for Col- lider Phenomenology”. In: Comput. Phys. Commun. 184 (2013), pp. 222–256. DOI : 10 . 1016 / j . cpc . 2012.09.009. arXiv: 1206.1599 [hep-ph]
2013 arXiv
-
[24]
Designing and recasting LHC analyses with MadAnalysis 5
Eric Conte et al. “Designing and recasting LHC analyses with MadAnalysis 5”. In: Eur . Phys. J.C74.10 (2014), p. 3103. DOI : 10.1140/epjc/s10052-014- 3103-0. arXiv: 1405.3982 [hep-ph]
2014 arXiv
-
[25]
Confronting new physics theories to LHC data with MADANALYSIS 5
Eric Conte and Benjamin Fuks. “Confronting new physics theories to LHC data with MADANALYSIS 5”. In: Int. J. Mod. Phys. A33.28 (2018), p. 1830027. DOI : 10.1142/S0217751X18300272 . arXiv: 1808. 00480 [hep-ph] . 9
2018 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.