REVIEW 4 major objections 5 minor 36 references
NEXUS shows that a 3-million-parameter autoencoder, pre-trained unsupervised on ATLAS tracks, transfers across collider and out-of-domain scientific tasks, matching a transformer's frozen-encoder probes within ~2% while pre-training ~46x fa
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 06:41 UTC pith:2Q5CAEGJ
load-bearing objection Solid in-domain transfer study with a useful efficiency comparison; the foundation-model framing overreaches where the out-of-domain evidence is thinnest. the 4 major comments →
A Lightweight Foundation Model for Collider Physics with Multi-Domain Adaptation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that a fully connected autoencoder with approximately 3.3 million parameters, pre-trained in a fully unsupervised manner on ATLAS inner-detector tracks (30 tracks x 6 features), learns a latent representation that generalizes across collider physics tasks and to out-of-domain waveform data. The paper demonstrates that frozen-encoder linear probes on the pre-trained backbone match a parameter-matched transformer within about 2% on 20-class SM classification while pre-training about 46 times faster per epoch and requiring about 31x fewer multiply-accumulate operations per event. The representation organizes events into physically meaningful latent clusters (as shown by
What carries the argument
The central object is NEXUS, a fully connected autoencoder (three-layer encoder of widths 2048-1024-512 to a 128-dimensional latent, asymmetric decoder) pre-trained with a masked Huber reconstruction loss on 20 million ATLAS collision events, with each event represented as a 30x6 tensor of track features (eta, pT, phi, d0, z0, chi2). The latent space itself is the transferable backbone used in all downstream tasks, either frozen with a linear probe or fine-tuned. For out-of-domain tasks, the key mechanism is the hand-authored feature mapping in Table 1 that associates collider track features to waveform descriptors (e.g., pT to standard deviation, phi to time centroid) to respect analogous s
Load-bearing premise
The out-of-domain transfer results rest on the hand-authored equivalence between collider track features and waveform summary statistics; if that mapping does not genuinely preserve the physical structure the encoder learned, the apparent cross-domain transfer is a property of the encoding rather than of a general-purpose learned representation.
What would settle it
Train NEXUS on the same ATLAS data but with a different random initialization of the encoder weights and repeat the gravitational-wave classification task under the same feature mapping; if the transfer advantage vanishes for multiple random seeds, the reported cross-domain benefit may be an artifact of a single favorable run rather than a robust property of pre-training.
If this is right
- If the claim holds, foundation models for scientific data do not require attention or transformer-scale parameter counts; O(1) million-parameter autoencoders can provide transferable representations at a fraction of the computational cost.
- Pre-trained track-based representations can improve performance in low-label regimes for collider tasks, potentially reducing the amount of labeled data needed for rare-signal searches and jet reconstruction.
- Out-of-domain transfer to gravitational waves, coastal flooding, and neural activity indicates that collider physics priors can be leveraged for waveform-like data from other scientific fields, provided a physically motivated feature mapping exists.
- The ~46x faster pre-training and ~31x fewer MACs per event relative to a transformer suggest practical feasibility for FPGA implementation and real-time or edge inference in data acquisition systems.
- The benefit of pre-training is data-regime dependent: with sufficient labeled data, from-scratch training matches or exceeds pre-trained initialization, so the advantage concentrates in low-statistics settings.
Where Pith is reading between the lines
- The success of NEXUS may be attributed to the simplicity of the track-feature modality (sparse, low-dimensional, set-based), which does not require the long-range dependencies that attention provides; a similar approach might fail for dense 2D or 3D data, as the paper itself notes.
- The hand-authored feature mapping could be viewed as a form of inductive bias that injects physics knowledge into the transfer process; if this mapping were learned or optimized automatically, transfer might extend to even more heterogeneous datasets.
- A testable extension would be to evaluate NEXUS on a larger variety of out-of-domain tasks with different feature mappings, or to compare against a small CNN or transformer on the same tasks to quantify when the simple autoencoder's inductive bias is beneficial.
- The paper's latent-space analysis (UMAP) suggests that the learned representation captures event topology; one could probe whether the latent dimensions correspond to interpretable physical quantities (e.g., jet multiplicity, missing energy) and use that for anomaly score interpretation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces NEXUS, a fully connected autoencoder with ~3.3M total parameters (encoder ~3.1M), pre-trained in an unsupervised manner on ~20 million ATLAS Open Data collision events represented as 30×6 track-feature tensors (η, pT, φ, d0, z0, χ²). Downstream evaluation covers in-domain collider tasks (BSM/SM classification, leading-jet multi-task reconstruction, latent-space anomaly detection) and out-of-domain scientific tasks (gravitational-wave classification, coastal-flooding regression, neural forecasting). The central claim is that the pre-trained backbone provides transferable representations that improve low-label downstream performance relative to from-scratch training, while being far cheaper than a parameter-matched transformer (≈46× faster per epoch, ~3.1 vs ~96.9 MMACs per event). The paper also argues that the out-of-domain results, obtained via a hand-authored mapping between waveform summary statistics and track features, support a general-purpose foundation-model capability. The methods and data are publicly available.
Significance. If the claims hold, the paper would make a meaningful contribution: it challenges the assumption that foundation models require attention-based architectures, demonstrates systematic low-label transfer gains with a lightweight MLP autoencoder, and provides a reproducible experimental protocol with 10-seed training curves, identical splits across schemes, and a parameter-matched transformer baseline. The explicit public code and data availability strengthen the work's utility to the community. However, the broader foundation-model claim rests on the out-of-domain transfer, whose feature mapping is not validated as physically faithful, and on the neural-forecasting result, which currently has no comparison baseline. These gaps are central to the paper's title and Discussion, so the significance is currently conditional.
major comments (4)
- [§2.2, Table 1] The out-of-domain transfer is the basis for the claim of a 'multi-domain adaptation' foundation model, but the mapping in Table 1 is never shown to preserve the physical structure the encoder is claimed to have learned. The permutation control in Fig. 4 swaps the assignment of the same six descriptors to track-feature columns, showing that the specific ordering matters; it does not test whether the descriptors respect the relevant symmetries and scale structure (e.g., φ periodicity, log-pT heavy-tail compression, d0/z0 displacement). A self-consistent but non-physical hash could produce the same pattern. Please add a direct validation: for example, show that the latent representation responds correctly to physical transformations of the waveform inputs (time shifts, amplitude rescales, phase rotations), or compare against an encoder pre-trained on scrambled inputs with the same marginal
- [§2.1.1, Figure 2] The anomaly-detection AUC is computed from a KDE fit to a 2D UMAP projection of the latent space, but no held-out split is described for the UMAP fit, the KDE fit, or the AUC evaluation. If the same events are used to fit the density and to compute the AUC, the reported values (>0.95) are optimistically biased. Please specify how out-of-sample events are used, or provide a cross-validated or separate-test-set evaluation. This is necessary to support the anomaly-detection capability claimed in the text.
- [§2.2.3, Eq. (1)] The neural-forecasting result reports R²=0.891/0.892 without any baseline comparison—neither a from-scratch version of the same two-stage architecture, nor a randomly initialized backbone, nor a dedicated forecasting method. As written, the sentence 'These results indicate strong predictive performance, demonstrating another successful mapping...' cannot be evaluated: the reader cannot tell whether the pre-trained NEXUS encoder contributes anything. Please add a from-scratch or randomly-initialized control and, ideally, a training-data scan analogous to Figures 3 and 5.
- [§3, parameter-matched transformer] The efficiency comparison is quoted as '~46× faster per epoch,' but the total pre-training cost to reach a given reconstruction quality is not reported. The transformer uses a much smaller batch size (4,096 vs 92,160), so per-epoch wall-clock is not necessarily a reliable proxy for total training time to convergence. Please report the total number of epochs (or total wall-clock time) each model needed to reach comparable validation loss, or present a convergence curve. This is load-bearing for the 'substantially lower computational overhead' claim.
minor comments (5)
- [Abstract, §4.2] The abstract says 'approximately 3 million parameters' while Section 4.2 gives 3.3 million for the full autoencoder and ~3.1 million for the encoder. Consider stating the encoder size explicitly in the abstract to avoid ambiguity.
- [Figure 4] The legend label 'GW SGLF' is inconsistent with the text's 'GW SG' (and the corresponding challenge terminology). Please harmonize the label.
- [Figure 3 caption] Typo in captions: 'F oundation model scaling' should be 'Foundation model scaling.'
- [§2.1.3, Figure 3e] The 'Calo reco' reference line is described as the resolution of standard calorimeter-based reconstruction, but the exact definition (e.g., whether it is mean relative error, median, or a resolution from a specific ATLAS Open Data object) is not given. Please specify the computation and the uncertainty on it, since it is used as a non-ML reference.
- [§4.3.2, neural forecasting] The two-stage chained model is only briefly described. Please provide details on how the sliding window of 30 neighboring channels is constructed (overlap, stride, ordering), and on the Stage-2 feature-to-signal mapping, so that the result is reproducible from the public code description.
Circularity Check
No significant circularity: pre-training and downstream tasks are independent, and the hand-designed out-of-domain mapping is an explicit input tested by controls rather than a result derived from itself.
full rationale
The derivation chain is self-contained with respect to circularity. NEXUS is pre-trained with an unsupervised masked Huber reconstruction loss on 30×6 ATLAS track tensors (Section 4.2); the downstream objectives—BSM/SM classification, jet pT/mass/b-tag regression, GW classification, coastal-flooding regression, neural forecasting—are all measured with task-specific metrics (accuracy, MAE, R^2) that do not appear in the pre-training loss. No parameter is fitted to a downstream quantity and then reported as a prediction of it. The out-of-domain transfer is conditional on the hand-authored Table 1 mapping; the paper is transparent that this mapping is 'designed to capture the same underlying detector response, while respecting analogous symmetries and information content' (Section 2.2) and tests it with a feature-permutation control (Figure 4), which is a genuine falsification attempt. The neural-forecasting result (R^2≈0.89, Section 2.2.3) lacks a from-scratch or random-backbone comparator, but that is a missing control/evidence limitation, not a circular reduction: the mapping is not fitted to the neural data, and the reported R^2 is not used to define the encoder. There are no load-bearing self-citations: the cited transfer-learning precedents (e.g., [16]) are external, and no uniqueness theorem or prior author result is invoked to force the architecture. The paper's own caveat that NEXUS 'is not expected... to provide competitive performance... compared to a dedicated ML approach' further indicates the out-of-domain numbers are presented as transfer probes, not as self-verifying derivations.
Axiom & Free-Parameter Ledger
free parameters (6)
- Input feature transforms (log p_T, linear rescales)
- Track truncation K=30 and zero-padding =
30
- Latent dimension =
128
- Encoder widths 2048-1024-512
- Out-of-domain feature mapping (Table 1)
- Fine-tuning learning rates =
1e-5 / 1e-3
axioms (5)
- domain assumption ATLAS track features (eta, pT, phi, d0, z0, chi2) are an adequate low-level representation for learning transferable structure.
- domain assumption Pre-training on the first 8 of 186 ATLAS 2016 run periods generalizes to the rest of Run 2 and to MC signal samples.
- ad hoc to paper The waveform-to-track feature mapping preserves analogous symmetries and information content.
- domain assumption UMAP dimensionality reduction and KDE density estimates on the latent space faithfully represent anomaly structure.
- domain assumption Fine-tuning with small learning rates preserves pre-trained features rather than overwriting them.
read the original abstract
We present a lightweight approach to foundation modeling (\textbf{NEXUS}) that leverages pre-trained learning from collider physics data towards out-of-domain tasks in other scientific datasets, using a fully connected autoencoder model with approximately 3 million parameters. The model pre-trains with no supervision over a large-scale collision dataset from the Large Hadron Collider modeled by charged particle track features. Downstream tasks for collider analyses, such as kinematic regression and event classification, are developed on pre-trained model weights and achieve improved accuracy with only small labeled datasets when compared to equivalent architectures trained from scratch. The benefits of pre-training are additionally investigated through latent space interpretation and application to other domains, including gravitational waves, flood forecasting, and neural activity. Furthermore, the relative computational simplicity of NEXUS is demonstrated compared to transformer approaches at comparable scale, opening the door to power-efficient inference and real-time or edge applications of foundation models in scientific experiments.
Reference graph
Works this paper leans on
-
[1]
URL https://arxiv.org/abs/1807.02876
Albertsson, K.et al.Machine learning in high energy physics community white paper (2019). URL https://arxiv.org/abs/1807.02876. arXiv:1807.02876
Pith/arXiv arXiv 2019
-
[2]
Duarte, J. M., Seljak, U. & Terao, K. Machine learning (2025). URL https: //arxiv.org/abs/2512.11133. Particle Data Group Review of Machine Learning, 2025 update, arXiv:2512.11133
arXiv 2025
-
[3]
Karagiorgi, G., Kasieczka, G., Kravitz, S., Nachman, B. & Shih, D. Machine learning in the search for new fundamental physics.Nature Reviews Physics4, 399–412 (2022). URL https://doi.org/10.1038/s42254-022-00455-1
-
[4]
URL https://doi.org/10.1038/ s41586-018-0361-2
Radovic, A.et al.Machine learning at the energy and intensity frontiers of particle physics.Nature560, 41–48 (2018). URL https://doi.org/10.1038/ s41586-018-0361-2
2018
-
[5]
URL https://arxiv.org/abs/2108.07258
Bommasani, R.et al.On the opportunities and risks of foundation models (2022). URL https://arxiv.org/abs/2108.07258. arXiv:2108.07258
Pith/arXiv arXiv 2022
-
[6]
URL https://arxiv.org/abs/2302.09419
Zhou, C.et al.A comprehensive survey on pretrained foundation models: A history from bert to chatgpt (2023). URL https://arxiv.org/abs/2302.09419. arXiv:2302.09419
Pith/arXiv arXiv 2023
-
[7]
Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment940, 156–167 (2019)
Kuchera, M.et al.Machine learning methods for track classification in the at-tpc. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment940, 156–167 (2019). URL https://www.sciencedirect.com/science/article/pii/S0168900219308046
2019
-
[8]
Chappell, A. & Whitehead, L. H. Application of transfer learning to neutrino interaction classification.The European Physical Journal C82(2022). URL http://dx.doi.org/10.1140/epjc/s10052-022-11066-6
-
[9]
Dreyer, F. A., Grabarczyk, R. & Monni, P. F. Leveraging universality of jet taggers through transfer learning.The European Physical Journal C82, 564 (2022). URL https://doi.org/10.1140/epjc/s10052-022-10469-9. 15
-
[10]
URL https://arxiv.org/abs/2401
Golling, T.et al.Masked particle modeling on sets: Towards self-supervised high energy physics foundation models (2024). URL https://arxiv.org/abs/2401. 13537. arXiv:2401.13537
Pith/arXiv arXiv 2024
-
[11]
& Nachman, B
Mikuni, V. & Nachman, B. Method to simultaneously facilitate all jet physics tasks.Phys. Rev. D111, 054015 (2025). URL https://link.aps.org/doi/10.1103/ PhysRevD.111.054015
2025
-
[12]
Bhimji, W., Harris, C., Mikuni, V. & Nachman, B. Foundation model framework for all tasks involving jet physics.Physical Review D113(2026). URL http: //dx.doi.org/10.1103/knmd-f5jm
-
[13]
Birk, J., Hallin, A. & Kasieczka, G. OmniJet-α: the first cross-task foundation model for particle physics.Machine Learning: Science and Technology5, 035031 (2024). URL http://dx.doi.org/10.1088/2632-2153/ad66ad
-
[14]
URL https://arxiv.org/abs/2601.17126
Hsu, T.-H.et al.Evenet: A foundation model for particle collision data analysis (2026). URL https://arxiv.org/abs/2601.17126. arXiv:2601.17126
arXiv 2026
-
[15]
& Nachman, B
Mikuni, V. & Nachman, B. Solving key challenges in collider physics with foun- dation models.Physical Review D111(2025). URL http://dx.doi.org/10.1103/ PhysRevD.111.L051504
2025
-
[16]
Mikuni, V., Elsharkawy, I. & Nachman, B. Omnicosmos: Transferring particle physics knowledge across the cosmos (2025). URL https://arxiv.org/abs/2512. 24422. arXiv:2512.24422
arXiv 2025
-
[17]
URL https://arxiv.org/abs/ 1706.03762
Vaswani, A.et al.Attention is all you need (2023). URL https://arxiv.org/abs/ 1706.03762. arXiv:1706.03762
Pith/arXiv arXiv 2023
-
[18]
URL https://arxiv.org/abs/2105.01601
Tolstikhin, I.et al.Mlp-mixer: An all-mlp architecture for vision (2021). URL https://arxiv.org/abs/2105.01601. arXiv:2105.01601
Pith/arXiv arXiv 2021
-
[19]
URL https://arxiv.org/abs/2105.03404
Touvron, H.et al.Resmlp: Feedforward networks for image classification with data-efficient training (2021). URL https://arxiv.org/abs/2105.03404. arXiv:2105.03404
Pith/arXiv arXiv 2021
-
[20]
Liu, H., Dai, Z., So, D. R. & Le, Q. V. Pay attention to mlps (2021). URL https://arxiv.org/abs/2105.08050. arXiv:2105.08050
Pith/arXiv arXiv 2021
-
[21]
JINST3, S08003 (2008)
Aad, G.et al.The ATLAS Experiment at the CERN Large Hadron Collider. JINST3, S08003 (2008)
2008
-
[22]
& Aarrestad, T
Belis, V., Odagiu, P. & Aarrestad, T. K. Machine learning for anomaly detection in particle physics.Reviews in Physics12, 100091 (2024). URL https://www. sciencedirect.com/science/article/pii/S2405428324000017. 16
2024
-
[23]
& McInnes, L
Healy, J. & McInnes, L. Uniform manifold approximation and projection. Nature Reviews Methods Primers4, 82 (2024). URL https://doi.org/10.1038/ s43586-024-00363-x
2024
-
[24]
Kumar, A., Raghunathan, A., Jones, R., Ma, T. & Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution (2022). URL https://arxiv.org/abs/2202.10054. arXiv:2202.10054
Pith/arXiv arXiv 2022
-
[25]
Anomaly detection ml challenge
NSF HDR. Anomaly detection ml challenge. https://www.nsfhdr.org/html/ mlchallenge-y1/index.html (2024)
2024
-
[26]
Scientific modeling out of distribution (mood) ml challenge
NSF HDR. Scientific modeling out of distribution (mood) ml challenge. https: //www.nsfhdr.org/html/mlchallenge-y2/index.html (2025)
2025
-
[27]
Computing and Software for Big Science3, 7 (2019)
Albrecht, J.et al.A roadmap for hep software and computing r&d for the 2020s. Computing and Software for Big Science3, 7 (2019). URL https://doi.org/10. 1007/s41781-018-0018-8
2019
-
[28]
Deiana, A. M.et al.Applications and techniques for fast machine learning in science.Frontiers in Big Data5(2022). URL http://dx.doi.org/10.3389/fdata. 2022.787421
arXiv 2022
-
[29]
https://opendata.atlas.cern/
ATLAS Open Data. https://opendata.atlas.cern/
-
[30]
Alwall, J., Herquet, M., Maltoni, F., Mattelaer, O. & Stelzer, T. Madgraph 5: going beyond.Journal of High Energy Physics2011(2011). URL http: //dx.doi.org/10.1007/JHEP06(2011)128
-
[31]
The pythia event generator: Past, present and future.Computer Physics Communications246, 106910 (2020)
Sj¨ ostrand, T. The pythia event generator: Past, present and future.Computer Physics Communications246, 106910 (2020). URL https://www.sciencedirect. com/science/article/pii/S0010465519302899
2020
-
[32]
Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization (2017). URL https://arxiv.org/abs/1412.6980. arXiv:1412.6980
Pith/arXiv arXiv 2017
-
[33]
URL http://dx.doi.org/10.3847/1538-4365/acdc9f
Abbott, R.et al.Open Data from the Third Observing Run of LIGO, Virgo, KAGRA, and GEO.The Astrophysical Journal Supplement Series267, 29 (2023). URL http://dx.doi.org/10.3847/1538-4365/acdc9f
-
[34]
& Shlizerman, E
Le, T. & Shlizerman, E. Stndt: Modeling neural population activity with spa- tiotemporal transformers.Advances in Neural Information Processing Systems 35, 17926–17939 (2022)
2022
-
[35]
Li, J.et al.Amag: Additive, multiplicative and adaptive graph neural network for forecasting neuron activity.Advances in Neural Information Processing Systems 36, 8988–9014 (2023). 17
2023
-
[36]
Pandarinath, C.et al.Inferring single-trial neural population dynamics using sequential auto-encoders.Nature methods15, 805–815 (2018). 18
2018
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.