Pith. sign in

REVIEW 4 major objections 5 minor 36 references

NEXUS shows that a 3-million-parameter autoencoder, pre-trained unsupervised on ATLAS tracks, transfers across collider and out-of-domain scientific tasks, matching a transformer's frozen-encoder probes within ~2% while pre-training ~46x fa

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 06:41 UTC pith:2Q5CAEGJ

load-bearing objection Solid in-domain transfer study with a useful efficiency comparison; the foundation-model framing overreaches where the out-of-domain evidence is thinnest. the 4 major comments →

arxiv 2607.27501 v1 pith:2Q5CAEGJ submitted 2026-07-29 cs.LG hep-ph

A Lightweight Foundation Model for Collider Physics with Multi-Domain Adaptation

classification cs.LG hep-ph MSC 68T07
keywords foundation modelautoencodertransfer learningcollider physicsATLASanomaly detectionmulti-domain adaptationself-supervised pre-training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a small, fully connected autoencoder—pre-trained without labels on about 20 million LHC collision events represented by charged-particle track features—can serve as a foundation model for collider physics and even for other scientific domains. The authors show that its learned 128-dimensional latent representation improves downstream classification and regression performance in low-label regimes compared to training from scratch, across tasks such as BSM signal classification, jet reconstruction, gravitational-wave classification, coastal flood forecasting, and neural activity prediction. The core claim is that attention mechanisms are not necessary to obtain transferable representations; a simple DNN autoencoder with O(1) million parameters suffices, delivering comparable in-domain transfer quality to a parameter-matched transformer while being far cheaper to train and deploy. If true, this challenges the prevailing assumption that foundation models require transformer-scale architectures and opens a path toward power-efficient, real-time scientific inference.

Core claim

The central discovery is that a fully connected autoencoder with approximately 3.3 million parameters, pre-trained in a fully unsupervised manner on ATLAS inner-detector tracks (30 tracks x 6 features), learns a latent representation that generalizes across collider physics tasks and to out-of-domain waveform data. The paper demonstrates that frozen-encoder linear probes on the pre-trained backbone match a parameter-matched transformer within about 2% on 20-class SM classification while pre-training about 46 times faster per epoch and requiring about 31x fewer multiply-accumulate operations per event. The representation organizes events into physically meaningful latent clusters (as shown by

What carries the argument

The central object is NEXUS, a fully connected autoencoder (three-layer encoder of widths 2048-1024-512 to a 128-dimensional latent, asymmetric decoder) pre-trained with a masked Huber reconstruction loss on 20 million ATLAS collision events, with each event represented as a 30x6 tensor of track features (eta, pT, phi, d0, z0, chi2). The latent space itself is the transferable backbone used in all downstream tasks, either frozen with a linear probe or fine-tuned. For out-of-domain tasks, the key mechanism is the hand-authored feature mapping in Table 1 that associates collider track features to waveform descriptors (e.g., pT to standard deviation, phi to time centroid) to respect analogous s

Load-bearing premise

The out-of-domain transfer results rest on the hand-authored equivalence between collider track features and waveform summary statistics; if that mapping does not genuinely preserve the physical structure the encoder learned, the apparent cross-domain transfer is a property of the encoding rather than of a general-purpose learned representation.

What would settle it

Train NEXUS on the same ATLAS data but with a different random initialization of the encoder weights and repeat the gravitational-wave classification task under the same feature mapping; if the transfer advantage vanishes for multiple random seeds, the reported cross-domain benefit may be an artifact of a single favorable run rather than a robust property of pre-training.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, foundation models for scientific data do not require attention or transformer-scale parameter counts; O(1) million-parameter autoencoders can provide transferable representations at a fraction of the computational cost.
  • Pre-trained track-based representations can improve performance in low-label regimes for collider tasks, potentially reducing the amount of labeled data needed for rare-signal searches and jet reconstruction.
  • Out-of-domain transfer to gravitational waves, coastal flooding, and neural activity indicates that collider physics priors can be leveraged for waveform-like data from other scientific fields, provided a physically motivated feature mapping exists.
  • The ~46x faster pre-training and ~31x fewer MACs per event relative to a transformer suggest practical feasibility for FPGA implementation and real-time or edge inference in data acquisition systems.
  • The benefit of pre-training is data-regime dependent: with sufficient labeled data, from-scratch training matches or exceeds pre-trained initialization, so the advantage concentrates in low-statistics settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The success of NEXUS may be attributed to the simplicity of the track-feature modality (sparse, low-dimensional, set-based), which does not require the long-range dependencies that attention provides; a similar approach might fail for dense 2D or 3D data, as the paper itself notes.
  • The hand-authored feature mapping could be viewed as a form of inductive bias that injects physics knowledge into the transfer process; if this mapping were learned or optimized automatically, transfer might extend to even more heterogeneous datasets.
  • A testable extension would be to evaluate NEXUS on a larger variety of out-of-domain tasks with different feature mappings, or to compare against a small CNN or transformer on the same tasks to quantify when the simple autoencoder's inductive bias is beneficial.
  • The paper's latent-space analysis (UMAP) suggests that the learned representation captures event topology; one could probe whether the latent dimensions correspond to interpretable physical quantities (e.g., jet multiplicity, missing energy) and use that for anomaly score interpretation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces NEXUS, a fully connected autoencoder with ~3.3M total parameters (encoder ~3.1M), pre-trained in an unsupervised manner on ~20 million ATLAS Open Data collision events represented as 30×6 track-feature tensors (η, pT, φ, d0, z0, χ²). Downstream evaluation covers in-domain collider tasks (BSM/SM classification, leading-jet multi-task reconstruction, latent-space anomaly detection) and out-of-domain scientific tasks (gravitational-wave classification, coastal-flooding regression, neural forecasting). The central claim is that the pre-trained backbone provides transferable representations that improve low-label downstream performance relative to from-scratch training, while being far cheaper than a parameter-matched transformer (≈46× faster per epoch, ~3.1 vs ~96.9 MMACs per event). The paper also argues that the out-of-domain results, obtained via a hand-authored mapping between waveform summary statistics and track features, support a general-purpose foundation-model capability. The methods and data are publicly available.

Significance. If the claims hold, the paper would make a meaningful contribution: it challenges the assumption that foundation models require attention-based architectures, demonstrates systematic low-label transfer gains with a lightweight MLP autoencoder, and provides a reproducible experimental protocol with 10-seed training curves, identical splits across schemes, and a parameter-matched transformer baseline. The explicit public code and data availability strengthen the work's utility to the community. However, the broader foundation-model claim rests on the out-of-domain transfer, whose feature mapping is not validated as physically faithful, and on the neural-forecasting result, which currently has no comparison baseline. These gaps are central to the paper's title and Discussion, so the significance is currently conditional.

major comments (4)
  1. [§2.2, Table 1] The out-of-domain transfer is the basis for the claim of a 'multi-domain adaptation' foundation model, but the mapping in Table 1 is never shown to preserve the physical structure the encoder is claimed to have learned. The permutation control in Fig. 4 swaps the assignment of the same six descriptors to track-feature columns, showing that the specific ordering matters; it does not test whether the descriptors respect the relevant symmetries and scale structure (e.g., φ periodicity, log-pT heavy-tail compression, d0/z0 displacement). A self-consistent but non-physical hash could produce the same pattern. Please add a direct validation: for example, show that the latent representation responds correctly to physical transformations of the waveform inputs (time shifts, amplitude rescales, phase rotations), or compare against an encoder pre-trained on scrambled inputs with the same marginal
  2. [§2.1.1, Figure 2] The anomaly-detection AUC is computed from a KDE fit to a 2D UMAP projection of the latent space, but no held-out split is described for the UMAP fit, the KDE fit, or the AUC evaluation. If the same events are used to fit the density and to compute the AUC, the reported values (>0.95) are optimistically biased. Please specify how out-of-sample events are used, or provide a cross-validated or separate-test-set evaluation. This is necessary to support the anomaly-detection capability claimed in the text.
  3. [§2.2.3, Eq. (1)] The neural-forecasting result reports R²=0.891/0.892 without any baseline comparison—neither a from-scratch version of the same two-stage architecture, nor a randomly initialized backbone, nor a dedicated forecasting method. As written, the sentence 'These results indicate strong predictive performance, demonstrating another successful mapping...' cannot be evaluated: the reader cannot tell whether the pre-trained NEXUS encoder contributes anything. Please add a from-scratch or randomly-initialized control and, ideally, a training-data scan analogous to Figures 3 and 5.
  4. [§3, parameter-matched transformer] The efficiency comparison is quoted as '~46× faster per epoch,' but the total pre-training cost to reach a given reconstruction quality is not reported. The transformer uses a much smaller batch size (4,096 vs 92,160), so per-epoch wall-clock is not necessarily a reliable proxy for total training time to convergence. Please report the total number of epochs (or total wall-clock time) each model needed to reach comparable validation loss, or present a convergence curve. This is load-bearing for the 'substantially lower computational overhead' claim.
minor comments (5)
  1. [Abstract, §4.2] The abstract says 'approximately 3 million parameters' while Section 4.2 gives 3.3 million for the full autoencoder and ~3.1 million for the encoder. Consider stating the encoder size explicitly in the abstract to avoid ambiguity.
  2. [Figure 4] The legend label 'GW SGLF' is inconsistent with the text's 'GW SG' (and the corresponding challenge terminology). Please harmonize the label.
  3. [Figure 3 caption] Typo in captions: 'F oundation model scaling' should be 'Foundation model scaling.'
  4. [§2.1.3, Figure 3e] The 'Calo reco' reference line is described as the resolution of standard calorimeter-based reconstruction, but the exact definition (e.g., whether it is mean relative error, median, or a resolution from a specific ATLAS Open Data object) is not given. Please specify the computation and the uncertainty on it, since it is used as a non-ML reference.
  5. [§4.3.2, neural forecasting] The two-stage chained model is only briefly described. Please provide details on how the sliding window of 30 neighboring channels is constructed (overlap, stride, ordering), and on the Stage-2 feature-to-signal mapping, so that the result is reproducible from the public code description.

Circularity Check

0 steps flagged

No significant circularity: pre-training and downstream tasks are independent, and the hand-designed out-of-domain mapping is an explicit input tested by controls rather than a result derived from itself.

full rationale

The derivation chain is self-contained with respect to circularity. NEXUS is pre-trained with an unsupervised masked Huber reconstruction loss on 30×6 ATLAS track tensors (Section 4.2); the downstream objectives—BSM/SM classification, jet pT/mass/b-tag regression, GW classification, coastal-flooding regression, neural forecasting—are all measured with task-specific metrics (accuracy, MAE, R^2) that do not appear in the pre-training loss. No parameter is fitted to a downstream quantity and then reported as a prediction of it. The out-of-domain transfer is conditional on the hand-authored Table 1 mapping; the paper is transparent that this mapping is 'designed to capture the same underlying detector response, while respecting analogous symmetries and information content' (Section 2.2) and tests it with a feature-permutation control (Figure 4), which is a genuine falsification attempt. The neural-forecasting result (R^2≈0.89, Section 2.2.3) lacks a from-scratch or random-backbone comparator, but that is a missing control/evidence limitation, not a circular reduction: the mapping is not fitted to the neural data, and the reported R^2 is not used to define the encoder. There are no load-bearing self-citations: the cited transfer-learning precedents (e.g., [16]) are external, and no uniqueness theorem or prior author result is invoked to force the architecture. The paper's own caveat that NEXUS 'is not expected... to provide competitive performance... compared to a dedicated ML approach' further indicates the out-of-domain numbers are presented as transfer probes, not as self-verifying derivations.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

No new physical entities are proposed. NEXUS is a neural architecture, and the Table 1 feature mapping is a data-encoding choice, not a physical object. The load-bearing assumptions are domain assumptions about input representativeness, the pre-training subset, and the hand-designed cross-domain feature mapping.

free parameters (6)
  • Input feature transforms (log p_T, linear rescales)
    Hand-tuned to ATLAS tracker acceptance (Section 4.1); defines the input representation the entire model is built on.
  • Track truncation K=30 and zero-padding = 30
    Fixed input size chosen by hand; events with fewer than 30 tracks are zero-padded (Section 4.1).
  • Latent dimension = 128
    Bottleneck width, hand-chosen; defines the transferred representation used by all downstream heads.
  • Encoder widths 2048-1024-512
    Architecture capacity chosen by hand; approximately 3.1M encoder parameters.
  • Out-of-domain feature mapping (Table 1)
    Hand-designed correspondence between waveform descriptors and track features; load-bearing for the cross-domain transfer claims.
  • Fine-tuning learning rates = 1e-5 / 1e-3
    Discriminative learning rates chosen by hand (Section 4.3); affects whether pre-trained features are preserved.
axioms (5)
  • domain assumption ATLAS track features (eta, pT, phi, d0, z0, chi2) are an adequate low-level representation for learning transferable structure.
    Used as the sole input modality; the paper acknowledges tracks do not fully characterize the final state (Section 2.1).
  • domain assumption Pre-training on the first 8 of 186 ATLAS 2016 run periods generalizes to the rest of Run 2 and to MC signal samples.
    Selection of pre-training periods is arbitrary; no justification is given that this subset spans the needed phase space (Section 4.1).
  • ad hoc to paper The waveform-to-track feature mapping preserves analogous symmetries and information content.
    Table 1 is an invented equivalence; the random-permutation control is the only evidence it is meaningful (Section 2.2).
  • domain assumption UMAP dimensionality reduction and KDE density estimates on the latent space faithfully represent anomaly structure.
    The AUC>0.95 is computed on the 2D UMAP projection, not the 128-dim latent; stability and held-out status are not assessed (Section 2.1.1).
  • domain assumption Fine-tuning with small learning rates preserves pre-trained features rather than overwriting them.
    Relies on standard transfer-learning practice (Section 4.3); not independently verified in this paper.

pith-pipeline@v1.3.0-daily-deepseek · 11296 in / 12765 out tokens · 133863 ms · 2026-08-01T06:41:47.960563+00:00 · methodology

0 comments
read the original abstract

We present a lightweight approach to foundation modeling (\textbf{NEXUS}) that leverages pre-trained learning from collider physics data towards out-of-domain tasks in other scientific datasets, using a fully connected autoencoder model with approximately 3 million parameters. The model pre-trains with no supervision over a large-scale collision dataset from the Large Hadron Collider modeled by charged particle track features. Downstream tasks for collider analyses, such as kinematic regression and event classification, are developed on pre-trained model weights and achieve improved accuracy with only small labeled datasets when compared to equivalent architectures trained from scratch. The benefits of pre-training are additionally investigated through latent space interpretation and application to other domains, including gravitational waves, flood forecasting, and neural activity. Furthermore, the relative computational simplicity of NEXUS is demonstrated compared to transformer approaches at comparable scale, opening the door to power-efficient inference and real-time or edge applications of foundation models in scientific experiments.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 4 canonical work pages

  1. [1]

    URL https://arxiv.org/abs/1807.02876

    Albertsson, K.et al.Machine learning in high energy physics community white paper (2019). URL https://arxiv.org/abs/1807.02876. arXiv:1807.02876

  2. [2]

    M., Seljak, U

    Duarte, J. M., Seljak, U. & Terao, K. Machine learning (2025). URL https: //arxiv.org/abs/2512.11133. Particle Data Group Review of Machine Learning, 2025 update, arXiv:2512.11133

  3. [3]

    & Shih, D

    Karagiorgi, G., Kasieczka, G., Kravitz, S., Nachman, B. & Shih, D. Machine learning in the search for new fundamental physics.Nature Reviews Physics4, 399–412 (2022). URL https://doi.org/10.1038/s42254-022-00455-1

  4. [4]

    URL https://doi.org/10.1038/ s41586-018-0361-2

    Radovic, A.et al.Machine learning at the energy and intensity frontiers of particle physics.Nature560, 41–48 (2018). URL https://doi.org/10.1038/ s41586-018-0361-2

  5. [5]

    URL https://arxiv.org/abs/2108.07258

    Bommasani, R.et al.On the opportunities and risks of foundation models (2022). URL https://arxiv.org/abs/2108.07258. arXiv:2108.07258

  6. [6]

    URL https://arxiv.org/abs/2302.09419

    Zhou, C.et al.A comprehensive survey on pretrained foundation models: A history from bert to chatgpt (2023). URL https://arxiv.org/abs/2302.09419. arXiv:2302.09419

  7. [7]

    Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment940, 156–167 (2019)

    Kuchera, M.et al.Machine learning methods for track classification in the at-tpc. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment940, 156–167 (2019). URL https://www.sciencedirect.com/science/article/pii/S0168900219308046

  8. [8]

    & Whitehead, L

    Chappell, A. & Whitehead, L. H. Application of transfer learning to neutrino interaction classification.The European Physical Journal C82(2022). URL http://dx.doi.org/10.1140/epjc/s10052-022-11066-6

  9. [9]

    A., Grabarczyk, R

    Dreyer, F. A., Grabarczyk, R. & Monni, P. F. Leveraging universality of jet taggers through transfer learning.The European Physical Journal C82, 564 (2022). URL https://doi.org/10.1140/epjc/s10052-022-10469-9. 15

  10. [10]

    URL https://arxiv.org/abs/2401

    Golling, T.et al.Masked particle modeling on sets: Towards self-supervised high energy physics foundation models (2024). URL https://arxiv.org/abs/2401. 13537. arXiv:2401.13537

  11. [11]

    & Nachman, B

    Mikuni, V. & Nachman, B. Method to simultaneously facilitate all jet physics tasks.Phys. Rev. D111, 054015 (2025). URL https://link.aps.org/doi/10.1103/ PhysRevD.111.054015

  12. [12]

    & Nachman, B

    Bhimji, W., Harris, C., Mikuni, V. & Nachman, B. Foundation model framework for all tasks involving jet physics.Physical Review D113(2026). URL http: //dx.doi.org/10.1103/knmd-f5jm

  13. [13]

    & Kasieczka, G

    Birk, J., Hallin, A. & Kasieczka, G. OmniJet-α: the first cross-task foundation model for particle physics.Machine Learning: Science and Technology5, 035031 (2024). URL http://dx.doi.org/10.1088/2632-2153/ad66ad

  14. [14]

    URL https://arxiv.org/abs/2601.17126

    Hsu, T.-H.et al.Evenet: A foundation model for particle collision data analysis (2026). URL https://arxiv.org/abs/2601.17126. arXiv:2601.17126

  15. [15]

    & Nachman, B

    Mikuni, V. & Nachman, B. Solving key challenges in collider physics with foun- dation models.Physical Review D111(2025). URL http://dx.doi.org/10.1103/ PhysRevD.111.L051504

  16. [16]

    & Nachman, B

    Mikuni, V., Elsharkawy, I. & Nachman, B. Omnicosmos: Transferring particle physics knowledge across the cosmos (2025). URL https://arxiv.org/abs/2512. 24422. arXiv:2512.24422

  17. [17]

    URL https://arxiv.org/abs/ 1706.03762

    Vaswani, A.et al.Attention is all you need (2023). URL https://arxiv.org/abs/ 1706.03762. arXiv:1706.03762

  18. [18]

    URL https://arxiv.org/abs/2105.01601

    Tolstikhin, I.et al.Mlp-mixer: An all-mlp architecture for vision (2021). URL https://arxiv.org/abs/2105.01601. arXiv:2105.01601

  19. [19]

    URL https://arxiv.org/abs/2105.03404

    Touvron, H.et al.Resmlp: Feedforward networks for image classification with data-efficient training (2021). URL https://arxiv.org/abs/2105.03404. arXiv:2105.03404

  20. [20]

    Liu, H., Dai, Z., So, D. R. & Le, Q. V. Pay attention to mlps (2021). URL https://arxiv.org/abs/2105.08050. arXiv:2105.08050

  21. [21]

    JINST3, S08003 (2008)

    Aad, G.et al.The ATLAS Experiment at the CERN Large Hadron Collider. JINST3, S08003 (2008)

  22. [22]

    & Aarrestad, T

    Belis, V., Odagiu, P. & Aarrestad, T. K. Machine learning for anomaly detection in particle physics.Reviews in Physics12, 100091 (2024). URL https://www. sciencedirect.com/science/article/pii/S2405428324000017. 16

  23. [23]

    & McInnes, L

    Healy, J. & McInnes, L. Uniform manifold approximation and projection. Nature Reviews Methods Primers4, 82 (2024). URL https://doi.org/10.1038/ s43586-024-00363-x

  24. [24]

    & Liang, P

    Kumar, A., Raghunathan, A., Jones, R., Ma, T. & Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution (2022). URL https://arxiv.org/abs/2202.10054. arXiv:2202.10054

  25. [25]

    Anomaly detection ml challenge

    NSF HDR. Anomaly detection ml challenge. https://www.nsfhdr.org/html/ mlchallenge-y1/index.html (2024)

  26. [26]

    Scientific modeling out of distribution (mood) ml challenge

    NSF HDR. Scientific modeling out of distribution (mood) ml challenge. https: //www.nsfhdr.org/html/mlchallenge-y2/index.html (2025)

  27. [27]

    Computing and Software for Big Science3, 7 (2019)

    Albrecht, J.et al.A roadmap for hep software and computing r&d for the 2020s. Computing and Software for Big Science3, 7 (2019). URL https://doi.org/10. 1007/s41781-018-0018-8

  28. [28]

    M.et al.Applications and techniques for fast machine learning in science.Frontiers in Big Data5(2022)

    Deiana, A. M.et al.Applications and techniques for fast machine learning in science.Frontiers in Big Data5(2022). URL http://dx.doi.org/10.3389/fdata. 2022.787421

  29. [29]

    https://opendata.atlas.cern/

    ATLAS Open Data. https://opendata.atlas.cern/

  30. [30]

    & Stelzer, T

    Alwall, J., Herquet, M., Maltoni, F., Mattelaer, O. & Stelzer, T. Madgraph 5: going beyond.Journal of High Energy Physics2011(2011). URL http: //dx.doi.org/10.1007/JHEP06(2011)128

  31. [31]

    The pythia event generator: Past, present and future.Computer Physics Communications246, 106910 (2020)

    Sj¨ ostrand, T. The pythia event generator: Past, present and future.Computer Physics Communications246, 106910 (2020). URL https://www.sciencedirect. com/science/article/pii/S0010465519302899

  32. [32]

    Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization (2017). URL https://arxiv.org/abs/1412.6980. arXiv:1412.6980

  33. [33]

    URL http://dx.doi.org/10.3847/1538-4365/acdc9f

    Abbott, R.et al.Open Data from the Third Observing Run of LIGO, Virgo, KAGRA, and GEO.The Astrophysical Journal Supplement Series267, 29 (2023). URL http://dx.doi.org/10.3847/1538-4365/acdc9f

  34. [34]

    & Shlizerman, E

    Le, T. & Shlizerman, E. Stndt: Modeling neural population activity with spa- tiotemporal transformers.Advances in Neural Information Processing Systems 35, 17926–17939 (2022)

  35. [35]

    Li, J.et al.Amag: Additive, multiplicative and adaptive graph neural network for forecasting neuron activity.Advances in Neural Information Processing Systems 36, 8988–9014 (2023). 17

  36. [36]

    Pandarinath, C.et al.Inferring single-trial neural population dynamics using sequential auto-encoders.Nature methods15, 805–815 (2018). 18