REVIEW 5 cited by
Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Self-Supervised Learning (SSL) is at the core of training modern large machine learning models, providing a scheme for learning powerful representations that can be used in a variety of downstream tasks. However, SSL strategies must be adapted to the type of training data and downstream tasks required. We propose RS3L ("Re-simulation-based self-supervised representation learning"), a novel simulation-based SSL strategy that employs a method of re-simulation to drive data augmentation for contrastive learning in the physical sciences, particularly, in fields that rely on stochastic simulators. By intervening in the middle of the simulation process and re-running simulation components downstream of the intervention, we generate multiple realizations of an event, thus producing a set of augmentations covering all physics-driven variations available in the simulator. Using experiments from high-energy physics, we explore how this strategy may enable the development of a foundation model; we show how RS3L pre-training enables powerful performance in downstream tasks such as discrimination of a variety of objects and uncertainty mitigation. In addition to our results, we make the RS3L dataset publicly available for further studies on how to improve SSL strategies.
Forward citations
Cited by 5 Pith papers
-
Explicit or Implicit? Encoding Physics at the Precision Frontier
On three precision classification tasks — reweighting-based unfolding, likelihood-ratio estimation, and weakly supervised anomaly detection — a Lorentz-equivariant transformer and a pretrained foundation model perform...
-
Enhancing next token prediction based pre-training for jet foundation models
Using continuous particle features as input and combining next-token with masked-token pre-training markedly improves classification accuracy of the OmniJet jet foundation model without visibly hurting its generative quality.
-
Pretrained Event Classification Model for High Energy Physics Analysis
A pretrained GNN on 120M simulated LHC events improves downstream event-classification accuracy when training data are scarce, with benefits shrinking as data grow.
-
Learning Symmetry-Independent Jet Representations via Jet-Based Joint Embedding Predictive Architecture
J-JEPA pretraining on 1M jets modestly improves top jet tagging versus from-scratch training, but gains are inconsistent for the strongest baseline model.
-
Are We Ready for AI-Driven Discovery? AI Verification Before the Next Fundamental Physics Breakthrough
Verification of ML in fundamental physics is essential precisely when models enter statistical modeling, inference, or hypothesis testing, and is bounded by unavoidable inductive bias, sample complexity, and experimen...
Discussion (0). Continue with ORCID to comment.