REVIEW 17 cited by
Multiple Physics Pretraining for Physical Surrogate Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling of spatiotemporal systems with transformers. In MPP, rather than training one model on a specific physical system, we train a backbone model to predict the dynamics of multiple heterogeneous physical systems simultaneously in order to learn features that are broadly useful across systems and facilitate transfer. In order to learn effectively in this setting, we introduce a shared embedding and normalization strategy that projects the fields of multiple systems into a shared embedding space. We validate the efficacy of our approach on both pretraining and downstream tasks over a broad fluid mechanics-oriented benchmark. We show that a single MPP-pretrained transformer is able to match or outperform task-specific baselines on all pretraining sub-tasks without the need for finetuning. For downstream tasks, we demonstrate that finetuning MPP-trained models results in more accurate predictions across multiple time-steps on systems with previously unseen physical components or higher dimensional systems compared to training from scratch or finetuning pretrained video foundation models. We open-source our code and model weights trained at multiple scales for reproducibility.
Forward citations
Cited by 17 Pith papers
-
TIDE: A Physically Diverse 3D Turbulence Benchmark Dataset for Advancing Scientific Machine Learning
TIDE is a DNS-verified, physically diverse 3D turbulence benchmark with independent ensembles that shows current neural operators barely beat persistence and that low pointwise error does not guarantee physical fidelity.
-
Scale-Consistent Learning for Partial Differential Equations
Scale-consistency training, which enforces agreement between global and rescaled sub-domain predictions, enables neural PDE solvers to extrapolate to unseen scale parameters such as Reynolds number or wavenumber.
-
Kernel Methods for Learning Operators with Multiple Inputs and Outputs
A kernel encoder-decoder framework with error bounds is developed for multi-input/multi-output operator learning, and the KernelMO implementations beat or match neural operators on five parametric PDE families.
-
Neural operator discovery from heterogeneous trajectories
Trajectory grouping plus a low-dimensional latent bottleneck lets a neural operator discover each system's hidden governing factors and extrapolate to unseen systems.
-
Towards a Physics Foundation Model
A single transformer-based model, GPhyT, trained on diverse 2D simulation data, predicts next states across several fluid and heat-transfer systems and extrapolates to similar unseen regimes with plausible results.
-
Probabilistic operator learning: generative modeling and uncertainty quantification for foundation models of differential equations
ICON is shown to compute the posterior predictive mean of differential equation solutions, and a generative extension, GenICON, provides samples from this distribution for uncertainty quantification.
-
Pixel-Resolved Long-Context Learning for Turbulence at Exascale: Resolving Small-scale Eddies Toward the Viscous Limit
A multiscale transformer with a new collective-based parallel attention method is claimed to be the first deep-learning model to reproduce small-scale turbulence statistics down to the viscous limit in 3D flow.
-
Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Models
A self-supervised framework that discovers governing equations from short observed data windows and uses them to regularize autoregressive PDE foundation models, improving long-term forecast accuracy.
-
HEP-JEPA: A foundation model for collider physics using joint embedding predictive architecture
A JEPA-style self-supervised transformer for collider jets improves few-shot classification and transfers to top and quark-gluon tagging, yet remains behind specialized taggers.
-
MATEY: multiscale adaptive foundation models for spatiotemporal physical systems
Adaptive patch sizes chosen by local variance let spatiotemporal transformers match uniform high-resolution accuracy with about half the token sequence length.
-
The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning
The Well provides 16 diverse physics simulation datasets totaling 15TB of data, with a unified interface and baselines showing standard surrogate models fail on many of the tasks.
-
NeuralDEM -- Real-time Simulation of Industrial Particulate Flows
A field-based multi-branch transformer predicts macroscopic granular-flow fields, enabling real-time surrogate simulation of hoppers and 500k-particle fluidized bed reactors.
-
BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics
BCAT, a block causal transformer for next-frame prediction, achieves state-of-the-art accuracy on 2D fluid dynamics PDE benchmarks, beating larger foundation models with fewer parameters.
-
Materials Behavior as Mechanism Ensembles: A Probabilistic Framework for Emergent Behaviors
Materials phenomena such as fatigue crack growth are framed as conditional probability landscapes over competing unit mechanisms, to be inferred from multiscale simulation and multimodal data and then optimized toward...
-
Improving Data Fidelity via Diffusion Model-based Correction and Super-Resolution
A two-stage diffusion pipeline, DCSR, removes solver and noise biases from low-resolution data using an imbalanced SDEdit step, then upscales the corrected fields with cascaded conditional diffusion models.
-
Foundation Models for Astrophysics
Astronomical 'foundation models' largely reuse transformers and self-supervised pretraining, but evidence of transfer to new instruments, populations, or tasks remains rare; the paper argues such evidence, not archite...
-
Machine learning for modelling unstructured grid data in computational physics: a review
A broad review of machine learning techniques for modeling unstructured mesh data in computational physics, with a taxonomy, a qualitative comparison, and a list of public benchmarks.
Discussion (0). Continue with ORCID to comment.