REVIEW 15 cited by
DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Pre-training has been investigated to improve the efficiency and performance of training neural operators in data-scarce settings. However, it is largely in its infancy due to the inherent complexity and diversity, such as long trajectories, multiple scales and varying dimensions of partial differential equations (PDEs) data. In this paper, we present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data and generalizes to various downstream tasks. Moreover, by designing a flexible and scalable model architecture based on Fourier attention, we can easily scale up the model for large-scale pre-training. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories. Extensive experiments show that we achieve SOTA on these benchmarks and validate the strong generalizability of our model to significantly enhance performance on diverse downstream PDE tasks like 3D data. Code is available at \url{https://github.com/thu-ml/DPOT}.
Forward citations
Cited by 15 Pith papers
-
TIDE: A Physically Diverse 3D Turbulence Benchmark Dataset for Advancing Scientific Machine Learning
TIDE is a DNS-verified, physically diverse 3D turbulence benchmark with independent ensembles that shows current neural operators barely beat persistence and that low pointwise error does not guarantee physical fidelity.
-
From Points to Edges: Edge-Conditioned Spectral Operators for Physics-Sensitive PDE Learning
ESO conditions spectral mode mixing on local pairwise variation statistics, improving neural operator accuracy on PDEs with sharp local structures.
-
Hybrid Lagrangian-Eulerian Model for Lagrangian Fluid Simulation
A hybrid Lagrangian-Eulerian graph neural simulator with adaptive downsampling and cross-attention achieves state-of-the-art accuracy and rollout stability on particle-based fluid benchmarks.
-
Neural operator discovery from heterogeneous trajectories
Trajectory grouping plus a low-dimensional latent bottleneck lets a neural operator discover each system's hidden governing factors and extrapolate to unseen systems.
-
Neptuna: A Comprehensive Machine Learning Framework for Benchmarking Complex Multiphase Flows
A new 2.4 TB benchmark shows no single ML surrogate dominates on shock-driven multiphase flows, and composite losses with SoftAdapt weighting improve interface and spectral fidelity.
-
Physics-Informed Neural Quantum Control for Rovibrational Photoassociation in a Morse Molecular System
PINQC optimizes neural-network laser fields via differentiable Schrödinger propagation to photoassociate a Morse molecule, stably reaching l_max=6 versus prior TBQCP limits near l_max=4.
-
Generative Neural Operators through Diffusion Last Layer
Diffusion in a learned Karhunen-Loève coefficient space turns deterministic neural operators into generative surrogates with predictive uncertainty and improved rollout stability.
-
Probabilistic operator learning: generative modeling and uncertainty quantification for foundation models of differential equations
ICON is shown to compute the posterior predictive mean of differential equation solutions, and a generative extension, GenICON, provides samples from this distribution for uncertainty quantification.
-
Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias
Aspect ratio of weight matrices biases heavy-tail spectral metrics; the new FARMS subsampling method removes this bias and improves downstream layer-wise tuning.
-
MATEY: multiscale adaptive foundation models for spatiotemporal physical systems
Adaptive patch sizes chosen by local variance let spatiotemporal transformers match uniform high-resolution accuracy with about half the token sequence length.
-
BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics
BCAT, a block causal transformer for next-frame prediction, achieves state-of-the-art accuracy on 2D fluid dynamics PDE benchmarks, beating larger foundation models with fewer parameters.
-
FourierFlow: Frequency-aware Flow Matching for Generative Turbulence Modeling
FourierFlow improves multi-step generative turbulence modeling by adding a differential-attention branch, a frequency-weighted Fourier mixing branch, and MAE-based feature alignment to a flow matching model.
-
Latent Mamba Operator for Partial Differential Equations
LaMO replaces attention in latent-token neural operators with bidirectional state-space models and reports consistent accuracy gains on six PDE benchmarks.
-
Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates
Predicting the temporal derivative and integrating it with an ODE solver improves accuracy and stability of neural PDE surrogates compared with direct next-state prediction.
-
Machine learning for modelling unstructured grid data in computational physics: a review
A broad review of machine learning techniques for modeling unstructured mesh data in computational physics, with a taxonomy, a qualitative comparison, and a list of public benchmarks.
Discussion (0). Continue with ORCID to comment.