Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
read the original abstract
Physics-Informed Neural Networks (PINNs) have emerged recently as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs). However, it has been recognized that adaptive procedures are needed to force the neural network to fit accurately the stubborn spots in the solution of "stiff" PDEs. In this paper, we propose a fundamentally new way to train PINNs adaptively, where the adaptation weights are fully trainable and applied to each training point individually, so the neural network learns autonomously which regions of the solution are difficult and is forced to focus on them. The self-adaptation weights specify a soft multiplicative soft attention mask, which is reminiscent of similar mechanisms used in computer vision. The basic idea behind these SA-PINNs is to make the weights increase as the corresponding losses increase, which is accomplished by training the network to simultaneously minimize the losses and maximize the weights. In addition, we show how to build a continuous map of self-adaptive weights using Gaussian Process regression, which allows the use of stochastic gradient descent in problems where conventional gradient descent is not enough to produce accurate solutions. Finally, we derive the Neural Tangent Kernel matrix for SA-PINNs and use it to obtain a heuristic understanding of the effect of the self-adaptive weights on the dynamics of training in the limiting case of infinitely-wide PINNs, which suggests that SA-PINNs work by producing a smooth equalization of the eigenvalues of the NTK matrix corresponding to the different loss terms. In numerical experiments with several linear and nonlinear benchmark problems, the SA-PINN outperformed other state-of-the-art PINN algorithm in L2 error, while using a smaller number of training epochs.
This paper has not been read by Pith yet.
Forward citations
Cited by 9 Pith papers
-
Loss-Conditional PINNs for Parametric PDE Families
A single physics-informed neural network learns a continuous family of PDE solutions indexed by loss weights or coefficients by sampling the conditioning vector during optimization.
-
When PINNs Go Wrong: Pseudo-Time Stepping Against Spurious Solutions
PINNs fail on spurious solutions admitted by the residual loss; adaptive pseudo-time stepping with Jacobian-based step selection improves accuracy and robustness on PDE benchmarks.
-
Local Well-Posedness of a Modified NSCH-Oldroyd System: PINN-Based Numerical Computation
A diffusion-enhanced NSCH-Oldroyd system is proven locally well-posed and illustrated with PINN numerics for thrombus formation using energy-decay sampling.
-
Adaptive Hard-Soft Physics-Informed Neural Networks for Robust Boundary-Constrained PDE Solving
HSPINN enforces Dirichlet and periodic BCs exactly via analytical lifting and masking, applies adaptive softmax weighting to soft loss terms for PDE residuals, and reports faster convergence and higher accuracy than s...
-
Physics-Informed Residuals for Adaptive Mesh Refinement in Finite-Difference PDE Solvers
PINN residuals serve as an off-grid probe to adaptively refine meshes before a finite-difference solve, yielding lower error with fewer degrees of freedom than uniform refinement on the 1D Burgers equation.
-
Physics-Informed Neural Networks with Learnable Loss Balancing and Transfer Learning
A PINN with learnable loss balancing and transfer learning predicts heat transfer in miniature heat sinks to under 8% error using only 87 data points, outperforming standard baselines.
-
Local Well-Posedness of a Modified NSCH-Oldroyd System: PINN-Based Numerical Computation
A diffusion-enhanced version of the NSCH-Oldroyd system is locally well-posed, with PINN numerics confirming energy decay for representative thrombus models.
-
Bayesian Reasoning for Physics Informed Neural Networks
Introduces Laplace-approximated Bayesian PINNs for automatic loss-weight optimization when solving PDEs such as heat, wave, and Burgers equations.
-
A Practitioner's Guide to Kolmogorov-Arnold Networks
A systematic review of Kolmogorov-Arnold Networks that maps their relation to Kolmogorov superposition theory, MLPs, and kernels, examines basis-function design choices, summarizes performance advances, and supplies a...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.