Pith. sign in

REVIEW 4 major objections 5 minor 27 references

Improving physics-informed neural network extrapolation via transfer learning and adaptive activation functions

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that retraining only the final layer of a physics-informed neural network on a few dozen high-residual collocation points from a validation interval, together with an adaptive linear combination of activation functions…

desk verdict The headline 40/50% numbers are oracle estimates—best AF selected on the test interval plus TL versus vanilla tanh—so the practical gains are real but smaller than advertised. read the letter →

arxiv 2507.12659 v1 pith:WPLGH33T submitted 2025-07-16 cs.LG cs.AIcs.NAmath.DSmath.NAstat.ML

Athanasios Papastathopoulos-Katsaros , Alexandra Stavrianidi , Zhandong Liu This is my paper
classification cs.LGcs.AIcs.NAmath.DSmath.NAstat.ML MSC 68T0765M99
keywords physics-informedneuralnetworksextrapolationtransferlearningadaptiveactivationfunctionscollocationpointsamplingpartialdifferentialequationsAllen-CahnequationBurgers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Physics-informed neural networks solve PDEs by training on the equation's residual, but they extrapolate poorly beyond the training time window. This paper argues that the extrapolation failure can be largely repaired at almost no extra cost: freeze all network layers except the last, pick the 80 collocation points in a neighboring validation interval with the largest PDE residual, and retrain only the final layer on those points while letting its activation function be a learnable linear combination of standard activations. On the Allen-Cahn, Korteweg-de Vries, and viscous Burgers equations, the authors report an average 40% reduction in relative L2 error and a 50% reduction in mean absolute error in the held-out extrapolation domain, with only a few extra minutes of training. If the transfer step works as claimed, it would make PINN extrapolation a cheap calibration problem rather than a full retraining problem.

What carries the argument

The machinery is two-part: a transfer-learning protocol and an adaptive activation function. The protocol samples collocation points uniformly in [0,T_val), ranks them by PDE residual, keeps the 80 highest-loss points, freezes every layer except the last, and fine-tunes that layer with Adam and L2 regularization for 120-150 epochs. The activation function in the final layer is f(x)=sum_i G(alpha_i) sigma_i(beta_i x), where G is the softmax gate and alpha_i, beta_i are learnable weights and slopes; the best performers in the benchmarks are linear combinations of copies of one PDE-suitable activation, such as tanh for Allen-Cahn and Burgers and x+$sin^{2}$(x) for KdV. Hard-constraint changes of variables such as u(t,x)=$x^{2}$ cos(pi x)+t(1-$x^{2}$)v(t,x) remove the data-loss versus residual-loss trade-off, so the transfer phase is driven by the PDE residual alone.

What would settle it

Run the identical protocol on a 1D PDE outside the three tested, such as Fisher's equation or a Kuramoto-Sivashinsky variant, with the same time split and the same 80-point budget; if the final-layer fine-tune does not reduce the relative L2 error in (0.8,1.0] compared with no transfer learning, or if the training-domain error rises by more than the few percent reported here, the transfer premise fails.

Watch

Extended reading notes

Core claim

The central claim is that targeted transfer learning plus an adaptive activation function substantially improves PINN extrapolation. The time domain is split into training [0,0.5], validation (0.5,0.8], and test (0.8,1.0]. After training with hard-encoded initial and boundary conditions, the authors freeze all weights except the final layer, sample 4,000 collocation points uniformly in [0,0.8), keep the 80 with the highest PDE loss, and fine-tune the final layer with Adam and L2 regularization for 120 to 150 epochs. In that phase the final-layer activation is a softmax-weighted linear combination of standard activations, and the best performer is a linear combination of copies of the single activation type best suited to each PDE, such as tanh for Allen-Cahn and Burgers and x + $sin^{2}$(x) for KdV. Across the three benchmarks this reduces the relative L2 extrapolation error by an average of 40% and the mean absolute error by 50% compared with the same network without transfer learning, and it outperforms the DPM, SA-PINN, and causal sweeping baselines reported in the paper.

Load-bearing premise

The load-bearing premise is that fine-tuning only the final layer on 80 high-residual collocation points drawn from the validation interval (0.5,0.8] transfers knowledge to the truly unseen extrapolation interval (0.8,1.0] without disrupting the dynamics learned on [0,0.5].

Editorial extensions

If this is right

  • If the method transfers, any PINN trained on an initial time window can be patched for extended-time prediction by a short final-layer fine-tune on a few dozen residual-maximizing points, without re-optimizing the whole network.
  • The best extrapolation accuracy came from adaptive linear combinations of the activation function already suited to the PDE, not from mixing many diverse activations, which suggests activation selection should be shaped by the PDE's qualitative behavior.
  • The method's computational overhead is small: on the Allen-Cahn benchmark, reported training time increases from 51.6 to 60.7 minutes with the improved activation and transfer learning.
  • Across the three test equations the method reports lower extrapolation error than the DPM, SA-PINN, and causal sweeping baselines on the same benchmarks.
  • Because initial and boundary conditions are encoded exactly, the method avoids balancing data versus residual weights in the transfer phase, simplifying the loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the premise generalizes, any time-dependent PINN could be patched for longer prediction by periodically selecting high-residual points at the edge of the solved window and re-fitting only the readout layer, a cheap continual-learning schedule.
  • The reported averages hide per-benchmark heterogeneity: with tanh alone, transfer learning moves L2 from 0.20 to 0.17 on Allen-Cahn, 0.15 to 0.13 on KdV, and 0.09 to 0.08 on Burgers, so the headline gain depends on pairing transfer learning with the adaptive activation.
  • Because the paper itself flags that hard-constraint encoding and point selection become harder in higher dimensions, a natural test is whether the same 80-point budget and final-layer-only update survive a two-dimensional or irregular-domain PDE.
  • One could also check whether validation-interval residual magnitude predicts test-interval improvement: if the 80 highest-loss points do not lie where extrapolation later fails, the selection criterion would need to be revised.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a transfer learning (TL) method to improve PINN extrapolation. After training on the temporal domain [0,0.5] with hard-enforced initial/boundary conditions, the method samples 80 collocation points with the highest PDE residual in the validation interval (0.5,0.8], then fine-tunes only the final layer and the parameters of an adaptive activation function on those points, using L2 regularization to limit catastrophic forgetting. It also studies adaptive linear combinations of standard activation functions in the final layer. Experiments on the Allen-Cahn, Korteweg-de Vries, and viscous Burgers equations report relative L2 and MAE errors on the extrapolation interval (0.8,1], and the abstract claims an average 40% reduction in relative L2 error and 50% reduction in MAE. The code is released.

Significance. If the claimed gains are robust, the method would be a low-cost and practically useful way to improve PINN extrapolation, and the release of code plus the hard-constraint encoding are useful contributions. However, the central quantitative claim is currently not supported as stated because the headline numbers mix the effect of the transfer-learning step with a test-selected activation-function choice, and the same-activation-function TL gains are considerably smaller and sometimes negligible. The paper is transparent about its 1D, regular-domain scope, which is a strength, but the evaluation protocol needs to be reworked before the headline claims can be accepted.

major comments (4)
  1. [§5.1–5.3, Tables 1, 3, 5; Abstract] The headline 40%/50% reductions are computed by comparing the best per-equation AF with TL against vanilla tanh without TL, so the improvement conflates changing the activation function with the TL contribution. For the same AF, the with-/without-TL differences are smaller and in one case zero: AC lctanh L2 0.16 to 0.12, KdV lc(x + sin2(x)) L2 0.10 to 0.08, Burgers lctanh 0.07 to 0.06, and Burgers x + sin2(x) 0.08 to 0.08. Please report paired same-AF comparisons as the primary evidence for the TL benefit and revise the abstract and introduction claims accordingly.
  2. [§5.1, §5.2, §5.3] The choice of AF and of linear-combination size is made after evaluating extrapolation error on the test interval (0.8,1]; for example, the text reports that 'the best ABU-PINN was...' and that 'a linear combination of tanh provided the best extrapolation accuracy.' Since a practitioner solving a new PDE has no reference solution on the extrapolation interval, the reported best-case numbers are oracle estimates. Provide a selection rule based only on training/validation data, or report the average over all candidate AFs, and show the resulting average gain.
  3. [§4.2] The method's central premise is that the 80 highest-loss collocation points in the validation interval are the most informative points for fine-tuning, but the paper does not compare this selection against random or uniformly spaced points with the same budget. Without such an ablation, the contribution of the residual-based selection, as opposed to the TL fine-tuning itself, is not established.
  4. [§5, Tables 1, 3, 5] Results are reported as averages over ten runs with no standard deviations or per-run values. Some same-AF TL differences (e.g., Burgers x + sin2(x) L2 0.08 to 0.08) are within plausible run-to-run variability, so the reader cannot assess whether the TL gains are statistically meaningful. Add uncertainty estimates and, preferably, paired comparisons across seeds.
minor comments (5)
  1. [Appendix A, Table S1] The terms 'catastrophic forgetting rate' and 'extrapolation error reduction' are used without definitions; please define them and state exactly how they are computed.
  2. [Appendix D] The claim that altering the AF throughout the network can negatively affect convergence is supported by 'our findings (omitted)'; either include the supporting data or remove the unsupported assertion.
  3. [Appendix D, Figure S16] The layer-wise gradient analysis for the AC equation shows that the final three layers have the largest gradient norms, but the method freezes all but the final layer; the text does not explain why fine-tuning only the final layer rather than the last three follows from this figure.
  4. [Abstract, Table S2] The claim that the method adds no significant computational cost is supported only for the AC equation; please qualify the claim or provide timings for KdV and Burgers.
  5. [Section 3 and Section 4.2] The abstract describes TL as being applied 'within an extended training domain,' while Section 4.2 states that points are sampled exclusively from the validation interval; please align this terminology.

Circularity Check

1 steps flagged · score 4.0 of 10

The headline 40%/50% gain is an oracle estimate: the best activation function per PDE was chosen after measuring extrapolation error on the test interval, so the central quantitative claim partly reduces to test-set model selection; the TL mechanism itself remains independent.

  1. fitted input called prediction [Abstract; Section 5 (Experimental results) and Tables 1, 3, 5]
    "As mentioned previously, we evaluated the extrapolation performance of different AFs in the final layer while all other layers used tanh. ... Overall, a linear combination of tanh provided the best extrapolation accuracy while introducing a similar number of trainable parameters as other linear combinations. ... our method achieves an average of 40% reduction in relative L2 error and an average of 50% reduction in mean absolute error in the extrapolation domain."

    The 40%/50% headline is not a prediction for a fixed method: per PDE, the AF family and combination size were selected after measuring extrapolation error on (0.8,1] (Tables 1, 3, 5 report 'best extrapolation accuracy'), and the headline compares that test-selected best AF+TL to vanilla tanh without TL. The reported reduction is therefore an in-sample optimum over the extrapolation test distribution, not an out-of-sample result: the AF choice is fit to the same target quantity whose improvement is then claimed.

full rationale

The core TL procedure is not circular: the 80 high-loss collocation points are sampled from the validation interval (0.5,0.8], which is disjoint from the extrapolation interval (0.8,1]; retraining only the final layer is a standard transfer-learning heuristic; the AF weights are genuinely learned parameters; no uniqueness theorem is imported; and the citations to the authors' earlier work ([15], [27]) are not load-bearing. The circular component is confined to the headline metric: the AF family/combination size was selected after evaluating extrapolation performance on the test interval, and then the same extrapolation error is reported as the achieved reduction. Because same-AF with/without TL comparisons in Tables 1, 3, and 5 still show real, though smaller, gains, the paper has independent content; the score is set to 4 rather than higher.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The method contributes no new physical entities or conserved quantities. The improvement rests on a collection of tuned hyperparameters (regularization weights, number of transfer points, epochs, learning rates) and on three empirical domain assumptions (transferability of final-layer fine-tuning, informativeness of high-residual validation points, sufficiency of L2 regularization). The per-PDE choice of preferred activation function is a post-hoc selection on the test benchmarks, which is a free choice that inflates reported gains.

free parameters (7)
  • L2 regularization weight lambda (TL phase) = 0.01
    Tuned in supplementary A; reported as optimal across all PDEs tested.
  • EWC regularization weight lambda = 0.001
    Tuned in supplementary A; EWC was the worst TL method, so this parameter is not used in the main results.
  • TL collocation point count = 80
    Section 4.1: 80 points selected from 4k uniformly sampled candidates; chosen by hand.
  • TL training epochs = 120 to 150
    Section 4.1; the paper says 'an additional 120 to 150 epochs'.
  • TL learning rate = 5e-3 (AC), 5e-2 (KdV, Burgers)
    Section 4.1 lists range 1e-3 to 1e-2; per-equation values in Sections 5.1-5.3.
  • Preferred activation function per PDE = tanh for AC and Burgers; x + sin^2(x) for KdV
    Selected after evaluating extrapolation performance in Section 5; post-hoc choice baked into reported results.
  • Linear combination size = 3 for lctanh/lcsin (AC, Burgers); 2 for lc(x+sin^2) (KdV)
    Sections 5.1-5.3: 'the combination of three AFs was shown to give the best results'; selected post-hoc per PDE.
assumptions (5)
  • standard math The hard-constraint change of variables, e.g. u(t,x) = x^2 cos(pi x) + t(1-x^2)v(t,x), maps the original PDE to a well-posed PDE for v.
    Section 3 and supplementary Section B; exact algebraic transformation, but the well-posedness of the transformed PDE is assumed.
  • domain assumption Finite-difference/ode15s reference solutions with tolerances 1e-6 and 1e-8 are accurate enough for L2/MAE evaluation.
    Section 4.3; no convergence study for the reference solver is reported.
  • ad hoc to paper High-loss collocation points in the validation interval are the most informative points for fine-tuning toward the extrapolation interval.
    Sections 4.2 and 5; heuristic supported only by three empirical cases (Figures S1, S6, S11).
  • domain assumption Freezing all layers except the final one preserves learned dynamics while allowing adaptation.
    Section 3 and supplementary D; gradient-norm evidence is computed only for the AC equation with tanh AF.
  • domain assumption L2 regularization with lambda=0.01 prevents catastrophic forgetting without sacrificing extrapolation improvement.
    Supplementary A; empirical selection among vanilla TL, L2, and EWC.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving physics-informed neural network extrapolation via transfer learning and adaptive activation functions." pith.science (2026). https://pith.science/paper/WPLGH33T

@misc{pith2026250712659,
  author       = {Pith},
  title        = {Pith review of: Improving physics-informed neural network extrapolation via transfer learning and adaptive activation functions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WPLGH33T}},
  note         = {Machine review of arXiv:2507.12659}
}
read the original abstract

Physics-Informed Neural Networks (PINNs) are deep learning models that incorporate the governing physical laws of a system into the learning process, making them well-suited for solving complex scientific and engineering problems. Recently, PINNs have gained widespread attention as a powerful framework for combining physical principles with data-driven modeling to improve prediction accuracy. Despite their successes, however, PINNs often exhibit poor extrapolation performance outside the training domain and are highly sensitive to the choice of activation functions (AFs). In this paper, we introduce a transfer learning (TL) method to improve the extrapolation capability of PINNs. Our approach applies transfer learning (TL) within an extended training domain, using only a small number of carefully selected collocation points. Additionally, we propose an adaptive AF that takes the form of a linear combination of standard AFs, which improves both the robustness and accuracy of the model. Through a series of experiments, we demonstrate that our method achieves an average of 40% reduction in relative L2 error and an average of 50% reduction in mean absolute error in the extrapolation domain, all without a significant increase in computational cost. The code is available at https://github.com/LiuzLab/PINN-extrapolation .

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 24 canonical work pages

  1. [1]

    Abbasi and P.Ø

    J. Abbasi and P.Ø. Andersen. Physical activation functions: An approach for more efficient induction of physics into physics-informed neural networks. Neurocomputing 608 (2024)

    work page 2024

  2. [2]

    Alkhalifah and X

    T. Alkhalifah and X. Huang. Direct Imaging Using Physics Informed Neural Networks. 2022 IEEE International Conference on Image Processing (ICIP) (2022), 2781–2785

    work page 2022

  3. [3]

    Borate et al

    P. Borate et al. Using a physics-informed neural network and fault zone acoustic monitoring to predict lab earthquakes. Nat. Commun. 14 (2023)

    work page 2023

  4. [4]

    Byrd et al

    R.H. Byrd et al. A Limited Memory Algorithm for Bound Constrained Optimization. SIAM J. Sci. Comput. 16 (1995), pp. 1190–1208

    work page 1995

  5. [5]

    Cai et al

    S. Cai et al. Physics-informed neural networks for fluid mechanics: a review. Acta Mech. Sin. 37 (2021), pp. 1727– 1738

    work page 2021

  6. [6]

    Desai et al

    S. Desai et al. One-Shot Transfer Learning of Physics-Informed Neural Networks. arXiv e-prints 2110.11286v2 (2021)

    arXiv 2021

  7. [7]

    Faroughi et al

    S.A. Faroughi et al. Physics-Informed Neural Networks with Periodic Activation Functions for Solute Transport in Heterogeneous Porous Media. Mathematics 12.1 (2024)

    work page 2024

  8. [8]

    Fesser, L.D’Amico-Wong, and R

    L. Fesser, L.D’Amico-Wong, and R. Qiu. Understanding and mitigating extrapolation failures in physics-informed neural networks. arXiv e-prints 2306.09478 (2023)

    arXiv 2023

Show all 27 references
  1. [9]

    Jagtap, K

    A. Jagtap, K. Kawaguchi, and G. Karniadakis. Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks. Proc. R. Soc. A (2020)

    2020

  2. [10]

    Jalili et al

    D. Jalili et al. Transfer learning through physics-informed neural networks for bubble growth in superheated liquid domains. Int. J. Heat Mass Transf. 232 (2024), pp. 125940

    2024

  3. [11]

    Kapoor et al

    T. Kapoor et al. Transfer learning for improved generalizability in causal physics-informed neural networks for beam simulations. Eng. Appl. Artif. Intell. 133 (2024), pp. 108085

    2024

  4. [12]

    Kim et al

    J. Kim et al. DPM:A Novel Training Method for Physics-Informed Neural Networks in Extrapolation. 35th AAAI Conference on Artificial Intelligence (2021), 8146–8154

    2021

  5. [13]

    Kirkpatrick et al

    J. Kirkpatrick et al. Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114 (2016), pp. 3521–3526

    2016

  6. [14]

    Kissas et al

    G. Kissas et al. Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non- invasive 4D flow MRI data using physics-informed neural networks. Comput. Methods Appl. Mech. Eng. 358 (2020), pp. 112623

    2020

  7. [15]

    Z. Liu, H. Tilman, and U. Masahito. Neural Networks Fail to Learn Periodic Functions and How to Fix It. Adv. Neural Inf. Process. Syst. 33 (2020). 7

    2020

  8. [16]

    L. D. McClenny and U.M. Braga-Neto. Self-adaptive physics-informed neural networks. Journal of Computational Physics 474 (2023), 111722

    2023

  9. [17]

    A. H. Mustajab et al. Physics-Informed Neural Networks for High-Frequency and Multi-Scale Problems Using Trans- fer Learning. Appl. Sci. 14.8 (2024), pp. 3204

    2024

  10. [18]

    Penwarden et al

    M. Penwarden et al. A unified scalable framework for causal sweeping strategies for physics-informed neural networks (PINNs) and their temporal decompositions. J. Comput. Phys. 493 (2023)

    2023

  11. [19]

    Prantikos et al

    K. Prantikos et al. Physics-informed neural network with transfer learning (TL-PINN) based on domain similarity measure for prediction of nuclear reactor transients. Sci Rep 13.1 (2023), pp. 16840

    2023

  12. [20]

    Raissi, P

    M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378 (2019), pp. 686–707

    2019

  13. [21]

    Shima et al

    A. Shima et al. Modeling unobserved geothermal structures using a physics-informed neural network with transfer learning of prior knowledge. Geotherm Energy 12.1 (2024), pp. 38

    2024

  14. [22]

    Learning Specialized Activation Functions for Physics-informed Neural Networks

    Honghui Wang et al. Learning Specialized Activation Functions for Physics-informed Neural Networks. arXiv e-prints 2308.04073 (2023)

    2023 arXiv

  15. [23]

    Y . Wang, Y . Yao, and Z. Gao. An extrapolation-driven network architecture for physics-informed deep learning.Neural Netw. 183 (2025), pp. 106998

    2025

  16. [24]

    Wight and J

    C.L. Wight and J. Zhao. Solving Allen-Cahn and Cahn-Hilliard equations using the adaptive physics informed neural networks. Commun. Comput. Phys. 29 (2021), pp. 930–954

    2021

  17. [25]

    Wu et al

    C. Wu et al. A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks. Comput. Methods Appl. Mech. Eng. 403 (2023), pp. 115671

    2023

  18. [26]

    Zapf et al

    B. Zapf et al. Investigating molecular transport in the human brain from MRI with physics-informed neural networks. Sci. Rep. 12 (2022)

    2022

  19. [27]

    J. Zhu, Y . Xue, and Z. Liu. A transfer learning enhanced physics-informed neural network for parameter identification in soft materials. Appl. Math. Mech.-Engl. Ed. 45.10 (2024), pp. 1685–1704. 8 Supplementary materials A Comparison of TL Methods In this study, we examined th...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.