Pith. sign in

REVIEW 3 major objections 4 minor 34 references

Low-rank adaptive physics-informed HyperDeepONets for solving differential equations

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A low-rank factorization makes HyperDeepONets up to 70% cheaper while matching or beating full models on differential-equation benchmarks.

desk verdict A useful incremental result: applying LoRA to HyperDeepONets cuts parameters and often improves accuracy, but the 'regularization' story needs a controlled capacity baseline. read the letter →

arxiv 2507.18346 v1 pith:XBPQCMX2 submitted 2025-07-24 cs.LG cs.NAmath.NA

Etienne Zeudong , Elsa Cardoso-Bihlo , Alex Bihlo This is my paper
classification cs.LGcs.NAmath.NA
keywords HyperDeepONetlow-rankadaptationphysics-informedneuralnetworksoperatorlearningdifferentialequationsparameterefficiencyimplicitregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the main cost of HyperDeepONets—a hypernetwork that generates all weights of the trunk network—can be removed by low-rank factorization without losing accuracy. Replacing the hypernetwork's output weight matrix $W^{\text{branch}}_{\text{out}}$ by a product $W^{1}_{\text{LoRA}} W^{2}_{\text{LoRA}}$ with rank $r$ smaller than the hidden width cuts trainable parameters by up to 70% and, in the physics-informed setting, acts as an implicit regularizer. The authors test this on two ordinary differential equation systems (harmonic oscillator, rigid body) and three partial differential equations (advection, Burgers, shallow-water), training without paired data by minimizing differential-equation residuals. Across these benchmarks the low-rank variants match or beat both full HyperDeepONets and standard DeepONets, with lower errors per parameter and better long-time iterative predictions. The practical stake is that operator learning for differential equations can be made substantially cheaper and more stable than the full HyperDeepONet design suggests.

What carries the argument

The load-bearing mechanism is the low-rank factorization of the hypernetwork's final weight matrix, $W^{\text{branch}}_{\text{out,LoRA}} = W^{1}_{\text{LoRA}} W^{2}_{\text{LoRA}}$ with $r < n_{\text{hidden}}$, applied inside a physics-informed DeepONet whose trunk weights are generated entirely by the branch net. It carries the argument by cutting the dominant parameter block from $n_{\text{trunk}}\times n_{\text{hidden}}$ to $r(n_{\text{trunk}}+n_{\text{hidden}})$ and by introducing correlations among generated trunk weights that serve as a regularizer. The physics-informed loss then measures PDE residuals at collocation points, with periodic and initial conditions enforced as hard constraints in most experiments.

What would settle it

Train a full HyperDeepONet on a multi-scale or turbulent benchmark, compute the singular-value decomposition of its trained output weight matrix $W^{\text{branch}}_{\text{out}}$, and show that the singular values decay slowly with no small-rank cutoff; then a PI-LoRA-HyperDeepONet with the same trunk width should fail to match the full model at any rank $r$ below $n_{\text{hidden}}$.

Watch

Extended reading notes

Core claim

In the physics-informed setting, the paper's central claim is that the enormous output layer of a HyperDeepONet's branch net is not needed at full rank. If $W^{\text{branch}}_{\text{out}} \in \mathbb{R}^{n_{\text{trunk}}\times n_{\text{hidden}}}$ is replaced by $W^{1}_{\text{LoRA}} W^{2}_{\text{LoRA}}$ with $W^{1}_{\text{LoRA}}\in\mathbb{R}^{n_{\text{trunk}}\times r}$, $W^{2}_{\text{LoRA}}\in\mathbb{R}^{r\times n_{\text{hidden}}}$ and $r<n_{\text{hidden}}$, the parameter count drops from $n_{\text{trunk}}n_{\text{hidden}}$ to $r(n_{\text{trunk}}+n_{\text{hidden}})$. The factorization couples all trunk weights, which the authors identify as an implicit regularization that also appears to simplify the physics-informed loss surface. On the five benchmark equations, the low-rank versions consistently match or outperform the full HyperDeepONet and the standard DeepONet; for example, a rank-4 LoRA-HyperDeepONet reaches lower one-step and 10-step errors than both baselines for the harmonic oscillator and rigid body with less than 30% of their parameters. The authors conclude that constraining the hypernetwork output to low rank is not a compromise but a benefit in this setting, though the best rank is problem-dependent.

Load-bearing premise

The central assumption is that the weight matrix the hypernetwork must produce has low effective rank; if a target problem genuinely needs many independent trunk-weight directions, the low-rank bottleneck could lose accuracy, and the reported gains would not transfer.

Editorial extensions

If this is right

  • Training a physics-informed operator network needs far less memory: up to 70% fewer trainable parameters, so larger trunk nets or longer training runs fit on the same hardware.
  • Lower rank can improve generalization, including iterative rollout: the rank-4 harmonic-oscillator model reaches a 10-step $l^2$-error of $0.045$ versus $0.075$ for the full HyperDeepONet.
  • The best rank varies by equation, with small ranks around 2–4 for ODEs and larger ranks around 8–32 for PDEs, so rank tuning is a genuine hyperparameter.
  • Full-rank generation of trunk weights is not necessary for these benchmarks; a low-rank bottleneck can match or beat it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the benefit is a genuine low-rank inductive bias, a testable extension is to make the rank adaptive during training, letting the optimizer grow or shrink $r$ with the apparent complexity of the solution operator; the paper lists this as future work.
  • The same factorization could be applied to other hypernetwork-generated weights, including biases and hidden layers, and to chunked HyperDeepONets, potentially removing the extra hyperparameters and overhead that chunking introduced.
  • A likely boundary condition for the method is that on problems whose solution operators require high-rank trunk-weight matrices, such as strongly turbulent or multi-scale flows, the low-rank approximation may lose capacity; measuring the singular-value spectrum of a trained full HyperDeepONet would indicate where this limit lies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes PI-LoRA-HyperDeepONets, a modification of HyperDeepONets in which the hypernetwork's output weight matrix is replaced by a product of two low-rank matrices, Eq. (3). The authors apply this to physics-informed operator learning and report experiments on the harmonic oscillator, rigid body equations, linear advection, viscous Burgers, and shallow-water equations. They claim up to a 70% reduction in trainable parameters while matching or improving predictive accuracy and generalization relative to full HyperDeepONets and standard DeepONets. The experimental setup uses 10 seeds, hard/soft initial-condition constraints, exact or spectral-RK reference solutions, and one-step and iterative evaluation.

Significance. If the central claim holds, the paper offers a simple and practical way to reduce the dominant parameter cost of HyperDeepONets and extends HyperDeepONets to the physics-informed setting, which had not been done in the original data-driven proposal. The authors should be credited for a reasonable experimental protocol: repeated runs with 10 seeds, different constraint formulations, exact or high-fidelity references, and parameter parity between standard DeepONets and full HyperDeepONets. The reported parameter reductions are real arithmetic consequences of Eq. (3). The main weakness is that the accuracy/generalization improvement is claimed on the basis of per-benchmark selection of the best LoRA rank, without a matched-capacity baseline that would isolate the effect of low-rank factorization from generic parameter reduction. These issues are fixable with additional experiments and statistical reporting.

major comments (3)
  1. [Section 4, Tables 1-5; Section 5] The headline claim that LoRA-HyperDeepONets 'consistently outperform' full HyperDeepONets rests on selecting, for each benchmark, the best LoRA rank from a small sweep after observing test errors. Section 5 explicitly acknowledges that the optimal rank differs for each equation. Since no validation-based rank-selection rule or correction for multiple comparisons is provided, the reported improvements are not established as a property of the method. For example, in Table 2 the rank-16 variant is not better than the full HyperDeepONet (one-step 0.0012 vs 0.0012; 10-step 0.0120 vs 0.0110), and several differences in Tables 1 and 4 are within one standard deviation.
  2. [Section 3.3 and Section 5] The paper attributes the accuracy gains to an 'extra regularization' arising specifically from the low-rank factorization, but no experiment controls for the reduction in the number of effective parameters. A natural control is a full HyperDeepONet whose branch network has a reduced last-hidden-layer width chosen to match the total parameter count of the LoRA variant. Without such a baseline, the claim that the low-rank structure itself, rather than a smaller parameter budget, improves accuracy and generalization is not supported. This is a load-bearing point because the abstract and conclusions present the accuracy improvement as a central benefit, not just the parameter reduction.
  3. [Section 4, experimental statistics] All comparisons are based on 10 seeds and reported as mean l2-error plus/minus standard deviation, with no significance tests or confidence intervals. Given that the best rank is selected from the reported sweep, the probability of at least one spurious 'improvement' across the five tables is nontrivial. Reporting paired tests across seeds, or at least error bars with a correction for the number of ranks compared, would substantially strengthen the consistency claim. Otherwise the statement 'consistently outperform' is stronger than the statistical evidence supports.
minor comments (4)
  1. [Section 3.3] The name 'LoRA' is potentially misleading: Eq. (3) replaces the hypernetwork output matrix with a low-rank factorization, whereas the original LoRA formulation adds a low-rank update to a frozen pretrained weight matrix. The paper should clarify that no pretrained base matrix is used and that this is a low-rank reparameterization rather than adaptation of an existing weight.
  2. [Section 4] The paper claims reduced memory and training costs, but only parameter counts are reported. No wall-clock training time, memory footprint, or FLOPs measurements are given, so the computational-cost claim is not directly evidenced.
  3. [General] There are several typos and inconsistencies, for example 'hyperbolic activation functions is used' in Section 4, 'LoRa-HyperDeepoNets' in Section 4.4, and 'closesly' in the same section. Also, the abstract cites Lee, Cho and Hwang as 'ICLR 2023' while reference [22] is listed as arXiv:2312.15949, 2023 with no venue; please reconcile.
  4. [Section 4.1] The statement that 'all the LoRA-HyperDeepONet can outperform these networks' in the harmonic oscillator discussion is slightly ambiguous because it refers to both one-step and 10-step errors; the table shows the rank-2 and rank-6 variants have larger one-step errors than the rank-4 variant, so 'all' is only true relative to the DeepONet and full HyperDeepONet, not within the LoRA family.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: Equation (3) is an architectural definition and all reported errors are measured against external exact/numerical references; the only caveats are test-set rank selection and a partially overstrong attribution of gains to low-rank regularization, neither of which reduces the claim to its inputs.

full rationale

The paper's central claim is that replacing the hypernetwork output matrix Wbranch_out by the product W1_Lora W2_Lora (Eq. 3) reduces parameters and improves accuracy in physics-informed operator learning. This is not a derivation from, nor a prediction of, the method's own outputs: the reduction in parameter count is an arithmetic identity for the factorization, while the accuracy claims are evaluated against exact solutions (harmonic oscillator, rigid body) and high-fidelity spectral/RK reference solutions (advection, Burgers, shallow water) on held-out initial conditions, independent of the training loss. No fitted parameter is renamed as a prediction; no equation is equivalent by construction to another. The paper's self-citations (e.g., [4], [8], [9]) are used for standard components such as periodic hard constraints and conservative formulations, and none carries the load of the central claim. The most defensible caveat is that the rank r is tuned per benchmark and the best rank differs across equations (Section 5), meaning reported 'consistently outperform' results partly reflect favorable test-set hyperparameter selection; this is a data-dependent tuning concern, not circular reasoning. A second caveat is Section 5's attribution of the gains to low-rank regularization without a matched-parameter full HyperDeepONet baseline, which is a missing-control issue rather than a reduction of the conclusion to the method's inputs. Under the stated rules, the paper is self-contained against external benchmarks and receives a low circularity score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The empirical contribution rests on standard operator approximation theory and PI training assumptions; no new entities are introduced. The main tunable is the LoRA rank, chosen per benchmark, which is the primary source of potential bias. The direct LoRA vs full-HyperDeepONet comparison is fair because the architectures differ only in the output-layer factorization.

free parameters (3)
  • LoRA rank r = r=4 (harmonic), r=4 (rigid body), r=8 (advection), r=32 (Burgers), r=16 (shallow-water)
    Selected by parameter sweep per benchmark; the best-performing rank is reported in Tables 1-5.
  • Initial-condition loss weight lambda_I (Burgers) = 100
    Set for the soft initial-condition enforcement in the Burgers equation example (Section 4.4).
  • Architecture sizes (layer depths and widths) = varies per model (e.g., DeepONet branch 4x80, trunk 2x10 for harmonic oscillator)
    Chosen to achieve approximate parameter parity between DeepONet and HyperDeepONet and inspired by the architectures in Lee et al. [22].
assumptions (3)
  • standard math Universal approximation theorem for operators (Chen and Chen 1995)
    Justifies representing the solution operator with a DeepONet/HyperDeepONet architecture; cited in Section 1.
  • domain assumption Minimizing the physics-informed loss over collocation points yields accurate solution operators for the considered IBVPs
    The paper relies on the standard PI training paradigm without rigorous convergence guarantees (Section 3.1).
  • domain assumption Hard constraints exactly enforce initial and boundary conditions, and the training IC class covers the test ICs
    Used in most examples; test initial conditions are drawn from the same distribution as training (Section 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Low-rank adaptive physics-informed HyperDeepONets for solving differential equations." pith.science (2026). https://pith.science/paper/XBPQCMX2

@misc{pith2026250718346,
  author       = {Pith},
  title        = {Pith review of: Low-rank adaptive physics-informed HyperDeepONets for solving differential equations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XBPQCMX2}},
  note         = {Machine review of arXiv:2507.18346}
}
read the original abstract

HyperDeepONets were introduced in Lee, Cho and Hwang [ICLR, 2023] as an alternative architecture for operator learning, in which a hypernetwork generates the weights for the trunk net of a DeepONet. While this improves expressivity, it incurs high memory and computational costs due to the large number of output parameters required. In this work we introduce, in the physics-informed machine learning setting, a variation, PI-LoRA-HyperDeepONets, which leverage low-rank adaptation (LoRA) to reduce complexity by decomposing the hypernetwork's output layer weight matrix into two smaller low-rank matrices. This reduces the number of trainable parameters while introducing an extra regularization of the trunk networks' weights. Through extensive experiments on both ordinary and partial differential equations we show that PI-LoRA-HyperDeepONets achieve up to 70\% reduction in parameters and consistently outperform regular HyperDeepONets in terms of predictive accuracy and generalization.

Figures

Figures reproduced from arXiv: 2507.18346 by the authors.

Figure 1
Figure 1. Illustration of the architectures of a full HyperDeepONet [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. A case study for the harmonic oscillator [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. A case study for the rigid body equations [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: A case study for the linear advection equation [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: A case study for the viscous Burgers equation [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: A case study for the one-dimensional shallow-water equations [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 31 canonical work pages

  1. [1]

    Arora, A

    S. Arora, A. Bihlo, and F. Valiquette. Invariant physics-informed neural networks for ordinary differential equations. J. Mach. Learn. Res., 25(233):1–24, 2023

    work page 2023

  2. [2]

    K. Bi, L. Xie, H. Zhang, X. Chen, X. Gu, and Q. Tian. Accurate medium-range global weather forecasting with 3D neural networks. Nature, 619(7970):533–538, 2023

    work page 2023

  3. [3]

    A. Bihlo. Improving physics-informed neural networks with meta-learned optimization. J. Mach. Learn. Res., 24:1–26, 2024

    work page 2024

  4. [4]

    Bihlo and R

    A. Bihlo and R. O. Popovych. Physics-informed neural networks for the shallow-water equations on the sphere. J. of Comput. Phys., 456:111024, 2022

    work page 2022

  5. [5]

    Bradbury, R

    J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018

    2018

  6. [6]

    Brecht and A

    R. Brecht and A. Bihlo. M-ENIAC: A machine learning recreation of the first successful numerical weather forecasts. Geophys. Res. Lett., 51:e2023GL107718, 2024

    work page 2024

  7. [7]

    Brecht, E

    R. Brecht, E. Cardoso-Bihlo, and A. Bihlo. Physics-informed neural networks for tsunami inundation modeling. J. Comput. Phys., page 114066, 2025

    work page 2025

  8. [8]

    Improving physics-informed DeepONets with hard constraints

    R. Brecht, D. R. Popovych, A. Bihlo, and R. O. Popovych. Improving physics-informed DeepONets with hard constraints. arXiv preprint arXiv:2309.07899, 2023

    work page Pith review arXiv 2023

Show all 34 references
  1. [9]

    Cardoso-Bihlo and A

    E. Cardoso-Bihlo and A. Bihlo. Exactly conservative physics-informed neural networks and deep operator networks for dynamical systems. Neural Netw., 181:106826, 2025. 13

    2025

  2. [10]

    Chen and H

    T. Chen and H. Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Trans. Neural Netw., 6(4):911–917, 1995

    1995

  3. [11]

    Cuomo, V

    S. Cuomo, V . S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli. Scientific machine learning through physics–informed neural networks: where we are and what’s next. J. Sci. Comput., 92(3):88, 2022

    2022

  4. [12]

    D. R. Durran. Numerical methods for fluid dynamics: With applications to geophysics, volume 32. Springer Science & Business Media, 2010

    2010

  5. [13]

    D. Ha, A. M. Dai, and Q. V . Le. Hypernetworks. InInternational Conference on Learning Representations, 2017

    2017

  6. [14]

    P. S. Hadorn. Shift-Deeponet: Extending deep operator networks for discontinuous output functions. ETH Zurich, Seminar for Applied Mathematics, 2022

    2022

  7. [15]

    Hairer, C

    E. Hairer, C. Lubich, and G. Wanner. Geometric numerical integration: structure-preserving algorithms for ordinary differential equations. Springer, Berlin, 2006

    2006

  8. [16]

    D. D. Holm, T. Schmah, and C. Stoica. Geometric mechanics and symmetry: from finite to infinite dimensions, volume 12. Oxford University Press, 2009

    2009

  9. [17]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  10. [18]

    Kochkov, J

    D. Kochkov, J. Yuval, I. Langmore, P. Norgaard, J. Smith, G. Mooers, M. Klöwer, J. Lottes, S. Rasp, P. Düben, et al. Neural general circulation models for weather and climate. Nature, 632(8027):1060–1066, 2024

    2024

  11. [19]

    N. B. Kovachki, S. Lanthaler, and A. M. Stuart. Operator learning: Algorithms and analysis. Handbook of Numerical Analysis, 25:419–467, 2024

    2024

  12. [20]

    Krishnapriyan, A

    A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney. Characterizing possible fail- ure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548–26560, 2021

    2021

  13. [21]

    I. E. Lagaris, A. Likas, and D. I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Trans. Neural Netw., 9(5):987–1000, 1998

    1998

  14. [22]

    J. Y . Lee, S. W. Cho, and H. J. Hwang. HyperDeepONet: learning operator with complex target function space using the limited resources via hypernetwork. arXiv:2312.15949, 2023

    2023 arXiv

  15. [23]

    Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021

    2021

  16. [24]

    Z. Li, H. Zheng, N. Kovachki, D. Jin, H. Chen, B. Liu, K. Azizzadenesheli, and A. Anandkumar. Physics- informed neural operator for learning partial differential equations. ACM/JMS Journal of Data Science, 1(3):1–27, 2024

    2024

  17. [25]

    L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nat. Mach. Intell., 3(3):218–229, 2021

    2021

  18. [26]

    McClenny and U

    L. McClenny and U. Braga-Neto. Self-adaptive physics-informed neural networks using a soft attention mechanism. J. Comput. Phys., 474:111722, 2023

    2023

  19. [27]

    P. J. Olver. Application of Lie groups to differential equations. Springer, New York, 2000

    2000

  20. [28]

    Raissi, P

    M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys., 378:686–707, 2019

    2019

  21. [29]

    Seidman, G

    J. Seidman, G. Kissas, P. Perdikaris, and G. J. Pappas. NOMAD: Nonlinear manifold decoders for operator learning. Advances in Neural Information Processing Systems, 35:5601–5613, 2022

    2022

  22. [30]

    Venturi and T

    S. Venturi and T. Casey. SVD perspectives for augmenting DeepONet flexibility and interpretability. Comput. Methods Appl. Mech. Eng., 403:115718, 2023

    2023

  23. [31]

    Wang and P

    S. Wang and P. Perdikaris. Long-time integration of parametric evolution equations with physics-informed DeepONets. J. Comput. Phys., 475:111855, 2023

    2023

  24. [32]

    S. Wang, S. Sankaran, and P. Perdikaris. Respecting causality is all you need for training physics-informed neural networks. arXiv preprint arXiv:2203.07404, 2022

    2022 arXiv

  25. [33]

    S. Wang, H. Wang, and P. Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Sci. Adv., 7(40):eabi8605, 2021

    2021

  26. [34]

    S. Wang, X. Yu, and P. Perdikaris. When and why PINNs fail to train: a neural tangent kernel perspective. J. Comput. Phys., 449:110768, 2022. 14

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.