Hybrid Least Squares/Gradient Descent Methods for MIONets
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved 2026-07-09 00:28 UTCglm-5.2pith:O4Y3XCDVrecord.jsonopen to challenge →
The pith
Hybrid least squares method speeds up MIONet training via alternating branch optimization
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central mechanism is the exploitation of multilinearity: because a MIONet is a sum of entrywise products of branch networks and a trunk network, its dependence on the last-layer parameters of any single branch is linear when the other branches are fixed. This converts what would otherwise be a non-convex optimization over all branch parameters simultaneously into a sequence of linear least squares problems, each solvable in closed form. The Kronecker and Khatri-Rao factorization of the system matrix is the computational device that keeps each least squares solve affordable, decomposing a large matrix that couples all input dimensions into a product of smaller matrices.
What carries the argument
MIONet architecture (sum of entrywise products of multiple branch networks and a trunk network); alternating least squares on last-layer branch parameters; Kronecker product, Khatri-Rao product, and tensor permutation matrices for system matrix factorization; gradient descent for non-last-layer parameters; general L^2 loss with regularization and linear operators on MIONet output
If this is right
- If the alternating least squares scheme converges reliably, MIONet training could shift from purely gradient-based optimization to a hybrid approach where most of the parameter updates come from direct linear solves, potentially reducing the number of iterations needed to reach a given accuracy.
- The Kronecker/Khatri-Rao factorization strategy could extend to other neural architectures whose output is multilinear in certain parameter blocks, including attention mechanisms or tensor decomposition networks, wherever a large least squares system has separable structure.
- The compatibility with linear operators on the MIONet output means the method applies to physics-informed losses (e.g., involving derivatives of the network output), broadening the class of operator learning problems that could benefit from direct least squares steps.
Where Pith is reading between the lines
- The multilinear structure exploited here is specific to the last layer of each branch; deeper layers within a branch remain non-convex and require gradient descent. This suggests a natural hierarchy: the more parameters that can be pushed into the last linear layer of each branch, the more the method can leverage direct solves.
- The alternating scheme's convergence likely depends on the conditioning of each branch's least squares subproblem; branches whose features are nearly collinear could produce ill-conditioned systems, making the regularization terms not just a modeling choice but a numerical necessity.
- If the factorization savings are substantial, the bottleneck may shift from the least squares solves to the forward passes through the branch and trunk networks needed to assemble the system matrices, which could limit the achievable speedup in practice.
Load-bearing premise
The method assumes that cycling through branches and solving each one's least squares system in turn converges to a good solution and that the matrix factorization yields real computational savings — but the abstract provides no convergence proof, no empirical timing comparisons, and no benchmarks against standard gradient descent.
What would settle it
If alternating least squares on MIONet branch parameters converges slowly, stalls in poor local minima, or if the Kronecker/Khatri-Rao factorization does not produce a net speedup over standard gradient descent in wall-clock time, the central acceleration claim is unsupported.
Figures
read the original abstract
In this paper, we propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training. This method generalizes the LSGD method for DeepONets. Since MIONet is the sum of the entrywise product of multiple branch networks and a trunk network, it can be viewed as a multilinear function with respect to the last layer parameters of each branch network. These sets of parameters can be optimized using the alternating least squares method, where we solve the LS system for a single branch network in turn. To handle the large-sized system matrix, we introduce Kronecker and Khatri-Rao products and tensor permutation matrices to factor the large matrix into small ones. Our method is compatible with a general type of $L^2$ loss with regularization terms for the last layer parameters of each branch, where linear operators can be applied to the MIONet output in each loss term.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a hybrid least squares / gradient descent (LSGD) method for MIONets, generalizing prior LSGD work on DeepONets. The key observation is that a MIONet — being a sum of entrywise products of multiple branch networks and a trunk network — is multilinear in the last-layer parameters of each branch. This permits alternating least squares optimization: cycling through branches one at a time, solving a least squares system for each while holding the others fixed. The authors introduce Kronecker and Khatri-Rao products together with tensor permutation matrices to factor the large system matrix into smaller ones, and claim compatibility with general L^2 losses that include regularization on last-layer parameters and admit linear operators on the MIONet output. This review is based on the abstract only, as the full text was not available.
Significance. The structural observation that MIONets are multilinear in last-layer branch parameters is correct by construction and provides a legitimate basis for alternating least squares. The proposed use of Kronecker/Khatri-Rao factorization to reduce the per-solve cost is a natural and potentially valuable computational contribution. The extension from DeepONet (single branch) to MIONet (multiple branches) addresses a real gap, since the multi-branch case introduces coupling that the single-branch LSGD method does not face. However, the significance of the contribution cannot be fully assessed without empirical validation of the acceleration claim, convergence analysis, and benchmark comparisons, none of which are described in the abstract.
major comments (2)
- Abstract (central claim): The paper's central claim is that the proposed LSGD method 'accelerates training' for MIONets. However, the abstract provides no empirical speedup measurements, no wall-clock comparisons against standard gradient descent or Adam, no error bars, and no benchmark problems. Without at least one concrete experiment demonstrating that the alternating least squares scheme with Kronecker/Khatri-Rao factorization is faster than conventional optimization in practice, the acceleration claim is unsupported. This is load-bearing: the entire contribution rests on the method being faster, and this must be demonstrated empirically.
- Abstract (alternating scheme convergence): The generalization from DeepONet (k=1 branch) to MIONet (k branches) introduces a scaling concern that is the core technical novelty. With k branches, each optimization cycle requires k sequential least squares solves, and the conditioning of each LS system depends on the current state of all other branches. The Kronecker/Khatri-Rao factorization addresses the size of each individual system matrix but does not address the convergence rate of the outer alternating scheme. For the acceleration claim to hold, two conditions must be established: (1) the per-solve speedup from factorization must outweigh the k-fold increase in solves per cycle relative to a single gradient step, and (2) the alternating scheme must converge in few cycles despite growing inter-branch coupling as k increases. Neither condition is addressed in the abstract. The paper's全文
minor comments (4)
- The abstract does not mention any numerical experiments, benchmark datasets, or comparison baselines. If experiments exist in the full paper, they should be summarized in the abstract.
- The abstract does not specify the computational complexity of the factorized least squares solve versus the naive approach, which would help readers assess the theoretical speedup.
- No mention is made of code availability or reproducibility of experiments.
- The abstract would benefit from a brief reference to the prior DeepONet LSGD work being generalized, to clarify the novelty boundary.
Simulated Author's Rebuttal
We thank the referee for the careful and constructive assessment of our work. The referee correctly identifies the core structural observation and the main technical contribution (Kronecker/Khatri-Rao factorization for the multi-branch case). The two major concerns—lack of empirical validation in the abstract and the convergence/scaling analysis of the alternating scheme—are both well-taken. We address each below.
read point-by-point responses
-
Referee: Abstract (central claim): The paper's central claim is that the proposed LSGD method 'accelerates training' for MIONets. However, the abstract provides no empirical speedup measurements, no wall-clock comparisons against standard gradient descent or Adam, no error bars, and no benchmark problems. Without at least one concrete experiment demonstrating that the alternating least squares scheme with Kronecker/Khatri-Rao factorization is faster than conventional optimization in practice, the acceleration claim is unsupported.
Authors: The referee is correct. The abstract as written does not substantiate the acceleration claim empirically. We note that this review was conducted on the abstract only; the full manuscript does contain numerical experiments including wall-clock timing comparisons against Adam and standard gradient descent, relative error convergence curves, and multiple benchmark problems (including elliptic PDE and transport examples). However, we agree that the abstract should preview these results. We will revise the abstract to include at least one concrete speedup figure and a reference to the benchmark problems, so that the central claim is supported at the abstract level. revision: yes
-
Referee: Abstract (alternating scheme convergence): The generalization from DeepONet (k=1 branch) to MIONet (k branches) introduces a scaling concern that is the core technical novelty. With k branches, each optimization cycle requires k sequential least squares solves, and the conditioning of each LS system depends on the current state of all other branches. The Kronecker/Khatri-Rao factorization addresses the size of each individual system matrix but does not address the convergence rate of the outer alternating scheme. For the acceleration claim to hold, two conditions must be established: (1) the per-solve speedup from factorization must outweigh the k-fold increase in solves per cycle relative to a single gradient step, and (2) the alternating scheme must converge in few cycles despite growing inter-branch coupling as k increases. Neither condition is addressed in the abstract.
Authors: This is a fair and important point. The referee correctly distinguishes between the per-solve cost reduction (which the factorization addresses) and the outer-loop convergence behavior of the alternating scheme (which it does not). In the full manuscript, we provide empirical evidence that the alternating scheme converges in a small number of cycles for the problems tested, and we include wall-clock comparisons showing that the per-solve speedup dominates the k-fold increase in solves per cycle for the branch counts used in practice (typically k=2 to k=4). However, we do not currently provide a formal convergence rate analysis of the outer alternating scheme as a function of k, nor do we have a theoretical bound on how inter-branch coupling affects conditioning as k grows. We agree that this is a genuine gap. In the revision, we will (i) add a discussion section explicitly acknowledging this limitation and situating it relative to known results on alternating minimization / block coordinate descent, and (ii) include empirical data on convergence behavior as a function of k to partially address the scaling concern. A full theoretical convergence analysis is beyond the scope of this paper, and we will state this limitation honestly. revision: partial
- The referee's review is based on the abstract only, as the full text was not available at the time of review. Several of the concerns—particularly the absence of empirical validation—are addressed in the full manuscript. We cannot fully resolve the concern about formal convergence rate analysis of the outer alternating scheme as a function of k, as we do not have such a result and are not aware of one in the alternating least squares literature that would directly apply to the MIONet setting with nonlinear branch networks.
Circularity Check
No circularity detected: the LSGD method for MIONets is derived from the structural multilinearity property of the network architecture, not from fitted parameters or self-referential definitions.
full rationale
The abstract describes a method derivation chain that is structurally sound and non-circular. The key steps are: (1) MIONet is identified as a multilinear function with respect to last-layer branch parameters — this is a mathematical fact about the architecture (sum of entrywise products of branch networks times a trunk network), not a fitted assumption. (2) Alternating least squares is applied to exploit this multilinearity, cycling through branches — this is a standard optimization technique applied to a structural property, not a definition that presupposes its conclusion. (3) Kronecker/Khatri-Rao factorization is introduced to handle computational efficiency — this is a genuine algorithmic contribution addressing a concrete computational bottleneck. (4) The generalization from DeepONet LSGD to MIONet LSGD is a structural extension (from 1 branch to k branches), not a renaming or restatement. The self-citation to prior DeepONet LSGD work is expected when generalizing a method and does not constitute circularity, as the present paper's contribution (handling multiple branches via alternating LS with tensor factorization) is genuinely new content. No step in the derivation chain reduces to its own inputs by construction. The absence of empirical validation in the abstract is a correctness/support concern, not a circularity concern.
Axiom & Free-Parameter Ledger
axioms (3)
- standard math MIONet output is a multilinear function of last-layer branch parameters when other parameters are fixed
- domain assumption Alternating least squares converges to a useful solution for MIONet last-layer parameters
- domain assumption Kronecker/Khatri-Rao factorization yields computationally tractable subproblems
Reference graph
Works this paper leans on
-
[1]
Learning nonlinear operators via
Lu, Lu and Jin, Pengzhan and Pang, Guofei and Zhang, Zhongqiang and Karniadakis, George Em , journal=. Learning nonlinear operators via. 2021 , publisher=
work page 2021
-
[2]
IEEE Transactions on Neural Networks , volume=
Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems , author=. IEEE Transactions on Neural Networks , volume=. 1995 , publisher=
work page 1995
-
[3]
Mathematical and Scientific Machine Learning , pages=
Robust training and initialization of deep neural networks: An adaptive basis viewpoint , author=. Mathematical and Scientific Machine Learning , pages=. 2020 , organization=
work page 2020
-
[4]
Learning the solution operator of parametric partial differential equations with physics-informed
Wang, Sifan and Wang, Hanwen and Perdikaris, Paris , journal=. Learning the solution operator of parametric partial differential equations with physics-informed. 2021 , publisher=
work page 2021
-
[5]
Jin, Pengzhan and Meng, Shuai and Lu, Lu , journal=. 2022 , publisher=
work page 2022
-
[6]
Adam: A Method for Stochastic Optimization
Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[7]
Understanding the difficulty of training deep feedforward neural networks , author=. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages=. 2010 , organization=
work page 2010
-
[8]
Proceedings of the IEEE International Conference on Computer Vision , pages=
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification , author=. Proceedings of the IEEE International Conference on Computer Vision , pages=
-
[9]
Mathematical Proceedings of the Cambridge Philosophical Society , author=
A practical method for numerical evaluation of solutions of partial differential equations of the heat-conduction type , volume=. Mathematical Proceedings of the Cambridge Philosophical Society , author=. 1947 , pages=
work page 1947
- [10]
-
[11]
Selected Papers Volume I , pages=
Systems of conservation laws , author=. Selected Papers Volume I , pages=. 2005 , publisher=
work page 2005
-
[12]
James Bradbury and Roy Frostig and Peter Hawkins and Matthew James Johnson and Chris Leary and Dougal Maclaurin and George Necula and Adam Paszke and Jake Vander
-
[13]
Partitioned neural network approximation for partial differential equations enhanced with
Deok-Kyu Jang and Kyungsoo Kim and Hyea Hyun Kim , keywords =. Partitioned neural network approximation for partial differential equations enhanced with. Computer Methods in Applied Mechanics and Engineering , volume =. 2024 , issn =
work page 2024
-
[14]
Jan R. Magnus and H. Neudecker , title =. The Annals of Statistics , number =. 1979 , _doi =
work page 1979
- [15]
-
[16]
The solution of the matrix equations
Chu, King-wah Eric , journal=. The solution of the matrix equations. 1987 , publisher=
work page 1987
-
[17]
Bartels, Richard H and Stewart, George W , journal=. Algorithm 432. 1972 , publisher=
work page 1972
-
[18]
Doklady Akademii Nauk , volume=
On the solution of ill-posed problems and the method of regularization , author=. Doklady Akademii Nauk , volume=. 1963 , organization=
work page 1963
-
[19]
SIAM Journal on Matrix Analysis and Applications , volume=
Tikhonov regularization and total least squares , author=. SIAM Journal on Matrix Analysis and Applications , volume=. 1999 , publisher=
work page 1999
-
[20]
A Multilinear (Tensor) Algebraic Framework for Computer Graphics, Computer Vision and Machine Learning , author=. 2009 , school=
work page 2009
-
[21]
Solutions to some functional equations and their applications to characterization of probability distributions , author=. Sankhy. 1968 , publisher=
work page 1968
-
[22]
Journal of the American Statistical Association , volume=
Estimation of heteroscedastic variances in linear models , author=. Journal of the American Statistical Association , volume=. 1970 , publisher=
work page 1970
-
[23]
Computer Methods in Applied Mechanics and Engineering , volume=
A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data , author=. Computer Methods in Applied Mechanics and Engineering , volume=. 2022 , publisher=
work page 2022
-
[24]
Mathematics of Control, Signals and Systems , volume=
Approximation by superpositions of a sigmoidal function , author=. Mathematics of Control, Signals and Systems , volume=. 1989 , publisher=
work page 1989
-
[25]
Multilayer feedforward networks are universal approximators , author=. Neural Networks , volume=. 1989 , publisher=
work page 1989
-
[26]
Applied and Computational Harmonic Analysis , volume=
Universality of deep convolutional neural networks , author=. Applied and Computational Harmonic Analysis , volume=. 2020 , publisher=
work page 2020
-
[27]
Refinement and universal approximation via sparsely connected
Heinecke, Andreas and Ho, Jinn and Hwang, Wen-Liang , journal=. Refinement and universal approximation via sparsely connected. 2020 , publisher=
work page 2020
-
[28]
IEEE Transactions on Information Theory , volume=
Universal approximation bounds for superpositions of a sigmoidal function , author=. IEEE Transactions on Information Theory , volume=. 1993 , publisher=
work page 1993
-
[29]
Multilayer feedforward networks with a nonpolynomial activation function can approximate any function , author=. Neural Networks , volume=. 1993 , publisher=
work page 1993
-
[30]
Advances in Neural Information Processing Systems , volume=
The expressive power of neural networks: A view from the width , author=. Advances in Neural Information Processing Systems , volume=
-
[31]
Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
Deep residual learning for image recognition , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=
-
[32]
Advances in Neural Information Processing Systems , volume=
Resnet with one-neuron hidden layers is a universal approximator , author=. Advances in Neural Information Processing Systems , volume=
-
[33]
International Journal of Neural Systems , volume=
Recurrent neural networks are universal approximators , author=. International Journal of Neural Systems , volume=. 2007 , publisher=
work page 2007
-
[34]
Banach space theory: The basis for linear and nonlinear analysis , author=. 2011 , publisher=
work page 2011
-
[35]
A Survey on Universal Approximation Theorems
A survey on universal approximation theorems , author=. arXiv preprint arXiv:2407.12895 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[36]
Introduction to applied linear algebra: vectors, matrices, and least squares , author=. 2018 , publisher=
work page 2018
-
[37]
IEEE Signal Processing Letters , volume=
Partitioned alternating least squares technique for canonical polyadic tensor decomposition , author=. IEEE Signal Processing Letters , volume=. 2016 , publisher=
work page 2016
-
[38]
Enhanced line search: A novel method to accelerate
Rajih, Myriam and Comon, Pierre and Harshman, Richard A , journal=. Enhanced line search: A novel method to accelerate. 2008 , publisher=
work page 2008
- [39]
-
[40]
Journal of Computational Physics , volume=
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , author=. Journal of Computational Physics , volume=. 2019 , publisher=
work page 2019
-
[41]
Journal of Machine Learning Research , volume=
Automatic differentiation in machine learning: a survey , author=. Journal of Machine Learning Research , volume=
-
[42]
Fourier neural operator for parametric partial differential equations , author=
-
[43]
Neural operator: Graph kernel network for partial differential equations , author=
-
[44]
Accelerating deep learning with shrinkage and recall , author=. 2016 , organization=
work page 2016
-
[45]
Sparse networks from scratch: Faster training without losing performance , author=
-
[46]
Advances in Neural Information Processing Systems , volume=
Weight normalization: A simple reparameterization to accelerate training of deep neural networks , author=. Advances in Neural Information Processing Systems , volume=
- [47]
-
[48]
Domain Decomposition Methods in Science and Engineering XXVI , pages=
A Parareal Architecture for Very Deep Convolutional Neural Networks , author=. Domain Decomposition Methods in Science and Engineering XXVI , pages=. 2023 , publisher=
work page 2023
-
[49]
Two-level group convolution , author=. Neural Networks , volume=. 2022 , publisher=
work page 2022
-
[50]
A fast general thermal simulation model based on Multi-Branch Physics-Informed deep operator neural network , author=. Physics of Fluids , volume=
-
[51]
Searching for Activation Functions
Searching for activation functions , author=. arXiv preprint arXiv:1710.05941 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[52]
Liu, Dong C and Nocedal, Jorge , journal=. On the limited memory. 1989 , publisher=
work page 1989
-
[53]
Son, Hwijae , journal=
-
[54]
Hybrid Least Squares/Gradient Descent Methods for
Choi, Jun and Lee, Chang-Ock and Moon, Minam , journal=. Hybrid Least Squares/Gradient Descent Methods for
-
[55]
A hybrid iterative method based on
Hu, Jun and Jin, Pengzhan , journal=. A hybrid iterative method based on
- [56]
-
[57]
Zur theorie stetiger abbildungen in funktionalr
Schauder, Julius , journal=. Zur theorie stetiger abbildungen in funktionalr. 1927 , publisher=
work page 1927
-
[58]
Faber, Georg , journal=
-
[59]
Glenn Byrenheid , title =
-
[60]
Linear and Multilinear Algebra , volume =
Changqing Xu and Lingling He and Zerong Lin , title =. Linear and Multilinear Algebra , volume =. 2020 , publisher =. doi:10.1080/03081087.2018.1556242 , URL =
-
[61]
The Annals of Statistics , number =
Elizabeth Chase MacRae , title =. The Annals of Statistics , number =. 1974 , doi =
work page 1974
-
[62]
arXiv preprint math.GM/0508053 , year=
Tensor permutation matrices in finite dimensions , author=. arXiv preprint math.GM/0508053 , year=
-
[63]
Bhattacharya, Kaushik and Hosseini, Bamdad and Kovachki, Nikola B and Stuart, Andrew M , journal=
-
[64]
Advances in Neural Information Processing Systems , volume=
Multiwavelet-based operator learning for differential equations , author=. Advances in Neural Information Processing Systems , volume=
-
[65]
Yang, Bo and Li, Xingquan and Zhao, Jie and Jiang, Ying , journal=
-
[66]
He, Junyan and Koric, Seid and Abueidda, Diab and Najafi, Ali and Jasiuk, Iwona , journal=. 2024 , publisher=
work page 2024
-
[67]
Physical Review Research , volume=
Multifidelity deep neural operators for efficient learning of partial differential equations with application to fast inverse design of nanoscale heat transport , author=. Physical Review Research , volume=. 2022 , publisher=
work page 2022
-
[68]
Hadorn, Patrik Simon , year=
-
[69]
Lee, Jae Yong and Cho, Sung Woong and Hwang, Hyung Ju , journal=
-
[70]
Seidman, Jacob and Kissas, Georgios and Perdikaris, Paris and Pappas, George J. , booktitle =
-
[71]
Tensor Transpose and Its Properties
Tensor transpose and its properties , author=. arXiv preprint arXiv:1411.1503 , year=
work page internal anchor Pith review Pith/arXiv arXiv
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.