REVIEW 4 major objections 5 minor 28 references
Rank Inspired Neural Network for solving linear partial differential equations
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that decorrelating hidden-layer outputs before a least-squares solve, and stopping early on the PDE residual, makes the physics-informed extreme learning machine robust to random initialization.
desk verdict A plausible incremental improvement to PIELM whose abstract overclaims: only the early-stopping variant RINN-es is supported by the paper's own Table 7, and the evidence is single-run. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the sample covariance matrix of the last hidden layer's outputs, $C = \Phi^\top\Phi/(K-1)$, whose off-diagonal entries approximate the $L^2$ inner products between neural basis functions at the collocation points. The machinery is the covariance-driven loss $L_{\text{total}} = \varepsilon L_{\text{diag}} + L_{\text{ortho}}$, where $L_{\text{ortho}} = \|C \odot (J - E)\|_F$ penalizes off-diagonal correlation and $L_{\text{diag}} = \sum_i |\log_{10}(C_{ii}^2)|$ pulls each basis function toward unit norm; training this loss is the preconditioning stage. The second stage is the classic PIELM least-squares solve for $\beta = H^\dagger S$. The early-stopping mechanism is the PDE residual $L_{\text{pde}}$ from equation (25), which the paper shows tracks the evaluation error during training and is therefore used to select the stopping epoch.
What would settle it
A direct check would be to compute the condition number of $H$ at the RINN-es optimal epoch and compare it with the condition number from a random initialization on the same PDE; if RINN-es improves accuracy while $\mathrm{cond}(H)$ does not improve, the claimed conditioning mechanism is not the cause of the improvement.
Extended reading notes
Core claim
On the paper's own terms: PIELM is accurate but fragile because randomly initialized hidden weights produce correlated neural basis functions. RINN adds a pretraining stage that minimizes the off-diagonal entries of the sample covariance matrix $C = \Phi^\top\Phi/(K-1)$ of the hidden-layer outputs, enforcing pairwise orthogonality of the basis at the collocation points, and also penalizes diagonal entries that deviate from unit norm. After this decorrelation stage, the hidden parameters are frozen and the output weights are obtained from the linear least-squares problem $H(\theta,X)\beta = S$. Because the orthogonalization loss carries no PDE information, overtraining can degrade the solution, and the paper's early-stopping variant RINN-es computes the PDE residual $L_{\text{pde}} = L_{\text{res}} + L_{\text{bcs}} + L_{\text{ics}}$ after each epoch and restores the parameters at which that residual was minimal. The claim is that this combination turns PIELM from an initialization-sensitive solver into one that maintains high accuracy across wide initialization ranges.
Load-bearing premise
The load-bearing premise is that making the raw hidden-layer outputs $\Phi$ orthogonal at the collocation points is enough to make the PDE collocation matrix $H = [A\Phi; B\Phi; \Phi]$ well-conditioned and accurate, even though $H$ applies differential operators to $\Phi$ and the paper gives no analysis linking orthogonality of $\Phi$ to conditioning of $H$.
Editorial extensions
If this is right
- For the linear PDEs tested, RINN-es reports relative $L^2$ errors several orders of magnitude below PIELM on the same collocation points and network architecture.
- The optimal early-stopping epochs differ strongly across initializations (epochs 6, 672, and 999 in Table 7), implying that fixed-epoch training is not a reliable protocol for this class of neural least-squares solvers.
- The covariance-driven orthogonalization can be inserted as a preconditioning stage into the existing two-step ELM pipeline without changing the least-squares solve, so it is a drop-in modification for PIELM-type methods.
- The experiments cover elliptic, advection, heat, and wave equations, so the claimed stabilization is not limited to a single equation type or regime.
- Because the early-stopping criterion uses only the PDE residual, it requires no extra data beyond the collocation points already used by the solver.
Reading between the lines
- A natural extension the paper does not test is to apply the same decorrelation loss to the operator-applied matrix $H = [A\Phi; B\Phi; \Phi]$ rather than to $\Phi$ alone, since the conditioning that actually determines the least-squares error is the conditioning of $H$.
- The reported failure of plain RINN under small initializations suggests that decorrelation alone can be harmful, and that early stopping may be acting as a regularizer rather than simply locating an optimum of the orthogonalization loss; the paper leaves this distinction open.
- The two-stage structure is general enough to serve as an initialization scheme for other random-feature PDE solvers, such as randomized Petrov-Galerkin or discontinuous Galerkin variants, though that transfer is not demonstrated here.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript introduces Rank Inspired Neural Networks (RINN) as a two-stage alternative to PIELM for linear PDEs. In the first stage, hidden-layer parameters are trained using a loss that penalizes off-diagonal entries of the sample covariance matrix C = Φ^T Φ/(K−1) and deviations of diagonal entries from unity; in the second stage, the PDE collocation system H(θ,X)β = S is solved by least squares. A variant, RINN-es, adds an early-stopping rule based on the PDE residual loss Lpde defined in Eq. (25). Experiments on 1D and 2D Poisson equations, advection, heat, and wave equations report relative L2 and L1 errors and compare with PIELM under several initialization ranges. The paper claims that RINN significantly reduces performance variability due to parameter initialization compared to PIELM.
Significance. If borne out, the proposed covariance-driven pretraining would be a practically useful, inexpensive way to stabilize PIELM, and the residual-based early-stopping mechanism is a sensible addition. The paper also documents the known initialization sensitivity of PIELM in Table 1. The central claim, however, is not currently supported by the evidence: plain RINN fails badly in part of Table 7, and no repeated-seed statistics are reported. The contribution therefore needs substantial strengthening before it can be assessed.
major comments (4)
- [Sec. 3.2, Table 7] The abstract's claim that 'RINN significantly reduces performance variability due to parameter initialization compared to PIELM' is contradicted by the plain RINN rows. Under U(−0.5, 0.5), RINN has EL2 = 1.16×10^1 versus PIELM's 7.80×10^−2, and under U(−2, 2), RINN has EL2 = 3.46×10^0 whereas PIELM has 8.36×10^0. Only the RINN-es variant is uniformly competitive. The paper should either restrict the central claim to RINN-es and explain why plain RINN is not covered, or report fixed-epoch RINN results for all tested problems and show that the failure in Table 7 is atypical.
- [Tables 1–7] Each table reports a single run per configuration. Since the paper's central claim is about variability across random initializations, single-run numbers cannot establish it. Please report means and standard deviations (or medians with quartiles) over multiple independent initializations, and clarify whether the reported errors are the best, median, or a single realization. This is essential for Table 7, where the conclusion is described in terms of robustness.
- [Sec. 2.2, Eqs. (10)–(15) and Eq. (6)] The orthogonalization objective acts on the raw hidden-output matrix Φ, whereas the least-squares system (6) is governed by H = [AΦ; BΦ; Φ], which contains derivatives. The manuscript asserts that decorrelating Φ improves representational ability and numerical stability, but no analysis, estimate, or numerical diagnostic connects orthogonality of Φ to the conditioning of H or to the resulting error. In addition, the text states that the 'primary objective' is to maximize the rank of Φ, but Algorithm 1 never maximizes a rank; it minimizes off-diagonal covariance. Either provide a supporting argument for the transfer from Φ-orthogonality to H-conditioning, or soften the claim and present evidence that the objective is the right one.
- [Sec. 3.1 and Algorithm 2] The early-stopping criterion selects the epoch that minimizes Lpde on the training points. Figure 7 shows this residual tracks the test error in one example, but no robustness evidence is given for other problems, and the procedure is a form of model selection on the training set. Moreover, in Table 7 under U(−2,2), the best epoch is 999 with patience 250 and maximum 1000 epochs, so no early stopping actually occurs; the algorithm simply returns the final iterate. Please report for each case whether termination was due to patience or to reaching E, and consider a small validation split or a separate stopping rule.
minor comments (5)
- [Sec. 2.2, page 6] 'potimize' should be 'optimize'; also 'PyTorch 3.8.20' is not a valid PyTorch version and should be corrected.
- [Example 2.5, Eq. (23)] The spatial domain is written as Ω=[0,1] but the boundary condition is u(−1,t)=u(1,t)=0; these are inconsistent and the intended interval or boundary indices should be fixed.
- [Sec. 2.3] The section promises a computational efficiency analysis, but no runtime, iteration-count, or floating-point operation data are reported; please add the promised comparison or remove the claim.
- [Figures 7–8 and Table 7] The panels would benefit from explicit labels, and the caption or text should state whether the reported errors are for a single seed; see Major Comment 2.
- [Eq. (14)] The definition of Ldiag uses log10(C_ii^2), which can be unstable if a diagonal entry is near zero; a small floor or a different normalization should be specified.
Circularity Check
No significant circularity: RINN's covariance orthogonalization and early stopping are tested against analytic solutions, and no load-bearing derivation reduces to its own inputs.
full rationale
RINN's derivation chain does not exhibit circular reasoning. The covariance-driven loss L_total = epsilon * L_diag + L_ortho in Eq. (15) is minimized over hidden parameters theta before the PDE system H * beta = S in Eq. (6) is solved by least squares; the PDE constraints enter only in stage 2, so the orthogonalization objective is not defined in terms of the PDE solution or the reported errors. All accuracy claims are measured against analytic solutions (Tables 2-6) or a fixed reference solution (Table 7); no parameter is fitted to the target error. The early-stopping epoch in Algorithm 2 is selected by monitoring the PDE residual L_pde on the training collocation points, which is a standard model-selection heuristic and not a circular prediction: the reported EL2 and EL1 are evaluated on the analytic solution independently of the selection criterion, and the stopping rule does not use the test error. The only self-citation with author overlap, Ref. [17] cited in the introduction for a generalization error bound on PINNs, is contextual and not load-bearing for RINN's construction. The paper's internal inconsistency, where plain RINN performs worse than PIELM under U(-0.5,0.5) in Table 7 while the abstract claims reduced variability, is a correctness and reproducibility concern, not a circularity. No equation in the paper reduces by construction to its own input, and no prior-work uniqueness theorem or ansatz is invoked to force the proposed form.
Assumptions & free parameters
free parameters (7)
- regularization coefficient epsilon =
0.1 or 0.01 per example
- learning rate eta =
1e-3 throughout
- max epochs E =
500, 1000, or 2000 depending on example
- early stopping patience P =
250 (RINN-es)
- network architecture =
[1,128,1], [2,512,1024,1], etc.
- initialization distribution width a =
U(-0.5,0.5), U(-1,1), U(-2,2), U(-20,20)
- collocation point counts =
Kres=1024/2048, Kbcs=2/2048/4096, Kics=1024
assumptions (4)
- domain assumption Neural basis orthogonality improves the conditioning and approximation power of the collocation system
- domain assumption Discrete sample covariance on collocation points is a valid proxy for L2 orthogonality, and minimizing its off-diagonal entries improves PDE solves
- domain assumption PDE residual loss minimum during orthogonalization is a safe early-stopping signal
- standard math Universal approximation capability of neural networks justifies the ELM ansatz
Cite this review
Pith. "Pith review of Rank Inspired Neural Network for solving linear partial differential equations." pith.science (2026). https://pith.science/paper/UT4CSWN2
@misc{pith2026250617654,
author = {Pith},
title = {Pith review of: Rank Inspired Neural Network for solving linear partial differential equations},
year = {2026},
howpublished = {\url{https://pith.science/paper/UT4CSWN2}},
note = {Machine review of arXiv:2506.17654}
}
read the original abstract
This paper proposes a rank inspired neural network (RINN) to tackle the initialization sensitivity issue of physics informed extreme learning machines (PIELM) when numerically solving partial differential equations (PDEs). Unlike PIELM which randomly initializes the parameters of its hidden layers, RINN incorporates a preconditioning stage. In this stage, covariance-driven regularization is employed to optimize the orthogonality of the basis functions generated by the last hidden layer. The key innovation lies in minimizing the off-diagonal elements of the covariance matrix derived from the hidden-layer output. By doing so, pairwise orthogonality constraints across collocation points are enforced which effectively enhances both the numerical stability and the approximation ability of the optimized function space.The RINN algorithm unfolds in two sequential stages. First, it conducts a non-linear optimization process to orthogonalize the basis functions. Subsequently, it solves the PDE constraints using linear least-squares method. Extensive numerical experiments demonstrate that RINN significantly reduces performance variability due to parameter initialization compared to PIELM. Incorporating an early stopping mechanism based on PDE loss further improves stability, ensuring consistently high accuracy across diverse initialization settings.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
F. Calabr` o, G. Fabiani and C. Siettos. Extreme learning machine collocation for the numerical solution of elliptic PDEs with sharp gradients. Computer Methods in Applied Mechanics and Engineering, 387:114188, 2021
work page 2021
-
[2]
G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303–314, 1989
work page 1989
-
[3]
V. Dwivedi and B. Srinivasan. Physics-informed extreme learning machine (PIELM)—a rapid method for the numerical solution of partial differential equations. Neurocomputing, 391:96–118, 2020
work page 2020
-
[4]
S. Dong and Z. Li. Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations. Computer Methods in Applied Mechanics and Engineering, 387:114129, 2021
work page 2021
-
[5]
S. Dong and J. Yang. On computing the hyperparameter of extreme learning machines: Algorithm and ap- plication to computational PDEs, and comparison with classical and high-order finite elements. Journal of Computational Physics, 463:111290, 2022
work page 2022
-
[6]
G. B. Huang, Q. Y. Zhu and C. K. Siew. Extreme learning machine: theory and applications. Neurocomputing, 70(1–3):489–501, 2006
work page 2006
-
[7]
G. B. Huang, L. Chen and C. K. Siew. Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Transactions on Neural Networks, 17(4):879–892, 2006
work page 2006
-
[8]
A. D. Jagtap, E. Kharazmi and G. E. Karniadakis. Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365:113028, 2020
work page 2020
Show all 28 references
-
[9]
A. D. Jagtap and G. E. Karniadakis. Extended physics-informed neural networks (XPINNs): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. Communications in Computational Physics, 28(5):2002–2041, 2020
2002
-
[10]
Kharazmi, Z
E. Kharazmi, Z. Zhang and G. E. Karniadakis. Variational physics-informed neural networks for solving partial differential equations. arXiv preprint arXiv:1912.00873, 2019
1912 arXiv
-
[11]
Kharazmi, Z
E. Kharazmi, Z. Zhang and G. E. Karniadakis. hp-VPINNs: Variational physics-informed neural networks with domain decomposition. Computer Methods in Applied Mechanics and Engineering, 374:113547, 2021
2021
-
[12]
I. E. Lagaris, A. Likas and D. I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5):987–1000, 1998
1998
-
[13]
I. E. Lagaris, A. C. Likas and D. G. Papageorgiou. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 11(5):1041–1049, 2000
2000
-
[14]
R. J. LeVeque. Finite difference methods for ordinary and partial differential equations: steady-state and time-dependent problems. SIAM, 2007
2007
-
[15]
Y. Liu, W. Liu, X. Yan, S. Guo and C. Zhang. Adaptive transfer learning for PINN. Journal of Computational Physics, 490:112291, 2023
2023
-
[16]
Ming and Y
P. Ming and Y. Liao. Deep Nitsche Method: Deep Ritz Method with Essential Boundary Conditions. Commu- nications in Computational Physics, 29(5):1365–1384, 2021
2021
-
[17]
Y. Qian, Y. Zhang, Y. Huang and S. Dong. Physics-informed neural networks for approximating dynamic (hyperbolic) PDEs of second order in time: Error analysis and algorithms. Journal of Computational Physics, 495:112527, 2023
2023
-
[18]
Raissi, P
M. Raissi, P. Perdikaris and G. E. Karniadakis. Physics-informed neural networks: A deep learning frame- work for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019
2019
-
[19]
Rahaman, A
N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio and A. Courville. On the spectral bias of neural networks. In Proceedings of the International Conference on Machine Learning, pages 5301–5310, 2019
2019
-
[20]
S. S. Rao. The finite element method in engineering. Elsevier, 2010
2010
-
[21]
Shang, F
Y. Shang, F. Wang and J. Sun. Deep Petrov-Galerkin method for solving partial differential equations. arXiv preprint arXiv:2201.12995, 2022
2022 arXiv
-
[22]
Sirignano and K
J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339–1364, 2018
2018
-
[23]
J. Sun, S. Dong and F. Wang. Local randomized neural networks with discontinuous Galerkin methods for partial differential equations. Journal of Computational and Applied Mathematics, 445:115830, 2024
2024
-
[24]
H. K. Versteeg. An introduction to computational fluid dynamics: the finite volume method. Pearson Education India, 2007. RINN FOR PDES 19
2007
-
[25]
Wang and S
Y. Wang and S. Dong. An extreme learning machine-based method for computational PDEs in higher dimen- sions. Computer Methods in Applied Mechanics and Engineering, 418:116578, 2024
2024
-
[26]
C. Wu, M. Zhu, Q. Tan, Y. Kartha and L. Lu. A comprehensive study of non-adaptive and residual-based adap- tive sampling for physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 403:115671, 2023
2023
-
[27]
W. E and B. Yu. The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems. Communications in Mathematics and Statistics, 6(1):1–12, 2018
2018
-
[28]
Y. Zang, G. Bao, X. Ye and H. Zhou. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411:109409, 2020. 1 School of Mathematics and Computational Science, Xiangtan University, Xiangtan 411105, P.R.China 2 National ...
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.