Pith. sign in

REVIEW 3 major objections 6 minor 21 references

Rethinking Neural-based Matrix Inversion: Why can't, and Where can

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Neural networks cannot learn a general matrix inverse: near any singular matrix, inversion blows up while the network stays bounded, so no polynomial Lipschitz network can cover such domains.

desk verdict The negative result is real but elementary, and as stated the main theorem is vacuous because a dataset cannot contain a ball and exclude all singular matrices; the paper is repairable and worth a serious referee. read the letter →

arxiv 2506.00642 v1 pith:X353TUIU submitted 2025-05-31 cs.LG

Yuliang Ji , Jian Wu , Yuanzhe Xi This is my paper
classification cs.LG
keywords matrixinversionneuralnetworkspolynomialLipschitzcontinuityend-to-endlearningsingularmatricesscientificcomputingReLUuniversalapproximation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a neural network trained end-to-end can approximate the matrix inverse over a general domain, and answers no for a broad class of networks it calls polynomial Lipschitz continuous. This class includes fully connected, convolutional, attention, transformer, residual, spline, and recurrent layers, so the negative result covers most modern architectures. The obstruction is analytic: near any singular matrix of rank $n-1$, the inverse is unbounded, while every polynomial Lipschitz continuous function is bounded on bounded sets. Hence, over any training domain that accumulates at a singular matrix, no network in this class can keep the error below a fixed threshold everywhere, and certain expected moments of the error diverge. The paper then shows where the task becomes feasible: on compact regions kept a positive distance from the singular set, inversion is smooth, and a two-layer ReLU network with enough hidden units can reproduce the linear approximation and achieve small test error.

What carries the argument

Polynomial Lipschitz continuity: a function $f$ satisfies $\| f(x)-f(y)\|_{L_*} \le \sum_{i=0}^{n_f} f_i(\|x\|_{L_+},\|y\|_{L_+}) \|x-y\|_{L_+}^i$, where each $f_i$ is a polynomial. This generalizes ordinary Lipschitz continuity and includes architectures whose Jacobian entries grow polynomially, such as multi-head dot-product attention, transformers, residual blocks, and spline layers. The paper proves closure under composition and concatenation, so a trained network built from such layers inherits the property, and proves such functions are bounded on bounded sets. The other load-bearing object is the adjugate formula $A^{-1}=\mathrm{adj}(A)/\det(A)$, which yields the blow-up estimate near a rank-$(n-1)$ singular matrix and, expanded about a nonsingular $A_0$, gives the first-order formula that the constructive two-layer ReLU network is designed to match.

What would settle it

Fix $n=2$ and train a ReLU network end-to-end on nonsingular matrices sampled from a box whose closure contains the singular matrix with all entries equal to 1, while avoiding exact singulars in the training labels. Take a sequence of nonsingular matrices $x_m$ in that box converging to that singular matrix and measure $\| \mathrm{Inv}(x_m)-F(x_m)\|_2$. The theorem predicts this error grows without bound as $m$ increases, since the inverse entries blow up while $F$ stays bounded; observing bounded error along such a sequence would disprove the claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that no universal end-to-end neural matrix inversion model exists within the polynomial Lipschitz class. Theorem 3.7 states that for any such function $F$ and any error threshold $E>0$, there exists a data point $x$ in the nonsingular dataset with $\| \mathrm{Inv}(x)-F(x)\|_L > E$; Theorem 3.8 extends this to subsets of positive measure, and Theorem 3.9 shows that for any $k>n^2$ the $k$-th moment of the error over the whole dataset is infinite. The proof combines a lower bound for inversion near a rank-$(n-1)$ singular matrix $A_0$, namely $\| \mathrm{Inv}(A)\|_L > C_{A_0}/\|A-A_0\|_L$, with the fact that polynomial Lipschitz functions are bounded on bounded sets. The authors therefore conclude that a general matrix inversion model fails because the target function blows up where these functions necessarily stay bounded.

Load-bearing premise

The load-bearing premise is that the training domain contains points arbitrarily close to a singular matrix; the paper states this as a ball that both lies in the dataset and contains a rank-$(n-1)$ singular matrix after all singular matrices are excluded, which is contradictory as written, but the essential requirement is accumulation at a singular matrix, and the negative theorems do not apply to domains kept a positive distance away from all singular matrices.

Editorial extensions

If this is right

  • Any network built from polynomial Lipschitz layers misses the matrix inverse by an arbitrarily large amount somewhere on any training domain that approaches a singular matrix; width and depth within this class cannot fix that.
  • For such domains, training with the loss $\| \mathrm{Inv}(x)-F(x)\|_L^k$ with $k>n^2$ gives infinite expected loss, and for smaller $k$ a subset of positive measure still pushes the expected loss above any chosen threshold.
  • Feasible training sets must exclude the $\epsilon$-neighborhood $M_\epsilon$ of the singular set; on compact boxes $M_0$ with $M_0 \cap M_\epsilon = \emptyset$, inversion is well behaved and neural approximation is possible.
  • A two-layer ReLU network with at least $2n^2$ hidden units can implement the linearization $(A_0+A')^{-1} \approx f^0(A_0) + \sum_{i,j} f^1_{i,j}(A_0) A'_{i,j}$, with error $O(\|A'\|^2)$, matching first-order accuracy.
  • Experiments on $2\times2$, $3\times3$, and $16\times16$ matrices sampled in safe regions show test errors near $10^{-5}$ to $10^{-4}$, and the trained network's coefficients align with the linear approximation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This is an editorial extension: the obstruction is not specific to neural networks; any approximation scheme whose outputs are bounded on bounded sets fails on domains that accumulate at a singular matrix, so the real boundary is a domain condition rather than an architecture condition.
  • This is an editorial extension: the distance of training data to the singular set (equivalently, its condition number) should predict end-to-end generalization, and adding even a few near-singular training examples should sharply increase worst-case error; that is testable.
  • This is an editorial extension: the impossibility applies to one-shot end-to-end maps; hybrid schemes that use a neural network as a warm start and then apply Newton or Schulz iterations could escape the bound, because the refinement steps handle the neighborhood where the inverse is unbounded.
  • This is an editorial extension: whether non-polynomial-Lipschitz architectures, such as networks with normalization or features that break the bounded-Jacobian condition, can approximate inversion on such domains remains open, as the authors themselves note.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies end-to-end neural approximation of matrix inversion. It introduces polynomial Lipschitz continuity as a function class covering common neural network components, proves negative results (Theorems 3.7–3.9) that no such function can approximate Inv pointwise or in expectation over datasets that contain neighborhoods of singular matrices, and then identifies feasible training regions away from singular matrices. It proposes a two-layer network that is intended to realize the first-order Taylor expansion of the inverse, and validates performance on 2×2, 3×3, and 16×16 datasets. A post-hoc analysis of a trained 2×2 model shows that its piecewise-linear form closely matches the linear approximation.

Significance. If the dataset assumptions are repaired, the negative result is a clean and useful observation: Inv is unbounded near rank-(n−1) singular matrices, while polynomial Lipschitz functions are bounded on bounded sets, so universal approximation over any domain whose closure contains a singular matrix is impossible. The positive part correctly identifies that local approximation away from the singular set is feasible, and the experiments support that claim. Strengths of the paper include that the negative bounds do not depend on fitted parameters, that the Taylor coefficients in Section 5.3 are computed independently of the trained network, and that the authors inspect the trained network rather than only reporting aggregate errors. The main concerns are formal: the contradictory assumption in Theorem 3.7 and the incorrect sign in Eq. (11).

major comments (3)
  1. [Section 3.1, Theorems 3.7–3.9] The hypothesis that M contains an open ball B(a,c) while excluding all singular matrices is inconsistent, because every open ball in R^{n×n} intersects the zero set of the nonzero polynomial det. The theorem is therefore vacuous as stated. The proof actually only needs a rank-(n−1) singular matrix A0 in the closure of M together with nonsingular points of M arbitrarily close to A0; I recommend restating the theorems with a closure assumption (e.g., A0 ∈ closure(M) and M ∩ B(A0,δ) \ S_M ≠ ∅) and updating the 'without loss of generality' remark in Section 2 accordingly.
  2. [Section 4.2, Eq. (11)] The proposed two-layer construction computes h2_{k,l} = f^0_{k,l} + ReLU(s) + ReLU(−s) = f^0_{k,l} + |s|, not f^0_{k,l} + s, because both second-layer weights are set to +1. With this choice the network cannot reproduce the linear term from Eq. (10). The construction works only if the second-layer weights are +1 and −1 on the two branches; please correct Eq. (11) and the surrounding sentence accordingly.
  3. [Section 3, Lemma 3.4 and Theorem 3.7 proof] Lemma 3.4 is stated for Lipschitz functions, but Theorem 3.7 applies it to polynomial Lipschitz functions when asserting that F is bounded on B(A0,δ). The proof of Lemma 3.4 also refers to Definition 2.2, which is not the definition being used in that lemma. Please state and prove boundedness of polynomial Lipschitz functions on bounded sets (the argument goes through because the defining polynomials are bounded on bounded sets), and cite that result in Theorems 3.7–3.9.
minor comments (6)
  1. [Section 3.2, Theorem 3.8] The symbol Mε is reused with a different meaning from Section 4.1: in the proof Mε = B(A0, ε) is a single ball around a singular matrix, whereas Section 4.1 defines Mε as the union of ε-neighborhoods of all singular matrices. Please use different notation.
  2. [Section 3.2, Theorem 3.8 proof] The phrase 'the integral diverges as ε→0' is imprecise for k < n^2: the integral ∫_0^ε r^{n^2−1−k} dr converges, but the normalized expectation lower bound C ε^{−n^2} ∫_0^ε r^{n^2−1−k} dr ∼ C ε^{−k} diverges as ε→0. The conclusion is correct, but the wording should be adjusted.
  3. [Appendix A.6, Theorem A.6] The sentence 'obviously the last formula larger than any real number when ε → ∞' should read 'as ε → 0', and for K ≥ n^2 the integral over (0, ε) is already infinite for every ε > 0.
  4. [Section 4.2, Eq. (9)] The notation A^{adj,k,l}_{0,i,σ(i)} and A'^{adj,k,l}_{i,σ(i)} is not defined; please define the adjugate deletion notation explicitly.
  5. [Section 2.2, Definition 2.2] The norms L+ and L* are introduced in the definition but later results refer to L1, L2, and L∞ norms; please clarify the relationship or unify the notation.
  6. [Section 5.3 and Appendix E] The analysis omits h2: the first row of W1 is zero and h2 is never defined, yet the ReLU pattern descriptions list h1,h3,...,h7. Please clarify whether h2 is identically zero or simply unused.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the negative bounds follow from analytic blow-up near singular matrices, and the positive construction reads weights from Taylor coefficients rather than from fitted data.

full rationale

The paper's main negative results are derived from an analytic property of the inverse map near rank-(n-1) singular matrices (Lemma 3.6) combined with boundedness of polynomial Lipschitz functions on bounded balls, not from fitted parameters or from the authors' prior results. The constructive positive result in Section 4.2 sets neural-network weights directly from the Taylor coefficients f^0_k,l({A0}) and f^1_k,l,i,j({A0}) of the inverse, so the resulting network is a designed local approximant, not a fitted parameter renamed as a prediction; the construction is self-contained. Section 5.3 inspects a trained network post hoc and compares its local affine behavior with the Taylor linearization; this is empirical verification rather than circular reasoning. The main caveat is a correctness issue, not a circularity: Theorem 3.7 assumes that the dataset M simultaneously contains a ball B(a,c) and excludes all singular matrices, which cannot both hold since every open ball in R^{n x n} contains singular matrices; the intended argument only requires a rank-(n-1) singular matrix in the closure of M. The self-citation to Cai et al. (2022) in Lemma 3.5 is used only to illustrate that neural spline layers are polynomial Lipschitz and is not load-bearing for the central no-go claim. Overall, no circular step could be identified: the derivation chain does not reduce to its own inputs. The vacuous-assumption issue should be treated as a correctness risk, not as circularity, under the rule that only explicit reductions count.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The theoretical results introduce no fitted constants; the free parameters of trained networks are not used in the proofs. The main added premise is the dataset condition near singular matrices, which is stated inconsistently.

assumptions (4)
  • ad hoc to paper The dataset M contains a ball B(a, c) of sufficiently large radius and contains no singular matrices.
    Invoked in Theorems 3.7, 3.8, and 3.9; the two clauses are mutually inconsistent because every ball contains singular matrices. The proofs require a rank-(n-1) singular matrix in the closure of M.
  • standard math Jacobian entries bounded by polynomials imply polynomial Lipschitz continuity via the mean value theorem.
    Used in Lemma 3.3 and applied to attention layers in Lemma 3.5.
  • domain assumption Neural network components such as fully connected layers, ReLU, attention, spline, and transformer layers are polynomial Lipschitz continuous.
    Lemma 3.5 relies on prior work for Lipschitz components and on Jacobian bounds for attention; the paper does not prove this for every modern architecture.
  • standard math The matrix inverse is given by the adjugate over determinant formula, and the determinant is a polynomial in the entries.
    Used in Lemma 3.6 and in the Taylor expansion of Section 4.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Neural-based Matrix Inversion: Why can't, and Where can." pith.science (2026). https://pith.science/paper/X353TUIU

@misc{pith2026250600642,
  author       = {Pith},
  title        = {Pith review of: Rethinking Neural-based Matrix Inversion: Why can't, and Where can},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X353TUIU}},
  note         = {Machine review of arXiv:2506.00642}
}
read the original abstract

Deep neural networks have achieved substantial success across various scientific computing tasks. A pivotal challenge within this domain is the rapid and parallel approximation of matrix inverses, critical for numerous applications. Despite significant progress, there currently exists no universal neural-based method for approximating matrix inversion. This paper presents a theoretical analysis demonstrating the fundamental limitations of neural networks in developing a general matrix inversion model. We expand the class of Lipschitz functions to encompass a wider array of neural network models, thereby refining our theoretical approach. Moreover, we delineate specific conditions under which neural networks can effectively approximate matrix inverses. Our theoretical results are supported by experimental results from diverse matrix datasets, exploring the efficacy of neural networks in addressing the matrix inversion challenge.

Figures

Figures reproduced from arXiv: 2506.00642 by the authors.

Figure 1
Figure 1. Left: The Mϵ region (shaded area) for fixed a11 = 1, a12 = 2. Right: The Mϵ region (shaded area) for fixed a11 = 1, a22 = 2. Example 1: Fixed a11 = 1 and a12 = 2 When a11 = 1 and a12 = 2 are fixed, Mϵ consists of the ϵ-neighborhoods around the line a22 = 2a21 in the (a21, a22) plane. The shaded area in the left plot of [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Blue surface: The Mϵ region for fixed a11 = 1. Yellow ball: A training dataset area avoiding Mϵ. 4.2 Theoretical Analysis of Neural Network Approximation In this section, we theoretically describe how to design the coefficients of a neural network to approximate the matrix inversion in a specific region M0 ⊂ R n×n. The region M0 is defined as: M0 = Yn i=1 Yn j=1 [A0,i,j − c, A0,i,j + c], (7) where A0 ∈ R n×n is a fi… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 20 canonical work pages

  1. [1]

    Almasadeh, Khawla A

    Ali J. Almasadeh, Khawla A. Alnajjar, and Mahmoud A. Albreem. Enhanced deep learning for massive mimo detection using approximate matrix inversion. In 2022 5th International Conference on Communications, Signal Processing, and their Applications (ICCSPA), 2022

    work page 2022

  2. [2]

    Sorting out lipschitz function approximation

    Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97, pages 291--301. PMLR, 2019

    work page 2019

  3. [3]

    Deep Learning Solution of the Eigenvalue Problem for Differential Operators

    Ido Ben-Shaul, Leah Bar, Dalia Fishelov, and Nir Sochen. Deep Learning Solution of the Eigenvalue Problem for Differential Operators . Neural Computation, 35 0 (6): 0 1100--1134, 05 2023

    work page 2023

  4. [4]

    Autm flow: Atomic unrestricted time machine for monotonic normalizing flows

    Difeng Cai, Yuliang Ji, Huan He, Qiang Ye, and Yuanzhe Xi. Autm flow: Atomic unrestricted time machine for monotonic normalizing flows. In UAI, 2022

    work page 2022

  5. [5]

    Design and analysis of a hybrid gnn-znn model with a fuzzy adaptive factor for matrix inversion

    Jianhua Dai, Yuanmeng Chen, Lin Xiao, Lei Jia, and Yongjun He. Design and analysis of a hybrid gnn-znn model with a fuzzy adaptive factor for matrix inversion. IEEE Transactions on Industrial Informatics, 18 0 (4): 0 2434--2442, 2022

    work page 2022

  6. [6]

    A fuzzy adaptive zeroing neural network model with event-triggered control for time-varying matrix inversion

    Jianhua Dai, Ping Tan, Lin Xiao, Lei Jia, Yongjun He, and Jiajie Luo. A fuzzy adaptive zeroing neural network model with event-triggered control for time-varying matrix inversion. IEEE Transactions on Fuzzy Systems, 31 0 (11): 0 3974--3983, 2023

    work page 2023

  7. [7]

    Dongarra, Iain S

    Jack J. Dongarra, Iain S. Duff, Danny C. Sorensen, and Henk A. van der Vorst. Solving linear systems on vector and shared memory computers. SIAM Journal on Scientific and Statistical Computing, 11 0 (5): 0 1096--1127, 1990

    work page 1990

  8. [8]

    Neural spline flows

    Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, pages 7509--7520, 2019

    work page 2019

Show all 21 references
  1. [9]

    Neural network approach to computing matrix inversion

    Luo Fa-Long and Bao Zheng. Neural network approach to computing matrix inversion. Applied Mathematics and Computation, 47: 0 109--120, 1992

    1992

  2. [10]

    From zhang neural network to newton iteration for matrix inversion

    Luo Fa-Long and Bao Zheng. From zhang neural network to newton iteration for matrix inversion. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—I, 56 0 (7): 0 1405--1414, 2009

    2009

  3. [11]

    Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J. R. Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, David Silver, Demis Hassabis, and Pushmeet Kohli. Discovering faster matrix multiplicat...

    2022

  4. [12]

    Stanimirović, Panagiotis Tzekis, and Vasilios N

    Dimitrios Gerontitis, Changxin Mo, Predrag S. Stanimirović, Panagiotis Tzekis, and Vasilios N. Katsikis. A novel extended li zeroing neural network for matrix inversion. Neural Computing and Applications, 35: 0 14129–14152, 2023

    2023

  5. [13]

    Golub and Charles F

    Gene H. Golub and Charles F. Van Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, MD, 3rd edition, 1996

    1996

  6. [14]

    Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion monte carlo like approach

    Jiequn Han, Jianfeng Lu, and Mo Zhou. Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion monte carlo like approach. Journal of Computational Physics, 423: 0 109792, 2020. ISSN 0021-9991

    2020

  7. [15]

    An optimization network for matrix inversion

    Ju-Seog Jang, Soo-Young Lee, and Sang-Yung Shin. An optimization network for matrix inversion. In Neural Information Processing Systems, 1987

    1987

  8. [16]

    The lipschitz constant of self-attention

    Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In ICML, 2021

    2021

  9. [17]

    Latorre, P

    F. Latorre, P. Rolland, , and V. Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization. In International Conference on Learning Representations, 2020

    2020

  10. [18]

    An efficient second-order neural network model for computing the moore–penrose inverse of matrices

    Lin Li and Jianhao Hu. An efficient second-order neural network model for computing the moore–penrose inverse of matrices. IET Signal Processing, 16 0 (9): 0 1106--1117, 2022

    2022

  11. [19]

    SGDR: stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR, 2017

    2017

  12. [20]

    Steriti, J

    R. Steriti, J. Coleman, and M. A. Fiddy. A neural network based matrix inversion algorithm. In IJCNN, 1990

    1990

  13. [21]

    Virmaux and K

    A. Virmaux and K. Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. In Advances in Neural Information Processing Systems, 2018

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.