{"id":"04f1838a-599f-47e5-8ad1-249277a5b9c7","arxiv_id":"2506.00642","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Neural networks cannot approximate matrix inversion over domains approaching singular matrices; in safe regions they merely learn the local linear Taylor approximation of the inverse.","lead":"The paper tries to prove that neural networks cannot learn a universal matrix inversion model, and that they can only approximate inversion in small regions away from singular matrices. The main impossibility argument reduces to the fact that matrix inversion is unbounded near singular matrices, which any bounded function (including any continuous neural network) fails to approximate.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.7's dataset assumption is self-contradictory: every open ball in R^{n×n} contains singular matrices, so 'B(a,c) ⊂ M' and 'exclude all singular matrices' cannot both hold; the central no-go theorem is vacuous as stated.","rationale":"I agree with the reader that the weakest point is the dataset assumption. This is the single most load-bearing concern because it invalidates the statement of the paper's headline negative theorem, even though the underlying proof idea is sound. The positive construction in Eq. (11) also contains a sign error: h1+ + h1- computes ReLU(x)+ReLU(-x)=|x| rather than x, so the network as written does not reproduce the linear term; replacing the plus with a minus repairs the construction. Theorem 3.8's proof has a minor misstatement about the divergence of the integral for small k, but the normalized bound still diverges after the 1/epsilon^{n^2} factor. These issues are all repairable, so a conditional verdict is appropriate. The novelty overstatement is a presentation issue rather than a correctness issue. Because the central theorem is vacuous as written, the paper cannot be accepted without revision; the reader's CONDITIONAL verdict is the right call.","tokens_in":22146,"tokens_out":12730,"duration_ms":120884,"concrete_test":"Check whether any nonempty open ball in R^{n×n} is disjoint from the singular set; since det is a nonzero polynomial, its zero set has empty interior, so no such ball exists, confirming the assumptions of Theorem 3.7 are contradictory. Then re-derive Theorem 3.7 under the corrected assumption that the closure of M contains a singular matrix of rank n-1 while M itself excludes singular matrices. If the existing proof of Lemma 3.6, together with the generalized boundedness of F on B(A0,δ), yields the conclusion without further hypotheses, the concern is a repairable statement flaw; if additional conditions on M are required, the no-go theorem is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative result (Theorem 3.7) assumes M contains an open ball B(a,c) and also that all singular matrices are excluded from M. Because det is a nonzero polynomial on R^{n×n}, its zero set has empty interior and therefore intersects every nonempty open ball. Hence no dataset M can satisfy both hypotheses for any c>0, making the theorem vacuous. The proof itself only needs a rank-(n-1) singular matrix A0 in the closure of M, together with Lemma 3.6 (unboundedness of Inv near A0) and boundedness of the polynomial Lipschitz function F on B(A0,δ). The boundedness claim is drawn from Lemma 3.4, which is stated for Lipschitz functions; it does extend to polynomial Lipschitz functions because the defining polynomials are bounded on bounded sets, but the paper does not state or prove this generalization. Thus the central no-go claim is not established as written, though the intended argument is likely repairable by replacing the contradictory ball condition with a closure condition. The same inconsistent assumption propagates to Theorems 3.8 and 3.9, which inherit the 'dataset contains a ball' hypothesis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies end-to-end neural approximation of matrix inversion. It introduces polynomial Lipschitz continuity as a function class covering common neural network components, proves negative results (Theorems 3.7–3.9) that no such function can approximate Inv pointwise or in expectation over datasets that contain neighborhoods of singular matrices, and then identifies feasible training regions away from singular matrices. It proposes a two-layer network that is intended to realize the first-order Taylor expansion of the inverse, and validates performance on 2×2, 3×3, and 16×16 datasets. A post-hoc analysis of a trained 2×2 model shows that its piecewise-linear form closely matches the linear approximation.","tokens_in":22403,"tokens_out":6652,"duration_ms":59997,"significance":"If the dataset assumptions are repaired, the negative result is a clean and useful observation: Inv is unbounded near rank-(n−1) singular matrices, while polynomial Lipschitz functions are bounded on bounded sets, so universal approximation over any domain whose closure contains a singular matrix is impossible. The positive part correctly identifies that local approximation away from the singular set is feasible, and the experiments support that claim. Strengths of the paper include that the negative bounds do not depend on fitted parameters, that the Taylor coefficients in Section 5.3 are computed independently of the trained network, and that the authors inspect the trained network rather than only reporting aggregate errors. The main concerns are formal: the contradictory assumption in Theorem 3.7 and the incorrect sign in Eq. (11).","major_comments":[{"comment":"The hypothesis that M contains an open ball B(a,c) while excluding all singular matrices is inconsistent, because every open ball in R^{n×n} intersects the zero set of the nonzero polynomial det. The theorem is therefore vacuous as stated. The proof actually only needs a rank-(n−1) singular matrix A0 in the closure of M together with nonsingular points of M arbitrarily close to A0; I recommend restating the theorems with a closure assumption (e.g., A0 ∈ closure(M) and M ∩ B(A0,δ) \\ S_M ≠ ∅) and updating the 'without loss of generality' remark in Section 2 accordingly.","section":"Section 3.1, Theorems 3.7–3.9"},{"comment":"The proposed two-layer construction computes h2_{k,l} = f^0_{k,l} + ReLU(s) + ReLU(−s) = f^0_{k,l} + |s|, not f^0_{k,l} + s, because both second-layer weights are set to +1. With this choice the network cannot reproduce the linear term from Eq. (10). The construction works only if the second-layer weights are +1 and −1 on the two branches; please correct Eq. (11) and the surrounding sentence accordingly.","section":"Section 4.2, Eq. (11)"},{"comment":"Lemma 3.4 is stated for Lipschitz functions, but Theorem 3.7 applies it to polynomial Lipschitz functions when asserting that F is bounded on B(A0,δ). The proof of Lemma 3.4 also refers to Definition 2.2, which is not the definition being used in that lemma. Please state and prove boundedness of polynomial Lipschitz functions on bounded sets (the argument goes through because the defining polynomials are bounded on bounded sets), and cite that result in Theorems 3.7–3.9.","section":"Section 3, Lemma 3.4 and Theorem 3.7 proof"}],"minor_comments":[{"comment":"The symbol Mε is reused with a different meaning from Section 4.1: in the proof Mε = B(A0, ε) is a single ball around a singular matrix, whereas Section 4.1 defines Mε as the union of ε-neighborhoods of all singular matrices. Please use different notation.","section":"Section 3.2, Theorem 3.8"},{"comment":"The phrase 'the integral diverges as ε→0' is imprecise for k < n^2: the integral ∫_0^ε r^{n^2−1−k} dr converges, but the normalized expectation lower bound C ε^{−n^2} ∫_0^ε r^{n^2−1−k} dr ∼ C ε^{−k} diverges as ε→0. The conclusion is correct, but the wording should be adjusted.","section":"Section 3.2, Theorem 3.8 proof"},{"comment":"The sentence 'obviously the last formula larger than any real number when ε → ∞' should read 'as ε → 0', and for K ≥ n^2 the integral over (0, ε) is already infinite for every ε > 0.","section":"Appendix A.6, Theorem A.6"},{"comment":"The notation A^{adj,k,l}_{0,i,σ(i)} and A'^{adj,k,l}_{i,σ(i)} is not defined; please define the adjugate deletion notation explicitly.","section":"Section 4.2, Eq. (9)"},{"comment":"The norms L+ and L* are introduced in the definition but later results refer to L1, L2, and L∞ norms; please clarify the relationship or unify the notation.","section":"Section 2.2, Definition 2.2"},{"comment":"The analysis omits h2: the first row of W1 is zero and h2 is never defined, yet the ReLU pattern descriptions list h1,h3,...,h7. Please clarify whether h2 is identically zero or simply unused.","section":"Section 5.3 and Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be acceptable after revision: the contradictory dataset condition and the sign error in Eq. (11) are local and repairable, and the intended arguments are sound. I would not require new experiments. The claim of being the 'first proofs' may be somewhat strong relative to elementary unboundedness arguments, but I do not see a need for additional related-work verification beyond what the authors cite."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the central negative claim is correct in spirit, but the main theorem as written is vacuous, and the positive ReLU construction has a sign error. Both are fixable. Send it to review with a referee who will push on the hypotheses.\n\nWhat's actually new: the polynomial Lipschitz class is a sensible generalization that covers attention layers and residual networks, and the composition lemmas in Section 2 are fine. Once the hypotheses are mended, the no-go argument is simple and correct: polynomial Lipschitz functions are bounded on bounded sets, inversion is unbounded near a rank-(n-1) singular matrix, so no such network can approximate inversion on a domain whose closure contains one. The paper also gives a clear feasible-region analysis (stay away from singular matrices) and a Taylor-expansion-based explanation of what a trained two-layer network learns. Section 5.3's post-hoc inspection, with full parameters in Appendix E, is reproducible and honest. That is genuinely useful.\n\nSoft spots, in order. First, Theorem 3.7 assumes M contains a ball B(a,c) and excludes all singular matrices. That is contradictory: every open ball in R^{n×n} contains singular matrices, since det is a nonzero polynomial. The proof itself only needs a rank-(n-1) singular matrix in the closure of M. The same bad assumption infects Theorems 3.8 and 3.9. This is a load-bearing flaw in the statements, but easy to repair.\n\nSecond, Eq. (11) defines h2 = f0 + 1*h1+ + 1*h1-, where h1+ = ReLU(s), h1- = ReLU(-s). That computes f0 + |s|, not f0 + s. The claimed two-layer construction doesn't approximate the linear term; it takes its absolute value. The standard fix is ReLU(s) - ReLU(-s) = s, so this is repairable with the same widths. Minor: Lemma 3.4 is stated for Lipschitz functions; the boundedness claim extends to polynomial Lipschitz functions, but the paper doesn't say so.\n\nNovelty is modest. The negative bound follows from boundedness of the network on compact sets; the polynomial Lipschitz definition isn't needed for the lower bound. The 'first proofs' claim oversells it. Still, the paper documents the limitation cleanly and the feasible-region/'what the net learns' analysis is a real contribution.\n\nFor whom: anyone working on neural linear algebra or learned solvers. I'd cite it only if the hypotheses get fixed; right now I'd hesitate. But it deserves a serious referee; the flaws are not fatal, and the honest experimental section argues for engagement.","headline":"The negative result is real but elementary, and as stated the main theorem is vacuous because a dataset cannot contain a ball and exclude all singular matrices; the paper is repairable and worth a serious referee.","tokens_in":22885,"tokens_out":3476,"would_cite":false,"duration_ms":31995,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neural networks cannot learn a general matrix inverse: near any singular matrix, inversion blows up while the network stays bounded, so no polynomial Lipschitz network can cover such domains.","keywords":["matrix inversion","neural networks","polynomial Lipschitz continuity","end-to-end learning","singular matrices","scientific computing","ReLU networks","universal approximation"],"falsifier":"Fix $n=2$ and train a ReLU network end-to-end on nonsingular matrices sampled from a box whose closure contains the singular matrix with all entries equal to 1, while avoiding exact singulars in the training labels. Take a sequence of nonsingular matrices $x_m$ in that box converging to that singular matrix and measure $\\| \\mathrm{Inv}(x_m)-F(x_m)\\|_2$. The theorem predicts this error grows without bound as $m$ increases, since the inverse entries blow up while $F$ stays bounded; observing bounded error along such a sequence would disprove the claim.","tokens_in":21947,"feed_emoji":"🧮","tokens_out":11228,"duration_ms":92909,"temperature":0.7,"pith_summary":"The paper asks whether a neural network trained end-to-end can approximate the matrix inverse over a general domain, and answers no for a broad class of networks it calls polynomial Lipschitz continuous. This class includes fully connected, convolutional, attention, transformer, residual, spline, and recurrent layers, so the negative result covers most modern architectures. The obstruction is analytic: near any singular matrix of rank $n-1$, the inverse is unbounded, while every polynomial Lipschitz continuous function is bounded on bounded sets. Hence, over any training domain that accumulates at a singular matrix, no network in this class can keep the error below a fixed threshold everywhere, and certain expected moments of the error diverge. The paper then shows where the task becomes feasible: on compact regions kept a positive distance from the singular set, inversion is smooth, and a two-layer ReLU network with enough hidden units can reproduce the linear approximation and achieve small test error.","feed_headline":"No universal neural matrix-inversion model exists","feed_subtitle":"Near singular matrices, inversion blows up, so every bounded network fails on general domains.","key_machinery":"Polynomial Lipschitz continuity: a function $f$ satisfies $\\| f(x)-f(y)\\|_{L_*} \\le \\sum_{i=0}^{n_f} f_i(\\|x\\|_{L_+},\\|y\\|_{L_+}) \\|x-y\\|_{L_+}^i$, where each $f_i$ is a polynomial. This generalizes ordinary Lipschitz continuity and includes architectures whose Jacobian entries grow polynomially, such as multi-head dot-product attention, transformers, residual blocks, and spline layers. The paper proves closure under composition and concatenation, so a trained network built from such layers inherits the property, and proves such functions are bounded on bounded sets. The other load-bearing object is the adjugate formula $A^{-1}=\\mathrm{adj}(A)/\\det(A)$, which yields the blow-up estimate near a rank-$(n-1)$ singular matrix and, expanded about a nonsingular $A_0$, gives the first-order formula that the constructive two-layer ReLU network is designed to match.","core_discovery":"The paper's central claim is that no universal end-to-end neural matrix inversion model exists within the polynomial Lipschitz class. Theorem 3.7 states that for any such function $F$ and any error threshold $E>0$, there exists a data point $x$ in the nonsingular dataset with $\\| \\mathrm{Inv}(x)-F(x)\\|_L > E$; Theorem 3.8 extends this to subsets of positive measure, and Theorem 3.9 shows that for any $k>n^2$ the $k$-th moment of the error over the whole dataset is infinite. The proof combines a lower bound for inversion near a rank-$(n-1)$ singular matrix $A_0$, namely $\\| \\mathrm{Inv}(A)\\|_L > C_{A_0}/\\|A-A_0\\|_L$, with the fact that polynomial Lipschitz functions are bounded on bounded sets. The authors therefore conclude that a general matrix inversion model fails because the target function blows up where these functions necessarily stay bounded.","pith_inferences":["This is an editorial extension: the obstruction is not specific to neural networks; any approximation scheme whose outputs are bounded on bounded sets fails on domains that accumulate at a singular matrix, so the real boundary is a domain condition rather than an architecture condition.","This is an editorial extension: the distance of training data to the singular set (equivalently, its condition number) should predict end-to-end generalization, and adding even a few near-singular training examples should sharply increase worst-case error; that is testable.","This is an editorial extension: the impossibility applies to one-shot end-to-end maps; hybrid schemes that use a neural network as a warm start and then apply Newton or Schulz iterations could escape the bound, because the refinement steps handle the neighborhood where the inverse is unbounded.","This is an editorial extension: whether non-polynomial-Lipschitz architectures, such as networks with normalization or features that break the bounded-Jacobian condition, can approximate inversion on such domains remains open, as the authors themselves note."],"forward_implications":["Any network built from polynomial Lipschitz layers misses the matrix inverse by an arbitrarily large amount somewhere on any training domain that approaches a singular matrix; width and depth within this class cannot fix that.","For such domains, training with the loss $\\| \\mathrm{Inv}(x)-F(x)\\|_L^k$ with $k>n^2$ gives infinite expected loss, and for smaller $k$ a subset of positive measure still pushes the expected loss above any chosen threshold.","Feasible training sets must exclude the $\\epsilon$-neighborhood $M_\\epsilon$ of the singular set; on compact boxes $M_0$ with $M_0 \\cap M_\\epsilon = \\emptyset$, inversion is well behaved and neural approximation is possible.","A two-layer ReLU network with at least $2n^2$ hidden units can implement the linearization $(A_0+A')^{-1} \\approx f^0(A_0) + \\sum_{i,j} f^1_{i,j}(A_0) A'_{i,j}$, with error $O(\\|A'\\|^2)$, matching first-order accuracy.","Experiments on $2\\times2$, $3\\times3$, and $16\\times16$ matrices sampled in safe regions show test errors near $10^{-5}$ to $10^{-4}$, and the trained network's coefficients align with the linear approximation."],"supporting_citations":[{"why":"Supplies the Jacobian bound for multi-head dot-product attention, the key example that motivates the polynomial Lipschitz class and underpins Lemma 3.5.","marker":"Kim et al., 2021"},{"why":"Introduces neural spline flows whose element-wise polynomial layers are used as polynomial Lipschitz components in Lemma 3.5.","marker":"Durkan et al., 2019"},{"why":"Provides the automatic (spline-based) flow layers that the paper cites alongside Durkan et al. as polynomial Lipschitz building blocks.","marker":"Cai et al., 2022"},{"why":"Documents residual architectures that are not Lipschitz continuous, motivating the need for the generalized definition.","marker":"Anil et al., 2019"},{"why":"Supplies the classical matrix inversion algorithms that serve as the baseline for the neural alternatives discussed in the introduction.","marker":"Golub and Van Loan, 1996"}],"fun_headline_variants":["No universal neural matrix inversion exists","Neural nets can't invert all matrices","Singular blowup defeats neural inversion","Bounded nets fail on unbounded inversion","Inversion blows up, networks stay bounded"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the training domain contains points arbitrarily close to a singular matrix; the paper states this as a ball that both lies in the dataset and contains a rank-$(n-1)$ singular matrix after all singular matrices are excluded, which is contradictory as written, but the essential requirement is accumulation at a singular matrix, and the negative theorems do not apply to domains kept a positive distance away from all singular matrices.","fun_headline_variants_meta":{"raw":{"variants":["No universal neural matrix inversion exists","Neural nets can't invert all matrices","Singular blowup defeats neural inversion","Bounded nets fail on unbounded inversion","Inversion blows up, networks stay bounded"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1479,"prompt_tokens":867,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":483,"tokens_out":612,"duration_ms":6336,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:01:29.452637+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix $n=2$ and train a ReLU network end-to-end on nonsingular matrices sampled from a box whose closure contains the singular matrix with all entries equal to 1, while avoiding exact singulars in the training labels. Take a sequence of nonsingular matrices $x_m$ in that box converging to that singular matrix and measure $\\| \\mathrm{Inv}(x_m)-F(x_m)\\|_2$. The theorem predicts this error grows without bound as $m$ increases, since the inverse entries blow up while $F$ stays bounded; observing bounded error along such a sequence would disprove the claim.","supporting_citations":[{"cited_title":"The lipschitz constant of self-attention","cited_arxiv_id":null,"evidence_quote":"Supplies the Jacobian bound for multi-head dot-product attention, the key example that motivates the polynomial Lipschitz class and underpins Lemma 3.5."},{"cited_title":"Neural spline flows","cited_arxiv_id":null,"evidence_quote":"Introduces neural spline flows whose element-wise polynomial layers are used as polynomial Lipschitz components in Lemma 3.5."},{"cited_title":"Autm flow: Atomic unrestricted time machine for monotonic normalizing flows","cited_arxiv_id":null,"evidence_quote":"Provides the automatic (spline-based) flow layers that the paper cites alongside Durkan et al. as polynomial Lipschitz building blocks."},{"cited_title":"Sorting out lipschitz function approximation","cited_arxiv_id":null,"evidence_quote":"Documents residual architectures that are not Lipschitz continuous, motivating the need for the generalized definition."}],"review_version":1}