{"id":"2ae620f8-a919-4796-8a41-cab0d4477731","arxiv_id":"2502.06300","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a teacher-student setup, maximal expressive power for a fixed learnable-weight budget is characterized by even row or column distribution in linear RNNs and feedforward networks.","lead":"This paper studies where to place a fixed number of trainable weights in a neural network so it can imitate a teacher network. It proves clean all-or-nothing conditions for linear networks and suggests spreading learnable weights broadly improves expressivity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proof of Theorem 3.5 infers a unique F from two rank computations without proving the row spaces are disjoint, and then infers solvability for W from equal equation/variable counts; both steps are unproven.","rationale":"The paper's most important claimed result is Theorem 3.5: certain row- or column-concentrated allocations of the recurrent weights of a linear RNN achieve match probability 1. The proof strategy introduces nTb auxiliary variables F (Lemma B.3), converting the polynomial system into linear-plus-quadratic equations. Everything then hinges on a square linear system in F being uniquely solvable and on the subsequent linear systems for W being solvable. The reader flagged the joint-rank gap in the first step; I agree with that identification. I also find a second, equally load-bearing gap in the passage from a unique F to solvable W: equality of equation and variable counts is not solvability unless the relevant submatrix of gB(F) is invertible. Neither gap is filled by the empirical sections, since the numerics use approximate matching (MSE 1e-2 in the ReLU model) and solver restarts, and they only probe small n. That said, I see no counterexample to the theorem itself; in the small n=2, T=2 case the combined determinant is a nonzero polynomial, which suggests the theorem is true and the proof is repairable with a generic-determinant argument. A separate proof-rigor concern is Lemma B.12, which underpins Theorems 3.2 and 4.1 and is argued heuristically; I did not make it the primary attack because the joint-rank and solvability gaps in Theorem 3.5 are more directly tied to the paper's strongest claim. Because the current manuscript lacks the required linear-algebra genericity proof, the paper is not ready for acceptance-level confidence, matching the reader's CONDITIONAL verdict. The broader 'distribute learnable weights' principle is explicitly a conjecture (Section 6), so I would not treat that as an unacknowledged flaw. If the symbolic determinant checks in concrete_test come out nonzero, the verdict should remain CONDITIONAL or be upgraded only after the proof is repaired; if they come out identically zero, the theorem's proof route collapses and the verdict would need to move to REJECT. For now, my independent read does not change the reader's verdict.","tokens_in":25141,"tokens_out":25625,"duration_ms":239751,"concrete_test":"For a minimal nontrivial case, e.g., n=3, b=1, d=1, T=2, m=2, r=2 with one learned row (say W11 and W13 learned, W12 fixed), form the coefficient matrix of the combined system (B.5)+(B.6) in vec(F) and compute its determinant symbolically with SymPy using generic symbolic entries for X, D, B, and W_BOT. Then solve that linear system for F, form the learned-row coefficient matrix gB(F)[{1,3}, :], and compute its determinant. Repeat for a second within-row choice of learned positions. If either determinant is identically zero, the proposed linearization argument cannot establish MP=1 and a different proof would be required; if both determinants are nonzero polynomials, the gap is repairable by a generic-determinant argument and the CONDITIONAL verdict stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central maximality theorem (Theorem 3.5, proved as Theorem B.6/B.9) is not established by the argument given. After splitting W into learned TOP rows and constant BOT rows, the proof forms the linear system (B.5) (rank dm) and (B.6) (rank nTb−dm) in the nTb entries of F, and concludes there are nTb linearly independent equations. This conclusion requires the two row spaces to have trivial intersection; individual rank computations do not imply that, and no argument is supplied. The same gap appears in Theorem B.9. A second independent gap follows: once F is (hypothetically) unique, substituting into (B.3) gives, for each learned row, a Tb×Tb linear system in the Tb learnable weights (the fixed entries move to the right-hand side). The proof says solvability follows because there are Tb equations and Tb variables; this is also a non-sequitur unless the relevant submatrix of gB(F) is shown invertible. Both facts may well be true for generic Gaussian parameters, but they are the load-bearing steps that turn a calculable polynomial system into a linear one, and they are not proven. If either fails, the proof does not deliver match probability 1; if both hold, the proof needs a generic-determinant argument rather than the current rank-sum inference.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies subset learning: given a network and a fixed budget r of learnable weights, which allocation of those weights maximizes expressive power? The authors define a teacher–student benchmark, match probability, and evaluate it at m = r/d so that the number of output constraints equals the number of learnable weights. They prove exact maximal/minimal allocation conditions for a linear estimator, for the encoder, decoder, and recurrent weights of linear RNNs, and for intermediate layers of linear feedforward networks. For allocations that fall between the extremes they propose heuristic principles, which are extended to shallow ReLU networks and tested numerically. The central claim is that spreading the same number of learnable weights over more rows or columns generally increases expressive power.","tokens_in":25466,"tokens_out":12385,"duration_ms":113502,"significance":"If the proofs are completed, this is a valuable new benchmark for subset-learning expressivity. The paper cleanly separates the counting/existence question from optimization, and it makes falsifiable predictions: allocations satisfying Theorem 3.5 should reach match probability 1, while allocations violating Theorem 3.4 should have match probability 0. The numerical experiments and included code strengthen the empirical part. The main theorems, however, depend on appendix arguments that are not yet rigorous; in particular, the proof of Theorem 3.5 has two unproven linear-algebra steps, and the encoder/feedforward theorems rest on an informal lemma. The results are plausible and likely fixable, but the manuscript in its current form does not fully support the central maximality claim.","major_comments":[{"comment":"The proof computes ranks dm and nTb−dm for Equations (B.5) and (B.6) and concludes that the two systems jointly give nTb linearly independent equations in the nTb entries of F. This conclusion requires that the row spaces of the two systems have trivial intersection; individual rank computations do not imply this. The same unsupported step appears in Theorem B.9. Since the uniqueness of F is the first load-bearing step of the maximality proof, Theorem 3.5 is not established by the argument given.","section":"Theorem B.6 (proof of Theorem 3.5)"},{"comment":"Once F is fixed, the proof substitutes it into W_TOP gB(F) = F_TOP and asserts that each learned row is solvable because there are Tb equations and Tb variables per row. Solvability of a square linear system is not guaranteed by equal dimensions: the Tb×Tb submatrix of gB(F) selected by the learned columns must be shown invertible, or the right-hand side must be shown to lie in its column space. No such generic-invertibility argument is provided. The analogous step in Theorem B.9 has the same gap.","section":"Theorem B.6, step after Eq. (B.3)"},{"comment":"The second direction of Lemma B.12 is not written as a rigorous proof. It uses an induction on the number of distinct entries, but invokes 'uncorrelated' in places that require genuine independence, and it discusses probability-zero events without a measure-theoretic justification. This lemma is the foundation for Theorem B.14 (Theorem 3.2) and Theorem C.1 (Theorem 4.1), so the encoder and feedforward classifications are currently conditional on an unproved statement.","section":"Lemma B.12"},{"comment":"Lemma B.13 shows that correlations between entries of powers of W are either zero or tend to zero as n→∞, whereas Theorem 3.2 is a fixed-n statement asserting MP equal to 0 or 1. Applying Lemma B.12 requires the entries of U_i to be uncorrelated at finite n, which has not been shown. Either a finite-n argument or an explicitly asymptotic reformulation of Theorem 3.2 is needed.","section":"Theorem B.14 / Lemma B.13"},{"comment":"In the reduction from a row with ri>Tb learnable weights to a smaller allocation, the proof solves for the remaining Tb learnable weights by inverting the selected Tb×Tb submatrix of gB(F*). Invertibility of this submatrix is asserted without proof, and the relation between the constants in the original and reduced allocations is not specified. The claim that every solvable instance of A yields a solvable instance of A2 therefore does not follow as written.","section":"Theorem B.5 (first part of Theorem 3.4)"}],"minor_comments":[{"comment":"The phrase 'follows one of the following' should likely be 'satisfies both of the following'; under the literal OR reading, an allocation with no row exceeding m but one column exceeding d would be claimed maximal, which contradicts the proof in Theorem C.1.","section":"Theorem 4.1"},{"comment":"The theorem statement should explicitly note that condition 1 requires r to be divisible by Tb, and condition 2 requires r to be divisible by Td, for the respective clause to be applicable.","section":"Theorem 3.5"},{"comment":"The choice of the number of fsolve initializations is described as based on a manually observed threshold; this is acceptable as a heuristic, but it should be labeled as such in the main text.","section":"Appendix E.1"},{"comment":"The complement of an allocation is written as an overline on A, which is easy to confuse with a closure; a more explicit symbol such as A^c would improve readability.","section":"Definition 2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a machine learning theory journal, and the framing is original. The stress-test concern reported by the reader is real and lands on the proof of the central maximality theorem. I do not see circularity or a novelty problem. The gaps are likely repairable with a generic-rank or algebraic-geometry argument, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe paper is worth a serious look, but the main proof of the recurrent-weight maximality theorem is not currently made. What is genuinely new is the match-probability benchmark and the clean dichotomy results for the decoder and the linear estimator. Those proofs are simple and solid, and the transfer to feedforward nets is an honest extension. The authors also deserve credit for labeling the general 'spread the weights' principle as a conjecture and for running the small-model example that shows sub-maximal allocations can still have match probability between 0 and 1.\n\nThe soft spot is where the reader's report lands, and I think it lands correctly. In Theorem B.6 the proof builds two linear systems in the auxiliary variables F, with ranks dm and nTb-dm, and concludes the combined system has nTb linearly independent equations. That inference requires the two row spaces to be disjoint, and no argument for that is given. The same gap appears in Theorem B.9. Then, after the (hypothetical) unique F, the proof claims that each row of W can be solved because there are Tb equations in Tb unknowns. A square system is solvable only if its matrix is invertible; the matrix in question is a submatrix of gB(F), and the proof does not show it is invertible. These may well be true for generic Gaussian weights, but for a theorem claiming match probability 1, they are load-bearing and currently unproven.\n\nA second, softer issue is Lemma B.12. The first direction is fine; the second direction is informal and leans on a probabilistic argument that needs to be made precise. Theorem 3.2's encoder result depends on it. The empirical section is honest about using solver restarts and a loose MSE threshold for the ReLU case, so the numerics are suggestive rather than conclusive.\n\nIs the paper worth a serious referee? Yes. The framework is clean, the negative results (Theorem 3.4) appear correct, and the gaps look repairable, probably with a generic-determinant argument over Gaussian parameters. I would send it to review and ask the authors to fix the two missing pieces in the maximality proofs, and to tighten Lemma B.12. If those get fixed, the paper becomes a solid contribution that people working on parameter-efficient fine-tuning and sparse plasticity will want to cite.","headline":"A clean benchmark and solid linear results, but the main maximality theorem for recurrent weights has a real, likely repairable proof gap.","tokens_in":25901,"tokens_out":2588,"would_cite":true,"duration_ms":21823,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fixed budget of learnable weights can be maximally expressive or completely useless depending only on how the weights are spread across rows and columns.","keywords":["allocation strategy","subset learning","expressive power","teacher-student setup","match probability","linear recurrent neural network","parameter-efficient learning"],"falsifier":"Take a small linear recurrent network satisfying the hypotheses of Theorem 3.5, form the coefficient matrices of equations (B.5) and (B.6) from the proof, and compute the rank of the stacked system: if the rank is less than $nTb$, the unique-solution step fails and a maximal allocation could still miss the teacher on a set of positive probability.","tokens_in":24937,"feed_emoji":"🧠","tokens_out":6623,"duration_ms":673599,"temperature":0.7,"pith_summary":"The paper asks a resource-constrained question: if only r of a network's weights can be learned and the rest stay frozen at random values, which choices of learnable positions maximize the chance that the student exactly reproduces a teacher's outputs on m samples? It answers with a benchmark, the match probability MP(A), and a set of conditions in linear recurrent and feedforward networks. The central finding is that allocations can be maximal (match probability 1, exact reproduction) or minimal (match probability 0) depending only on how the r weights are spread across rows and columns, with a sharp transition between the two. The paper also argues that, as a rule, distributing the same fixed budget over more neurons raises expressive power, and it gives empirical evidence for this in shallow ReLU networks.","feed_headline":"Where you place a fixed weight budget flips expressivity from 0 to 1","feed_subtitle":"Spreading r learnable weights one block per row or column lets a student exactly clone the teacher","key_machinery":"The machinery is the student-teacher match probability MP(A,m), defined as the probability that a student whose learnable positions are fixed by allocation A and whose other weights are random can exactly match the teacher's outputs on m samples. The proof engine is Lemma 3.3, which rewrites the degree-T polynomial matching equations as a linear-quadratic system in the recurrent weights W and auxiliary variables F; when whole rows or columns of W are frozen, the auxiliary equations become linear. Counting the ranks of these linear systems with Kronecker products (the tensor product operation that turns matrix equations into ordinary linear systems) shows when F is forced to a unique value, after which the learnable blocks in W are determined by square linear systems.","core_discovery":"On the paper's own terms, the discovery is that in a linear recurrent network, learning the recurrent matrix W with r = dm learnable weights is maximal whenever each active row of W contains exactly Tb learnable weights, or each active column contains exactly Td learnable weights, and no allocation with any row above Tb or any column above Td can match the teacher. This structure turns a system of high-degree polynomial equations into a forced linear system, so a student with only r learnable weights reproduces all m teacher outputs with probability 1. The same block-counting logic gives exact conditions for the decoder (m weights per row), the encoder (no row above Tm and no column above Td), and intermediate feedforward layers (no row above m and no column above d).","pith_inferences":["A testable prediction follows for neuroscience: if the brain has a fixed budget of synaptic changes, behavioral competence should show near-threshold transitions as the fraction of modified neurons crosses a row or column bound, rather than smooth improvement.","The row and column bounds could give a first-order estimate for parameter-efficient fine-tuning: when a constrained update fits data exactly, it should be visible as a square linear system after the same auxiliary-variable reduction used here.","Because match probability only asks whether a solution exists, not whether gradient descent finds it, the practical gain from spreading weights may be smaller than the theory suggests; the gap between exact solvability and trainability is a natural next benchmark.","A concrete extension of the paper's conjecture would hold r fixed and vary only the number of rows used by the allocation: match probability should increase monotonically with the row count in nonlinear networks as well."],"forward_implications":["A fixed budget r that concentrates too many learnable weights in one row or column makes the student unable to match the teacher at all, whereas the same budget spread one block per row or column succeeds with probability 1.","The decoder result is rigid: in a linear RNN readout, every row must have exactly m learnable weights, so moving a single weight between rows flips the match probability from 1 to 0.","For recurrent connections, maximality is reached at the minimal allowed number of active rows, r/(Tb), so the transition from zero to full expressivity is sharp at that threshold.","In linear feedforward networks, the same row and column conditions hold per layer, with all allocations that violate them being minimal.","If the paper's conjecture about random polynomial systems is right, larger networks should make even suboptimal allocations approach match probability 1, so resource constraints become less damaging as model size grows."],"supporting_citations":[{"why":"Supplies the Kronecker product rank identity used to count the ranks of the auxiliary linear systems in the maximality proofs.","marker":"[Horn and Johnson, 1991]"},{"why":"Used in Lemma B.4 to assert that a polynomial system with fewer variables than equations has zero probability of a solution, which underlies the minimality arguments.","marker":"[Azas and Wschebor, 2009]"},{"why":"Provides the student-teacher setup in which the match probability benchmark is defined.","marker":"[Gardner and Derrida, 1989]"},{"why":"Motivates measuring expressivity by the probability of realizing labels on samples, the basis of the match probability definition.","marker":"[Cover, 1965]"},{"why":"Supplies the result on random polynomial systems that motivates the conjecture that match probability grows with the number of polynomial equations.","marker":"[Subag, 2024]"}],"fun_headline_variants":["Weight placement flips expressivity from zero to one","Strategic weight spread lets a student clone the teacher","Fixed weight budget: placement dictates perfect teacher cloning","Allocation of learnable weights decides net's expressive power"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The maximality proof adds up the number of independent equations in two separate linear systems and assumes no equation is shared between them; if the two systems overlap, the unique-solution argument collapses.","fun_headline_variants_meta":{"raw":{"variants":["Weight placement flips expressivity from zero to one","Strategic weight spread lets a student clone the teacher","Fixed weight budget: placement dictates perfect teacher cloning","Allocation of learnable weights decides net's expressive power"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001312,"raw_usage":{"total_tokens":5325,"prompt_tokens":901,"completion_tokens":4424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":4362}},"tokens_in":517,"tokens_out":4424,"duration_ms":26189,"temperature":1.0,"reasoning_tokens":4362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:53:13.816889+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small linear recurrent network satisfying the hypotheses of Theorem 3.5, form the coefficient matrices of equations (B.5) and (B.6) from the proof, and compute the rank of the stacked system: if the rank is less than $nTb$, the unique-solution step fails and a maximal allocation could still miss the teacher on a set of positive probability.","supporting_citations":[{"cited_title":"Level sets and extrema of random processes and fields","cited_arxiv_id":null,"evidence_quote":"Used in Lemma B.4 to assert that a polynomial system with fewer variables than equations has zero probability of a solution, which underlies the minimality arguments."},{"cited_title":"Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition","cited_arxiv_id":null,"evidence_quote":"Motivates measuring expressivity by the probability of realizing labels on samples, the basis of the match probability definition."},{"cited_title":"Concentration for the zero set of large random polynomial systems","cited_arxiv_id":"2303.11924","evidence_quote":"Supplies the result on random polynomial systems that motivates the conjecture that match probability grows with the number of polynomial equations."}],"review_version":1}