{"id":"d9030511-90e1-4865-84f9-7df0f798f0ce","arxiv_id":"2411.09004","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A unified geometric and thermodynamic account of deep linear network training, combining invariant manifolds, an exact Boltzmann entropy, and Riemannian Langevin dynamics.","lead":"This paper surveys how deep linear network training can be described by Riemannian geometry, entropy, and stochastic Langevin dynamics. It collects prior theorems into a thermodynamic picture of implicit regularization in linear networks.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central thermodynamic formulas—entropy (6.2), orthonormal basis (8.20)–(8.24), and RLE (12.10)–(12.13)—are deferred to an unpublished companion paper [34], so the main claim is not verifiable from this manuscript.","rationale":"The reader's verdict is CONDITIONAL, and my concern reinforces that condition rather than changing it. My focus is slightly different from the reader's stated weakest assumption: rather than the rank-deficient limit, the more load-bearing gap is that the paper's own positive results—the entropy formula and the RLE—are asserted with proof deferred to the companion reference [34]. The rank-deficient limitation is explicitly acknowledged in §14.4 as an open problem, so it weakens the breadth of the thermodynamic claim but does not threaten its validity in the regime where it is stated. By contrast, the full-rank distinct-singular-value regime is precisely where the paper asserts exactness, yet the proofs are not in the manuscript. The paper is an expository synthesis; if [34] is a reliable published source, the exposition may be acceptable. But with only a bare 2024 citation, the exact formulas (6.2), (8.20)–(8.24), and (12.10)–(12.13) cannot be checked by the reader. The concrete test above would settle whether this concern lands: if the deferred derivations check out, the central claim stands; if not, the manuscript should be revised to include the proofs or to mark these results as conditional on [34].","tokens_in":28807,"tokens_out":17188,"duration_ms":156092,"concrete_test":"Obtain the companion paper [34] and independently verify two identities: (i) the determinant of the pullback metric z♯ι restricted to the O_d^{N-1} directions in Lemma 2 equals the square of the right-hand side of (6.2); and (ii) the Itô SDE (12.10) has generator β^{-1}Δ_{g_N} and the drift in (12.12)–(12.13) is −grad_{g_N}F_β. If [34] is not publicly available, the central thermodynamic claim remains unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's novel thermodynamic assertions are not proved in this manuscript. Theorem 10 (entropy formula, Eq. 6.2) is dispatched with 'may be computed explicitly using Theorem 14, yielding Theorem 10'; Theorem 14 is called the 'technical core of [34]' and is stated without proof; and the RLE for the end-to-end matrix, Eq. (12.12) with free-energy gradient (12.13), is justified by 'Then we show in [34] that...'. Reference [34] appears only as 'G. Menon and T. Yu, An entropy formula for the deep linear network, 2024', with no venue or preprint identifier. The sketches in Sections 6.4 and 8.3 do not supply the determinant evaluation leading to (6.2) nor the generator calculation leading to (12.10). Since the central claim—an exact thermodynamic description with explicit entropy and RLE—rests entirely on this deferred material, the manuscript's argument is incomplete: if [34] is unavailable or contains an error, the main claim has no support in this paper. This is not a charge of incorrectness; it is a verification gap that currently prevents independent assessment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This expository paper develops a geometric theory of training dynamics in deep linear networks (DLNs). The proofs in Section 7 establish Theorems 1–3: the phase space M_d^N is foliated by invariant G-balanced varieties; on balanced manifolds the end-to-end matrix evolves by the Riemannian gradient flow of the loss E with respect to the exactly computable metric g_N in (4.16), diagonalized explicitly in Lemma 1. The paper then constructs a thermodynamic picture: the fiber of the projection phi over W in the balanced manifold is an O_d^{N-1} orbit whose volume is claimed to be the Vandermonde-type formula (6.2) (Theorem 10); the metric g_N arises by Riemannian submersion (Theorem 13); an explicit orthonormal basis for the balanced manifold is given in (8.20)–(8.24) (Theorem 14); and a Riemannian Langevin equation with explicit free-energy gradient (12.12)–(12.13) is proposed as a stochastic extension, modeled on Dyson Brownian motion (Section 11). Theorems 10, 13, 14, and 18 are attributed to the author's unpublished companion manuscript [34] and are presented only as sketches.","tokens_in":29107,"tokens_out":24069,"duration_ms":182782,"significance":"If correct, this framework is a remarkably complete exact description of DLN training: an invariant foliation, a parameter-free Riemannian metric, entropy as the logarithm of a group-orbit volume, and an explicit noise model with a Coulomb-like repulsion term (12.8). The proofs that are actually present (Section 7, Lemmas 1–2) are direct, reproducible matrix computations, and the paper is candid about its modeling axiom (the RLE is phenomenological, Section 10.1) and about the rank-deficient gap (Section 14.4). I independently verified several central formulas: (7.13)–(7.14) follow from a geometric sum; (7.15) is the correct reciprocal metric; the interior (N−1)×(N−1) blocks of the pullback metric (8.10) have determinant 2^{N−1}(σ_k^2−σ_l^2)/(σ_k^{2/N}−σ_l^{2/N}), which yields (6.2); and (12.13) matches the g_N-gradient of −β^{−1}S computed from (6.2). The main weakness is verifiability: the headline quantitative claims are deferred to an unpublished, unlocatable companion [34], so the manuscript cannot currently serve as a standalone rigorous reference.","major_comments":[{"comment":"The paper's central new quantitative claims—Theorem 10 (entropy formula (6.2)), Theorem 14 (orthonormal basis (8.20)–(8.24)), and Theorem 18 (Brownian motion SDE (12.10)), together with the projected RLE (12.12)–(12.13)—are quoted from the author's companion manuscript [34], which has no venue or preprint identifier. The manuscript itself locates the omissions: Section 8 calls Theorem 14 'the technical core of [34]' and Section 12.2 says 'Then we show in [34] that…'. The determinant evaluation converting the block-tridiagonal metric (8.9)–(8.10) into (6.2), the Chebyshev-polynomial diagonalization behind (8.20)–(8.24), and the Itô/generator calculation behind (12.10) are not supplied. Because the entropy formula and the RLE are advertised in the abstract as the paper's contribution, this is a load-bearing verification gap rather than a presentation detail. I verified that the deferred claims are consistent with what is shown: the interior (N−1)×(N−1) blocks of (8.10) have determinant 2^{N−1}(σ_k^2−σ_l^2)/(σ_k^{2/N}−σ_l^{2/N}), which yields (6.2), and (12.13) agrees with the g_N-gradient of the entropy from (6.2). The concern is therefore completeness, not evident error. Please include the proofs (an appendix or supplement would suffice) or update [34] to a publicly verifiable form with results keyed to specific statements.","section":"§6.2, §8, §8.3, §12.2; ref. [34]"},{"comment":"The entropy and RLE results are established only on the full-rank stratum with distinct singular values, where the SVD parametrization z is a local bijection (Section 5.3), and Section 14.4 explicitly concedes that singularities at repeated and repeated-zero singular values are not understood. Yet the motivating phenomena—rank drops during training (Section 13.3), convergence to low-rank energy minimizers (Section 14.1), and the suggested 'entropic origin for implicit regularization' (Sections 1 and 13.4)—live in the excluded rank-deficient regime, and the entropic selection principle (6.9) is explicitly conjectural (Section 6.3). I credit the paper for flagging the limitation in Sections 6.1 and 14.4; however, the abstract and introduction claim 'exact formulas for a Boltzmann entropy' and a thermodynamic framework without this scoping caveat. Please state in the abstract and introduction that the exact entropy and RLE formulas hold on the full-rank, distinct-singular-value stratum and that the rank-deficient and repeated-spectrum cases are open.","section":"§5.3, §6.1, §13.3–§13.4, §14.1, §14.4; abstract"}],"minor_comments":[{"comment":"The theorem as stated ('For each W ∈ (M_d, g_N)') omits the hypothesis that the singular values of W be distinct; formula (6.2) and the parametrization (5.15)–(5.17) degenerate at repeated singular values (cf. Section 5.3). State the hypothesis explicitly.","section":"§6.2, Theorem 10"},{"comment":"The range '1 ≤ p ≤ n' should be '1 ≤ p ≤ N − 1'; the integer n is not defined in this section.","section":"§4.2, Eq. (4.8)"},{"comment":"The title 'RLE for stochastic gradient descent' and the abstract's phrase 'stochastic gradient descent of free energy' should be qualified: Section 10.1 states that the RLE is a phenomenological model that is 'not stochastic gradient descent (SGD) corresponding to batch processing'. Suggest rewording to avoid the implication of equivalence to practical SGD.","section":"§10.1, §12.1, abstract"},{"comment":"The symbol 'a ∈ R\\{0}' is printed after the limiting matrix (14.4), but it parameterizes the family of rank-one matrices (14.3); move it to (14.3).","section":"§14.1, Eqs. (14.3)–(14.4)"},{"comment":"Reference [34] should be updated to include a preprint identifier or a 'to appear' venue; as printed it cannot be located, which compounds the verification gap described in Major Comment 1.","section":"Ref. [34]"},{"comment":"The symbol M_r is used both for the rank-r leaf of the balanced variety (Section 3.3) and for the rank-r matrix manifold (4.20); the relationship is explained, but the double use is a readability hazard.","section":"§3.3, §4.2, Eq. (4.20)"},{"comment":"The phrase 'van(A) denoted' should read 'van(A) denotes'.","section":"§6.2, Eq. (6.3)"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is an expository survey that draws heavily on the author's own prior work ([14], [22]) and on the unpublished companion [34]. Self-citation as such is not the problem—theorems quoted from published sources are standard in surveys—but acceptance should be conditioned on [34] being made publicly locatable, since Theorems 10, 14, and 18 carry the paper's advertised novelty. My spot-checks found the deferred formulas internally consistent with the material that is proved (Lemma 1, the block structure (8.9)–(8.10), and the entropy-gradient identity), so the risk is verification rather than evident error. The paper reads as a monograph chapter or survey for an applied-mathematics or dynamical-systems audience; its claims of relevance to practical deep learning (Sections 13.3–13.4) are heuristic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing before you open it: this is an expository synthesis, not a new-results paper, and its main thermodynamic claims—the entropy formula, the orthonormal basis, and the Riemannian Langevin equation—are deferred to an unpublished companion paper [34] with no preprint identifier. The manuscript is not self-contained on the point that matters most.\n\nThe paper does a lot well. The Riemannian-submersion viewpoint is used consistently, and the proofs of the invariant manifold theorems and the explicit diagonalization of the metric are readable. The author is honest about what is his and what is a composite of other people's results, and he openly lists the rank-deficient case and repeated singular values as unresolved. The connections to the Simons cone, Dyson Brownian motion, and Bures-Wasserstein geometry are genuinely illuminating.\n\nThe soft spot is the deferred material. Theorem 10 (entropy), Theorem 13 (Riemannian submersion), Theorem 14 (orthonormal basis), and Theorem 18 (RLE) are quoted from [34] with only sketches. The determinant evaluation behind (6.2) and the generator calculation behind (12.10)–(12.13) are not given. Since these are the foundation of the 'thermodynamic framework,' a referee cannot verify the central claim from this manuscript. This is a verification gap, not a sign of error; the author's other published papers contain independent proofs of related results. But [34] itself is unavailable, so the gap is real. A revision that posts [34] or includes the missing proofs in an appendix would remove it.\n\nOne minor note: the reader's worry about Lemma 1 seems misplaced. I checked the spectral computation; the formula for the eigenvalues of A is correct, and the metric in (7.15) uses the reciprocal, so there's no inconsistency.\n\nThe rank-deficient limit is a limitation, but it's not hidden—Section 14.4 says the singularities are not understood, and that's treated as an open problem.\n\nWho gets value: mathematicians and theoretically oriented ML researchers who want the geometry of DLNs in one place. It is a good survey and deserves a serious referee, but the review should require access to [34] or an appendix with the proofs. I would not cite it in my own work until the companion is available; I'd cite the original sources instead.","headline":"A valuable expository synthesis of deep linear network geometry, but its central thermodynamic theorems are deferred to an unavailable companion paper, so the manuscript is not self-contained.","tokens_in":29582,"tokens_out":5411,"would_cite":false,"duration_ms":43730,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","53C20","60H10","58J65"],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep linear network training is exactly Riemannian gradient descent combined with a Boltzmann entropy that selects among degenerate minimizers.","keywords":["deep linear network","gradient flow","Riemannian submersion","balanced manifold","Boltzmann entropy","Riemannian Langevin equation","matrix completion","implicit regularization"],"falsifier":"Run the balanced gradient flow with $E(W)=\\frac12((W_{11}-1)^2+(W_{22}-1)^2)$ for $d=2$ from random full-rank balanced initial conditions and compare the empirical low-rank limits with the minimizers of $F_\\beta(W)=E(W)-\\beta^{-1}S(W)$. If the selected limit is not among the free-energy minimizers as $\\beta$ and $N$ vary, or if rank does not drop, the entropic selection principle fails.","tokens_in":28624,"feed_emoji":"🧮","tokens_out":10458,"duration_ms":177828,"temperature":0.7,"pith_summary":"This paper gives a unified, expository account of training dynamics in the deep linear network (DLN): a deep network whose layers are matrices and whose output is their product. It claims that the gradient flow of any loss is constrained to invariant G-balanced varieties, and that on the balanced manifold the end-to-end matrix evolves by Riemannian gradient descent with an explicitly computable metric. The article then attaches a thermodynamics to this geometry: the volume of the group of weight factors giving the same end-to-end matrix defines a Boltzmann entropy with a closed Vandermonde formula, and stochastic gradient descent is modeled by a Riemannian Langevin equation whose free energy gradient is explicit. A sympathetic reader would care because this turns the vague notion of implicit regularization for degenerate losses, such as matrix completion, into an exact selection principle: entropy, not just the loss, governs which low-rank limit is reached.","feed_headline":"Boltzmann entropy selects the limit of deep linear training","feed_subtitle":"Invariant manifolds and orbit volumes turn the training flow into Riemannian descent plus an entropic selection rule.","key_machinery":"The machinery is a chain of geometric identifications. The G-balanced varieties are the invariant sets of the gradient flow; the SVD parametrization $W_p = Q_p \\Lambda Q_{p-1}^T$, with $\\Lambda=\\Sigma^{1/N}$, exhibits each balanced manifold as orbits of the orthogonal group and makes tangent directions explicit. The metric $g_N$ is carried from the Frobenius inner product upstairs by Riemannian submersion, and its inverse operator $A_{N,W}$ is diagonalized by the singular-vector matrices $u_k v_l^T$ with eigenvalue $(\\sigma_k^2-\\sigma_l^2)/(\\sigma_k^{2/N}-\\sigma_l^{2/N})$ for $k\\neq l$ and $N \\sigma_k^{2-2/N}$ on the diagonal blocks. The entropy $S(W)=\\log \\mathrm{vol}(O_W)$ is the logarithm of the group-orbit volume, computed from the pullback metric through an orthonormal basis built from Chebyshev polynomials; its closed form is a ratio of Vandermonde determinants. The stochastic side is carried by Brownian motion on the quotient manifold and an explicit Itô SDE whose drift contains the mean-curvature correction that becomes Coulomb repulsion in singular-value coordinates.","core_discovery":"The central discovery is that overparametrization in a DLN does not merely add redundant coordinates; it organizes phase space into invariant algebraic varieties $M_G$ defined by the quadratic balance equations $W_{p+1}^T W_{p+1} = W_p W_p^T - G_p$, with the case $G=0$ giving balanced manifolds $M_r$ of rank $r$. On $M_r$ the map from weights to the end-to-end matrix $W$ is a Riemannian submersion, so the flow of $W$ is exactly the Riemannian gradient flow of $E$ with respect to the metric $g_N(Z,Z)=\\mathrm{Tr}(Z^T A_{N,W}^{-1} Z)$, where $A_{N,W}$ is a positive operator diagonalized in singular-vector coordinates by explicit eigenvalues. The same submersion structure makes the volume of a group orbit $O_W = \\varphi^{-1}(W) \\cap M$ computable in closed form as $c_d^{N-1} \\sqrt{\\mathrm{van}(\\Sigma^2)/\\mathrm{van}(\\Sigma^{2/N})}$, and this volume defines the Boltzmann entropy $S(W)=\\log \\mathrm{vol}(O_W)$. With Brownian motion on $(M_d,g_N)$, these ingredients produce the Riemannian Langevin equation for $W$, whose drift is $-\\mathrm{grad}_{g_N}F_\\beta$ with $F_\\beta = E - \\beta^{-1} S$; the entropy gradient is expressed through a diagonal matrix $\\Sigma'$ whose off-diagonal terms act like Coulombic repulsion between singular values. This gives a complete thermodynamic description of training in the full-rank, distinct-singular-value regime.","pith_inferences":["The paper leaves implicit that if the thermodynamic description extends past full rank, implicit regularization is an entropic force: the volume of weight symmetries biases training toward macrostates with larger orbit volume, and this bias should be visible in the distribution of low-rank limits even before any explicit penalty is added.","One testable extension the paper does not run is to compare the empirical limits of balanced gradient flow with tiny gauge noise against the minimizers of $F_\\beta$ for the $d=2$ matrix-completion energy $E(W)=\\frac12((W_{11}-1)^2+(W_{22}-1)^2)$; the theory predicts that the selected rank-one matrix $\\begin{pmatrix}1&a\\\\1/a&1\\end{pmatrix}$ is the free-energy minimizer for each $\\beta$ and $N$.","The anisotropic splitting of noise into gauge and observable directions predicts a concrete experimental signature: adding isotropic weight noise should change the selected low-rank limit in a computable way, whereas noiseless gradient flow should approach the entropic selection only through round-off errors.","The Simons-cone connection suggests that the rank-deficient balanced variety may carry singular minimal-surface structure; one could test whether tangential noise on low-rank varieties drives them toward the balanced manifold by motion by curvature, which would explain rank collapse as a geometric phenomenon rather than a pure optimization effect."],"forward_implications":["On balanced manifolds, the end-to-end matrix follows a Riemannian gradient flow, so overparametrization does not change the energy-decay identity $dE/dt = -\\|\\mathrm{grad}_{g_N}E\\|^2$; convergence to critical points happens at a rate set by the geometry, not by the number of parameters.","For degenerate losses, minimizers of the free energy $F_\\beta = E - \\beta^{-1} S$ supply an entropic selection principle; the paper conjectures that the minimizing set is a single point $A_\\beta$ and that training with round-off noise selects its $\\beta\\to\\infty$ limit.","Stochastic noise in the gauge directions is not a nuisance but the microscopic source of entropy: the Riemannian Langevin equation shows that tangential fluctuations produce a deterministic motion by mean curvature of the group orbits, exactly as in Dyson Brownian motion.","The explicit formulas extend to an infinite-depth limit $N\\to\\infty$, where the metric and entropy survive through the operator $A_{\\infty,W}=\\int_0^1 (WW^T)^{1-s} Z (W^T W)^s ds$ even though the upstairs group-orbit picture disappears.","The Bures-Wasserstein geometry of positive definite matrices arises as the $N=2$, $W_2=W_1^T$ specialization, connecting DLN training to optimal transport and to known gradient flows on matrix manifolds."],"supporting_citations":[{"why":"It proves that every G-balanced variety is invariant under the gradient flow and derives the balanced equation for the end-to-end matrix.","marker":"[3]"},{"why":"It introduces the metric $g_N$ on rank manifolds and establishes that the balanced flow is the Riemannian gradient flow of $E$.","marker":"[5]"},{"why":"It supplies the closed-form Boltzmann entropy, the orthonormal-basis computation of the pullback metric, and the Riemannian Langevin equation for the DLN.","marker":"[34]"},{"why":"It provides numerical evidence for rank collapse in matrix completion and the explicit infinite-depth limit operator $A_{\\infty,W}$.","marker":"[14]"},{"why":"It gives the Riemannian-submersion construction of Dyson Brownian motion and motion by mean curvature that motivates the Riemannian Langevin structure.","marker":"[22]"}],"fun_headline_variants":["Orbit volumes define Boltzmann entropy in deep linear nets","Deep linear training as Riemannian Langevin flow","Geometry of deep linear nets yields entropic selection","Boltzmann entropy from orbit volumes in deep linear training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The exact entropy formula, the metric identification, and the Riemannian Langevin equation are proved on the full-rank balanced manifold with distinct singular values; the rank-deficient limit that training actually approaches is not covered by these proofs.","fun_headline_variants_meta":{"raw":{"variants":["Orbit volumes define Boltzmann entropy in deep linear nets","Deep linear training as Riemannian Langevin flow","Geometry of deep linear nets yields entropic selection","Boltzmann entropy from orbit volumes in deep linear training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000698,"raw_usage":{"total_tokens":3173,"prompt_tokens":981,"completion_tokens":2192,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2131}},"tokens_in":597,"tokens_out":2192,"duration_ms":15963,"temperature":1.0,"reasoning_tokens":2131,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:12:18.931248+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the balanced gradient flow with $E(W)=\\frac12((W_{11}-1)^2+(W_{22}-1)^2)$ for $d=2$ from random full-rank balanced initial conditions and compare the empirical low-rank limits with the minimizers of $F_\\beta(W)=E(W)-\\beta^{-1}S(W)$. If the selected limit is not among the free-energy minimizers as $\\beta$ and $N$ vary, or if rank does not drop, the entropic selection principle fails.","supporting_citations":[{"cited_title":"Arora, N","cited_arxiv_id":null,"evidence_quote":"It proves that every G-balanced variety is invariant under the gradient flow and derives the balanced equation for the end-to-end matrix."},{"cited_title":"Menon and T","cited_arxiv_id":null,"evidence_quote":"It supplies the closed-form Boltzmann entropy, the orthonormal-basis computation of the pullback metric, and the Riemannian Langevin equation for the DLN."},{"cited_title":"Cohen, G","cited_arxiv_id":null,"evidence_quote":"It provides numerical evidence for rank collapse in matrix completion and the explicit infinite-depth limit operator $A_{\\infty,W}$."},{"cited_title":"Huang, D","cited_arxiv_id":null,"evidence_quote":"It gives the Riemannian-submersion construction of Dyson Brownian motion and motion by mean curvature that motivates the Riemannian Langevin structure."}],"review_version":1}