{"id":"7cdc993c-6544-4788-80b5-5a99993e1f5c","arxiv_id":"2506.13714","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Under MSE training of deep linear networks, data augmentation and hard-wired invariance share global optima and critical points; regularization adds saddles and its solution path limits to the hard-wired optimum.","lead":"This paper proves that for rank-bounded linear networks trained with squared error, data augmentation and hard-wired invariant constraints produce the same optimal solutions and the same critical points, while regularization adds extra saddle points. It also claims the regularized solution path is continuous and reaches the hard-wired optimum as the penalty grows.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.6 is false as stated: a full-rank 3x3 example exhibits a crossing sigma_1(lambda)=sigma_2(lambda), where no continuous single-valued global-optimum path exists.","rationale":"The reader's weakest assumption identified the same mechanism: Theorem 3.6's proof of continuity relies on Lemma A.4 without verifying that the relevant singular values remain in non-merging groups. My stress test goes further by exhibiting a concrete instance satisfying every stated hypothesis of Theorem 3.6 (full rank of Z_reg for all lambda>=0, r=0. Its singular values are 2a, 1, epsilon; at lambda0=1/4, a=1/2 and sigma_1=sigma_2=1. For lambdalambda0 it is e1 e2^T (entry (1,2)=1). The one-sided limits as lambda -> lambda0 are distinct, so no continuous single-valued selection of global optima exists. The theorem can be repaired by adding the missing non-crossing assumption, but as stated it is false.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies mean squared error regression with deep linear networks under cyclic group invariance, comparing three ways of imposing invariance: hard-wired constraints, data augmentation, and explicit regularization. The main claimed results are that hard-wiring and data augmentation have identical global optima (Theorems 3.3, 3.8, 3.9), that the regularization path is continuous and converges to the hard-wired solution (Theorem 3.6), and that the critical points of the hard-wired and data-augmented problems coincide and consist of saddles plus the global optimum, while regularization introduces additional saddles (Proposition 3.10). The paper also includes MNIST experiments and an extension to shallow nonlinear networks via the neural tangent kernel. The writing is clear and the paper addresses a timely question, but several load-bearing technical claims require repair.","tokens_in":33567,"tokens_out":9102,"duration_ms":93048,"significance":"If the stated results hold, the paper gives a valuable theoretical explanation of when data augmentation, regularization, and hard-wired invariance are interchangeable in a non-convex rank-constrained linear model. The exact characterizations of global optima and critical points in a determinantal variety are useful, and the treatment is essentially parameter-free, deriving closed-form solutions from Eckart-Young theory and known critical-point results for low-rank approximation. The experiments provide supporting evidence for the linear setting and an honest discussion of limitations. However, the correctness of the central claims currently depends on unproved or false technical assumptions, so the significance can only be assessed after the issues below are resolved.","major_comments":[{"comment":"Theorem 3.6 is false as stated. The proof invokes Lemma A.4 (Dieci et al.), whose hypothesis requires the singular values to split into disjoint groups that never merge, but the theorem only assumes that Z_reg(λ) = Z B(λ)^{-1} has full rank for all λ ≥ 0. Full rank does not prevent σ_r(λ) and σ_{r+1}(λ) from crossing at the truncation boundary, and at such a crossing the best rank-r approximation is not unique and a single-valued continuous regularization path need not exist. Concretely, take d0 = dL = 3, X = I_3, Y = Z with columns (0,2,0), (1,0,0), (0,0,ε) for 0 < ε < 1, and eG = diag(2,0,0). Then r = 1, nullity(eG) = 2, so the stated setting applies. With B(λ)^{-1} = diag(a,1,1) and a = 1/sqrt(1+12λ), the singular values of Z_reg are 2a, 1, ε. At λ0 = 1/4 they satisfy σ_1 = σ_2 = 1; for λ < λ0 the unique global optimum is 2a^2 e2 e1^T, and for λ > λ0 it is e1 e2^T. The one-sided limits at λ0 are distinct, so no continuous single-valued selection of global optima exists. The theorem can likely be repaired by adding a non-crossing assumption for the singular values, but as written the statement and its proof do not establish the advertised continuity.","section":"Theorem 3.6 and Appendix A.5"},{"comment":"The claim that the critical points of the constrained problem and the data-augmented problem are identical is not proved. The proof says the statement follows directly from Proposition A.10 and the fact that Z_inv and Z_da both have rank d, but the argument only counts C(d,r) critical points in each case. Equal cardinality does not imply equal sets. The two descriptions, U_inv Σ_inv_I V_inv^T P^{-1} and U_da Σ_da_I V_da^T Q^{-1}, are not shown to coincide for arbitrary subsets I; Theorem 3.9 establishes equality of the global optima, i.e., only for I = [r]. To support the set-identity claim, the authors need either to prove that the two objective functions agree up to an additive constant on the invariant subspace, or to provide a direct identification of the critical sets. Otherwise the statement should be weakened to an equality of cardinalities or to a statement about correspondence under the coordinate change used in the proofs.","section":"Proposition 3.10 and Appendix A.8"},{"comment":"The proof of Theorem 3.9 contains an algebraic error in the simplification of the averaged matrix. The step (Λ_g A Λ_g^†)_{ij} = (diag(Λ_g) diag(Λ_g)^†)_{ij} A_{ij} is not valid off the diagonal: in fact (Λ_g A Λ_g^†)_{ij} = λ_i A_{ij} \\bar{λ}_j. Averaging over the group gives 1 whenever λ_i = λ_j, even when i ≠ j, so the averaged matrix is block diagonal with respect to the eigenspaces of ρ_X(g), not merely the 1-eigenspace. The final identity in Eq. (47) may still be salvageable because the subsequent multiplication by G on the left retains only the trivial block, but as written the proof needs a corrected derivation of the block structure. This is a load-bearing point because Theorem 3.9 is the central equivalence result between data augmentation and hard-wiring.","section":"Appendix A.7, Eq. (50)"}],"minor_comments":[{"comment":"Remark 3.7 states that the full-rank assumption in Theorem 3.6 is mild and holds with probability 1, but the discussion after the theorem should acknowledge that full rank alone does not imply the non-crossing condition actually needed for continuity; the remark should be revised accordingly.","section":"Remark 3.7"},{"comment":"In Case 1 of the proof of Theorem 3.3, the sentence 'We assume Z has rank d' introduces an assumption that is not stated in the theorem; the case should be justified or the theorem statement amended.","section":"Theorem 3.3 proof, Appendix A.3"},{"comment":"There is a typo in the first paragraph of Appendix A.12: 'Besdies' should be 'Besides'.","section":"Appendix A.12"},{"comment":"The symbol G is used both for the group and for the matrix I_{d0} - ρ_X(g); although the context distinguishes them, a different symbol for the matrix would improve readability.","section":"Notation, Proposition 3.1"},{"comment":"The sentence beginning 'We observe that fully-connected linear networks have no spurious local minima... suggesting that also for our three optimization problems there are no spurious local minima' should be phrased as a conjecture or supported by a precise reference, since the preceding proposition concerns function-space critical points and the parameter-space statement is only heuristic.","section":"After Proposition 3.10"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting question, and the intended results are plausible, but the current version contains a false continuity theorem and an unsupported critical-point identity. I recommend major revision rather than rejection because both issues appear repairable: the continuity claim needs a non-crossing assumption, and the critical-point identity needs a direct proof or a weakened statement. The proof of Theorem 3.9 also requires correction of the block-diagonal computation. The relationship to the earlier work of Nordenfors et al. on equivariant dynamics should be sharpened to make the novelty precise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's what you need to know. This paper compares hard-wiring, data augmentation, and regularization for imposing invariance in deep linear networks under MSE loss. The genuinely new result is a characterization of all critical points in the rank-bounded function space, showing that data augmentation and hard-wiring share global optima and, allegedly, all critical points. That is a real advance over Nordenfors et al. (2024), which only compared stationary points inside the equivariant set. The global-optimum equivalence (Theorem 3.9) is believable; the reparameterization argument is sound and the proof is intricate but plausible.\n\nThe soft spots are real. Theorem 3.6, the claim that the regularization path is continuous, is false as stated. There is a concrete 3x3 counterexample where the singular values of Z_reg cross at the truncation boundary; the best rank-1 approximation jumps, so no continuous single-valued path exists. The proof invokes Dieci et al.'s SVD continuation lemma without checking the disjoint-singular-value-groups hypothesis. This is not a minor gap—it is a counterexample. The theorem could be repaired by adding a non-crossing assumption, and the convergence-to-hard-wired limit might still hold separately, but the abstract's continuity claim as written is wrong.\n\nProposition 3.10 also overclaims. It says the DA and constrained critical point sets are identical, but the proof only counts them—both have C(d,r) points. Counting does not establish set equality. One direction likely holds: DA critical points are invariant (the paper shows this), so they lie in the constrained space. The reverse containment is not proved. As it stands, the identity claim is not established.\n\nWhat the paper does well: it is honest about scope (deep linear, MSE, finite cyclic unitary representations), the related work is well placed, and there is no self-citation circularity or hidden fitting. The experiments are illustrative rather than conclusive—no code shipped, but reproducible in principle.\n\nBottom line: this is a serious theoretical paper with a load-bearing flaw in one theorem and an incomplete proof in another. The core global-optimum result is likely salvageable. I would send it to peer review, with a clear request for major revision: fix Theorem 3.6 (add the missing assumption or restate as a set-valued continuity claim) and fill the gap in Proposition 3.10. Without those, the advertised results do not stand.","headline":"Genuine extension of prior work on invariance in deep linear networks, but the regularization-path continuity theorem is false as stated and the critical-point identity proof is incomplete.","tokens_in":34221,"tokens_out":8444,"would_cite":false,"duration_ms":79614,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","15A18"],"pacs":[],"model":"deepseek-v4-flash","headline":"For deep linear networks with MSE, hard-wiring, data augmentation, and regularization share the same attainable solution.","keywords":["invariance","equivariance","data augmentation","regularization","deep linear networks","loss landscape","critical points","low-rank approximation"],"falsifier":"Construct a target matrix $Z$ and an invariance projector so that the $r$-th and $(r+1)$-th singular values of $Z_{\\mathrm{reg}}(\\lambda)$ cross at some $\\lambda^*$, then evaluate the rank-$r$ truncation on both sides of $\\lambda^*$; a discontinuous jump in $\\widehat W_{\\mathrm{reg}}(\\lambda)$ would refute Theorem 3.6 as stated, while a continuous limit would support it.","tokens_in":33030,"feed_emoji":"🎯","tokens_out":5198,"duration_ms":47346,"temperature":0.7,"pith_summary":"The paper tries to establish that, for mean-squared-error regression with deep linear networks, three standard ways of imposing a symmetry on the learned model — hard-wiring the invariant architecture, training on a data-augmented orbit, and adding a penalty that pushes toward invariance — lead to the same achievable solution. It characterizes the global optimum of each problem and proves that the set of critical points for hard-wiring and data augmentation is identical: every non-global critical point is a saddle, and the global optimum is the unique local minimum. Regularization admits a larger critical set, but the extra points are saddles too, and the regularized solution varies continuously and converges to the hard-wired optimum as the penalty grows. If the results hold, engineers can choose among these methods based on compute and data cost rather than on what the model can ultimately learn.","feed_headline":"Hard-wiring and data augmentation converge to the same model","feed_subtitle":"In MSE deep linear networks, all extra critical points are saddles, so the real choice is data and compute cost.","key_machinery":"The function space is the determinantal variety of rank-at-most-$r$ linear maps, which is non-convex when $r<\\min\\{d_0,d_L\\}$. Each method produces a target matrix to be approximated by a rank-$r$ truncation: the orthogonal projection $Z_{\\mathrm{inv}}=Z(I-\\tilde G\\tilde G^+)$ for hard-wiring, the orbit-averaged $Z_{\\mathrm{da}}$ for augmentation, and the rescaled $Z_{\\mathrm{reg}}(\\lambda)=ZB(\\lambda)^{-1}$ for regularization, with $B(\\lambda)^2=I+n\\lambda\\tilde G\\tilde G^T$. The Eckart–Young truncation of these targets, followed by a change of variables back to the original coordinates, yields the optima; the critical-point classification follows from the normal-space description of the determinantal variety.","core_discovery":"Under MSE loss and a unitary representation of a finite cyclic group, the paper proves that data augmentation and hard-wiring have the same global optima and the same critical points in function space (Theorems 3.6, 3.9, Proposition 3.10). In all three formulations every non-global critical point is a saddle; the only local minimum is the global one. The regularized problem has more critical points — $\\binom{m}{r}$ instead of $\\binom{d}{r}$ — but they remain saddles, and the regularization path continuously connects the unregularized optimum to the constrained invariant optimum.","pith_inferences":["The paper does not claim it, but its rank-truncation argument suggests a practical priority: if the symmetry group is known and a hard-wired architecture is feasible, augmentation adds data and parameter cost without improving the optimum.","A testable nonlinear analogue follows: train two-layer networks under MSE with a bottleneck and compare late-training losses of augmented versus hard-wired models; the paper's appendix observations suggest the gap widens as capacity grows.","The continuity theorem is only proven under a full-rank assumption, so a synthetic check where $\\sigma_r(\\lambda)$ and $\\sigma_{r+1}(\\lambda)$ cross would reveal whether the regularization path can jump; the paper leaves this case open."],"forward_implications":["For any unitary representation, a model trained on all group-orbit copies of the data converges to a map that is exactly invariant and identical to the hard-wired optimum.","Because all non-global critical points are saddles and the global optimum is the unique local minimum, gradient-based training on any of the three objectives is not trapped by spurious local minima.","Increasing the regularization strength $\\lambda$ moves the solution along a continuous path that ends at the hard-wired invariant solution.","The extra critical points introduced by regularization are harmless in the sense that none of them is a local minimum.","In this linear setting, the choice among hard-wiring, augmentation, and regularization changes computational cost, not the attainable function."],"supporting_citations":[{"why":"Supplies the classical two-layer linear network loss-landscape result that this work extends to invariant networks.","marker":"Baldi & Hornik (1989)"},{"why":"Gives the rank-r truncation formula used to write all global optima in closed form.","marker":"Eckart & Young (1936b)"},{"why":"Provides the normal-space characterization of critical points on determinantal varieties that Proposition 3.10 adapts.","marker":"Trager et al. (2020)"},{"why":"Supplies the smooth continuation lemma for singular value decompositions used in the proof of Theorem 3.6.","marker":"Dieci et al. (2005)"},{"why":"Gives the manifold-regularization solution that Proposition 3.5 builds on for the regularized objective.","marker":"Zhang & Zhao (2013)"},{"why":"Empirically motivates the comparison of data-driven invariance methods with hard-wired invariances.","marker":"Moskalev et al. (2023)"}],"fun_headline_variants":["Augmentation and hard-wiring give identical critical points","Deep linear nets: augmentation = hard-wiring at every critical point","All non-global critical points are saddles in invariant deep linear nets","Regularization path converges to hard-wired solution via saddles","Data augmentation matches hard-wiring in MSE deep linear nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For the regularization-path continuity theorem, the paper assumes $Z_{\\mathrm{reg}}(\\lambda)=ZB(\\lambda)^{-1}$ has full rank for every $\\lambda\\ge 0$, but the proof additionally requires its singular values to fall into separated groups that never merge; full rank alone does not prevent $\\sigma_r(\\lambda)$ and $\\sigma_{r+1}(\\lambda)$ from crossing, where the best rank-$r$ approximation could jump.","fun_headline_variants_meta":{"raw":{"variants":["Augmentation and hard-wiring give identical critical points","Deep linear nets: augmentation = hard-wiring at every critical point","All non-global critical points are saddles in invariant deep linear nets","Regularization path converges to hard-wired solution via saddles","Data augmentation matches hard-wiring in MSE deep linear nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3114,"prompt_tokens":813,"completion_tokens":2301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":2215}},"tokens_in":429,"tokens_out":2301,"duration_ms":13914,"temperature":1.0,"reasoning_tokens":2215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:00:04.536009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a target matrix $Z$ and an invariance projector so that the $r$-th and $(r+1)$-th singular values of $Z_{\\mathrm{reg}}(\\lambda)$ cross at some $\\lambda^*$, then evaluate the rank-$r$ truncation on both sides of $\\lambda^*$; a discontinuous jump in $\\widehat W_{\\mathrm{reg}}(\\lambda)$ would refute Theorem 3.6 as stated, while a continuous limit would support it.","supporting_citations":[],"review_version":2}