{"id":"c5a90daa-79c5-40e7-84ae-f954b24512fa","arxiv_id":"2505.24668","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Two-layer linear denoising autoencoders show a bias-variance trade-off in bottleneck width, and skip connections reduce variance near the interpolation peak.","lead":"This paper derives exact formulas for the test error of two-layer linear denoising autoencoders, both with and without a low-dimensional bottleneck and a skip connection. The formulas show that bottleneck width sets a bias-variance trade-off, and that skip connections dampen the error peak near the overparameterization threshold.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.5's J_sc has a spurious 1/c: the proof of Lemma E.13 yields diag(((c-1)σ_i²+1)/(1+η_trn⁻²σ_i²)²), so Eq. (10) is not derived by the paper's own argument.","rationale":"The paper's central contribution is a pair of closed-form test-risk formulas (Theorems 3.2 and 3.5). The proof of Theorem 3.5 is internally inconsistent: Lemma E.13's statement, its proof's final algebraic expression, and the definition of J_sc in Theorem 3.5 give three different coefficients for the term involving Tr(LLᵀ). Carrying out the algebra from the intermediate means in Lemma E.13's proof shows the correct coefficient has no 1/c factor; hence Eq. (10) is not established. This is a concrete, checkable mathematical error rather than a matter of interpretation or a gap that might be filled by additional concentration arguments. The reader's weakest_assumption focused on the concentration-of-products step in Appendix E.1; that concern is legitimate but different. My finding is more decisive: even if the concentration step is accepted, Theorem 3.5's formula is false as stated. The qualitative conclusion — skip connections dampen the (c-1)⁻¹ variance peak by a factor 1/d — remains plausible after correcting J_sc, so the paper's high-level story is not destroyed. But because the exact risk formula is the paper's main deliverable, the manuscript should not be accepted in its present form; a corrected derivation and re-checked experiments are required.","tokens_in":38998,"tokens_out":24457,"duration_ms":253316,"concrete_test":"Analytic check: recompute the second term of Eq. (10) directly from the mean expressions in Lemma E.13's proof (first term η⁴|I_a|/n · (c-1)/c · (η²D⁻²+I)⁻²D⁻²; fourth term η⁴|I_a|/d · D⁻⁴(η²D⁻²+I)⁻²); substituting n=d/c yields (|I_a|/d)((c-1)D²+I)(1+η⁻²D²)⁻² with no denominator c. Numerical check: simulate the skip-connected linear DAE for r=k=1, σ=η=1, c=2, d large, N_tst large; estimate the coefficient of Tr(LLᵀ)/(d N_tst). If it is 0.5 rather than 0.375, the published J_sc is wrong.","verdict_should_be":"REJECT","load_bearing_attack":"In Theorem 3.5 (Eq. 10), the second term is (|I_a|/(d N_tst)) Tr(J_sc L Lᵀ) with J_sc_ii = [c+(c-1)σ_i²] / [c(1+η_trn⁻²σ_i²)²]. The proof's Lemma E.13 states E∥W_sc X_tst∥²_F = (|I_a|/d) Tr( ((c-1)D²+I_d)(I_r+η_trn⁻²D²)⁻² L Lᵀ ) + O(d⁻¹). Using instead the mean expressions inside the proof of Lemma E.13 — first term η⁴_trn (|I_a|/n) (c-1)/c (η²D⁻²+I)⁻²D⁻² and fourth term η⁴_trn (|I_a|/d) D⁻⁴(η²D⁻²+I)⁻² — and substituting n=d/c, the two terms sum to (|I_a|/d)(η²+D²)⁻²[(c-1)D²+I] times η⁴, which equals (|I_a|/d)(1+η⁻²D²)⁻²[(c-1)D²+I]. There is no 1/c factor. Thus the stated J_sc is not the coefficient following from the paper's own derivation; the correct diagonal is ((c-1)σ_i²+1)/(1+η_trn⁻²σ_i²)². Since Eq. (10) is one of the two central closed-form risk formulas claimed in the abstract, the theorem as stated is incorrect, even though the qualitative variance-damping conclusion likely survives after the factor is fixed.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies two-layer linear denoising autoencoders trained by gradient flow on the squared reconstruction loss with product regularization, in the overparameterized regime d ≥ n + r. For a low-rank Gaussian noise data model (Assumption 3.1), it derives closed-form expressions for all critical points (Theorem 2.1) and specializes them to DAEs with and without an identity skip connection (Corollaries 2.4 and 2.5). It then states asymptotic test-risk formulas (Theorems 3.2 and 3.5) and decomposes them into bias and variance terms, arguing that the bottleneck width k induces a bias-variance trade-off and that skip connections damp the variance peak near c = d/n ≈ 1. Section 4 uses random matrix theory on a rank-1 additive model to explain the variance suppression via weaker eigenvector alignment, and CIFAR-10 experiments support the qualitative curves.","tokens_in":39254,"tokens_out":15369,"duration_ms":158654,"significance":"The paper makes a useful step beyond single-layer analyses: it gives a complete critical-point characterization for two-layer linear DAEs with bottleneck and skip connection, identifies the minimum-norm global minimizer, and produces interpretable closed-form risk formulas. The bias-variance decomposition of Eq. (9) and the variance-damping comparison between Eq. (9) and Eq. (10) are conceptually valuable, and the numerical experiments appear to match the intended predictions. However, the main theorem for the skip-connection model, Theorem 3.5, contains an algebraic error: the stated J_sc is inconsistent with the paper's own Lemma E.13 and with the mean computations in its proof. Because Eq. (10) is advertised in the abstract as one of the two central risk formulas, the paper cannot be accepted without correcting this formula and re-checking the figures that plot it.","major_comments":[{"comment":"The stated coefficient J_sc does not follow from the paper's own derivation. Lemma E.13 states E∥W_sc X_tst∥²_F = |I_a|/d Tr(((c−1)D²+I_d)(I_r+η_{trn}^{-2}D²)^{-2}LL^T)+O(d^{-1}); substituting this into the decomposition in Eq. (31) yields the bias term |I_a|/(dN_tst) Tr(J_correct LL^T) with J_correct,ii = ((c−1)σ_i²+1)/(1+η_{trn}^{-2}σ_i²)^2. Theorem 3.5 instead has J_sc,ii = (c+(c−1)σ_i²)/(c(1+η_{trn}^{-2}σ_i²)^2), which differs by a factor 1/c on the σ_i² term. The proof of Lemma E.13 also ends with a third, different form ((c−1)D²+cI_d)(cI_r+cη^{-2}D²)^{-2}. The theorem as stated is therefore not established by the preceding argument; please correct Eq. (10), align the closing display of Lemma E.13 with its statement, and verify that the theoretical curves in Figure 3 use the corrected formula.","section":"Theorem 3.5, Eq. (10), and Lemma E.13"},{"comment":"The central derivations rely on replacing expectations of products of random matrix factors by products of expectations, justified by element-wise variance O(d^{-1}) for each factor. For products of four dependent factors such as D(P^T P)^{-1}D, K_1^{-1}, ZD^{-1}, and HH^T, the error involves joint fourth-order cumulants, not just pairwise variances. The manuscript cites [27] and [11] but does not state or prove a uniform lemma controlling these cross-terms. Since the closed-form risk formulas depend on this concentration step, please provide the missing concentration lemma or an explicit reference with the exact statement used.","section":"Appendix E.1, Eqs. (28)-(30) and Lemmas E.2-E.13"},{"comment":"The alignment-suppression theorem is proved for the additive model S=XX^T+AA^T, whereas the DAE actually corresponds to the information-plus-noise model (X+A)(X+A)^T. Remark 4.2 only matches first moments of the two models; it does not imply that eigenvector alignment properties transfer. The paper acknowledges the simplification, but as written the 'origin' of the variance discrepancy is a heuristic supported by a related model, not a theorem for the actual model. Please either prove the analogous statement for (X+A)(X+A)^T (or a precise asymptotic equivalence) or explicitly label this part as a heuristic.","section":"Section 4, Definition 4.1 and Theorem 4.3"}],"minor_comments":[{"comment":"Defining bias as the component that decreases with k and variance as the component that increases with k makes the bias-variance trade-off true by definition. The concrete decompositions in Eq. (28), Eq. (31), Lemmas E.3/E.4, and E.11/E.12 are the substantive statements; please present Definition E.14 as an interpretive summary rather than a formal definition.","section":"Appendix E.2, Definition E.14"},{"comment":"The case j=1 is dismissed 'for brevity' in a proof of a theorem about alignment; since the largest eigenvalue can dominate variance, this case should be either proved or explicitly excluded from the theorem statement.","section":"Remark F.4"},{"comment":"In Appendix E, D is used both as a d×d matrix and as an r×r block, and Lemma E.13 mixes I_d and I_r; please make the dimensions of all identity matrices and diagonal blocks explicit.","section":"Appendix E, notation"},{"comment":"The theorems state asymptotic o(1/d) and O(1/(dN_tst)) remainders while the text calls the setting 'non-asymptotic'; please clarify whether these are statements about sequences with d,n→∞ or finite-d bounds with explicit constants.","section":"Theorems 3.2 and 3.5"}],"recommendation":"major_revision","confidential_remarks":"The algebraic error in Theorem 3.5 is real and fixable; I would not reject the paper on this basis, but the theorem must be corrected and the numerical predictions re-verified. The concentration step is the main technical risk; the authors should be asked for a precise lemma. The paper's contribution is useful and within scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does two genuinely new things. It characterizes all critical points of two-layer linear denoising autoencoders with bottleneck and skip connections under product regularization, and it uses that characterization to derive asymptotic test-risk formulas. The critical point analysis (Theorem 2.1 and Corollaries 2.4–2.5) is solid and is a real advance over the one-layer treatments in [38,27]. The bias–variance reading of bottleneck width is a useful framing, and the observation that skip connections damp the variance near the interpolation peak is interesting.\n\nThe main problem is algebraic, not conceptual. In Theorem 3.5 the stated J_sc has a 1/c factor in the numerator: J_sc_ii = (c + (c-1)σ_i²)/(c(1+η_trn^{-2}σ_i²)²). But the paper's own Lemma E.13 and the mean expressions inside its proof give ((c-1)σ_i² + 1)/(1+η_trn^{-2}σ_i²)², with no 1/c. Equation (10) is one of the two central closed-form results, so as stated it is wrong. The qualitative variance-damping conclusion will very likely survive because the missing factor is O(1) for c away from 0, but the exact formula needs to be fixed. The proof of Lemma E.13 also has an internal inconsistency: its final display contains extra c's that match neither the lemma statement nor the intermediate terms.\n\nThe other caveats are softer. The concentration-of-products argument in Appendix E.1 replaces expectations of products with products of expectations; the claimed O(1/d) element-wise variances are asserted rather than fully established for all cross-terms, though this is common in this literature. Section 4's random matrix explanation is proved for the simplified additive model and for left singular vectors, not for the actual information-plus-noise model with right singular vectors; the causal claim in the main text overreaches. The numerical evidence is single-trial with no error bars, adequate for illustration only.\n\nWho will get value from this? Anyone working on benign overfitting in unsupervised settings or on theory of linear autoencoders. It deserves a serious referee. The right decision is major revision: correct J_sc and the proof of Lemma E.13, tighten or clearly state the concentration step, and soften the Section 4 conclusion. After that, it is a solid contribution. My own verdict: conditional, leaning positive once the algebra is fixed.","headline":"A real step forward for linear DAE theory, but the skip-connection risk formula in Theorem 3.5 has a spurious 1/c that must be fixed.","tokens_in":39883,"tokens_out":5971,"would_cite":false,"duration_ms":59654,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["60B20","62H25","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Bottleneck width creates a bias-variance trade-off in linear denoising autoencoders, and skip connections damp the overparameterization peak, according to closed-form test-risk formulas.","keywords":["denoising autoencoders","bottleneck layers","skip connections","generalization","double descent","bias-variance trade-off","linear neural networks","random matrix theory"],"falsifier":"Fix a deterministic rank-$r$ signal with well-conditioned spectrum, draw Gaussian training and test noise with variances $\\eta_{\\mathrm{trn}}^2/d$ and $\\eta_{\\mathrm{tst}}^2/d$, and train both models to their ridgeless global minimizers. Compare the measured test risk with Eqs. (9) and (10) as $d$ grows with $c$ fixed near $1$: the formulas are falsified if the rescaled variance term $d(c-1)R$ does not converge to $\\eta_{\\mathrm{tst}}^2 \\sum_{j\\in[k]} \\alpha_j^2/(1+\\alpha_j^2)$, or if the skip-connected variance is not smaller by an extra factor of $1/d$.","tokens_in":38698,"feed_emoji":"📉","tokens_out":8606,"duration_ms":111628,"temperature":0.7,"pith_summary":"The paper derives closed-form test-risk formulas for two-layer linear denoising autoencoders with a bottleneck layer, both with and without a skip connection, in the overparameterized regime $d \\geq n+r$. It shows that the global minimizer is $W_* = P_{[k]}(X)(X+A)^\\dagger$ and that the bottleneck width $k$ acts as a complexity dial: increasing $k$ reduces bias but increases variance, creating a modern bias-variance trade-off. It further shows that a skip connection multiplies the dangerous $(c-1)^{-1}$ variance term by an extra $1/d$ factor, flattening the peak near $c=d/n \\approx 1$. A random-matrix analysis traces this flattening to weaker alignment between the noise singular vectors and the corrupted-data eigenvectors. A sympathetic reader would care because this gives the first analytic picture of how two architectural staples, bottlenecks and skip connections, shape generalization in an unsupervised setting.","feed_headline":"Skip connections dampen the overfitting peak in linear denoisers","feed_subtitle":"Closed-form test-risk formulas trace the peak near d=n to variance and show skip connections reduce it by a factor of 1/d.","key_machinery":"The load-bearing object is the closed-form characterization of all critical points of the product-regularized two-layer linear loss: every critical point has the form $\\hat W_c = U_{G,I}U_{G,I}^\\top YZ^\\top(\\tilde Z\\tilde Z^\\top)^{-1}$, and in the ridgeless limit $W_c = P_I(Y)Z^\\dagger$, with the global minimizer $W_* = P_{[k]}(X)(X+A)^\\dagger$. The risk proofs expand $(X+A)^\\dagger$ through a pseudo-inverse perturbation formula and then replace expectations of products of the random matrices $D(P^\\top P)^{-1}D$, $K_1^{-1}$, $ZD^{-1}$, and $HH^\\top$ by products of expectations, relying on element-wise variances of order $O(1/d)$. For the skip-connection comparison, the paper introduces the rank-$1$ additive model $S=XX^\\top+AA^\\top$ and proves the alignment ratio identity $E[\\langle u_i^A,u_j^S\\rangle^2 / \\langle u_1,u_j^S\\rangle^2] = \\Theta\\left(\\frac{1}{d(\\lambda_i^A-\\lambda_j^S)^2}\\right)$, which is what makes the noise-alignment contribution to the variance parametrically smaller.","core_discovery":"Under a deterministic low-rank signal model with Gaussian noise of variances $\\eta_{\\mathrm{trn}}^2/d$ and $\\eta_{\\mathrm{tst}}^2/d$, and with $d \\geq n+r$, the paper claims exact control of the test risk of every critical point of the two-layer linear DAE. For the global minimizer $W_* = P_{[k]}(X)(X+A)^\\dagger$ without a skip connection, the test risk is $R(W_*, X_{\\mathrm{tst}}) = \\frac{1}{N_{\\mathrm{tst}}}\\mathrm{Tr}(JLL^\\top) + \\frac{\\eta_{\\mathrm{tst}}^2 c}{d(c-1)}\\sum_{j=1}^k \\frac{\\alpha_j^2}{1+\\alpha_j^2} + o(1/d)$, where the diagonal entries of $J$ are $(1+\\alpha_i^2)^{-2}$ for the selected directions and $1$ otherwise. The first term falls as $k$ grows, while the second rises, giving the bottleneck width a genuine bias-variance trade-off. With a skip connection, whose training objective is equivalent to predicting the noise $-A$ rather than the clean signal, the paper derives the corresponding risk formula whose variance term is smaller by an extra factor of $1/d$, so the $(c-1)^{-1}$ peak is damped, most visibly in the mildly overparameterized regime. The mechanism is identified in a rank-$1$ additive model $S = XX^\\top + AA^\\top$: the alignment between singular vectors of the noise and of $S$ is $\\Theta(1/d)$ weaker than the alignment between singular vectors of the signal and of $S$, which suppresses the dominant small-eigenvalue contribution to the variance.","pith_inferences":["Editorial inference, beyond the paper: the same concentration-plus-pseudo-inverse pipeline should apply to other minimum-norm interpolating autoencoders, such as masked autoencoders where the mask plays the role of the additive noise; the paper does not discuss this extension.","Editorial inference, beyond the paper: the paper's own remark about a four-layer linear model with the skip connection between hidden layers suggests a direct test of whether the constant $\\eta_{\\mathrm{tst}}^2$ bias can be removed while preserving the variance damping, since the two-layer input-output skip forces the network to predict full-rank noise.","Editorial inference, beyond the paper: the theory predicts that the optimal bottleneck width for fixed $n$ should grow with $d$ in the mildly overparameterized regime; a grid search over $k$ and $d$ on low-rank data would provide a sharp quantitative check of the predicted U-shape.","Editorial inference, beyond the paper: the alignment mechanism likely transfers to nonlinear denoisers trained with denoising score matching, where the effective target is also noise; the paper's linear analysis gives a concrete baseline for how much variance reduction one can attribute to the skip connection alone."],"forward_implications":["For fixed input dimension $d$ and sample size $n$, the test risk as a function of bottleneck width $k$ is U-shaped: larger $k$ lowers bias but raises variance, and the optimal $k$ depends on the overparameterization ratio $c=d/n$.","Double descent in $d/n$ persists even when the bottleneck is small: the variance term diverges like $(c-1)^{-1}$ as $c$ approaches $1$, so the peak is a genuine feature of the architecture, not of unlimited rank.","Adding a skip connection changes the effective target from the clean signal to the noise, and the resulting variance term carries an extra $1/d$ factor; this dampens the peak near $c\\approx 1$ without eliminating it.","In the skip-connected model, different critical points have nearly the same test error as the global minimizer, whereas in the bottleneck-only model different critical points can differ substantially in both bias and variance.","The variance reduction is traced to singular-vector alignment: noise singular vectors align with corrupted-data eigenvectors a factor $\\Theta(1/d)$ more weakly than signal singular vectors do, and this suppresses the contribution of the smallest eigenvalues that dominate the variance."],"supporting_citations":[{"why":"Supplies the classical critical-point characterization for two-layer linear networks that the paper extends to the overparameterized, regularized DAE setting.","marker":"[8]"},{"why":"Provides the product-regularization formulation and linear-DAE learning dynamics used in the training objective.","marker":"[33]"},{"why":"Gives the initial rank-1 DAE generalization analysis and the bias-variance decomposition whose techniques are extended here.","marker":"[38]"},{"why":"Establishes the low-rank deterministic data model and the concentration argument for replacing expectations of products by products of expectations.","marker":"[27]"},{"why":"Supplies the pseudo-inverse expansion used to write $(X+A)^\\dagger$ in the tractable form needed for the risk computations.","marker":"[45]"},{"why":"Provides the covariance formula for products of several random variables that justifies the concentration step in the proofs of Theorems 3.2 and 3.5.","marker":"[11]"},{"why":"Supplies the Marchenko-Pastur law and resolvent tools used in Section 4 to locate eigenvalues and derive the alignment bound.","marker":"[6]"},{"why":"Gives the spiked-model eigenvalue convergence used for the largest noise eigenvalue in the alignment argument.","marker":"[7]"},{"why":"Provides the $\\Theta(d^{-2})$ scaling of the smallest singular value of a rectangular Gaussian matrix, which drives the variance contribution from small eigenvalues.","marker":"[36]"}],"fun_headline_variants":["Bottleneck width trades bias for variance in linear denoisers","Skip connections cut variance in linear denoising autoencoders","Bottleneck adds bias-variance trade-off to linear DAEs","Skip connections damp the peak in linear denoiser risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The risk formulas rest on the assumption that the random matrices appearing inside the trace concentrate quickly enough that the expectation of a product equals the product of expectations up to $o(1)$ noise; if that joint concentration fails, the closed-form curves could acquire extra terms.","fun_headline_variants_meta":{"raw":{"variants":["Bottleneck width trades bias for variance in linear denoisers","Skip connections cut variance in linear denoising autoencoders","Bottleneck adds bias-variance trade-off to linear DAEs","Skip connections damp the peak in linear denoiser risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000253,"raw_usage":{"total_tokens":1685,"prompt_tokens":1185,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":801,"completion_tokens_details":{"reasoning_tokens":428}},"tokens_in":801,"tokens_out":500,"duration_ms":5799,"temperature":1.0,"reasoning_tokens":428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:16:23.705081+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix a deterministic rank-$r$ signal with well-conditioned spectrum, draw Gaussian training and test noise with variances $\\eta_{\\mathrm{trn}}^2/d$ and $\\eta_{\\mathrm{tst}}^2/d$, and train both models to their ridgeless global minimizers. Compare the measured test risk with Eqs. (9) and (10) as $d$ grows with $c$ fixed near $1$: the formulas are falsified if the rescaled variance term $d(c-1)R$ does not converge to $\\eta_{\\mathrm{tst}}^2 \\sum_{j\\in[k]} \\alpha_j^2/(1+\\alpha_j^2)$, or if the skip-connected variance is not smaller by an extra factor of $1/d$.","supporting_citations":[{"cited_title":"Neural networks and principal component analysis: Learning from examples without local minima","cited_arxiv_id":null,"evidence_quote":"Supplies the classical critical-point characterization for two-layer linear networks that the paper extends to the overparameterized, regularized DAE setting."},{"cited_title":"Learning dynamics of linear denoising autoencoders","cited_arxiv_id":null,"evidence_quote":"Provides the product-regularization formulation and linear-DAE learning dynamics used in the training objective."},{"cited_title":"Training Data Size Induced Double Descent For Denoising Neural Networks and the Role of Training Noise Level","cited_arxiv_id":null,"evidence_quote":"Gives the initial rank-1 DAE generalization analysis and the bias-variance decomposition whose techniques are extended here."},{"cited_title":"On the Exact Covariance of Products of Random Variables","cited_arxiv_id":null,"evidence_quote":"Provides the covariance formula for products of several random variables that justifies the concentration step in the proofs of Theorems 3.2 and 3.5."},{"cited_title":"Spectral analysis of large dimensional random matrices","cited_arxiv_id":null,"evidence_quote":"Supplies the Marchenko-Pastur law and resolvent tools used in Section 4 to locate eigenvalues and derive the alignment bound."},{"cited_title":"Eigenvalues of Large Sample Covariance Matrices of Spiked Population Models","cited_arxiv_id":"math/0408165","evidence_quote":"Gives the spiked-model eigenvalue convergence used for the largest noise eigenvalue in the alignment argument."},{"cited_title":"Smallest singular value of a random rectangular matrix","cited_arxiv_id":null,"evidence_quote":"Provides the $\\Theta(d^{-2})$ scaling of the smallest singular value of a rectangular Gaussian matrix, which drives the variance contribution from small eigenvalues."}],"review_version":1}