{"id":"18f1a890-2860-4de2-a420-b5f26ebb5ca8","arxiv_id":"2502.01347","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"For high-dimensional ridge regression, spurious correlations concentrate on a deterministic function of the data covariance and ridge penalty, creating a provable trade-off with in-distribution test loss.","lead":"This paper proves that in high-dimensional ridge regression, the amount of spurious correlation a model learns is governed by the data covariance and the regularization strength, not by the optimizer's bias. It also shows that the regularization level that minimizes in-distribution loss sits exactly in a range where spurious correlations grow, and that over-parameterized random features behave like regularized linear regression.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The trade-off theorem (Prop. 5.3) is proven only for Σxx=I and the small-ratio condition (5.8); the paper's own synthetic experiments violate (5.8), so the headline claim that ridge-optimal ERM necessarily learns spurious correlations is not established in the generic n=Θ(d) regime.","rationale":"The paper does substantial rigorous work: Theorem 1 is a genuine non-asymptotic concentration result with an explicit rate, Proposition 5.2 is a careful adaptation of [19], and Theorem 2 is nontrivial as a pointwise predictor-equivalence statement. The experiments are reproducible and support the qualitative story. My concern is not with these results but with the bridge from them to the headline claim. The abstract presents the trade-off as general ('the value of λ that minimizes L lies in an interval where C is increasing'), but Proposition 5.3 establishes it only for Σxx=I and under the small-ratio sufficient condition (5.8). The paper's own synthetic and Color-MNIST experiments use 2d/n values that violate (5.8), so they do not actually test the theorem's regime. The random-features section also asserts that Theorem 2 allows one to 'quantify spurious correlations' in RF models, yet Theorem 2 is a pointwise bound on a single test point and no separate theorem converts it into convergence of C(θRF). This second gap is real but secondary to the trade-off issue. None of this invalidates the main concentration results; it means the central claim is established only conditionally, which matches the reader's CONDITIONAL verdict. I therefore recommend no change.","tokens_in":41003,"tokens_out":12084,"duration_ms":113457,"concrete_test":"Use the released code (or a direct evaluation of (4.8)-(4.9) and (5.7)) to compute CΣ(λ) and LΣ(λ) for the synthetic model of Appendix F with the core covariance changed from Σxx=I to Σxx=diag(2,1,...,1), keeping all other parameters (n=2000, d=400, θ*=e1) fixed, and also for n/d=50. Locate λ* = argmin LΣ(λ) and the largest λC such that dCΣ/dλ ≥ 0 on (0,λC]. If in any run λ* > λC or CΣ(λ*) ≤ 0, the trade-off theorem's restriction to Σxx=I is essential and the headline claim is unsupported for generic core covariance; if λ* ≤ λC and CΣ is positive and increasing up to λ*, the restriction appears to be a proof artifact rather than a substantive limitation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion that the loss-optimal ridge penalty lies in the interval where CΣ is increasing rests entirely on Proposition 5.3, whose final statement (λC ≥ λL) is proved only under two assumptions: Σxx = I and the sample-ratio bound (5.8). For Σxx ≠ I, the derivative computation (B.35) does not apply; no argument is given that CΣ is non-negative or monotone on [0,λC], and the formula (B.15) can change sign depending on the alignment of θ* with Σxy. The condition (5.8) is also not generic in the n=Θ(d) regime: in the paper's own synthetic setup (d=400, n=2000, Σxx=I, β=0.5, σ²=0.25), λmin(Σ)=0.5 and λmax(Σ)=2.5, so RHS(5.8) ≈ 0.5/4 · min(1, 2·2.5/0.25/6²) ≈ 0.069, while 2d/n=0.4. Thus the main numerical demonstration lies outside the range in which Prop. 5.3 is proven. The concentration theorems (Thm 1, Prop 5.2) are not in question; the gap is that the monotonicity/trade-off step, which converts them into the headline claim, is conditional on non-generic sufficient conditions. The random-features section presents a related but secondary gap: Theorem 2 is a pointwise predictor equivalence, and the paper does not state a theorem converting it to convergence of C(θRF), despite asserting such a transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies spurious correlations in high-dimensional ridge regression and in a random-features model. It defines a metric C(θ) as the covariance between the model output on a fresh independent core feature together with the spurious feature and the label, and gives a non-asymptotic characterization of C for ridge-regularized linear regression: Theorem 1 shows that C(θLR(λ)) concentrates around a deterministic quantity CΣ(λ) defined through the data covariance Σ and an implicit regularization parameter τ(λ). Proposition 5.1 interprets CΣ(λ) in terms of the Schur complement of Σ and the spectrum of the spurious-feature covariance. Proposition 5.3 states a trade-off between CΣ(λ) and the in-distribution test loss LΣ(λ), claiming that the loss-minimizing λ lies in a region where CΣ(λ) is increasing. Theorem 2 gives a pointwise equivalence between a random-features predictor and a linear ridge predictor with an effective regularization. The theoretical results are complemented by experiments on synthetic Gaussian data, Color-MNIST, and CIFAR-10.","tokens_in":41391,"tokens_out":12231,"duration_ms":109810,"significance":"If fully established, the paper would provide a valuable quantitative handle on spurious correlations and simplicity bias in high-dimensional regression, connecting these phenomena to the spectrum of the data covariance and to ridge regularization. The concentration result Theorem 1 is a clean application of the Han–Xu machinery and appears correct; Proposition 5.2 is likewise a sound adaptation. The deterministic expression CΣ(λ) is elegant and falsifiable, and the experiments on multiple datasets support the qualitative trends. The paper also makes a useful contribution by proving a pointwise predictor-level equivalence for random features rather than only an equivalence of training/test losses. However, the headline trade-off claim is proved only under restrictive auxiliary assumptions, and the transfer from the random-features predictor equivalence to convergence of the C metric is not proved. The core concentration results are solid, but the advertised conclusions go beyond what is established.","major_comments":[{"comment":"The central trade-off claim—that the regularization minimizing the in-distribution loss lies in an interval where CΣ(λ) is increasing—is conditional on two restrictive hypotheses that are not reflected in the abstract or in the discussion following Proposition 5.3. Monotonicity and non-negativity of CΣ are proved only for Σxx = I (the derivative computation (B.35) is specific to that choice), and the inequality λC ≥ λL is proved only under the sample-ratio bound (5.8). For Σxx ≠ I, the representation (B.15) shows that the sign of CΣ(λ) depends on the alignment of θx* with Σxy, and no monotonicity argument is supplied. Moreover, (5.8) is not a generic n = Θ(d) condition: it forces 2d/n to be small relative to the spectral condition number of Σ, and in the paper's own synthetic example of Figure 2 (d = 400, n = 2000, λmax(Σyy) = 2, β = 0.5, σ² = 0.25) one has 2d/n = 0.4 while the right-hand side of (5.8) evaluates to roughly 3.5×10^{-3}. Thus the numerical demonstration of the trade-off lies outside the regime in which Proposition 5.3 is proved. The result may be true, but the manuscript as written does not establish it at the claimed level of generality; either the proof must be extended to general Σxx and to the full proportional regime, or the statements must be explicitly restricted.","section":"§5, Proposition 5.3 and abstract/introduction"},{"comment":"Theorem 2 is a pointwise equivalence between the random-features predictor and the linear predictor on a single fresh input z. It does not imply convergence of the spurious-correlation metric C(θRF(λ)), because C is a covariance over the joint distribution of the core, spurious, and independent core features, and pointwise closeness at one draw does not control this second-order expectation. The sentences after Theorem 2—'Thus, we expect the presence of spurious correlations' and 'C(θRF(0)) approaches CΣ(λ~)'—are assertions rather than consequences of the stated theorem. To make the over-parameterization claim rigorous, the paper needs either a uniform version of the equivalence over the relevant population (for example, control of E_z[|fRF(z) - fLR(z)|²], or a direct bound on the covariance functional) or an explicit statement that the transfer to C is conjectural.","section":"§6, Theorem 2"},{"comment":"The proof of Proposition 5.3 itself relies on a sequence of inequalities that establish monotonicity of LΣ(λ) only for sufficiently large τ, and the final statement λC ≥ λL is obtained by requiring both the monotonicity threshold and the small-ratio condition (5.8). The paper's remark that (5.8) is 'implied by taking n = ω(d)' is asymptotic and does not cover the n = Θ(d) regime in which Theorem 1 and the experiments operate; for any fixed ratio n/d = c, the condition (5.8) fails when c is below a constant threshold. The manuscript should either weaken the claims to the asymptotic regime where it is proved or provide a finite-ratio analysis.","section":"§5 and Appendix B, Proposition 5.3 proof"}],"minor_comments":[{"comment":"The notation EE[C(θLR(0))] for the expectation over the label noise is easy to confuse with the noise vector E used in the same proof; a clearer notation such as E_E or E_ϵ would improve readability.","section":"§4, proof of Proposition 4.1"},{"comment":"The sentence 'These trends in turn imply that the optimal value λ*_L that minimizes the test loss is s.t. λ*_L ∈ (0, λC]' is stated without repeating the hypotheses of Proposition 5.3; it should explicitly say 'under the assumptions of Proposition 5.3'.","section":"§5, discussion after Proposition 5.3"},{"comment":"The synthetic data construction sets Σxy = (Σyy - βI)^{1/2}, which gives a Schur complement βI; however, the resulting full matrix Σ has a smallest eigenvalue strictly smaller than β. This affects any numerical check of condition (5.8) and should be stated explicitly so that readers can verify the regime.","section":"Appendix F, synthetic Gaussian data"},{"comment":"The red-curve labels in the figure captions appear to omit the symbol C (for example, '(LR(λ)), n = 1000' should presumably read 'C(θLR(λ)), n = 1000'); the captions should be corrected for clarity.","section":"Figures 2 and 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's abstract and Section 5 present the trade-off between test loss and spurious correlations as unconditional, but the proof of Proposition 5.3 is limited to Σxx = I and to the small-ratio condition (5.8), and the paper's own synthetic experiments violate (5.8). The RF section likewise asserts a transfer to C that Theorem 2 does not prove. The concentration results Theorem 1 and Proposition 5.2 appear correct and are a solid contribution, so the appropriate outcome is a major revision rather than rejection: the authors should either extend the proofs to the claimed regimes or carefully restrict the claims. The manuscript is transparent about several technical restrictions in Appendix B.2, which is to its credit, but the main text does not consistently carry those caveats."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a solid core and an overstated headline. The concentration results for C and L in ridge regression look right; the trade-off theorem that converts them into the punchline is proved only under a narrow condition, and the paper's own experiments do not satisfy that condition.\n\nWhat is new: a quantitative target C_Sigma(lambda) for spurious correlations, a non-asymptotic concentration bound for it via Han-Xu's machinery, an interpretation of simplicity through the spectrum of the spurious covariance and correlation through the Schur complement, and a pointwise random-features-to-ridge equivalence with an explicit effective regularization. That last item is a real step beyond the existing error-only equivalence results. The proofs are careful, the Gaussian assumptions are stated, and the paper is honest about borrowing the main concentration hammer from Han-Xu [19]. No parameters are fitted to data; tau is determined by the covariance. As far as I can tell, the citation pattern is fair.\n\nThe soft spots are in the interpretive layer. Proposition 5.3, which yields the claim that the loss-optimal lambda sits in an interval where C is positive and increasing, requires Sigma_xx=I for non-negativity and monotonicity of C_Sigma, and condition (5.8) for lambda_C >= lambda_L. That condition is not generic in the n=Theta(d) regime: in the paper's own synthetic setup (d=400, n=2000, sigma^2=0.25), 2d/n=0.4 while RHS(5.8) is roughly 0.003. So the headline is a theorem about a restricted regime, not about the regime the experiments demonstrate. The abstract and conclusion do not flag this restriction. That is a real overclaim, and it is the main thing a referee should push on.\n\nThe RF section has a smaller gap of the same shape: Theorem 2 proves pointwise predictor equivalence, not convergence of C(theta_RF) itself. The paper says the spurious correlation is expected to follow, but no theorem transfers the metric. Fine as a conjecture, not as a proved claim.\n\nWho this is for: people working on spurious correlations, simplicity bias, or high-dimensional ridge asymptotics will get value from the C_Sigma formula and the RF equivalence. I would send it to a serious referee rather than desk-reject, but I would ask for a major revision that either extends Prop 5.3 to generic covariance or ratio, or explicitly reframes the contribution as a sufficient-condition analysis. As is, I would not cite the trade-off as a general fact.","headline":"Solid concentration results, but the headline trade-off is proved only under Sigma_xx=I and a sample-ratio condition that the paper's own experiments violate; the RF equivalence is pointwise, not metric-level.","tokens_in":41875,"tokens_out":3808,"would_cite":true,"duration_ms":36877,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62J07","62H20","60B20"],"pacs":[],"model":"deepseek-v4-flash","headline":"Ridge regularization—not interpolation—makes high-dimensional regression learn spurious correlations, and the loss-optimal penalty sits exactly in the growing-spurious-correlation range.","keywords":["spurious correlations","high-dimensional regression","ridge regularization","simplicity bias","random features","over-parameterization","test loss trade-off","Schur complement"],"falsifier":"Simulate the paper's Gaussian linear model with $n=\\Theta(d)$, a strongly non-isotropic core covariance $\\Sigma_{xx}$ with eigenvalues spread over two decades, and a sample ratio $2d/n$ violating the bound (5.8); find the ridge that minimizes the in-distribution test loss by cross-validation and estimate $C(\\hat{\\theta}_{\\mathrm{LR}}(\\lambda))$. If the minimizing $\\lambda$ falls outside the range where $C_\\Sigma(\\lambda)$ is increasing, or if $C$ is decreasing at the optimal $\\lambda$, the trade-off claim is false in that regime.","tokens_in":40796,"feed_emoji":"📊","tokens_out":9331,"duration_ms":79937,"temperature":0.7,"pith_summary":"This paper tries to establish a quantitative, first-principles account of when and why high-dimensional regression models learn spurious correlations. Its core claim is that for linear regression with Gaussian data and $n=\\Theta(d)$, the covariance-based measure $C$ of spurious correlation learned by the ridge-regularized empirical risk minimizer concentrates around a deterministic function $C_\\Sigma(\\lambda)$ of the data covariance and the regularization strength. From that function the paper proves a trade-off: the ridge value that minimizes the in-distribution test loss lies in an interval where $C$ is positive and increasing, so the best in-distribution model necessarily relies on spurious features. It also shows that an over-parameterized random-features model is asymptotically equivalent to ridge regression with a positive effective regularization, which explains why over-parameterized models keep spurious correlations even with no explicit penalty. A sympathetic reader should care because this turns a phenomenon usually discussed qualitatively—shortcut learning and simplicity bias—into a computable statement about spectra of covariances.","feed_headline":"Ridge regularization creates spurious correlations in regression","feed_subtitle":"Unregularized ERM learns none; the loss-optimal penalty sits exactly where spurious correlations grow.","key_machinery":"The load-bearing object is the deterministic functional $C_\\Sigma(\\lambda)$, the population analogue of the learned spurious correlation. It is a quadratic form in the projected covariance $P_y\\Sigma$ and the resolvent $(\\Sigma+\\tau(\\lambda)I)^{-1}$, with the auxiliary parameter $\\tau(\\lambda)$ fixed by the trace equation $1-\\lambda/\\tau=\\frac1n\\operatorname{tr}((\\Sigma+\\tau I)^{-1}\\Sigma)$. The Schur complement $S_x^\\Sigma$ enters as the conditional covariance of $y$ given $x$ and encodes how strongly the spurious feature is tied to the core feature, while $\\lambda_{\\max}(\\Sigma_{yy})$ encodes spectral simplicity of the spurious feature. In the random-features half, the mechanism is the predictor-level equivalence $f_{\\mathrm{RF}}(\\hat{\\theta}_{\\mathrm{RF}}(\\lambda),z)\\approx f_{\\mathrm{LR}}(\\hat{\\theta}_{\\mathrm{LR}}(\\tilde\\lambda),z)$ with the explicit effective regularization $\\tilde\\lambda$, derived by splitting the activation into its first Hermite component and a negligible nonlinear remainder.","core_discovery":"On the paper's own terms, the central discovery is that spurious correlations are created by regularization, not by interpolation. For unregularized linear regression with $n=\\omega(d)$, the expected amount of spurious correlation is zero and the random fluctuations vanish as $O(\\log d/\\sqrt{d})$. With ridge regularization, however, the learned spurious correlation $C(\\hat{\\theta}_{\\mathrm{LR}}(\\lambda))$ concentrates around $C_\\Sigma(\\lambda)=\\theta^{*\\top}\\Sigma(\\Sigma+\\tau(\\lambda)I)^{-1}P_y\\Sigma\\theta^*$, where $\\tau(\\lambda)$ is the unique positive solution of $1-\\lambda/\\tau=\\frac1n\\operatorname{tr}((\\Sigma+\\tau I)^{-1}\\Sigma)$. This deterministic proxy lets the paper read the data geometry directly: the cross-covariance $\\Sigma_{yx}$ controls the bare correlation between features, the Schur complement $S_x^\\Sigma=\\Sigma_{yy}-\\Sigma_{yx}\\Sigma_{xx}^{-1}\\Sigma_{xy}$ measures how much of $y$ is predictable from $x$, and the largest eigenvalue of $\\Sigma_{yy}$ measures how spectrally simple the spurious feature is. Under the conditions of Proposition 5.3, $C_\\Sigma(\\lambda)$ is non-negative and increasing up to $\\lambda_C$, while the in-distribution loss $L_\\Sigma(\\lambda)$ is minimized at some $\\lambda^*_L\\in(0,\\lambda_C]$, so the loss-optimal regularized model is provably in the regime of growing spurious correlation. The random-features result completes the picture: $f_{\\mathrm{RF}}(\\hat{\\theta}_{\\mathrm{RF}}(\\lambda),z)$ is asymptotically equal to $f_{\\mathrm{LR}}(\\hat{\\theta}_{\\mathrm{LR}}(\\tilde\\lambda),z)$ with $\\tilde\\lambda=\\frac{2\\tilde\\mu_2 d}{\\mu_1^2 n}+\\frac{2d}{\\mu_1^2 p}\\lambda$, so over-parameterization itself acts as ridge regularization.","pith_inferences":["Editorial inference: because $C_\\Sigma(\\lambda)$ is computable from the covariance $\\Sigma$, one could screen a dataset before training—estimate $\\Sigma_{yy}$ and the Schur complement, compute the predicted spurious-correlation profile, and decide whether ERM will need group annotations or decorrelation.","Editorial inference: early stopping and ridge regularization should follow the same $C$-versus-$\\lambda$ trade-off, so a network stopped early at the loss-optimal epoch should exhibit the same positive and increasing spurious correlation as the ridge-optimal linear predictor.","Editorial inference: the random-features equivalence suggests that the persistent spurious correlations seen in deep networks at small explicit $\\lambda$ are an effective-regularization effect rather than a feature-learning effect; measuring the empirical NTK or Hessian regularization would yield a testable effective $\\tilde\\lambda$.","Editorial inference: if the trade-off is as sharp as stated, methods that reduce $C$ by reweighting or invariant objectives will generically incur an in-distribution loss penalty unless they also change the covariance structure that couples $y$ to $x$."],"forward_implications":["Unregularized linear interpolation with $n=\\omega(d)$ does not learn spurious correlations: the expected $C$ is exactly zero and the magnitude is $O(\\log d/\\sqrt{d})$ with high probability.","Any ridge level that minimizes the in-distribution test loss lies in the interval $(0,\\lambda_C]$ where the deterministic spurious correlation $C_\\Sigma(\\lambda)$ is positive and increasing, so in-distribution optimality and spurious reliance cannot be separated in this model.","Spurious features that are spectrally simpler, meaning larger $\\lambda_{\\max}(\\Sigma_{yy})$, or more tightly correlated with the core feature, meaning smaller $\\lambda_{\\min}(S_x^\\Sigma)$, produce larger learned spurious correlations.","A random-features model with zero explicit ridge is equivalent to ridge regression with effective regularization $\\tilde\\lambda>0$, so over-parameterization alone generates spurious correlations; the effect is stronger for activations with larger $\\tilde\\mu_2/\\mu_1^2$.","Increasing $\\lambda$ beyond the intermediate range suppresses $C$ but only by shrinking the whole predictor, which raises the in-distribution test loss."],"supporting_citations":[{"why":"Supplies the non-asymptotic Gaussian comparison theorem that Theorem 1 and Proposition 5.2 use to concentrate $C$ and $L$ on their deterministic surrogates.","marker":"[19]"},{"why":"Provides the universality setting for random-features models that the paper extends from loss-level to predictor-level equivalence in Theorem 2.","marker":"[24]"},{"why":"Provides concentration lemmas for random matrices with sub-Gaussian and Lipschitz-concentrated data that the random-features proof reuses.","marker":"[8]"},{"why":"Supplies the matrix concentration inequalities used for the spectral estimates in the linear-regression and random-features proofs.","marker":"[58]"},{"why":"Supplies the sub-Gaussian and Lipschitz-concentration definitions and Bernstein-type inequalities behind Assumptions 1 and 4.","marker":"[59]"}],"fun_headline_variants":["Ridge regularization, not overfitting, causes spurious correlations","Spurious correlations arise from regularization, not interpolation","Over-parameterization acts like ridge, fueling spurious links","Loss-optimal ridge penalty lands in spurious-correlation zone"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The trade-off proof assumes the core-feature covariance is exactly the identity and that the sample-to-dimension ratio $2d/n$ is small enough to satisfy the bound in (5.8); outside that regime the paper does not prove that the loss-optimal ridge sits in the interval where spurious correlations are increasing.","fun_headline_variants_meta":{"raw":{"variants":["Ridge regularization, not overfitting, causes spurious correlations","Spurious correlations arise from regularization, not interpolation","Over-parameterization acts like ridge, fueling spurious links","Loss-optimal ridge penalty lands in spurious-correlation zone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000831,"raw_usage":{"total_tokens":3749,"prompt_tokens":1184,"completion_tokens":2565,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":800,"completion_tokens_details":{"reasoning_tokens":2497}},"tokens_in":800,"tokens_out":2565,"duration_ms":17208,"temperature":1.0,"reasoning_tokens":2497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:35:34.448562+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate the paper's Gaussian linear model with $n=\\Theta(d)$, a strongly non-isotropic core covariance $\\Sigma_{xx}$ with eigenvalues spread over two decades, and a sample ratio $2d/n$ violating the bound (5.8); find the ridge that minimizes the in-distribution test loss by cross-validation and estimate $C(\\hat{\\theta}_{\\mathrm{LR}}(\\lambda))$. If the minimizing $\\lambda$ falls outside the range where $C_\\Sigma(\\lambda)$ is increasing, or if $C$ is decreasing at the optimal $\\lambda$, the trade-off claim is false in that regime.","supporting_citations":[{"cited_title":"Introduction to the non-asymptotic analysis of random matrices, page 210–268","cited_arxiv_id":null,"evidence_quote":"Supplies the matrix concentration inequalities used for the spectral estimates in the linear-regression and random-features proofs."},{"cited_title":"High-dimensional probability: An introduction with applications in data science","cited_arxiv_id":null,"evidence_quote":"Supplies the sub-Gaussian and Lipschitz-concentration definitions and Bernstein-type inequalities behind Assumptions 1 and 4."}],"review_version":1}