REVIEW 3 major objections 4 minor 67 references
Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization
T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Ridge regularization—not interpolation—makes high-dimensional regression learn spurious correlations, and the loss-optimal penalty sits exactly in the growing-spurious-correlation range.
desk verdict Solid concentration results, but the headline trade-off is proved only under Sigma_xx=I and a sample-ratio condition that the paper's own experiments violate; the RF equivalence is pointwise, not metric-level. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the deterministic functional $C_\Sigma(\lambda)$, the population analogue of the learned spurious correlation. It is a quadratic form in the projected covariance $P_y\Sigma$ and the resolvent $(\Sigma+\tau(\lambda)I)^{-1}$, with the auxiliary parameter $\tau(\lambda)$ fixed by the trace equation $1-\lambda/\tau=\frac1n\operatorname{tr}((\Sigma+\tau I)^{-1}\Sigma)$. The Schur complement $S_x^\Sigma$ enters as the conditional covariance of $y$ given $x$ and encodes how strongly the spurious feature is tied to the core feature, while $\lambda_{\max}(\Sigma_{yy})$ encodes spectral simplicity of the spurious feature. In the random-features half, the mechanism is the predictor-level equivalence $f_{\mathrm{RF}}(\hat{\theta}_{\mathrm{RF}}(\lambda),z)\approx f_{\mathrm{LR}}(\hat{\theta}_{\mathrm{LR}}(\tilde\lambda),z)$ with the explicit effective regularization $\tilde\lambda$, derived by splitting the activation into its first Hermite component and a negligible nonlinear remainder.
What would settle it
Simulate the paper's Gaussian linear model with $n=\Theta(d)$, a strongly non-isotropic core covariance $\Sigma_{xx}$ with eigenvalues spread over two decades, and a sample ratio $2d/n$ violating the bound (5.8); find the ridge that minimizes the in-distribution test loss by cross-validation and estimate $C(\hat{\theta}_{\mathrm{LR}}(\lambda))$. If the minimizing $\lambda$ falls outside the range where $C_\Sigma(\lambda)$ is increasing, or if $C$ is decreasing at the optimal $\lambda$, the trade-off claim is false in that regime.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that spurious correlations are created by regularization, not by interpolation. For unregularized linear regression with $n=\omega(d)$, the expected amount of spurious correlation is zero and the random fluctuations vanish as $O(\log d/\sqrt{d})$. With ridge regularization, however, the learned spurious correlation $C(\hat{\theta}_{\mathrm{LR}}(\lambda))$ concentrates around $C_\Sigma(\lambda)=\theta^{*\top}\Sigma(\Sigma+\tau(\lambda)I)^{-1}P_y\Sigma\theta^*$, where $\tau(\lambda)$ is the unique positive solution of $1-\lambda/\tau=\frac1n\operatorname{tr}((\Sigma+\tau I)^{-1}\Sigma)$. This deterministic proxy lets the paper read the data geometry directly: the cross-covariance $\Sigma_{yx}$ controls the bare correlation between features, the Schur complement $S_x^\Sigma=\Sigma_{yy}-\Sigma_{yx}\Sigma_{xx}^{-1}\Sigma_{xy}$ measures how much of $y$ is predictable from $x$, and the largest eigenvalue of $\Sigma_{yy}$ measures how spectrally simple the spurious feature is. Under the conditions of Proposition 5.3, $C_\Sigma(\lambda)$ is non-negative and increasing up to $\lambda_C$, while the in-distribution loss $L_\Sigma(\lambda)$ is minimized at some $\lambda^*_L\in(0,\lambda_C]$, so the loss-optimal regularized model is provably in the regime of growing spurious correlation. The random-features result completes the picture: $f_{\mathrm{RF}}(\hat{\theta}_{\mathrm{RF}}(\lambda),z)$ is asymptotically equal to $f_{\mathrm{LR}}(\hat{\theta}_{\mathrm{LR}}(\tilde\lambda),z)$ with $\tilde\lambda=\frac{2\tilde\mu_2 d}{\mu_1^2 n}+\frac{2d}{\mu_1^2 p}\lambda$, so over-parameterization itself acts as ridge regularization.
Load-bearing premise
The trade-off proof assumes the core-feature covariance is exactly the identity and that the sample-to-dimension ratio $2d/n$ is small enough to satisfy the bound in (5.8); outside that regime the paper does not prove that the loss-optimal ridge sits in the interval where spurious correlations are increasing.
Editorial extensions
If this is right
- Unregularized linear interpolation with $n=\omega(d)$ does not learn spurious correlations: the expected $C$ is exactly zero and the magnitude is $O(\log d/\sqrt{d})$ with high probability.
- Any ridge level that minimizes the in-distribution test loss lies in the interval $(0,\lambda_C]$ where the deterministic spurious correlation $C_\Sigma(\lambda)$ is positive and increasing, so in-distribution optimality and spurious reliance cannot be separated in this model.
- Spurious features that are spectrally simpler, meaning larger $\lambda_{\max}(\Sigma_{yy})$, or more tightly correlated with the core feature, meaning smaller $\lambda_{\min}(S_x^\Sigma)$, produce larger learned spurious correlations.
- A random-features model with zero explicit ridge is equivalent to ridge regression with effective regularization $\tilde\lambda>0$, so over-parameterization alone generates spurious correlations; the effect is stronger for activations with larger $\tilde\mu_2/\mu_1^2$.
- Increasing $\lambda$ beyond the intermediate range suppresses $C$ but only by shrinking the whole predictor, which raises the in-distribution test loss.
Reading between the lines
- Editorial inference: because $C_\Sigma(\lambda)$ is computable from the covariance $\Sigma$, one could screen a dataset before training—estimate $\Sigma_{yy}$ and the Schur complement, compute the predicted spurious-correlation profile, and decide whether ERM will need group annotations or decorrelation.
- Editorial inference: early stopping and ridge regularization should follow the same $C$-versus-$\lambda$ trade-off, so a network stopped early at the loss-optimal epoch should exhibit the same positive and increasing spurious correlation as the ridge-optimal linear predictor.
- Editorial inference: the random-features equivalence suggests that the persistent spurious correlations seen in deep networks at small explicit $\lambda$ are an effective-regularization effect rather than a feature-learning effect; measuring the empirical NTK or Hessian regularization would yield a testable effective $\tilde\lambda$.
- Editorial inference: if the trade-off is as sharp as stated, methods that reduce $C$ by reweighting or invariant objectives will generically incur an in-distribution loss penalty unless they also change the covariance structure that couples $y$ to $x$.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies spurious correlations in high-dimensional ridge regression and in a random-features model. It defines a metric C(θ) as the covariance between the model output on a fresh independent core feature together with the spurious feature and the label, and gives a non-asymptotic characterization of C for ridge-regularized linear regression: Theorem 1 shows that C(θLR(λ)) concentrates around a deterministic quantity CΣ(λ) defined through the data covariance Σ and an implicit regularization parameter τ(λ). Proposition 5.1 interprets CΣ(λ) in terms of the Schur complement of Σ and the spectrum of the spurious-feature covariance. Proposition 5.3 states a trade-off between CΣ(λ) and the in-distribution test loss LΣ(λ), claiming that the loss-minimizing λ lies in a region where CΣ(λ) is increasing. Theorem 2 gives a pointwise equivalence between a random-features predictor and a linear ridge predictor with an effective regularization. The theoretical results are complemented by experiments on synthetic Gaussian data, Color-MNIST, and CIFAR-10.
Significance. If fully established, the paper would provide a valuable quantitative handle on spurious correlations and simplicity bias in high-dimensional regression, connecting these phenomena to the spectrum of the data covariance and to ridge regularization. The concentration result Theorem 1 is a clean application of the Han–Xu machinery and appears correct; Proposition 5.2 is likewise a sound adaptation. The deterministic expression CΣ(λ) is elegant and falsifiable, and the experiments on multiple datasets support the qualitative trends. The paper also makes a useful contribution by proving a pointwise predictor-level equivalence for random features rather than only an equivalence of training/test losses. However, the headline trade-off claim is proved only under restrictive auxiliary assumptions, and the transfer from the random-features predictor equivalence to convergence of the C metric is not proved. The core concentration results are solid, but the advertised conclusions go beyond what is established.
major comments (3)
- [§5, Proposition 5.3 and abstract/introduction] The central trade-off claim—that the regularization minimizing the in-distribution loss lies in an interval where CΣ(λ) is increasing—is conditional on two restrictive hypotheses that are not reflected in the abstract or in the discussion following Proposition 5.3. Monotonicity and non-negativity of CΣ are proved only for Σxx = I (the derivative computation (B.35) is specific to that choice), and the inequality λC ≥ λL is proved only under the sample-ratio bound (5.8). For Σxx ≠ I, the representation (B.15) shows that the sign of CΣ(λ) depends on the alignment of θx* with Σxy, and no monotonicity argument is supplied. Moreover, (5.8) is not a generic n = Θ(d) condition: it forces 2d/n to be small relative to the spectral condition number of Σ, and in the paper's own synthetic example of Figure 2 (d = 400, n = 2000, λmax(Σyy) = 2, β = 0.5, σ² = 0.25) one has 2d/n = 0.4 while the right-hand side of (5.8) evaluates to roughly 3.5×10^{-3}. Thus the numerical demonstration of the trade-off lies outside the regime in which Proposition 5.3 is proved. The result may be true, but the manuscript as written does not establish it at the claimed level of generality; either the proof must be extended to general Σxx and to the full proportional regime, or the statements must be explicitly restricted.
- [§6, Theorem 2] Theorem 2 is a pointwise equivalence between the random-features predictor and the linear predictor on a single fresh input z. It does not imply convergence of the spurious-correlation metric C(θRF(λ)), because C is a covariance over the joint distribution of the core, spurious, and independent core features, and pointwise closeness at one draw does not control this second-order expectation. The sentences after Theorem 2—'Thus, we expect the presence of spurious correlations' and 'C(θRF(0)) approaches CΣ(λ~)'—are assertions rather than consequences of the stated theorem. To make the over-parameterization claim rigorous, the paper needs either a uniform version of the equivalence over the relevant population (for example, control of E_z[|fRF(z) - fLR(z)|²], or a direct bound on the covariance functional) or an explicit statement that the transfer to C is conjectural.
- [§5 and Appendix B, Proposition 5.3 proof] The proof of Proposition 5.3 itself relies on a sequence of inequalities that establish monotonicity of LΣ(λ) only for sufficiently large τ, and the final statement λC ≥ λL is obtained by requiring both the monotonicity threshold and the small-ratio condition (5.8). The paper's remark that (5.8) is 'implied by taking n = ω(d)' is asymptotic and does not cover the n = Θ(d) regime in which Theorem 1 and the experiments operate; for any fixed ratio n/d = c, the condition (5.8) fails when c is below a constant threshold. The manuscript should either weaken the claims to the asymptotic regime where it is proved or provide a finite-ratio analysis.
minor comments (4)
- [§4, proof of Proposition 4.1] The notation EE[C(θLR(0))] for the expectation over the label noise is easy to confuse with the noise vector E used in the same proof; a clearer notation such as E_E or E_ϵ would improve readability.
- [§5, discussion after Proposition 5.3] The sentence 'These trends in turn imply that the optimal value λ*_L that minimizes the test loss is s.t. λ*_L ∈ (0, λC]' is stated without repeating the hypotheses of Proposition 5.3; it should explicitly say 'under the assumptions of Proposition 5.3'.
- [Appendix F, synthetic Gaussian data] The synthetic data construction sets Σxy = (Σyy - βI)^{1/2}, which gives a Schur complement βI; however, the resulting full matrix Σ has a smallest eigenvalue strictly smaller than β. This affects any numerical check of condition (5.8) and should be stated explicitly so that readers can verify the regime.
- [Figures 2 and 5] The red-curve labels in the figure captions appear to omit the symbol C (for example, '(LR(λ)), n = 1000' should presumably read 'C(θLR(λ)), n = 1000'); the captions should be corrected for clarity.
Circularity Check
No significant circularity: the central characterization is a genuine concentration result built on an external benchmark, and the trade-off is a direct calculus argument; the few self-citations in the random-features appendix are auxiliary, not load-bearing.
full rationale
The derivation chain is self-contained relative to its announced inputs. Theorem 1 and Proposition 5.2 are concentration results obtained by applying Han and Xu [19], an external non-asymptotic characterization of ridge estimators; the target statistic C(theta) = theta^T P_y Sigma theta* enters only as the linear functional being concentrated, and C_Sigma(lambda) is its expectation under the Gaussian sequence model (Eqs. B.9-B.13), not a quantity fitted to the data or defined in terms of the claim. The trade-off Proposition 5.3 is a direct calculus argument: C_Sigma is rewritten via the Schur-complement identity (B.15), and under Sigma_xx = I the derivative (B.35) is manifestly nonnegative for tau <= sqrt(lambda_min(Sigma_x)), giving monotonicity on [0, lambda_C]; the final lambda_C >= lambda_L claim follows from the explicit bound (5.8) and Lemma B.3, an independent matrix inequality. No step in this chain assumes the conclusion it is meant to prove. The random-features section derives a pointwise predictor equivalence (Theorem 2) through concentration lemmas; although Lemmas C.4 and C.5 borrow technical arguments from the authors' earlier work [6,8], those are auxiliary operator-norm and Hermite-expansion bounds whose assumptions do not include the RF equivalence and which are not the result being proved, so the citation is real evidence rather than circular import. The main caveat is one of scope, not circularity: the monotonicity/trade-off conclusion is proven only for Sigma_xx = I and under the small-ratio condition (5.8), while the paper's own synthetic experiment uses 2d/n = 0.4, outside (5.8). That is a limitation of the theorem's applicability, not a reduction of the prediction to its inputs.
Assumptions & free parameters
assumptions (6)
- domain assumption Gaussian data with invertible well-conditioned covariance and trace 2d (Assumption 1)
- domain assumption Odd, Lipschitz activation with nonzero first Hermite coefficient (Assumption 2)
- domain assumption p=ω(n log^4 n) over-parameterization (Assumption 3)
- domain assumption Lipschitz-concentrated data distribution (Assumption 4)
- standard math Han-Xu [19] non-asymptotic distribution of ridge estimators (Theorems 2.3 and 3.1)
- ad hoc to paper Σxx=I for the monotonicity of CΣ(λ) (Prop 5.3)
Cite this review
Pith. "Pith review of Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization." pith.science (2026). https://pith.science/paper/OXKN63NX
@misc{pith2026250201347,
author = {Pith},
title = {Pith review of: Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization},
year = {2026},
howpublished = {\url{https://pith.science/paper/OXKN63NX}},
note = {Machine review of arXiv:2502.01347}
}
abstract
Learning models have been shown to rely on spurious correlations between non-predictive features and the associated labels in the training data, with negative implications on robustness, bias and fairness. In this work, we provide a statistical characterization of this phenomenon for high-dimensional regression, when the data contains a predictive core feature $x$ and a spurious feature $y$. Specifically, we quantify the amount of spurious correlations $C$ learned via linear regression, in terms of the data covariance and the strength $\lambda$ of the ridge regularization. As a consequence, we first capture the simplicity of $y$ through the spectrum of its covariance, and its correlation with $x$ through the Schur complement of the full data covariance. Next, we prove a trade-off between $C$ and the in-distribution test loss $L$, by showing that the value of $\lambda$ that minimizes $L$ lies in an interval where $C$ is increasing. Finally, we investigate the effects of over-parameterization via the random features model, by showing its equivalence to regularized linear regression. Our theoretical results are supported by numerical experiments on Gaussian, Color-MNIST, and CIFAR-10 datasets.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[19]
The distribution of ridgeless least squares interpolators.arXiv preprint arXiv:2307.02044, 2023
Qiyang Han and Xiaocong Xu. The distribution of ridgeless least squares interpolators.arXiv preprint arXiv:2307.02044, 2023
arXiv 2023
-
[1]
Systematic generalisation with group invariant predictions
Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. InInternational Conference on Learning Representations, 2021
work page 2021
-
[2]
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2020
arXiv 1907
-
[3]
High-dimensional asymptotics of feature learning: How one gradient step improves the representation
Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. InAdvances in Neural Information Processing Systems (NeurIPS), 2022
work page 2022
-
[4]
Deep learning: a statistical viewpoint
Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica, 30:87–201, 2021. 14
work page 2021
-
[5]
Reconciling modern machine-learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off.Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019
2019
-
[6]
Memorization and optimization in deep neural networks with minimum over-parameterization
Simone Bombari, Mohammad Hossein Amani, and Marco Mondelli. Memorization and optimization in deep neural networks with minimum over-parameterization. InAdvances in Neural Information Processing Systems (NeurIPS), 2022
work page 2022
-
[7]
Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels
Simone Bombari, Shayan Kiyani, and Marco Mondelli. Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels. InProceedings of the 40th International Conference on Machine Learning, 2023
work page 2023
Show all 67 references
-
[8]
Privacy for free in the over-parameterized regime.arXiv preprint arXiv:2410.14787, 2024
Simone Bombari and Marco Mondelli. Privacy for free in the over-parameterized regime.arXiv preprint arXiv:2410.14787, 2024
2024 arXiv
-
[9]
A universal law of robustness via isoperimetry
Sebastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. InAdvances in Neural Information Processing Systems (NeurIPS), 2021
2021
-
[10]
Chang, G
C. Chang, G. Adam, and A. Goldenberg. Towards robust classification model by counterfactual and invariant data generation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021
2021
-
[11]
Provable benefits of overparameterization in model compression: From double descent to pruning neural networks
Xiangyu Chang, Yingcong Li, Samet Oymak, and Christos Thrampoulidis. Provable benefits of overparameterization in model compression: From double descent to pruning neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6974–6983, 2021
2021
-
[12]
Dimension free ridge regression.The Annals of Statistics, 52(6):2879 – 2912, 2024
Chen Cheng and Andrea Montanari. Dimension free ridge regression.The Annals of Statistics, 52(6):2879 – 2912, 2024
2024
-
[13]
Neural networks can learn representations with gradient descent
Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent. InConference on Learning Theory (COLT), 2022
2022
-
[14]
On the (non-) robustness of two-layer neural networks in different learning regimes
Elvis Dohmatob and Alberto Bietti. On the (non-) robustness of two-layer neural networks in different learning regimes. arXiv preprint arXiv:2203.11864, 2022
2022 arXiv
-
[15]
Wichmann
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020
2020
-
[16]
Wichmann, and Wieland Brendel
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. InInternational Conference on Learning Representations (ICLR), 2019
2019
-
[17]
The gaussian equivalence of generative models for learning with shallow neural networks
Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. The gaussian equivalence of generative models for learning with shallow neural networks. InMathematical and Scientific Machine Learning, pages 426–471. PMLR, 2022
2022
-
[18]
Modeling the influence of data structure on learning in neural networks: The hidden manifold model.Physical Review X, 10(4):041044, 2020
Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure on learning in neural networks: The hidden manifold model.Physical Review X, 10(4):041044, 2020
2020
-
[20]
The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression.The Annals of Statistics, 52(2):441 – 465, 2024
Hamed Hassani and Adel Javanmard. The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression.The Annals of Statistics, 52(2):441 – 465, 2024
2024
-
[21]
Hastie, Andrea Montanari, Saharon Rosset, and Ryan J
Trevor J. Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation.Annals of statistics, 50 2:949–986, 2022. 15
2022
-
[22]
What shapes feature representations? Exploring datasets, architectures, and training
Katherine Hermann and Andrew Lampinen. What shapes feature representations? Exploring datasets, architectures, and training. InAdvances in Neural Information Processing Systems (NeurIPS), 2020
2020
-
[23]
On the foundations of shortcut learning
Katherine Hermann, Hossein Mobahi, Thomas FEL, and Michael Curtis Mozer. On the foundations of shortcut learning. InThe Twelfth International Conference on Learning Representations, 2024
2024
-
[24]
Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features.IEEE Transactions on Information Theory, 69(3):1932–1964, 2023
1932
-
[25]
On feature learning in the presence of spurious correlations
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson. On feature learning in the presence of spurious correlations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information Processing Systems, 2022
2022
-
[26]
Sgd on neural networks learns functions of increasing complexity
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. InAdvances in Neural Information Processing Systems, 2019
2019
-
[27]
Last layer re-training is sufficient for ro- bustness to spurious correlations
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for ro- bustness to spurious correlations. InThe Eleventh International Conference on Learning Representations, 2023
2023
-
[28]
Demystifying disagreement-on-the-line in high dimensions
Donghwan Lee, Behrad Moniri, Xinmeng Huang, Edgar Dobriban, and Hamed Hassani. Demystifying disagreement-on-the-line in high dimensions. InProceedings of the 40th International Conference on Machine Learning, 2023
2023
-
[29]
Just train twice: Improving group robustness without training group information
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Proceedings of the 38th International Conference on Machine Learning, 2021
2021
-
[30]
Avoiding spurious correlations via logit correction
Sheng Liu, Xu Zhang, Nitesh Sekhar, Yue Wu, Prateek Singhal, and Carlos Fernandez-Granda. Avoiding spurious correlations via logit correction. In The Eleventh International Conference on Learning Representations, 2023
2023
-
[31]
Learning curves of generic features maps for realistic datasets with a teacher-student model
Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. In Advances in Neural Information Processing Systems, 2021
2021
-
[32]
Minimum-norm interpolation under covariate shift
Neil Rohit Mallinar, Austin Zane, Spencer Frei, and Bin Yu. Minimum-norm interpolation under covariate shift. InForty-first International Conference on Machine Learning, 2024
2024
-
[33]
Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration.Applied and Computational Harmonic Analysis, 59:3–84, 2022
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration.Applied and Computational Harmonic Analysis, 59:3–84, 2022. Special Issue on Harmonic Analysis and Machine Learning
2022
-
[34]
The generalization error of random features regression: Precise asymptotics and the double descent curve.Communications on Pure and Applied Mathematics, 75(4):667– 766, 2022
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve.Communications on Pure and Applied Mathematics, 75(4):667– 766, 2022
2022
-
[35]
Hard imagenet: Segmentations for objects with strong spurious cues
Mazda Moayeri, Sahil Singla, and Soheil Feizi. Hard imagenet: Segmentations for objects with strong spurious cues. InAdvances in Neural Information Processing Systems, 2022
2022
-
[36]
A theory of non-linear feature learning with one gradient step in two-layer neural networks
Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. InInternational Conference on Machine Learning (ICML), 2024
2024
-
[37]
The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime. arXiv preprint arXiv:1911.01544, 2019. 16
1911 arXiv
-
[38]
Universality of empirical risk minimization
Andrea Montanari and Basil N Saeed. Universality of empirical risk minimization. InConference on Learning Theory, pages 4310–4312. PMLR, 2022
2022
-
[39]
Simplicity bias in 1-hidden layer neural networks
Depen Morwani, jatin batra, Prateek Jain, and Praneeth Netrapalli. Simplicity bias in 1-hidden layer neural networks. InThirty-seventh Conference on Neural Information Processing Systems, 2023
2023
-
[40]
Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep ReLU networks
Quynh Nguyen, Marco Mondelli, and Guido Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep ReLU networks. InInternational Conference on Machine Learning (ICML), 2021
2021
-
[41]
Analysis of Boolean Functions
Ryan O’Donnell. Analysis of Boolean Functions. Cambridge University Press, 2014
2014
-
[42]
Gradient starvation: A learning proclivity in neural networks
Mohammad Pezeshki, Sékou-Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. InAdvances in Neural Information Processing Systems, 2021
2021
-
[43]
Finding and fixing spurious patterns with explanations
Gregory Plumb, Marco Tulio Ribeiro, and Ameet Talwalkar. Finding and fixing spurious patterns with explanations. Transactions on Machine Learning Research, 2022
2022
-
[44]
Complexity matters: Dynamics of feature learning in the presence of spurious correlations
GuanWen Qiu, Da Kuang, and Surbhi Goel. Complexity matters: Dynamics of feature learning in the presence of spurious correlations. InInternational Conference on Machine Learning (ICML), 2024
2024
-
[45]
On the spectral bias of neural networks
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. InProceedings of the 36th International Conference on Machine Learning, 2019
2019
-
[46]
Random features for large-scale kernel machines
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. InAdvances in Neural Information Processing Systems, 2007
2007
-
[47]
Early stopping and non-parametric regression: an optimal data-dependent stopping rule.The Journal of Machine Learning Research, 15(1):335–366, 2014
Garvesh Raskutti, Martin J Wainwright, and Bin Yu. Early stopping and non-parametric regression: an optimal data-dependent stopping rule.The Journal of Machine Learning Research, 15(1):335–366, 2014
2014
-
[48]
Hashimoto, and Percy Liang
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2020
2020
-
[49]
An investigation of why overpa- rameterization exacerbates spurious correlations
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overpa- rameterization exacerbates spurious correlations. InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pag...
2020
-
[50]
Information-theoretic bias reduction via causal view of spurious correlation
Seonguk Seo, Joon-Young Lee, and Bohyung Han. Information-theoretic bias reduction via causal view of spurious correlation. InAAAI Conference on Artificial Intelligence, 2022
2022
-
[51]
The pitfalls of simplicity bias in neural networks
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. InAdvances in Neural Information Processing Systems, 2020
2020
-
[52]
Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2022
Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2022
2022
-
[53]
Generalization error of min-norm interpolators in transfer learning
Yanke Song, Sohom Bhattacharya, and Pragya Sur. Generalization error of min-norm interpolators in transfer learning. arXiv preprint arXiv:2406.13944, 2024
2024 arXiv
-
[54]
Regularized linear regression: A precise analysis of the estimation error
Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. Regularized linear regression: A precise analysis of the estimation error. InConference on Learning Theory, pages 1683–1709. PMLR, 2015
2015
-
[55]
Overcoming simplicity bias in deep networks using a feature sieve
Rishabh Tiwari and Pradeep Shenoy. Overcoming simplicity bias in deep networks using a feature sieve. In Proceedings of the 40th International Conference on Machine Learning, 2023
2023
-
[56]
Overparameterization improves robustness to covariate shift in high dimensions
Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Overparameterization improves robustness to covariate shift in high dimensions. InAdvances in Neural Information Processing Systems, 2021. 17
2021
-
[57]
Counterfactual invariance to spurious correlations in text classification
Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations in text classification. InAdvances in Neural Information Processing Systems, 2021
2021
-
[58]
Introduction to the non-asymptotic analysis of random matrices, page 210–268
Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices, page 210–268. Cambridge University Press, 2012
2012
-
[59]
High-dimensional probability: An introduction with applications in data science
Roman Vershynin. High-dimensional probability: An introduction with applications in data science. Cambridge university press, 2018
2018
-
[60]
Noise or signal: The role of image backgrounds in object recognition
Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. InInternational Conference on Learning Representations (ICLR), 2021
2021
-
[61]
Zhang, Sen Wu, Christopher Ré, and Weijie J
Fan Yang, Hongyang R. Zhang, Sen Wu, Christopher Ré, and Weijie J. Su. Precise high-dimensional asymptotics for quantifying heterogeneous transfers.arXiv preprint arXiv:2010.11750, 2023
2010 arXiv
-
[62]
Spurious correlations in machine learning: A survey.arXiv preprint arXiv:2402.12715, 2024
Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, and Aidong Zhang. Spurious correlations in machine learning: A survey.arXiv preprint arXiv:2402.12715, 2024
2024
-
[63]
Coping with label shift via distributionally robust optimisation
Jingzhao Zhang, Aditya Krishna Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. InInternational Conference on Learning Representations, 2021
2021
-
[64]
Examining and combating spurious features under distribution shift
Chunting Zhou, Xuezhe Ma, Paul Michel, and Graham Neubig. Examining and combating spurious features under distribution shift. InInternational Conference on Machine Learning (ICML), 2021
2021
-
[65]
On the relation between accuracy and fairness in binary classification
Indre Zliobaite. On the relation between accuracy and fairness in binary classification. In2nd ICML Workshop on Fairness, Accountability, and Transparency in Machine Learning (FATML), 2015. 18 A Additional Notation We define a sub-Gaussian random variable according to Proposit...
2015
-
[66]
its covarianceΣ ∈ R2d×2d is invertible, withλmax (Σ) = O (1), λmin (Σ) = Ω(1), and tr(Σ) = 2d
-
[67]
orthogonal features
for z ∼ PZ, the random variableΣ−1/2z has independent, mean-0, unit variance, sub-Gaussian entries. This assumption resembles the requirements A-B in Section 2.2 in [19], where we also included the scaling of the trace. To formally state the equivalent of Theorem 1 and Proposi...
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.