Pith. sign in

REVIEW 2 major objections 3 minor 86 references

Benign Overfitting in Out-of-Distribution Generalization of Linear Models

T0 review · 2 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows that over-parameterized ridge regression can exhibit benign overfitting under covariate shift, provided the target distribution's covariance is not too large along the source's minor directions.

desk verdict The main ridge-regression bound omits a bias term that can be nonzero even when the stated bound is zero; the paper is not correct as written, though the PCR side may be salvageable. read the letter →

arxiv 2412.14474 v1 pith:6DIJID6Y submitted 2024-12-19 cs.LG stat.ML

Shange Tang , Jiayun Wu , Jianqing Fan , Chi Jin This is my paper · ORCID
classification cs.LGstat.ML MSC 62J0762H25
keywords benignoverfittingout-of-distributiongeneralizationcovariateshiftridgeregressionprincipalcomponentexcessriskover-parameterizationeffectiverank
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that benign overfitting—where an over-parameterized model fits training noise yet still generalizes—survives out-of-distribution covariate shift in linear ridge regression. It proves a non-asymptotic excess-risk bound that reduces to known in-distribution and under-parameterized guarantees in the right special cases. The bound identifies concrete source-target covariance ratios that control whether overfitting stays benign. A sympathetic reader would care because the result draws a clear line between shifts where interpolation-style learning is safe and shifts where it fails.

What carries the argument

The central objects are the covariance comparison matrices T = Sigma_S,$k^{{-1/2}}$ Sigma_T,k Sigma_S,$k^{{-1/2}}$, U = Sigma_S,-k Sigma_T,-k, and V = Sigma_S,-$k^{2}$, together with the effective ranks r_k and R_k of the minor directions. The argument decomposes the ridge estimator through a block form of X_-k X_-k^T, showing that when the source's minor directions are high-dimensional, the n-dimensional training subspace is nearly orthogonal to test points, so noise fitted in those directions stays harmless. That orthogonality is what lets only the total magnitude of the target's minor covariance enter the bound.

What would settle it

Simulate the instance of Theorem 4 with growing n and test set covariance equal to the identity; if ridge regression's excess risk decays faster than 1/sqrt(n) for some choice of the ridge parameter, the claimed slow-rate lower bound is false. Alternatively, find a target covariance with large minor-direction mass where ridge regression still achieves a 1/n rate, contradicting the paper's distinction between benign and non-benign regimes.

Watch

Extended reading notes

Core claim

For over-parameterized ridge regression under covariate shift, the expected excess risk is bounded by quantities that compare the target covariance to the source covariance in the major and minor directions. When the target's minor-direction variance is comparable to or smaller than the source's, ridge regression keeps the same fast rate as in the in-distribution setting, and only the overall magnitude of the target covariance in the minor directions matters, not its spectral structure. In contrast, when the target has large variance in directions the source treats as minor, ridge regression can be stuck at a slow O(1/sqrt(n)) rate. Principal component regression achieves O(1/n) in that difficult regime when the true signal lies in the major subspace, and it does not require the source's minor directions to have high effective rank.

Load-bearing premise

The load-bearing premise is Assumption CondNum(k, delta, L): the source's minor-direction Gram matrix must have bounded condition number, which is equivalent to the minor directions having effective rank much larger than the sample size n; if this fails, the variance suppression that makes overfitting benign collapses.

Editorial extensions

If this is right

  • When the target covariance in the minor directions scales like the source, over-parameterized ridge regression retains the in-distribution benign-overfitting rate under covariate shift.
  • If the test distribution has substantial variance in source-minor directions, ridge regression's excess risk can be as slow as O(1/sqrt(n)) regardless of the ridge parameter.
  • Principal component regression reaches O(1/n) in that difficult regime when the signal lies in the major subspace, and it does not need high effective rank in the minor directions.
  • The bound recovers the known in-distribution ridge result when source and target coincide, and the known under-parameterized covariate-shift result when minor components vanish.
  • Larger effective rank in the source's minor directions relaxes the condition on the target's minor-direction bias, meaning over-parameterization improves robustness to minor-direction shift.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical diagnostic suggested by the bound is to estimate the ratios tr[T]/k, tr[U]/tr[V], and n ||Sigma_T,-k|| / (r_k ||Sigma_S,-k||) from data and use them to predict whether ridge regression will keep its fast rate under a given shift.
  • The results imply that the choice between ridge and PCR in OOD settings should hinge on whether the target's extra variance lies in source-minor directions; PCR discards those directions and is therefore safer when the signal is sparse in them.
  • One could test the sharpness of the slow-rate lower bound by seeking an analogous lower bound for any target with large minor mass; the paper constructs one concrete instance rather than a general matching lower bound.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper studies benign overfitting for over-parameterized ridge regression under covariate shift. Under a source covariance with k large eigenvalues and high effective rank in the minor directions, Theorem 2 gives non-asymptotic upper bounds on the expected excess risk in terms of the in-distribution bound of Tsigler and Bartlett (2023) multiplied by factors involving the target covariance. The authors claim that benign overfitting persists when the overall magnitude of the target minor covariance is comparable to that of the source, and that only this overall magnitude matters, not the target's spectral structure. Section 4 provides a lower bound showing that ridge regression can degrade to a O(1/√n) rate under large shifts in minor directions, while PCR achieves O(1/n). The appendix contains detailed concentration proofs and explicit sample-complexity calculations.

Significance. If correct, the result would be a substantial advance: it would give the first non-asymptotic OOD benign-overfitting guarantee for general target covariances and would recover both the in-distribution result of Tsigler and Bartlett (2023) and the under-parameterized covariate-shift result of Ge et al. (2024). The paper is also careful in making the sample complexity explicit and in providing simulation support. However, the central theorem is not correct as stated: the proof drops a direct minor-direction bias term using an inequality that fails on zero-variance source directions, and a counterexample satisfying all stated assumptions contradicts Theorem 2. The paper is therefore in need of substantial revision before the main claim can be accepted.

major comments (2)
  1. [Appendix A.3.4 and proof of Theorem 25] The direct minor-direction bias term is not carried into the final bound. The proof decomposes the minor bias as 3(β*_-k)^T Σ_{T,-k} β*_-k plus two further terms, and in the proof of Theorem 25 the direct term is bounded by ‖Σ_{S,-k}^{-1/2}Σ_{T,-k}Σ_{S,-k}^{-1/2}‖ · ‖β*_-k‖²_{Σ_{S,-k}}. This inequality is false when β*_-k has a component in the null space of Σ_{S,-k}: the left-hand side can be positive while the right-hand side is zero. Since B_ID in Theorem 2 and Corollary 26 is proportional to ‖β*_-k‖²_{Σ_{S,-k}}, the stated bound omits an error that is not controlled by the theorem's assumptions.
  2. [Theorem 2 / Corollary 26, with Section 3.2] The omission is load-bearing. Concrete instance: take k=1, Σ_S = diag(1, 1, …, 1, 0) with m unit minor eigenvalues and one zero eigenvalue, Σ_T = I_d, β* = e_d, λ = 1, n = m/2, and v = 0. Then A_k = I_n + X_-k X_-k^T is a Wishart-plus-identity matrix with condition number O(1) for m ≫ n, so CondNum holds; n < r_k = m+1; n/R_k ≈ 1/2; ‖T‖ = tr[U]/tr[V] = 1. The ridge estimator has zero coefficient in the zero-variance direction, so the true excess risk is ‖β*_-k‖²_{Σ_{T,-k}} = 1, while the claimed bound gives 0. This also refutes the Section 3.2 assertion that only the overall magnitude of the target minor covariance matters: the spectral alignment of the target with low-variance source directions controls the direct bias term. The theorem needs either an explicit additive term ‖β*_-k‖²_{Σ_{T,-k}} in B_ID or an additional assumption such as supp(β*_-k) ⊆ supp(Σ_{S,-k}) together with a relative-norm condition; the qualitative claims in the abstract and Section 3.2 must be revised accordingly.
minor comments (3)
  1. [Throughout] The assumption is referred to as both CondNum and condNum; a single consistent name should be used.
  2. [Lemma 18] Lemma 18 contains typographical errors, including '∥∆11∥, ∥∆12, ∥∆∥∥' and 'probablity'; a careful proofread is needed.
  3. [Section 5] The text says 'The fast rate O(1) of minimum norm interpolation is confirmed' and that PCR 'achieves the fast rate of O(1)', but the plotted quantity is the excess risk, which decays as O(1/n); the notation should be corrected to O(1/n).

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: OOD benign-overfitting bound is derived from fresh concentration lemmas and an external spectral assumption; self-citations are comparative, not load-bearing.

full rationale

The central bound (Theorem 2 / Corollary 26) is obtained by decomposing the expected excess risk into variance and bias terms and bounding each through Lemmas 19-24, assembled in Theorem 25. The key spectral condition, Assumption 1 (CondNum), is imported from Tsigler and Bartlett [2023], an external source, rather than from the authors' own prior work. The claimed recovery of the in-distribution rate is a comparison against that external benchmark. The only author-overlapping citations are Ge et al. [2024] and Ge et al. [2023]; these are used to state that Theorem 2 recovers a known under-parameterized rate and as proof inspiration ('the proof idea is similar to [Ge et al., 2023] and [Tripuraneni et al., 2021b]'), not as the justification for any asserted bound. No free parameter is fitted to data; the simulation is illustrative. The skeptic's concern that Corollary 26 omits a direct minor-direction bias term is a correctness/validity issue rather than circularity, because it does not involve a bound reducing to its own inputs by definition, fitting, or self-citation. Accordingly, no circular step is present.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The theory rests on a standard covariate-shift linear model with sub-gaussian source covariates, a high-effective-rank minor subspace (CondNum), and the over-parameterized regime n < r_k. These are domain assumptions, not free parameters; no constants are fitted to data. No new entities are postulated.

assumptions (6)
  • domain assumption Source covariate x has zero mean with covariance Σ_S and Σ_S^{-1/2} x is σ-sub-gaussian for a fixed constant σ.
    Stated in Section 2.1; needed for all concentration inequalities (Lemmas 7-15, 28).
  • domain assumption Outcome follows y = x^T β* + ε with ε independent of x, zero-mean, variance v^2; conditional distribution of y|x is invariant across source and target.
    Stated in Section 2.1; defines the covariate shift setting and allows bias-variance decomposition.
  • domain assumption Source covariance has k major eigenvalues and d-k minor eigenvalues with effective rank r_k > n; formally Assumption CondNum(k,δ,L) holds.
    Assumption 1, Section 3.1 and Appendix A; this is the key structural condition enabling benign overfitting.
  • domain assumption Over-parameterized regime: cN < n < r_k, where N is a polynomial in k + log(1/δ), λ1/λk, and 1 + (λ + sum_{j>k} λ_j)/λ_k; the ridge parameter λ is such that eλ = λ + sum_{j>k} λ_j = O(√n) in the benign regime.
    Theorem 2 statement and Remark 2; the lower bound on n is a sample-complexity condition, the upper bound n < r_k marks the over-parameterized regime.
  • domain assumption For the PCR results (Theorem 5), the true signal satisfies β*_{-k}=0, the eigenvalue gap λ_k - λ_{k+1} is positive, and independent data halves are used for PCA and regression.
    Section 4.2 and Lemma 31; without β*_{-k}=0 the PCR bound contains an extra bias term.
  • standard math Standard concentration results (sub-gaussian Hanson-Wright, Bernstein, Davis-Kahan) are used without proof, as cited.
    Appendix A and B; these are established results from Vershynin 2010/2018, Chen et al. 2021, Wedin 1973.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benign Overfitting in Out-of-Distribution Generalization of Linear Models." pith.science (2026). https://pith.science/paper/6DIJID6Y

@misc{pith2026241214474,
  author       = {Pith},
  title        = {Pith review of: Benign Overfitting in Out-of-Distribution Generalization of Linear Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6DIJID6Y}},
  note         = {Machine review of arXiv:2412.14474}
}
abstract

Benign overfitting refers to the phenomenon where an over-parameterized model fits the training data perfectly, including noise in the data, but still generalizes well to the unseen test data. While prior work provides some theoretical understanding of this phenomenon under the in-distribution setup, modern machine learning often operates in a more challenging Out-of-Distribution (OOD) regime, where the target (test) distribution can be rather different from the source (training) distribution. In this work, we take an initial step towards understanding benign overfitting in the OOD regime by focusing on the basic setup of over-parameterized linear models under covariate shift. We provide non-asymptotic guarantees proving that benign overfitting occurs in standard ridge regression, even under the OOD regime when the target covariance satisfies certain structural conditions. We identify several vital quantities relating to source and target covariance, which govern the performance of OOD generalization. Our result is sharp, which provably recovers prior in-distribution benign overfitting guarantee [Tsigler and Bartlett, 2023], as well as under-parameterized OOD guarantee [Ge et al., 2024] when specializing to each setup. Moreover, we also present theoretical results for a more general family of target covariance matrix, where standard ridge regression only achieves a slow statistical rate of $O(1/\sqrt{n})$ for the excess risk, while Principal Component Regression (PCR) is guaranteed to achieve the fast rate $O(1/n)$, where $n$ is the number of samples.

Figures

Figures reproduced from arXiv: 2412.14474 by the authors.

Figure 1
Figure 1. Simulation results for excess risks across varying training sample sizes. The shaded regions [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

86 extracted references · 50 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Importance sampling: Intrinsic dimension and computational cost

    Sergios Agapiou, Omiros Papaspiliopoulos, Daniel Sanz-Alonso, and Andrew M Stuart. Importance sampling: Intrinsic dimension and computational cost. Statistical Science, pages 405--431, 2017

    2017

  3. [3]

    On robustness of principal component regression

    Anish Agarwal, Devavrat Shah, Dennis Shen, and Dogyoon Song. On robustness of principal component regression. Advances in Neural Information Processing Systems, 32, 2019

    2019

  4. [5]

    Determining the number of factors in approximate factor models

    Jushan Bai and Serena Ng. Determining the number of factors in approximate factor models. Econometrica, 70 0 (1): 0 191--221, 2002

    2002

  5. [6]

    Prediction by supervised principal components

    Eric Bair, Trevor Hastie, Debashis Paul, and Robert Tibshirani. Prediction by supervised principal components. Journal of the American Statistical Association, 101 0 (473): 0 119--137, 2006

    2006

  6. [7]

    Benign overfitting in linear regression

    Peter L Bartlett, Philip M Long, G \'a bor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117 0 (48): 0 30063--30070, 2020

    2020

  7. [8]

    Deep learning: a statistical viewpoint

    Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica, 30: 0 87--201, 2021

    work page 2021

  8. [9]

    Laplacian eigenmaps for dimensionality reduction and data representation

    Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15 0 (6): 0 1373--1396, 2003

    2003

Show all 86 references
  1. [10]

    Two models of double descent for weak features

    Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 2 0 (4): 0 1167--1180, 2020

    2020

  2. [11]

    Analysis of representations for domain adaptation

    Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006

    2006

  3. [12]

    A new look at an old problem: A universal learning approach to linear regression

    Koby Bibas, Yaniv Fogel, and Meir Feder. A new look at an old problem: A universal learning approach to linear regression. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 2304--2308. IEEE, 2019

    2019

  4. [13]

    Project cost estimation using principal component regression

    Swee Lean Chan and Moonseo Park. Project cost estimation using principal component regression. Construction Management and Economics, 23 0 (3): 0 295--304, 2005

    2005

  5. [14]

    High-dimensional kernel methods under covariate shift: Data-dependent implicit regularization

    Yihang Chen, Fanghui Liu, Taiji Suzuki, and Volkan Cevher. High-dimensional kernel methods under covariate shift: Data-dependent implicit regularization. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net,...

    2024

  6. [15]

    Spectral methods for data science: A statistical perspective

    Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, et al. Spectral methods for data science: A statistical perspective. Foundations and Trends in Machine Learning , 14 0 (5): 0 566--806, 2021

    2021

  7. [16]

    On the robustness of the minimim l2 interpolator

    Geoffrey Chinot and Matthieu Lerasle. On the robustness of the minimim l2 interpolator. Bernoulli, 2022

    2022

  8. [17]

    Learning bounds for importance weighting

    Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. Advances in neural information processing systems, 23, 2010

    2010

  9. [18]

    Genetic algorithms applied to the selection of factors in principal component regression

    Uwe Depczynski, VJ Frost, and K Molt. Genetic algorithms applied to the selection of factors in principal component regression. Analytica Chimica Acta, 420 0 (2): 0 217--227, 2000

    2000

  10. [19]

    High-dimensional asymptotics of prediction: Ridge regression and classification

    Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46 0 (1): 0 247--279, 2018

    2018

  11. [20]

    Factor augmented sparse throughput deep relu neural networks for high dimensional regression

    Jianqing Fan and Yihong Gu. Factor augmented sparse throughput deep relu neural networks for high dimensional regression. Journal of the American Statistical Association, pages 1--15, 2023

    2023

  12. [21]

    Factor-adjusted regularized model selection

    Jianqing Fan, Yuan Ke, and Kaizheng Wang. Factor-adjusted regularized model selection. Journal of Econometrics, 216 0 (1): 0 71--85, 2020

    2020

  13. [22]

    Robust high dimensional factor models with applications to statistical machine learning

    Jianqing Fan, Kaizheng Wang, Yiqiao Zhong, and Ziwei Zhu. Robust high dimensional factor models with applications to statistical machine learning. Statistical science: a review journal of the Institute of Mathematical Statistics, 36 0 (2): 0 303, 2021

    2021

  14. [23]

    On the provable advantage of unsupervised pretraining

    Jiawei Ge, Shange Tang, Jianqing Fan, and Chi Jin. On the provable advantage of unsupervised pretraining. arXiv preprint arXiv:2303.01566, 2023

    2023 arXiv

  15. [24]

    Maximum likelihood estimation is all you need for well-specified covariate shift

    Jiawei Ge, Shange Tang, Jianqing Fan, Cong Ma, and Chi Jin. Maximum likelihood estimation is all you need for well-specified covariate shift. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024....

    2024

  16. [25]

    When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33: 0 14820--14830, 2020

    Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33: 0 14820--14830, 2020

    2020

  17. [26]

    Linearized two-layers neural networks in high dimension

    Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 49 0 (2), 2021

    2021

  18. [27]

    Domain adaptation for medical image analysis: a survey

    Hao Guan and Mingxia Liu. Domain adaptation for medical image analysis: a survey. IEEE Transactions on Biomedical Engineering, 69 0 (3): 0 1173--1185, 2021

    2021

  19. [28]

    Some cautionary notes on the use of principal components regression

    Ali S Hadi and Robert F Ling. Some cautionary notes on the use of principal components regression. The American Statistician, 52 0 (1): 0 15--19, 1998

    1998

  20. [29]

    On the value of target data in transfer learning

    Steve Hanneke and Samory Kpotufe. On the value of target data in transfer learning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  21. [30]

    On the benefits of over-parameterization for out-of-distribution generalization

    Yifan Hao, Yong Lin, Difan Zou, and Tong Zhang. On the benefits of over-parameterization for out-of-distribution generalization. CoRR, abs/2403.17592, 2024. doi:10.48550/ARXIV.2403.17592. URL https://doi.org/10.48550/arXiv.2403.17592

    2024 doi arXiv

  22. [31]

    Surprises in high-dimensional ridgeless least squares interpolation

    Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. Annals of statistics, 50 0 (2): 0 949, 2022

    2022

  23. [32]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019

    1903 arXiv

  24. [33]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international c...

    2021

  25. [34]

    Alternative principal components regression procedures for dendrohydrologic reconstructions

    Hugo G Hidalgo, Thomas C Piechota, and John A Dracup. Alternative principal components regression procedures for dendrohydrologic reconstructions. Water Resources Research, 36 0 (11): 0 3241--3249, 2000

    2000

  26. [35]

    Improved principal component regression for face recognition under illumination variations

    Shih-Ming Huang and Jar-Ferr Yang. Improved principal component regression for face recognition under illumination variations. IEEE signal processing letters, 19 0 (4): 0 179--182, 2012

    2012

  27. [36]

    Investigation of alternative regressions: Some practical examples

    JNR Jeffers. Investigation of alternative regressions: Some practical examples. Journal of the Royal Statistical Society. Series D (The Statistician), 30 0 (2): 0 79--88, 1981

    1981

  28. [37]

    Two case studies in the application of principal component analysis

    John NR Jeffers. Two case studies in the application of principal component analysis. Journal of the Royal Statistical Society: Series C (Applied Statistics), 16 0 (3): 0 225--236, 1967

    1967

  29. [38]

    A note on the use of principal components in regression

    Ian T Jolliffe. A note on the use of principal components in regression. Journal of the Royal Statistical Society Series C: Applied Statistics, 31 0 (3): 0 300--303, 1982

    1982

  30. [39]

    Double descent and overfitting under noisy inputs and distribution shift for linear denoisers

    Chinmaya Kausik, Kashvi Srivastava, and Rishi Sonthalia. Double descent and overfitting under noisy inputs and distribution shift for linear denoisers. Trans. Mach. Learn. Res., 2024, 2024. URL https://openreview.net/forum?id=HxfqTdLIRF

    2024

  31. [40]

    Multivariate concentration determination using principal component regression with residual analysis

    Richard B Keithley, R Mark Wightman, and Michael L Heien. Multivariate concentration determination using principal component regression with residual analysis. TrAC Trends in Analytical Chemistry, 28 0 (9): 0 1127--1136, 2009

    2009

  32. [41]

    The optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization

    Dmitry Kobak, Jonathan Lomond, and Benoit Sanchez. The optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization. Journal of Machine Learning Research, 21 0 (169): 0 1--16, 2020

    2020

  33. [42]

    Uniform convergence of interpolators: Gaussian width, norm bounds and benign overfitting

    Frederic Koehler, Lijia Zhou, Danica J Sutherland, and Nathan Srebro. Uniform convergence of interpolators: Gaussian width, norm bounds and benign overfitting. Advances in Neural Information Processing Systems, 34: 0 20657--20668, 2021

    2021

  34. [43]

    Wilds: A benchmark of in-the-wild distribution shifts

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, p...

    2021

  35. [44]

    Marginal singularity, and the benefits of labels in covariate-shift

    Samory Kpotufe and Guillaume Martinet. Marginal singularity, and the benefits of labels in covariate-shift. In Conference On Learning Theory, pages 1882--1886. PMLR, 2018

    2018

  36. [45]

    Forecasting of air quality in delhi using principal component regression technique

    Anikender Kumar and Pramila Goyal. Forecasting of air quality in delhi using principal component regression technique. Atmospheric Pollution Research, 2 0 (4): 0 436--444, 2011

    2011

  37. [46]

    Near-optimal linear regression under distribution shift

    Qi Lei, Wei Hu, and Jason Lee. Near-optimal linear regression under distribution shift. In International Conference on Machine Learning, pages 6164--6174. PMLR, 2021

    2021

  38. [47]

    On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels

    Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai. On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels. In Conference on Learning Theory, pages 2683--2711. PMLR, 2020

    2020

  39. [48]

    Principal component regression analysis with spss

    RX Liu, J Kuang, Qiong Gong, and XL Hou. Principal component regression analysis with spss. Computer methods and programs in biomedicine, 71 0 (2): 0 141--147, 2003

    2003

  40. [49]

    Optimally tackling covariate shift in rkhs-based nonparametric regression

    Cong Ma, Reese Pathak, and Martin J Wainwright. Optimally tackling covariate shift in rkhs-based nonparametric regression. The Annals of Statistics, 51 0 (2): 0 738--761, 2023

    2023

  41. [50]

    Minimum-norm interpolation under covariate shift

    Neil Mallinar, Austin Zane, Spencer Frei, and Bin Yu. Minimum-norm interpolation under covariate shift. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=Zw7TcnTmHj

    2024

  42. [51]

    Principal components regression in exploratory statistical research

    William F Massy. Principal components regression in exploratory statistical research. Journal of the American Statistical Association, 60 0 (309): 0 234--256, 1965

    1965

  43. [52]

    The generalization error of random features regression: Precise asymptotics and the double descent curve

    Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75 0 (4): 0 667--766, 2022

    2022

  44. [53]

    Generalization error of random feature and kernel methods: hypercontractivity and kernel matrix concentration

    Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Generalization error of random feature and kernel methods: hypercontractivity and kernel matrix concentration. Applied and Computational Harmonic Analysis, 59: 0 3--84, 2022

    2022

  45. [54]

    Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization

    John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International conferen...

    2021

  46. [55]

    The interpolation phase transition in neural networks: Memorization and generalization under lazy training

    Andrea Montanari and Yiqiao Zhong. The interpolation phase transition in neural networks: Memorization and generalization under lazy training. The Annals of Statistics, 50 0 (5): 0 2816--2847, 2022

    2022

  47. [56]

    Minimax lower bounds for transfer learning with linear and one-hidden layer neural networks

    Mohammadreza Mousavi Kalan, Zalan Fabian, Salman Avestimehr, and Mahdi Soltanolkotabi. Minimax lower bounds for transfer learning with linear and one-hidden layer neural networks. Advances in Neural Information Processing Systems, 33: 0 1959--1969, 2020

    1959

  48. [57]

    Harmless interpolation of noisy data in regression

    Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpolation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1 0 (1): 0 67--83, 2020

    2020

  49. [58]

    Principal component regression in nir analysis: viewpoints, background details and selection of components

    Tormod N s and Harald Martens. Principal component regression in nir analysis: viewpoints, background details and selection of components. Journal of chemometrics, 2 0 (2): 0 155--167, 1988

    1988

  50. [59]

    More data can hurt for linear regression: Sample-wise double descent

    Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent. arXiv preprint arXiv:1912.07242, 2019

    1912 arXiv

  51. [60]

    In defense of uniform convergence: Generalization via derandomization with an application to interpolating predictors

    Jeffrey Negrea, Gintare Karolina Dziugaite, and Daniel Roy. In defense of uniform convergence: Generalization via derandomization with an application to interpolating predictors. In International Conference on Machine Learning, pages 7263--7272. PMLR, 2020

    2020

  52. [61]

    Manifold regularization and semi-supervised learning: Some theoretical analyses

    Partha Niyogi. Manifold regularization and semi-supervised learning: Some theoretical analyses. Journal of Machine Learning Research, 14 0 (5), 2013

    2013

  53. [62]

    A new similarity measure for covariate shift with applications to nonparametric regression

    Reese Pathak, Cong Ma, and Martin Wainwright. A new similarity measure for covariate shift with applications to nonparametric regression. In International Conference on Machine Learning, pages 17517--17530. PMLR, 2022

    2022

  54. [63]

    Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389--5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389--5400. PMLR, 2019

    2019

  55. [64]

    Asymptotics of ridge (less) regression under general source condition

    Dominic Richards, Jaouad Mourtada, and Lorenzo Rosasco. Asymptotics of ridge (less) regression under general source condition. In International Conference on Artificial Intelligence and Statistics, pages 3889--3897. PMLR, 2021

    2021

  56. [65]

    The implicit bias of benign overfitting

    Ohad Shamir. The implicit bias of benign overfitting. Journal of Machine Learning Research, 24 0 (113): 0 1--40, 2023

    2023

  57. [66]

    Improving predictive inference under covariate shift by weighting the log-likelihood function

    Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90 0 (2): 0 227--244, 2000

    2000

  58. [67]

    More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory

    James B Simon, Dhruva Karkada, Nikhil Ghosh, and Mikhail Belkin. More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory. arXiv preprint arXiv:2311.14646, 2023

    2023 arXiv

  59. [68]

    Forecasting using principal components from a large number of predictors

    James H Stock and Mark W Watson. Forecasting using principal components from a large number of predictors. Journal of the American statistical association, 97 0 (460): 0 1167--1179, 2002

    2002

  60. [69]

    A correlation principal component regression analysis of nir data

    Jianguo Sun. A correlation principal component regression analysis of nir data. Journal of Chemometrics, 9 0 (1): 0 21--29, 1995

    1995

  61. [70]

    Overparameterization improves robustness to covariate shift in high dimensions

    Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Overparameterization improves robustness to covariate shift in high dimensions. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Informatio...

    2021

  62. [71]

    Provable meta-learning of linear representations

    Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. In International Conference on Machine Learning, pages 10434--10443. PMLR, 2021 b

    2021

  63. [72]

    Bartlett

    Alexander Tsigler and Peter L. Bartlett. Benign overfitting in ridge regression. J. Mach. Learn. Res., 24: 0 123:1--123:76, 2023. URL http://jmlr.org/papers/v24/22-1398.html

    2023

  64. [73]

    An inequality for trace ideals

    J Leo van Hemmen and Tsuneya Ando. An inequality for trace ideals. Communications in Mathematical Physics, 76: 0 143--148, 1980

    1980

  65. [74]

    Introduction to the non-asymptotic analysis of random matrices

    Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. CoRR, abs/1011.3027, 2010. URL http://arxiv.org/abs/1011.3027

    2010 arXiv

  66. [75]

    High-dimensional probability: An introduction with applications in data science, volume 47

    Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

    2018

  67. [76]

    Principal component regression, ridge regression and ridge principal component regression in spectroscopy calibration

    Evelyne Vigneau, MF Devaux, EM Qannari, and P Robert. Principal component regression, ridge regression and ridge principal component regression in spectroscopy calibration. Journal of Chemometrics: A Journal of the Chemometrics Society, 11 0 (3): 0 239--249, 1997

    1997

  68. [77]

    Perturbation theory for pseudo-inverses

    Per-Ake Wedin. Perturbation theory for pseudo-inverses. BIT Numerical Mathematics, 13: 0 217--232, 1973

    1973

  69. [78]

    Assaying out-of-distribution generalization in transfer learning

    Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of-distribution generalization in transfer learning. Advances in Neural Information Processing Syst...

    2022

  70. [79]

    On the optimal weighted l2 regularization in overparameterized linear regression

    Denny Wu and Ji Xu. On the optimal weighted l2 regularization in overparameterized linear regression. Advances in Neural Information Processing Systems, 33: 0 10112--10123, 2020

    2020

  71. [80]

    On the number of variables to use in principal component regression

    Ji Xu and Daniel J Hsu. On the number of variables to use in principal component regression. Advances in neural information processing systems, 32, 2019

    2019

  72. [81]

    Understanding why generalized reweighting does not improve over erm

    Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Understanding why generalized reweighting does not improve over erm. arXiv preprint arXiv:2201.12293, 2022

    2022 arXiv

  73. [82]

    A class of geometric structures in transfer learning: Minimax bounds and optimality

    Xuhui Zhang, Jose Blanchet, Soumyadip Ghosh, and Mark S Squillante. A class of geometric structures in transfer learning: Minimax bounds and optimality. In International Conference on Artificial Intelligence and Statistics, pages 3794--3820. PMLR, 2022

    2022

  74. [83]

    On uniform convergence and low-norm interpolation learning

    Lijia Zhou, Danica J Sutherland, and Nati Srebro. On uniform convergence and low-norm interpolation learning. Advances in Neural Information Processing Systems, 33: 0 6867--6877, 2020

    2020

  75. [84]

    Unsupervised domain adaptation for semantic segmentation via class-balanced self-training

    Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pages 289--305, 2018

    2018

  76. [85]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  77. [86]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  78. [87]

    scenario, where the last d-k components of the target covariance matrix _T scale similarly to or smaller than those of the source. However, when the shift is ``high-dimensional

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.