Pith. sign in

REVIEW 2 major objections 6 minor 4 cited by

Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization

T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proves that early-stopped gradient descent dominates ridge regression for all well-specified linear regression problems, while remaining incomparable with SGD overall.

desk verdict Substantial new bounds for GD plus a surprising GD-loses-to-SGD construction, but the headline GD-dominates-ridge claim is formally proven only for a Bayesian-averaged class, and the paper's own extension step beyond that class is not valid. read the letter →

arxiv 2509.17251 v2 pith:B3N35MC7 submitted 2025-09-21 stat.ML cs.LG

Jingfeng Wu , Peter L. Bartlett , Sham M. Kakade , Jason D. Lee , Bin Yu This is my paper · ORCID
classification stat.MLcs.LG MSC 62J0562J0768T05
keywords gradientdescentridgeregressionstochasticimplicitregularizationexcessrisklinearearlystoppingbenignoverfitting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper compares the finite-sample excess risks of three estimators on every well-specified linear regression problem: gradient descent with early stopping, ridge regression, and online SGD. Its central claim is that GD's implicit regularization dominates explicit ridge regularization: with comparable regularization (stopping time inversely proportional to the ridge penalty), GD's excess risk is always within a constant factor of ridge's, and for natural problems ridge can be polynomially worse even after optimal tuning. Against SGD, the picture is mixed: GD is incomparable overall, being polynomially better on some problems and polynomially worse on specially constructed ones, but GD dominates SGD on problems whose covariance spectra decay fast and continuously. If true, this means a simple unregularized early-stopped algorithm is a statistically safe, often superior substitute for a tuned regularized estimator.

What carries the argument

The argument runs through a scalar-parameter family of bounds indexed by an effective regularization strength: for GD this is $1/(\eta t)$ plus a tail quantity, exactly matching ridge's $\lambda$ at the critical index $k^*$ where bias and variance balance. The load-bearing object is the GD shrinkage matrix $\tilde A = (I - (I - \eta XX^\top/n)^t)^{-1} XX^\top$, which the paper proves is comparable to ridge's shrinkage matrix $XX^\top + n\lambda I$; this reduction yields a new ridge-type upper bound for GD (Theorem 3.1), a new SGD-type upper bound (Theorem 4.3) that decomposes risk into bias, variance, and an effective-variance term controlled by the order-1 effective dimension $D_1$, and a new lower bound for GD (Theorem 4.1). Assumption 3, that the covariance spectrum decays fast and continuously, brings $D_1$ within a constant factor of the effective dimension $D$, which is exactly what lets GD match SGD's bias decay in that subclass.

What would settle it

Exhibit a well-specified linear regression problem satisfying Assumptions 1 and 2 where, for some fixed ridge penalty $\lambda\ge 0$, every GD stopping time $t$ gives expected excess risk larger than a constant times the ridge expected excess risk; a simulated Gaussian spike-covariance problem whose tail spectrum is slow is the natural place to look. Such an instance would directly contradict Theorem 3.2.

Watch

Extended reading notes

Core claim

For the class of well-specified linear regression problems satisfying the paper's conditions, the paper proves (Theorem 3.2) that for every ridge penalty $\lambda \ge 0$ there is a GD stopping time $t$ such that, with high probability, $E[E_\mu(\hat w^{\mathrm{GD}}_t)|X] \le c\, E[E_\mu(\hat w^{\mathrm{ridge}}_\lambda)|X]$, where $c$ depends only on the subgaussian constant and the signal-to-noise ratio, not on the problem or sample size. It also proves that the reverse inequality can fail polynomially: in $(a,r)$-power-law classes with source condition $r>1$, ridge's optimally tuned excess risk is $\Omega(n^{-2a/(1+2a)})$, while GD attains the minimax rate $n^{-2ar/(1+2ar)}$. The paper further shows GD and SGD are incomparable by constructing a spike-covariance family where GD's risk is $\Omega(n^{-0.2})$ while SGD achieves $O(\log n / n)$, and it shows GD dominates SGD under Assumption 3, a fast continuously decaying spectrum condition that every power-law spectrum satisfies.

Load-bearing premise

The proof that GD dominates ridge needs a symmetry condition on the optimal parameter's prior (Assumption 2C') or, without it, a comparison between a high-probability GD upper bound and an expectation ridge lower bound; all upper bounds also require the whitened covariates to have independent, subgaussian entries, and if those fail the dominance statements are unproven.

Editorial extensions

If this is right

  • For any well-specified linear regression problem, a user can switch from tuned ridge regression to early-stopped GD and never pay more than a constant factor in excess risk, while sometimes gaining polynomially in sample size.
  • In power-law classes with smooth source conditions ($r>1$), ridge regression and SGD are polynomially suboptimal, whereas GD is minimax optimal for all $r\ge 0$, so early stopping alone achieves the best possible rate.
  • GD and SGD occupy incomparable regimes: benign-overfitting-style spike problems favor SGD by a polynomial gap, while fast-decay-spectrum problems favor GD, so the batch versus online choice must depend on the covariance structure.
  • Within the fast continuously decaying subclass, which contains all standard capacity-condition models, GD is at least as good as SGD up to constants, making early-stopped GD a robust default there.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural empirical extension would be to test whether data-driven stopping-time selection, such as validation, preserves the dominance over tuned ridge in finite samples; the paper's fixed-proportional stopping proof does not cover that selection rule.
  • The paper notes that multi-epoch SGD is no worse than either GD or SGD, which suggests the dominance results may extend to multi-epoch procedures, but that extension is not established here.
  • Because the hard GD-versus-SGD examples are built from benign-overfitting spike spectra, the paper can be read as delineating when early stopping should be preferred to online averaging; a practical rule of thumb might pick between them based on the empirical eigenvalue decay rate.
  • The comparison framework could be tested on kernel regression with nontrivial spectra, though all bounds here rely on analytic formulas special to linear regression.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies finite-sample excess-risk comparisons among gradient descent (GD), ridge regression, and online SGD in well-specified linear regression. It contributes two new upper bounds for GD (a ridge-type bound in Theorem 3.1 and an SGD-type bound in Theorem 4.3) and a new lower bound for GD (Theorem 4.1). Combining these with existing tight bounds for ridge and SGD, the paper claims (i) GD dominates ridge regression for all well-specified problems, (ii) GD and SGD are incomparable, and (iii) GD dominates SGD for problems with fast, continuously decaying covariance spectra. It also derives exact rates for the (a,r)-power-law class, recovering known minimax rates for GD and the suboptimality regions for ridge and SGD.

Significance. If the results hold as stated, this would be a valuable instance-wise complement to the classical worst-case minimax theory, identifying explicit cases where implicit regularization is provably at least as good as explicit regularization. The new finite-sample bounds for GD, especially the SGD-type bound in Theorem 4.3 and the lower bound in Theorem 4.1, are potentially reusable technical tools. The appendix proofs are detailed and, as far as I checked, internally coherent. However, the headline claim of universal dominance of GD over ridge is currently not supported by the formal statements, because the only proved theorem for that claim uses a Bayesian prior over the target parameter and a comparison of high-probability and expectation bounds that is not justified as written. This limits the significance of the paper's central conceptual message, although the individual bounds and rate computations remain of interest.

major comments (2)
  1. [Section 3, Theorem 3.2 and its proof (pp.8-9)] The theorem is stated for the Bayesian class L'_b, and the proof explicitly says 'we take an additional expectation over w*' before combining Theorem 3.1 with Proposition 2.1. This yields a bound that is averaged over the prior on w*, not a guarantee for each fixed w* in the class. The subsequent claim that 'if one accepts comparing the high probability upper bound in Theorem 3.1 with the expectation lower bound in Proposition 2.1, then Assumption 2C' can be replaced by Assumption 2C' is not a valid step: Theorem 3.1 gives a high-probability upper bound on E[E_GD|X], whereas the third bullet of Proposition 2.1 is an unconditional expectation lower bound on E[E_ridge]. On the high-probability event on which the GD upper bound holds, the conditional expectation E[E_ridge|X] may be far below its unconditional average, so the claimed inequality E[E_GD|X] ≤ c E[E_ridge|X] does not follow. Because the abstract and Table 1 advertise 'all well-specified linear regression problems' and the formal L_b class is meant to be the non-Bayesian version, this is a load-bearing gap in the central dominance claim. The authors should either prove a genuine high-probability ridge lower bound under Assumption 2C for fixed w*, or reformulate the dominance result as a Bayesian-averaged statement and adjust the abstract and table accordingly.
  2. [Abstract and Section 1 vs. Eq. (1)] The abstract states the first finding for 'any well-specified linear regression problem', but the formal class L_b in Eq. (1) requires Assumption 1A (independent sigma_x^2-subgaussian entries of Sigma^{-1/2}x), Assumption 2B, Assumption 2C (symmetric x), and the signal-to-noise bound ||w*||_Sigma^2 ≤ b sigma^2. These are substantial restrictions that exclude, for example, designs with dependent whitened coordinates or strongly skewed x distributions. The dominance claim is also never proved for the full non-Bayesian L_b, as noted above. The authors should either prove the result under weaker assumptions or explicitly qualify the headline claim so that the abstract's 'any well-specified linear regression problem' is replaced by the formal class actually covered.
minor comments (6)
  1. [Section 3, proof of Theorem 3.2] The phrase 'we take an additional expectation over w*' needs a precise probability-space statement: the high-probability event in Theorem 3.1 is over X for fixed w*, and averaging over w* can change the probability measure with respect to which the inequality holds. Please clarify whether the theorem's 'with probability' is over (X,w*) jointly or over X for the prior-averaged risk.
  2. [Notation, Section 2] The definition of f ≲ g says 'for every x, f(x) ≤ c g(x)', but x is overloaded with the covariate; it would be clearer to say 'for all arguments' or to use a different placeholder.
  3. [Theorem 4.3 (p.11)] The effective variance error is stated twice in the same display ('EffectiveVariance≤ ...' followed by 'where EffectiveVariance≤ ...'), and the brace-heavy decomposition of the bound is difficult to parse. Restating the bound with named components (e.g., B_var, B_eff) would improve readability.
  4. [Appendix A, p.21] Typo: 'varaince' should be 'variance'.
  5. [Appendix B, Lemma B.4 proof; Appendix C; Appendix E] Minor typos: 'subGussian' should be 'subgaussian'; 'diagnoal' should be 'diagonal'; 'becasue' should be 'because'; 'defintion' should be 'definition'.
  6. [Section 5 and Figure 1] The claim that 'the best of ridge regression and SGD is (nearly) optimal for all power law classes' relies on Corollary 5.3 whose lower bound is eOmega(n^{-2r}), i.e., up to log factors; the statement and the figure should make the logarithmic factor explicit.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: GD-vs-ridge and GD-vs-SGD comparisons combine new GD bounds with independent published ridge/SGD bounds; self-citations are supporting lemmas, not conclusion-presupposing.

full rationale

The paper's central claims are comparisons: Theorem 3.2 bounds GD's excess risk by a constant times ridge's using a new GD upper bound (Theorem 3.1) and a previously published ridge lower bound (Proposition 2.1, from Tsigler and Bartlett 2023 with an expectation variant by Zou et al. 2021). These bounds are not the conclusion being derived: neither says GD dominates ridge, and Theorem 3.1 is proved from matrix concentration (Lemmas B.2-B.8) without assuming any comparison. Likewise Theorem 4.4 combines the new SGD-type upper bound for GD (Theorem 4.3) with the published SGD lower bound (Proposition 2.2) to show GD is no worse than SGD, again a genuine comparison of independent quantities. The cited works overlap with the present authors (Wu, Kakade, Bartlett), but they are prior, published, parameterized by stated assumptions (Assumptions 1-2) that do not include the target dominance statements, and are used as external benchmarks; thus the self-citations do not make the derivation circular. The paper's own caveat that the non-Bayesian L_b extension relies on comparing a high-probability upper bound with an expectation lower bound, and the restriction of the formal Theorem 3.2 to L'_b, are scope/technical-gap issues, not reductions of the result to its inputs. No fitted parameter is relabeled as a prediction; the matching of 1/(eta t) to lambda is a comparison device, not a definition of the result.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

This paper introduces no fitted constants or hand-tuned parameters; the only constants are problem-class parameters (b, a, r) and algorithm hyperparameters (step size, stopping time) that are set as functions of the problem to match the stated rates. The derivation relies on previously established tight bounds for ridge regression and SGD, and standard concentration inequalities.

assumptions (7)
  • domain assumption Tight ridge regression bounds as established by Tsigler and Bartlett (2023) and Bartlett et al. (2020), stated as Proposition 2.1.
    Used as the external benchmark for ridge's risk in the dominance comparison (Theorem 3.2).
  • domain assumption Tight SGD bounds as established by Wu et al. (2022b, 2022a), stated as Proposition 2.2.
    Used as the external benchmark for SGD's risk in the GD versus SGD comparisons.
  • domain assumption Minimax lower bound for the (a,r)-power law class from Zhang et al. (2024), stated as Proposition 5.1.
    Used to conclude minimax optimality of GD in Corollary 5.4.
  • domain assumption Independent and sigma_x^2-subgaussian entries of whitened covariates (Assumption 1A).
    Required by the new GD upper bounds and by the cited ridge and SGD bounds.
  • domain assumption Well-specified noise and bounded conditional variance (Assumptions 2B and 1B).
    Defines the problem class and enables the bias-variance decomposition.
  • domain assumption Technical symmetry for lower bounds: Assumption 2C or Bayesian variant 2C'.
    Used to cancel off-diagonal terms in the GD lower bound and to compare high-probability bounds for ridge.
  • standard math Standard matrix concentration inequalities, e.g., Vershynin's results invoked in Lemma B.4.
    Used to prove the new upper bounds for GD.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization." pith.science (2026). https://pith.science/paper/B3N35MC7

@misc{pith2026250917251,
  author       = {Pith},
  title        = {Pith review of: Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B3N35MC7}},
  note         = {Machine review of arXiv:2509.17251}
}
read the original abstract

Existing theory suggests that for linear regression problems categorized by capacity and source conditions, gradient descent (GD) is always minimax optimal, while both ridge regression and online stochastic gradient descent (SGD) are polynomially suboptimal for certain categories of such problems. Moving beyond minimax theory, this work provides instance-wise comparisons of the finite-sample risks for these algorithms on any well-specified linear regression problem. Our analysis yields three key findings. First, GD dominates ridge regression: with comparable regularization, the excess risk of GD is always within a constant factor of that of ridge, but ridge can be polynomially worse even when tuned optimally. Second, GD is incomparable with SGD. While it is known that for certain problems GD can be polynomially better than SGD, the reverse is also true: we construct problems, inspired by benign overfitting theory, where optimally stopped GD is polynomially worse. Finally, GD dominates SGD for a significant subclass of problems -- those with fast and continuously decaying covariance spectra -- which includes all problems satisfying the standard capacity condition.

Figures

Figures reproduced from arXiv: 2509.17251 by the authors.

Figure 1
Figure 1. A summary of results in Section 5. Left: rates for learning (𝑎, 𝑟)-power law class (see P𝑎,𝑟 defined in (3)). Right: exponents of 1/𝑛 in the rates when fixing 𝑎 and varying 𝑟. GD is minimax optimal for all 𝑎 > 1 and 𝑟 ≥ 0 (Corollary 5.4). Ridge regression is minimax optimal for 0 ≤ 𝑟 ≤ 1 but is polynomially suboptimal otherwise (Corollary 5.2). SGD is (nearly) minimax optimal for 𝑟 ≥ (𝑎 − 1)/(2𝑎) but is polynomially… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Minimax Optimal Early-Stopped Gradient Descent for Gaussian Mixture Classification

    stat.ML 2026-08 accept novelty 7.0 of 10

    Early-stopped gradient descent achieves the minimax-optimal classification error for Gaussian mixtures with label noise under fast-decaying covariance spectra, while interpolating classifiers can be exponentially worse.

  2. Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent

    cs.LG 2026-07 accept novelty 7.0 of 10

    Early-stopped negative-shifted gradient descent beats every stable negative-ridge endpoint and every positive shrinker in gapped high-dimensional linear models, by polynomial risk factors.

  3. A Defense of the Quadratic Model

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Local Taylor-expanded quadratic models reproduce a 150M-parameter LLM's validation loss for up to 10% of training late in the run, and LLM pretraining operates within a factor of 2 of a stochastic or deterministic edg...

  4. Statistical Inference on Gradient Flows

    math.ST 2026-05 unverdicted novelty 7.0 of 10

    Proves uniform CLT for gradient flows in ERM and constructs an algorithm-aware, inversion-free covariance estimator for asymptotically valid time-uniform confidence intervals.

Reference graph

Works this paper leans on

33 extracted references · 17 canonical work pages · cited by 4 Pith papers

  1. [1]

    A continuous-time view of early stopping for least squares regression

    Alnur Ali, J Zico Kolter, and Ryan J Tibshirani. A continuous-time view of early stopping for least squares regression. In The 22nd international conference on artificial intelligence and statistics, pages 1370--1378. PMLR, 2019

    work page 2019

  2. [2]

    For valid generalization the size of the weights is more important than the size of the network

    Peter L Bartlett. For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems, 9, 1996

    work page 1996

  3. [3]

    Benign overfitting in linear regression

    Peter L Bartlett, Philip M Long, G \'a bor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117 0 (48): 0 30063--30070, 2020

    2020

  4. [4]

    Deep learning: a statistical viewpoint

    Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica, 30: 0 87--201, 2021

    2021

  5. [5]

    On regularization algorithms in learning theory

    Frank Bauer, Sergei Pereverzev, and Lorenzo Rosasco. On regularization algorithms in learning theory. Journal of complexity, 23 0 (1): 0 52--72, 2007

    2007

  6. [6]

    Optimal rates for regularization of statistical inverse learning problems

    Gilles Blanchard and Nicole M \"u cke. Optimal rates for regularization of statistical inverse learning problems. Foundations of Computational Mathematics, 18 0 (4): 0 971--1013, 2018

    2018

  7. [7]

    Boosting with the L_2 loss: regression and classification

    Peter B \"u hlmann and Bin Yu. Boosting with the L_2 loss: regression and classification. Journal of the American Statistical Association, 98 0 (462): 0 324--339, 2003

    work page 2003

  8. [8]

    Optimal rates for the regularized least-squares algorithm

    Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7: 0 331--368, 2007

    2007

Show all 33 references
  1. [9]

    Dimension free ridge regression

    Chen Cheng and Andrea Montanari. Dimension free ridge regression. The Annals of Statistics, 52 0 (6): 0 2879--2912, 2024

    2024

  2. [10]

    A risk comparison of ordinary least squares vs ridge regression

    Paramveer S Dhillon, Dean P Foster, Sham M Kakade, and Lyle H Ungar. A risk comparison of ordinary least squares vs ridge regression. The Journal of Machine Learning Research, 14 0 (1): 0 1505--1511, 2013

    2013

  3. [11]

    Kernel ridge vs

    Lee H Dicker, Dean P Foster, and Daniel Hsu. Kernel ridge vs. principal component regression: minimax bounds and adaptability of regularization operators. arXiv preprint arXiv:1605.08839, 2016

    2016 arXiv

  4. [12]

    Nonparametric stochastic approximation with large step-sizes

    Aymeric Dieuleveut and Francis Bach. Nonparametric stochastic approximation with large step-sizes . The Annals of Statistics, 44 0 (4): 0 1363 -- 1399, 2016. doi:10.1214/15-AOS1391

    2016 doi

  5. [13]

    The step decay schedule: A near optimal, geometrically decaying learning rate procedure for least squares

    Rong Ge, Sham M Kakade, Rahul Kidambi, and Praneeth Netrapalli. The step decay schedule: A near optimal, geometrically decaying learning rate procedure for least squares. Advances in neural information processing systems, 32, 2019

    2019

  6. [14]

    Estimation with quadratic loss

    William James and Charles Stein. Estimation with quadratic loss. In Breakthroughs in statistics: Foundations and basic theory, pages 443--460. Springer, 1992

    1992

  7. [15]

    Risk and parameter convergence of logistic regression

    Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300, 2018

    2018 arXiv

  8. [16]

    On the saturation effect of kernel ridge regression

    Yicheng Li, Haobo Zhang, and Qian Lin. On the saturation effect of kernel ridge regression. In The Eleventh International Conference on Learning Representations, 2023

    2023

  9. [17]

    Optimal rates for multi-pass stochastic gradient methods

    Junhong Lin and Lorenzo Rosasco. Optimal rates for multi-pass stochastic gradient methods. Journal of Machine Learning Research, 18 0 (97): 0 1--47, 2017

    2017

  10. [18]

    Improved scaling laws in linear regression via data reuse

    Licong Lin, Jingfeng Wu, and Peter L Bartlett. Improved scaling laws in linear regression via data reuse. arXiv preprint arXiv:2506.08415, 2025

    2025

  11. [19]

    A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator

    Theodor Misiakiewicz and Basil Saeed. A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator. arXiv preprint arXiv:2403.08938, 2024

    2024 arXiv

  12. [20]

    Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes

    Loucas Pillaud-Vivien, Alessandro Rudi, and Francis Bach. Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems, 31, 2018

    2018

  13. [21]

    Acceleration of stochastic approximation by averaging

    Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30 0 (4): 0 838--855, 1992

    1992

  14. [22]

    The implicit bias of gradient descent on separable data

    Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19 0 (1): 0 2822--2878, 2018

    2018

  15. [23]

    Prediction, generalization, and complexity: Revisiting the view from classical statistics

    Ryan J Tibshirani. Prediction, generalization, and complexity: Revisiting the view from classical statistics. Lecture Note for Simons MPG Bootcamp, 2024. URL https://www.stat.berkeley.edu/ ryantibs/talks/simons-mpg-2024.pdf

    2024

  16. [24]

    Benign overfitting in ridge regression

    Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning Research, 24 0 (123): 0 1--76, 2023

    2023

  17. [25]

    High-dimensional probability: An introduction with applications in data science, volume 47

    Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

    2018

  18. [26]

    Last iterate risk bounds of sgd with decaying stepsize for overparameterized linear regression

    Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. Last iterate risk bounds of sgd with decaying stepsize for overparameterized linear regression. In International Conference on Machine Learning, pages 24280--24314. PMLR, 2022 a

    2022

  19. [27]

    The power and limitation of pretraining-finetuning for linear regression under covariate shift

    Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. The power and limitation of pretraining-finetuning for linear regression under covariate shift. The 36th Conference on Neural Information Processing Systems, 2022 b

    2022

  20. [28]

    Benefits of early stopping in gradient descent for overparameterized logistic regression

    Jingfeng Wu, Peter L Bartlett, Matus Telgarsky, and Bin Yu. Benefits of early stopping in gradient descent for overparameterized logistic regression. In Forty-second International Conference on Machine Learning, 2025

    2025

  21. [29]

    On early stopping in gradient descent learning

    Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26 0 (2): 0 289--315, 2007

    2007

  22. [30]

    The optimality of (accelerated) sgd for high-dimensional quadratic optimization

    Haihan Zhang, Yuanshi Liu, Qianwen Chen, and Cong Fang. The optimality of (accelerated) sgd for high-dimensional quadratic optimization. arXiv preprint arXiv:2409.09745, 2024

    2024 arXiv

  23. [31]

    The benefits of implicit regularization from sgd in least squares problems

    Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, Dean P Foster, and Sham M Kakade. The benefits of implicit regularization from sgd in least squares problems. Advances in Neural Information Processing Systems, 34: 0 5456--5468, 2021

    2021

  24. [32]

    Risk bounds of multi-pass sgd for least squares in the interpolation regime

    Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. Risk bounds of multi-pass sgd for least squares in the interpolation regime. Advances in Neural Information Processing Systems, 35: 0 12909--12920, 2022

    2022

  25. [33]

    Benign overfitting of constant-stepsize sgd for linear regression

    Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. Benign overfitting of constant-stepsize sgd for linear regression. Journal of Machine Learning Research, 24 0 (326): 0 1--58, 2023

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.