Pith. sign in

REVIEW 3 major objections 3 minor 51 references

Scaling Law for Stochastic Gradient Descent in Quadratically Parameterized Linear Regression

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Quadratic parameterization of linear regression changes SGD's scaling law, yielding strictly faster excess-risk decay than linear SGD when the ground truth opposes the covariance spectrum.

desk verdict Substantial new rates for feature-learning-style SGD, but the quoted minimax bound and the paper's own upper rate contradict in the α≤β regime – that conflict needs to be resolved before the optimality claim stands. read the letter →

arxiv 2502.09106 v1 pith:3QIVCRNO submitted 2025-02-13 cs.LG

Shihong Ding , Haihan Zhang , Hanzhen Zhao , Cong Fang This is my paper · ORCID
classification cs.LG MSC 68T0762J0568Q32
keywords scalinglawstochasticgradientdescentquadraticparameterizationfeaturelearningimplicitregularizationspectraldecayexcessriskeffectivedimension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper derives a scaling law for stochastic gradient descent on a quadratically parameterized linear regression, where the predictor is $\langle x, v^{\odot 2}\rangle$ and both the covariance spectrum and the ground truth decay as power laws. It establishes an upper bound on the excess risk that is piecewise in the effective dimension $D = \min\{T^{1/\max\{\beta,(\alpha+\beta)/2\}}, M\}$, with approximation, variance, and bias terms. The central finding is a separation: when the ground truth opposes the covariance spectrum ($\alpha > \beta$), the quadratic model achieves $\tilde{O}(T^{-(2\beta-2)/(\alpha+\beta)})$, strictly faster than the best linear SGD rate $\tilde{O}(T^{-(\beta-1)/\alpha})$; when $\alpha \le \beta$, it matches the optimal rate $\tilde{O}(T^{-(\beta-1)/\beta})$. The paper argues the speedup comes from feature learning: coordinates with strong signal are adapted in an initial phase, after which estimation behaves like linear SGD on re-scaled features.

What carries the argument

The proof decomposes SGD into two phases. In Phase I (adaptation), the dynamics select an effective dimension $D$ without explicit thresholding: coordinates $1,\ldots,D$ climb to within a constant factor of $v^*$, while coordinates beyond $D$ stay bounded by a constant multiple of $v^*$. This is shown through a b-capped coupling sequence and coordinate-wise sub-Gaussian supermartingales. In Phase II (estimation), iterates remain confined near $v^*$, and the update is approximated as linear SGD on reparameterized features $\Pi_M x \odot v^*_{1:M}$, whose per-coordinate rescaling by $v^*$ accelerates bias decay. The geometrically decaying step size and an auxiliary truncated sequence carry the linear-regression analysis through the nonlinear phase.

What would settle it

Generate data with the same power-law spectra but with correlated Gaussian covariates (for example, a fixed Toeplitz coupling) or heavy-tailed coordinates, run Algorithm 1, and measure the excess-risk exponent: a rate significantly worse than $\tilde{O}(T^{-(2\beta-2)/(\alpha+\beta)})$ in the $\alpha > \beta$ regime would show the scaling law depends on coordinate-wise independence.

Watch

Extended reading notes

Core claim

The paper's central claim is that the quadratic parameterization itself acts as an implicit regularizer that adapts per-coordinate learning rates, and this adaptation changes the SGD scaling law. Concretely, with eigenvalues $\lambda_i \asymp i^{-\alpha}$ and ground-truth alignment $\lambda_i (v^*_i)^4 \asymp i^{-\beta}$, the last-iterate excess risk of SGD with a warm-up constant step size followed by geometric decay is at most $$R_M(v_T) - \mathbb{E}[\$xi^{2}$] \lesssim $M^{{-(\beta-1)}}$ + \frac{\$sigma^{2}$ D}{T} + \frac{D}{T} + $D^{{-(\beta-1)}}$ \mathbf{1}_{D<M},$$ with $D = \min\{T^{1/\max\{\beta,(\alpha+\beta)/2\}}, M\}$. In the large-model regime this yields $\tilde{O}(T^{-(\beta-1)/\beta})$ when $\beta \ge \alpha$ and $\tilde{O}(T^{-(2\beta-2)/(\alpha+\beta)})$ when $\alpha > \beta$. The $\alpha > \beta$ rate is strictly faster than the best rate the paper quotes for linear SGD, $\tilde{O}(T^{-(\beta-1)/\alpha})$. The paper also proves a matching algorithmic lower bound for this SGD algorithm, showing the rate is intrinsic to the method, and notes that both quadratic and linear SGD miss the information-theoretic bound when $\alpha > \beta$.

Load-bearing premise

The load-bearing assumption is that the infinitely many covariates are independent Gaussians with variances $\lambda_i$; if covariates are correlated or non-Gaussian, the concentration estimates and the two-phase decomposition in the proof no longer go through.

Editorial extensions

If this is right

  • In the aligned regime $\beta \ge \alpha$, quadratic SGD reaches the same optimal rate as linear SGD, so feature learning does not degrade worst-case scaling.
  • In the opposed regime $\alpha > \beta$, the quadratic model's rate $\tilde{O}(T^{-(2\beta-2)/(\alpha+\beta)})$ is strictly faster than the linear rate $\tilde{O}(T^{-(\beta-1)/\alpha})$, yielding an explicit separation between models with and without feature learning.
  • The excess risk is governed by $D = \min\{T^{1/\max\{\beta,(\alpha+\beta)/2\}}, M\}$: for fixed $T$, adding parameters helps only until $D$ saturates, after which the approximation term $M^{-(\beta-1)}$ dominates.
  • For a compute budget $B=MT$, the optimal allocation follows $M \asymp B^{1/(1+\beta)}$, $T \asymp B^{\beta/(1+\beta)}$ when $\beta \ge \alpha$, and $M \asymp B^{1/(1+(\alpha+\beta)/2)}$, $T \asymp B^{(\alpha+\beta)/2/(1+(\alpha+\beta)/2)}$ when $\alpha > \beta$.
  • The matching algorithmic lower bound implies the quoted rates are intrinsic to this SGD algorithm, and in the $\alpha > \beta$ regime the algorithm remains above the information-theoretic bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension — If the two-phase mechanism is generic, other positive-homogeneous parameterizations (for example cubing or matrix factorization) should show a similar acceleration with exponents set by the degree of the parameterization; this is directly testable in the paper's synthetic setup.
  • Extension — The effective dimension $D$ emerges from the dynamics rather than from an explicit threshold, suggesting a principled early-stopping proxy: stop once the learned coordinates have plateaued near their $v^*$-neighborhoods; the paper does not explore this.
  • Extension — The algorithmic lower bound's slow-ascent argument implies the gap to the information-theoretic bound in the $\alpha > \beta$ regime is an optimization artifact, so preconditioned or momentum variants of SGD might close part of that gap.
  • Extension — The compute-optimal allocation implies that in opposed-spectrum problems data, not model size, is the bottleneck; practitioners should favor more samples over wider models in such regimes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper studies SGD for a quadratically parameterized linear regression model f(x)=<x,v^{\odot 2}> under infinite-dimensional Gaussian covariates whose covariance eigenvalues decay as \lambda_i\asymp i^{-\alpha} and whose ground truth satisfies \lambda_i(v^*_i)^4\asymp i^{-\beta}. It proposes a two-phase analysis of SGD with a warm-up plus tail-geometric step-size schedule, proving an upper bound on the last-iterate excess risk in Theorem 4.1 with an effective dimension D, and deriving Corollaries 4.2 and 4.3 for the large- and small-model regimes. The paper claims that when \alpha\le\beta the quadratic model achieves the information-theoretic rate and that when \alpha>\beta it strictly outperforms the linear-SGD rate quoted from prior work. An algorithmic lower bound matching the upper bound is stated in Appendix B, and simulations are provided in Appendix D.

Significance. If the proof is correct, the result is valuable: it gives a nontrivial two-phase convergence analysis for a simple feature-learning model with anisotropic spectra, and it provides a concrete rate separation between linear and quadratically parameterized SGD. The appendix contains explicit coupling constructions, submartingale/supermartingale arguments, and a matching algorithmic lower bound, which are substantive technical contributions. However, the central optimality claim is currently undermined by an internal contradiction between the paper's own upper bound and the information-theoretic lower bound quoted in Remark 4.4, so the rates cannot be accepted as stated without revision.

major comments (3)
  1. [§4, Remark 4.4 and Corollary 4.2] There is a direct internal contradiction in the claimed optimality. For \beta\ge\alpha, Corollary 4.2 and Theorem 4.1 give excess risk \tilde{O}((\sigma^2+1)/T^{(\beta-1)/\beta}), while Remark 4.4 quotes the information-theoretic lower bound as T^{-1/\beta} for any algorithm. For every \beta>2, (\beta-1)/\beta>1/\beta, so the upper bound is polynomially faster than the quoted lower bound; this cannot be reconciled by constants or logarithmic factors. Since the same remark concludes that SGD 'hits the lower bound' in the \alpha\le\beta regime, either the proof of Theorem 4.1 contains an error in the Phase II bias/variance decomposition or the lower bound quoted in Remark 4.4 is misstated or misapplied. The authors must correct one of these statements and then re-derive the optimality and separation conclusions.
  2. [Appendix B, Theorem B.1] The algorithmic lower bound in Theorem B.1 has the same polynomial shape as the upper bound, namely 1/M^{\beta-1}+\bar{\sigma}^2D/T+D^{-(\beta-1)}\mathbf{1}_{M>D}. When D\asymp T^{1/\beta}, this yields \Omega(T^{-(\beta-1)/\beta}), which is the exponent of the paper's own upper bound, not the T^{-1/\beta} quoted in Remark 4.4. Therefore this appendix cannot validate the claim that the upper bound hits the information-theoretic lower bound. The manuscript needs to distinguish clearly between the algorithmic lower bound, which matches the SGD trajectory analysis, and the minimax/information-theoretic lower bound, and it needs to identify which evidence supports the optimality assertion in the \alpha\le\beta regime.
  3. [§3.3 and Appendix C] The entire probabilistic engine relies on Assumption 3.1 [A1]: the covariates are independent Gaussian coordinates. Lemmas C.3, C.4, C.9 and the b-capped and neighbor coupling constructions in Appendix A use Gaussian fourth moments and coordinate-wise independence. No result is proved for correlated or non-Gaussian covariates, despite the broad framing in the Introduction and Abstract about 'feature learning' and 'scaling law'. This is not by itself an error, because the theorem states the assumption explicitly, but the scope limitation should be stated as prominently as the rate results, and Remark 3.2 should not imply that the extension is routine.
minor comments (3)
  1. [Algorithm 1 and Theorem 4.1] The algorithm's initialization v_0=\Omega(\min\{1,M^{-(\beta-\alpha)/4}\})\mathbf{1}_M and the step size \eta\asymp D^{\min\{0,(\alpha-\beta)/4\}} depend on the unknown exponents \alpha,\beta and on D. The text should qualify statements such as 'automatically adapt' and 'without explicitly selecting D' so that readers understand the adaptation is along the trajectory, not parameter-free.
  2. [Appendix B, Theorem B.1] The displayed condition 'h>\lceil(T-h)/\log(T-h)\rceil' in Theorem B.1 is inconsistent with the earlier definitions h=\lceil T/\log T\rceil and T_1=\lfloor(T-h)/\log(T-h)\rfloor; please correct this typo and state the exact relation between h and T_1 used in the lower bound proof.
  3. [Appendix D, Simulations] The simulation figures are visually consistent with the claimed rates, but the text reports no fitted exponents or error bars. Adding fitted slopes with confidence intervals for the quadratic and linear curves would make the empirical support for the scaling-law exponents quantitative.

Circularity Check

1 steps flagged · score 4.0 of 10

The upper-bound derivation is self-contained, but the advertised optimality and linear-model separation claims rest on a self-cited preprint whose quoted lower bound contradicts the paper's own Corollary 4.2.

  1. self citation load bearing [Remark 4.4; also Section 1 comparison with linear SGD]
    "Under the covariate distribution in Assumptions 3.1 and 3.3, and with the true parameter set meeting Assumption 3.3 regularity constraints, prior work (Zhang, Liu, et al., 2024) established the info-theoretic lower bound as T^{-1/β}. Our analysis shows two distinct regimes: When α ≤ β, SGD in linear and quadratic models hits the lower bound, proving statistical optimality."

    The 'proving statistical optimality' claim is not established in this paper: the only information-theoretic lower bound is imported from Zhang, Liu, Chen, and Fang (2024), a preprint whose author list overlaps with the present paper (Haihan Zhang and Cong Fang). Appendix B's Theorem B.1 lower-bounds Algorithm 1 only, not all algorithms, so the 'agnostic to parametrization method and algorithm' lower bound is entirely external to the proofs here. The citation is load-bearing for the headline optimality claim and is internally problematic: Corollary 4.2 states the upper bound T^{-(β−1)/β} in the α≤β regime, which is faster than the quoted T^{-1/β} for every β>2, so the paper's own equations cannot simultaneously support the quoted lower bound and the claimed 'hits' statement.

full rationale

The derivation of Theorem 4.1 is technically self-contained: the step-size η and effective dimension D are selected from the assumed exponents α,β rather than fitted to simulated outcomes, the Phase I/II proof decomposes the risk into approximation, variance, and bias terms, and the algorithmic lower bound in Appendix B is proved with coupling and submartingale arguments. I found no place where a quantity is defined in terms of the target rate or where a fitted parameter is renamed as a prediction. The only circularity is the load-bearing use of the coauthored preprint for the information-theoretic lower bound and the linear-SGD comparator; that citation is not machine-checked or otherwise independently verified here, and Remark 4.4's quoted lower bound conflicts with the paper's own Corollary 4.2 for β>2, so the 'hits the lower bound' conclusion cannot be taken as established by the manuscript. Under the rubric, this is a self-citation that is load-bearing for the optimality/separation claims while the main upper-bound analysis retains independent content, corresponding to a score of 4 rather than a higher construction-level circularity score. Correctness concerns about Gaussian independence and the quoted minimax rate are noted as risks but are not themselves circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central bounds rest on four distributional assumptions (independent Gaussian covariates, sub-Gaussian noise, power-law spectrum, power-law source condition) plus algorithm hyperparameters chosen with knowledge of α and β. No new physical or mathematical entities are introduced; the effective dimension D is an analysis construct rather than an invented entity.

free parameters (3)
  • initialization scale v0 = Ω(min{1, M^{-(β-α)/4}})
    Algorithm 1 sets v0 = Ω(min{1, M^{-(β-α)/4}}) 1_M; the proof needs v0 small enough to avoid escaping the basin in Phase I and large enough for the lower-bound argument. The choice depends on the unknown exponents α and β.
  • initial step size η = ≍ D^{min{0,(α-β)/4}}
    Theorem 4.1 and Appendix Theorem A.19 require η in a range depending on D, σ², M²(b), and spectral quantities. The paper gives only the scaling form, with hidden constants, and the algorithm needs this knowledge to achieve the stated bound.
  • effective dimension D = min{T^{1/max{β,(α+β)/2}}, M}
    D is chosen by the analysts to balance the bias and variance terms in the bound; it is not estimated from data. It is an analysis parameter that carries the piecewise structure of the main theorem.
assumptions (5)
  • domain assumption Assumption 3.1 [A1]: the covariates xi are independent Gaussians with variance λi.
    Section 3.3. This assumption underlies Lemmas C.3, C.4, and C.9 and the high-probability coupling in Appendix A; the authors note it is stronger than a low-correlation/RIP condition and say weakening it is future work.
  • domain assumption Assumption 3.1 [A2]: the noise ξ is zero-mean sub-Gaussian with parameter σξ.
    Section 3.3. Used in the event-based concentration inequalities and in the variance term σ²D/T.
  • domain assumption Assumption 3.3 [A3]: the covariance eigenvalues satisfy λi ≍ i^{-α} for α > 1.
    Section 3.3. The power-law spectrum fixes the decay of the covariance and enters every bound through D and the effective dimension.
  • domain assumption Assumption 3.3 [A4]: the ground truth satisfies λi (v*_i)^4 ≍ i^{-β} for β > 1.
    Section 3.3. This source condition sets the decay of the target and determines the piecewise rates in Corollary 4.2.
  • domain assumption Well-specified target: y = ⟨x, v^{*⊙2}⟩ + ξ with access only to the top M projected covariates.
    Section 3.2. The risk decomposition RM(v) - E[ξ²] and the approximation term 1/M^{β-1} rely on the response being generated exactly by the quadratic parameterization up to additive noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Law for Stochastic Gradient Descent in Quadratically Parameterized Linear Regression." pith.science (2026). https://pith.science/paper/3QIVCRNO

@misc{pith2026250209106,
  author       = {Pith},
  title        = {Pith review of: Scaling Law for Stochastic Gradient Descent in Quadratically Parameterized Linear Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3QIVCRNO}},
  note         = {Machine review of arXiv:2502.09106}
}
read the original abstract

In machine learning, the scaling law describes how the model performance improves with the model and data size scaling up. From a learning theory perspective, this class of results establishes upper and lower generalization bounds for a specific learning algorithm. Here, the exact algorithm running using a specific model parameterization often offers a crucial implicit regularization effect, leading to good generalization. To characterize the scaling law, previous theoretical studies mainly focus on linear models, whereas, feature learning, a notable process that contributes to the remarkable empirical success of neural networks, is regretfully vacant. This paper studies the scaling law over a linear regression with the model being quadratically parameterized. We consider infinitely dimensional data and slope ground truth, both signals exhibiting certain power-law decay rates. We study convergence rates for Stochastic Gradient Descent and demonstrate the learning rates for variables will automatically adapt to the ground truth. As a result, in the canonical linear regression, we provide explicit separations for generalization curves between SGD with and without feature learning, and the information-theoretical lower bound that is agnostic to parametrization method and the algorithm. Our analysis for decaying ground truth provides a new characterization for the learning dynamic of the model.

Figures

Figures reproduced from arXiv: 2502.09106 by the authors.

Figure 1
Figure 1. Empirical results on the convergence rate of quadratic model with spectral decay v.s. traditional [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Numerical simulation results. • Figure D (e): We compare the curve of mean error of SGD against the number of sample size for quadratic models with model size M = 10, 30, 50, 100, 200, under the setting α = 3, β = 2 and T ranging from 1 to 10000. The results show that for a fixed M, when T is small, the convergence rate approaches the rate observed as M → ∞. As T increases sufficiently, the convergence rate stabiliz… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 42 canonical work pages

  1. [1]

    Ali, A., Dobriban, E., & Tibshirani, R. (2020). The implicit regularization of stochastic gradient flow for least squares. In International conference on machine learning

    work page 2020

  2. [2]

    S., Hu, W., Li, Z., Salakhutdinov, R

    Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., & Wang, R. (2019). On exact computation with an infinitely wide neural net. In Neural information processing systems

    work page 2019

  3. [3]

    A., Suzuki, T., Wang, Z., Wu, D., & Yang, G

    Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., & Yang, G. (2022). High-dimensional asymptotics of feature learning: How one gradient step improves the representation. In Neural information processing systems

    work page 2022

  4. [4]

    Bach, F., & Moulines, E. (2013). Non-strongly-convex smooth stochastic approximation with convergence rate o (1/n). In Neural information processing systems

    work page 2013

  5. [5]

    Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2024). Explaining neural scaling laws. Proceedings of the National Academy of Sciences , 121 (27), e2311878121

    work page 2024

  6. [6]

    L., Long, P

    Bartlett, P. L., Long, P. M., Lugosi, G., & Tsigler, A. (2020). Benign overfitting in linear regression. Proceedings of the National Academy of Sciences

    work page 2020

  7. [7]

    L., Montanari, A., & Rakhlin, A

    Bartlett, P. L., Montanari, A., & Rakhlin, A. (2021). Deep learning: a statistical viewpoint. Acta numerica

    work page 2021

  8. [8]

    Bietti, A., & Bach, F. (2021). Deep equals shallow for reLU networks in kernel regimes. In International conference on learning representations. 39

    work page 2021

Show all 51 references
  1. [9]

    Bietti, A., & Mairal, J. (2019). On the inductive bias of neural tangent kernels. In Neural information processing systems

    2019

  2. [10]

    Bordelon, B., Atanasov, A., & Pehlevan, C. (2024). A dynamical model of neural scaling laws. In International conference on machine learning

    2024

  3. [11]

    Bordelon, B., & Pehlevan, C. (2021). Learning curves for SGD on structured features. arXiv preprint arXiv:2106.02713

    2021 arXiv

  4. [12]

    Bubeck, S., et al. (2015). Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning

    2015

  5. [13]

    T., & Hall, P

    Cai, T. T., & Hall, P. (2006). Prediction in functional linear regression. The Annals of Statistics

    2006

  6. [14]

    J., & Tao, T

    Candes, E. J., & Tao, T. (2005). Decoding by linear programming. IEEE transactions on information theory

    2005

  7. [15]

    Caponnetto, A., & De Vito, E. (2007). Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics

    2007

  8. [16]

    Chizat, L., & Bach, F. (2018). On the global convergence of gradient descent for over-parameterized models using optimal transport. In Neural information processing systems

    2018

  9. [17]

    Damian, A., Lee, J., & Soltanolkotabi, M. (2022). Neural networks can learn representations with gradient descent. In Conference on learning theory

    2022

  10. [18]

    Dieuleveut, A., & Bach, F. (2016). Nonparametric stochastic approximation with large step-sizes. The Annals of Statistics

    2016

  11. [19]

    Dieuleveut, A., Flammarion, N., & Bach, F. (2017). Harder, better, faster, stronger convergence rates for least-squares regression. Journal of Machine Learning Research

    2017

  12. [20]

    M., Kidambi, R., & Netrapalli, P

    Ge, R., Kakade, S. M., Kidambi, R., & Netrapalli, P. (2019). The step decay schedule: A near optimal, geometrically decaying learning rate procedure for least squares. In Neural information processing systems

    2019

  13. [21]

    Ghorbani, B., Mei, S., Misiakiewicz, T., & Montanari, A. (2021). Linearized two-layers neural networks in high dimension. The Annals of Statistics

    2021

  14. [22]

    Goyal, P. (2017). Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677

    2017 arXiv

  15. [23]

    E., Bhojanapalli, S., Neyshabur, B., & Srebro, N

    Gunasekar, S., Woodworth, B. E., Bhojanapalli, S., Neyshabur, B., & Srebro, N. (2017). Implicit regularization in matrix factorization. In Neural information processing systems

    2017

  16. [24]

    Z., Wei, C., Lee, J., & Ma, T

    HaoChen, J. Z., Wei, C., Lee, J., & Ma, T. (2021). Shape matters: Understanding the implicit bias of the noise covariance. In Conference on learning theory

    2021

  17. [25]

    Hastie, T., Montanari, A., Rosset, S., & Tibshirani, R. J. (2022). Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics

    2022

  18. [26]

    Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. In Neural information processing systems

    2018

  19. [27]

    M., Kidambi, R., Netrapalli, P., & Sidford, A

    Jain, P., Kakade, S. M., Kidambi, R., Netrapalli, P., & Sidford, A. (2018). Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of machine learning research

    2018

  20. [28]

    B., Chess, B., Child, R.,

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., . . . Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  21. [29]

    LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. nature, 521 (7553), 436–444

    2015

  22. [30]

    D., Oko, K., Suzuki, T., & Wu, D

    Lee, J. D., Oko, K., Suzuki, T., & Wu, D. (2024). Neural network learns low-dimensional polynomials with sgd near the information-theoretic limit. arXiv preprint arXiv:2406.01581

    2024 arXiv

  23. [31]

    S., Hu, W., Salakhutdinov, R., & Arora, S

    Li, Z., Wang, R., Yu, D., Du, S. S., Hu, W., Salakhutdinov, R., & Arora, S. (2019). Enhanced convolutional neural tangent kernels. arXiv preprint arXiv:1911.00809

    2019 arXiv

  24. [32]

    Liang, T., Rakhlin, A., & Zhai, X. (2020). On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels. In Conference on learning theory

    2020

  25. [33]

    Lin, J., & Rosasco, L. (2017). Optimal rates for multi-pass stochastic gradient methods. Journal of Machine Learning Research

    2017

  26. [34]

    M., Bartlett, P., & Lee, J

    Lin, L., Wu, J., Kakade, S. M., Bartlett, P., & Lee, J. D. (2024). Scaling laws in linear regression: Compute, parameters, and data. In Neural information processing systems

    2024

  27. [35]

    Lu, W., Zhang, H., Li, Y., Xu, M., & Lin, Q. (2023). Optimal rate of kernel regression in large dimensions. arXiv preprint arXiv:2309.04268

    2023 arXiv

  28. [36]

    Mei, S., & Montanari, A. (2022). The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics

    2022

  29. [37]

    Mei, S., Montanari, A., & Nguyen, P.-M. (2018). A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences . 40

    2018

  30. [38]

    Mousavi-Hosseini, A., Park, S., Girotti, M., Mitliagkas, I., & Erdogdu, M. A. (2022). Neural networks efficiently learn low-dimensional representations with sgd. arXiv preprint arXiv:2209.14863

    2022 arXiv

  31. [39]

    Natarajan, B. K. (1995). Sparse approximate solutions to linear systems. SIAM journal on computing

    1995

  32. [40]

    J., & Yu, B

    Raskutti, G., Wainwright, M. J., & Yu, B. (2014). Early stopping and non-parametric regression: an optimal data-dependent stopping rule. The Journal of Machine Learning Research

    2014

  33. [41]

    Tsigler, A., & Bartlett, P. L. (2023). Benign overfitting in ridge regression. Journal of Machine Learning Research

    2023

  34. [42]

    V., Pillaud-Vivien, L., & Flammarion, N

    Varre, A. V., Pillaud-Vivien, L., & Flammarion, N. (2021). Last iterate convergence of SGD for least-squares in the interpolation regime. In Neural information processing systems

    2021

  35. [43]

    Vaskevicius, T., Kanade, V., & Rebeschini, P. (2019). Implicit regularization for optimal sparse recovery. In Neural information processing systems

    2019

  36. [44]

    M., & Erdogdu, M

    Vural, N. M., & Erdogdu, M. A. (2024). Pruning is optimal for learning sparse features in high-dimensions. arXiv preprint arXiv:2406.08658

    2024 arXiv

  37. [45]

    Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint . Cambridge university press

    2019

  38. [46]

    D., Moroshko, E., Savarese, P., Golan, I.,

    Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., . . . Srebro, N. (2020). Kernel and rich regimes in overparametrized models. In Conference on learning theory

    2020

  39. [47]

    Wu, J., Zou, D., Braverman, V., Gu, Q., & Kakade, S. (2022). Last iterate risk bounds of SGD with decaying stepsize for overparameterized linear regression. In International conference on machine learning

    2022

  40. [48]

    Zhang, H., Li, Y., Lu, W., & Lin, Q. (2024). Optimal rates of kernel ridge regression under source condition in large dimensions. arXiv preprint arXiv:2401.01270

    2024 arXiv

  41. [49]

    Zhang, H., Liu, Y., Chen, Q., & Fang, C. (2024). The optimality of (accelerated) SGD for high-dimensional quadratic optimization. arXiv preprint arXiv:2409.09745

    2024 arXiv

  42. [50]

    P., & Kakade, S

    Zou, D., Wu, J., Braverman, V., Gu, Q., Foster, D. P., & Kakade, S. (2021). The benefits of implicit regularization from SGD in least squares problems. In Neural information processing systems

    2021

  43. [51]

    Zou, D., Wu, J., Braverman, V., Gu, Q., & Kakade, S. (2021). Benign overfitting of constant-stepsize SGD for linear regression. In Conference on learning theory. 41

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.