REVIEW 3 cited by
More Data Can Hurt for Linear Regression: Sample-wise Double Descent
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In this expository note we describe a surprising phenomenon in overparameterized linear regression, where the dimension exceeds the number of samples: there is a regime where the test risk of the estimator found by gradient descent increases with additional samples. In other words, more data actually hurts the estimator. This behavior is implicit in a recent line of theoretical works analyzing "double-descent" phenomenon in linear models. In this note, we isolate and understand this behavior in an extremely simple setting: linear regression with isotropic Gaussian covariates. In particular, this occurs due to an unconventional type of bias-variance tradeoff in the overparameterized regime: the bias decreases with more samples, but variance increases.
Forward citations
Cited by 3 Pith papers
-
Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics
Under Gaussian design with n ≍ d, the empirical distribution of leave-one-out influences for convex M-estimators converges to the pushforward of a four-dimensional Gaussian through an explicit nonlinear map built from...
-
The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classification
The saturation index S(K)=erank(pooled within-class covariance)/K tracks when few-shot labels become redundant for fixed linear probes, with within-task median Spearman ρ≈0.81 on 17 binary tasks.
-
Transformers Don't In-Context Learn Least Squares Regression
In-context regression transformers do not approximate OLS: they underperform it even in-distribution, fail on out-of-subspace prompts, and their failures correlate with a low-rank spectral signature in the residual stream.
Discussion (0). Sign in to comment.