Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining

Christos Thrampoulidis; Lalitha Sankar; Nathan Stromberg

REVIEW 1 major objections 4 minor 30 references

Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining

T0 review · 1 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Weighting the minority class in last layer retraining is effective, and the optimal weight grows with overparameterization.

desk verdict The equal-error weight in Theorem 2 is a real contribution, but the 'optimal' claim in Theorem 3 is overbroad and needs a separation condition. read the letter →

arxiv 2506.20025 v1 pith:QRKUOLMS submitted 2025-06-24 cs.LG stat.ML

Nathan Stromberg , Christos Thrampoulidis , Lalitha Sankar This is my paper · ORCID

classification cs.LGstat.ML MSC 62H3068T0768T10

keywords lastlayerretraininglossweightingworst-classerrorclassimbalanceoverparameterizationconvexGaussianminimaxtheoremsquareeffectivedimension

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies last layer retraining (LLR), where a pretrained model is frozen and only its final linear layer is refit on new data. In the regime where the number of retraining samples $n$ and the latent dimension $d$ are comparable (ratio $\delta=d/n$ in $(0,1)$), the paper argues that class-weighted empirical risk minimization is effective, unlike in the separable overparameterized regime where weighting has no effect. For square loss on class-conditional Gaussian data, it derives the asymptotically optimal minority-class weight $\tilde{\rho}=\pi_-/\pi_+ + (\pi_-/\pi_+-1)\delta/(2\pi_+-\delta)$, which equalizes per-class errors and, under a monotonicity assumption, minimizes worst-class error. This weight is larger than the classical ratio of priors, and the gap grows with $\delta$. In practice, using an effective latent dimension makes the formula outperform ratio-of-priors weighting on image-retraining tasks.

What carries the argument

The central machinery is a reduction of the $(d+1)$-dimensional weighted ERM problem to four scalar equations in $(\alpha,\gamma,\lambda,b)$, obtained through the convex Gaussian minimax theorem and expressed with Moreau envelopes of the loss. For square loss the Moreau envelope is $M(x;\lambda)=(x-1)^2/(2(1+\lambda))$, which turns the system into algebraic equations. The key object derived from that system is $\tilde{\rho}$: the weight that zeroes the bias, thereby equalizing the two class risks.

What would settle it

Run the Gaussian experiment of Section 3 with strong separation ($s=4$), $\delta<2\pi_+$, plot per-class errors against $\rho$ over a fine grid, and locate the weight minimizing worst-class error; if it is not $\tilde{\rho}$, the monotonicity assumption behind Theorem 3 is false in that regime. A direct calculation of $dR_+/d\rho$ and $dR_-/d\rho$ at that setting would settle the matter without simulation.

Watch

Extended reading notes

Core claim

Under proportional asymptotics $d,n\to\infty$ with $d/n\to\delta\in(0,1)$, the paper reduces weighted square-loss ERM on a Gaussian mixture to a closed-form system, and identifies the weight $\tilde{\rho}=\pi_-/\pi_+ + (\pi_-/\pi_+-1)\delta/(2\pi_+-\delta)$ as the unique weight that drives the asymptotic bias to zero, $b^*=0$. At this weight the per-class risks are equal, so worst-class error becomes $Q(s^2\sqrt{1-\Delta}/(\sqrt{\Delta}+s^2))$ with $\Delta=\delta/(4\pi_+)+\delta/(4\pi_-)$. The paper claims this weight is optimal for worst-class error provided the per-class risks are monotone in $\rho$, and that it strictly outperforms both ratio-of-priors weighting and downsampling for $\delta>0$. The formula reduces to the classical ratio of priors as $\delta\to 0$, and it diverges as $\delta$ approaches $2\pi_+$ from below, marking the point where no finite weight can equalize the classes.

Load-bearing premise

The claim that $\tilde{\rho}$ actually minimizes worst-class error depends on the unproven assumption that the minority-class error falls and the majority-class error rises monotonically as the minority weight increases; when class separation is large, the paper's own simulations show the worst-class minimizer lies below $\tilde{\rho}$, so that assumption fails there.

Editorial extensions

If this is right

As the ratio $\delta=d/n$ grows within $(0,1)$, the optimal minority weight rises above $\pi_-/\pi_+$; using the classical ratio underweights the minority in small-sample retraining.
When $\delta\ge 2\pi_+$, no finite weight equalizes the classes; the minority class keeps dominating worst-class error and the recommended strategy is to upweight it without bound.
Downsampling the majority class in this regime is equivalent to increasing $\delta$ to $\delta/(2\pi_+)$, so weighted ERM with $\tilde{\rho}$ achieves lower worst-class error than downsampling at the same data size.
On real image-retraining tasks, the practical weight should be computed from an effective latent dimension (the number of principal components capturing most variance), not from the nominal last-layer width.
In the population limit $\delta\to 0$, $\tilde{\rho}$ reduces to $\pi_-/\pi_+$, recovering the classical ratio-of-priors rule.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The monotonicity assumption behind optimality can be tested directly from the system (8): if $dR_+/d\rho$ changes sign for large signal strength $s$, the optimal-weight claim fails there, and a corrected weight would be the actual minimizer of the maximum of the two risks.
The same scalar-system approach likely applies to other convex losses, but only square loss gives a closed-form $\tilde{\rho}$; for other losses one would need to solve the four equations numerically, so the simple formula may not survive.
A practical adaptive rule could estimate the effective dimension from the retraining data itself rather than by a PCA threshold, making the weighting self-tuning and removing the main heuristic in the paper's real-data recipe.
The results suggest a general principle: optimal cost-sensitive corrections depend on the ratio of trainable parameters to retraining samples, so any method that reports a single best weight for a dataset should be expected to change with sample size.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The thing worth knowing up front: the δ-dependent equal-error weight ρ̃ in Theorem 2 is a genuine new result, and the CGMT derivation is competently executed. But the 'optimal' claim in Theorem 3 is not safe as stated; the paper's own formulas violate the required monotonicity in a regime the theorem allows, so ρ̃ should be called an equal-error weight, not the worst-class minimizer, until that's fixed.

What's actually new and good. The paper studies weighted ERM for last-layer retraining in the proportional regime δ=d/n∈(0,1), which sits between the population limit and the separable overparameterized limit. Theorem 1 reduces weighted ERM to four scalar equations via CGMT; for square loss, Corollary 1 gives a concrete system, and Theorem 2 solves it in closed form for the weight ρ̃ = π−/π+ + (π−/π+−1)δ/(2π+−δ) that makes b*=0 and equalizes the class errors. That closed form is, as far as I can tell from the cited literature, new, and it is internally consistent: the WCE expression Q(s²√(1−Δ)/(√Δ+s²)) follows. The simulation matches across δ, and the comparison with downsampling is informative. The real-data experiments are honest, including the admission that the PCA effective dimension is a heuristic.

Where it gets soft. Theorem 3 claims ρ̃ minimizes WCE under the assumption that R+ is decreasing and R− is increasing in ρ. That assumption is load-bearing, not proven, and in fact the paper's own asymptotic formulas contradict it for large signal separation. Working from Corollary 2 and Theorem 2 at δ=0.2, π+=0.2, s=4, the unweighted minority-class argument is about 3.33 and the weighted equal-error argument at ρ̃=7 is about 3.27, so R+(1)

Minor issues: no code is provided, and the effective-dimension estimation is ad hoc. Neither undercuts the theory, but they matter for a practical recipe.

Bottom line: this deserves a serious referee. The Theorem 2 result is useful and new; the optimality claim is fixable with a qualified statement. I'd cite the equal-error weight and would want to see a revision that fixes Theorem 3.

Referee Report

1 major / 4 minor

Summary. This paper analyzes weighted empirical risk minimization (wERM) for last-layer retraining in the proportional asymptotic regime d/n = δ ∈ (0,1), under a class-conditional Gaussian model. Using the convex Gaussian minimax theorem, the authors reduce the (d+1)-dimensional optimization to a scalar system for general convex losses (Theorem 1), specialize to square loss (Corollary 1), and derive closed-form solutions for unweighted ERM (Corollary 2) and for the weight ρ̃ that equalizes per-class errors (Theorem 2). Theorem 3 claims that ρ̃ minimizes worst-class error under a monotonicity assumption on the per-class risks. The paper compares wERM with downsampling and reports experiments on CelebA and CIFAR-10 in which the weight predicted from a PCA-based effective dimension outperforms the classical ratio of priors.

Significance. The equal-error characterization is a useful contribution: it bridges the population and separable-overparameterized regimes and predicts that the balancing weight must grow with δ, an effect confirmed by the synthetic simulations and qualitatively by the vision experiments. The CGMT reduction and the closed-form calculations are careful, the internal algebra in Corollary 2 and Theorem 2 is consistent, and the paper makes a falsifiable prediction about the dependence of the optimal weight on overparameterization. I also note that ρ̃ is derived from the generative model and optimality conditions rather than fitted to empirical crossover points, so the comparisons in Figures 2 and 6 are not circular. The practical demonstrations are heuristic but support the qualitative message. However, the central optimality claim is currently overbroad: it relies on an unproven and, by the paper's own asymptotics, false monotonicity assumption in a regime the theorem allows, so the word 'optimal' in the abstract and contributions needs to be scoped down or supported by an additional proof.

major comments (1)

[§3.1 (Theorem 3) and Appendix A.6, Eq. (52), Fig. 8b] The optimality claim for ρ̃ is load-bearing for the paper's main message, but Theorem 3's monotonicity hypothesis is not proven and is contradicted by the paper's own closed-form formulas. For δ=0.2, π+=0.2, s=4, which satisfies δ<2π+=0.4, Corollary 2 and Theorem 2 give unweighted minority-class argument (γ*s+b*)/α* ≈ 3.33 and weighted equal-error argument γ̃s/α̃ ≈ 3.28. Since Q is decreasing and b*<0, this gives WCE(ρ=1)=R+(1)=Q(3.33) < Q(3.28)=R+(ρ̃)=WCE(ρ̃); hence ρ̃ is not the WCE minimizer at this allowed parameter point, and R+ is not monotonically decreasing on [1,ρ̃]. This is exactly the behavior shown in Appendix A.6 and Figure 8b, and it is captured quantitatively by Eq. (52). The theorem as a conditional statement is valid, but the surrounding text—abstract contribution 2, Section 3.1, and the claim that 'this choice of ρ is optimal in terms of WCE'—overstates it. The authors should either prove monotonicity under explicit conditions on the parameters (especially s) or restrict the optimality claim to a parameter regime in which it is verified, and state what the optimal weight is in the large-separation regime.

minor comments (4)

[Section 1, Related Work] There is a typo in 'Our work focuses weighted learning on inseparable dala' — 'dala' should be 'data'.
[Section 3.1, after Theorem 2] The statement that for δ>2π+ 'the optimal choice of weighting is ρ→∞' is asserted without proof or a limiting argument; since Theorem 3 is restricted to δ<2π+, this claim should either be proved or explicitly labeled as a conjecture.
[Section 4 and Appendix B.1] The practical prescription depends on the choice of the 99% PCA variance threshold for the effective dimension; a sensitivity analysis over this threshold, or a more principled estimator, would substantially strengthen the claim that ρ̃ is a practical default weighting.
[Appendix A.6, Eq. (52)] Eq. (52) only compares ρ=1 with ρ̃ and does not identify the global WCE minimizer in the large-separation regime; the main text should make clear that the conjecture 'the optimal weight will decrease from ρ̃ to 1' is not established by the given condition.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the equal-error weight is derived algebraically from the model and the CGMT citations are external machinery; the optimality caveats are scope issues, not circularity.

full rationale

The derivation chain is self-contained and none of the central claims reduce to an input or to a self-citation. Theorem 2 defines rho-tilde in equation (10) and then verifies by substitution into system (8) that it yields b* = 0; equal per-class errors then follow from the risk formulas in (1), not from any fitted quantity. This is a genuine algebraic derivation, not a prediction that is forced by construction. Theorem 3 is explicitly conditional: 'Assume that R+ and R- are monotonically decreasing and increasing respectively in the weight parameter rho,' and the proof is a standard quasiconvexity argument under that assumption. Whether the monotonicity hypothesis actually holds for all parameters allowed by the theorem is a correctness/scope concern, not a circularity one; indeed Appendix A.6 states that 'if the separation of the classes is too large, then the unweighted model will outperform the weighted model,' which indicates the hypothesis can fail, but this does not make the theorem circular. The CGMT-based reduction in Theorem 1 relies on prior results [21, 22, 23] that are external mathematical machinery with stated assumptions that do not include the paper's target conclusions; the fact that one coauthor is also an author of those prior papers does not make the citation load-bearing. The unweighted closed form in Corollary 2 is credited to the external literature [24]. The empirical effective-dimension estimate from PCA is a heuristic input, not a parameter tuned to match the reported per-class errors, so it is not a fitted input renamed as a prediction. Therefore there is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four axioms: the CGMT mathematical tool, the Gaussian isotropic generative model, the proportional asymptotics, and an unproven monotonicity assumption for the optimality proof. The only hand-chosen numeric input is the 99% PCA threshold used to estimate the effective dimension in practice. No new physical or algorithmic entities are postulated.

free parameters (1)

PCA variance threshold = 99% (effective dimension d_eff = 3 for CelebA, 2-9 for CIFAR10 splits)
Used to estimate the dimension d entering δ = d/n and hence ρ̃. The 99% threshold is chosen by the authors, acknowledged as heuristic in Limitations, and materially changes the recommended weight.

assumptions (4)

standard math CGMT (convex Gaussian minimax theorem) and its high-dimensional asymptotic equivalence as stated in [22,23,29]
Basis for reducing weighted ERM to a scalar system in the proof of Theorem 1. It requires convexity and Gaussianity, which hold for the square-loss setup.
domain assumption Latent data follows class-conditional Gaussian with isotropic within-class covariance: X|Y=y ~ N(yμ, I)
Central generative model in Section 2.1; real validation data is not Gaussian isotropic, and the paper notes this in Limitations.
domain assumption Asymptotic regime d,n→∞ with fixed δ = d/n ∈ (0,1)
The closed forms are asymptotic; applying them to small finite n requires treating the PCA-estimated effective dimension as the true d.
ad hoc to paper Per-class risks R+(ρ) and R−(ρ) are monotonically decreasing and increasing in ρ, respectively
Assumed in Theorem 3 for the optimality of ρ̃; not proven, and the paper's Figure 8b shows the WCE-minimizing weight can be smaller than ρ̃ for large separation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining." pith.science (2026). https://pith.science/paper/QRKUOLMS

@misc{pith2026250620025,
  author       = {Pith},
  title        = {Pith review of: Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QRKUOLMS}},
  note         = {Machine review of arXiv:2506.20025}
}

read the original abstract

While machine learning models become more capable in discriminative tasks at scale, their ability to overcome biases introduced by training data has come under increasing scrutiny. Previous results suggest that there are two extremes of parameterization with very different behaviors: the population (underparameterized) setting where loss weighting is optimal and the separable overparameterized setting where loss weighting is ineffective at ensuring equal performance across classes. This work explores the regime of last layer retraining (LLR) in which the unseen limited (retraining) data is frequently inseparable and the model proportionately sized, falling between the two aforementioned extremes. We show, in theory and practice, that loss weighting is still effective in this regime, but that these weights \emph{must} take into account the relative overparameterization of the model.

Figures

Figures reproduced from arXiv: 2506.20025 by the authors.

**Figure 2.** Per-class test error plotted against the weight ratio [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Effect of ρ on the per-class test error on a synthetic dataset. We see the restriction for δ < 2π+ for ρ˜ (red) to be defined has an operational meaning regarding the per-class errors. Indeed if δ > 2π+, the per-class errors never meet and thus cannot be balanced. It is worth noting that this ρ˜ for WCE does not align with the conventional wisdom of the ratio of the priors except as δ → 0, which is the case that n →… view at source ↗

Figures from the paper (8 more)

**Figure 5.** Figure 5: A comparison of wERM using the optimal weight ρ˜ and downsampling. We see that for a fixed δ, the error for wERM is much lower than that of downsampling but that both achieve balanced per-class errors. Proof. For convenience, we overload WCE to be a function of weight…

**Figure 6.** Figure 6: Per-class accuracies for binary hair texture classification on the CelebA dataset, with the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Per-class accuracies for binary classification on the CIFAR10 dataset (planes vs trucks), [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: With mild separation, the weighted model (red) does significantly better than the un [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

**Figure 9.** Figure 9: PCA Spectra. We see that most of the variance is captured by very few features. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

**Figure 10.** Figure 10: Per-class errors on CelebA dataset. Even with a lower latent dimension, [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

**Figure 11.** Figure 11: Per-class errors on CIFAR10 dataset, cats vs dogs. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

**Figure 12.** Figure 12: Per-class errors on CIFAR10 dataset, deer vs horses. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 20 canonical work pages

[1]

Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. In The Eleventh International Conference on Learning Representa- tions, 2023. URL https://openreview.net/forum?id=Zb6c8A-Fghk

work page 2023
[2]

On Feature Learning in the Presence of Spurious Correlations, October 2022

Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson. On Feature Learning in the Presence of Spurious Correlations, October 2022. URL http://arxiv.org/abs/2210.11369. arXiv:2210.11369 [cs]

arXiv 2022
[3]

Theoretical guarantees of data augmented last layer retraining methods

Monica Welfert, Nathan Stromberg, and Lalitha Sankar. Theoretical guarantees of data augmented last layer retraining methods. In 2024 IEEE International Symposium on Information Theory (ISIT) , pages 581–586. IEEE, 2024

work page 2024
[4]

Kamalika Chaudhuri, Kartik Ahuja, Martin Arjovsky, and David Lopez-Paz. Why does throwing away data improve worst-group error? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Lear...

work page 2023
[5]

Long-tail learning via logit adjustment, July 2021

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and San- jiv Kumar. Long-tail learning via logit adjustment, July 2021. URL http://arxiv.org/abs/2007. 07314. arXiv:2007.07314 [cs, stat]

arXiv 2021
[6]

Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, October 2019

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, October 2019. URL http://arxiv.org/abs/1906. 07413. arXiv:1906.07413 [cs, stat]

arXiv 2019
[7]

Towards a Theoretical Framework of Out-of-Distribution Generalization

Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards a Theoretical Framework of Out-of-Distribution Generalization. November 2021. URL https://openreview.net/ forum?id=kFJoj7zuDVi

work page 2021
[8]

A statistical theory of overfitting for imbalanced classi- fication, February 2025

Jingyang Lyu, Kangjie Zhou, and Yiqiao Zhong. A statistical theory of overfitting for imbalanced classi- fication, February 2025. URL http://arxiv.org/abs/2502.11323. arXiv:2502.11323 [math]

arXiv 2025

Show all 30 references

[9]

Label- imbalanced and group-sensitive classification under overparameterization

Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis. Label- imbalanced and group-sensitive classification under overparameterization. Advances in Neural Infor- mation Processing Systems, 34:18970–18983, 2021

2021
[10]

importance- weighted

Kuo-Wei Lai and Vidya Muthukumar. Sharp analysis of out-of-distribution error for “importance- weighted” estimators in the overparameterized regime. In 2024 IEEE International Symposium on In- formation Theory (ISIT), pages 3701–3706, 2024. doi: 10.1109/ISIT57864.2024.10619252

2024
[11]

Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In Ka- malika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research...

2019
[12]

Understanding the role of importance weighting for deep learn- ing

Da Xu, Yuting Ye, and Chuanwei Ruan. Understanding the role of importance weighting for deep learn- ing. In International Conference on Learning Representations, 2021

2021
[13]

The implicit bias of gradient descent on separable data.The Journal of Machine Learning Research, 19(1):2822–2878, January 2018

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data.The Journal of Machine Learning Research, 19(1):2822–2878, January 2018. ISSN 1532-4435

2018
[14]

The implicit bias of gradient descent on nonseparable data

Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Proceedings of the Thirty-Second Conference on Learning Theory, pages 1772–1798. PMLR, June 2019. URLhttps: //proceedings.mlr.press/v99/ji19a.html. ISSN: 2640-3498

2019
[15]

Characterizing Implicit Bias in Terms of Optimization Geometry

Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing Implicit Bias in Terms of Optimization Geometry. In Proceedings of the 35th International Conference on Machine Learning , pages 1832–1841. PMLR, July 2018. URL https://proceedings.mlr.press/v80/guna...

2018
[16]

Understanding the role of importance weighting for deep learn- ing, March 2021

Da Xu, Yuting Ye, and Chuanwei Ruan. Understanding the role of importance weighting for deep learn- ing, March 2021. URL http://arxiv.org/abs/2103.15209. arXiv:2103.15209 [cs]

2021 arXiv
[17]

An Investigation of Why Overpa- rameterization Exacerbates Spurious Correlations, 2020

Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An Investigation of Why Overpa- rameterization Exacerbates Spurious Correlations, 2020. URLhttps://arxiv.org/abs/2005.04345. 10

2020 arXiv
[18]

On how to avoid exacerbating spurious correla- tions when models are overparameterized, June 2022

Tina Behnia, Ke Wang, and Christos Thrampoulidis. On how to avoid exacerbating spurious correla- tions when models are overparameterized, June 2022. URL http://arxiv.org/abs/2206.12739. arXiv:2206.12739 [cs]

2022 arXiv
[19]

Towards last-layer retraining for group robust- ness with fewer annotations

Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Towards last-layer retraining for group robust- ness with fewer annotations. In Thirty-seventh Conference on Neural Information Processing Systems , 2023

2023
[20]

For robust worst-group accuracy, ignore group annotations.Transactions on Machine Learning Research,

Nathan Stromberg, Rohan Ayyagari, Monica Welfert, Sanmi Koyejo, Richard Nock, and Lalitha Sankar. For robust worst-group accuracy, ignore group annotations.Transactions on Machine Learning Research,
[21]

A framework to characterize performance of LASSO algorithms, March 2013

Mihailo Stojnic. A framework to characterize performance of LASSO algorithms, March 2013. URL http://arxiv.org/abs/1303.7291. arXiv:1303.7291 [cs]

2013 arXiv
[22]

Regularized Linear Regression: A Precise Analysis of the Estimation Error

Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. Regularized Linear Regression: A Precise Analysis of the Estimation Error. In Proceedings of The 28th Conference on Learning Theory , pages 1683–1709. PMLR, June 2015. URL https://proceedings.mlr.press/v40/Thrampoulidis15...

2015
[23]

Precise error analysis of regularized m- estimators in high dimensions

Christos Thrampoulidis, Ehsan Abbasi, and Babak Hassibi. Precise error analysis of regularized m- estimators in high dimensions. IEEE Transactions on Information Theory, 64(8):5592–5628, 2018

2018
[24]

The Role of Regularization in Classification of High-dimensional Noisy Gaussian Mixture

Francesca Mignacco, Florent Krzakala, Yue Lu, Pierfrancesco Urbani, and Lenka Zdeborova. The Role of Regularization in Classification of High-dimensional Noisy Gaussian Mixture. In Proceedings of the 37th International Conference on Machine Learning , pages 6874–6883. PMLR, No...

2020
[25]

Deep learning face attributes in the wild

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015

2015
[26]

Learning multiple layers of features from tiny images

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Techni- cal Report 0, University of Toronto, Toronto, Ontario, 2009. URL https://www.cs.toronto.edu/ ~kriz/learning-features-2009-TR.pdf

2009
[27]

Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks

L Hui. Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. In The Ninth International Conference on Learning Representations (ICLR 2021), 2020

2021
[28]

LQF: Linear Quadratic Fine-Tuning

Alessandro Achille, Aditya Golatkar, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. LQF: Linear Quadratic Fine-Tuning . In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 15724–15734, Los Alamitos, CA, USA, June 2021. IEEE Computer...

2021
[29]

effective

Hossein Taheri, Ramtin Pedarsani, and Christos Thrampoulidis. Sharp Asymptotics and Optimal Perfor- mance for Inference in Binary Models, February 2020. URL http://arxiv.org/abs/2002.07284. arXiv:2002.07284 [math]. 11 A Proofs A.1 Proof of Risk in (1) Proof. Note we can decomp...

2020 arXiv
[30]

We see that up toρ = 60, the per-class errors do not meet

Note that for n = 20 , ˜ρ is undefined, suggesting that the correct weighting strategy is to push ρ→∞ . We see that up toρ = 60, the per-class errors do not meet. 10 20 30 40 50 60 0.0 0.1 0.2 0.3 0.4 0.5Error Backbone: resnet34, Latent Dim: 256, n: 90 Class +1 -1 (a)n = 90 10...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. In The Eleventh International Conference on Learning Representa- tions, 2023. URL https://openreview.net/forum?id=Zb6c8A-Fghk

work page 2023

[2] [2]

On Feature Learning in the Presence of Spurious Correlations, October 2022

Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson. On Feature Learning in the Presence of Spurious Correlations, October 2022. URL http://arxiv.org/abs/2210.11369. arXiv:2210.11369 [cs]

arXiv 2022

[3] [3]

Theoretical guarantees of data augmented last layer retraining methods

Monica Welfert, Nathan Stromberg, and Lalitha Sankar. Theoretical guarantees of data augmented last layer retraining methods. In 2024 IEEE International Symposium on Information Theory (ISIT) , pages 581–586. IEEE, 2024

work page 2024

[4] [4]

Kamalika Chaudhuri, Kartik Ahuja, Martin Arjovsky, and David Lopez-Paz. Why does throwing away data improve worst-group error? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Lear...

work page 2023

[5] [5]

Long-tail learning via logit adjustment, July 2021

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and San- jiv Kumar. Long-tail learning via logit adjustment, July 2021. URL http://arxiv.org/abs/2007. 07314. arXiv:2007.07314 [cs, stat]

arXiv 2021

[6] [6]

Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, October 2019

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, October 2019. URL http://arxiv.org/abs/1906. 07413. arXiv:1906.07413 [cs, stat]

arXiv 2019

[7] [7]

Towards a Theoretical Framework of Out-of-Distribution Generalization

Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards a Theoretical Framework of Out-of-Distribution Generalization. November 2021. URL https://openreview.net/ forum?id=kFJoj7zuDVi

work page 2021

[8] [8]

A statistical theory of overfitting for imbalanced classi- fication, February 2025

Jingyang Lyu, Kangjie Zhou, and Yiqiao Zhong. A statistical theory of overfitting for imbalanced classi- fication, February 2025. URL http://arxiv.org/abs/2502.11323. arXiv:2502.11323 [math]

arXiv 2025