{"id":"bcbe3303-fe1e-4fc6-8c78-dec78be03c06","arxiv_id":"2506.20025","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"For square-loss weighted ERM in the proportional asymptotic regime, the class weight that equalizes per-class errors is ρ̃ = π−/π+ + (π−/π+ − 1) δ/(2π+ − δ), exceeding the ratio of priors and growing with δ.","lead":"This paper studies last layer retraining for imbalanced binary classification when the number of retraining samples and the feature dimension are comparable, and derives a class-weight formula that grows with the overparameterization ratio. It shows the classical ratio-of-priors weight is suboptimal and validates the correction on image classification tasks.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The optimality claim is not safe: the monotonicity hypothesis of Theorem 3 fails in a parameter regime the theorem allows, so rho-tilde is not the worst-class-error minimizer there.","rationale":"The reader's weakest assumption identifies the same load-bearing point: Theorem 3's optimality rests on unproven monotonicity of per-class risks in rho. My stress test sharpens this from 'unproven' to 'contradicted by the paper's own asymptotic analysis.' I verified from the paper's closed forms that at a parameter point satisfying the theorem's hypotheses, R+(rho) increases between rho = 1 and rho = rho-tilde; hence the monotonicity assumption is false there, and rho-tilde is not the WCE minimizer in that regime. This is the single most load-bearing concern because the title and abstract promise an optimal weighting, and that promise is conditional at best. I do not see a comparably serious problem with the CGMT reduction, the equal-error derivation, or the practical experiments; those remain valuable. The practical impact of the monotonicity failure is softened by the fact that it occurs at large signal separation where all errors are tiny, and the paper's own Appendix A.6 already notes this. The correct remedy is to qualify the optimality claim or characterize when monotonicity holds. Since the reader's conditional verdict already demands this, my read does not change the verdict.","tokens_in":16304,"tokens_out":10538,"duration_ms":114563,"concrete_test":"Recompute R+(1) from Corollary 2 and R+(rho-tilde) from Theorem 2 at delta = 0.2, pi+ = 0.2, s = 4, using the closed forms for (gamma*, b*, alpha*) and (gamma-tilde, alpha-tilde). If R+(1) < R+(rho-tilde), then R+ is not monotone decreasing on [1, rho-tilde], and the hypothesis of Theorem 3 fails; this single comparison decides whether the optimality claim needs an s-domain restriction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is Theorem 3: for delta < 2*pi+, the weight rho-tilde from Theorem 2 minimizes WCE. The proof depends entirely on the stated hypothesis that R+(rho) is decreasing and R-(rho) is increasing in rho. Without that hypothesis, WCE = max(R+, R-) need not be quasiconvex and the equal-error point need not be the minimizer. This hypothesis is load-bearing, and it is contradicted by the paper's own asymptotic formulas. At delta = 0.2, pi+ = 0.2, s = 4, we have delta < 2*pi+ = 0.4, so Theorem 3 claims to apply. Using the closed forms in Corollary 2 and Theorem 2, the unweighted minority-class argument is t_u = (gamma* s + b*) / alpha* ≈ 3.33, while the weighted equal-error argument at rho-tilde = 7 is t_w = gamma-tilde s / alpha-tilde ≈ 3.27. Since Q is decreasing, R+(1) = Q(3.33) < Q(3.27) = R+(rho-tilde). Thus R+ increases between rho = 1 and rho = rho-tilde, directly violating the required monotonicity. This is exactly why Appendix A.6 and Figure 8b find that unweighted ERM can beat rho-tilde for large separation. The equal-error characterization in Theorem 2 remains valid, and the practical damage is limited because the errors in this high-separation regime are very small, but the paper's 'optimal' claim is overbroad. It needs an explicit condition on s, or a proof that monotonicity holds in the regimes where optimality is claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper analyzes weighted empirical risk minimization (wERM) for last-layer retraining in the proportional asymptotic regime d/n = δ ∈ (0,1), under a class-conditional Gaussian model. Using the convex Gaussian minimax theorem, the authors reduce the (d+1)-dimensional optimization to a scalar system for general convex losses (Theorem 1), specialize to square loss (Corollary 1), and derive closed-form solutions for unweighted ERM (Corollary 2) and for the weight ρ̃ that equalizes per-class errors (Theorem 2). Theorem 3 claims that ρ̃ minimizes worst-class error under a monotonicity assumption on the per-class risks. The paper compares wERM with downsampling and reports experiments on CelebA and CIFAR-10 in which the weight predicted from a PCA-based effective dimension outperforms the classical ratio of priors.","tokens_in":16664,"tokens_out":9994,"duration_ms":101266,"significance":"The equal-error characterization is a useful contribution: it bridges the population and separable-overparameterized regimes and predicts that the balancing weight must grow with δ, an effect confirmed by the synthetic simulations and qualitatively by the vision experiments. The CGMT reduction and the closed-form calculations are careful, the internal algebra in Corollary 2 and Theorem 2 is consistent, and the paper makes a falsifiable prediction about the dependence of the optimal weight on overparameterization. I also note that ρ̃ is derived from the generative model and optimality conditions rather than fitted to empirical crossover points, so the comparisons in Figures 2 and 6 are not circular. The practical demonstrations are heuristic but support the qualitative message. However, the central optimality claim is currently overbroad: it relies on an unproven and, by the paper's own asymptotics, false monotonicity assumption in a regime the theorem allows, so the word 'optimal' in the abstract and contributions needs to be scoped down or supported by an additional proof.","major_comments":[{"comment":"The optimality claim for ρ̃ is load-bearing for the paper's main message, but Theorem 3's monotonicity hypothesis is not proven and is contradicted by the paper's own closed-form formulas. For δ=0.2, π+=0.2, s=4, which satisfies δ<2π+=0.4, Corollary 2 and Theorem 2 give unweighted minority-class argument (γ*s+b*)/α* ≈ 3.33 and weighted equal-error argument γ̃s/α̃ ≈ 3.28. Since Q is decreasing and b*<0, this gives WCE(ρ=1)=R+(1)=Q(3.33) < Q(3.28)=R+(ρ̃)=WCE(ρ̃); hence ρ̃ is not the WCE minimizer at this allowed parameter point, and R+ is not monotonically decreasing on [1,ρ̃]. This is exactly the behavior shown in Appendix A.6 and Figure 8b, and it is captured quantitatively by Eq. (52). The theorem as a conditional statement is valid, but the surrounding text—abstract contribution 2, Section 3.1, and the claim that 'this choice of ρ is optimal in terms of WCE'—overstates it. The authors should either prove monotonicity under explicit conditions on the parameters (especially s) or restrict the optimality claim to a parameter regime in which it is verified, and state what the optimal weight is in the large-separation regime.","section":"§3.1 (Theorem 3) and Appendix A.6, Eq. (52), Fig. 8b"}],"minor_comments":[{"comment":"There is a typo in 'Our work focuses weighted learning on inseparable dala' — 'dala' should be 'data'.","section":"Section 1, Related Work"},{"comment":"The statement that for δ>2π+ 'the optimal choice of weighting is ρ→∞' is asserted without proof or a limiting argument; since Theorem 3 is restricted to δ<2π+, this claim should either be proved or explicitly labeled as a conjecture.","section":"Section 3.1, after Theorem 2"},{"comment":"The practical prescription depends on the choice of the 99% PCA variance threshold for the effective dimension; a sensitivity analysis over this threshold, or a more principled estimator, would substantially strengthen the claim that ρ̃ is a practical default weighting.","section":"Section 4 and Appendix B.1"},{"comment":"Eq. (52) only compares ρ=1 with ρ̃ and does not identify the global WCE minimizer in the large-separation regime; the main text should make clear that the conjecture 'the optimal weight will decrease from ρ̃ to 1' is not established by the given condition.","section":"Appendix A.6, Eq. (52)"}],"recommendation":"major_revision","confidential_remarks":"The core CGMT derivation and equal-error formulas are sound and worth publishing, but the 'optimal' language in the abstract and contributions must be scoped to match what Theorem 3 actually proves. The monotonicity issue is not a finite-sample artifact: it follows from the paper's own asymptotic closed forms, so it cannot be dismissed as a practical detail. The PCA threshold is a second, smaller gap between theory and practice. Please ensure the revision either proves a restricted optimality statement or rephrases the contributions accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing worth knowing up front: the δ-dependent equal-error weight ρ̃ in Theorem 2 is a genuine new result, and the CGMT derivation is competently executed. But the 'optimal' claim in Theorem 3 is not safe as stated; the paper's own formulas violate the required monotonicity in a regime the theorem allows, so ρ̃ should be called an equal-error weight, not the worst-class minimizer, until that's fixed.\n\nWhat's actually new and good. The paper studies weighted ERM for last-layer retraining in the proportional regime δ=d/n∈(0,1), which sits between the population limit and the separable overparameterized limit. Theorem 1 reduces weighted ERM to four scalar equations via CGMT; for square loss, Corollary 1 gives a concrete system, and Theorem 2 solves it in closed form for the weight ρ̃ = π−/π+ + (π−/π+−1)δ/(2π+−δ) that makes b*=0 and equalizes the class errors. That closed form is, as far as I can tell from the cited literature, new, and it is internally consistent: the WCE expression Q(s²√(1−Δ)/(√Δ+s²)) follows. The simulation matches across δ, and the comparison with downsampling is informative. The real-data experiments are honest, including the admission that the PCA effective dimension is a heuristic.\n\nWhere it gets soft. Theorem 3 claims ρ̃ minimizes WCE under the assumption that R+ is decreasing and R− is increasing in ρ. That assumption is load-bearing, not proven, and in fact the paper's own asymptotic formulas contradict it for large signal separation. Working from Corollary 2 and Theorem 2 at δ=0.2, π+=0.2, s=4, the unweighted minority-class argument is about 3.33 and the weighted equal-error argument at ρ̃=7 is about 3.27, so R+(1)<R+(ρ̃). This directly violates the monotonicity the theorem requires. The paper's Appendix A.6 and Figure 8b acknowledge the failure—unweighted ERM beats ρ̃ at s=4—and derive a condition (52) for when weighting hurts. So the correct statement is: ρ̃ is the weight that equalizes class errors, and it beats the ratio of priors in the moderate-separation regime; it is not the WCE minimizer for all δ<2π+. The practical damage is limited because the offending regime has vanishingly small errors, but a theorem that claims optimality should carry an explicit condition on s or a proof of monotonicity where claimed.\n\nMinor issues: no code is provided, and the effective-dimension estimation is ad hoc. Neither undercuts the theory, but they matter for a practical recipe.\n\nBottom line: this deserves a serious referee. The Theorem 2 result is useful and new; the optimality claim is fixable with a qualified statement. I'd cite the equal-error weight and would want to see a revision that fixes Theorem 3.","headline":"The equal-error weight in Theorem 2 is a real contribution, but the 'optimal' claim in Theorem 3 is overbroad and needs a separation condition.","tokens_in":17203,"tokens_out":4034,"would_cite":true,"duration_ms":39746,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H30","68T07","68T10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Weighting the minority class in last layer retraining is effective, and the optimal weight grows with overparameterization.","keywords":["last layer retraining","loss weighting","worst-class error","class imbalance","overparameterization","convex Gaussian minimax theorem","square loss","effective dimension"],"falsifier":"Run the Gaussian experiment of Section 3 with strong separation ($s=4$), $\\delta<2\\pi_+$, plot per-class errors against $\\rho$ over a fine grid, and locate the weight minimizing worst-class error; if it is not $\\tilde{\\rho}$, the monotonicity assumption behind Theorem 3 is false in that regime. A direct calculation of $dR_+/d\\rho$ and $dR_-/d\\rho$ at that setting would settle the matter without simulation.","tokens_in":16095,"feed_emoji":"⚖️","tokens_out":8311,"duration_ms":76256,"temperature":0.7,"pith_summary":"This paper studies last layer retraining (LLR), where a pretrained model is frozen and only its final linear layer is refit on new data. In the regime where the number of retraining samples $n$ and the latent dimension $d$ are comparable (ratio $\\delta=d/n$ in $(0,1)$), the paper argues that class-weighted empirical risk minimization is effective, unlike in the separable overparameterized regime where weighting has no effect. For square loss on class-conditional Gaussian data, it derives the asymptotically optimal minority-class weight $\\tilde{\\rho}=\\pi_-/\\pi_+ + (\\pi_-/\\pi_+-1)\\delta/(2\\pi_+-\\delta)$, which equalizes per-class errors and, under a monotonicity assumption, minimizes worst-class error. This weight is larger than the classical ratio of priors, and the gap grows with $\\delta$. In practice, using an effective latent dimension makes the formula outperform ratio-of-priors weighting on image-retraining tasks.","feed_headline":"Optimal minority weight grows with overparameterization","feed_subtitle":"A new formula beats the classical ratio of priors when retraining data is scarce.","key_machinery":"The central machinery is a reduction of the $(d+1)$-dimensional weighted ERM problem to four scalar equations in $(\\alpha,\\gamma,\\lambda,b)$, obtained through the convex Gaussian minimax theorem and expressed with Moreau envelopes of the loss. For square loss the Moreau envelope is $M(x;\\lambda)=(x-1)^2/(2(1+\\lambda))$, which turns the system into algebraic equations. The key object derived from that system is $\\tilde{\\rho}$: the weight that zeroes the bias, thereby equalizing the two class risks.","core_discovery":"Under proportional asymptotics $d,n\\to\\infty$ with $d/n\\to\\delta\\in(0,1)$, the paper reduces weighted square-loss ERM on a Gaussian mixture to a closed-form system, and identifies the weight $\\tilde{\\rho}=\\pi_-/\\pi_+ + (\\pi_-/\\pi_+-1)\\delta/(2\\pi_+-\\delta)$ as the unique weight that drives the asymptotic bias to zero, $b^*=0$. At this weight the per-class risks are equal, so worst-class error becomes $Q(s^2\\sqrt{1-\\Delta}/(\\sqrt{\\Delta}+s^2))$ with $\\Delta=\\delta/(4\\pi_+)+\\delta/(4\\pi_-)$. The paper claims this weight is optimal for worst-class error provided the per-class risks are monotone in $\\rho$, and that it strictly outperforms both ratio-of-priors weighting and downsampling for $\\delta>0$. The formula reduces to the classical ratio of priors as $\\delta\\to 0$, and it diverges as $\\delta$ approaches $2\\pi_+$ from below, marking the point where no finite weight can equalize the classes.","pith_inferences":["The monotonicity assumption behind optimality can be tested directly from the system (8): if $dR_+/d\\rho$ changes sign for large signal strength $s$, the optimal-weight claim fails there, and a corrected weight would be the actual minimizer of the maximum of the two risks.","The same scalar-system approach likely applies to other convex losses, but only square loss gives a closed-form $\\tilde{\\rho}$; for other losses one would need to solve the four equations numerically, so the simple formula may not survive.","A practical adaptive rule could estimate the effective dimension from the retraining data itself rather than by a PCA threshold, making the weighting self-tuning and removing the main heuristic in the paper's real-data recipe.","The results suggest a general principle: optimal cost-sensitive corrections depend on the ratio of trainable parameters to retraining samples, so any method that reports a single best weight for a dataset should be expected to change with sample size."],"forward_implications":["As the ratio $\\delta=d/n$ grows within $(0,1)$, the optimal minority weight rises above $\\pi_-/\\pi_+$; using the classical ratio underweights the minority in small-sample retraining.","When $\\delta\\ge 2\\pi_+$, no finite weight equalizes the classes; the minority class keeps dominating worst-class error and the recommended strategy is to upweight it without bound.","Downsampling the majority class in this regime is equivalent to increasing $\\delta$ to $\\delta/(2\\pi_+)$, so weighted ERM with $\\tilde{\\rho}$ achieves lower worst-class error than downsampling at the same data size.","On real image-retraining tasks, the practical weight should be computed from an effective latent dimension (the number of principal components capturing most variance), not from the nominal last-layer width.","In the population limit $\\delta\\to 0$, $\\tilde{\\rho}$ reduces to $\\pi_-/\\pi_+$, recovering the classical ratio-of-priors rule."],"supporting_citations":[{"why":"Defines last layer retraining as a robustness correction and supplies the observation that the effective latent dimension is much smaller than the nominal width.","marker":"[1]"},{"why":"Establishes the population-regime baseline where ratio-of-priors weighting is optimal, the $\\delta\\to 0$ limit this paper extends.","marker":"[3]"},{"why":"Provides the downsampling analysis in separable settings that this paper compares against.","marker":"[4]"},{"why":"Shows overparameterized CS-SVM behavior for $\\delta>1$, the contrast regime motivating the underparameterized analysis.","marker":"[9]"},{"why":"Documents that importance weighting loses effect for large overparameterized models, the puzzle this paper resolves for $\\delta\\in(0,1)$.","marker":"[11]"},{"why":"Similar evidence that weighting does not change the learned model in overparameterized settings.","marker":"[12]"},{"why":"Supplies the convex Gaussian minimax theorem used to reduce wERM to the scalar system.","marker":"[22]"},{"why":"Provides the Moreau-envelope properties and CGMT scalarization details used in the proof of Theorem 1.","marker":"[23]"},{"why":"Gives the closed-form unweighted Gaussian-mixture solution that Corollary 2 reproduces as a special case.","marker":"[24]"}],"fun_headline_variants":["Optimal loss weight now depends on overparameterization","New formula for loss weight beats ratio of priors","Last layer retraining: loss weight adjusts to overparameterization","Loss weighting works in retraining when weighted by delta","Optimal weight formula: ratio of priors plus correction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that $\\tilde{\\rho}$ actually minimizes worst-class error depends on the unproven assumption that the minority-class error falls and the majority-class error rises monotonically as the minority weight increases; when class separation is large, the paper's own simulations show the worst-class minimizer lies below $\\tilde{\\rho}$, so that assumption fails there.","fun_headline_variants_meta":{"raw":{"variants":["Optimal loss weight now depends on overparameterization","New formula for loss weight beats ratio of priors","Last layer retraining: loss weight adjusts to overparameterization","Loss weighting works in retraining when weighted by delta","Optimal weight formula: ratio of priors plus correction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000809,"raw_usage":{"total_tokens":3528,"prompt_tokens":902,"completion_tokens":2626,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":2547}},"tokens_in":518,"tokens_out":2626,"duration_ms":19572,"temperature":1.0,"reasoning_tokens":2547,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:22:49.174219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Gaussian experiment of Section 3 with strong separation ($s=4$), $\\delta<2\\pi_+$, plot per-class errors against $\\rho$ over a fine grid, and locate the weight minimizing worst-class error; if it is not $\\tilde{\\rho}$, the monotonicity assumption behind Theorem 3 is false in that regime. A direct calculation of $dR_+/d\\rho$ and $dR_-/d\\rho$ at that setting would settle the matter without simulation.","supporting_citations":[{"cited_title":"Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations","cited_arxiv_id":null,"evidence_quote":"Defines last layer retraining as a robustness correction and supplies the observation that the effective latent dimension is much smaller than the nominal width."},{"cited_title":"Theoretical guarantees of data augmented last layer retraining methods","cited_arxiv_id":null,"evidence_quote":"Establishes the population-regime baseline where ratio-of-priors weighting is optimal, the $\\delta\\to 0$ limit this paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the downsampling analysis in separable settings that this paper compares against."},{"cited_title":"Label- imbalanced and group-sensitive classification under overparameterization","cited_arxiv_id":null,"evidence_quote":"Shows overparameterized CS-SVM behavior for $\\delta>1$, the contrast regime motivating the underparameterized analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that importance weighting loses effect for large overparameterized models, the puzzle this paper resolves for $\\delta\\in(0,1)$."},{"cited_title":"Understanding the role of importance weighting for deep learn- ing","cited_arxiv_id":null,"evidence_quote":"Similar evidence that weighting does not change the learned model in overparameterized settings."},{"cited_title":"Regularized Linear Regression: A Precise Analysis of the Estimation Error","cited_arxiv_id":null,"evidence_quote":"Supplies the convex Gaussian minimax theorem used to reduce wERM to the scalar system."},{"cited_title":"Precise error analysis of regularized m- estimators in high dimensions","cited_arxiv_id":null,"evidence_quote":"Provides the Moreau-envelope properties and CGMT scalarization details used in the proof of Theorem 1."},{"cited_title":"The Role of Regularization in Classification of High-dimensional Noisy Gaussian Mixture","cited_arxiv_id":null,"evidence_quote":"Gives the closed-form unweighted Gaussian-mixture solution that Corollary 2 reproduces as a special case."}],"review_version":2}