{"id":"dc13460b-9aab-4d65-a3c4-c2b4c293e911","arxiv_id":"2507.10536","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Under heavy-tail class imbalance, subtracting the DP noise variance from Adam's second moment (DP-AdamBC) substantially improves learning of rare classes compared with DP gradient descent.","lead":"This paper studies four differentially private optimizers on datasets where some classes are very rare, and finds that plain DP gradient descent learns rare classes poorly while Adam with bias correction (DP-AdamBC) learns them noticeably better. The finding suggests a simple optimizer fix for privacy-preserving training on imbalanced natural language and image data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DP-AdamBC's rare-class gains may come from the tuned floor γ' in the bias-corrected denominator rather than from unbiased curvature estimation; an ablation is needed to support the paper's causal claim.","rationale":"The reader's weakest assumption (Assumption 1 of Kunstner et al., large output probability p) is a real limitation of the scaling argument, but the most load-bearing issue for the paper's causal claim is the algorithm's update rule itself. Section 2's argument that DP-AdamBC 'magnifies/shrinks' updates by dividing by E[g_t^2] assumes the bias-corrected second moment is positive and proportional to true curvature. Algorithm 2's max(·, γ') floor means that for classes where E[g_t^2] is comparable to or smaller than (σC/L)^2, the denominator is set by the tunable γ'. Since Section 3 says the stability constants are tuned per optimizer, a small γ' can produce large rare-class updates irrespective of curvature. This is not an external disagreement; it is a potential confound inside the reported experimental setup. The paper reports no variation of γ' or comparison with a floor-only control, so the empirical evidence does not yet distinguish 'unbiased curvature estimation helps' from 'a small floor constant helps rare classes.' I am not claiming the results are false; I am claiming the current experiments cannot support the specific mechanism highlighted in the strongest claim. The reader's verdict of CONDITIONAL already requires additional evidence, so my concern does not change the verdict, but it sharpens the condition: an ablation of the floor is necessary before accepting the causal interpretation.","tokens_in":12014,"tokens_out":8808,"duration_ms":106534,"concrete_test":"Re-run the Barcode MNIST experiment (and, if possible, the synthetic linear model) with DP-AdamBC's denominator changed from sqrt(max(v_t - (σC/L)^2, γ')) to sqrt(max(v_t - (σC/L)^2, 0)) + γ, using a fixed γ = 1e-8 and the same tuned learning rate; also run a control with γ' fixed to a value comparable to the rare-class second-moment scale. If the rare-class training accuracy gain over DP-Adam persists, the bias correction is the active ingredient; if it vanishes, the gain comes from the tuned floor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The causal claim that DP-AdamBC's bias correction is 'crucial' rests on the assumption in §2 that after subtracting the DP noise variance from the second-moment estimate, the residual E[g_t^2] is positive and informative for low-frequency classes. In Algorithm 2 the denominator is sqrt(max(v_t - (σC/L)^2, γ')), so whenever the true signal is at or below the noise floor, the update is controlled by the tuned constant γ' rather than by curvature. Section 3 states that the numerical stability constants are tuned separately for DP-Adam and DP-AdamBC. A small γ' gives low-frequency classes a large normalized step size even when the bias-corrected second moment is uninformative, so the reported ≈8% and ≈5% rare-class accuracy gains could be an artifact of this extra degree of freedom rather than evidence that removing DP bias restores curvature estimation. The paper provides no ablation separating these effects.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies differentially private optimization under heavy-tail class imbalance, a setting common in language modeling. Building on the linear-model analysis of Kunstner et al. [8], it argues that DP-GD is ill-conditioned for low-frequency classes because gradients and Hessians both scale with class frequency, while Adam-type methods that estimate curvature can mitigate this. The paper further argues that DP-Adam's second-moment estimate is dominated by DP noise variance, so the bias-corrected variant DP-AdamBC is crucial for recovering the curvature signal. Experiments on a synthetic linear model, Barcode MNIST, TinyPTB, and E2E show a consistent trend that DP-AdamBC improves training accuracy on rare classes, with headline gains of about 8% on the synthetic task and about 5% on Barcode MNIST. The theoretical discussion is heuristic and imports Assumption 1 from [8], and the empirical evaluation is based on single runs with tuned stability constants.","tokens_in":12081,"tokens_out":5302,"duration_ms":66715,"significance":"If the central claim holds, the paper identifies a practical and cheap algorithmic fix: under DP, using DP-AdamBC instead of DP-Adam can substantially improve training on rare classes in heavy-tail imbalanced data, which is directly relevant to private language modeling. The strengths are the controlled synthetic setup, the inclusion of several DP hyperparameter settings in Appendix C, and the fact that the empirical results are not fit to the theory. The main weaknesses are the absence of multiple seeds or error bars, the lack of an ablation for the numerical floor constant in DP-AdamBC, and the unverified reliance on Assumption 1 of [8]. These issues do not invalidate the trend, but they do mean the specific quantitative claims and the causal attribution to bias correction are not fully supported. The paper is a contribution to an incremental research line, but the evidence as presented is not yet at journal strength.","major_comments":[{"comment":"The central causal claim that DP-AdamBC's bias correction is 'crucial' is confounded by the numerical floor gamma-prime in the denominator. In Algorithm 2, the update divides by sqrt(max(v_hat_t - (sigma*C/L)^2, gamma-prime)); when the true signal E[g_t^2] is at or below the DP noise variance (sigma*C/L)^2, the denominator is exactly gamma-prime, so the update is controlled by the tuned constant rather than by unbiased curvature. The paper states in §3 that numerical stability constants are tuned separately for DP-Adam and DP-AdamBC, but it does not report the tuned values or run an ablation. Without an ablation that varies gamma-prime while holding the bias-correction term fixed (or vice versa), the reported ≈8% and ≈5% rare-class gains cannot be attributed to bias removal. I request an ablation reporting the tuned gamma and gamma-prime for each method, DP-AdamBC with gamma-prime matched to DP-Adam's gamma, and a version with a standard small gamma-prime.","section":"§3, Algorithm 2"},{"comment":"All quantitative claims are based on a single run per configuration. No seeds, error bars, or confidence intervals are reported, and hyperparameters are selected as those with the lowest overall training loss on the same data used for evaluation. Given that the headline figures concern rare classes with as few as 5–10 samples, the differences could plausibly lie within run-to-run variation. The paper should report multiple seeds (at least for the synthetic and Barcode MNIST experiments), or explicitly quantify the variance and show that the ordering of optimizers is stable. Without this, the abstract's precise figures (≈8%, ≈5%) are not fully supported.","section":"§3, Appendix B"},{"comment":"The theoretical argument imports Assumption 1 of Kunstner et al. [8] without verification in the private setting. The assumption that the model assigns a large output probability p to the correct class is unlikely to hold early in training, for rare classes, or after clipping, and the derivation of the scaling of gradients and Hessians with class frequency pi_k depends on it. The analysis also takes expectations only over the DP noise z_t, not over the optimization trajectory or data subsampling. As a concrete check, the authors could measure p for low-frequency classes during DP training or test whether the predicted rank ordering of optimizers is robust when Assumption 1 is violated. If the empirical result holds regardless, the paper should say so and relegate the theory to a heuristic.","section":"§2"},{"comment":"The abstract's claim of '≈5% increase in training accuracy ... on ... real data' is supported by only one of the three real datasets. In Barcode MNIST, DP-AdamBC reaches 11.5% vs 6.6% for DP-GD on the lowest-frequency group, but the TinyPTB results (Figures 5–6) show no clear advantage for DP-AdamBC on the lowest-frequency tokens, and the E2E results (Figure 7) report only training loss, not accuracy. The paper should either restrict the claim to Barcode MNIST or provide accuracy numbers and a consistent definition of the reported percentage gain across all datasets.","section":"Abstract, Appendix B"}],"minor_comments":[{"comment":"There are several typos in the author affiliation block, including 'V ancouver' and 'T raining' in the workshop name; these should be corrected.","section":"Title page"},{"comment":"The captions for Figures 3(a), 5(a), and 7(a) all describe the plot as 'The class distribution of the synthetic dataset', but these figures show Barcode MNIST, TinyPTB, and E2E, which are not synthetic. The captions should be corrected.","section":"Figures 3, 5, 7"},{"comment":"The axis labels use 'Epsilon' and 'Step'; the paper should use the standard notation epsilon and 'steps' for consistency, and should define what privacy accountant is used to compute epsilon.","section":"Figures 2–11"},{"comment":"The description of the synthetic class sizes is unclear: 'm groups of classes with class sizes equals c = 2^{m+1} − 1' does not match the histogram in Figure 2(a), which shows sizes 8, 128, 256, 512, and 1024. A precise table of the class-size groups would be helpful.","section":"§3"},{"comment":"No code or repository link is provided. Making the code available would substantially improve reproducibility, especially because the hyperparameter tuning procedure and the exact DP accountant details are not fully specified.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short workshop-style manuscript, and the central idea is a reasonable extension of prior work [8, 14]. The main risk is the gamma-prime confound: without an ablation, the headline 'bias correction is crucial' claim is not established. The lack of seeds and code is also a serious hindrance for a journal-level submission, though it may be acceptable for a workshop. The self-citation to [14] is relevant and appropriate; the novelty of the present paper is the application and analysis in the heavy-tail imbalance setting, not the DP-AdamBC algorithm itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper brings together [8]'s heavy-tail imbalance curvature analysis and [14]'s DP-Adam bias correction, and reports that DP-AdamBC gives the best rare-class training accuracy across a synthetic linear model, Barcode MNIST, and TinyPTB. That is a sensible question and a new empirical comparison. But the causal claim that bias correction is 'crucial' is not yet supported: no ablation separates the effect of subtracting the DP noise variance from the effect of the tuned floor γ' in the denominator. There is also no code, no multiple seeds, no error bars, and the theory is a heuristic scaling argument borrowed from [8] rather than a derivation. The abstract overstates things a bit, since the data show DP-GDM (momentum) and even biased DP-Adam also help on rare classes.\n\nWhat's actually good: the comparison is direct, the synthetic setup is clean, and grouping results by class frequency is the right way to expose rare-class behavior. The consistent trend across three datasets—DP-AdamBC ahead on low-frequency classes, DP-GD best on the most frequent class—is a plausible and useful observation. The paper is honest about the milder effect in TinyPTB, which I appreciate.\n\nSoft spots. The stress-test concern about γ' is legitimate. In Algorithm 2, when the true second moment for a rare class is near the noise floor, the update is controlled by the max(..., γ') constant rather than by curvature. The authors tune γ' separately for DP-Adam and DP-AdamBC, so a small γ' alone could produce the large rare-class steps. An ablation that sweeps γ' (or sets it equal to the Adam ε) is needed to pin the effect on bias correction. I would call this a moderate weakness, not a fatal one: DP-AdamBC also beats DP-GDM, and that gap is harder to explain by the floor alone. Still, 'crucial' is too strong until that ablation exists.\n\nThe missing reproducibility artifacts are a problem for an empirical paper. If this goes beyond a workshop, the authors should release code and report variance.\n\nWho it's for: anyone working on DP fine-tuning or private learning under label imbalance. The paper deserves referee time, but it needs an ablation, artifacts, and a more measured abstract before the central claim is credible.","headline":"Plausible claim that DP-AdamBC helps rare classes under heavy-tail imbalance, but the 'crucial' role of bias correction is confounded by the tuned floor γ' and needs an ablation.","tokens_in":12734,"tokens_out":3464,"would_cite":false,"duration_ms":39493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under heavy-tail class imbalance, DP-GD underfits rare classes; DP-AdamBC, which subtracts DP noise from Adam's curvature estimate, lifts training accuracy on the least frequent classes by roughly 8% and 5% in synthetic and real-data…","keywords":["differential privacy","heavy-tail class imbalance","DP-AdamBC","bias correction","Adam optimizer","second-moment curvature estimation","next-token prediction","ill-conditioned optimization"],"falsifier":"Track per-class gradient and Hessian norms during DP training on a heavy-tail dataset while starting from a deliberately low-confidence initialization (correct-class probability near $1/c$). If rare-class gradients do not scale with class frequency when confidence is low, or if DP-AdamBC's rare-class accuracy edge disappears in that regime, the ill-conditioning account fails.","tokens_in":11697,"feed_emoji":"🔒","tokens_out":8952,"duration_ms":98092,"temperature":0.7,"pith_summary":"This paper asks why differentially private training struggles when class labels follow a heavy-tail (Zipf-like) distribution, and whether the optimizer can fix it. It argues that DP-GD's updates inherit the class-frequency scaling of the loss: frequent classes receive large steps in high-curvature regions and rare classes receive tiny steps in flat regions, so rare classes are learned slowly or not at all. Adam can relieve this by normalizing with a curvature estimate, but the Gaussian noise added for DP biases Adam's second-moment estimate so heavily that DP-Adam behaves like DP-SGD. The paper's remedy is DP-AdamBC, which subtracts the known noise variance from the second moment before normalizing; in experiments this raises training accuracy on the least frequent classes by about 8% on a synthetic linear model and about 5% on real data with a CNN. The setting matters because next-token-prediction language models have exactly this heavy-tail label distribution, where private training is most difficult.","feed_headline":"Bias-corrected DP-Adam wins rare classes by ~8%","feed_subtitle":"Subtracting DP noise from Adam's curvature estimate beats DP-GD on least frequent classes, with modest cost on frequent ones.","key_machinery":"The machinery is the DP-AdamBC update rule, which changes Adam's denominator from $\\sqrt{\\hat v_t}+\\gamma$ to $\\sqrt{\\max(\\hat v_t - (\\sigma C/L)^2, \\gamma')}$, subtracting the DP noise variance from the curvature estimate. It is powered by the scaling identities $g_t \\approx (1-p)\\pi_k \\bar x_k$ and $H_t \\approx p(1-p)\\pi_k \\bar H_k$ for class $k$, which hold when the model assigns a large probability $p$ to the correct class and show that the ill-conditioning is set by class frequency. Because the noise variance is identical in every coordinate, it disproportionately contaminates the small second moments of low-frequency classes; removing it is what lets the normalization shrink updates for frequent classes and magnify updates for rare classes, mimicking sign-descent behavior.","core_discovery":"In a stylized linear model with Zipf-distributed labels, the paper shows that both the private gradient and the loss Hessian for class $k$ scale with the class frequency $\\pi_k = n_k/n$, so DP-GD updates are too large for frequent classes and too small for rare classes under a single learning rate. Adam preconditions the gradient by the square root of the second moment of the gradient, a cheap proxy for curvature, but additive DP noise contributes a per-coordinate variance $(\\sigma C/L)^2$ that dominates the second moment and makes the normalization class-frequency-blind. DP-AdamBC repairs the estimate by using $\\max(\\hat v_t - (\\sigma C/L)^2, \\gamma')$ in the denominator, restoring the curvature signal. The paper reports that this bias correction produces the best fit to the least frequent classes in controlled linear experiments (about 8% higher training accuracy than DP-GD on the rarest class group) and on Barcode MNIST with a CNN (about 5% higher on the rarest group), at a modest cost on the most frequent class. Clipping each sample's gradient before aggregation does not remove the ill-conditioning, because gradients within a class remain aligned and still accumulate in proportion to $\\pi_k$.","pith_inferences":["A direct extension is to test DP-AdamBC in full DP fine-tuning of larger language models; the paper's TinyPTB results suggest pretrained representations mute the effect, so a randomly initialized head or a model without pretrained embeddings should show a larger rare-token gap if the class-frequency mechanism is correct.","The denominator $\\max(\\hat v_t - (\\sigma C/L)^2, \\gamma')$ suggests a practical rule beyond Adam: any DP optimizer that estimates a second moment from noisy gradients should subtract the known noise variance before using that moment for normalization or step-size adaptation.","One could falsify the mechanism by varying only the confidence assumption: train the linear model from a deliberately low-confidence initialization where the correct-class probability $p$ is small, and check whether the per-class gradient scaling and DP-AdamBC's rare-class advantage both disappear."],"forward_implications":["Under a fixed privacy budget and learning rate, DP-GD will underfit low-frequency classes, whereas DP-AdamBC reduces this bias: in the synthetic linear setting the rarest class group reached about 9.5% training accuracy with DP-AdamBC versus roughly 9 percentage points less for DP-GD.","The advantage is concentrated on rare classes; on the highest-frequency class DP-GD can reach 100% training accuracy where DP-AdamBC reaches about 87.5%, so the optimizer choice trades a little frequent-class accuracy for large rare-class gains.","Gradient clipping cannot fix the ill-conditioning by itself, because per-sample clipped gradients stay aligned within a class and sum proportionally to class frequency; curvature information must be estimated accurately instead.","The value of bias correction grows with the DP noise level: with a smaller noise multiplier ($\\sigma=5$ rather than 10), plain DP-Adam approaches DP-AdamBC, whereas with large $\\sigma$ or large clipping norm $C$, DP-Adam degrades toward DP-SGD behavior.","On real next-token prediction data (TinyPTB), the paper observes a similar but milder pattern, with DP-AdamBC highest for medium- and high-frequency tokens and all optimizers struggling equally on the rarest tokens."],"supporting_citations":[{"why":"Supplies the heavy-tail class imbalance model and the assumption that gradient and Hessian scale with class frequency, which the paper's linear-model analysis adopts.","marker":"[8]"},{"why":"Introduces DP-AdamBC and the observation that DP noise dominates Adam's second-moment estimate, the correction the paper claims is necessary.","marker":"[14]"},{"why":"Defines DP-(S)GD with per-sample clipping and Gaussian noise, the baseline private optimizer whose ill-conditioning is the paper's target.","marker":"[1]"},{"why":"Defines Adam and its second-moment preconditioning, which the paper uses as the cheap proxy for loss curvature.","marker":"[6]"},{"why":"Provides the E2E restaurant-domain dataset used in the real-data next-token prediction experiments.","marker":"[12]"},{"why":"Supplies the DP fine-tuning implementation and LoRA hyperparameters used for the E2E experiments.","marker":"[16]"}],"fun_headline_variants":["Bias-corrected DP-Adam lifts rare-class accuracy 8%","DP-AdamBC: subtract DP noise, gain 8% on rare classes","Private Adam's curvature fix beats GD on rare labels","Heavy-tail imbalance? DP-AdamBC corrects curvature bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument depends on a stated assumption, adopted without verification: the model assigns a large output probability to the correct class, so gradients and Hessians scale linearly with class frequency; early in training or for rare classes the model has not learned, this can fail and the predicted benefit of curvature normalization may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Bias-corrected DP-Adam lifts rare-class accuracy 8%","DP-AdamBC: subtract DP noise, gain 8% on rare classes","Private Adam's curvature fix beats GD on rare labels","Heavy-tail imbalance? DP-AdamBC corrects curvature bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1477,"prompt_tokens":911,"completion_tokens":566,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":489}},"tokens_in":527,"tokens_out":566,"duration_ms":6579,"temperature":1.0,"reasoning_tokens":489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:28:42.222303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track per-class gradient and Hessian norms during DP training on a heavy-tail dataset while starting from a deliberately low-confidence initialization (correct-class probability near $1/c$). If rare-class gradients do not scale with class frequency when confidence is low, or if DP-AdamBC's rare-class accuracy edge disappears in that regime, the ill-conditioning account fails.","supporting_citations":[{"cited_title":"DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)","cited_arxiv_id":"2312.14334","evidence_quote":"Introduces DP-AdamBC and the observation that DP noise dominates Adam's second-moment estimate, the correction the paper claims is necessary."}],"review_version":1}