Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

On Design Principles for Private Adaptive Optimizers

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that trying to keep Adam's second-moment estimate unbiased under differential privacy is the wrong design goal, and that scale-then-privatize, which clips and noises gradients in the optimizer's own geometry, is…

desk verdict Credible empirical case that scale-then-privatize beats unbiased-moment variants on a small transformer, but the theory has a constant-gradient gap and an arithmetic error in the regime threshold. read the letter →

arxiv 2507.01129 v1 pith:W3C7RDA5 submitted 2025-07-01 cs.LG cs.CR

Arun Ganesh , Brendan McMahan , Abhradeep Thakurta This is my paper
classification cs.LGcs.CR
keywords differentialprivacyadaptiveoptimizersAdamsecond-momentestimationbiascorrectionscale-then-privatizecorrelatednoisematrixfactorizationmechanisms
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Differentially private training corrupts the second-moment estimate $\nu$ that Adam and AdaGrad use to set per-coordinate learning rates, and several recent optimizers try to repair it by making $\nu$ unbiased. This paper argues that unbiasedness is the wrong design goal: it cannot separate the variance that comes from the data from the variance injected by privacy noise, and in high dimensions it drives a constant fraction of $\nu$'s coordinates negative, forcing a choice between unstable large learning rates, slowed progress, or re-introducing the very bias being corrected. The paper instead studies scale-then-privatize, which clips and noises each gradient in the non-isotropic geometry defined by the previous $\hat{\nu}$, so the noisy gradient distribution matches the noiseless one. In a 4.4-million-parameter transformer fine-tuning task, scale-then-privatize outperforms post-processing, bias correction, and independent moment estimation, while the 'unbiased' methods that win in one dimension degrade badly, evidence that conclusions from low-dimensional experiments may not transfer to realistic model training.

What carries the argument

The central object is $\hat{\nu}$, Adam's decayed, decay-normalized average of squared gradients, which acts as a per-coordinate preconditioner through the learning rates $1/\sqrt{\hat{\nu}}$. The mechanism that carries the argument is scale-then-privatize (Algorithm 8): using the previous round's $\hat{\nu}_{t-1}$ it defines a coordinate scaling $s_t = 1/(\sqrt{\hat{\nu}_{t-1}} + \varepsilon_{s1})$, clips the scaled gradients $s_t \odot g$ in $\ell_2$ norm, adds isotropic Gaussian noise in that scaled space, then divides back by $s_t$ before the update, equivalently, clipping to an ellipsoid and adding noise shaped like it. Two quantitative results do the theoretical work: the steady-state contraction of $\mathbb{E}[\hat{\nu}_t]$ toward a multiple of the noiseless $\hat{\nu}^*$ (Section 4.3.1), and the variance lower bound on unbiased $\hat{\nu}$ that yields the regime inequality separating low-dimensional settings where unbiased methods help from high-dimensional settings where they produce many negative coordinates (Section 4.4.1). A third result connects the two: after warmup, $\sqrt{\hat{\nu}_j/\hat{\nu}_{j-1}} \approx 1$ under scale-then-privatize, making the effective update noise nearly isotropic so that offline DP-MF noise schedules for SGD remain near-optimal.

What would settle it

On a synthetic quadratic with a drifting Hessian (so per-coordinate gradient magnitudes change over time), track whether $\mathbb{E}[\hat{\nu}_t]$ under scale-then-privatize stays proportional to the noiseless $\hat{\nu}^*$: if a coordinate whose recent gradients are small still has $\hat{\nu}$ inflated toward the steady state $g^2/(1 - \sigma^2/B^2)$, the contraction fails. Equally decisive: run the same private-Adam comparison at a dimension low enough that $B/\sigma \gtrsim 0.15\sqrt{m}$ holds and check whether bias correction overtakes scale-then-privatize, since the paper's regime prediction says it should.

Watch

Extended reading notes

Core claim

The paper's central claim is that the prevailing design goal for private adaptive optimizers, making the second-moment estimate $\hat{\nu}$ unbiased, is misguided, and that scale-then-privatize is a better design. For a fixed gradient $g$ (with $\zeta = 1$, $\beta_2 = 0.999$), scale-then-privatize's $\hat{\nu}$ converges in expectation to a steady state proportional to the noiseless value $\hat{\nu}^* = g^2$, namely $g^2/(1 - \sigma^2/B^2)$, with the multiplicative error shrinking by a factor $0.999 + 0.001\,\sigma^2/B^2$ each round. Unbiased estimators, by contrast, carry variance at least $(g^2\sigma^2/B^2 + 2\sigma^4/B^4)/1999$, large enough that in dimension $m$ a constant fraction of $\hat{\nu}$'s coordinates go negative unless the batch-to-noise ratio satisfies $B/\sigma \gtrsim 0.15\sqrt{m}$ under optimistic assumptions, or $\sqrt{B}/\sigma \gtrsim 0.15\sqrt{m}$ under pessimistic ones; the unavoidable corrections either destabilize learning, slow it down, or reproduce the original bias. The TinyBERT experiments confirm the regime picture: roughly 45 to 50 percent of bias-corrected coordinates are negative, scale-then-privatize beats every surveyed variant, and a noiseless-preconditioner ablation helps only when combined with scale-then-privatize, because only then do the noiseless and noisy geometries match. The paper also argues that scale-then-privatize makes the noise on Adam's updates roughly isotropic, so correlated-noise (DP-MF) schedules designed offline for DP-SGD carry over to private Adam.

Load-bearing premise

The theoretical case for scale-then-privatize rests on a derivation that assumes the true gradient is the same fixed vector $g$ in every round, with $\zeta = 1$ and $\beta_2 = 0.999$; in real training the gradients move, and if the steady-state contraction does not extend to non-stationary gradients, the method's advantage rests only on the single TinyBERT experiment.

Editorial extensions

If this is right

  • Bias correction and independent moment estimation should be expected to lose their advantage as model dimension grows, so their reported gains on low-dimensional benchmarks like CIFAR, SNLI, and QNLI may not transfer to token-prediction language models.
  • Scale-then-privatize recovers, up to a constant factor rescaled into the learning rate, the preconditioner that non-private Adam would compute on the same gradient stream, restoring adaptivity that black-box DP-Adam loses.
  • With scale-then-privatize, practitioners can reuse noise-correlating matrices optimized offline for DP-SGD (such as banded matrix factorization mechanisms) for private Adam with little loss, avoiding the need to re-solve the matrix factorization for the adaptive optimizer.
  • The regime condition comparing $B/\sigma$ with $\sqrt{m}$ gives a concrete rule of thumb: below the threshold, unbiased-moment methods may genuinely help (as in the paper's one-dimensional logistic regression); above it, they risk roughly half of preconditioner coordinates going negative and worse training.
  • The paper's observation that noise-mechanism RMSE (root-mean-square error of its prefix sums) does not predict utility for post-processing private Adam implies utility comparisons among DP mechanisms must be re-run per optimizer rather than inherited from DP-SGD.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The steady-state argument assumes a constant gradient $g$; a natural extension would test whether proportional recovery of $\hat{\nu}^*$ survives on non-stationary streams where per-coordinate gradient magnitudes drift, since the TinyBERT experiment alone cannot distinguish the theory from a favorable empirical coincidence.
  • Because the experiments fix the noise multiplier rather than the end-to-end $\varepsilon$ (privacy amplification from subsampling is explicitly ignored), an inference, not the paper's claim, is that a full privacy-accounted comparison could reorder the variants; the practical recommendation should be re-validated with amplification and user-level adjacency included.
  • Scale-then-privatize reinterprets the clip norm as a bound in a moving preconditioned geometry; a testable extension is to adapt the clip norm or the stability floor $\varepsilon_{s1}$ per coordinate to control worst-case update sizes, which the paper does not explore.
  • The correlated-noise argument suggests a concrete prediction: for scale-then-privatize, the RMSE of the effective per-update noise, rather than the gradient-level noise, should predict utility, a quantity the paper names but does not compute.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper surveys several differentially private (DP) variants of adaptive optimizers such as Adam and Adagrad, develops theoretical intuition for their behavior, and compares them empirically on a one-dimensional logistic regression task and on masked token prediction with TinyBERT. The central claim is that the commonly pursued goal of unbiased estimation of the second moment of gradients is misguided in high-dimensional settings, and that a scale-then-privatize variant—which clips and noises gradients in the geometry defined by the current preconditioner—has more desirable theoretical properties and outperforms the other variants on the TinyBERT task. The paper also argues that scale-then-privatize is better suited than post-processing for use with correlated noise mechanisms (DP-MF).

Significance. If the claims hold, the paper reframes an active design question for private adaptive optimization: it challenges the prevailing emphasis on unbiased second-moment estimates and provides a concrete, simple alternative. The empirical comparison is a genuine strength: TinyBERT is a non-trivial transformer task, each configuration is run 30 times, and the reported standard deviations are at most 0.001 for most methods, so the rank ordering among variants is well supported. The theoretical sections are unusually honest about their idealizations (constant gradients, large t, specific settings of beta_2 and zeta), and the paper includes useful baselines such as a noiseless preconditioner. However, one of the central quantitative regime claims contains a large arithmetic error, and the main theoretical justification for scale-then-privatize is proven only in a constant-gradient setting that does not directly cover the tested non-stationary training regime. These issues are fixable, but they currently weaken the paper's strongest conceptual claims.

major comments (3)
  1. [Section 4.4.1, Eq. (2) and (3), and the example with B = 2048, sigma = 1] The numerical claims about the optimistic and pessimistic regimes are arithmetically incorrect. For Eq. (2), B/sigma >= 0.15 sqrt(m) with B = 2048, sigma = 1 gives sqrt(m) <= 2048/0.15, i.e. m <= (2048/0.15)^2 ≈ 1.9e8, not m ≲ 10^6 as stated. The claimed boundary is off by roughly two orders of magnitude. For Eq. (3), sqrt(B)/sigma >= 0.15 sqrt(m) gives m <= (sqrt(2048)/0.15)^2 ≈ 9.1e4, not the stated m ≲ 2e4. This is not a cosmetic typo: under the corrected optimistic bound, the TinyBERT model with m ≈ 4.4e6 is comfortably inside the regime where bias correction should rarely produce negative coordinates, yet Fig. 4 reports 45–50% negative coordinates. The paper's statement that these bounds predict the TinyBERT result is therefore not supported by the theory as written. The negative-coordinate observation remains empirically interesting, but it must be presented as an empirical finding or be backed by a corrected calculation.
  2. [Section 4.3.1 and Appendix B, Eq. (4)-(5)] The derivation that scale-then-privatize retrieves a preconditioner proportional to the noiseless one assumes g_t is a fixed vector g in every round, with zeta = 1, beta_2 = 0.999, and beta_2^t ≈ 0. In the TinyBERT experiment the gradients are non-stationary, and with beta_2 = 0.999 the estimator nu_t is roughly a 1000-step moving average. The paper gives no tracking bound of the form ||E[nu_t] - c_t nu*_t|| for time-varying g_t, and the noise term in Eq. (4) has variance proportional to nu_{t-1}, so a lagging nu can distort the scaling s_t that is applied before adding noise. Thus the theoretical reason scale-then-privatize should outperform post-processing in the tested setting is not established; only the empirical result in Section 5.2 remains. The authors should either supply a non-stationary analysis (even in a simplified model) or explicitly scope the theoretical claim to the constant-gradient regime.
  3. [Section 4.5] The argument that scale-then-privatize makes correlated noise (DP-MF) as effective as in the non-adaptive setting relies on the approximation sqrt(nu_j / nu_{j-1}) ≈ 1. No quantitative bound is given for how close this ratio is to 1 when gradients change, and the ratio depends on the data through the noisy squared-gradient term. The text partially acknowledges this ("reasonably confident"), but the takeaway box states that scale-then-privatize "permits an offline objective that reasonably approximates the true objective." As written, this is a heuristic rather than a proven design property. Please either provide a bound on the deviation of sqrt(nu_j / nu_{j-1}) from 1 in a tractable setting, or soften the takeaway so it does not overclaim the theoretical support for the DP-MF benefit.
minor comments (5)
  1. [Section 5.1] There is a typo in "modern large-scale DP training pipelnies" — "pipelnies" should be "pipelines."
  2. [Algorithm 8 and Section 5.2] The algorithm uses two stability constants, epsilon_{s1} for the scaling step and epsilon_{s2} for the final Adam update, but the text does not clearly specify which of these was tuned in each experiment; please state the tuned values or explain how they were set.
  3. [Section 4.4.1 and Appendix C] The notation "2 · 104" and "5 · 105" is ambiguous; these should be typeset as 2 x 10^4, 5 x 10^5, etc., to avoid confusion with a decimal point.
  4. [Section 5.2, Fig. 4-6] The captions for Figures 4-6 are minimal; the reader must infer from the text that the x-axis is the training round and the y-axis is the fraction of negative coordinates (Fig. 4), the normalized gradient norm (Fig. 5), and the value of nu (Fig. 6). Adding axis labels and a short caption would improve readability.
  5. [Abstract and Section 1.1] The abstract and contribution bullet list say scale-then-privatize "outperforms all other variants we study," which is supported for the TinyBERT task but only for one task; the claim would be more precise if it stated the empirical scope explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: design principles are derived from explicit idealized recurrences and checked against external baselines; overlapping-author citations are background or credited heuristics, not load-bearing.

full rationale

The load-bearing derivations are self-contained. Section 4.3.1 and Appendix B start from Algorithm 8's recurrence (Eqs. (1) and (4)) and solve for an expectation fixed point proportional to g^2 under the stated constant-gradient, large-t, beta2 = 0.999 assumptions; the fixed point is a property of the defined scaling rule, not an input relabeled as an output. Section 4.4.1 computes the variance of the unrolled bias-corrected estimator and derives the B/sigma thresholds directly; the thresholds are not fitted to the TinyBERT measurements, and the negative-coordinate fractions are reported as empirical checks. Section 4.5 derives the near-one scaling ratio from the beta2 coefficient before invoking DP-MF as a tool. References to the authors' own DP-MF and delayed-preconditioner work define mechanisms or supply heuristics (e.g., using an SGD-momentum-optimized C^{-1}), but the central argument does not reduce to those citations; the empirical section compares against post-processing, independent moment estimation, bias correction, SGD, and noiseless-preconditioner baselines, and reports direct measurements. The main caveats are rigor issues rather than circularity: the scale-then-privatize theory assumes a constant gradient and gives no tracking bound for non-stationary TinyBERT gradients, and the numerical translation of Eq. (2) to m ~ 10^6 appears inconsistent with B/sigma = 2048. These affect correctness, not circular structure.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims do not rely on fitted free parameters; the steady-state and variance analyses are parameter-free under stated idealizations. The empirical comparisons do depend on tuned learning rates and stability constants, which are listed here. No new scientific entities are postulated.

free parameters (4)
  • learning rate η per method = tuned, not reported
    Each variant's learning rate is tuned separately in Section 5; the reported test losses depend on these choices. No values are listed.
  • stability constant εs for bias correction and independent moment estimation = 10^-4 (best in all settings)
    Tuned; the paper argues this floor effectively reintroduces the post-processing bias, which is part of the empirical argument in Section 5.2.
  • clip norm ζ for scale-then-privatize = re-tuned, not reported
    The paper re-tunes clip norm because clipping is no longer in the ℓ2 norm; the exact value is not listed.
  • noise multipliers σ in TinyBERT experiment = 0.134 and 1.0
    Chosen to equate the RMSE of prefix sums and example-level ε with DP-BAND-MF; this is an experimental design choice, not derived from theory.
assumptions (4)
  • domain assumption Fixed-gradient idealization g_t = g for all t, ζ=1, β2=0.999, β2^t≈0
    Used in Section 4.3.1 to derive the steady-state proportionality of scale-then-privatize; the paper acknowledges this is a simple setting.
  • domain assumption Gradient-norm regimes for batch averages: either ||g_t||=ζ (optimistic) or ||g_t||≤ζ/√B (pessimistic)
    Underpins Eqs. (2) and (3) in Section 4.4.1; real gradients lie between these extremes and the measured value is about 0.27.
  • domain assumption sqrt(ν_j/ν_{j-1})≈1 for large t
    Used in Section 4.5 to argue that DP-MF objectives become offline and coordinate-independent; the paper calls this a reasonable approximation but does not prove uniform bounds.
  • standard math Zero-out adjacency and Gaussian mechanism for DP guarantees
    Standard DP background from [21]; assumed throughout the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Design Principles for Private Adaptive Optimizers." pith.science (2026). https://pith.science/paper/W3C7RDA5

@misc{pith2026250701129,
  author       = {Pith},
  title        = {Pith review of: On Design Principles for Private Adaptive Optimizers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W3C7RDA5}},
  note         = {Machine review of arXiv:2507.01129}
}
read the original abstract

The spherical noise added to gradients in differentially private (DP) training undermines the performance of adaptive optimizers like AdaGrad and Adam, and hence many recent works have proposed algorithms to address this challenge. However, the empirical results in these works focus on simple tasks and models and the conclusions may not generalize to model training in practice. In this paper we survey several of these variants, and develop better theoretical intuition for them as well as perform empirical studies comparing them. We find that a common intuition of aiming for unbiased estimates of second moments of gradients in adaptive optimizers is misguided, and instead that a simple technique called scale-then-privatize (which does not achieve unbiased second moments) has more desirable theoretical behaviors and outperforms all other variants we study on a small-scale language model training task. We additionally argue that scale-then-privatize causes the noise addition to better match the application of correlated noise mechanisms which are more desirable to use in practice.

Figures

Figures reproduced from arXiv: 2507.01129 by the authors.

Figure 1
Figure 1. Pseudocode for ADAM and its private variants. 3.1 Post-Processing (“Standard” DP-ADAM) Arguably the simplest and most common way to make ADAM private is post-processing (Algorithm 5): privatize the computation of gt, and then use the privatized gt in ADAM in a black-box manner. The main issue with the algorithm is that while the Gaussian noise is mean-zero and hence µt is unbiased, νt is biased. In particular, if ν … view at source ↗
Figure 2
Figure 2. Test loss of different variants of private A [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Test loss of different variants of private A [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: The norm of the averaged clipped gradient for [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: The values of non-private ν compared to the bias correction term. Finally, we note that unlike what was observed in past works for DP-SGD, for DP-ADAM the RMSE of the noising mechanism is not a good predictor of learning performance, in spite of the observations made i…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Correlating Cross-Iteration Noise for DP-SGD using Model Curvature

    cs.LG 2025-10 conditional novelty 7.0 of 10

    Using Hessian eigenvalues from public data to design correlated noise for DP-SGD improves accuracy by 1–4% over current DP-MF methods.

Reference graph

Works this paper leans on

25 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Goodfellow, H

    Mart ´ın Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proc. of the 2016 ACM SIGSAC Conf. on Computer and Communications Security (CCS’16), pages 308–318, 2016

    work page 2016

  2. [2]

    Private adaptive gradient methods for 20 convex optimization

    Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar. Private adaptive gradient methods for 20 convex optimization. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 383–392. PMLR, 18–24 Jul 2021. URL https://proceed...

    work page 2021

  3. [3]

    Adapting to function difficulty and growth conditions in private optimization

    Hilal Asi, Daniel Asher Nathan Levy, and John Duchi. Adapting to function difficulty and growth conditions in private optimization. In Advances in Neural Information Processing Systems, 2021

    work page 2021

  4. [4]

    Private empirical risk minimization: Efficient algorithms and tight error bounds

    Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Proc. of the 2014 IEEE 55th Annual Symp. on Foundations of Computer Science (FOCS), pages 464–473, 2014

    work page 2014

  5. [5]

    Brendan McMahan, Nicole Mitchell, Krishna Pillutla, and Keith Rush

    Zachary Charles, Arun Ganesh, Ryan McKenna, H. Brendan McMahan, Nicole Mitchell, Krishna Pillutla, and Keith Rush. Fine-tuning large language models with user-level differential privacy, 2024. URL https:// arxiv.org/abs/2407.07737

    arXiv 2024

  6. [6]

    Multi-epoch matrix factorization mechanisms for private machine learning

    Christopher A Choquette-Choo, H Brendan McMahan, Keith Rush, and Abhradeep Thakurta. Multi-epoch matrix factorization mechanisms for private machine learning. arXiv preprint arXiv:2211.06530, 2022

    arXiv 2022

  7. [7]

    (amplified) banded matrix factorization: A unified approach to private training

    Christopher A Choquette-Choo, Arun Ganesh, Ryan McKenna, H Brendan McMahan, John Rush, Abhradeep Guha Thakurta, and Zheng Xu. (amplified) banded matrix factorization: A unified approach to private training. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  8. [8]

    Improved dif- ferential privacy for sgd via optimal private linear operators on adaptive streams.Advances in Neural Information Processing Systems, 35:5910–5924, 2022

    Sergey Denisov, H Brendan McMahan, John Rush, Adam Smith, and Abhradeep Guha Thakurta. Improved dif- ferential privacy for sgd via optimal private linear operators on adaptive streams.Advances in Neural Information Processing Systems, 35:5910–5924, 2022

    work page 2022

Show all 25 references
  1. [9]

    BERT: Pre-training of deep bidirec- tional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter ...

    2019

  2. [10]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011

    2011

  3. [11]

    Calibrating noise to sensitivity in private data analysis

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proc. of the Third Conf. on Theory of Cryptography (TCC), pages 265–284, 2006. URL http://dx.doi.org/10.1007/11681878_14

    2006 doi

  4. [12]

    Practical and private (deep) learning without sampling or shuffling

    Peter Kairouz, Brendan McMahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling or shuffling. In ICML, 2021

    2021

  5. [13]

    Kalinin, Jalaj Upadhyay, and Christoph H

    Nikita P. Kalinin, Jalaj Upadhyay, and Christoph H. Lampert. Continual release moment estimation with differ- ential privacy, 2025. URL https://arxiv.org/abs/2502.06597

    2025 arXiv

  6. [14]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL https://api.semanticscholar.org/CorpusID:6628106

    2014 arXiv

  7. [15]

    Private adaptive optimization with side information

    Tian Li, Manzil Zaheer, Sashank Reddi, and Virginia Smith. Private adaptive optimization with side information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine ...

    2022

  8. [16]

    Reddi, Hugh Brendan McMahan, and Virginia Smith

    Tian Li, Manzil Zaheer, Ken Liu, Sashank J. Reddi, Hugh Brendan McMahan, and Virginia Smith. Differen- tially private adaptive optimization with delayed preconditioners. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/foru...

    2023

  9. [17]

    Scaling up the banded matrix factorization mechanism for differentially private ml, 2025

    Ryan McKenna. Scaling up the banded matrix factorization mechanism for differentially private ml, 2025. URL https://arxiv.org/abs/2405.15913

    2025 arXiv

  10. [18]

    Brendan McMahan and Matthew Streeter

    H. Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. In Proceedings of the 23rd Annual Conference on Learning Theory (COLT), 2010

    2010

  11. [19]

    Adaptive bound optimization for online convex optimization

    H Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. arXiv preprint arXiv:1002.4908, 2010

    2010 arXiv

  12. [20]

    A hassle-free algorithm for strong differential privacy in federated learning systems

    Hugh Brendan McMahan, Zheng Xu, and Yanxiang Zhang. A hassle-free algorithm for strong differential privacy in federated learning systems. In Franck Dernoncourt, Daniel Preot ¸iuc-Pietro, and Anastasia Shimorina, editors, Proceedings of the 2024 Conference on Empirical Methods...

    2024

  13. [21]

    Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta

    Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H. Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta. How to dp-fy ml: A practical guide to machine learning with differential privacy. Journal of Artificial Intelligence...

    2023 doi

  14. [22]

    Stochastic gradient descent with differentially private updates

    Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, pages 245–248. IEEE, 2013

    2013

  15. [23]

    Dp-adambc: Your dp-adam is actually dp-sgd (unless you apply bias correction), 2023

    Qiaoyue Tang, Frederick Shpilevskiy, and Mathias L ´ecuyer. Dp-adambc: Your dp-adam is actually dp-sgd (unless you apply bias correction), 2023. URL https://arxiv.org/abs/2312.14334

    2023 arXiv

  16. [24]

    Position: Considerations for differentially private learn- ing with large-scale public pretraining

    Florian Tram `er, Gautam Kamath, and Nicholas Carlini. Position: Considerations for differentially private learn- ing with large-scale public pretraining. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenka...

    2024

  17. [25]

    steady state

    Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstructing what makes a good optimizer for language models, 2025. URL https://arxiv.org/abs/2407.07972. A Bias Correction for DP-MF We focus on the one-dimensional case, i.e. the gradients are s...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.