REVIEW 1 major objections 1 minor 12 references
DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum
T0 review · 1 major / 1 minor · reviewed 2026-06-28 · grok-4.3
Pith's one-line read DP-MacAdam reuses the same gradient mean and variance estimates for both adaptive clipping and momentum updates.
desk verdict DP-MacAdam reuses the same mean/variance estimates for adaptive clipping and momentum, with an analysis claiming bias-free variance and empirical gains over separate baselines. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
DP-MacAdam algorithm that applies the same empirical gradient mean and variance estimates to both adaptive clipping and momentum acceleration.
What would settle it
An experiment that measures nonzero bias in the variance estimates produced by DP-MacAdam on controlled synthetic gradients, or a trial showing that its accuracy gains vanish when separate estimates are used for clipping and momentum.
Extended reading notes
Core claim
DP-MacAdam estimates the gradient variances in a bias-free manner by reusing the same empirical mean and variance estimates for both clipping and momentum, leading to improved model utility in differentially private settings without requiring manual tuning of the clipping threshold.
Load-bearing premise
Reusing identical empirical mean and variance estimates for clipping and momentum preserves differential privacy and introduces neither bias nor instability.
Editorial extensions
If this is right
- Higher test accuracy than DP-SGD, AdaClip, and DP-Adam on standard benchmarks.
- No manual selection of clipping threshold is required.
- Variance estimates remain unbiased under the shared computation.
- Differential privacy guarantees hold when the estimates are reused.
Reading between the lines
- The shared-estimate design could reduce the number of gradient passes needed per step.
- Similar reuse might apply to other first-order adaptive methods beyond Adam.
- The approach may simplify hyper-parameter search in large-scale private training pipelines.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DP-MacAdam, which unifies adaptive clipping (via shift/scale parameters derived from empirical gradient mean and variance) with Adam-style momentum by reusing the same estimates for both. It claims an analysis establishing bias-free variance estimation and reports empirical results showing improved model utility over DP-SGD, AdaClip, and DP-Adam baselines without manual clipping-threshold tuning.
Significance. If the bias-free claim can be established rigorously despite the shared estimates, the work would reduce a key source of hyperparameter sensitivity in DP training and offer a practical unification of two lines of adaptive DP-SGD research. The direct comparison to three baselines is a positive feature; significance is otherwise limited by the narrow scope of the theoretical analysis.
major comments (1)
- [§4] §4 (bias-free variance analysis): the derivation treats the clipping threshold as exogenous to the gradient statistics, yet the algorithm applies the threshold (computed from the same empirical mean/variance) before the variance estimator is updated. This creates a dependence that standard unbiasedness arguments do not cover; the post-clipping expectation is no longer unconditional. The central claim that the variance estimator remains bias-free therefore requires an explicit correction or conditioning argument that is not supplied.
minor comments (1)
- [§5] The experimental section should report the exact clipping-norm schedule and privacy-accounting parameters used for each baseline so that the “no manual tuning” claim can be reproduced.
Simulated Author's Rebuttal
We thank the referee for the careful reading of the manuscript and for highlighting the need to strengthen the conditioning argument in the bias-free variance analysis. We address the single major comment below.
read point-by-point responses
-
Referee: [§4] §4 (bias-free variance analysis): the derivation treats the clipping threshold as exogenous to the gradient statistics, yet the algorithm applies the threshold (computed from the same empirical mean/variance) before the variance estimator is updated. This creates a dependence that standard unbiasedness arguments do not cover; the post-clipping expectation is no longer unconditional. The central claim that the variance estimator remains bias-free therefore requires an explicit correction or conditioning argument that is not supplied.
Authors: We agree that the current write-up of the bias-free claim in §4 would benefit from an explicit conditioning argument. In DP-MacAdam the shift and scale parameters (hence the clipping threshold) are computed from the exponential moving-average estimates of mean and variance that were formed at the end of the previous iteration. Consequently, at step t the threshold is a deterministic function of the history up to t−1 and is therefore exogenous with respect to the fresh gradient sample g_t. The expectation that establishes unbiasedness of the variance estimator can therefore be taken conditionally on the filtration F_{t−1}; the post-clipping gradient is then an unbiased (scaled) version of the conditional expectation of g_t given F_{t−1}. We will insert this short conditioning paragraph into the revised §4 and will also add a one-sentence remark in the algorithm description to make the temporal separation explicit. revision: yes
Circularity Check
No circularity: bias-free variance claim rests on separate analysis, not self-definition or fitted inputs.
full rationale
The paper's central claim is an analysis establishing bias-free gradient variance estimation in DP-MacAdam, which reuses empirical mean/variance for both clipping and momentum. No quoted equations or self-citations reduce this result to its inputs by construction; the abstract presents the bias-free property as arising from independent analysis rather than tautological fitting or renaming. The derivation chain is self-contained against external benchmarks, with empirical comparisons to baselines providing falsifiable support outside any internal fit.
Assumptions & free parameters
Cite this review
Pith. "Pith review of DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum." pith.science (2026). https://pith.science/paper/XLJMJYLA
@misc{pith2026260605435,
author = {Pith},
title = {Pith review of: DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum},
year = {2026},
howpublished = {\url{https://pith.science/paper/XLJMJYLA}},
note = {Machine review of arXiv:2606.05435}
}
read the original abstract
Differentially private stochastic gradient descent (DP-SGD) has become the standard framework for privacy-preserving machine learning, yet its reliance on a fixed gradient clipping threshold to limit sensitivity remains a significant practical limitation. Adaptive clipping algorithms such as AdaClip shift and scale the gradient prior to clipping and adding noise so that the clipped gradient yields a more informative descent direction. The shift and scaling parameters are selected adaptively based on the empirical mean and variance. However, in existing adaptive clipping algorithms, these empirical estimates have not been also used for momentum to accelerate training itself. On the other hand, DP-Adam is an algorithm that exploits Adam-like momentum updates based on the gradient mean and variance to accelerate training, but does not exploit these estimates for adaptive clipping. In this work, we propose Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum (DP-MacAdam), a novel algorithm that combines these two approaches so as to use the same mean and variance estimates for both clipping and momentum. We perform an analysis showing that DP-MacAdam estimates the gradient variances in a bias-free manner. In addition, we empirically evaluate the privacy and accuracy of DP-MacAdam, demonstrating that it achieves improved model utility compared to DP-SGD, AdaClip, and DP-Adam baselines, without requiring manual tuning of the clipping threshold.
Reference graph
Works this paper leans on
-
[1]
Calibrating Noise to Sensitivity in Private Data Analysis,
C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating Noise to Sensitivity in Private Data Analysis,” inTheory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3. Springer, 2006, pp. 265–284
2006
-
[2]
Deep Learning with Differential Privacy,
M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep Learning with Differential Privacy,” inProceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 308–318
2016
-
[3]
Adam: A Method for Stochastic Optimization
D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980, 2014
work page Pith review arXiv 2014
-
[4]
An improvement of the convergence proof of the ADAM-Optimizer
S. Bock, J. Goppold, and M. Weiß, “An Improvement of the Convergence Proof of the ADAM-Optimizer,”arXiv preprint arXiv:1804.10587, 2018
work page Pith review arXiv 2018
-
[5]
DP-AdamBC: Your DP- Adam Is Actually DP-SGD (Unless You Apply Bias Correction),
Q. Tang, F. Shpilevskiy, and M. Lécuyer, “DP-AdamBC: Your DP- Adam Is Actually DP-SGD (Unless You Apply Bias Correction),” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 14, 2024, pp. 15 276–15 283
2024
-
[6]
AdaCliP: Adaptive Clipping for Private SGD,
V . Pichapati, A. T. Suresh, F. X. Yu, S. J. Reddi, and S. Ku- mar, “AdaCliP: Adaptive Clipping for Private SGD,”arXiv preprint arXiv:1908.07643, 2019
-
[7]
TensorFlow Privacy,
TensorFlow Privacy Authors, “TensorFlow Privacy,” https://github.com/ tensorflow/privacy/tree/master/tensorflow_privacy/privacy/optimizers, 2019, accessed: 2026, dp_optimizer.py, line 386
2019
-
[8]
DP-AdamW: Investigating De- coupled Weight Decay and Bias Correction in Private Deep Learning,
J. Chooi, K. Cong, R. Li, and L. Sun, “DP-AdamW: Investigating De- coupled Weight Decay and Bias Correction in Private Deep Learning,” arXiv preprint arXiv:2511.07843, 2025
Show all 12 references
-
[9]
GeoClip: Geometry- Aware Clipping for Differentially Private SGD,
A. Gilani, N. Tasnim, L. Sankar, and O. Kosut, “GeoClip: Geometry- Aware Clipping for Differentially Private SGD,”Advances in Neural Information Processing Systems, vol. 38, pp. 6233–6257, 2026
2026
-
[10]
Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributions,
V . Doroshenko, B. Ghazi, P. Kamath, R. Kumar, and P. Manurangsi, “Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributions,” inPrivacy Enhancing Technologies Symposium (PETS), 2022
2022
-
[11]
Gradient-Based Learning Applied to Document Recognition,
Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,”Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998
1998
-
[12]
Learning Multiple Layers of Features from Tiny Im- ages,
A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Im- ages,” University of Toronto, Tech. Rep., 2009
2009
Reviewed June 28, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.