Pith. sign in

REVIEW 4 major objections 4 minor 60 references

Curvature-aware noise correlations improve DP-SGD accuracy by 1–4% at the same privacy budget.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:19 UTC pith:DZE5M26N

load-bearing objection Genuinely new curvature-aware DP-MF objective with a clean quadratic theorem, but the reported privacy budget ignores the cost of tuning on a private validation split, so the equal-privacy comparison is optimistic. the 4 major comments →

arxiv 2510.05416 v3 pith:DZE5M26N submitted 2025-10-06 cs.LG

Correlating Cross-Iteration Noise for DP-SGD using Model Curvature

Xin Gu , Yingtai Xiao , Guanlin He , Jiamu Bai , Daniel Kifer , Kiwan Maeng This is my paper
classification cs.LG
keywords differential privacyDP-SGDDP-MFnoise correlationHessian eigenvaluesmodel curvaturematrix factorizationprivacy-preserving training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DP-SGD lags behind ordinary SGD because the noise injected into each gradient update distorts not only the current step but also every later step, since later gradients are evaluated at shifted parameters. The paper shows that, for a quadratic loss, this compounding effect is captured exactly by a trace term involving the Hessian eigenvalues and the noise correlation matrix. It replaces the existing data-independent DP-MF objective with this curvature-aware objective, called NoiseCurve, and solves the same banded matrix factorization problem. Across CIFAR-10 and ChestX-ray14, with several model families and privacy budgets, the resulting noise correlations yield 1–4% higher accuracy than DP-SGD and the state-of-the-art DP-BandMF at the same privacy guarantee. The practical recipe is to estimate the Hessian spectrum once from public unlabeled data after self-supervised pretraining, truncate negative eigenvalues, and fit a power-law tail for large models.

Core claim

The paper's central claim is that cross-iteration noise correlation for DP-SGD should minimize a curvature-aware trace objective rather than the data-independent energy objective used by prior DP-MF methods. For a quadratic loss the paper proves E[L(w̃_T)] − E[L(w_T)] = (η²/2) Tr(X⁻¹(Vᵀ M V)), where X is the Gram matrix of the noise-mixing matrix, M is the diagonal matrix of Hessian eigenvalues, and V encodes the powers (1−ηµᵢ)^{T−j−1}. Minimizing this trace over diagonal-one, positive-definite, banded X yields the mixing matrix C. The paper then argues a single representative Hessian—estimated from public unlabeled data after self-supervised pretraining, with negative eigenvalues zeroed and

What carries the argument

The load-bearing object is the identity of Theorem 1: E[L(w̃_T)] − E[L(w_T)] = (η²/2) Tr(X⁻¹(Vᵀ M V)). Here X = CᵀC is the Gram matrix of the noise-mixing matrix, M = Diag(µ₀,…,µ_{p−1}) holds the Hessian eigenvalues, and V[i,j] = (1−ηµᵢ)^{T−j−1} propagates earlier noise through the iterates. This identity reduces the design of noise correlations to a convex trace-minimization problem with banded, diagonal-one, positive-definite constraints, and it is why only Hessian eigenvalues, not the full Hessian, are needed. The supporting machinery makes this practical: pretraining on public data yields a stable representative spectrum, zeroing negative eigenvalues restores convexity, and a power-law c

Load-bearing premise

The Hessian eigenvalue spectrum estimated once from public data after pretraining remains representative of the spectra seen during private training; the paper gives empirical support but no formal analysis, and its own limitations section says it is unclear whether this holds beyond vision.

What would settle it

Measure the actual Hessian spectra along the private training trajectory on a language or tabular task and compare the accuracy of correlations optimized from the public-data spectrum against the previous data-independent objective; if spectra drift markedly or the accuracy gain disappears, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Using NoiseCurve as the objective inside the DP-MF framework yields 1–4% higher test accuracy than DP-SGD and the previous banded-MF baseline across every privacy budget tested, with no additional privacy loss.
  • Curvature information from public unlabeled data—even from a different domain, such as a general vision dataset for chest X-rays—remains informative enough to improve noise correlations.
  • The method works across convex last-layer finetuning, small non-convex CNNs, full VGG training, and LoRA finetuning of a vision transformer, indicating it is not tied to one architecture.
  • Because the change is only in the objective and the feasible set is unchanged, NoiseCurve can be dropped into existing DP-MF implementations and privacy accounting.
  • The eigenvalue curve-fitting approximation extends the method to models where full Hessian eigen-decomposition is infeasible, at a small accuracy cost compared to using the true spectrum.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the Hessian spectrum is indeed governed mostly by architecture rather than data, the same optimized correlation matrix could be reused across tasks for a fixed architecture, turning the one-time public-data computation into a reusable artifact.
  • Inference: The quadratic derivation suggests the curvature-weighted trace objective could also improve other correlated-noise mechanisms, not only the banded form tested here.
  • Inference: The paper's stability evidence is short-horizon; a natural extension is to re-estimate the spectrum periodically during training and check whether adaptive correlation matrices beat the fixed one.
  • Inference: The power-law tail fit, treated by the authors as an orthogonal contribution, could serve other applications that need Hessian spectral information; its main risk is sensitivity to the chosen endpoint (p₊, µ_p₊).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes NoiseCurve, a method for improving DP-SGD by correlating the privacy noise across iterations using Hessian curvature information. In the idealized quadratic-loss case, Theorem 1 derives the expected loss gap between correlated-noise gradient descent and noise-free gradient descent in terms of the Hessian eigenvalues and the matrix X = C^T C, yielding Problem P1: minimize Tr((V^T M V)X^{-1}) subject to diag(X)=1, X positive definite, and b-banded. The authors then propose practical solutions for using public unlabeled data to estimate eigenvalues, coping with non-constant Hessians, zeroing negative eigenvalues, and approximating spectra via curve fitting. Experiments on CIFAR-10 and ChestX-ray14 with various models and privacy budgets report consistent accuracy improvements over DP-SGD and DP-BandMF. The paper explicitly acknowledges limitations regarding public-data dependence and eigenvalue approximation.

Significance. If the empirical comparisons are at the claimed privacy level, this is a useful contribution: the quadratic derivation is clean and the proposed objective is a natural, principled replacement for the DP-MF heuristic. The paper also includes a self-contained proof of Theorem 1, a drop-in integration with the DP-BandMF framework, and a range of experiments. However, the central equal-privacy-budget claim is currently undermined by validation-set hyperparameter tuning on private data that is not charged to the privacy budget. The representative-spectrum assumption is also load-bearing and is only empirically supported in a limited vision setting, as the limitations section itself concedes. With corrections to the privacy accounting and a sensitivity analysis of the spectrum assumption, the contribution would be significant.

major comments (4)
  1. [§5, §D.1] The privacy accounting for hyperparameter selection is missing. The paper states 'we split 10% of data samples from training sets as validation sets and tune hyperparameters on validation sets' and Appendix D.1 says Optuna searches learning rate, clip norm, and band size (and, in effect, p+ and mu_p+). These validation data are drawn from the private training set. The reported (epsilon,delta) guarantee is computed for the final training run only; the selection of hyperparameters is a data-dependent computation on private data with no privacy charge. Therefore the claim that NoiseCurve improves over DP-SGD and DP-BandMF at the same privacy budget is not currently supported. The fact that all baselines are tuned the same way does not restore the absolute guarantee. Please either (a) include the selection procedure in the privacy analysis and report the total epsilon, (b) tune on public dat
  2. [§4.2, §6] The load-bearing premise that a single Hessian eigenspectrum computed once from public data after SimCLR pretraining is representative along the private training trajectory is supported only by Figure 3 for one small CNN, and the limitations section explicitly states there is no formal analysis and it is unclear whether it holds outside vision. Since Problem P1 fixes M, an unrepresentative spectrum will miscalibrate C. I ask for a sensitivity analysis: vary the spectrum (e.g., using spectra taken at different training epochs, or perturbing the eigenvalues) and report the resulting accuracy. This is a correctness-risk concern, not a circularity objection, but it is central to the claim that NoiseCurve is a generally applicable drop-in replacement.
  3. [§4.4, Table 3] The curve-fitting validation in Table 3 is not convincing as presented. The 'Fit' row values (122, 5.59, 479, 20, 140, 285 for k=100,...,3000) are non-monotone in k, and no error bars, standard deviations, or number of runs are given. Some entries make it appear that more eigenvalues can hurt the fitted solution. Please clarify what these numbers represent (one run? mean over runs?), report variance, and explain the non-monotonicity. This matters because S4 is the scalability contribution for large models.
  4. [§4.3] Replacing negative eigenvalues with zero is justified empirically by the observation that they are small after pretraining, but Problem P1 is derived for a positive semidefinite Hessian. The paper should state explicitly that clamping negative eigenvalues to zero is a heuristic modification of the quadratic-loss objective, and provide evidence on how sensitive P1's solution is to the exact truncation. An ablation with different thresholds or with small positive floors would clarify whether the convexification step introduces a significant gap between Theorem 1 and the algorithm used in practice.
minor comments (4)
  1. [Table 3 / Appendix C] The row entries in Table 3 lack column separators, and the introductory sentence 'reduction-in-objective of different with respect to k' is garbled. Please reformat and rephrase.
  2. [Eq. (2)] State the domain of alpha (presumably negative) and clarify that C and alpha are fitted constants. Note that at i = p+, the formula is independent of C and alpha, so the endpoint mu_p+ is a free parameter.
  3. [Figure 3] The caption 'epoch 0 and epoch 1' is ambiguous; clarify whether these are checkpoints at initialization and after one epoch of training, and whether pretraining is included.
  4. [Appendix D.2] Table 5, which tunes on the test set, is explicitly acknowledged as bad practice. Please add a sentence noting that these results are not comparable to the main tables and should not be used to support the central equal-privacy claim.

Circularity Check

0 steps flagged

No material circularity: closed-form objective, public-data curvature, and external accuracy benchmarks; only a minor non-load-bearing self-citation.

full rationale

The paper's central derivation is self-contained rather than circular. Theorem 1 is a closed-form computation of expected excess loss for quadratic losses with correlated noise; it does not assume the accuracy improvements it is later used to motivate. Problem P1 is then obtained by substituting this exact expression into the DP-BandMF feasibility constraints, so the objective is not defined in terms of the accuracy numbers it is claimed to predict. The curvature input M is estimated from unlabeled public data (TinyImageNet) and not fit to test accuracy; the spectrum curve-fitting in Section 4.4 fits only eigenvalue tails and is validated separately in Table 3 by objective reduction, not by circular reuse of the target result. The accuracy comparisons in Section 5 are external benchmark measurements, so the central claim does not reduce to its inputs. The only overlapping self-citation is Gu et al. (2025), used in Appendix D.1 as a recommendation to finetune parameters on public data first; this is not load-bearing because the paper supplies independent empirical support for pretraining (Figures 3 and 5, Table 7) and because it does not justify the core curvature-correlation claim. The paper's stated limitations—no formal analysis of public-data dependence and limited validation of the eigenvalue approximation—are empirical-validity gaps, not circularity. A separate privacy-accounting concern (validation-set hyperparameter tuning not charged to epsilon) is a correctness risk rather than a derivation-equivalence or circularity issue, so it does not raise the circularity score.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The derivation's only exact input is the eigenvalue spectrum; every other load-bearing choice is an approximation or hyperparameter. The most important, unproven choices are the representative-spectrum assumption and the private-validation-tuning assumption.

free parameters (7)
  • Hessian eigenvalues μ_i = estimated from public TinyImageNet
    Representative curvature used to build M in P1; the entire method depends on this measured spectrum.
  • Top-k count k = 300–2000
    Number of Lanczos eigenvalues computed; changes the fitted tail and therefore the optimized X.
  • Curve-fit coefficients C and α = not reported
    Fitted to the top-k eigenvalues in Eq. (2); determine the extrapolated spectrum and thus M.
  • Tail endpoint index p+ = 100k–200k (range)
    Hand-chosen deep-learning hyperparameter marking where the eigenvalue tail ends.
  • Tail endpoint eigenvalue μp+ = 10^-5–10^-6
    Hand-chosen near-zero magnitude at the tail endpoint; controls the fitted curve.
  • Learning rate η = searched [0.1, 1.0]
    Appears inside V in Theorem 1 and P1, so it changes the optimal correlation matrix; tuned on validation data.
  • Band size b = searched [1, 20]
    Controls the banded structure of X, the privacy amplification, and the objective; tuned as a hyperparameter.
axioms (5)
  • domain assumption Quadratic loss with fixed PSD Hessian is an adequate proxy for designing C in real DNN training.
    Theorem 1 is exact only for constant Hessian; Section 4 applies it to nonconvex DNNs heuristically, with no formal guarantee that P1 matches the expected loss for real networks.
  • domain assumption Hessian eigenspectrum estimated from public data after SimCLR pretraining is representative of the private training trajectory and is stable.
    Used in Sections 4.1–4.2 and Figures 2–3. The limitations section explicitly says no formal analysis supports this and generalizability beyond vision is unknown.
  • domain assumption Negative Hessian eigenvalues can be replaced by zero without materially changing the quality of C.
    Section 4.3 relies on the empirical claim that large negative eigenvalues vanish near local minima, citing prior work and Figure 3b.
  • standard math DP-BandMF privacy amplification and Opacus accounting remain valid when X is optimized with the new objective.
    External results; the new objective only changes C, while the privacy constraints, banding, and partition scheme are unchanged.
  • ad hoc to paper Tuning hyperparameters on a validation set split from private training data does not consume the privacy budget.
    Appendix D.1 splits 10% of private training data for tuning and reports ε as if only training consumed privacy; this is an unflagged assumption that is not standard DP composition.

pith-pipeline@v1.3.0-alltime-deepseek · 18380 in / 16636 out tokens · 129997 ms · 2026-08-04T11:19:46.716359+00:00 · methodology

0 comments
read the original abstract

Differentially private stochastic gradient descent (DP-SGD) offers the promise of training deep learning models while mitigating many privacy risks. However, there is currently a large accuracy gap between DP-SGD and normal SGD training. This has resulted in different lines of research investigating orthogonal ways of improving privacy-preserving training. One such line of work, known as DP-MF, correlates the privacy noise across different iterations of stochastic gradient descent -- allowing later iterations to cancel out some of the noise added to earlier iterations. In this paper, we study how to improve this noise correlation. We propose a technique called NoiseCurve that uses model curvature, estimated from public unlabeled data, to improve the quality of this cross-iteration noise correlation. Our experiments on various datasets, models, and privacy parameters show that the noise correlations computed by NoiseCurve offer consistent and significant improvements in accuracy over the correlation scheme used by DP-MF.

Figures

Figures reproduced from arXiv: 2510.05416 by Daniel Kifer, Guanlin He, Jiamu Bai, Kiwan Maeng, Xin Gu, Yingtai Xiao.

Figure 1
Figure 1. Figure 1: Test accuracy (averaged over 3 runs) on CIFAR-10 for δ = 10−5 as ϵ varies. Details in Sec￾tion 5. Curvature is obtained in an unsupervised way from TinyImageNet. To improve the accuracy of DP-SGD, several recent works, which we call DP-MF methods, explored replacing indepen￾dent Gaussian noise with noise that is correlated across iter￾ations. The correlations are carefully designed so that noise in some it… view at source ↗
Figure 2
Figure 2. Figure 2: Largest 10,000 eigenvalues of four datasets estimated on two CNNs. Setup details are in Appendix D.1. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Change of the eigenvalues during training, (a) without and (b) with pretraining. Only values between -0.05 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The true (positive) eigenvalues (in descending order) of a small CNN ( [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Change of the Hessian eigenvalues during training, a full display of Figure 3a. Despite significant negative [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    2026

  5. [5]

    Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang

    Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS '16, pp.\ 308–318, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450341394. doi:10.1145/2976749...

    arXiv 2016

  6. [6]

    Optuna: A next-generation hyperparameter optimization framework

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2019

    2019

  7. [7]

    Privacy amplification via random check-ins

    Borja Balle, Peter Kairouz, Brendan McMahan, Om Thakkar, and Abhradeep Guha Thakurta. Privacy amplification via random check-ins. Advances in neural information processing systems, 33: 0 4623--4634, 2020

    2020

  8. [8]

    Automatic clipping: Differentially private deep learning made easier and stronger

    Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Automatic clipping: Differentially private deep learning made easier and stronger. Advances in Neural Information Processing Systems, 36: 0 41727--41764, 2023 a

    2023

  9. [9]

    Differentially private optimization on large model at small cost

    Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Differentially private optimization on large model at small cost. In International Conference on Machine Learning, pp.\ 3192--3218. PMLR, 2023 b

    2023

  10. [10]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 1597--1607. PMLR, 13--18 Jul 2020. URL https://...

    2020

  11. [11]

    Choquette-Choo, Arun Ganesh, Ryan McKenna, Hugh Brendan McMahan, J Keith Rush, Abhradeep Guha Thakurta, and Zheng Xu

    Christopher A. Choquette-Choo, Arun Ganesh, Ryan McKenna, Hugh Brendan McMahan, J Keith Rush, Abhradeep Guha Thakurta, and Zheng Xu. (amplified) banded matrix factorization: A unified approach to private training. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 a . URL https://openreview.net/forum?id=zEm6hF97Pz

    2023

  12. [12]

    Choquette-Choo, H

    Christopher A. Choquette-Choo, H. Brendan McMahan, Keith Rush, and Abhradeep Thakurta. Multi-epoch matrix factorization mechanisms for private machine learning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023 b

    2023

  13. [13]

    Mind the privacy unit! user-level differential privacy for language model fine-tuning

    Lynn Chua, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Daogao Liu, Pasin Manurangsi, Amer Sinha, and Chiyuan Zhang. Mind the privacy unit! user-level differential privacy for language model fine-tuning. arXiv preprint arXiv:2406.14322, 2024 a

    Pith/arXiv arXiv 2024

  14. [14]

    Scalable DP - SGD : Shuffling vs

    Lynn Chua, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, and Chiyuan Zhang. Scalable DP - SGD : Shuffling vs. poisson subsampling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b . URL https://openreview.net/forum?id=6gMnj9oc6d

    2024

  15. [15]

    Unlocking high-accuracy differentially private image classification through scale

    Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650, 2022

    Pith/arXiv arXiv 2022

  16. [16]

    Brendan McMahan, John Rush, Adam Smith, and Abhradeep Guha Thakurta

    Sergey Denisov, H. Brendan McMahan, John Rush, Adam Smith, and Abhradeep Guha Thakurta. Improved differential privacy for sgd via optimal private linear operators on adaptive streams. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 5910--5924. Curran Associates...

    2022

  17. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010.11929, 2020. URL https://arxiv.org/abs/2010.11929

    Pith/arXiv arXiv 2010

  18. [18]

    Brendan McMahan, Krishna Pillutla, Thomas Steinke, and Abhradeep Thakurta

    Krishnamurthy Dj Dvijotham, H. Brendan McMahan, Krishna Pillutla, Thomas Steinke, and Abhradeep Thakurta. Efficient and near-optimal noise generation for streaming differential privacy. In 2024 IEEE 65th Annual Symposium on Foundations of Computer Science (FOCS), pp.\ 2306--2317, 2024. doi:10.1109/FOCS61266.2024.00135

    arXiv 2024

  19. [19]

    Our data, ourselves: privacy via distributed noise generation

    Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: privacy via distributed noise generation. In Proceedings of the 24th Annual International Conference on The Theory and Applications of Cryptographic Techniques, EUROCRYPT'06, pp.\ 486–503, Berlin, Heidelberg, 2006 a . Springer-Verlag. ISBN 3540345469. d...

    doi:10.1007/11761679_29 2006

  20. [20]

    Calibrating noise to sensitivity in private data analysis

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin (eds.), Theory of Cryptography, pp.\ 265--284, Berlin, Heidelberg, 2006 b . Springer Berlin Heidelberg. ISBN 978-3-540-32732-5

    2006

  21. [21]

    Privacy amplification by random allocation, 2025

    Vitaly Feldman and Moshe Shenfeld. Privacy amplification by random allocation, 2025. URL https://arxiv.org/abs/2502.08202

    arXiv 2025

  22. [22]

    Tighter privacy analysis for truncated poisson sampling

    Arun Ganesh. Tighter privacy analysis for truncated poisson sampling. arXiv preprint arXiv:2508.15089, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Faster differentially private convex optimization via second-order methods

    Arun Ganesh, Mahdi Haghifam, Thomas Steinke, and Abhradeep Thakurta. Faster differentially private convex optimization via second-order methods. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc

    2023

  24. [24]

    On design principles for private adaptive optimizers, 2025

    Arun Ganesh, Brendan McMahan, and Abhradeep Thakurta. On design principles for private adaptive optimizers, 2025. URL https://arxiv.org/abs/2507.01129

    Pith/arXiv arXiv 2025

  25. [25]

    An investigation into neural net optimization via hessian eigenvalue density

    Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.\ 2232--2241. PMLR, 09--15 Jun 2019. URL http...

    2019

  26. [26]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp.\ 249--256, Chia Laguna Resort, Sardinia, Italy,...

    2010

  27. [27]

    G rammarly: F ree A I W riting A ssistance --- grammarly.com

    Grammarly. G rammarly: F ree A I W riting A ssistance --- grammarly.com. www.grammarly.com. [Accessed 24-09-2025]

    2025

  28. [28]

    pfl-research: simulation framework for accelerating research in Private Federated Learning

    Filip Granqvist, Congzheng Song, Áine Cahill, Rogier van Dalen, Martin Pelikan, Yi Sheng Chan, Xiaojun Feng, Natarajan Krishnaswami, Vojta Jina, and Mona Chitnis. pfl-research: simulation framework for accelerating research in Private Federated Learning . arXiv preprint arXiv:2404.06430, April 2024. doi:10.48550/arXiv.2404.06430

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2404.06430 2024

  29. [29]

    Choosing public datasets for private machine learning via gradient subspace distance

    Xin Gu, Gautam Kamath, and Zhiwei Steven Wu. Choosing public datasets for private machine learning via gradient subspace distance. In IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2025, Copenhagen, Denmark, April 9-11, 2025 , pp.\ 879--900. IEEE , 2025. doi:10.1109/SATML64287.2025.00054. URL https://doi.org/10.1109/SaTML64287.2025.00054

    arXiv 2025

  30. [30]

    Roberts, and Ethan Dyer

    Guy Gur - Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. CoRR, abs/1812.04754, 2018. URL http://arxiv.org/abs/1812.04754

    Pith/arXiv arXiv 2018

  31. [31]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015. URL https://arxiv.org/abs/1502.01852

    Pith/arXiv arXiv 2015

  32. [32]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pp.\ 770--778. IEEE Computer Society, 2016. doi:10.1109/CVPR.2016.90. URL https://doi.org/10.1109/CVPR.2016.90

    doi:10.1109/cvpr.2016.90 2016

  33. [33]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  34. [34]

    Practical and private (deep) learning without sampling or shuffling

    Peter Kairouz, Brendan Mcmahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling or shuffling. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 5213--5225. PMLR, 18--24 Ju...

    2021

  35. [35]

    Back to square roots: An optimal bound on the matrix factorization error for multi-epoch differentially private sgd

    Nikita P Kalinin, Ryan McKenna, Jalaj Upadhyay, and Christoph H Lampert. Back to square roots: An optimal bound on the matrix factorization error for multi-epoch differentially private sgd. arXiv preprint arXiv:2505.12128, 2025

    arXiv 2025

  36. [36]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2017

    2017

  37. [37]

    Gradient descent with linearly correlated noise: theory and applications to differential privacy

    Anastasia Koloskova, Ryan McKenna, Zachary Charles, Keith Rush, and Brendan McMahan. Gradient descent with linearly correlated noise: theory and applications to differential privacy. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc

    2023

  38. [38]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  39. [39]

    Toward training at imagenet scale with differential privacy

    Alexey Kurakin, Shuang Song, Steve Chien, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta. Toward training at imagenet scale with differential privacy. arXiv preprint arXiv:2201.12328, 2022

    Pith/arXiv arXiv 2022

  40. [40]

    An iteration method for the solution of the eigenvalue problem of linear differential and integral operators

    Cornelius Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. Journal of Research of the National Bureau of Standards, 45 0 (4): 0 255--282, 1950. doi:10.6028/jres.045.019

    work page doi:10.6028/jres.045.019 1950

  41. [41]

    Tiny imagenet visual recognition challenge

    Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7 0 (7): 0 3, 2015

    2015

  42. [42]

    Concentrated differentially private gradient descent with adaptive per-iteration privacy budget, 2018

    Jaewoo Lee and Daniel Kifer. Concentrated differentially private gradient descent with adaptive per-iteration privacy budget, 2018. URL https://arxiv.org/abs/1808.09501

    Pith/arXiv arXiv 2018

  43. [43]

    Scaling up differentially private deep learning with fast per-example gradient clipping

    Jaewoo Lee and Daniel Kifer. Scaling up differentially private deep learning with fast per-example gradient clipping. Proceedings on Privacy Enhancing Technologies, 2021

    2021

  44. [44]

    Reddi, Hugh Brendan McMahan, and Virginia Smith

    Tian Li, Manzil Zaheer, Ken Liu, Sashank J. Reddi, Hugh Brendan McMahan, and Virginia Smith. Differentially private adaptive optimization with delayed preconditioners. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=j1zQGmQQOX1

    2023

  45. [45]

    Large language models can be strong differentially private learners

    Xuechen Li, Florian Tram\`er, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In Proceedings of the 10th International Conference on Learning Representations, ICLR '22, 2022 a

    2022

  46. [46]

    Large language models can be strong differentially private learners

    Xuechen Li, Florian Tram \` e r, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022 b . URL https://openreview.net/forum?id=bVuP3ltATMz

    2022

  47. [47]

    An inversion theorem for buffered linear toeplitz (blt) matrices and applications to streaming differential privacy

    H Brendan McMahan and Krishna Pillutla. An inversion theorem for buffered linear toeplitz (blt) matrices and applications to streaming differential privacy. arXiv preprint arXiv:2504.21413, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf

    2011

  49. [49]

    Updating quasi-newton matrices with limited storage

    Jorge Nocedal. Updating quasi-newton matrices with limited storage. Mathematics of computation, 35 0 (151): 0 773--782, 1980

    1980

  50. [50]

    The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size, 2019

    Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size, 2019. URL https://arxiv.org/abs/1811.07062

    Pith/arXiv arXiv 2019

  51. [51]

    PyTorch: an imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K\" o pf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: an imperative style, high-perf...

    2019

  52. [52]

    Correlated noise mechanisms for differentially private learning

    Krishna Pillutla, Jalaj Upadhyay, Christopher A Choquette-Choo, Krishnamurthy Dvijotham, Arun Ganesh, Monika Henzinger, Jonathan Katz, Ryan McKenna, H Brendan McMahan, Keith Rush, et al. Correlated noise mechanisms for differentially private learning. arXiv preprint arXiv:2506.08201, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    Eigenvalues of the hessian in deep learning: Singularity and beyond

    Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint arXiv:1611.07476, 2016

    Pith/arXiv arXiv 2016

  54. [54]

    Very deep convolutional networks for large-scale image recognition, 2015

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015. URL https://arxiv.org/abs/1409.1556

    Pith/arXiv arXiv 2015

  55. [55]

    Deep image prior

    Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In CVPR, 2018

    2018

  56. [56]

    Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases

    Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pp.\ 3462--3471, 2017

    2017

  57. [57]

    Opacus: U ser-friendly differential privacy library in PyTorch

    Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: U ser-friendly differential privacy library in PyTorch . arXiv preprint arXiv:2109.12298, 2021

    Pith/arXiv arXiv 2021

  58. [58]

    Do not let privacy overbill utility: Gradient embedding perturbation for private learning

    Da Yu, Huishuai Zhang, Wei Chen, and Tie - Yan Liu. Do not let privacy overbill utility: Gradient embedding perturbation for private learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=7aogOj\_VYO0

    2021

  59. [59]

    Differentially private fine-tuning of language models

    Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. Differentially private fine-tuning of language models. In Proceedings of the 10th International Conference on Learning Representations, ICLR '22, 2022

    2022

  60. [60]

    Bypassing the ambient dimension: Private \ sgd \ with gradient subspace identification

    Yingxue Zhou, Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private \ sgd \ with gradient subspace identification. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=7dpmlkBuJFC

    2021