Pith. sign in

REVIEW 3 major objections 8 minor 47 references

Whitened-space trick unifies DP and natural gradient

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 21:49 UTC pith:DQZZTRZ6

load-bearing objection DP-NGD combines K-FAC curvature from public data with a whitened-space DP mechanism that provably recovers natural-gradient updates at zero extra privacy cost. The theory is clean and the experiments are solid, but the OOD curvature transferability assumption is argued informally and the code link is a placeholder. the 3 major comments →

arxiv 2607.05866 v1 pith:DQZZTRZ6 submitted 2026-07-07 cs.LG cs.AI

Differentially Private Natural Gradient Descent

Pan Li , Kai Chen , Shuai Chang , Shengzhi Zhang , Peizhuo Lv , Jinwen He This is my paper
classification cs.LG cs.AI
keywords curvatureprivacytrainingprivateutilityanisotropicbudgetbudgets
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Differentially private training wastes privacy budget on inefficient gradient steps. Natural gradient descent converges faster by preconditioning gradients with curvature, but its anisotropic scaling conflicts with the isotropic noise and clipping that differential privacy requires. This paper resolves that conflict by performing all DP operations—clipping and Gaussian noise injection—in an F^(-1/2)-whitened space, then projecting back. The central result (Theorem 3.1) proves that this whitened-space procedure recovers the curvature-aligned natural gradient in expectation and transforms isotropic DP noise into anisotropic noise with covariance proportional to F^(-1), preserving the geometric advantage of second-order optimization under strict privacy guarantees. The paper also establishes a KL-DP duality: the DP clipping threshold C simultaneously bounds the KL trust region of natural gradient descent, giving the clipping parameter a principled geometric meaning rather than treating it as an arbitrary tuning knob. To avoid training instability in flat directions where inverse curvature explodes, the authors derive a dynamic eigenvalue clamping floor from the expected step size of standard DP-SGD. The entire curvature matrix is estimated on a small public auxiliary dataset (~1% of training data), incurring zero additional privacy cost. The paper argues this works because DP noise already corrupts fine-grained gradient components, so only coarse-grained curvature structure—transferable across datasets—matters for acceleration.

Core claim

The whitening matrix F^(-1/2) is the unique linear map that makes isotropic L2-norm clipping in the whitened space mathematically equivalent to bounding the Fisher norm (KL trust region) in the original parameter space. This equivalence means standard DP-SGD operations—clip, aggregate, add Gaussian noise—can be performed in the whitened space without corrupting the anisotropic curvature alignment that natural gradient descent requires. The proof shows the resulting parameter update equals the adaptively clipped natural gradient in expectation, with DP noise covariance becoming F^(-1)-shaped rather than spherical.

What carries the argument

The F^(-1/2 whitened-space update mechanism (Algorithm 1), the KL-DP duality showing C bounds both DP sensitivity and KL trust region, dynamic eigenvalue clamping via Euclidean step equivalence to DP-SGD, and K-FAC approximation for tractable curvature estimation on public data.

Load-bearing premise

The argument that out-of-distribution public data suffices for curvature estimation rests on the informal claim that DP noise corrupts fine-grained gradient components, making only coarse-grained structural priors necessary. If the dominant eigenspaces of the Fisher matrix are not transferable across datasets for a given architecture, the curvature preconditioner would mislead rather than merely be imprecise, and the acceleration would not materialize.

What would settle it

If the dominant eigenspaces of the Fisher matrix estimated on OOD public data systematically differ from those of the private training data, the preconditioner would steer updates in wrong directions, and both convergence speedup and accuracy gains would disappear or reverse.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Second-order optimization becomes viable for privacy-preserving deep learning without splitting the privacy budget between curvature estimation and gradient updates.
  • The KL-DP duality suggests that DP clipping thresholds in other contexts may have implicit geometric interpretations worth investigating.
  • If coarse-grained curvature transfer holds more broadly, public auxiliary data could accelerate DP training in domains beyond image classification with minimal data requirements.
  • The whitened-space mechanism provides a template for reconciling other anisotropic optimization methods (e.g., quasi-Newton, trust-region) with DP constraints.
  • Dynamic clamping derived from baseline step-size equivalence could generalize to stabilize other preconditioned DP optimizers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The claim that OOD public data suffices because DP noise corrupts fine-grained components is an informal argument; if dominant eigenspaces of the Fisher matrix are not transferable across datasets for a given architecture, the preconditioner could be misleading rather than merely coarse.
  • The 10x speedup on SVHN versus 2.3x on CIFAR-10 suggests the acceleration benefit is dataset-dependent, potentially correlating with how ill-conditioned the loss landscape is for each task.
  • If the approach extends to larger models, the K-FAC approximation's assumptions about activation-gradient independence may need revalidation, as these assumptions could break for architectures with complex skip connections or attention mechanisms.
  • The insensitivity to public data size (50 vs 5000 samples) supports the coarse-grained transferability claim but also raises the question of whether a randomly initialized network's curvature would work equally well, which would eliminate the public data requirement entirely.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper proposes DP-NGD, a framework for integrating Natural Gradient Descent (NGD) with Differential Privacy (DP). The approach has three components: (1) curvature estimation on a small out-of-distribution public dataset to avoid privacy cost, (2) a whitened-space update mechanism that performs isotropic DP clipping and noise injection in an F^{-1/2}-whitened space, provably recovering anisotropic curvature-aligned updates in the original parameter space (Theorem 3.1), and (3) a dynamic eigenvalue clamping schedule to prevent parameter explosion in flat directions. Experiments on CIFAR-10, SVHN, and UTKFace show accuracy improvements over DP-SGD, AdaDPS, and GEP across privacy budgets, with convergence speedups of up to 10x.

Significance. The paper addresses a well-motivated problem: the geometric inefficiency of first-order DP optimizers. The whitened-space mechanism (Theorem 3.1) is a clean algebraic result showing that isotropic DP operations in F^{-1/2}-whitened space recover the natural gradient in expectation and transform isotropic noise into anisotropic noise with covariance proportional to F^{-1}. The KL-DP duality (Eqs. 11-12) connecting the clipping threshold to the NGD trust region is an elegant observation. The privacy analysis (Section 3.5) is correct: since F^{-1/2} is computed on public data and fixed, the projection is post-processing. The empirical results are comprehensive, with controlled baselines, ablations, and hyperparameter sensitivity analyses. The framework is practical and the code availability statement is noted.

major comments (3)
  1. Section 3.1 and Theorem 3.1: Theorem 3.1 proves curvature alignment for the *estimated* F, but the paper's headline claim of 'curvature-aligned updates' that accelerate convergence requires that F estimated on OOD public data approximates the true Fisher on private data. The informal argument in Section 3.1 ('coarse-grained curvature transferability') asserts that DP noise dominates fine-grained components and that top eigenspaces transfer across datasets, but no quantitative bound or theorem is provided for the error introduced by using F_public^{-1} as a preconditioner. This is the load-bearing premise of the entire framework. The paper should either (a) provide a formal bound on the preconditioning error as a function of the spectral alignment between F_public and F_private, or (b) explicitly reframe Theorem 3.1 as a statement about the estimated F (not the true curvature) and add a讨论
  2. Figure 3 (Section 4.4): The sample efficiency experiment varies |D_pub| from 50 to 5000 but uses the same OOD source (CIFAR-100/ImageNet for UTKFace). This tests sample size, not distributional shift. The claim that 'coarse-grained curvature priors are sufficient' would be substantially strengthened by an experiment varying the *distribution* of D_pub (e.g., natural images vs. non-natural images, or different domains) to test whether the top eigenspaces of F are truly transferable. Without this, the central assumption remains supported only by informal argument.
  3. Table 2 and Section 4.1: The clipping threshold C differs across methods (C=10 for DP-NGD, C=1 for DP-SGD, C=2 for AdaDPS). The paper argues this is fair because C cancels in the privacy loss. However, C also affects optimization dynamics (gradient bias from clipping). The paper should discuss whether the gains could partly stem from less aggressive clipping (C=10 vs C=1) rather than curvature alignment per se. An ablation with DP-SGD at C=10, or DP-NGD at C=1, would help isolate the contribution of curvature preconditioning from the clipping threshold.
minor comments (8)
  1. Equation (3): The asymptotic scaling sigma ~ q*sqrt(T*log(1/delta))/epsilon is presented without derivation or citation. A reference to the specific result in RDP or Moments Accountant literature would help readers verify this scaling.
  2. Section 3.3, Eq. (14): The expected squared Euclidean step size for DP-SGD uses E[||xi||^2] = d*sigma^2*C^2, where d is the parameter dimension. For DP-NGD, the Rayleigh quotient bound gives 1/lambda_min, but the dimension d appears in both Eqs. (14) and (15) and cancels. It would help to state explicitly that the same d appears in both, as readers may wonder about the effective dimension in the block-diagonal setting.
  3. Table 2 caption: 'Best results are in bold' — the bolding in the table appears correct, but the caption could note that all methods use the same public auxiliary dataset for clarity.
  4. Section 4.3: The '10x speedup' claim on SVHN compares DP-NGD reaching 89.38% at step 340 vs. baselines reaching 89.25% at step 3400. However, Table 7 shows DP-NGD's total steps T for SVHN at epsilon=6.0 is 2380, not 340. The relationship between the speedup comparison (which uses a truncated run) and the final reported accuracy (which uses the full T) should be clarified. The speedup is to reach baseline peak, not DP-NGD's own peak.
  5. Figure 2: The y-axis label in panel (a) shows accuracy ranging from 45-62.5%, but Table 2 reports CIFAR-10 at epsilon=1.0 as 60.78% for DP-NGD. The figure appears to show lower accuracies than the table — please reconcile.
  6. Section 3.1: The phrase 'structurally homomorphic' is unusual in this context; 'structurally similar' or 'sharing the same block-diagonal structure' would be more precise.
  7. Appendix A: The derivation of W = F^{-1/2} from W^T W = F^{-1} states that the unique symmetric PSD solution is F^{-1/2}. This is correct, but the uniqueness claim should note that non-symmetric solutions also exist (e.g., via Cholesky). The paper uses the symmetric solution, which is appropriate, but should state this requirement explicitly.
  8. Reference [25] (Kunstner et al., 2019) discusses limitations of the empirical Fisher approximation. Since this paper uses the empirical Fisher (computed on D_pub), citing this work and discussing its implications would strengthen the paper.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee raises three major points: (1) the lack of a formal bound on preconditioning error from using OOD public data, (2) the absence of an experiment varying the distribution of D_pub, and (3) the potential confound of different clipping thresholds across methods. We address each below and commit to revisions for all three.

read point-by-point responses

  1. Referee: Theorem 3.1 proves curvature alignment for the estimated F, but the paper's headline claim of 'curvature-aligned updates' that accelerate convergence requires that F estimated on OOD public data approximates the true Fisher on private data. The informal argument in Section 3.1 ('coarse-grained curvature transferability') asserts that DP noise dominates fine-grained components and that top eigenspaces transfer across datasets, but no quantitative bound or theorem is provided for the error introduced by using F_public^{-1} as a preconditioner. This is the load-bearing premise of the entire framework. The paper should either (a) provide a formal bound on the preconditioning error as a function of the spectral alignment between F_public and F_private, or (b) explicitly reframe Theorem 3.1 as a statement about the estimated F (not the true curvature) and add a discussion.

    Authors: The referee is correct that Theorem 3.1 is a statement about the estimated F, not the true Fisher, and the manuscript should make this explicit. We will revise the theorem statement and surrounding text to clarify that the curvature alignment guarantee holds with respect to the estimated curvature matrix F_public, not the true F_private. We will also add a formal discussion of the preconditioning error. Specifically, we can provide a bound on the convergence rate degradation as a function of the spectral alignment between F_public and F_private. The key observation is that in the whitened space defined by F_public^{-1/2}, the effective condition number is governed by the generalized eigenvalue ratio between F_public and F_private. When the top eigenspaces align (which our empirical results and the K-FAC low-pass filtering argument support), the preconditioning error is bounded by the ratio of eigenvalues in the aligned subspace. We will formalize this as a proposition and discuss its implications, including the regime where the bound is loose (large distributional shift) and how the dynamic clamping mechanism provides additional robustness. We agree this is the load-bearing premise and the revision will strengthen the paper's theoretical foundation. revision: yes

  2. Referee: Figure 3 (Section 4.4): The sample efficiency experiment varies |D_pub| from 50 to 5000 but uses the same OOD source (CIFAR-100/ImageNet for UTKFace). This tests sample size, not distributional shift. The claim that 'coarse-grained curvature priors are sufficient' would be substantially strengthened by an experiment varying the *distribution* of D_pub (e.g., natural images vs. non-natural images, or different domains) to test whether the top eigenspaces of F are truly transferable. Without this, the central assumption remains supported only by informal argument.

    Authors: This is a fair and well-taken point. The current experiment in Figure 3 varies the quantity of public data but not its distribution, so it does not directly test the curvature transferability assumption. We will add a new experiment varying the distribution of D_pub. Specifically, for the UTKFace task (face images), we will construct public datasets from: (a) CIFAR-100 (natural images, different domain), (b) ImageNet (natural images, broader domain), (c) a text-based or synthetic non-natural image dataset, and (d) a random noise dataset as a negative control. We will report the resulting test accuracy and, importantly, measure the spectral alignment (e.g., principal angle between top-k eigenspaces) between F_public and F_private for each source. This will directly test whether the top eigenspaces transfer across distributions and whether performance degrades gracefully with increasing distributional shift. We expect the random noise control to show significant degradation, which would validate that the gains stem from genuine curvature transfer rather than an arbitrary preconditioning effect. This experiment will be added to Section 4.4 in the revision. revision: yes

  3. Referee: Table 2 and Section 4.1: The clipping threshold C differs across methods (C=10 for DP-NGD, C=1 for DP-SGD, C=2 for AdaDPS). The paper argues this is fair because C cancels in the privacy loss. However, C also affects optimization dynamics (gradient bias from clipping). The paper should discuss whether the gains could partly stem from less aggressive clipping (C=10 vs C=1) rather than curvature alignment per se. An ablation with DP-SGD at C=10, or DP-NGD at C=1, would help isolate the contribution of curvature preconditioning from the clipping threshold.

    Authors: The referee raises a valid concern about the confounding effect of different clipping thresholds on optimization dynamics. While C cancels in the privacy accounting, it indeed affects the bias-variance tradeoff of the clipped gradient estimator. We will address this in two ways. First, we will add an explicit discussion in Section 4.1 acknowledging that C affects optimization dynamics and explaining our rationale: C=10 for DP-NGD is not arbitrary but is inherited from the standard NGD KL trust-region constraint, and the whitened-space clipping means that C=10 in the whitened space corresponds to a curvature-aware Mahalanobis norm bound, not a raw Euclidean bound. Second, and more importantly, we will add the requested ablation: DP-SGD with C=10 and DP-NGD with C=1, on at least one dataset and privacy budget. We expect that DP-SGD with C=10 will perform worse than DP-SGD with C=1 (since the optimal C for DP-SGD is typically small, as confirmed by our grid search), and that DP-NGD with C=1 will still outperform DP-SGD with C=1, demonstrating that the gains are not solely due to the clipping threshold. This ablation will isolate the contribution of curvature preconditioning from the clipping threshold effect. revision: yes

Circularity Check

0 steps flagged

No circularity found: Theorem 3.1 is a parameter-free algebraic identity, the KL-DP duality follows by substitution, and the Euclidean safety bound is derived independently from DP-SGD step-size expectations.

full rationale

The paper's central theoretical result (Theorem 3.1) is a straightforward algebraic derivation: given the whitened-space update in Algorithm 1 (project via F^{-1/2}, clip isotropically, add isotropic Gaussian noise, project back via F^{-1/2}), the expectation and covariance of the update are computed by direct matrix algebra (Eqs. 8-9). No parameter is fitted to a target and then 'predicted.' The KL-DP duality (Eqs. 11-12) follows by substituting the update into the KL divergence formula and using F^{-1/2} F F^{-1/2} = I. The Euclidean safety bound (Eq. 16) is derived by comparing expected squared step sizes of DP-SGD and DP-NGD via Rayleigh quotient bounds, yielding a floor on eigenvalues. None of these steps reduce to their inputs by construction in a circular sense—they are genuine mathematical derivations from stated assumptions. The OOD public data assumption (Section 3.1) is an empirical claim supported by experiments (Figure 3, Table 3), not a circular definition. Self-citations are absent from the load-bearing theoretical arguments; the derivation in Appendix A is self-contained. The paper's weaknesses (unquantified OOD transfer error, informal curvature transferability argument) are correctness risks, not circularity.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

The paper introduces no new mathematical entities, particles, forces, or dimensions. It combines existing constructs (Fisher Information Matrix, K-FAC, whitening transformation, DP-SGD) in a new framework. All axioms are either standard math (K-FAC independence, KL Taylor expansion, DP post-processing) or domain assumptions about curvature transferability across datasets. The domain assumptions are the most fragile part of the framework.

free parameters (7)
  • C (clipping threshold) = 10.0
    Set to 10.0 for DP-NGD, inherited from standard NGD KL constraint. Baselines use different values (1.0 for DP-SGD, 2.0 for AdaDPS). Justified by KL-DP duality but still a chosen constant.
  • T_cur (curvature update interval) = 8
    Number of steps between curvature re-estimation. Chosen empirically; sensitivity analysis in Table 6a shows robustness from 1-16.
  • T1 (warmup fraction) = 0.1*T
    Fraction of total steps for the linear warmup phase of the clamping schedule. Chosen empirically; sensitivity in Table 6c.
  • p (polynomial exponent) = 10
    Controls steepness of the polynomial transition in the clamping schedule. Chosen empirically; sensitivity in Table 6b shows <1% variation from 2-15.
  • lambda_base = not specified
    Described as 'a standard numerical stabilizer adopted from classical NGD' but its specific value is not stated in the main text.
  • eta (learning rate) = varies by dataset/epsilon
    Grid-searched per dataset and privacy budget. Table 7 lists optimal values (0.01 or 0.02).
  • T (total steps) = varies by dataset/epsilon
    Grid-searched per dataset and privacy budget. Table 7 lists optimal values.
axioms (6)
  • domain assumption DP noise disproportionately corrupts fine-grained gradient components, making DP training inherently tolerant of curvature estimation errors in these directions.
    Section 3.1. This is the core justification for using OOD public data for curvature. It is argued informally, not proven as a theorem.
  • domain assumption Shared model architecture imposes strong inductive priors yielding covariance matrices that are structurally homomorphic across datasets.
    Section 3.1, citing [7, 30]. Used to justify why OOD public data captures relevant curvature.
  • domain assumption Natural images share universal low-level statistics (edges, textures, color smoothness) that dominate the top eigenspaces of the curvature matrix.
    Section 3.1, citing [21, 38]. Used to justify cross-dataset curvature transferability.
  • standard math K-FAC approximation: activations and pre-activation gradients of each layer are independent.
    Section 2.2. Standard K-FAC assumption from Martens and Grosse.
  • standard math KL divergence between predictive distributions can be approximated via second-order Taylor expansion: KL ≈ (1/2)Δθ^T F Δθ.
    Section 2.2, Eq. 4. Standard NGD assumption.
  • standard math The post-processing property of DP: applying a fixed linear transformation to a DP output does not incur additional privacy cost.
    Section 3.2 and 3.5. Standard DP property, correctly applied since F^(-1/2) is computed on public data.

pith-pipeline@v1.1.0-glm · 26994 in / 3878 out tokens · 444810 ms · 2026-07-08T21:49:34.125671+00:00 · methodology

0 comments
read the original abstract

Under a fixed privacy budget, the utility of differentially private (DP) training is ultimately determined by its optimization efficiency. Standard first-order DP optimizers such as DP-SGD rely solely on local gradients and ignore the underlying loss curvature. This geometric blindness causes severe zigzagging in ill-conditioned landscapes, squandering precious privacy budgets on inefficient iterations. Practitioners are thus trapped in a bind: either stop training prematurely or inject massive per-step noise, both of which critically compromise final model utility. Natural Gradient Descent (NGD) resolves this by preconditioning gradients with curvature, aligning updates with the loss geometry and extracting more efficient signal from every noisy step, offering a principled pathway to break the privacy-utility bottleneck. Despite its theoretical appeal, directly integrating NGD with DP introduces fundamental challenges: curvature estimation itself consumes prohibitive privacy budgets, isotropic DP operations conflict with the anisotropic scaling of NGD, and the inverse curvature catastrophically amplify parameter updates in flat directions, causing training instability. We propose DP-NGD, a practical framework that systematically addresses these obstacles by decoupling curvature estimation from private data, reconciling isotropic DP constraints with anisotropic second-order optimization via a whitened-space mechanism, and dynamically clamping the curvature to stabilize training. Extensive experiments on standard benchmarks demonstrate that DP-NGD achieves state-of-the-art accuracy, breaking through the utility ceilings of first-order baselines while delivering up to a $10\times$ convergence speedup under the same privacy budget.

Figures

Figures reproduced from arXiv: 2607.05866 by Jinwen He, Kai Chen, Pan Li, Peizhuo Lv, Shengzhi Zhang, Shuai Chang.

Figure 1
Figure 1. Figure 1: Why optimization efficiency dictates differential privacy utility. Here, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Convergence efficiency vs. total training steps. We compare the test accuracy of DP-NGD, DP-SGD-PT, and AdaDPS under [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Sample efficiency of public auxiliary data. Test accuracy [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation on the clamping floor using UTKFace ( [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 47 canonical work pages · 15 internal anchors

  1. [1]

    DeepLearningwithDifferentialPrivacy.Proceedings ofthe2016ACMSIGSACConferenceonComputerandCommunicationsSecurity (2016)

    MartínAbadi,AndyChu,IanJ.Goodfellow,H.B.McMahan,IlyaMironov,Kunal Talwar,andLiZhang.2016. DeepLearningwithDifferentialPrivacy.Proceedings ofthe2016ACMSIGSACConferenceonComputerandCommunicationsSecurity (2016). https://api.semanticscholar.org/CorpusID:207241585

    work page 2016

  2. [2]

    Yehya Abouelnaga, Ola S Ali, Hager Rady, and Mohamed Moustafa. 2016. Cifar-10: Knn-based ensemble of classifiers. In2016 International Conference on ComputationalScienceandComputationalIntelligence(CSCI).IEEE,1192–1195

    work page 2016

  3. [3]

    Adam with model exponential moving average is effective for nonconvex optimization

    KwangjunAhnandAshokCutkosky.2024. Adamwithmodelexponentialmoving average is effective for nonconvex optimization.ArXivabs/2405.18199 (2024). https://api.semanticscholar.org/CorpusID:270068108

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    Sukania, Vishal Vijayan, and Revanth Srinivasan

    BachinaHarishBabu,V.R.Vinothini,GautamSolaimalai,R.Vanitha,AjaySingh Yadav, P. Sukania, Vishal Vijayan, and Revanth Srinivasan. 2024. Theoretical optimizationofgroupsizeingroupnormalizationforenhanceddeepneuralnetwork training.AIP Conference Proceedings(2024). https://api.semanticscholar.org/ CorpusID:274008853

    work page 2024

  5. [5]

    DP-Mix: Mixup-based Data Augmentation for Differentially Private Learning

    Wenxuan Bao, F. Pittaluga, Vijay Kumar, and Vincent Bindschaedler. 2023. DP- Mix: Mixup-based Data Augmentation for Differentially Private Learning.ArXiv abs/2311.01295 (2023). https://api.semanticscholar.org/CorpusID:264935065

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    Adam Block, Mark Bun, Rathin Desai, Abhishek Shetty, and Zhiwei Steven Wu

    work page

  7. [7]

    Oracle-Efficient Differentially Private Learning with Public Data

    Oracle-Efficient Differentially Private Learning with Public Data.ArXiv abs/2402.09483 (2024). https://api.semanticscholar.org/CorpusID:267682343

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Enric Boix-Adserà. 2025. On the inductive bias of infinite-depth ResNets and the bottleneck rank.ArXivabs/2501.19149 (2025). https://api.semanticscholar.org/ CorpusID:276079304

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. 2022. Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger.ArXiv abs/2206.07136 (2022). https://api.semanticscholar.org/CorpusID:249674655

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  10. [10]

    Praveen Kumar Chandaliya, V.Chandra Kumar, Mayank Harjani, and Neeta Nain. 2019. SCDAE: Ethnicity and Gender Alteration on CLF and UTKFace Dataset. InInternational Conference on Computer Vision and Image Processing. https://api.semanticscholar.org/CorpusID:214730428

    work page 2019

  11. [11]

    Pratik Chaudhari, Anna Choromańska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Tour Chayes, Levent Sagun, and Ric- cardo Zecchina. 2016. Entropy-SGD: biasing gradient descent into wide val- leys.Journal of Statistical Mechanics: Theory and Experiment2019 (2016). https://api.semanticscholar.org/CorpusID:13807351

    work page 2016

  12. [12]

    Felix Dangel, Johannes Müller, and Marius Zeinhofer. 2024. Kronecker-factored approximate curvature for physics-informed neural networks.Advances in Neural Information Processing Systems37 (2024), 34582–34636

    work page 2024

  13. [13]

    Unlocking High-Accuracy Differentially Private Image Classification through Scale

    SohamDe,LeonardBerrada,JamieHayes,SamuelL.Smith,andBorjaBalle.2022. Unlocking High-Accuracy Differentially Private Image Classification through Scale.ArXivabs/2204.13650 (2022). https://api.semanticscholar.org/CorpusID: 248427073

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  14. [14]

    Wei Dong, Yuting Liang, and Ke Yi. 2022. Differentially Private Covariance Re- visited.ArXivabs/2205.14324 (2022). https://api.semanticscholar.org/CorpusID: 249191345

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  15. [15]

    Cynthia Dwork. 2006. Differential Privacy. InInternational Colloquium on Au- tomata, Languages and Programming. https://api.semanticscholar.org/CorpusID: 2565493

    work page 2006

  16. [16]

    Trends Theor

    CynthiaDworkandAaronRoth.2014.TheAlgorithmicFoundationsofDifferential Privacy.Found. Trends Theor. Comput. Sci.9 (2014), 211–407. https://api. semanticscholar.org/CorpusID:207178262

    work page 2014

  17. [17]

    Shuya Feng, Meisam Mohammady, Hanbin Hong, Shenao Yan, Ashish Kundu, Binghui Wang, and Yuan Hong. 2024. Harmonizing Differential Privacy Mecha- nisms for Federated Learning: Boosting Accuracy and Convergence.Proceedings of the Fifteenth ACM Conference on Data and Application Security and Privacy (2024). https://api.semanticscholar.org/CorpusID:278238496

    work page 2024

  18. [18]

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2020. Sharpness-Aware Minimization for Efficiently Improving Generalization.ArXiv abs/2010.01412 (2020). https://api.semanticscholar.org/CorpusID:222134093

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  19. [19]

    NumericalComposition of Differential Privacy

    SivakanthGopi,YinTatLee,andLukasWutschitz.2021. NumericalComposition of Differential Privacy. InNeural Information Processing Systems. https://api. semanticscholar.org/CorpusID:235358383

    work page 2021

  20. [20]

    Markos, and Elena Tuzhilina

    MichaelGreenacre,PatrickJ.F.Groenen,TrevorJ.Hastie,AlfonsoIodiceD’Enza, Angelos I. Markos, and Elena Tuzhilina. 2003. Principal Component Analysis. Technometrics45 (2003), 276 – 276. https://api.semanticscholar.org/CorpusID: 2534141

    work page 2003

  21. [21]

    Roger Grosse and James Martens. 2016. A kronecker-factored approximate fisher matrix for convolution layers. InInternational Conference on Machine Learning. PMLR, 573–582

    work page 2016

  22. [22]

    On the universality of neural encodings in CNNs

    FlorentinGuthandBriceM’enard.2024.Ontheuniversalityofneuralencodingsin CNNs.ArXivabs/2409.19460 (2024). https://api.semanticscholar.org/CorpusID: 272987466

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Tianjiao Huo, Jiaqi Fan, Xin Li, Hong Chen, Bingzhao Gao, and Xuesong Li. 2020. Traffic Sign Recognition Based on ResNet-20 and Deep Mutual Learning.2020 Chinese Automation Congress (CAC)(2020), 4770–4774. https: //api.semanticscholar.org/CorpusID:231737679

    work page 2020

  24. [24]

    Jolliffe and Jorge Cadima

    Ian T. Jolliffe and Jorge Cadima. 2016. Principal component analysis: a review and recent developments.Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences374 (2016). https://api. semanticscholar.org/CorpusID:20101754

    work page 2016

  25. [25]

    Daigo Kudo and Hiroyasu Tajima. 2022. Fisher information matrix as a resource measure in the resource theory of asymmetry with general connected-Lie-group symmetry.Physical Review A(2022). https://api.semanticscholar.org/CorpusID: 248562504

    work page 2022

  26. [26]

    Frederik Kunstner, Philipp Hennig, and Lukas Balles. 2019. Limitations of the empirical fisher approximation for natural gradient descent.Advances in neural information processing systems32 (2019)

    work page 2019

  27. [27]

    Reddi, and Virginia Smith

    Tian Li, Manzil Zaheer, Sashank J. Reddi, and Virginia Smith. 2022. Private Adaptive Optimization with Side Information. InInternational Conference on Machine Learning. https://api.semanticscholar.org/CorpusID:246823521

    work page 2022

  28. [28]

    Optimal Differentially Private Model Training with Public Data

    AndrewLowy,ZemanLi,TianjianHuang,andMeisamRazaviyayn.2023.Optimal Differentially Private Learning with Public Data.ArXivabs/2306.15056 (2023). https://api.semanticscholar.org/CorpusID:275750699

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  29. [29]

    PSS Madhulika and Nalini Sampath. 2022. An application of normalizer free neuralnetworksontheSVHNdataset.In2022Internationalconferenceonapplied artificial intelligence and computing (ICAAIC). IEEE, 238–242

    work page 2022

  30. [30]

    James Martens. 2020. New insights and perspectives on the natural gradient method.Journal of Machine Learning Research21, 146 (2020), 1–76

    work page 2020

  31. [31]

    ResNets Are Deeper Than You Think

    ChristianH.X.AliMehmeti-GöpelandMichaelWand.2025. ResNetsAreDeeper Than You Think.ArXivabs/2506.14386 (2025). https://api.semanticscholar.org/ CorpusID:279410014

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    Sebastian Meiser and Esfandiar Mohammadi. 2018. Tight on Budget?: Tight Bounds for r-Fold Approximate Differential Privacy.Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security(2018). https://api.semanticscholar.org/CorpusID:52504941

    work page 2018

  33. [33]

    RényiDifferentialPrivacy.2017IEEE30thComputerSecurity Foundations Symposium (CSF)(2017), 263–275

    IlyaMironov.2017. RényiDifferentialPrivacy.2017IEEE30thComputerSecurity Foundations Symposium (CSF)(2017), 263–275. https://api.semanticscholar.org/ CorpusID:9386213

    work page 2017

  34. [34]

    Ilya Mironov, Kunal Talwar, and Li Zhang. 2019. Rényi Differential Privacy of the Sampled Gaussian Mechanism.ArXivabs/1908.10530 (2019). https: //api.semanticscholar.org/CorpusID:201660159

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  35. [35]

    Daniel Morales-Brotons, Thijs Vogels, and Hadrien Hendrikx. 2024. Expo- nential Moving Average of Weights in Deep Learning: Dynamics and Benefits. Trans. Mach. Learn. Res.2024 (2024). https://api.semanticscholar.org/CorpusID: 270355881

    work page 2024

  36. [36]

    Joni Pajarinen, Hong Linh Thai, Riad Akrour, Jan Peters, and Gerhard Neumann

    work page

  37. [37]

    https://api.semanticscholar.org/CorpusID:59842948

    Compatible natural gradient policy search.Machine Learning108 (2019), 1443 – 1466. https://api.semanticscholar.org/CorpusID:59842948

    work page 2019

  38. [38]

    Physical Review D—Particles, Fields, Gravitation, and Cosmology88, 8 (2013), 084013

    CarlLRodriguez,BenjaminFarr,WillMFarr,andIlyaMandel.2013.Inadequacies of the Fisher information matrix in gravitational-wave parameter estimation. Physical Review D—Particles, Fields, Gravitation, and Cosmology88, 8 (2013), 084013

    work page 2013

  39. [39]

    Xinyu Tang, Ashwinee Panda, Vikash Sehwag, and Prateek Mittal. 2023. Differ- entially Private Image Classification by Learning Priors from Random Processes. J. Priv. Confidentiality15 (2023). https://api.semanticscholar.org/CorpusID: 259129508

    work page 2023

  40. [40]

    Ragav Venkatesan, Vijetha Gattupalli, and Baoxin Li. 2016. On the generality of neural image features.2016 IEEE International Conference on Image Processing (ICIP)(2016), 41–45. https://api.semanticscholar.org/CorpusID:9264808

    work page 2016

  41. [41]

    Hua Wang, Sheng yang Gao, Huanyu Zhang, Milan Shen, and Weijie Su. 2022. Edgeworth Accountant: An Analytical Approach to Differential Privacy Composi- tion.J. Amer. Statist. Assoc.(2022). https://api.semanticscholar.org/CorpusID: 249538149

    work page 2022

  42. [42]

    Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. 2018. Sub- sampled Rényi Differential Privacy and Analytical Moments Accountant. In International Conference on Artificial Intelligence and Statistics. https: //api.semanticscholar.org/CorpusID:51893629

    work page 2018

  43. [43]

    Yuxin Wu and Kaiming He. 2018. Group normalization. InProceedings of the European conference on computer vision (ECCV). 3–19

    work page 2018

  44. [44]

    Trustregionpolicy optimizationviaentropyregularizationforKullback-Leiblerdivergenceconstraint

    HaotianXu,JunyuXuan,GuangquanZhang,andJieLu.2024. Trustregionpolicy optimizationviaentropyregularizationforKullback-Leiblerdivergenceconstraint. Neurocomputing589 (2024), 127716. https://api.semanticscholar.org/CorpusID: 269195555

    work page 2024

  45. [45]

    Da Yu, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. 2021. Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning.ArXiv abs/2102.12677 (2021). https://api.semanticscholar.org/CorpusID:232046284

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  46. [46]

    Post-processing of Differentially Private Data: A Fairness Perspective

    KeyuZhu,FerdinandoFioretto,andPascalVanHentenryck.2022. Post-processing of Differentially Private Data: A Fairness Perspective.ArXivabs/2201.09425 (2022). https://api.semanticscholar.org/CorpusID:246240328

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  47. [47]

    Keyu Zhu, Pascal Van Hentenryck, and Ferdinando Fioretto. 2020. Bias and Variance of Post-processing in Differential Privacy.ArXivabs/2010.04327 (2020). https://api.semanticscholar.org/CorpusID:222272017 13

    work page internal anchor Pith review Pith/arXiv arXiv 2020