Pith. sign in

REVIEW 3 major objections 6 minor 21 references

Steps Adaptive Decay DPSGD: Enhancing Performance on Imbalanced Datasets with Differential Privacy with HAM10000

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A step-decay noise schedule lifts differentially private training on imbalanced skin images.

desk verdict A plausible schedule tweak with standard DP accounting, but the 1% accuracy claim rests on test-set tuning and a mismatched abstract; it needs a proper experiments pass before it earns trust. read the letter →

arxiv 2507.06619 v1 pith:ZW2XBIAF submitted 2025-07-09 cs.LG stat.ML

classification cs.LGstat.ML
keywords differentialprivacyDPSGDadaptivenoisemultiplierclippingthresholddecayimbalanceddatasetmedicalimageclassificationRényiHAM10000
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes SAD-DPSGD, a differentially private stochastic gradient descent variant that decays both the noise multiplier and the gradient clipping threshold in non-uniform steps over training. It argues that on small, imbalanced medical image datasets like HAM10000, standard DPSGD and existing adaptive DP algorithms clip away minority-class gradient information early, pushing the model into suboptimal solutions. By starting with a low noise multiplier and a high clipping threshold and then decaying both, SAD-DPSGD spends more privacy budget where learning is most informative. The authors report that on HAM10000 it reaches 80.68% accuracy at $\epsilon=3$, outperforming Auto-DPSGD-S by 1.00 percentage point and DPSGD by 2.15 percentage points.

What carries the argument

The load-bearing mechanism is the $\sigma$ clip estimation procedure (Algorithm 1): total training epochs $T$ are split into steps $D_0,\dots,D_n$ with $D_i\cdot\gamma=D_{i-1}$, and within each step the noise multiplier and clipping threshold are held constant while across steps $\sigma_i=\sigma_{i+1}\cdot\beta$ and $C_i=C_{i+1}\cdot a$. This produces a low-noise, high-clipping early phase and a high-noise, low-clipping late phase. The privacy accounting uses RDP composition (Proposition 1) to sum per-step RDP guarantees and converts the total to $(\epsilon,\delta)$-DP via Lemma 1, so the adaptive schedule still provides a formal privacy guarantee.

What would settle it

Re-run the HAM10000 comparison with hyperparameters selected on a held-out validation split (or via nested cross-validation) and report the test accuracies; if SAD-DPSGD no longer beats Auto-DPSGD-S at $\epsilon=3$, the central claim is not supported.

Watch

Extended reading notes

Core claim

The central claim is that the schedule of the noise multiplier and clipping threshold, not just their final values, controls how well differentially private training survives class imbalance. SAD-DPSGD divides training into $n$ steps with geometrically decreasing lengths, sets the noise multiplier to increase by a factor $\beta$ and the clipping threshold to decay by a factor $a$ across steps, and uses Rényi differential privacy composition to account for the changing noise. The result, the paper asserts, is that the model avoids the early suboptimal solutions caused by aggressive clipping of minority-class gradients and achieves higher accuracy than Auto-DPSGD and DPSGD at every privacy budget tested.

Load-bearing premise

The reported gain over Auto-DPSGD is not an artifact of choosing the decay parameters ($\beta,\gamma,n$) after seeing the test-set results; the paper does not describe a separate validation split.

Editorial extensions

If this is right

  • SAD-DPSGD is a drop-in training schedule change: it requires no extra data, no change to model architecture, and no post-processing, only a decaying noise and clipping schedule.
  • On HAM10000, SAD-DPSGD improves over Auto-DPSGD-S by 1.00, 0.99, and 0.80 percentage points at $\epsilon=3,8,16$ respectively with $\delta=10^{-3}$.
  • The method lifts both majority- and minority-class accuracy relative to Auto-DPSGD-S, indicating that the benefit is not simply a majority-class artifact.
  • The proposed parameter ranges ($\beta=0.8$, $\gamma=0.9$, $n=3$) give a starting point for applying the schedule to other imbalanced datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the schedule effect is real, it suggests a 'privacy curriculum': spending privacy budget early, when minority-class gradients are most informative, may be more valuable than spreading it uniformly.
  • A testable extension is to benchmark SAD-DPSGD on other imbalanced datasets (e.g., iNaturalist or UTKFace) with a fixed, pre-registered hyperparameter rule; currently the optimal $\beta,\gamma,n$ are chosen on HAM10000.
  • The mechanism may interact with class-weighted loss or data augmentation; combining a step-decay privacy schedule with those data-side fixes could compound the gain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SAD-DPSGD, a variant of DP-SGD in which the noise multiplier and the clipping threshold decay according to a pre-determined, geometrically non-uniform step schedule. The motivation is that on small imbalanced datasets, uniform schedules cause minority-class gradients to be clipped too aggressively early in training. The authors provide an RDP-based privacy accounting for the varying noise schedule, then compare SAD-DPSGD against DPSGD and two Auto-DPSGD variants on HAM10000, reporting accuracy gains of about one percentage point at epsilon = 3, 8, and 16. The central claim is that this simple schedule change improves DP accuracy on imbalanced medical image data without additional data or architectural changes.

Significance. If the central claim held, SAD-DPSGD would be a practically valuable, drop-in modification to DP-SGD that improves accuracy on small imbalanced datasets while preserving a formal privacy guarantee. The RDP composition framework used for the pre-determined schedule is the standard and appropriate tool, and the paper reasonably documents a hyperparameter exploration across several baseline algorithms. However, the empirical evidence as presented is not sufficient to support the claim: the reported gains are small, hyperparameters are selected on the test set, results are single runs, and the abstract's headline number misstates the comparison. The privacy accounting also has a load-bearing gap concerning step multiplicity. These issues are addressable in revision but currently undermine the paper's main conclusions.

major comments (3)
  1. [Section IV.B / Algorithm 1] The RDP composition in Section IV.B sums rdp_t only over the n steps, but each step D_i contains D_i training iterations and the Gaussian mechanism is applied once per iteration. Since the per-iteration RDP is alpha/(2*sigma_i^2), the correct budget for a step is D_i * alpha/(2*sigma_i^2), and the total is sum_i D_i * alpha/(2*sigma_i^2). As written, the reported epsilon = 3, 8, 16 undercount privacy by roughly a factor of T/n, which is not a conservative error: it makes SAD-DPSGD appear more private than it actually is. Please correct the accounting and re-run the experiments with adjusted noise schedules, or clarify that each D_i denotes a single iteration and revise Algorithm 1 accordingly.
  2. [Section V.B / Table II] Hyperparameters beta, gamma, and n are selected by accuracy on the HAM10000 test set (Figs. 6-7 and Table I), and the same test set is then used to report the final comparison in Table II. No held-out validation split, cross-validation, or multiple-seed statistics are described. With reported advantages of only 0.80-1.00 percentage points over Auto-DPSGD-S, the observed gap is within the range that test-set selection among the swept hyperparameters could produce. Please provide an out-of-sample evaluation protocol and report mean plus/minus standard deviation over multiple runs.
  3. [Abstract / Section V.C and Table II] The abstract claims a 2.15% improvement under epsilon = 3.0, but Table II shows SAD-DPSGD at 80.68% versus Auto-DPSGD-S at 79.68%, a 1.00 percentage-point gain; the 2.15% figure is the gap to DPSGD (78.53%). The abstract should state the comparison correctly, and the same correction applies to any summary of the results.
minor comments (6)
  1. [Algorithm 2] The input parameter names 'clipping decay parameter beta, noise multiplier decay parameter gamma' are inconsistent with Algorithm 1, where beta is the sigma decay parameter and gamma is the step decay parameter; the call sigma clip estimation(T, t, gamma, beta, a, sigma_n, C_n) matches Algorithm 1, so the input list should be corrected.
  2. [Algorithm 2, line 6] C_j is undefined; the clipping threshold should be C_t, consistent with the noise scale used in line 8.
  3. [Algorithm 2, line 2] 'with probability |B⊔|/N' should read 'with probability B/N'.
  4. [Section V.A] The mean noise multiplier formula mu = (1/n) sum s_i*sigma_i should be normalized by sum s_i = T, not by n; otherwise the reported means are not the average noise multiplier over training.
  5. [Section V.A.3] 'Figure 6 shows the results' for the effect of n should refer to Figure 5; Figure 6 is the beta-performance plot.
  6. [References and in-text citations] The text contains raw LaTeX citation commands (e.g., \cite{19}, \cite{8}) and at least one cited work (DPAdaMod-AGC) does not appear in the reference list; the bibliography and citation commands should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the privacy-accounting derivation is standard RDP composition from external results, and the empirical comparisons are measurements rather than predictions derived from fitted parameters.

full rationale

The paper's derivation chain is the privacy-budget argument in Section IV.B: each step uses the Gaussian mechanism with sensitivity C_t and noise scale C_t * sigma_t, giving (alpha, alpha/(2*sigma_t^2))-RDP via Definition 5; Proposition 1 from Mironov's RDP work justifies summing the per-step RDP values; and Lemma 1 converts the total to (epsilon, delta)-DP. This chain uses standard external results and does not feed the accuracy numbers back into the privacy computation; it is self-contained and independent of the empirical comparisons. There are no load-bearing self-citations by the present authors, no imported uniqueness theorem, and no ansatz smuggled in via citation. The empirical claim in Section V.C is an external measurement on HAM10000, not a fitted parameter renamed as a prediction. The main validity concern is that beta, gamma, and n are selected in Section V.B by maximizing accuracy on HAM10000 and then the same dataset is used for the Table II comparison, with no described validation split or error bars; this is a real selection-bias and experimental-design concern, but it is not circularity of the derivation. Section VI likewise concedes, as a limitation, that optimal hyperparameters may vary across datasets. The abstract's 2.15% improvement figure is internally inconsistent with Table II, where the gain over Auto-DPSGD-S is 1.00 percentage point and 2.15 percentage points is the gain over plain DPSGD; this is an accuracy/consistency error, not a circular step. Overall, the privacy derivation is first-principles and the empirical result is an external measurement, so no significant circularity is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard RDP composition and DP-SGD sensitivity assumptions, plus an unvalidated modeling choice about step scheduling. The experimental setup adds several tuned hyperparameters that are not fully reported, including beta, gamma, n, a, sigma_n, and C_n.

free parameters (6)
  • beta (noise multiplier decay parameter) = 0.8
    Chosen by accuracy sweep on HAM10000 in Section V.B.1; performance peaks at 0.8.
  • gamma (step decay parameter) = 0.9
    Chosen by accuracy sweep in Section V.B.2; the paper reports peaks at both 0.3 and 0.9, then uses 0.9.
  • n (number of steps) = 3
    Chosen by comparing n=3,4,5 in Table I; n=3 gives the highest accuracy.
  • a (clipping threshold decay parameter) = not reported
    Algorithm 1 includes clipping threshold decay parameter a, but no value or sweep is reported.
  • sigma_n (final noise multiplier) = not reported
    Calibrated to hit target epsilon, but the calibration procedure is not described.
  • C_n (final clipping threshold) = not reported
    Final clipping threshold set without documentation.
assumptions (5)
  • standard math Per-step RDP composition via Proposition 1 (Mironov 2017) with sum of per-step RDP values.
    Used in Section IV.B to derive total privacy budget from per-step Gaussian mechanism RDP.
  • standard math Gaussian mechanism with sensitivity C_t and noise std sigma_t C_t satisfies (alpha, alpha/(2 sigma_t^2))-RDP.
    Used for rdpt in Section IV.B.
  • domain assumption The clipping threshold C_t bounds the l2 sensitivity of each per-sample gradient at step t.
    Required for the Gaussian mechanism RDP bound; standard DP-SGD assumption.
  • domain assumption RDP analysis without subsampling amplification provides a valid upper bound on privacy loss for the Poisson-sampled batches.
    The paper sums alpha/(2 sigma_t^2) per step without accounting for subsampling q; this is conservative but not stated or justified in the paper.
  • ad hoc to paper The non-uniform step schedule (D_i gamma = D_{i-1}) and geometric decay of sigma and C improve learning on imbalanced data.
    Motivated by Figure 1 observation of minority accuracy growth, but no theoretical or empirical justification independent of the final results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Steps Adaptive Decay DPSGD: Enhancing Performance on Imbalanced Datasets with Differential Privacy with HAM10000." pith.science (2026). https://pith.science/paper/ZW2XBIAF

@misc{pith2026250706619,
  author       = {Pith},
  title        = {Pith review of: Steps Adaptive Decay DPSGD: Enhancing Performance on Imbalanced Datasets with Differential Privacy with HAM10000},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZW2XBIAF}},
  note         = {Machine review of arXiv:2507.06619}
}
abstract

When applying machine learning to medical image classification, data leakage is a critical issue. Previous methods, such as adding noise to gradients for differential privacy, work well on large datasets like MNIST and CIFAR-100, but fail on small, imbalanced medical datasets like HAM10000. This is because the imbalanced distribution causes gradients from minority classes to be clipped and lose crucial information, while majority classes dominate. This leads the model to fall into suboptimal solutions early. To address this, we propose SAD-DPSGD, which uses a linear decaying mechanism for noise and clipping thresholds. By allocating more privacy budget and using higher clipping thresholds in the initial training phases, the model avoids suboptimal solutions and enhances performance. Experiments show that SAD-DPSGD outperforms Auto-DPSGD on HAM10000, improving accuracy by 2.15% under $\epsilon = 3.0$ , $\delta = 10^{-3}$.

Figures

Figures reproduced from arXiv: 2507.06619 by the authors.

Figure 1
Figure 1. the performance of the model during the training [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. distribution of HAM10000 dataset and n = 3. The below [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. relationship between σt and β (a) (b) (c) (d) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Relationship between σt and γ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: relationship between σt and n [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Effects of σ decay parameter with ϵ = 3.0, δ = 1e−3, γ = 1, n = 3 balancing the trade-offs between the initial σ and the mean and variance of σt to achieve the best model performance. C. Performance of Different Algorithms within Different Pri￾vacy Setting [PITH_FULL_…
Figure 8
Figure 8. Figure 8: performance of different algorithms [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: comparison of accuracy of the majority group and the [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 12 canonical work pages

  1. [1]

    Neither private nor fair: Impact of data imbalance on utility and fairness in differential privacy,

    T. Farrand, F. Mireshghallah, S. Singh, and A. Trask, “Neither private nor fair: Impact of data imbalance on utility and fairness in differential privacy,” in Proceedings of the 2020 workshop on privacy-preserving machine learning in practice , 2020, pp. 15–19

  2. [2]

    Reconciling privacy and accuracy in ai for medical imaging,

    A. Ziller, T. T. Mueller, S. Stieger, L. F. Feiner, J. Brandt, R. Braren, D. Rueckert, and G. Kaissis, “Reconciling privacy and accuracy in ai for medical imaging,” Nature Machine Intelligence , vol. 6, no. 7, pp. 764–774, 2024

  3. [3]

    The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,

    P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” Scientific data, vol. 5, no. 1, pp. 1–9, 2018

  4. [4]

    Differential Privacy Under Class Imbalance: Methods and Empirical Insights

    L. Rosenblatt, Y . Lut, E. Turok, M. Avella-Medina, and R. Cummings, “Differential privacy under class imbalance: Methods and empirical insights,” arXiv preprint arXiv:2411.05733 , 2024

  5. [5]

    Adaptive privacy preserving deep learning algorithms for medical data,

    X. Zhang, J. Ding, M. Wu, S. T. Wong, H. Van Nguyen, and M. Pan, “Adaptive privacy preserving deep learning algorithms for medical data,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2021, pp. 1169–1178

  6. [6]

    A study on adaptive gradient clipping algorithms for differential privacy: Enhancing cyber security and trust,

    S. Chen and W. Liang, “A study on adaptive gradient clipping algorithms for differential privacy: Enhancing cyber security and trust,” in 2024 IEEE Cyber Science and Technology Congress (CyberSciTech) . IEEE, 2024, pp. 100–106

  7. [7]

    Deep learning with differential privacy,

    M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016, pp. 308–318

  8. [8]

    Learning rate adaptation for differentially private learning,

    A. Koskela and A. Honkela, “Learning rate adaptation for differentially private learning,” in International Conference on Artificial Intelligence and Statistics. PMLR, 2020, pp. 2465–2475

Show all 21 references
  1. [9]

    Dynamic differential- privacy preserving sgd,

    J. Du, S. Li, X. Chen, S. Chen, and M. Hong, “Dynamic differential- privacy preserving sgd,” arXiv preprint arXiv:2111.00173 , 2021

  2. [10]

    On the convergence and calibration of deep learning with differential privacy,

    Z. Bu, H. Wang, Z. Dai, and Q. Long, “On the convergence and calibration of deep learning with differential privacy,” Transactions on machine learning research, vol. 2023, pp. https–openreview, 2023

  3. [11]

    Disparate im- pact in differential privacy from gradient misalignment,

    M. S. Esipova, A. A. Ghomi, Y . Luo, and J. C. Cresswell, “Disparate im- pact in differential privacy from gradient misalignment,” arXiv preprint arXiv:2206.07737, 2022

  4. [12]

    Dpadamod agc: Adaptive gradient clipping-based differential privacy,

    J. Zhang, W. Yang, Y . Zhang, H. Zheng, and T. Zhang, “Dpadamod agc: Adaptive gradient clipping-based differential privacy,” in 2024 27th International Conference on Computer Supported Cooperative Work in Design (CSCWD). IEEE, 2024, pp. 950–955

  5. [13]

    Differential privacy has disparate impact on model accuracy,

    E. Bagdasaryan, O. Poursaeed, and V . Shmatikov, “Differential privacy has disparate impact on model accuracy,” Advances in neural informa- tion processing systems , vol. 32, 2019

  6. [14]

    Diversity in faces,

    M. Merler, N. Ratha, R. S. Feris, and J. R. Smith, “Diversity in faces,” arXiv preprint arXiv:1901.10436 , 2019

  7. [15]

    Privacy in deep learning: A survey,

    F. Mireshghallah, M. Taram, P. Vepakomma, A. Singh, R. Raskar, and H. Esmaeilzadeh, “Privacy in deep learning: A survey,” arXiv preprint arXiv:2004.12254, 2020

  8. [16]

    Demographic dialectal variation in social media: A case study of african-american english,

    S. L. Blodgett, L. Green, and B. O’Connor, “Demographic dialectal variation in social media: A case study of african-american english,” arXiv preprint arXiv:1608.08868 , 2016

  9. [17]

    The inaturalist species classifi- cation and detection dataset,

    G. Van Horn, O. Mac Aodha, Y . Song, Y . Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, “The inaturalist species classifi- cation and detection dataset,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 8769–8778

  10. [18]

    Differential privacy: A survey of results,

    C. Dwork, “Differential privacy: A survey of results,” in International conference on theory and applications of models of computation . Springer, 2008, pp. 1–19

  11. [19]

    Calibrating noise to sensitivity in private data analysis,

    C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 . Springer, 2006, pp. 265–284

  12. [20]

    R ´enyi divergence and kullback-leibler divergence,

    T. Van Erven and P. Harremos, “R ´enyi divergence and kullback-leibler divergence,” IEEE Transactions on Information Theory , vol. 60, no. 7, pp. 3797–3820, 2014

  13. [21]

    R ´enyi differential privacy,

    I. Mironov, “R ´enyi differential privacy,” in 2017 IEEE 30th computer security foundations symposium (CSF) . IEEE, 2017, pp. 263–275

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.