REVIEW 3 major objections 5 minor 28 references
Optimizing Noise Distributions for Differential Privacy
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Noise distributions optimized for a fixed composition count and variance deliver smaller (ε, δ)-DP guarantees than Gaussian, Laplace, Staircase, or Cactus noise in the moderate composition regime.
desk verdict A credible, reproducible empirical improvement over Gaussian/Laplace noise for moderate composition, with the 'optimal' label running ahead of the actual guarantees. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is $g_\alpha(P_Z) = \max_{t\in S} \mathbb{E}_{P_Z}\left[\left(P_Z/T_t P_Z\right)^{\alpha-1}\right]$, the worst-case term inside the logarithm of the Renyi divergence of order $\alpha$ over shifts $t$ within the sensitivity set $S$. Theorem 3.1 proves that $g_\alpha(P_Z)$ is convex in the distribution and that symmetry is sufficient, so after restricting to symmetric piecewise-constant densities (or PMFs) with geometric tails—Definitions 3.3 and 3.4—the search becomes a finite-dimensional convex problem over bin probabilities $p_0,\dots,p_N$ (Theorem 3.6). The solver is a preconditioned gradient descent with scaling matrix $M=\mathrm{diag}(p)^{-1}$, which keeps iterates strictly inside the positive orthant and thereby avoids the infinite Renyi divergence that arises when supports shrink, and a Newton update of $\alpha$ that follows the moments-accountant objective $\gamma_{p,N_c,\delta}(\alpha)$.
What would settle it
For $s=1$, $\sigma=8$, $N_c=10$, $\delta=10^{-6}$, search a much finer family—a high-resolution histogram with no geometric-tail restriction and a dense grid over $\alpha$—and evaluate every candidate with Connect-the-Dots; any distribution with variance $64$ and $\varepsilon$ below $1.62$ would refute the paper's optimality claim.
Extended reading notes
Core claim
For a fixed sensitivity $s$, noise variance $\sigma^2$, target $\delta$, and composition count $N_c$, the paper constructs a symmetric, piecewise-constant density (or, for integer outputs, a symmetric probability mass function) with geometric tails that minimizes the Renyi differential privacy parameter at an order $\alpha$ selected to track the moments-accountant optimum for $N_c$ and $\delta$. The central discovery is that this distribution dominates Gaussian, Laplace, Staircase, and Cactus noise at the same variance in the moderate composition regime (roughly 10 to 40 compositions), with the largest gaps appearing for moderate $\alpha$ values, and that the same framework automatically reproduces Cactus for very large $N_c$ and Staircase for a single composition. All reported $(\varepsilon,\delta)$ curves are evaluated with Connect-the-Dots accounting, so the comparison uses the same tight privacy accountant practitioners rely on.
Load-bearing premise
The paper assumes that the best noise distribution is well captured by a symmetric step-shaped density whose tails decay geometrically from a hand-chosen bin, and that a Newton shortcut finds the right Renyi order $\alpha$; if the true optimum lives outside this family or the shortcut stalls at a local optimum, the output is only a local optimum, not the global one.
Editorial extensions
If this is right
- For any deployment with a known composition count in the moderate regime, the optimized mechanism gives a smaller $\varepsilon$ than Gaussian or Laplace at the same variance, which is a privacy improvement at zero added noise.
- Practitioners no longer need to choose among Staircase, Cactus, Gaussian, and Laplace: the same algorithm selects the right shape automatically as $\alpha$ varies with $N_c$ and $\delta$.
- For integer-valued queries, the discrete version yields noise that beats discrete Gaussian and discrete Laplace noise while staying immune to floating-point attacks.
- Because the optimized Renyi DP depends only on the ratio $\sigma/s$, the same relative noise level transfers across query sensitivities.
- The reported gains translate into roughly 8 to 12% lower mean-squared error on real datasets when the privacy budget is held fixed.
Reading between the lines
- A testable extension is to relax the geometric-tail restriction: comparing against a much larger $N$ and finer bin width, or a non-geometric tail, would quantify whether the family restriction loses anything in the moderate regime.
- The same recipe—optimize a Renyi surrogate at a tuned $\alpha$, then convert via the moments accountant—could be applied to other cost functions such as $\ell^1$ cost or heterogeneous per-query costs, where symmetric variance-constrained shapes may not be optimal.
- A practical consequence the paper does not develop: for machine-learning training with a moderate number of update rounds, replacing Gaussian noise with this mechanism could lower the noise multiplier at a fixed privacy budget, provided adaptive composition and subsampling behave for this noise as they do for Gaussian.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a unified optimization framework for designing continuous and discrete noise distributions for differential privacy by minimizing Rényi divergence of a tunable order α under a cost (variance) constraint. The infinite-dimensional distribution search is reduced to a finite-dimensional convex problem over symmetric piecewise-constant (continuous) or PMF (discrete) distributions with geometric tails, controlled by hyperparameters N, r, and Δ. The resulting problem is solved by a preconditioned gradient descent algorithm that also updates α heuristically via a Newton step based on the moments accountant. The authors demonstrate that the method recovers the Staircase and Cactus mechanisms in the single- and large-composition limits, and report improved ε for fixed δ and σ in moderate composition regimes, with all final privacy guarantees computed using the Connect-the-Dots accountant.
Significance. If the reported gains are stable, the paper delivers a practical and unified recipe for noise design that outperforms the standard Gaussian and Laplace mechanisms at equal variance in moderate composition regimes, and the recovery of known optimal mechanisms in limiting regimes is a strong sanity check. The paper also ships code, evaluates with a state-of-the-art accountant, and provides a clean convexity reduction (Theorem 3.1) that is rigorous. The main caveats are that the 'optimal' label is tied to a surrogate moments-accountant objective and to a hand-chosen parametric family, and the numerical optimization lacks convergence guarantees; these issues do not invalidate the empirical comparison but do narrow the scope of the claims.
major comments (3)
- [§3.1, Definitions 3.3–3.4; §5] The finite-dimensional restriction to symmetric piecewise-constant distributions with geometric tails is asserted (Section 3.1: 'this tail assumption does not significantly impact the optimization, provided that the interval is sufficiently large') to closely approximate the true optimizer, but no approximation error bound or guidance on N, r, and Δ is given. The numerical sections fix values such as N=8000, Δ=0.01, r=0.9999 with no sensitivity analysis. If the reported ε at the target δ changes materially when N is doubled or Δ is halved, the headline improvements in Figures 4–5 and Table 1 are artifacts of under-optimization rather than properties of the true optimizer within the family. Please add a sensitivity study or prove an approximation bound before claiming optimality.
- [§4, Algorithm 3; Algorithm 1] The optimization minimizes the moments-accountant surrogate inf_α [N_c/(α−1) log g_α(p) + log(1/δ)/(α−1)] (Eq. (25)), not the true ε or the Connect-the-Dots accountant used for evaluation. Algorithm 1's output description ('Optimal distribution P* minimizing ε') is therefore overstated. Moreover, the preconditioned gradient descent on the nonsmooth objective max_t g_α(p,t) with a backtracking line search over a finite geometric step-size sequence, together with the Newton update for α (line 21), has no convergence or global-optimality guarantee. The statement in Section 4 that 'gradient descent will converge to the optimal objective value' is not justified. Please either provide a convergence analysis or rephrase to 'heuristic optimization' and add empirical evidence such as multiple restarts or a comparison with a certified convex solver for fixed α.
- [Theorem 3.6, Eq. (23); Appendix C] The reduction of the continuous optimization to shifts t in {1,...,s/Δ} relies on piecewise linearity of g_α in t. The derivation in Appendix C is plausible, but the final closed form in Eq. (23) is dense and I could not independently verify every index; for instance, the last sum mixes p^α_{|t+j|} with r^{-j(1−α)} and the correspondence with Eq. (75) of Appendix C is not immediate. Since (23) is exactly the objective that the convex solver minimizes, please provide a cleaner step-by-step derivation, a machine-checkable verification, or a statement that the algebra has been symbolically verified, and explicitly specify how the case a=0 is handled in the sums.
minor comments (5)
- [Algorithm 3, line 10] The definition of μ_ub assumes that at least one component of g_proj is positive; if all components are non-positive, μ_ub is undefined. Please specify the fallback behavior (e.g., terminate the line search or set μ_ub to a large value).
- [Algorithm 3, line 21] The Newton update for α is unconstrained and may push α below 1, which is outside the valid range for Rényi divergence. Please add a safeguard such as projecting α to [1+ε, ∞).
- [Figure 4 caption and surrounding text] The caption says RDP noise is marked as the winner when it achieves more than a 2% improvement, but the text then says 'Even when another distribution is marked as the best, our noise still consistently outperforms the others, although by less than 2%.' This is confusing; please clarify the exact criterion for marking the winner and what is meant by 'outperforms' in the latter sentence.
- [Table 1 and Appendix D] The text states that Gaussian noise consistently outperformed Laplace, Cactus, and Staircase across all tested settings, but the table only reports improvements over Gaussian. Since the paper's contribution is comparison with all baselines, please include the relative performance against the other baselines for at least one representative setting, or provide a supplementary table.
- [Eq. (25) and Algorithm 3] The notation γ' and γ'' for the Newton update refers to derivatives with respect to α, but γ_{p,N_c,δ}(α) depends on α both through the explicit denominator and through g_α(p). Please clarify whether the derivatives account for the α-dependence of g_α(p) and how the derivative is computed numerically.
Circularity Check
No significant circularity: the RDP/moments-accountant surrogate used for optimization is distinct from the Connect-the-Dots accountant used for evaluation, so the reported gains are empirical rather than true by construction.
full rationale
The paper's core empirical claim is that its optimized RDP noise achieves smaller (epsilon, delta) at fixed variance in moderate composition regimes. The optimization (Algorithms 1-3, Eq. 22, Eq. 25) minimizes a finite-dimensional convex surrogate based on Renyi divergence and the simple moments-accountant bound of Proposition 2.4. Section 5 explicitly states that 'All (epsilon, delta)-DP guarantees presented here are computed using the Connect-the-Dots accounting (Doroshenko et al., 2022)', a different, external accountant. Hence the reported epsilon values are not the optimized objective, and the superiority over Gaussian and Laplace is not an identity or a fitted tautology. Recovery of Staircase (alpha toward infinity) and Cactus (alpha toward 1) is presented as a numerical sanity check against independently known mechanisms, not as the source of the moderate-regime claim; no uniqueness theorem from the authors' prior work is invoked to exclude alternatives. The finite-dimensional piecewise-constant geometric-tail family (Definitions 3.3-3.4) is a computational restriction, and Section 3.1 concedes without proof that the tail restriction 'does not significantly impact the optimization, provided that the interval is sufficiently large.' That is a missing approximation bound, not a circular reduction; likewise, the Newton alpha update in Algorithm 3 has no convergence guarantee, and the hyperparameters N, r, delta are not subjected to sensitivity analysis. These are robustness and correctness concerns, not circularity. Self-citations to Alghamdi et al. (2022, 2023) and Asoodeh et al. (2021) provide baselines and context, but the central comparison is self-contained against external benchmarks, so the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- N (number of bins in distribution family) =
1600 to 8000 in experiments
- r (geometric tail decay rate) =
0.9999 in experiments
- Delta (bin width) =
0.005 to 0.01 in experiments, subject to s/Delta integer
- T (alpha update time step) and learning rate schedule =
not specified precisely
assumptions (6)
- standard math g_alpha(P) = max_t E_P[(P / T_t P)^(alpha-1)] is convex in P for alpha > 1
- standard math A symmetric distribution suffices for the optimal noise
- domain assumption The worst-case shift for a piecewise constant PDF occurs at integer multiples of the bin width
- ad hoc to paper The geometric-tail family (Definitions 3.3 and 3.4) approximates the true optimum well
- domain assumption Moments accountant conversion epsilon = gamma(alpha) + log(1/delta) / (alpha-1) with optimal alpha
- domain assumption Connect-the-Dots accounting correctly certifies the (epsilon, delta) guarantees of the optimized distributions
Cite this review
Pith. "Pith review of Optimizing Noise Distributions for Differential Privacy." pith.science (2026). https://pith.science/paper/QWM3OJOM
@misc{pith2026250414730,
author = {Pith},
title = {Pith review of: Optimizing Noise Distributions for Differential Privacy},
year = {2026},
howpublished = {\url{https://pith.science/paper/QWM3OJOM}},
note = {Machine review of arXiv:2504.14730}
}
abstract
We propose a unified optimization framework for designing continuous and discrete noise distributions that ensure differential privacy (DP) by minimizing R\'enyi DP, a variant of DP, under a cost constraint. R\'enyi DP has the advantage that by considering different values of the R\'enyi parameter $\alpha$, we can tailor our optimization for any number of compositions. To solve the optimization problem, we reduce it to a finite-dimensional convex formulation and perform preconditioned gradient descent. The resulting noise distributions are then compared to their Gaussian and Laplace counterparts. Numerical results demonstrate that our optimized distributions are consistently better, with significant improvements in $(\varepsilon, \delta)$-DP guarantees in the moderate composition regimes, compared to Gaussian and Laplace distributions with the same variance.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
https://github.com/SankarLab/Renyi-DP-Mechanism-Design, May 2025
Renyi DP mechanism design. https://github.com/SankarLab/Renyi-DP-Mechanism-Design, May 2025
work page 2025
-
[3]
B., Mironov, I., Talwar, K., and Zhang, L
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp.\ 308--318, 2016
2016
-
[4]
The 2020 Census Disclosure Avoidance System TopDown Algorithm
Abowd, J., Ashmead, R., Cumings-Menon, R., Garfinkel, S., Heineck, M., Heiss, C., Johns, R., Kifer, D., Leclerc, P., Machanavajjhala, A., Moran, B., Sexton, W., Spence, M., and Zhuravlev, P. The 2020 Census Disclosure Avoidance System TopDown Algorithm . Harvard Data Science Review, 0 (Special Issue 2), jun 24 2022. https://hdsr.mitpress.mit.edu/pub/7evz361i
work page 2020
-
[5]
P., Kosut, O., Sankar, L., and Wei, F
Alghamdi, W., Asoodeh, S., Calmon, F. P., Kosut, O., Sankar, L., and Wei, F. Cactus mechanisms: Optimal differential privacy mechanisms in the large-composition regime. In 2022 IEEE International Symposium on Information Theory (ISIT), pp.\ 1838--1843, 2022. doi:10.1109/ISIT50566.2022.9834438
arXiv 2022
-
[6]
P., Felipe Gomez, J., Kosut, O., and Sankar, L
Alghamdi, W., Asoodeh, S., Calmon, F. P., Felipe Gomez, J., Kosut, O., and Sankar, L. Schrödinger mechanisms: Optimal differential privacy mechanisms for small sensitivity. In 2023 IEEE International Symposium on Information Theory (ISIT), pp.\ 2201--2206, 2023. doi:10.1109/ISIT54713.2023.10206616
arXiv 2023
-
[7]
Asoodeh, S., Liao, J., Calmon, F. P., Kosut, O., and Sankar, L. Three variants of differential privacy: Lossless conversion and applications. IEEE Journal on Selected Areas in Information Theory, 2 0 (1): 0 208--222, 2021
work page 2021
-
[8]
Balcer, V. and Vadhan, S. P. Differential privacy on finite computers. In Information Technology Convergence and Services, 2017. URL https://api.semanticscholar.org/CorpusID:6692720
work page 2017
Show all 28 references
-
[9]
Hypothesis testing interpretations and R\'enyi differential privacy
Balle, B., Barthe, G., Gaboardi, M., Hsu, J., and Sato, T. Hypothesis testing interpretations and R\'enyi differential privacy. In Int. Conf. Art. Intelligence and Stat. (AISTAT), pp.\ 2496--2506, 2020
2020
-
[10]
and Steinke, T
Bun, M. and Steinke, T. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Proc. Theory of Cryptography, pp.\ 635--658, 2016
2016
-
[11]
L., Kamath, G., and Steinke, T
Canonne, C. L., Kamath, G., and Steinke, T. The discrete Gaussian for differential privacy. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546
2020
-
[12]
G., and Woodruff, D
Chen, Y., Thakurta, A. G., and Woodruff, D. P. Poisson subsampled rényi differential privacy. In Proceedings of the 52nd Annual ACM Symposium on Theory of Computing (STOC), pp.\ 849--860. ACM, 2019. doi:10.1145/3313276.3316369
2019
-
[13]
Connect the dots: Tighter discrete approximations of privacy loss distributions, 2022
Doroshenko, V., Ghazi, B., Kamath, P., Kumar, R., and Manurangsi, P. Connect the dots: Tighter discrete approximations of privacy loss distributions, 2022. URL https://arxiv.org/abs/2207.04380
2022 arXiv
-
[14]
Our data, ourselves: Privacy via distributed noise generation
Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., and Naor, M. Our data, ourselves: Privacy via distributed noise generation. In Vaudenay, S. (ed.), EUROCRYPT, pp.\ 486--503, 2006 a
2006
-
[15]
Calibrating noise to sensitivity in private data analysis
Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Proc. Theory of Cryptography (TCC), pp.\ 265--284, Berlin, Heidelberg, 2006 b . ISBN 3-540-32731-2, 978-3-540-32731-8
2006
-
[16]
N., and Rothblum, G
Feldman, V., Acharya, A., Dinesh, P. N., and Rothblum, G. N. On the rényi differential privacy of the shuffle model. In Proceedings of the 52nd Annual ACM Symposium on Theory of Computing (STOC), pp.\ 1232--1244. ACM, 2021. doi:10.1145/3460120.3484794
2021
-
[17]
V., Muthukrishnan, S., and Kuczynski, J
Feldman, V., Liu, Y., Karpov, D. V., Muthukrishnan, S., and Kuczynski, J. Stronger privacy amplification by shuffling for rényi and approximate differential privacy. arXiv preprint arXiv:2208.04591, 2022. URL https://arxiv.org/abs/2208.04591
2022 arXiv
-
[18]
The staircase mechanism in differential privacy
Geng, Q., Kairouz, P., Oh, S., and Viswanath, P. The staircase mechanism in differential privacy. IEEE Journal of Selected Topics in Signal Processing, 9 0 (7): 0 1176--1184, 2015. doi:10.1109/JSTSP.2015.2425831
2015
-
[19]
Universally utility-maximizing privacy mechanisms
Ghosh, A., Roughgarden, T., and Sundararajan, M. Universally utility-maximizing privacy mechanisms. SIAM J. Comput., 41 0 (6): 0 1673–1693, January 2012. ISSN 0097-5397. doi:10.1137/09076828X. URL https://doi.org/10.1137/09076828X
2012 doi
-
[20]
Heart disease data set
Janosi, A., Steinbrunn, W., Pfisterer, M., and Detrano, R. Heart disease data set. https://archive.ics.uci.edu/ml/datasets/Heart+Disease, 1988. Accessed: 2025-04-29
1988
-
[21]
Diabetes dataset
learn developers, S. Diabetes dataset. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_diabetes.html. Accessed: 2025-04-29
2025
-
[22]
M., Smith, A., and Vadhan, S
Lécuyer, M., Lee, J. M., Smith, A., and Vadhan, S. Practical privacy filters and odometers with rényi differential privacy and applications to differentially private deep learning. arXiv preprint arXiv:2103.01379, 2021. URL https://arxiv.org/abs/2103.01379
2021 arXiv
-
[23]
On significance of the least significant bits for differential privacy
Mironov, I. On significance of the least significant bits for differential privacy. In Proceedings of the 2012 ACM Conference on Computer and Communications Security, CCS '12, pp.\ 650–661, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450316514. doi:1...
2012
-
[24]
R\'enyi differential privacy
Mironov, I. R\'enyi differential privacy. In Proc. IEEE Comp. Security Foundations Symp. (CSF), pp.\ 263--275, 2017
2017
-
[25]
and Harremos, P
van Erven, T. and Harremos, P. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60 0 (7): 0 3797--3820, 2014. doi:10.1109/TIT.2014.2320500
2014
-
[26]
Moore–Penrose inverse
Wikipedia . Moore–Penrose inverse . https://en.wikipedia.org/wiki/Moore
-
[27]
Tonelli’s theorem for non-negative measurable functions
Wikipedia . Tonelli’s theorem for non-negative measurable functions . https://en.wikipedia.org/wiki/Fubini
-
[28]
H., Mangasarian, O
Wolberg, W. H., Mangasarian, O. L., and Street, W. N. Breast cancer wisconsin (diagnostic) data set. https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic), 1993. Accessed: 2025-04-29
1993
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.