REVIEW 2 major objections 6 minor 33 references
Curator Attack: When Blackbox Differential Privacy Auditing Loses Its Power
T0 review · 2 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Blackbox differential privacy auditing can be fooled by violations hidden in the small-probability region it ignores.
desk verdict A credible and useful formalization of a real blind spot in blackbox DP auditing, but the 'ubiquitous FPs' claim overshoots what the fixed-threshold analysis proves. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the false-positive condition $\xi^* \le \epsilon_c < \epsilon^*$, where $\epsilon^*$ is the strongest DP guarantee the mechanism actually affords, $\epsilon_c$ is the curator's claim, and $\xi^*$ is the auditor's maximal empirical power. The mechanism that makes the condition provable is probability/density truncation: DP-Sniper replaces $\Pr[M(A)\in S]$ with $\Pr_{\ge c}[M(A)\in S] = \max\{\Pr[M(A)\in S], c\}$, MPL replaces densities with $\max\{p(b|A), \tau\}$, and $\Delta$-Siege's surrogate $\rho(\epsilon,\delta)$ moves along contours that generally miss the mechanism's true $\epsilon$–$\delta$ tradeoff curve. Each theorem converts the two inequalities $\epsilon^*(\theta) > \epsilon_c$ and $\xi^*(\theta) \le \epsilon_c$ into explicit parameter ranges, with prerequisites such as $\Pr[M_\theta(a')\in S^*] < c$ ensuring that the auditor's empirical witness departs from the theoretical optimal one. What this machinery shows is that any violation small enough to fall inside the truncated region is invisible to the auditor by construction.
What would settle it
Run the benchmark Laplace mechanism with a noise scale $\theta$ inside the paper's Theorem 1 window, but have the auditor draw its threshold $c$ uniformly at random from $(0, 0.1)$ after the curator fixes $\theta$, so the curator cannot solve for $c$ in advance. If the audit still certifies some $\epsilon_c < \theta$, the ubiquity claim survives; if the false positives disappear, the attack is tied to the threshold being public and fixed.
Extended reading notes
Core claim
The paper's central claim is that false positives are structural, not accidental: a blackbox auditor that caps small probabilities at a threshold $c$ (DP-Sniper) or small densities at $\tau$ (MPL) can inspect only a segment of the mechanism's type-I/type-II error tradeoff curve, and the unobserved tail is a region where the curator can violate privacy freely. Definition 3 makes the failure precise: $M$ is a false positive against an auditor exactly when $\xi^* \le \epsilon_c < \epsilon^*$, so $M$ violates the claimed $(\epsilon_c,\delta_c)$-DP yet the audit certifies it. For the Laplace mechanism audited by DP-Sniper the paper proves this happens iff $\Pr[M_\theta(a') \in S^*] < c$, and it derives analogous inequalities for SVT with discrete output vectors, for Gaussian and one-step DPSGD mechanisms, and for $\Delta$-Siege where the choice of privacy surrogate $\rho(\epsilon,\delta)$ can produce both false positives and false negatives. These theoretical conditions are then confirmed with experiments that reuse the auditors' own implementations.
Load-bearing premise
The construction assumes the curator knows the auditor's fixed threshold for ignoring small probabilities and the exact adjacent-dataset heuristic the auditor uses; if the auditor randomizes or hides those choices, the derived false-positive windows no longer follow.
Editorial extensions
If this is right
- A curator can deploy a benchmark Laplace mechanism with scale $\theta > -\ln(2c)$ and claim any $\epsilon_c$ lying between the auditor's power and the true level; the audit certifies the claim even though $\theta > \epsilon_c$.
- With bounded-noise adaptations of Laplace and SVT, the true privacy level $\epsilon^*$ becomes infinite, yet the mechanisms pass DP-Sniper and MPL for finite claimed $\epsilon_c$, so the attack does not depend on borderline parameters.
- For Delta-Siege, the Gaussian mechanism can be a false positive for some noise scales and a false negative for others under the same privacy surrogate, so the same auditor both certifies violations and rejects valid claims.
- For one-step DPSGD, even a moderate threshold that gives a narrow confidence interval leaves false-positive claims, meaning the flaw extends to machine learning audits, not just statistical query mechanisms.
Reading between the lines
- An extension the paper leaves implicit: an auditor that randomizes its truncation threshold $c$ or $\tau$ on each run and keeps the value secret would break the curator's ability to solve the inequalities in advance, without opening the blackbox.
- A practical warning not spelled out: the width of the region below the auditor's threshold is itself a risk signal; reporting how much of the tradeoff curve was never examined would tell data owners how conditional the certification is.
- The same recipe should transfer to any future blackbox auditor with a fixed nonparametric estimator: locate the probability or density level below which estimation is capped or unreliable, then place a violation inside that region.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that blackbox differential privacy auditing is inherently unreliable because it ignores small probabilities or densities. It frames the problem as hypothesis testing over the auditor itself: the null hypothesis is that the curator's (ε_c, δ_c)-DP claim is valid, and a false positive is defined as the event ξ* ≤ ε_c < ε*, i.e., the mechanism violates the claimed guarantee but passes the audit. The authors derive analytic conditions (Theorems 1–9) under which benchmark mechanisms (Laplace, SVT, Gaussian, one-step DPSGD) and adapted mechanisms (with crafted noise distributions such as gLap and modified SVT noise) become false positives or false negatives against DP-Sniper, MPL, Delta-Siege, and DPSGD-Audit. They validate the construction experimentally using the official code of the auditors and conclude that false positives are ubiquitous in blackbox auditing.
Significance. If the central claim holds, this is a practically important cautionary result: it shows that fixed-threshold blackbox auditors can be evaded by mechanisms whose privacy violations lie in the probability/density region the auditor truncates. The main strength is that the theoretical conditions are derived from the auditors' published algorithms and thresholds rather than fitted to experimental outcomes, and the experiments reuse the auditors' official code, which reduces implementation-bias concerns. The paper also provides a systematic template (Prerequisites P1–P3 and Algorithm 1) for finding future false-positive attacks. However, the universality of the claim is currently bounded by the assumption that the curator knows the auditor's fixed threshold and adjacent-input heuristic, and several load-bearing proofs are deferred or omitted.
major comments (2)
- [Section 5.2, Section 6.2, Section 8] Several load-bearing theorems are not proved in the manuscript: Theorem 3's proof is deferred to an external repository ([2]), Theorem 4's proof is deferred to Appendix D [2], Theorem 6's proof is omitted, and Theorem 9's proof is deferred to Appendix I [2]. These theorems provide the analytic false-positive conditions for SVT and DPSGD, which are central to the paper's empirical claims. The manuscript as submitted is therefore not self-contained, and the reader cannot verify the main theoretical results without consulting the external supplement.
- [Section 9, Section 10] The inequalities in Theorems 1–9 concern the exact theoretical power ξ*, while the actual auditors return finite-sample confidence intervals. The paper states in Section 9 that only the lower bound of the confidence interval needs to be reported, but it does not prove that this empirical lower bound is close enough to the theoretical ξ* for the false-positive conditions to hold with high probability. A one-sided confidence interval with lower bound below ε_c does not imply that the true power is below ε_c; it only means the audit cannot reject the claim at the stated confidence level. The experiments provide illustrative evidence, but the formal relationship between the analytic FP conditions and the statistical decision rule of each auditor still needs to be stated.
minor comments (6)
- [Section 6.2, Theorem 6] The statement of Theorem 6 says 'with probability threshold τ,' but for MPL the threshold τ is a density threshold, as defined in Section 2.2 and Prerequisite 2. This should be corrected to 'density threshold τ.'
- [Figure 2] Figure 2 contains the stray text '在此处键入公式' ('insert formula here') in panels (a) and (c), which appears to be an editing artifact and should be removed.
- [Section 9] The text says DPSGD-Audit reports a two-sided interval '[ξ*, ξ*)', which as written is empty; this is presumably a typographical loss of the upper endpoint and should read '[ξ*_lower, ξ*_upper)' or the equivalent notation.
- [Section 11] The phrase 'As such, As such,' is duplicated in the Related Work section and should be reduced to a single occurrence.
- [Section 3, Definition 3] The paper's use of 'false positive' is nonstandard: in the usual hypothesis-testing vocabulary, the event 'mechanism violates DP but the auditor passes it' is a Type II error (false negative), not a Type I error. Although the definition is explicit and internally consistent, the Abstract's phrase 'false positives are ubiquitous' may mislead readers familiar with the standard convention; a brief remark explaining the opposite naming convention would improve clarity.
- [Section 5.1, proof of Theorem 1] The proof fixes the adjacent pair q(a)=0, q(a')=Δ and calls it optimal 'intuitively,' but it does not prove that this pair maximizes the ratio over all adjacent pairs nor that this is the pair selected by the auditors' GenerateInputs() heuristic. A short argument justifying the choice of the adjacent pair would make the 'iff' claim fully rigorous.
Circularity Check
No significant circularity: the FP constructions are derived analytically from the auditors' published algorithms and thresholds; the central claim does not reduce to a fitted input or a self-citation chain.
full rationale
The paper's central derivation—that certain mechanisms satisfy Definition 3's FP condition—does not reduce to its inputs. Definition 3 (Eq. 6) defines a false positive as xi* <= epsilon_c < epsilon*, and the paper then proves, rather than assumes, that specific benchmark and adapted mechanisms meet this condition. For example, Theorem 1 computes S* = (-inf, 0] for the Laplace mechanism, Pr[M_theta(a') in S*] = e^{-theta}/2, and DP-Sniper's empirical power xi* from its one-sided truncated interval, yielding Eq. (8) as a genuine analytical consequence. The prerequisite conditions (P1, P2, P3) are stated as iff conditions and used to construct adapted mechanisms, which is a standard constructive-attack methodology rather than circular reasoning. The experiments reuse the official code of DP-Sniper, MPL, and Delta-Siege and the published GenerateInputs() heuristics from Table 5, so the empirical validation is independent of the authors' own fitted values. No load-bearing self-citation was found: references to the authors' companion repository [2] are for deferred proofs and code, not for an unverified mathematical premise. The main caveats are that several proofs are deferred to appendices in [2] and that the attack assumes the curator knows the auditor's fixed truncation threshold and input-pattern heuristic; randomizing c or tau, or changing the adjacent-input pattern, would alter xi* and could shrink or eliminate the derived FP intervals. These are threat-model and completeness concerns, not circularity, because the derivation does not become equivalent to its assumptions merely because it is conditional on those assumptions. The Section 9 generalization is a recipe, but it is presented as a recipe and is not used to claim a proof of universality. Overall, the paper's construction is self-contained against the published auditor algorithms and does not fit a parameter and then rename it a prediction.
Assumptions & free parameters
free parameters (3)
- Auditor probability threshold c (DP-Sniper) and density threshold tau (MPL) =
c = 0.01 or 0.05; tau = 1e-4 in experiments
- Noise scale theta of benchmark attack mechanisms =
Values solving the theorem inequalities, e.g., theta in (4, 4.7) for Laplace against DP-Sniper at c = 0.01
- Adapted noise distribution parameters e_theta1, e_theta2 (gLap, uniform rho, rescaled Laplace nu) =
Chosen from solution sets of Theorems 2, 4, 5, and 6
assumptions (4)
- standard math The f-DP tradeoff characterization of (epsilon, delta)-DP from Dong-Roth-Su [15] applies to all mechanisms considered.
- domain assumption The procedural descriptions of DP-Sniper, MPL, Delta-Siege, and DPSGD-Audit in Table 2 are accurate representations of the published tools.
- domain assumption The auditor uses the input-pattern heuristics of Table 5 from [11] when selecting adjacent datasets.
- domain assumption The true privacy level epsilon* is analytically computable for the benchmark Laplace, SVT, Gaussian, and DPSGD mechanisms.
invented entities (2)
-
gLap bounded or truncated Laplace noise distribution
-
Adapted SVT noise modifications (uniform rho and rescaled Laplace nu)
Cite this review
Pith. "Pith review of Curator Attack: When Blackbox Differential Privacy Auditing Loses Its Power." pith.science (2026). https://pith.science/paper/4GLKG7K7
@misc{pith2026241116516,
author = {Pith},
title = {Pith review of: Curator Attack: When Blackbox Differential Privacy Auditing Loses Its Power},
year = {2026},
howpublished = {\url{https://pith.science/paper/4GLKG7K7}},
note = {Machine review of arXiv:2411.16516}
}
read the original abstract
A surge in data-driven applications enhances everyday life but also raises serious concerns about private information leakage. Hence many privacy auditing tools are emerging for checking if the data sanitization performed meets the privacy standard of the data owner. Blackbox auditing for differential privacy is particularly gaining popularity for its effectiveness and applicability to a wide range of scenarios. Yet, we identified that blackbox auditing is essentially flawed with its setting: small probabilities or densities are ignored due to inaccurate observation. Our argument is based on a solid false positive analysis from a hypothesis testing perspective, which is missed out by prior blackbox auditing tools. This oversight greatly reduces the reliability of these tools, as it allows malicious or incapable data curators to pass the auditing with an overstated privacy guarantee, posing significant risks to data owners. We demonstrate the practical existence of such threats in classical differential privacy mechanisms against four representative blackbox auditors with experimental validations. Our findings aim to reveal the limitations of blackbox auditing tools, empower the data owner with the awareness of risks in using these tools, and encourage the development of more reliable differential privacy auditing methods.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[2]
[n.d.]. Our Curator Attack. https://github.com/ShimingWang98/Curator-Attack- When-Blackbox-DP-Auditing-Loses-Its-Power
-
[1]
[n.d.]. DP-Sniper Code. https://github.com/eth-sri/dp-sniper. Accessed: 2023- 02-07
work page 2023
-
[3]
John M Abowd. 2018. The US Census Bureau adopts differential privacy. In Pro- ceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2867–2867
work page 2018
-
[4]
Aws Albarghouthi and Justin Hsu. 2017. Synthesizing coupling proofs of dif- ferential privacy. Proceedings of the ACM on Programming Languages 2, POPL (2017), 1–30
work page 2017
-
[5]
Önder Askin, Tim Kutta, and Holger Dette. 2022. Statistical quantification of differential privacy: a local approach. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 402–421
work page 2022
-
[6]
Gilles Barthe, Rohit Chadha, Vishal Jagannath, A Prasad Sistla, and Mahesh Viswanathan. 2020. Deciding differential privacy for programs with finite inputs and outputs. In Proceedings of the 35th Annual ACM/IEEE Symposium on Logic in Computer Science. 141–154
work page 2020
-
[7]
Gilles Barthe, George Danezis, Benjamin Grégoire, César Kunz, and Santiago Zanella-Beguelin. 2013. Verified computational differential privacy with ap- plications to smart metering. In 2013 IEEE 26th Computer Security Foundations Symposium. IEEE, 287–301
work page 2013
-
[8]
Gilles Barthe, Noémie Fong, Marco Gaboardi, Benjamin Grégoire, Justin Hsu, and Pierre-Yves Strub. 2016. Advanced probabilistic couplings for differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. 55–67
work page 2016
Show all 33 references
-
[9]
Gilles Barthe, Marco Gaboardi, Emilio Jesús Gallego Arias, Justin Hsu, César Kunz, and Pierre-Yves Strub. 2014. Proving differential privacy in Hoare logic. In 2014 IEEE 27th Computer Security Foundations Symposium . IEEE, 411–424
2014
-
[10]
Benjamin Bichsel, Timon Gehr, Dana Drachsler-Cohen, Petar Tsankov, and Martin Vechev. 2018. Dp-finder: Finding differential privacy violations by sampling and optimization. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. 508–524
2018
-
[11]
Benjamin Bichsel, Samuel Steffen, Ilija Bogunovic, and Martin Vechev. 2021. Dp- sniper: Black-box discovery of differential privacy violations using classifiers. In 2021 IEEE Symposium on Security and Privacy (SP) . IEEE, 391–409
2021
-
[12]
Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. 2017. Collecting telemetry data privately. Advances in Neural Information Processing Systems 30 (2017)
2017
-
[13]
Zeyu Ding, Yuxin Wang, Guanhong Wang, Danfeng Zhang, and Daniel Kifer
-
[14]
Kashyap Dixit, Madhav Jha, Sofya Raskhodnikova, and Abhradeep Thakurta. 2013. Testing the Lipschitz property over product distributions with applications to data privacy. In Theory of Cryptography: 10th Theory of Cryptography Conference, TCC 2013, Tokyo, Japan, March 3-6, 2013...
2013
-
[15]
Jinshuo Dong, Aaron Roth, and Weijie J Su. 2019. Gaussian differential privacy. arXiv preprint arXiv:1905.02383 (2019)
2019 arXiv
-
[16]
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. 2020. Auditing differen- tially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems 33 (2020), 22205–22216. Curator Attack: When Blackbox Differential Privacy Auditing Lose...
2020
-
[17]
Noah Johnson, Joseph P Near, and Dawn Song. 2018. Towards practical differential privacy for SQL queries. Proceedings of the VLDB Endowment 11, 5 (2018), 526– 539
2018
-
[18]
Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. 2024. Propile: Probing privacy leakage in large language models. Advances in Neural Information Processing Systems 36 (2024)
2024
-
[19]
Xiyang Liu and Sewoong Oh. 2019. Minimax optimal estimation of approximate differential privacy on neighboring databases. Advances in neural information processing systems 32 (2019)
2019
-
[20]
Johan Lokna, Anouk Paradis, Dimitar I Dimitrov, and Martin Vechev. 2023. Group and Attack: Auditing Differential Privacy. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security . 1905–1918
2023
-
[21]
Fred Lu, Joseph Munoz, Maya Fuchs, Tyler LeBlond, Elliott Zaresky-Williams, Edward Raff, Francis Ferraro, and Brian Testa. 2022. A general framework for auditing differentially private machine learning. Advances in Neural Information Processing Systems 35 (2022), 4165–4176
2022
-
[22]
Lyu Min, Su Dong, and Li Ninghui. 2016. Understanding the sparse vector technique for differential privacy
2016
-
[23]
Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tramèr, Matthew Jagielski, Nicholas Carlini, and Andreas Terzis. 2023. Tight Auditing of Differen- tially Private Machine Learning. arXiv preprint arXiv:2302.07956 (2023)
2023 arXiv
-
[24]
Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlin. 2021. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on security and privacy (SP) . IEEE, 866–882
2021
-
[25]
Vishal Jagannath Ravi. 2019. Automated methods for checking differential privacy. (2019)
2019
-
[26]
Thomas Steinke, Milad Nasr, and Matthew Jagielski. 2024. Privacy auditing with one (1) training run. Advances in Neural Information Processing Systems 36 (2024)
2024
-
[27]
Erlingsson Ulfar, Pihur Wasyl, and Korolova Aleksandra. 2014. RAPPOR: Ran- domized aggregatable privacy-preserving ordinal response. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security
2014
-
[28]
Yuxin Wang, Zeyu Ding, Daniel Kifer, and Danfeng Zhang. 2020. Checkdp: An automated and integrated approach for proving differential privacy or finding precise counterexamples. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security . 919–938
2020
-
[29]
Yuxin Wang, Zeyu Ding, Guanhong Wang, Daniel Kifer, and Danfeng Zhang. 2019. Proving differential privacy with shadow execution. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation . 655–669
2019
-
[30]
Royce J Wilson, Celia Yuxin Zhang, William Lam, Damien Desfontaines, Daniel Simmons-Marengo, and Bryant Gipson. 2020. Differentially private SQL with bounded user contribution. Proceedings on privacy enhancing technologies 2020, 2 (2020), 230–250
2020
-
[31]
Jungang Yang, Liyao Xiang, Ruidong Chen, Weiting Li, and Baochun Li. 2021. Differential privacy for tensor-valued queries. IEEE Transactions on Information Forensics and Security 17 (2021), 152–164
2021
-
[32]
Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Ahmed Salem, Victor Rühle, Andrew Paverd, Mohammad Naseri, Boris Köpf, and Daniel Jones. 2023. Bayesian estimation of differential privacy. InInternational Conference on Machine Learning. PMLR, 40624–40636
2023
-
[2018]
In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security
Detecting violations of differential privacy. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security . 475–489
2018
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.