REVIEW 2 major objections 6 minor 1 cited by
That which we call private
T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper argues that 'relaxed' differential-privacy definitions are tighter worst-case analyses, not weaker guarantees: the reported epsilon can drop while actual privacy loss stays fixed.
desk verdict A short, basically sound correction to how people read Jayaraman-Evans, plus a genuinely tighter membership-inference bound; treat the 'region of privacy' graphs as illustrative, not rigorous. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by three pieces. The first is 'rotating the table': re-indexing an existing empirical comparison by training noise (equivalently, training loss) instead of by the target epsilon, which reveals that attack success tracks noise, not the choice of analysis definition. The second is a sharper upper bound on attack advantage: starting from a hypothesis-testing proposition in [10], the paper derives that for an $(\varepsilon,\delta)$-DP mechanism, membership-inference advantage is at most $1-e^{-\varepsilon}+\delta e^{-\varepsilon}$. The third is its inversion, which turns an observed attack advantage $\gamma$ into a lower bound $\varepsilon \ge \log\left(\frac{1-\delta}{1-\gamma}\right)$ on the true epsilon. Together these produce the 'region of privacy' between the tightest analysis upper bound and the strongest attack lower bound.
What would settle it
Take a dataset where every training image has its first pixel set to 0 and every test image has it set to 1, train a model that outputs a constant independent of its input (true epsilon 0), and run the membership-inference attack from [20] at a 5% false-positive rate. If the attack advantage is positive, the lower-bound formula yields a positive epsilon for a perfectly private mechanism, showing that distribution shift alone can produce the signal the method interprets as privacy loss.
Extended reading notes
Core claim
The paper's central claim is a distinction between the privacy loss a mechanism actually has and the upper-bound epsilon reported by any analysis technique. A fixed model trained with DP-SGD has one unknown actual privacy guarantee; re-analyzing it under RDP or zCDP cannot change that guarantee, only provide a tighter upper bound. The apparent paradox in an earlier membership-inference study is therefore an artifact of comparing models trained to match a target epsilon under different definitions: the looser the analysis, the more noise must be added to meet the target, and that extra noise is what lowers both utility and measured attack success. When the comparison is re-centered on the actual training noise, attack success is approximately unchanged across analysis definitions while the reported epsilon drops sharply. The paper also tightens the known bound relating membership-inference advantage to $(\varepsilon,\delta)$-DP and inverts it into a lower bound on the true epsilon, defining a 'region of privacy' between the best upper and lower bounds.
Load-bearing premise
The load-bearing premise is that the training and test distributions are indistinguishable, so a nonzero membership-inference advantage can only come from the model's use of training data; if the two distributions differ for any unrelated reason, even a perfectly private model can show attack advantage, and the derived epsilon lower bound becomes an artifact.
Editorial extensions
If this is right
- If the paper is right, the higher attack success rates reported under RDP/zCDP in the earlier study are not evidence that those definitions weaken privacy; they reflect the smaller noise needed to hit the same reported epsilon under a tighter analysis.
- Reported epsilon values from different analyses of the same model are not comparable as measures of actual privacy; the same model can legitimately carry a much smaller RDP epsilon than a naive-composition epsilon.
- Membership-inference attacks can be repurposed as lower-bound estimators: combined with tight upper bounds they bracket the true privacy loss into a 'region of privacy' that can be narrowed on both sides.
- Practitioners who want comparable privacy reporting should specify the analysis technique and, ideally, the training noise scale, not just the epsilon target.
- For Gaussian DP-SGD training, zCDP and RDP coincide, so any reported difference between those two accounts is a difference in how tightly sampling is accounted for, not a difference in the actual guarantee.
Reading between the lines
- Editorial inference: the same fixed-noise comparison could be run with other attack families, such as per-class threshold attacks, to test whether attack success stays flat across analysis definitions; the paper's prediction is that it should.
- Editorial inference: when applying the lower-bound method, one should first verify that training and test features are statistically close; otherwise the computed epsilon lower bound may measure dataset shift rather than privacy loss, exactly the artifact the paper warns about.
- Editorial inference: cross-paper comparisons of privacy would become more meaningful if every reported epsilon were accompanied by the analysis definition and noise multiplier, since any fixed model carries a whole family of valid epsilon upper bounds.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This note argues that the "relaxed definitions" of differential privacy (zCDP and RDP) are not weaker-adversary relaxations but tighter worst-case analyses, and that the apparent paradox in Jayaraman and Evans (2019) arises from comparing models trained to the same upper-bound epsilon under different analysis techniques. The paper reinterprets and reformats tables from Jayaraman and Evans, reproduces a subset of their experiments, derives a strengthened bound on membership-inference advantage (Proposition 2) that is translated into a lower bound on epsilon (Proposition 3), and plots "region of privacy" graphs that allegedly bracket the true epsilon between RDP upper bounds and attack-derived lower bounds. Section 5.4 then acknowledges two caveats: the attack advantage can be nonzero for a perfectly private model when train and test distributions differ, and choosing a threshold to maximize TPR-FPR would break the rigorous nature of the lower-bound analysis. The central conceptual claim is independent of these caveats, but the empirical lower-bound contribution depends on them.
Significance. If the main conceptual claim holds, it provides a valuable correction to a common misreading of Jayaraman and Evans: attack success at a given reported epsilon does not demonstrate that RDP/zCDP are weak, because the actual privacy loss of a mechanism is fixed and the epsilon value is an analysis-dependent upper bound. The derivation of Propositions 2 and 3 is clean, parameter-free, and anchored to existing theorems, and the paper is unusually honest about the limitations of attack-based epsilon estimation. However, the empirical demonstration of a "tighter connection" relies on the validity of the Proposition 3 lower bounds, and the paper's own Section 5.4 concessions call that validity into question. The conceptual contribution is likely sound, but the empirical contribution needs substantial revision.
major comments (2)
- [Section 5.3 and Section 5.4] Section 5.3 states that the lower bound is computed by "choosing a threshold that maximizes TPR-FPR," while Section 5.4 concedes that such threshold choice "would break the rigorous nature of the analysis of ε-lower bounds." These statements are mutually inconsistent. Because Figures 1 and 2 plot exactly the lower-bound curves from this optimized-threshold procedure, the figures cannot be presented as rigorous lower bounds under the paper's own caveat. The authors should either use a fixed, pre-specified threshold (such as average loss) or explicitly label the curves as heuristic estimates, and remove the claim that the region between curves brackets the actual privacy loss.
- [Section 5.4 and Table 2] The lower bounds in Table 2 and Figures 1-2 are computed by applying Proposition 3 to a measured membership-inference advantage γ. As Section 5.4 concedes, a perfectly private model can exhibit γ > 0 if the training and test distributions are distinguishable, in which case Proposition 3 returns a positive lower bound on ε that is an artifact of distribution shift rather than a privacy loss. The paper does not provide any evidence that the Purchase-100 train/test split used in the experiments is distributionally indistinguishable, and the reformatted tables in Sections 3 and 4 rely on the same assumption. The authors should either test for and control this confound, or explicitly reframe Table 2 and Figures 1-2 as valid only under that assumption, which would substantially weaken the "region of privacy" claim.
minor comments (6)
- [Section 2] The phrase "viana¨ıve[6]" contains a typo and should read "via naïve [6]."
- [Section 4] The statement that "zCDP and RDP definitions are identical" is imprecise; zCDP and RDP are different privacy definitions, even though for Gaussian mechanisms they can be related by reparameterization. The likely intended remark is that the specific zCDP analysis in Jayaraman and Evans is weaker because it ignores subsampling amplification, whereas the RDP analysis includes it. Please rephrase to avoid misleading readers about the definitions themselves.
- [Table 1] Attack TPR values are reported as point estimates without confidence intervals. Several adjacent values (e.g., 5.0%, 5.3%, 5.6%) are very close, and without variance information it is difficult to assess the claim that empirical privacy loss is unchanged across analysis types. Reporting standard errors or per-trial results would strengthen the comparison.
- [Section 5.3] The sentence "The region between the two curves constitute the 'region of privacy'" has a subject-verb agreement error: "constitute" should be "constitutes."
- [Section 5.4] The phrase "These tests shed a more accurate light on the privacy of the model" is awkward; consider "shed more accurate light" or "shed light more accurately."
- [References] Reference [11] is cited as the arXiv v3 preprint; the final USENIX Security 2019 version should be cited if available.
Circularity Check
No significant circularity: the paper's bounds and re-interpretation are self-contained and anchored to external theorems.
full rationale
The derivation chain in Section 5 uses Proposition 1 from Hall et al. (an external result) to prove Proposition 2, an upper bound on membership-inference advantage for any (ε,δ)-DP mechanism, and Proposition 3 rearranges that bound to obtain ε ≥ log((1-δ)/(1-γ)) from measured advantage γ. This is a logical implication, not a construction that fits or defines the predicted quantity; the lower bound is parameter-free and does not use Jayaraman and Evans' numbers as inputs. The re-interpretation of Table 5 in Section 3 does not derive new predictions from fitted parameters; it re-orients existing measurements and observes that for fixed noise/accuracy, attack success is approximately unchanged, which is an interpretation consistent with the definition of actual privacy loss, not a circular derivation. The only self-referential element is that author Ilya Mironov is the inventor of RDP, and the paper defends RDP/zCDP; however, the defense rests on the mathematical bound and reproduced experiments, not on an unverified self-citation. Section 5.4 explicitly acknowledges limitations (train/test distribution shift and threshold choice) that could invalidate lower bounds in practice; that is a correctness caveat, not an input-to-output circularity. No step reduces by construction to its own inputs.
Assumptions & free parameters
assumptions (5)
- standard math Proposition 1 from Hall et al. [10, Proposition 4]: any hypothesis test against an (epsilon, delta)-DP mechanism with FPR alpha has TPR bounded by e^{epsilon} * alpha + delta.
- domain assumption Membership inference advantage is defined as TPR - FPR as in Yeom et al. [20].
- domain assumption Training and test distributions are assumed indistinguishable in the ideal case.
- domain assumption In DP-SGD with Gaussian noise and Poisson sampling, zCDP and RDP analyses are equivalent.
- domain assumption The selected values from Jayaraman and Evans [11] Table 5 are accurate and representative.
Cite this review
Pith. "Pith review of That which we call private." pith.science (2026). https://pith.science/paper/5NBN3HVX
@misc{pith2026190803566,
author = {Pith},
title = {Pith review of: That which we call private},
year = {2026},
howpublished = {\url{https://pith.science/paper/5NBN3HVX}},
note = {Machine review of arXiv:1908.03566}
}
read the original abstract
The guarantees of security and privacy defenses are often strengthened by relaxing the assumptions made about attackers or the context in which defenses are deployed. Such relaxations can be a highly worthwhile topic of exploration---even though they typically entail assuming a weaker, less powerful adversary---because there may indeed be great variability in both attackers' powers and their context. However, no weakening or contextual discounting of attackers' power is assumed for what some have called "relaxed definitions" in the analysis of differential-privacy guarantees. Instead, the definitions so named are the basis of refinements and more advanced analyses of the worst-case implications of attackers---without any change assumed in attackers' powers. Because they more precisely bound the worst-case privacy loss, these improved analyses can greatly strengthen the differential-privacy upper-bound guarantees---sometimes lowering the differential-privacy epsilon by orders-of-magnitude. As such, to the casual eye, these analyses may appear to imply a reduced privacy loss. This is a false perception: the privacy loss of any concrete mechanism cannot change with the choice of a worst-case-loss upper-bound analysis technique. Practitioners must be careful not to equate real-world privacy with differential-privacy epsilon values, at least not without full consideration of the context.
Figures
Forward citations
Cited by 1 Pith paper
-
Membership Inference Attacks and Defenses in Federated Learning: A Survey
A comprehensive survey of membership inference attacks and defenses in federated learning, proposing update-based and trend-based attack taxonomies and four defense categories.
Reference graph
Works this paper leans on
-
[1]
Mart ´ın Abadi, Andy Chu, Ian J. Goodfellow, H. Bren- dan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Computer and Communications Security Conference (CCS), pages 308–318, 2016
work page 2016
-
[2]
Learning with pri- vacy at scale
Apple’s Differential Privacy Team. Learning with pri- vacy at scale. Apple Machine Learning Journal , 1(9), December 2017
work page 2017
-
[3]
Concentrated differ- ential privacy: Simplifications, extensions, and lower bounds
Mark Bun and Thomas Steinke. Concentrated differ- ential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography Conference (TCC), Part I, pages 635–658, 2016. 4
work page 2016
-
[4]
The secret sharer: Measuring unintended neural network memorization & extracting secrets
Nicholas Carlini, Chang Liu, Jernej Kos, ´Ulfar Erlings- son, and Dawn Song. The secret sharer: Measuring unintended neural network memorization & extracting secrets. To appear at the 28th USENIX Security Sympo- sium, 2019
work page 2019
-
[5]
Collecting telemetry data privately
Bolin Ding, Janardhan Kulkarni, and Sergey Yekhanin. Collecting telemetry data privately. InAdvances in Neu- ral Information Processing Systems (NeurIPS) , pages 3574–3583, 2017
work page 2017
-
[6]
Our data, ourselves: Privacy via distributed noise generation
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSh- erry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology—EUROCRYPT , pages 486–503, 2006
work page 2006
-
[7]
Calibrating noise to sensitivity in private data analysis
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference (TCC), pages 265–284, 2006
work page 2006
-
[8]
Cynthia Dwork, Guy N. Rothblum, and Salil Vadhan. Boosting and differential privacy. In STOC 2010, pages 51–60, 2010
work page 2010
Show all 20 references
-
[9]
RAPPOR: Randomized aggregatable privacy- preserving ordinal response
´Ulfar Erlingsson, Vasyl Pihur, and Aleksandra Ko- rolova. RAPPOR: Randomized aggregatable privacy- preserving ordinal response. In Computer and Commu- nications Security (CCS), pages 1054–1067, 2014
2014
-
[10]
Differential privacy for functions and functional data
Rob Hall, Alessandro Rinaldo, and Larry Wasserman. Differential privacy for functions and functional data. Journal of Machine Learning Research , 14(Feb):703– 727, 2013
2013
-
[11]
Evaluat- ing differentially private machine learning in practice
Bargav Jayaraman and David Evans. Evaluat- ing differentially private machine learning in practice. In proceedings of the 28th USENIX Security Symposium , 2019. (Retrieved from https://arxiv.org/abs/1902.08874v3)
2019 arXiv
-
[12]
R ´enyi differential privacy
Ilya Mironov. R ´enyi differential privacy. In Proceed- ings of 30th IEEE Computer Security Foundations Sym- posium (CSF), pages 263–275, 2017
2017
-
[13]
Secureml: A sys- tem for scalable privacy-preserving machine learning
Payman Mohassel and Yupeng Zhang. Secureml: A sys- tem for scalable privacy-preserving machine learning. In 2017 IEEE Symposium on Security and Privacy (SP), pages 19–38, 2017
2017
-
[14]
Semi-supervised knowl- edge transfer for deep learning from private training data
Nicolas Papernot, Mart ´ın Abadi, ´Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowl- edge transfer for deep learning from private training data. In International Conference on Learning Repre- sentations, 2017
2017
-
[15]
In- troducing tensorflow privacy: Learning with differential privacy for training data
Carey Radebaugh and ´Ulfar Erlingsson. In- troducing tensorflow privacy: Learning with differential privacy for training data. URL: https://medium.com/tensorflow/introducing- tensorflow-privacy-learning-with-differential-privacy- for-training-data-b143c5e801b6, 2019
2019
-
[16]
Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00451, 2018
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00451, 2018
2018 arXiv
-
[17]
Membership inference attacks against machine learning models
Reza Shokri, Marco Stronati, Congzheng Song, and Vi- taly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3–18, 2017
2017
-
[18]
Machine learning models that remember too much
Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remember too much. In Conference on Computer and Commu- nications Security (CCS), pages 587–601, 2017
2017
-
[19]
Cold case: The lost mnist digits
Chhavi Yadav and L ´eon Bottou. Cold case: The lost mnist digits. arXiv preprint arXiv:1905.10498, 2019
1905 arXiv
-
[20]
Privacy risk in machine learning: An- alyzing the connection to overfitting
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: An- alyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pages 268–282, July 2018. 5
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.