{"id":"3dcd1437-22bc-4c6f-aeff-43da1439a5b4","arxiv_id":"1908.03566","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Newer 'relaxed' differential privacy analyses yield tighter upper-bound guarantees without changing actual privacy loss, and attack success can be translated into a lower bound on that loss.","lead":"This paper argues that newer, so-called relaxed differential privacy analyses are actually tighter worst-case bounds, not weaker definitions, and that a model's real privacy loss does not change when you switch analysis methods. It also derives a formula that turns membership-inference attack success into a lower bound on privacy loss, helping bracket a model's true privacy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prop. 3's attack-derived ε lower bound is invalid under the §5.4 concessions on train/test distribution shift and test-calibrated thresholds, undercutting the paper's empirical bracket of actual privacy.","rationale":"The paper's central conceptual claim is sound: the actual privacy loss of a fixed mechanism is independent of the analysis technique used to bound it, and RDP/zCDP are refinements rather than weaker-adversary definitions. The derivation of Proposition 2 is algebraically correct, and the RDP upper bounds are valid. However, the paper's new lower-bound contribution (Prop. 3 and the 'region of privacy' in §5.3) relies on measuring a membership-inference advantage γ that can be inflated by two factors the paper itself flags in §5.4: train/test distribution shift and threshold choice (e.g., calibrating FPR on test data). Since Proposition 3 is monotone in γ, an inflated γ yields an inflated ε lower bound, so the bracket may not contain the true privacy loss. This does not refute the abstract's conceptual point, but it means the paper's empirical claim to have bracketed actual privacy is unsupported without additional controls. The reader's weakest assumption correctly identifies the distribution-shift part; the threshold-calibration issue is an additional, more controllable failure mode. Because the reader already returns CONDITIONAL for these reasons, no verdict change is needed.","tokens_in":5989,"tokens_out":16875,"duration_ms":178915,"concrete_test":"Recompute Table 2's ε-lower bounds from Proposition 3 using an attack threshold fixed a priori from the training set (e.g., the average loss of the non-private baseline) and without calibrating the 5% FPR on test labels; if the inferred lower bounds drop materially from Table 2's values, the published lower bounds are artifacts of test-set threshold selection rather than rigorous privacy lower bounds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weak point is the attack-to-epsilon lower-bound translation (Prop. 3), not the abstract's conceptual point about upper bounds. Section 5.4 concedes that the Yeom et al. attack advantage can be non-zero for a perfectly private model if train/test distributions differ, and that choosing the threshold to maximize TPR−FPR (or calibrating FPR on test data) 'would break the rigorous nature of the analysis of ε-lower bounds.' The paper nevertheless presents Table 2 and Figures 1–2 as bracketing the actual ε between RDP upper bounds and these attack-derived lower bounds. If the measured γ is inflated by distribution shift or by test-set-calibrated thresholds, Proposition 3 returns an ε lower bound that is higher than the true privacy loss; the 'region of privacy' is then an artifact of the evaluation protocol rather than a property of the mechanism. This does not refute the claim that RDP/zCDP are tighter analyses of the same worst-case privacy loss, but it does undercut the paper's empirical demonstration of a 'tighter connection between privacy leakage and DP bounds.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This note argues that the \"relaxed definitions\" of differential privacy (zCDP and RDP) are not weaker-adversary relaxations but tighter worst-case analyses, and that the apparent paradox in Jayaraman and Evans (2019) arises from comparing models trained to the same upper-bound epsilon under different analysis techniques. The paper reinterprets and reformats tables from Jayaraman and Evans, reproduces a subset of their experiments, derives a strengthened bound on membership-inference advantage (Proposition 2) that is translated into a lower bound on epsilon (Proposition 3), and plots \"region of privacy\" graphs that allegedly bracket the true epsilon between RDP upper bounds and attack-derived lower bounds. Section 5.4 then acknowledges two caveats: the attack advantage can be nonzero for a perfectly private model when train and test distributions differ, and choosing a threshold to maximize TPR-FPR would break the rigorous nature of the lower-bound analysis. The central conceptual claim is independent of these caveats, but the empirical lower-bound contribution depends on them.","tokens_in":6158,"tokens_out":7252,"duration_ms":76854,"significance":"If the main conceptual claim holds, it provides a valuable correction to a common misreading of Jayaraman and Evans: attack success at a given reported epsilon does not demonstrate that RDP/zCDP are weak, because the actual privacy loss of a mechanism is fixed and the epsilon value is an analysis-dependent upper bound. The derivation of Propositions 2 and 3 is clean, parameter-free, and anchored to existing theorems, and the paper is unusually honest about the limitations of attack-based epsilon estimation. However, the empirical demonstration of a \"tighter connection\" relies on the validity of the Proposition 3 lower bounds, and the paper's own Section 5.4 concessions call that validity into question. The conceptual contribution is likely sound, but the empirical contribution needs substantial revision.","major_comments":[{"comment":"Section 5.3 states that the lower bound is computed by \"choosing a threshold that maximizes TPR-FPR,\" while Section 5.4 concedes that such threshold choice \"would break the rigorous nature of the analysis of ε-lower bounds.\" These statements are mutually inconsistent. Because Figures 1 and 2 plot exactly the lower-bound curves from this optimized-threshold procedure, the figures cannot be presented as rigorous lower bounds under the paper's own caveat. The authors should either use a fixed, pre-specified threshold (such as average loss) or explicitly label the curves as heuristic estimates, and remove the claim that the region between curves brackets the actual privacy loss.","section":"Section 5.3 and Section 5.4"},{"comment":"The lower bounds in Table 2 and Figures 1-2 are computed by applying Proposition 3 to a measured membership-inference advantage γ. As Section 5.4 concedes, a perfectly private model can exhibit γ > 0 if the training and test distributions are distinguishable, in which case Proposition 3 returns a positive lower bound on ε that is an artifact of distribution shift rather than a privacy loss. The paper does not provide any evidence that the Purchase-100 train/test split used in the experiments is distributionally indistinguishable, and the reformatted tables in Sections 3 and 4 rely on the same assumption. The authors should either test for and control this confound, or explicitly reframe Table 2 and Figures 1-2 as valid only under that assumption, which would substantially weaken the \"region of privacy\" claim.","section":"Section 5.4 and Table 2"}],"minor_comments":[{"comment":"The phrase \"viana¨ıve[6]\" contains a typo and should read \"via naïve [6].\"","section":"Section 2"},{"comment":"The statement that \"zCDP and RDP definitions are identical\" is imprecise; zCDP and RDP are different privacy definitions, even though for Gaussian mechanisms they can be related by reparameterization. The likely intended remark is that the specific zCDP analysis in Jayaraman and Evans is weaker because it ignores subsampling amplification, whereas the RDP analysis includes it. Please rephrase to avoid misleading readers about the definitions themselves.","section":"Section 4"},{"comment":"Attack TPR values are reported as point estimates without confidence intervals. Several adjacent values (e.g., 5.0%, 5.3%, 5.6%) are very close, and without variance information it is difficult to assess the claim that empirical privacy loss is unchanged across analysis types. Reporting standard errors or per-trial results would strengthen the comparison.","section":"Table 1"},{"comment":"The sentence \"The region between the two curves constitute the 'region of privacy'\" has a subject-verb agreement error: \"constitute\" should be \"constitutes.\"","section":"Section 5.3"},{"comment":"The phrase \"These tests shed a more accurate light on the privacy of the model\" is awkward; consider \"shed more accurate light\" or \"shed light more accurately.\"","section":"Section 5.4"},{"comment":"Reference [11] is cited as the arXiv v3 preprint; the final USENIX Security 2019 version should be cited if available.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short note whose main value is conceptual clarification. The analytical result (Propositions 2 and 3) is correct and cleanly presented. However, the empirical section is currently overclaimed relative to the paper's own caveats, and the internal contradiction between Section 5.3 and Section 5.4 must be resolved. The paper's position as a response to Jayaraman and Evans is fine, but the authors should ensure the tone remains a technical correction rather than a dismissal. For a full journal publication, the empirical evaluation would need to be substantially strengthened; the current form may be better suited to a workshop or short-paper venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's main claim is right: RDP and zCDP are not weaker-attacker definitions, they are tighter worst-case analyses, and reanalyzing a fixed mechanism should not change the actual privacy loss. That directly fixes a common misreading of Jayaraman-Evans. The rotated-table reinterpretation is the clearest part of the paper and it lands.\n\nWhat is new is mostly Proposition 2: the membership-inference advantage bound of 1 - e^{-ε} + δ e^{-ε}, which is genuinely tighter than the Yeom et al. bound. Proposition 3 follows by rearrangement and gives an ε lower bound. The derivations are short, correct, and anchored to Hall et al. No free parameters, no fitted values. Credit where it is due. The citation pattern is appropriate, and the authors are honest about some limitations in Section 5.4.\n\nNow the soft spots. The attack-to-ε lower-bound translation is only rigorous if the TPR-FPR advantage is measured under the same conditions as the DP hypothesis test. The authors concede in Section 5.4 that train/test distribution shift can produce large attack advantage even for a perfectly private model, and that threshold selection 'would break the rigorous nature' of the lower-bound analysis. That is not a footnote; it is the load-bearing assumption behind Table 2 and Figures 1-2. In their actual Purchase-100 experiment, the random split makes the distribution-shift concern secondary, but the 'region of privacy' framing overstates what the lower bound is. The lower bound is protocol-dependent, not a property of the mechanism. They should either soften that framing or derive bounds under explicitly controlled train/test distributions. The empirical section is also thin: point estimates without confidence intervals, no code, and selective reuse of the original table. Minor, but worth fixing.\n\nThe central conceptual point does not depend on the empirical bracket, so these weaknesses do not sink the paper. This is a useful note for practitioners and for anyone who cites Jayaraman-Evans as evidence that RDP/zCDP are weak. It deserves a serious referee, with revisions aimed at the Section 5.4 assumptions and the overclaimed region of privacy.\n\nI would bring it to reading group and would cite Proposition 2. Recommend engage.","headline":"A short, basically sound correction to how people read Jayaraman-Evans, plus a genuinely tighter membership-inference bound; treat the 'region of privacy' graphs as illustrative, not rigorous.","tokens_in":6741,"tokens_out":6469,"would_cite":true,"duration_ms":72010,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68P27","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that 'relaxed' differential-privacy definitions are tighter worst-case analyses, not weaker guarantees: the reported epsilon can drop while actual privacy loss stays fixed.","keywords":["differential privacy","Rényi differential privacy","concentrated differential privacy","membership inference attacks","epsilon lower bounds","DP-SGD","privacy loss accounting","worst-case analysis"],"falsifier":"Take a dataset where every training image has its first pixel set to 0 and every test image has it set to 1, train a model that outputs a constant independent of its input (true epsilon 0), and run the membership-inference attack from [20] at a 5% false-positive rate. If the attack advantage is positive, the lower-bound formula yields a positive epsilon for a perfectly private mechanism, showing that distribution shift alone can produce the signal the method interprets as privacy loss.","tokens_in":5771,"feed_emoji":"🔒","tokens_out":6418,"duration_ms":70128,"temperature":0.7,"pith_summary":"Across differentially private machine learning, the same trained model can be assigned very different reported epsilon values depending on which formal analysis is used. This paper argues that the definitions sometimes called 'relaxed'—Rényi differential privacy and concentrated differential privacy—are not weaker, less-powerful-adversary guarantees. They are sharper worst-case analyses of the same guarantee, so they can lower the reported upper bound by orders of magnitude without changing the mechanism's actual privacy loss. The paper shows that an empirical study's apparent finding of higher attack success under those definitions disappears once models are compared at the same training noise rather than at the same target epsilon. It then derives attack-based lower bounds on epsilon, so the true privacy loss sits between the tightest upper bound and the strongest attack lower bound.","feed_headline":"Lower epsilon can mean tighter analysis, not weaker privacy","feed_subtitle":"Attack success tracks training noise, not the DP analysis used; true privacy sits between upper and lower bounds.","key_machinery":"The argument is carried by three pieces. The first is 'rotating the table': re-indexing an existing empirical comparison by training noise (equivalently, training loss) instead of by the target epsilon, which reveals that attack success tracks noise, not the choice of analysis definition. The second is a sharper upper bound on attack advantage: starting from a hypothesis-testing proposition in [10], the paper derives that for an $(\\varepsilon,\\delta)$-DP mechanism, membership-inference advantage is at most $1-e^{-\\varepsilon}+\\delta e^{-\\varepsilon}$. The third is its inversion, which turns an observed attack advantage $\\gamma$ into a lower bound $\\varepsilon \\ge \\log\\left(\\frac{1-\\delta}{1-\\gamma}\\right)$ on the true epsilon. Together these produce the 'region of privacy' between the tightest analysis upper bound and the strongest attack lower bound.","core_discovery":"The paper's central claim is a distinction between the privacy loss a mechanism actually has and the upper-bound epsilon reported by any analysis technique. A fixed model trained with DP-SGD has one unknown actual privacy guarantee; re-analyzing it under RDP or zCDP cannot change that guarantee, only provide a tighter upper bound. The apparent paradox in an earlier membership-inference study is therefore an artifact of comparing models trained to match a target epsilon under different definitions: the looser the analysis, the more noise must be added to meet the target, and that extra noise is what lowers both utility and measured attack success. When the comparison is re-centered on the actual training noise, attack success is approximately unchanged across analysis definitions while the reported epsilon drops sharply. The paper also tightens the known bound relating membership-inference advantage to $(\\varepsilon,\\delta)$-DP and inverts it into a lower bound on the true epsilon, defining a 'region of privacy' between the best upper and lower bounds.","pith_inferences":["Editorial inference: the same fixed-noise comparison could be run with other attack families, such as per-class threshold attacks, to test whether attack success stays flat across analysis definitions; the paper's prediction is that it should.","Editorial inference: when applying the lower-bound method, one should first verify that training and test features are statistically close; otherwise the computed epsilon lower bound may measure dataset shift rather than privacy loss, exactly the artifact the paper warns about.","Editorial inference: cross-paper comparisons of privacy would become more meaningful if every reported epsilon were accompanied by the analysis definition and noise multiplier, since any fixed model carries a whole family of valid epsilon upper bounds."],"forward_implications":["If the paper is right, the higher attack success rates reported under RDP/zCDP in the earlier study are not evidence that those definitions weaken privacy; they reflect the smaller noise needed to hit the same reported epsilon under a tighter analysis.","Reported epsilon values from different analyses of the same model are not comparable as measures of actual privacy; the same model can legitimately carry a much smaller RDP epsilon than a naive-composition epsilon.","Membership-inference attacks can be repurposed as lower-bound estimators: combined with tight upper bounds they bracket the true privacy loss into a 'region of privacy' that can be narrowed on both sides.","Practitioners who want comparable privacy reporting should specify the analysis technique and, ideally, the training noise scale, not just the epsilon target.","For Gaussian DP-SGD training, zCDP and RDP coincide, so any reported difference between those two accounts is a difference in how tightly sampling is accounted for, not a difference in the actual guarantee."],"supporting_citations":[{"why":"The empirical study the paper re-interprets; its fixed-epsilon comparison is the apparent paradox the paper resolves by rotating the table.","marker":"[11]"},{"why":"Defines the membership-inference attack and the advantage notion the paper tightens and inverts into epsilon lower bounds.","marker":"[20]"},{"why":"Supplies the hypothesis-testing proposition from which the sharper attack-advantage upper bound is derived.","marker":"[10]"},{"why":"Supplies Rényi differential privacy, one of the refined analyses claimed to be a tighter upper bound rather than a weaker guarantee.","marker":"[12]"},{"why":"Supplies zero-Concentrated differential privacy, the other refined definition analyzed in the paper.","marker":"[3]"},{"why":"Describes DP-SGD with clipping and Gaussian noise, the mechanism class whose epsilon bounds and attacks the paper studies.","marker":"[1]"}],"fun_headline_variants":["Lower epsilon from tighter analysis isn't weaker privacy","Better DP bounds lower epsilon without changing actual privacy","Tighter DP analysis slashes epsilon, not the real privacy loss","True privacy lies between tight upper and lower bounds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the training and test distributions are indistinguishable, so a nonzero membership-inference advantage can only come from the model's use of training data; if the two distributions differ for any unrelated reason, even a perfectly private model can show attack advantage, and the derived epsilon lower bound becomes an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Lower epsilon from tighter analysis isn't weaker privacy","Better DP bounds lower epsilon without changing actual privacy","Tighter DP analysis slashes epsilon, not the real privacy loss","True privacy lies between tight upper and lower bounds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001083,"raw_usage":{"total_tokens":4531,"prompt_tokens":951,"completion_tokens":3580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":3517}},"tokens_in":567,"tokens_out":3580,"duration_ms":26275,"temperature":1.0,"reasoning_tokens":3517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:19:10.554941+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset where every training image has its first pixel set to 0 and every test image has it set to 1, train a model that outputs a constant independent of its input (true epsilon 0), and run the membership-inference attack from [20] at a 5% false-positive rate. If the attack advantage is positive, the lower-bound formula yields a positive epsilon for a perfectly private mechanism, showing that distribution shift alone can produce the signal the method interprets as privacy loss.","supporting_citations":[{"cited_title":"Privacy risk in machine learning: An- alyzing the connection to overfitting","cited_arxiv_id":null,"evidence_quote":"Defines the membership-inference attack and the advantage notion the paper tightens and inverts into epsilon lower bounds."},{"cited_title":"Differential privacy for functions and functional data","cited_arxiv_id":null,"evidence_quote":"Supplies the hypothesis-testing proposition from which the sharper attack-advantage upper bound is derived."},{"cited_title":"R ´enyi differential privacy","cited_arxiv_id":null,"evidence_quote":"Supplies Rényi differential privacy, one of the refined analyses claimed to be a tighter upper bound rather than a weaker guarantee."},{"cited_title":"Concentrated differ- ential privacy: Simplifications, extensions, and lower bounds","cited_arxiv_id":null,"evidence_quote":"Supplies zero-Concentrated differential privacy, the other refined definition analyzed in the paper."},{"cited_title":"Goodfellow, H","cited_arxiv_id":null,"evidence_quote":"Describes DP-SGD with clipping and Gaussian noise, the mechanism class whose epsilon bounds and attacks the paper studies."}],"review_version":1}