{"id":"656f0902-9b1b-453d-b99c-27a6bb67b52b","arxiv_id":"2508.07066","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A post-hoc wrapper, MIAFdR, converts any membership inference attack scores into conformal p-values and applies a Benjamini-Hochberg correction, guaranteeing that the expected proportion of non-members among flagged members is at most the significance level.","lead":"This paper adds a statistical error-control layer on top of existing membership inference attacks, bounding the expected fraction of non-members wrongly flagged as members. It shows the wrapper preserves or slightly improves attack accuracy across several attack families and image datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FDR guarantee hinges on exchangeability between victim-model test scores and surrogate-based calibration scores, which the assumption in Theorem 1 (exchangeable x's) does not deliver.","rationale":"The reader's weakest_assumption and my read coincide: the load-bearing step is the exchangeability of the victim-based test score with surrogate-based calibration scores. This is not a manufactured concern; the theorem statement's exchangeability assumption is about the raw inputs x_i, but the p-values are computed from a score transform that is different for the test point (uses θ*) and for calibration (uses θ~k). Therefore the proof of Theorem 1 cannot follow from the stated assumption alone, and Theorem 2 inherits the gap. The experimental FDR curves and Figure 2a provide some evidence that the violation is mild in the tested configurations, and the paper is honest that black-box relies on transferability, but the informal claim 'we can control FDR' is conditional on an unproven distributional assumption. A CONDITIONAL verdict with a required proof/caveat is appropriate; since the reader already reached that verdict, no change is needed.","tokens_in":17845,"tokens_out":15254,"duration_ms":161595,"concrete_test":"Reproduce the grey-box setup with identical architecture, optimizer, and equal training-set size for victim and all K surrogates. (1) On a holdout of true non-members, compare test scores S(f(xt;θ*);θbc) with calibration scores S(f(xj;θ~k);θbc) via two-sample KS; a large statistic contradicts the exchangeability needed by Theorem 1. (2) Run full MIAFdR at α=0.05,0.1,0.2 over many repetitions; if empirical FDR exceeds α·π0 beyond sampling error, the concern lands. (3) Positive control: generate test scores from a randomly selected surrogate instead of the victim; if FDR control holds there but fails with victim scores, victim/surrogate mismatch is the cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the FDR bound in Theorem 2 (Eq. 13), built on the p-values of Eq. 8. For Eq. 8 to give valid conformal p-values under H0, the test score S(f(xt; θ*); θbc) must be exchangeable with the calibration scores S(f(xj; θ~k); θbc). Theorem 1 instead assumes only that the raw sequence G=(x1,...,xm,xt) is exchangeable. That does not imply score exchangeability here, because test scores come from the victim model θ* while calibration scores come from surrogate models θ~k (Section 4, Eqs. 4-8), and θbc is itself fitted to surrogate predictions. Unless the victim and surrogates are statistically interchangeable (same architecture, same training-set size, same algorithm, and independence conditions that avoid the shared-surrogate dependence), the marginal guarantee of Theorem 1 and the FDR control of Theorem 2 do not follow. In the black-box experiments (Fig. 6a), architectures differ, so the assumption is violated by construction; in grey-box it is an unstated extra assumption. The empirical FDR plots suggest the violation may be mild, but the theorem as stated does not prove the advertised guarantee for the actual pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MIAFdR, a wrapper method for membership inference attacks that aims to provide false discovery rate (FDR) control. For each test point, a non-member relative probability is computed as a conformal p-value from scores produced by a binary classifier trained on surrogate-model predictions. The p-values are then adjusted via the Benjamini-Hochberg procedure (Eq. 10), and the paper claims a distribution-free bound E[FDP] ≤ α·π0 ≤ α (Theorem 2) together with a marginal guarantee on falsely labeling non-members as members (Theorem 1). Experiments on CIFAR-10/100 and Tiny-ImageNet with several base MIA methods report FDR curves below the analytical bound, plus applications to machine unlearning and lifelong learning.","tokens_in":18156,"tokens_out":5753,"duration_ms":60237,"significance":"If the stated guarantees hold as advertised, MIAFdR would be a useful and cheap post-hoc wrapper that turns existing membership inference attacks into procedures with FDR control, addressing a real gap in the MIA literature. The empirical FDR curves in the grey-box settings match the claimed α·π0 bound, and the additional runtime is negligible. However, the paper's contribution is primarily an application of conformal p-values plus BH adjustment; the theoretical novelty is limited, and the central guarantees are not actually proved in the manuscript. The main value is the framing and empirical demonstration, conditional on the theory being repaired.","major_comments":[{"comment":"The premise of Theorem 1 is insufficient for the claimed marginal guarantee in the actual pipeline. The theorem assumes exchangeability of the raw sequence G = (x1,...,xm,xt). But the p-value in Eq. (8) is computed from scores S(y;θbc), where the calibration scores use surrogate outputs f(xj;θ~k) and the test score uses victim outputs f(xt;θ*); θbc is itself fitted to surrogate predictions (Eqs. 4-6). Exchangeability of raw x's does not imply exchangeability of these scores. In the black-box experiments (Section 5, Fig. 6a) the surrogate and victim architectures differ, so score exchangeability is violated by construction. As written, Theorem 1 does not prove Eq. (9) for the proposed procedure; an explicit assumption of score exchangeability under H0, a bound on the violation, or a changed theorem is required. This gap also propagates to Theorem 2.","section":"Theorem 1 / Eq. (9)"},{"comment":"The FDR guarantee is asserted but its proof is deferred to the full version. The p-values are not independent: all use the same calibration set C2,ca_au and the same fitted θbc. BH control under dependence requires conditions such as PRDS; the manuscript cites [7] but does not state or verify that those conditions hold for these conformal p-values. Without this, the advertised 'distribution-free' FDR bound is not established. The empirical FDR curves in Figs. 3/5/7/8 are encouraging but do not substitute for the missing proof.","section":"Theorem 2 / Eq. (13)"},{"comment":"Both central theorems (Theorem 1 and Theorem 2) have proofs deferred to 'the full version of the paper.' Since the paper's claimed contribution is the theoretical guarantee, this is not merely a presentation issue: the claims cannot be checked from the manuscript. The authors should include complete proofs or detailed proof sketches, including the exact exchangeability and dependence assumptions needed at each step.","section":"Section 4, proofs"},{"comment":"The black-box experiments (Fig. 6a) report only attack accuracy, not FDR. This is the setting where the score-exchangeability premise is least plausible, because the surrogate and victim architectures differ. Since the central claim is FDR control, an empirical FDR check in this setting is needed; attack accuracy alone does not validate the guarantee.","section":"Section 5, black-box setting"}],"minor_comments":[{"comment":"The notation 'Sk ∈ C2,ca ∪ {S(yt;θbc)}' is ambiguous. The numerator should explicitly define whether it counts only calibration scores or also the test score; this affects the range and interpretation of the conformal p-value.","section":"Eq. (8)"},{"comment":"The union defining Y2,ca_au is written over |D2,tr|, which appears to be a typo; it should be over |D2,ca|.","section":"Eq. (4)"},{"comment":"The text says 'From D2, we then sample a fraction η = 3/7 to obtain {D1,k}', but earlier D1,k is constructed from D1. This is inconsistent and should be corrected.","section":"Section 5.1"},{"comment":"The overview figure is too compressed to read; labels such as 'Calibration Score Cau^2,ca' are unclear. A larger, cleaner diagram would help.","section":"Figure 1"},{"comment":"The statement conditions on the event xt ∉ Dtr, but Dtr is fixed. Please clarify the probability space and what is random in this statement.","section":"Theorem 1"}],"recommendation":"major_revision","confidential_remarks":"The core of the paper is the observation that conformal p-values plus BH can be used for FDR control in MIAs. This is a reasonable and potentially useful application, but in its current form the theoretical guarantee does not actually cover the implemented pipeline because the theorem assumes exchangeability of raw data while the procedure uses scores from different models. The missing proofs and the unverified dependence conditions for BH are also serious. I do not see an irreparable flaw; the authors could add correct assumptions, proofs, and black-box FDR experiments. For this reason I recommend major revision rather than rejection. I would also ask the editor to ensure the authors provide all proofs in the revised manuscript rather than referring to a 'full version'."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper does something genuinely useful—it wraps existing membership inference attacks so the output carries an FDR guarantee. The wrapper is conformal p-values plus Benjamini-Hochberg, and the authors don't overclaim novelty on the math; the new bit is applying that machinery to MIA, and that's a real contribution. The experiments are solid for a preprint: several datasets, multiple attack families (classifier, metric, LiRA), grey-box and black-box settings, plus machine unlearning and lifelong learning demos. The empirical FDR tracks the analytical bound α·π0 well.\n\nThe main soft spot is the theory. Theorem 1 assumes the sequence of inputs (calibration plus test) is exchangeable, but the p-value in Eq. 8 is computed from conformity scores, and those scores come from different models: the victim model for the test point, surrogate models for calibration. Exchangeability of inputs doesn't imply exchangeability of scores unless the models are statistically interchangeable. That extra assumption is never stated. In the black-box experiments, the surrogate architecture differs from the victim, so the theorem's premise is violated by construction. The empirical results suggest the violation is mild, but as stated the theorem doesn't prove the advertised guarantee for the actual pipeline. The proofs are deferred to the full version, which makes it hard to check whether that gap is addressed.\n\nThe BH adjustment itself is standard, so Theorem 2 is probably fine if the p-values are valid. The free parameters (λ, η, K) don't enter the bound, so there's no circularity. The self-citations are background.\n\nBottom line: worth publishing after revision. I'd send it to referees and ask them to require a rewrite of the theoretical assumption—either prove score-exchangeability under explicit conditions or weaken the claim and discuss when it fails. For people working on MIA or privacy auditing, the wrapper is worth knowing. I'd cite it if I wrote in that space.","headline":"A useful wrapper that gives MIAs FDR control by bolting conformal p-values and BH onto existing attacks, but the theory as stated assumes exchangeability on the wrong object and the proofs are deferred.","tokens_in":18648,"tokens_out":3086,"would_cite":true,"duration_ms":31102,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62G10","62G15","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Membership inference attacks can be wrapped in a post-hoc correction that caps the false discovery rate—the share of flagged members that are actually non-members—at any preset level, with negligible extra cost.","keywords":["membership inference attack","false discovery rate","conformal inference","conformal p-values","Benjamini-Hochberg","privacy auditing","machine unlearning","lifelong learning"],"falsifier":"In the black-box setting with a ResNet-18 victim and a MobileNetV2 surrogate on CIFAR-10, collect a set of known non-members and count the fraction with non-member probability below α=0.05; if that fraction clearly exceeds 0.05, the marginal guarantee of Theorem 1 is false. Similarly, with the true non-member proportion π0 known, run the full procedure at α=0.1 over ten random splits; empirical FDR values above α·π0 would falsify Theorem 2.","tokens_in":17740,"feed_emoji":"🛡️","tokens_out":8471,"duration_ms":78442,"temperature":0.7,"pith_summary":"Membership inference attacks try to decide whether a given record was part of a model's training data, but existing attacks give no statistical control over how many of the records they flag as 'members' are actually non-members. This paper proposes MIAFdR, a wrapper that attaches a non-member probability to every test point using a conformal score, then applies a Benjamini-Hochberg-style correction to those probabilities. It proves two guarantees: a true non-member is labeled a member with probability at most the significance level α, and the false discovery rate (the expected fraction of false member labels among all member labels) is bounded by α times the true non-member proportion, and hence by α itself. Because the wrapper is post-hoc, any score-producing membership inference attack can be converted into one with these guarantees. The empirical results show the bound holds across datasets, attack families, and even under defenses such as knowledge distillation, with negligible added runtime.","feed_headline":"Membership attacks get a provable false-discovery cap","feed_subtitle":"A conformal-p-value correction bounds false 'member' labels at a preset level, with negligible added compute.","key_machinery":"The central object is a conformal p-value. For each test point, the paper computes a conformity score from a binary classifier's output and defines the non-member probability as the fraction of calibration non-member scores no larger than the test score; this is exactly the conformal p-value, so exchangeability gives it its marginal validity. The second component is the adjustment in Eqn. (10), the Benjamini-Hochberg step-up procedure applied to these p-values, which is what converts many marginally valid but dependent p-values into an FDR-controlled decision set. The wrapper structure is the third component: because only scores are needed, any existing MIA can be placed underneath.","core_discovery":"The central claim is Theorem 2: for a test set with $T$ points, after ranking the conformal non-member probabilities $p_{(1)} \\le \\cdots \\le p_{(T)}$ and computing adjusted values $p_{(t)}^{\\mathrm{adj}} = \\min\\{1, \\min_{m \\in \\{t,\\ldots,T\\}} (T/m)\\, p_{(m)}\\}$, declaring every point with $p_{(t)}^{\\mathrm{adj}} \\le \\alpha$ to be a member controls the false discovery rate in the sense $E[ |R(D_{ts}) \\cap H_0^*(D_{ts})| / \\max\\{1, |R(D_{ts})|\\} ] \\le \\alpha \\cdot H_0^*(D_{ts})/T \\le \\alpha$. The bound is distribution-free and holds even though the individual probabilities are interdependent, because the adjustment is the Benjamini-Hochberg correction. Theorem 1 provides the per-point ingredie","pith_inferences":["The formal exchangeability premise is strongest in the grey-box setting; the black-box results rely on transferability, so the Theorem 2 bound should be read as approximate there unless the surrogate ensemble is demonstrably close to the victim.","The bound $\\alpha \\cdot \\pi_0$ is tighter than $\\alpha$ when few test points are non-members, meaning the procedure is conservative in member-heavy test sets; an adversary auditing a mostly-member set gets a stronger guarantee for free.","A natural next test is to replace the single calibration split with cross-conformal or split-conformal variants to reduce the dependence among p-values and potentially sharpen the FDR bound further.","Because the wrapper only consumes scores, it could be grafted onto label-only or query-efficient MIA variants, provided their scores can be made exchangeable with a calibration set."],"forward_implications":["Any score-based membership inference attack can be turned into one with FDR control by plugging its scores into MIAFdR; the extra inference cost is tiny (about 0.01 seconds for 7,000 test points).","Attack accuracy and AUROC are preserved or slightly improved across datasets: e.g., on CIFAR-100 classifier-based accuracy rises from 76.8% to 78.2%.","The FDR bound remains valid under common defenses such as knowledge distillation, so privacy auditing can report calibrated error rates even when the model is hardened.","In black-box settings where the surrogate architecture differs from the victim, the method still keeps empirical FDR near the analytical line across ResNet-18, ResNet-50, VGG-16, and MobileNetV2 combinations.","The wrapper also applies to memorization-based tasks: machine unlearning verification and lifelong learning assessment can report a controlled proportion of samples wrongly labeled as 'not unlearned' or 'memorized'."],"supporting_citations":[{"why":"Supplies the conformal p-value construction that makes each test point's non-member probability valid under exchangeability.","marker":"[7]"},{"why":"Supplies the Benjamini-Hochberg step-up correction that the paper adapts to control FDR under interdependent p-values.","marker":"[9]"},{"why":"Provides the distribution-free conformal prediction background that underpins the marginal guarantee.","marker":"[4]"},{"why":"Shadow-training classifier MIA is the base attack the method is demonstrated on as a wrapper.","marker":"[61]"},{"why":"Likelihood Ratio Attack is the alternative base attack used to show the wrapper preserves attack precision.","marker":"[12]"}],"fun_headline_variants":["Capping false 'member' labels in inference attacks","Provable cap on false member flags","Membership attacks with FDR guarantees","Tame false discovery in membership inference","Guaranteed false-discovery cap for MIAs"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The guarantees rest on treating the victim model's score on a test point as exchangeable with surrogate models' calibration scores, which in the black-box setting with different architectures is a formal violation rather than a proven fact.","fun_headline_variants_meta":{"raw":{"variants":["Capping false 'member' labels in inference attacks","Provable cap on false member flags","Membership attacks with FDR guarantees","Tame false discovery in membership inference","Guaranteed false-discovery cap for MIAs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1166,"prompt_tokens":815,"completion_tokens":351,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":285}},"tokens_in":559,"tokens_out":351,"duration_ms":3949,"temperature":1.0,"reasoning_tokens":285,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:21:55.947999+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In the black-box setting with a ResNet-18 victim and a MobileNetV2 surrogate on CIFAR-10, collect a set of known non-members and count the fraction with non-member probability below α=0.05; if that fraction clearly exceeds 0.05, the marginal guarantee of Theorem 1 is false. Similarly, with the true non-member proportion π0 known, run the full procedure at α=0.1 over ten random splits; empirical FDR values above α·π0 would falsify Theorem 2.","supporting_citations":[{"cited_title":"Testing for outliers with conformal p-values","cited_arxiv_id":null,"evidence_quote":"Supplies the conformal p-value construction that makes each test point's non-member probability valid under exchangeability."},{"cited_title":"Controlling the false discovery rate: a practical and powerful approach to multiple testing","cited_arxiv_id":null,"evidence_quote":"Supplies the Benjamini-Hochberg step-up correction that the paper adapts to control FDR under interdependent p-values."},{"cited_title":"Membership inference attacks against machine learning models","cited_arxiv_id":null,"evidence_quote":"Shadow-training classifier MIA is the base attack the method is demonstrated on as a wrapper."},{"cited_title":"Membership inference attacks from first principles","cited_arxiv_id":null,"evidence_quote":"Likelihood Ratio Attack is the alternative base attack used to show the wrapper preserves attack precision."}],"review_version":1}