{"id":"e5fdc9be-8032-4bd9-8c9f-158c70b87ac7","arxiv_id":"2507.21412","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CMIA cascades conditional shadow training to exploit membership dependencies, and PMIA uses proxy data to approximate Bayesian membership odds, both substantially outperforming prior MIAs in low false-positive regimes.","lead":"This paper introduces two new membership inference attacks against machine learning classifiers: a cascading attack that uses early confident guesses to sharpen later guesses, and a proxy-based attack that substitutes similar examples for the queried one. Both aim to reveal, more reliably than prior attacks, whether a given data point was in a model's training set, especially when only very few false positives are allowed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CMIA's headline 5x gain is measured at ~6x the baseline shadow-training budget, with no confidence intervals; the 'substantial outperformance' claim is not yet budget-matched.","rationale":"The reader's weakest assumption (PMIA proxy transfer) is reasonable but not the most load-bearing for the overall paper. The proxy concern is partially mitigated by the paper's own distribution-shift experiment and does not speak to CMIA. The compute mismatch, by contrast, affects the single quantitative result highlighted in the abstract and introduction, and is acknowledged by the paper's own efficiency section. The paper does provide an auxiliary matched-budget analysis (Figure 4) suggesting the framework can beat LiRA at equal wall-clock time, which is why this does not warrant rejection; it does mean the main tables overstate the algorithmic advantage. Adding budget-matched tables and bootstrap intervals would settle whether the headline claim holds.","tokens_in":34029,"tokens_out":8371,"duration_ms":98227,"concrete_test":"Rerun the adaptive comparison on MNIST and CIFAR-10 under a hard total-shadow-model budget (e.g., 256) for both LiRA and CMIA, choosing CMIA's per-iteration shadow count and early stopping to stay within budget, and report TPR@0.001% FPR with 95% bootstrap confidence intervals over at least 20 seeds. If CMIA no longer beats LiRA under matched budget, the '5x' headline should be reframed as compute-dependent; if it still beats LiRA with disjoint intervals, the compute confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the headline empirical claim is confounded by compute and unreported variance. In Table I, CMIA runs the base attack's full shadow-model count in each of up to K=10 cascading iterations, and all intermediate models are reused for the final score; Table III shows CMIA-LiRA takes 13.30 h vs 2.25 h for LiRA on MNIST. The 6.4x TPR gain at 0.001% FPR therefore coincides with a 5.9x compute increase. The matched-budget analysis is delegated to Figure 4 instead of the main tables, while Section V-A reports only five-seed point averages with no confidence intervals. At 0.001% FPR the threshold corresponds to a fraction of one false positive in the query's non-member set, so TPR estimates are highly seed-sensitive; without intervals, the size and even the existence of the advantage is not statistically pinned down. PMIA's proxy-transfer assumption is a real but separate risk; the compute/uncertainty confound touches the paper's central 'substantial outperformance' claim in both settings.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper formalizes membership inference as a query-set game, splitting attacks into adaptive (shadow models after queries) and non-adaptive. It proposes CMIA, an adaptive framework that iteratively trains shadow models conditioned on inferred 'anchor' members to exploit joint membership dependence, and PMIA, a non-adaptive attack that approximates the in-membership likelihood of a query using proxy instances from the adversary's dataset. Both are evaluated against six baselines on four image and two tabular datasets with four architectures, with claims of substantial TPR gains at low FPR, plus a Gibbs-convergence theorem and a posterior-odds derivation in the appendix.","tokens_in":34232,"tokens_out":10086,"duration_ms":119887,"significance":"The paper is likely useful for privacy auditing: it offers a clean distinction between adaptive and non-adaptive MIAs, an attack-agnostic wrapper (CMIA) that improves several baselines, and a low-cost non-adaptive attack (PMIA) that consistently improves TPR at low FPR. Table IX provides direct evidence that membership-dependence exploitation matters, and Figure 8 shows PMIA degrades gracefully with fewer shadow models. The release of code, the breadth of the benchmark (six datasets, five architectures), and the DP-SGD evaluation are strengths. However, the central 'substantially outperform' claim currently rests on comparisons that are not budget-matched in the main tables and on five-seed point averages without uncertainty, so the strength of the empirical claim is not yet fully pinned down.","major_comments":[{"comment":"The headline claim that 'CMIA improves LiRA by more than 5x' is made at an unequal compute budget. On MNIST, Table I reports a TPR increase from 0.12% to 0.77% at 0.001% FPR, but Table III reports 13.30 hours for CMIA versus 2.25 hours for LiRA, a 5.9x increase in wall-clock time. Since the main tables report the full CMIA with K=10 iterations, the gain may largely reflect the additional shadow models rather than the cascading mechanism. Figure 4's CMIA_opt and CMIA_loss partially address this, but only for LiRA on MNIST and CIFAR-10. Please make the same-budget comparison a primary result for all datasets and base attacks, or clearly separate the 'higher compute' claim from the 'same compute' claim in the main text.","section":"V-B, Tables I and III"},{"comment":"The evaluation reports point averages over five seeds with no variance, and the low-FPR metric is not robust to this omission. At 0.001% FPR on the adaptive MNIST setup, the non-member set is about 30,000 instances (D1 has 60,000 and the target is trained on 50%), so the threshold admits only about 0.3 false positives per seed; TPR estimates are therefore highly seed-sensitive and granular. To support the 'more than 5x' claim, the paper should report per-seed results or confidence intervals, and ideally the distribution of TPR across seeds for both baseline and CMIA/PMIA.","section":"V-A, Tables I and II"},{"comment":"Theorem 1 is an exact statement about the query instance, but PMIA replaces the in-membership likelihood N(mu_in, sigma_in^2) with the proxy-instance distribution N(mu_tilde_in, sigma_tilde_in^2) estimated from Algorithm 2, lines 15-22. No formal condition or error bound characterizes when proxy behavior approximates the query's in-membership behavior. This is a load-bearing approximation: Figure 7 shows a visible degradation under distribution shift, and the choice of top-10 CLIP/Wasserstein proxies is a heuristic. Please either provide an analysis of the approximation error or explicitly state that PMIA's proxy substitution is an empirically motivated heuristic rather than an instantiation of Theorem 1.","section":"IV-B, Theorem 1 and Algorithm 2"},{"comment":"The theoretical support for CMIA is currently limited to a standard convergence theorem for the exact Gibbs sampler (Appendix A), while the algorithm itself is a greedy, single-pass heuristic that conditions shadow models on pseudo-labeled anchors. No result shows that this procedure approximates the joint posterior or that the cascade improves MAP estimation; Section III-B explicitly calls CMIA 'a heuristic.' The abstract's claim of 'theoretical analyses for both attacks' should be tempered, or an approximation guarantee for the greedy cascade should be supplied.","section":"III-B, Algorithm 1 and Theorem 2"}],"minor_comments":[{"comment":"Several cells lack percent signs (e.g., '6.64', '11.5%', '80.4%'), and '6.64' should be '6.64%' for consistency with the rest of the table.","section":"Table II"},{"comment":"The stopping condition refers to M_t_in and M_t_out, which are not defined; it should refer to M^k_in and M^k_out.","section":"Algorithm 1, line 24"},{"comment":"The text says 'select the top 10 proxy models for each query instance'; this should read 'proxy instances'.","section":"V-E, first paragraph"},{"comment":"The Canary inference cost is listed as '>400,000' with no unit or explanation; please clarify or correct this entry.","section":"Table VI"},{"comment":"The phrase 'Closedly connected' is a typo and should be 'Closely connected'.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is strong and likely publishable, but the empirical core needs revision: the main tables should include matched-budget comparisons and uncertainty quantification, and the proxy-approximation and CMIA-greediness claims should be stated with appropriate caveats. These are fixable within the manuscript's scope, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this paper because it proposes two genuinely new membership inference mechanisms. CMIA cascades conditional shadow training to exploit membership dependencies between query instances; PMIA approximates the in-distribution likelihood ratio using proxy instances. Both are new relative to LiRA, RMIA, Canary, and RAPID, and the paper backs them with extensive experiments across six datasets and multiple architectures. Table IX is the most informative ablation: attack performance improves monotonically as the number of jointly queried instances grows, directly evidencing that modeling joint dependence matters. PMIA is also a clean, efficient application of the Bayesian posterior odds test, with no target-specific fitted constants; its extra cost over LiRA is marginal. The authors are honest that CMIA is a heuristic, and they ship code.\n\nThe soft spots are real but concentrated. The headline adaptive claim is not budget-matched. Table I runs the full shadow-model count per cascading iteration, so CMIA-LiRA uses about 6x the compute of LiRA alone (Table III: 13.30h vs 2.25h on MNIST). The 5x TPR gain at 0.001% FPR therefore comes with roughly 6x the training budget; the matched-budget comparison is only in Figure 4, not the main tables. Also, there are no confidence intervals anywhere, and at 0.001% FPR the TPR estimate is based on a fraction of one false positive, so seed-to-seed variance is high. The size of the adaptive advantage is not statistically pinned down. This is addressable: report intervals and put a budget-matched table in the main text. PMIA's proxy-transfer assumption is load-bearing, and the paper's own distribution-shift experiment shows noticeable degradation. That is a genuine limitation but it is stated clearly and the attack still beats baselines in that setting. The Gibbs-sampling theory in the appendix is a standard convergence re-derivation; it does not close the gap between the joint sampling ideal and the greedy cascade, but the paper labels the cascade a heuristic rather than overclaiming.\n\nBottom line: this deserves serious peer review. The contributions are real, the experiments are extensive, and the main weaknesses are methodological fixes rather than fatal flaws. I would send it to reviewers and would cite the attack constructions myself. For a reading group, this is a good paper to discuss because it highlights how compute budgets and evaluation granularity can shift conclusions.","headline":"Two new MIA mechanisms with solid experiments, but the headline adaptive gain is confounded by compute and missing confidence intervals.","tokens_in":34807,"tokens_out":2283,"would_cite":true,"duration_ms":25503,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that membership inference attacks become markedly stronger when the attacker exploits dependencies between query items and uses look-alike proxy samples to approximate in-training behavior, with the largest gains at very…","keywords":["membership inference attacks","shadow models","adaptive attacks","non-adaptive attacks","conditional shadow training","proxy data","posterior odds test","privacy auditing"],"falsifier":"Run PMIA against a target model while shadow models are trained on a different data distribution, and then isolate query instances whose nearest neighbors in the attacker's dataset are far away; if the membership-score distributions for true members and non-members overlap completely at 0.001% FPR, the proxy-similarity premise is falsified.","tokens_in":1477,"feed_emoji":"🔍","tokens_out":2395,"duration_ms":104370,"temperature":0.7,"pith_summary":"The paper tries to establish that existing membership inference attacks leave substantial signal unused, and that two new attacks can capture it. In the adaptive setting, it argues that a model's outputs on a whole query set create statistical dependencies between which instances were training members, so membership should be decided jointly rather than one instance at a time; its Cascading MIA (CMIA) feeds high-confidence memberships back into shadow training to amplify attack power. In the non-adaptive setting, where shadow models must be trained before queries arrive, it argues that an attacker can approximate the in-training confidence distribution of a query by watching proxy instances that behave similarly, and its Proxy MIA (PMIA) turns this into a likelihood-ratio test. If these claims hold, privacy auditors get stronger worst-case estimates of leakage, since the largest gains appear at the very low false-positive rates used to certify models.","feed_headline":"Cascading attack finds 5x more true positives at low FPR","feed_subtitle":"Two attacks exploit query dependencies and look-alike proxies to beat existing methods at low false-positive rates.","key_machinery":"CMIA uses conditional shadow training: each iteration trains shadow models on datasets that include confirmed anchor members and exclude confirmed non-members, then runs a base shadow-based attack to score all queries and select new anchors, approximating a single greedy step of Gibbs sampling over the joint membership vector. PMIA uses proxy likelihood approximation: before seeing queries it trains shadow models, then for each query it selects proxies (all data, same-class data, or top-10 similar instances by cosine similarity in a pretrained image-embedding space, or by Wasserstein distance on raw features), collects those proxies' logit-scaled confidence scores from the shadow models' in-training sets, fits Gaussian distributions to the in and out score sets, and computes a likelihood ratio $\\tilde{\\Lambda}$ as the membership score.","core_discovery":"The paper claims that membership inference is substantially stronger than previously measured once two overlooked signals are used: conditional dependencies among query memberships, and behavioral similarity between query and proxy data. In the adaptive setting, conditioning on the model's full output creates a collider that makes membership indicators dependent, and CMIA exploits this by cascading: it locks in high-confidence anchor members and retrains shadow models conditioned on those anchors, which improves inference on the rest of the query set. In the non-adaptive setting, PMIA approximates the likelihood ratio of the Bayesian posterior odds test by substituting proxy samples' in-training confidence distributions for the query's own, which recovers much of the gap between one-sided offline LiRA and the full posterior test.","pith_inferences":["I infer that the collider-dependency argument generalizes beyond shadow training: any attack that pools evidence across queries could gain from joint inference, though the paper only demonstrates the effect through conditional shadow training.","I infer that PMIA's accuracy will be uneven across a query population, since instances with no close proxies should show much lower attack success; privacy auditors using PMIA should therefore report subgroup or worst-case TPR rather than averages.","A testable extension is to replace the fixed proxy-selection metric with a learned similarity trained on shadow-model behaviors, which should improve PMIA under distribution shift because the paper's shift experiment identifies proxy quality as the bottleneck."],"forward_implications":["CMIA lifts every base attack it is applied to, with the largest gains in the low-FPR regime: on MNIST it raises LiRA's TPR at 0.001% FPR from 0.12% to 0.77%, a more than 5x improvement.","PMIA outperforms all non-adaptive baselines on the four image datasets; on CIFAR-100 it reaches 5.90% TPR at 0.001% FPR, more than twice the best baseline's 2.73%.","PMIA answers queries cheaply at inference time (about 15.8 seconds on MNIST), compared with RMIA's more than 400,000 seconds, making it practical for large query sets.","Both attacks weaken substantially under DP-SGD, confirming differential privacy as a mitigation.","The benefit of joint inference is monotone in query-set size: splitting 60,000 MNIST queries into 60,000 individual queries drops TPR at 0.001% FPR from 0.77% to 0.12%."],"supporting_citations":[{"why":"Introduces shadow training, the foundation technique that both CMIA and PMIA build on.","marker":"[1]"},{"why":"Defines LiRA, the base attack and Gaussian likelihood-ratio estimator that CMIA wraps and PMIA adapts.","marker":"[9]"},{"why":"Provides the Canary baseline and represents the class of adaptive attacks that CMIA boosts.","marker":"[7]"},{"why":"Provides RMIA, the strongest non-adaptive baseline that PMIA must beat.","marker":"[14]"},{"why":"Provides RAPID and the non-adaptive evaluation setup that PMIA is compared against.","marker":"[8]"},{"why":"Supplies the Bayes-optimal membership posterior odds test that PMIA explicitly approximates.","marker":"[24]"},{"why":"Provides the Calibration attack used as a baseline in both adaptive and non-adaptive settings.","marker":"[37]"},{"why":"Supplies the pretrained image embeddings used for instance-level proxy selection.","marker":"[25]"},{"why":"Provides the fast nearest-neighbor search used to retrieve top-10 proxy images.","marker":"[26]"}],"fun_headline_variants":["Cascading and proxy attacks boost membership inference 5x","Dependencies and proxies unlock 5x better membership attacks","Cascading attack exploits dependencies for 5x true positives","PMIA and CMIA: beating membership inference with proxies and cascades"],"cache_read_input_tokens":36992,"weakest_assumption_plain":"The load-bearing premise is that the confidence-score distribution of proxy samples, when those proxies are training members, is a faithful stand-in for the query sample's own confidence-score distribution when the query is a training member.","fun_headline_variants_meta":{"raw":{"variants":["Cascading and proxy attacks boost membership inference 5x","Dependencies and proxies unlock 5x better membership attacks","Cascading attack exploits dependencies for 5x true positives","PMIA and CMIA: beating membership inference with proxies and cascades"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000543,"raw_usage":{"total_tokens":2571,"prompt_tokens":885,"completion_tokens":1686,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":1614}},"tokens_in":501,"tokens_out":1686,"duration_ms":15597,"temperature":1.0,"reasoning_tokens":1614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:46:53.190797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PMIA against a target model while shadow models are trained on a different data distribution, and then isolate query instances whose nearest neighbors in the attacker's dataset are far away; if the membership-score distributions for true members and non-members overlap completely at 0.001% FPR, the proxy-similarity premise is falsified.","supporting_citations":[{"cited_title":"Membership inference attacks against machine learning models,","cited_arxiv_id":null,"evidence_quote":"Introduces shadow training, the foundation technique that both CMIA and PMIA build on."},{"cited_title":"Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries,","cited_arxiv_id":null,"evidence_quote":"Provides the Canary baseline and represents the class of adaptive attacks that CMIA boosts."},{"cited_title":"Low-Cost High-Power Mem- bership Inference Attacks,","cited_arxiv_id":null,"evidence_quote":"Provides RMIA, the strongest non-adaptive baseline that PMIA must beat."},{"cited_title":"Is Difficulty Calibration All We Need? Towards More Practical Member- ship Inference Attacks,","cited_arxiv_id":null,"evidence_quote":"Provides RAPID and the non-adaptive evaluation setup that PMIA is compared against."},{"cited_title":"White-box vs black-box: Bayes optimal strategies for membership inference,","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayes-optimal membership posterior odds test that PMIA explicitly approximates."},{"cited_title":"On the Impor- tance of Difficulty Calibration in Membership Inference Attacks,","cited_arxiv_id":null,"evidence_quote":"Provides the Calibration attack used as a baseline in both adaptive and non-adaptive settings."}],"review_version":1}