{"id":"147f7faa-9634-42ee-ae32-1cd8b6f2c052","arxiv_id":"2411.11144","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"CLMIA is a membership inference attack that pretrains an attack model on unlabeled classifier posteriors via contrastive learning and fine-tunes it with a small labeled set.","lead":"This paper introduces CLMIA, a membership inference attack that uses contrastive learning to train an attack model with very few labeled examples. It claims to outperform earlier attacks when labeled membership data is scarce, which matters for practical privacy auditing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported results contradict the central claim: CLMIA is not consistently better than baselines (CIFAR-100/ResNet-18: NN attack 0.970 vs CLMIA 0.894; CIFAR-10/VGG-19: prediction correctness 0.962 vs CLMIA 0.619).","rationale":"The reader's verdict of REJECT is appropriate, but I find the most load-bearing concern to be the direct contradiction between the central claim and the paper's own experimental tables, rather than the threat-model inconsistency emphasized in the reader's weakest_assumption. The threat-model issue is real: Section III-A says the attacker does not know the target model structure, yet Section III-C and Algorithm 1 require shadow models with the same structure as the target, so the positive-pair construction cannot be mounted as stated. However, even if that inconsistency were fixed by granting the attacker shadow-model access, the empirical evidence would still not support the headline claim. The reported numbers show CLMIA losing to NN attack on CIFAR-100 with ResNet-18 and losing badly to Prediction correctness on CIFAR-10 with VGG-19. The paper's own tables should settle the central claim, and they do not support it. I partially agree with the reader because their rationale mentions the table contradiction, but their weakest_assumption was focused on the threat model. The concrete test of re-running those two configurations would directly settle whether the numerical contradiction is real or an artifact of reporting.","tokens_in":13798,"tokens_out":2677,"duration_ms":23543,"concrete_test":"Rerun the evaluation for CIFAR-100 with ResNet-18 and CIFAR-10 with VGG-19 using the paper's stated protocol, including the same shadow-model dropout settings, attack model architecture, and NN attack baseline. If the balanced accuracies reproduce the reported values (NN attack 0.970 > CLMIA 0.894 on CIFAR-100; Prediction correctness 0.962 > CLMIA 0.619 on CIFAR-10), then the central 'performs better' claim is falsified on those configurations. If the numbers do not reproduce, the paper must still report full per-configuration results with variance across seeds before the claim can be assessed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that CLMIA 'performs better than existing attack methods for different datasets and model structures.' The paper's own tables refute this. In Table II, on CIFAR-100 with ResNet-18, the NN attack achieves balanced accuracy 0.970 and F1 score 0.970, while CLMIA achieves 0.894 and 0.903, respectively. In Appendix Table V, on CIFAR-10 with VGG-19, Prediction correctness achieves balanced accuracy 0.962 and F1 0.963, while CLMIA achieves 0.619 and 0.745. These are not marginal differences; they are large gaps on exactly the dataset/model combinations the claim is supposed to cover. The secondary claim about performance 'with data with less marked identity information' is supported only by Figure 4 for a single dataset/model pair (CIFAR-100, ResNet-18), with no reported error bars or multiple runs, so it is also not established. Because the central comparison fails on the paper's own reported numbers, the main contribution is not supported without corrected or additional experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLMIA, a membership inference attack that trains an attack model with unsupervised contrastive learning on posterior-probability vectors obtained from dropout-augmented shadow models, followed by supervised fine-tuning on a small labeled set. The authors claim that CLMIA outperforms existing MIAs across different datasets and model architectures, especially when little labeled identity information is available, and evaluate it on CIFAR-10, CIFAR-100, and STL-10 with CNN, VGG-19, and ResNet-18 target models.","tokens_in":14086,"tokens_out":4609,"duration_ms":42971,"significance":"If the central claims were supported, the paper would make a useful practical contribution by attacking models with very little labeled membership data, which is a realistic constraint. The experimental comparison against an 'Only FC layer' baseline (Table III) does suggest that contrastive pretraining adds value over using the fine-tuning head alone. However, the manuscript's own quantitative results contradict the headline claim of universal superiority over prior attacks, and the threat model is internally inconsistent. The core privacy contribution is therefore not established by the present experiments, although the underlying idea of contrastive learning on model posteriors may be worth further study.","major_comments":[{"comment":"The abstract and Section IV-B claim that CLMIA 'performs better than existing attack methods for different datasets and model structures.' This is directly contradicted by the paper's own tables. In Table II, on CIFAR-100 with ResNet-18, the NN attack reaches balanced accuracy 0.970 and F1 0.970, while CLMIA reaches only 0.894 and 0.903, respectively. In Table V, on CIFAR-10 with VGG-19, prediction correctness reaches balanced accuracy 0.962 and F1 0.963, while CLMIA reaches 0.619 and 0.745, respectively. These are substantial gaps on exactly the dataset/model combinations the claim is meant to cover. The main claim of the paper is thus not supported by its own reported data.","section":"Abstract and Section IV-B, Table II and Table V"},{"comment":"The threat model in Section III-A states that the attacker does not know the structure of the target model and only has access to its posterior probabilities. However, the attack pipeline in Section III-C and Figure 2 requires training shadow models that have the same structure as the target model and that incorporate dropout layers with rates d1 and d2. Without knowledge of the target's architecture, the positive-pair generation via such shadow models cannot be implemented. This is an internal inconsistency in the core method. Additionally, the paper does not specify what data are used to train the shadow models; the stated attacker resources Dt and Dl do not obviously include an auxiliary dataset drawn from the target's training distribution, which is normally required for shadow models.","section":"Section III-A vs. Section III-C and Figure 2"},{"comment":"The description of how the attack training features are built is ambiguous. Algorithm 1 line 3 indicates that two positive samples xi, xj are obtained from the two shadow models S+d1_1 and S+d2_2, but line 4 computes p* = p ⊕ max(p) ⊕ entropy(p) for a single p, without specifying whether p is xi, xj, or some other posterior vector. It is also unclear whether the attack model's contrastive loss operates on pairs of vectors (xi, xj) or on single vectors augmented with max and entropy features. This ambiguity prevents replication and obscures what exactly the contrastive learner sees.","section":"Algorithm 1 and Section III-C, positive-pair construction"},{"comment":"Section IV-B claims that CLMIA achieves the best performance 'in the low-FPR regime' across most scenarios, but no numeric TPR-at-low-FPR results are reported; ROC curves alone do not substantiate this. Likewise, the important secondary claim that CLMIA is especially effective when labeled identity information is scarce is supported only by Figure 4, which shows a single dataset/model combination (CIFAR-100, ResNet-18) with no error bars, no multiple runs, and no statistical test. These claims require more extensive evidence before they can be accepted.","section":"Section IV-B and Figure 4, low-FPR and low-label claims"}],"minor_comments":[{"comment":"The caption says additional results can be found in 'Section VIII-A', but the additional results are in the unnumbered Appendix A; the reference should be corrected.","section":"Table II caption"},{"comment":"The name 'Lein et al.' for prediction correctness should be 'Leino et al.' (reference [21]).","section":"Section IV-A, baselines"},{"comment":"The text says the adversary 'adds the dropout layer to the output of the last layer of the target model,' but the pipeline actually uses shadow models with dropout. This wording should be corrected to say the shadow models are augmented with dropout, not the target model itself.","section":"Section III-C and Figure 2"},{"comment":"The notation S1 + d1 → S+d1_1 is confusing; it is not clear what the '+' operation means, and the subscript/superscript formatting makes it hard to read. A clearer definition of the shadow models is needed.","section":"Equation (3)"},{"comment":"The main tables do not report the size of the labeled set Dl used in the primary comparisons; Figure 4 uses specific sizes, but the reader cannot tell whether Table II and Table V use the same sizes. This should be stated explicitly.","section":"Experimental setup"},{"comment":"The system model figure contains handwritten-style annotations such as '11,, nn xy xy' and is generally hard to read; a clean, typeset diagram would greatly improve clarity.","section":"Figure 2"}],"recommendation":"reject","confidential_remarks":"The paper's central advertised claim is contradicted by its own experimental tables, and the threat model contains a load-bearing inconsistency regarding knowledge of the target architecture. A re-scoped submission that presents CLMIA as a competitive or specialized attack in low-label settings, after fixing the threat model and adding the missing shadow-model training details, could be revisited, but in its current form the contribution is not supported. The paper also appears to be an extension of a CCS 2023 poster, so the novelty framing should be checked carefully if a future version is submitted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: the contrastive-learning angle is genuinely new for membership inference, but the paper as written overclaims. Its own tables show CLMIA losing to simpler baselines on two dataset/model combinations, which contradicts the abstract's central claim.\n\nWhat's new: using unsupervised contrastive learning to train an MIA attack model, with two dropout versions of a shadow model producing positive pairs and the standard NT-Xent loss, appears to be the first application in this space. That is a legitimate idea, and the ablation study (only FC layer, dropout rates, temperature) shows the authors thought through the design. The paper is an extension of their CCS 2023 poster, so the core novelty is incremental, but the experimental coverage here is substantially broader than a poster.\n\nNow the soft spots. The biggest one is the headline claim: the abstract and Section IV say CLMIA \"performs better than existing attack methods for different datasets and model structures.\" Table II shows the NN attack at 0.970 balanced accuracy on CIFAR-100 with ResNet-18, against CLMIA's 0.894. Appendix Table V shows prediction correctness at 0.962 on CIFAR-10 with VGG-19, against CLMIA's 0.619. These are not marginal gaps; they are large deficits on exactly the comparisons the claim is supposed to cover. The secondary claim about low labeled 'identity information' rests on Figure 4, which reports a single dataset/model pair with no error bars or repeated runs, so that result is not established either.\n\nThe threat model is also internally inconsistent. Section III-A says the attacker does not know the target model structure, but the method requires shadow models with the same structure (plus dropout layers) to generate positive pairs. If the attacker truly has black-box access, this step collapses. The paper needs to either adopt a known-architecture threat model or explain how the positive-pair construction works without that knowledge.\n\nNone of this kills the underlying idea. The problem is the presentation and the unsupported headline claim. With corrected claims and a realistic threat model, the method could be worth evaluating. As it stands, the evidence does not support what the paper advertises.\n\nThis is for a privacy-auditing reader who wants to track contrastive-learning attacks, not for someone looking for a ready-to-use attack. I would still send it to peer review rather than desk reject, because the idea is real and the experimental breadth is substantial; a good referee can force the authors to fix the threat model and the overclaims. But I wouldn't cite it in its current form.","headline":"Contrastive learning for MIA is a real idea, but the paper's own numbers undercut its advertised advantage.","tokens_in":14594,"tokens_out":1941,"would_cite":false,"duration_ms":17955,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unsupervised contrastive learning on a model's output probabilities can expose which samples were in its training set.","keywords":["membership inference attack","contrastive learning","black-box attack","shadow model","dropout augmentation","posterior probabilities","privacy leakage","NT-Xent loss"],"falsifier":"Use a target model whose architecture is kept secret and train the CLMIA shadow models with a deliberately wrong architecture; if balanced accuracy then falls to the level of the best threshold baseline rather than the reported figures, the claimed advantage depends on architecture knowledge. Alternatively, run CLMIA against a target whose inference path disables dropout and returns deterministic posteriors; the positive-pair construction would produce identical views and the contrastive training would collapse.","tokens_in":13617,"feed_emoji":"🕵️","tokens_out":5192,"duration_ms":43719,"temperature":0.7,"pith_summary":"This paper argues that membership inference attacks—determining whether a specific sample was in a model's training set—can be built almost entirely without labeled member/non-member data. The proposed attack, CLMIA, trains a contrastive encoder on the target model's output probabilities alone, treating two dropout-perturbed posterior vectors of the same input as a positive pair and all other inputs in the batch as negatives. Only a small labeled dataset is used, to fine-tune a single MLP head. The authors report that this beats six existing attacks across three datasets and three model architectures, with the largest gains when labeled identity information is scarce. If true, it means an attacker with very little prior knowledge can still effectively probe a model's training set.","feed_headline":"Contrastive learning attack finds training data with few labels","feed_subtitle":"An attacker needs only a small labeled sample to fine-tune, not a large balanced set.","key_machinery":"The machinery is dropout-based positive-pair generation combined with a contrastive objective (NT-Xent, the normalized temperature-scaled cross-entropy loss) over posterior-derived features. Because the target model returns only one output per input, the attacker builds two shadow models with the same architecture as the target plus dropout layers at different rates, so the same sample yields two distinct posterior vectors treated as a positive pair; every other sample in the batch serves as a negative. Each posterior is augmented before contrastive training by concatenating its maximum probability and entropy, which injects the classic MIA signals into the learned representation. This contrastively pretrained encoder is then frozen and topped with a small MLP trained on a handful of labeled members/non-members.","core_discovery":"On the paper's own terms, the central discovery is that the membership signal in a model's posterior probabilities is learnable by unsupervised contrastive learning. CLMIA constructs an attack training set from an unlabeled target dataset: each sample's posterior vector is passed through two shadow models that share the target's architecture plus dropout layers, producing two positive views; the input feature is the raw posterior concatenated with its maximum value and entropy. Training with the NT-Xent loss pulls these pairs together and pushes different inputs apart, without any membership labels. The resulting encoder already separates members from non-members; a final MLP layer, trained on a small labeled set, turns the embedding into a membership decision. The paper claims this outperforms threshold-based, prediction-correctness, and shadow-model baselines, especially in the low false-positive regime and when labeled data are scarce.","pith_inferences":["The stated black-box threat model is not fully met: creating the shadow models requires knowing the target's architecture and inserting dropout, so the practical assumption is knowledge of the model family, not just its outputs.","The improvement over the 'only FC layer' baseline may come partly from the max/entropy feature augmentation rather than from contrastive learning per se; a non-contrastive encoder trained on the same augmented features would isolate the contribution.","The same dropout-positive-pair trick could be applied to label-only or decision-based queries, where the attacker sees only the predicted class; whether contrastive learning still separates members there is a testable extension.","If contrastive pretraining does encode membership from unlabeled posteriors, then simply observing a model's API outputs over time—without any labeled data—could let an attacker train a reusable membership probe for any similarly structured model."],"forward_implications":["Realistic attackers, who typically have far fewer labeled members than non-members, can mount effective MIAs rather than needing a large balanced labeled set.","The attack transfers across model architectures (CNN, VGG-19, ResNet-18) and datasets, so it is not tied to one target configuration.","In the low-FPR regime—where prior work says MIAs must be evaluated—CLMIA claims the best performance, meaning a small number of false positives can still yield true membership disclosures.","Attack accuracy depends on the ratio of labeled members to non-members, rising as the member share grows; this gives defenders a lever, such as making member data less identifiable.","Only a single additional MLP layer is tuned during supervised fine-tuning, so the attack remains cheap even after unsupervised pretraining."],"supporting_citations":[{"why":"Supplies the NT-Xent contrastive loss and the positive/negative sampling strategy used to train the attack encoder.","marker":"[6]"},{"why":"Supplies the dropout-based positive-pair construction, here applied to posterior vectors.","marker":"[11]"},{"why":"Introduces shadow-model-based membership inference, which CLMIA builds on for generating attack data.","marker":"[36]"},{"why":"Provides the single-shadow-model NN attack used as a key baseline and the relaxed knowledge assumptions.","marker":"[34]"},{"why":"Establishes the low-FPR evaluation metric that frames the claimed improvements.","marker":"[2]"}],"fun_headline_variants":["Unsupervised contrastive learning exposes membership in models","Attack learns membership without labeled samples","Membership inference with contrastive learning and minimal labels","Contrastive attack reveals training data with few labels","Few labels suffice for contrastive membership inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's core premise is that the attacker can build shadow models with the same architecture as the target and add dropout layers to generate paired outputs; if the attacker truly does not know the target's structure, as the paper's threat model states, this premise does not hold.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised contrastive learning exposes membership in models","Attack learns membership without labeled samples","Membership inference with contrastive learning and minimal labels","Contrastive attack reveals training data with few labels","Few labels suffice for contrastive membership inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000494,"raw_usage":{"total_tokens":2414,"prompt_tokens":923,"completion_tokens":1491,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1421}},"tokens_in":539,"tokens_out":1491,"duration_ms":11125,"temperature":1.0,"reasoning_tokens":1421,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:51:57.569449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a target model whose architecture is kept secret and train the CLMIA shadow models with a deliberately wrong architecture; if balanced accuracy then falls to the level of the best threshold baseline rather than the reported figures, the claimed advantage depends on architecture knowledge. Alternatively, run CLMIA against a target whose inference path disables dropout and returns deterministic posteriors; the positive-pair construction would produce identical views and the contrastive training would collapse.","supporting_citations":[{"cited_title":"Simcse: Simple contrastive learning of sentence embeddings,","cited_arxiv_id":null,"evidence_quote":"Supplies the dropout-based positive-pair construction, here applied to posterior vectors."},{"cited_title":"Membership inference attacks against machine learning models,","cited_arxiv_id":null,"evidence_quote":"Introduces shadow-model-based membership inference, which CLMIA builds on for generating attack data."},{"cited_title":"Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models,","cited_arxiv_id":null,"evidence_quote":"Provides the single-shadow-model NN attack used as a key baseline and the relaxed knowledge assumptions."}],"review_version":1}