{"id":"63251aaa-f94b-4300-b49c-7e2191f5c547","arxiv_id":"2512.16292","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ICP-MIA infers membership in fine-tuned LLMs by measuring confidence improvement under in-context probes, beating prior black-box attacks at low false-positive rates.","lead":"This paper introduces a membership inference attack on fine-tuned language models that measures how much a model's confidence in a sample improves when extra context is added, using that improvement as a membership signal. The attack, ICP-MIA, outperforms earlier black-box attacks on several benchmarks, especially at low false-positive rates.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ICP proxy is validated only for reference-based semantically similar probes, but the headline ICP-MIA-SP attack uses random-mask/generation probes; no evidence connects those to a simulated fine-tuning step.","rationale":"The reader's weakest assumption was the broad fidelity of in-context probing to a real fine-tuning step, citing the moderate 0.547 correlation and the Pythia failure. My stress-test sharpens that: the only validation of the ICP-to-optimization-gap link uses reference-based probes selected by semantic similarity and likelihood maximization, while the attack that actually achieves the headline results (ICP-MIA-SP) uses random masks and generated paraphrases, aggregated by a min operation. Even the reference variant in the main table uses Dolly-15k rather than the domain-aligned pool from the validation. So the central theoretical grounding is not directly tested for the deployed attack. This does not refute the empirical MIA results — random perturbations can be an effective heuristic even if the optimization-gap story is wrong — but it weakens the paper's core novelty claim and reinforces the reader's CONDITIONAL verdict. No change in verdict is needed; the recommended check would settle whether the grounding concern actually lands.","tokens_in":30584,"tokens_out":4602,"duration_ms":48139,"concrete_test":"Run the Sec IV-C validation protocol with the exact ICP-MIA-SP probe generator used for Table I: for the same HealthcareMagic samples and LLaMA-3.2-3B-Instruct target, compute (a) true per-sample loss reduction from one additional gradient step and (b) the min-over-K ICP score under random masking (p=0.7) and under each of the four generation models. Report Spearman correlations separately. If the SP-probe correlations are not significantly positive (or are much lower than the 0.547 reference-probe correlation), the optimization-gap explanation for the headline attack fails; if they are comparable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that ICP-MIA is not another perturbation heuristic but estimates the Optimization Gap via in-context probing (Sec IV-B). The only direct evidence for that link is Sec IV-C / Fig 4: a Spearman correlation of 0.547 between true one-step gradient loss reduction and ICP-induced loss change, measured with probe contexts that are top-20 semantically similar iCliniq examples, selecting the one that maximizes the target's conditional likelihood. The attack that produces the headline numbers, ICP-MIA-SP (Table I), uses a different probe family: random masks at rates up to 70-100% and paraphrases generated by Llama-3.3-70B/Qwen/Mixtral/GPT-4.1-mini, scored with min over K probes (Eq 8). No experiment in the paper validates that this specific probe distribution or the min operation tracks a true gradient step. The paper's own explanation for why random masking works (Sec VI-C) is that multiple independent probes have a 'higher probability of finding an effective probe' — a perturbation-search story, not an optimization-gap story. ICP-MIA-Ref in the main table likewise uses Dolly-15k, not the aligned iCliniq pool used in validation. Thus the distinguishing theoretical grounding is unsupported for the actual attack variants. This is load-bearing because the stated novelty (Sec I) is the principled grounding; if the correlation is specific to aligned reference probes, the SP results could be another perturbation heuristic, and the Pythia-2.8B collapse to TPR@1%FPR=0.144 is consistent with that reading.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes ICP-MIA, a black-box membership inference attack against fine-tuned LLMs. The authors introduce the \"Optimization Gap\"—the remaining loss-reduction potential of a sample—as a membership signal, and propose to estimate it via in-context probing: prepending a probe context and measuring the change in log-likelihood of the target response. Two probe families are proposed: reference-data-based (semantically retrieved examples from a public dataset) and self-perturbation (randomly masked or LLM-generated variants). The paper reports that, on three datasets and three target models, ICP-MIA-SP outperforms reference-free baselines (e.g., AUC 0.942 vs 0.847 for ReCaLL on HealthCareMagic with Llama-3.2-3B-Instruct) and is competitive with reference-model attacks, while requiring no training. The paper also includes ablations for mask rate, K, generator choice, PEFT methods, and DP-SGD.","tokens_in":31052,"tokens_out":9077,"duration_ms":86077,"significance":"If the empirical results are robust, the proposed attack is practically useful: it is reference-free (SP variant), requires only logit access, and improves TPR at low FPR, which matters for real-world auditing. The underlying hypothesis—that membership leaves a trace in residual optimization potential—is interesting and the control experiment (Fig. 3) provides direct support. The paper also provides a detailed artifact and a thorough ablation suite. However, the central theoretical claim suffers from a validation gap between the proxy used to test the optimization-gap link and the probes used in the actual attack, and the reported gains come from point estimates with hyperparameters selected on the same benchmarks. These issues undercut the claim of a 'theoretically grounded' signal for the SP variant, but the attack itself may still be a strong heuristic.","major_comments":[{"comment":"The only empirical evidence that ICP scores approximate true gradient-based loss reduction (Fig. 4, Spearman ρ=0.547) is obtained with reference probes from the aligned iCliniq dataset, selecting the top-20 similar sample that maximizes the target's conditional likelihood. The actual attack variants—ICP-MIA-SP with random masking/generation, and ICP-MIA-Ref with Dolly-15k—use different probe distributions. No experiment connects random-mask or generated probes to a true fine-tuning step. Indeed, §VI-C explains the success of random masking by \"the probability of finding an effective probe\", which is a perturbation-search explanation, not an optimization-gap one. Thus the central novelty claim (\"we introduce the Optimization Gap as a fundamental signal\" and \"ICP-MIA is not another perturbation heuristic\") is unsupported for the attacks that produce the main results. Please either (a) vali","section":"Section IV-C vs. V-B.2, VI-C"},{"comment":"The hyperparameters of the attack—mask ratio p and number of probes K—are selected using the same evaluation datasets. Fig. 7 shows a non-monotonic sensitivity to p (optimal around 30-70%), and Fig. 11 shows gains from increasing K; the reported default p=0.7 and K=5/10 are chosen with knowledge of the test results. This is a form of test-data leakage that can inflate the reported performance relative to baselines with fixed/standard hyperparameters. Please use a separate validation split for hyperparameter selection, or report results across a grid of p and K without claiming optimality. Also explain the choice K=5 for SP when Table VIII shows K=20 frequently improves AUC/TPR.","section":"Section VI-C, Fig. 7; Section VI-G, Fig. 11"},{"comment":"Every metric is reported once, with no variance across seeds, data splits, or bootstrap resampling. This is especially problematic for TPR@1%FPR, which is computed on 500 non-members (5 expected false positives at 1%); the estimate has a standard error of roughly ±0.02–0.05, comparable to several reported differences. The claim of \"significantly outperforms\" (Abstract, §I) is not statistically supported. Please provide repeated runs with different seeds and/or bootstrap confidence intervals for the key comparisons, and state the number of samples and thresholding procedure.","section":"Table I"},{"comment":"The \"theoretical grounding\" is overclaimed. The power-law decay of training loss is cited but never used in the derivation; the Optimization Gap is defined but no formal connection between it and the ICP score is established beyond the moderate Spearman correlation in Fig. 4. The empirical control (Fig. 3) is supportive but only for the gap itself. The paper should either (a) provide a mechanistic derivation or a more rigorous justification of why prepending a probe context induces the same loss-reduction potential as a real gradient step, or (b) lower the strength of the claim from \"grounded in training dynamics\" to \"motivated by training dynamics.\" This issue is less about correctness than about calibration of claims, but it is load-bearing for the paper's framing.","section":"Section IV-A and IV-B"}],"minor_comments":[{"comment":"The sign convention in Eq. (4) is confusing: ΔLL(s) = LL(y|x;M) − LL(y|x;M′) is a negative number when the fine-tuned model assigns higher likelihood, yet the text calls it \"optimization gain.\" Eq. (7) defines ICP score similarly and later treats negative ICP scores as indicating larger gains. Please clarify the sign conventions so the relationship between ΔLL, the Optimization Gap, and the final membership score is unambiguous.","section":"Eq. (4) and (7)"},{"comment":"The model names are inconsistent: \"LLama-3.3-3B-instruct\" appears in §IV-C, but the experimental models are LLaMA-3.2-3B and LLaMA-3.2-3B-Instruct (Table I). Please correct.","section":"Section IV-C"},{"comment":"Generator model names are inconsistent: §VI-A lists \"Qwen2-72B-Instruct\" while Table II uses \"Qwen2.5-72B-Instruct.\" Please standardize.","section":"Section VI-A and Table II"},{"comment":"The x-axis label \"Number of Pretrain Data in Nonmembers\" is ambiguous, and the axis values (100, 70, 50, 30) decrease from left to right while the text says performance degrades \"as the proportion of pre-trained samples increases.\" Please clarify what the axis represents and the direction.","section":"Figure 9"},{"comment":"The appendix says to fine-tune for 3 epochs and use the 2nd epoch checkpoint, while §VI-A states \"all models are fully fine-tuned for two epochs.\" Please reconcile this discrepancy.","section":"Artifact Appendix A"},{"comment":"The min over K probes in Eq. (8) is presented as a central design choice, but no justification is given for why the minimum (rather than the mean or the maximum over probes) is the correct aggregation. If this follows from the optimization-gap interpretation, state it; otherwise, provide an empirical comparison of aggregation methods.","section":"Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong artifact and extensive experiments, but the central claim about the optimization gap is oversold. The validation gap between the probe family used to test the theory (aligned iCliniq reference probes) and the probe families used in the actual attacks (random masking/generation and Dolly-based retrieval) is the main technical obstacle. The lack of variance estimates and test-set hyperparameter selection also weaken the empirical claims. A major revision should address these issues; if the authors can either validate the SP probes against true gradient steps or reframe the SP variant as a perturbation heuristic, the paper could become publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about if you work on LLM membership inference. The empirical core is solid: the min-over-K probing score and the self-perturbation variant (random masking, generation) are genuinely new relative to ReCaLL and Neighborhood, and the gains at TPR@1%FPR on CNN-DM and MedInstruct under LLaMA models are real. The ablation suite is unusually thorough — PEFT configs, DP-SGD, label-only access, training order, and schedule effects — and the artifact is detailed enough that I expect the main numbers to reproduce.\n\nThe soft spots are real, though they don't sink the empirical contribution. Most important: the validation that in-context probing approximates a true gradient step (Sec IV-C) uses only semantically similar reference probes from iCliniq, with a Spearman rho of 0.547. The attack that actually produces the headline numbers, ICP-MIA-SP, uses random masks and generated paraphrases, and no experiment connects those probes to a simulated fine-tuning step. The paper's own explanation for why random masking works (Sec VI-C) is a search story — 'higher probability of finding an effective probe' — which is honest but undercuts the theoretical framing. The Pythia-2.8B collapse (TPR@1%FPR = 0.144 on CNN-DM) is consistent with SP being another perturbation heuristic. So the 'Optimization Gap' grounding is load-bearing only for the Ref variant, and even there the correlation is moderate.\n\nTwo smaller issues: Table I reports each metric once, no variance across seeds or splits; and the mask rate p=0.7 and K are selected using the same benchmark data, which risks a mild selection bias. The power-law discussion is decorative; the 'theory' is essentially a restatement of overfitting.\n\nOverall: worth a serious referee. The authors should either validate the SP variant against the true optimization gap or soften the theoretical claims, add error bars, and report hyperparameter selection more transparently. I'd take the paper to the reading group and would cite it if I were writing on MIA for LLMs.","headline":"Solid empirical MIA paper; the optimization-gap framing is validated only for the reference variant, and the headline self-perturbation attack works but is better described as a perturbation heuristic.","tokens_in":31512,"tokens_out":2850,"would_cite":true,"duration_ms":28927,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the decisive signal for membership inference in fine-tuned LLMs is the 'optimization gap' — the remaining loss-reduction potential of a sample — and that a training-free in-context probing technique can estimate this","keywords":["membership inference","optimization gap","in-context learning","in-context probing","fine-tuned LLMs","black-box attack","privacy auditing","diminishing returns"],"falsifier":"Compute true per-sample loss reductions from an actual fine-tuning step on a holdout set and compare them against ICP scores across a broad model family; if the Spearman correlation falls below some floor, or if a model with strong ICL still shows no gap, the claim fails. Concretely, the paper itself notes Pythia-2.8B with CNN-DM yields TPR@1%FPR 0.144 — a scenario where the proxy breaks.","tokens_in":30540,"feed_emoji":"🔍","tokens_out":2692,"duration_ms":24974,"temperature":0.7,"pith_summary":"The paper claims that the signal separating members from non-members in a fine-tuned LLM is the 'optimization gap': how much loss could still be reduced if training continued on that sample. Members have already been optimized, so their gap is near zero; non-members show large potential gains. The paper proposes estimating this gap in a black-box setting by in-context probing — prepending carefully chosen contexts and measuring the log-likelihood improvement. This yields a membership score that outperforms previous black-box attacks, especially with a self-perturbation strategy that requires no reference data. If correct, it provides a principled, training-free way to audit privacy exposure in deployed models.","feed_headline":"Optimization gap exposes training data in fine-tuned LLMs","feed_subtitle":"Attack scores a sample by how much extra optimization it still allows; members barely improve.","key_machinery":"The central object is the Optimization Gap, defined as the difference between the loss at a converged model and the loss after one additional optimization step on that sample. The machinery that carries the argument is In-Context Probing (ICP): a training-free, black-box proxy that prepends a probe context to the input and computes the log-likelihood change; the minimum score over multiple probes is the membership signal.","core_discovery":"The discovery is that the optimization gap is a fundamental and usable membership signal: at convergence, member samples have little remaining loss-reduction potential, while non-members retain substantial potential (measured per-sample: non-members mean loss reduction 0.368 vs members 0.125). The paper shows that in-context probing can approximate this gap without any parameter updates, and that the ICP score — the log-likelihood change induced by a probe context — correlates with the true single-step gradient loss reduction (Spearman 0.547). This grounding distinguishes the method from earlier heuristic perturbations.","pith_inferences":["The moderate correlation (0.547) suggests the grounding is partial; a more faithful proxy might come from multi-step in-context updates or looped inference, which could sharpen the signal at low FPR.","The mask-rate ablation's single-peaked curve implies the probe must remove exactly the right amount of information; this could be used as a principled design rule for future perturbation attacks.","The recency effect found under fixed learning rates (later training partitions more vulnerable) implies that fine-tuning order and schedule are not neutral: a deployed model's privacy risk may be unevenly distributed across its training data, which auditing tools should report per stratum."],"forward_implications":["Fine-tuned models' membership exposure can be measured with only query access and no reference models, using the self-perturbation variant.","The attack excels at high-precision settings: for example, TPR@1%FPR reaches 0.518 on CNN-DM, far above prior reference-free attacks.","Instruction-tuned models are more vulnerable than their base versions, because stronger in-context learning makes the probing approximation more faithful.","PEFT setups (LoRA, QLoRA) remain vulnerable; larger adapter rank increases memorization and attack success, while 4-bit quantization mitigates it.","Differential privacy (DP-SGD) suppresses all attacks including this one, but ICP-MIA still leads among reference-free methods."],"fun_headline_variants":["Optimization gap exposes training data in LLMs","In-context probing reveals LLM membership","Members leak via optimization gap in LLMs","Training-free attack uses optimization gap","LLM privacy: probing finds optimization gap"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire method assumes that in-context probing faithfully mimics a real fine-tuning step; the only direct evidence is a moderate correlation (0.547), and when that approximation degrades (as with Pythia-2.8B) the attack's low-FPR performance collapses.","fun_headline_variants_meta":{"raw":{"variants":["Optimization gap exposes training data in LLMs","In-context probing reveals LLM membership","Members leak via optimization gap in LLMs","Training-free attack uses optimization gap","LLM privacy: probing finds optimization gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2777,"prompt_tokens":761,"completion_tokens":2016,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1961}},"tokens_in":505,"tokens_out":2016,"duration_ms":14894,"temperature":1.0,"reasoning_tokens":1961,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T15:34:08.414760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute true per-sample loss reductions from an actual fine-tuning step on a holdout set and compare them against ICP scores across a broad model family; if the Spearman correlation falls below some floor, or if a model with strong ICL still shows no gap, the claim fails. Concretely, the paper itself notes Pythia-2.8B with CNN-DM yields TPR@1%FPR 0.144 — a scenario where the proxy breaks.","supporting_citations":[],"review_version":1}