{"id":"2775a34b-bcdd-4169-a573-011885b7ec13","arxiv_id":"2411.19876","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LUMIA uses per-layer linear probes on LLM activations to detect training-data membership, outperforming output-based attacks and extending to multimodal models.","lead":"Researchers trained small classifiers on the internal activations of language models to detect whether a text or image was in the training data. The method, LUMIA, outperforms existing membership inference attacks in most tested settings and works for both text-only and multimodal models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 15.71% AUC gain is an upper bound because the layer with maximum AUC is selected and evaluated on the same 80/20 split, with no held-out test set or error bars.","rationale":"The central claim is quantitative: LUMIA improves AUC by 15.71% on average over prior MIA methods. For this claim to hold, the reported AUC must be an unbiased estimate of performance on unseen data. The paper's own description defeats this: the layer achieving the maximum AUC is identified on the same 20% split used to report that AUC. This is textbook selection-on-the-validation-set bias. With many layers, the maximum of many correlated AUC estimates is systematically above the true value of any single layer, so the reported numbers are optimistic. The three repeats reduce sampling noise but do not fix the selection bias, and the absence of standard deviations means the reader cannot assess the stability of the 15.71% figure. No other concern is as directly tied to the headline claim: the method's use of MLPs rather than strictly linear probes affects interpretation but not the core empirical question, and the multimodal setup has additional distribution-shift risks but the unimodal claim is the paper's primary contribution. The reader's weakest assumption identifies the same protocol issue; I agree. Verdict should remain CONDITIONAL: the qualitative finding is likely correct, but the quantitative claims need re-evaluation with a clean test split or nested cross-validation before they can be accepted. Therefore no verdict change is needed.","tokens_in":17112,"tokens_out":5780,"duration_ms":48995,"concrete_test":"Re-run the unimodal comparison with a nested split: for each of the three repeats, train LPs on 70% of each dataset, select the layer l* on a disjoint 15% validation split, and compute the final AUC on the remaining 15% test split (or use 5-fold cross-validation with layer selection inside the loop). Report the mean and standard deviation of the test AUC for all 176 unimodal configurations and compare against the same SOTA baselines. If the average improvement over SOTA drops materially below 15.71% (e.g., below 5 points) or the AUC>0.6 frequency falls below 65.33%, the headline claim is inflated by selection bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the evaluation protocol for the headline unimodal result. Section 3.1 defines the reported AUC as the maximum over layers (layer l* where P_l* achieves the highest AUC). Section 4.3 states that data is split once into 80% training and 20% validation, with three repeats, and the average is computed; no separate test set is mentioned anywhere. Thus the same 20% validation split is used both to select l* and to report the final AUC. Because AUC is maximized over up to 36 layers (Pythia 12B) or 32 layers (GPT-Neo 2.7B), even a probe with no true membership signal would achieve a non-trivial max-over-layers AUC on the selection set. The reported 15.71% average gain and 65.33% AUC>0.6 frequency are therefore upper-bound estimates; the magnitude of the inflation is unknown, and the absence of standard deviations or confidence intervals (only three repeats) makes the improvement hard to distinguish from selection noise. The qualitative conclusion that internal activations carry membership signal is plausible, but the quantitative central claim is not supported as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces LUMIA, a white-box membership inference method that trains linear probes on per-layer internal activations of LLMs. It is evaluated on unimodal (Pythia, GPT-Neo; WikiMIA, ArXiv, Mimir, Gutenberg, etc.) and multimodal (LLaVA-OneVision; seven datasets) settings, comparing AUC against several SOTA baselines. The paper reports an average relative AUC improvement of 15.71% over SOTA, reaching AUC>0.6 in 65.33% of unimodal cases and 85.9% of multimodal experiments, and analyzes how layer depth, model size, deduplication, and dataset bias affect detection.","tokens_in":17322,"tokens_out":6545,"duration_ms":53762,"significance":"The core idea is timely and useful: using internal activations for MIA is a natural white-box direction, and the layer-by-layer analysis plus the extension to multimodal models are novel relative to prior work. The paper's strengths include broad model/dataset coverage, comparison with multiple baselines, and a public (reduced) code release. If the quantitative claims survive a corrected evaluation protocol, this would be a solid contribution to the MIA literature. However, the headline numbers are not yet reliable because of the evaluation protocol.","major_comments":[{"comment":"The reported unimodal AUC is the maximum over layers l* selected on the same 80/20 split that is then used to report the final AUC. Section 3.1 defines l* as the layer with highest AUC, and Section 4.3 states that all data are split once into 80% training and 20% validation with three repeats and no separate test set. Since l* is chosen and evaluated on the same validation data, the headline 15.71% average improvement and the 65.33% AUC>0.6 frequency are upper-bound estimates; the maximum over up to 36 layers (Pythia 12B) can inflate AUC even for a probe with no true signal. Please re-evaluate on a fresh held-out split after layer selection, or otherwise quantify the selection bias (e.g., by reporting the difference between max-layer and fixed-layer AUC on a test set).","section":"3.1 and 4.3"},{"comment":"The evaluation reports only the average of three repeats and no standard deviations, confidence intervals, or statistical significance tests. Many improvements in Tables 1 and 2 are small (e.g., 0.00% for Wikipedia Ref in Table 2, 0.30% for WikiMIA Heuristics in Table 1), so the claim that LUMIA improves on SOTA in 174 of 176 cases is not supported without a measure of variance. Please report per-repeat results, error bars, and a paired significance test (e.g., Wilcoxon or bootstrap) over the three repeats.","section":"4.3 and 5.1"},{"comment":"The multimodal summary is internally inconsistent. The text states that all configurations except Magpie achieve AUC>0.6, but Table 3 shows Textcaps 0.5B Textual+visual at 0.540 and MathV360k 0.5B Textual+visual at 0.599 and Visual at 0.584. If the 85.9% figure counts the best modality per dataset-size pair, the count is 12/14 (85.7%), not 85.9%; if it counts all modality entries, the fraction is 21/28 (75%). Please define the counting unit and correct the statements.","section":"5.1 and Table 3"}],"minor_comments":[{"comment":"The headline number appears as 15.71% in the abstract and Takeaway 1, but Section 5.1 states 15.75%; please harmonize.","section":"Abstract and Section 5.1"},{"comment":"The 15.71% figure is an average of relative improvements (each entry is computed as (Ours - SOTA)/SOTA), not an average gain in AUC units; please state this explicitly to avoid overclaiming.","section":"Tables 1 and 2"},{"comment":"The code release is described as a 'reduced version' of LUMIA; for reproducibility, a full release with the exact train/validation split and layer-selection procedure is needed.","section":"Footnote 5"},{"comment":"Linear probes are defined in Section 2.2 as MLPs, while Section 2.2 also calls them linear classifiers; please clarify whether the probe is a single linear layer or a multi-layer perceptron, since this affects the interpretation of 'linear probing'.","section":"2.2 and 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea and broad evaluation, but the quantitative claims need a corrected evaluation protocol. The reduced code release and the lack of a separate test set should be addressed during revision. The fit with a security/privacy venue is good."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nYou should read this as a useful but methodologically imperfect empirical study. The core finding—that linear probes on internal activations separate members from non-members without fine-tuning or prompts, in both text and multimodal models—is probably real. The layer-wise analysis is genuinely new, and the multimodal results (6 of 7 datasets above 0.6 AUC) are the first of their kind. The paper deserves a serious referee.\n\nWhat it does well: It removes the two crutches from Liu et al.—fine-tuning and a fixed prompt—and shows the signal survives. The layer-depth findings (early layers for temporal-bias datasets, deeper layers for n-gram-bias) are a nice observation. The evaluation is broad: 14 text datasets, 7 multimodal, three model families, including dedup/non-dedup Pythia. That breadth is a real contribution.\n\nThe soft spot is the evaluation protocol for the headline numbers. Section 3.1 defines the reported AUC as the maximum over layers, and Section 4.3 describes a single 80/20 split, repeated three times, with no held-out test set for choosing the layer. So the same 20% is used for both layer selection and final scoring. With 32–36 layers, max-over-layers inflates AUC even with no true signal. The 15.71% average gain and 65.33% frequency are upper bounds, and the lack of standard deviations or confidence intervals means we cannot tell how much of the gain is selection noise. The qualitative conclusions are plausible, but the quantitative claims need a clean test split or nested cross-validation.\n\nMinor issues: abstract says 15.71%, body says 15.75%; the code release is explicitly reduced; and the comparison against Liu et al. is not apples-to-apples, though they do address this.\n\nBottom line: worth peer review, but it should be conditional on redoing the evaluation with a proper held-out selection procedure and reporting error bars. If you work on MIA or auditing, this is a useful paper to cite for the layer map and the multimodal extension, not for the exact numbers.","headline":"Useful broad study of linear probes for MIA, but the headline AUC gains are inflated by picking the best layer on the same split used to report results.","tokens_in":17890,"tokens_out":2704,"would_cite":true,"duration_ms":24166,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LUMIA claims that training a linear probe on every layer of an LLM's internal activations detects training-set membership better than output-based attacks, with average AUC gains of 15.71% on text models and strong results on multimodal…","keywords":["membership inference attacks","linear probes","internal activations","large language models","multimodal models","training data detection","layer-wise analysis","privacy auditing"],"falsifier":"Take one of the paper's setups, such as Pythia 12B on a temporal-bias dataset, hold out a third partition never touched during training or layer selection, and recompute the AUC at the previously chosen layer $l^*$. If the average improvement over the best state-of-the-art method drops below the claimed 15.71% or the AUC>0.6 rate falls below 65.33%, the headline numbers are an artifact of layer selection on the test split.","tokens_in":16883,"feed_emoji":"🔍","tokens_out":7186,"duration_ms":54094,"temperature":0.7,"pith_summary":"LUMIA proposes that membership inference attacks on large language models can be mounted by training a linear probe on the internal activations at every layer, rather than relying only on model outputs. The paper claims this white-box strategy outperforms previous output-based and grey-box attacks on unimodal text models, with an average gain of 15.71% in AUC, reaching AUC above 0.6 in 65.33% of tested configurations. It extends the same approach to multimodal models, where probes on activations from the visual encoder alone often detect membership, with 85.9% of experiments exceeding the 0.6 AUC threshold. The paper also identifies where in the network membership information concentrates, which layers vary with dataset bias type, and how model size, deduplication, and task nature change vulnerability. If these results hold, internal activations give auditors a practical tool for assessing whether sensitive or copyrighted data was used in training.","feed_headline":"Probing hidden LLM layers beats membership attacks by 15.7% AUC","feed_subtitle":"LUMIA reads internal activations layer by layer, beating output-based detectors on text and image+text models.","key_machinery":"The central mechanism is the linear probe: a lightweight classifier (implemented as a small MLP) placed after a model layer, taking as input the average of the activation vectors $A_l(x_i)$ over all tokens of a sample at layer $l$, and trained to predict membership status $y_i$. Its output is an AUC per layer, and LUMIA's detector is the layer $l^*$ with maximum AUC. The probes reveal where membership information concentrates and turn internal activations into a per-layer membership score.","core_discovery":"The central discovery is that per-layer hidden states of an LLM separate training members from non-members with a simple linear classifier, and this separation is stronger than what can be read from loss, perplexity, or output probabilities. LUMIA trains one linear probe per layer on the mean activation vector across tokens, computes AUC for each layer, and takes the layer with the highest AUC as the detector. Across 14 textual and seven multimodal datasets and 15 model configurations, the paper reports that LUMIA improves on the best state-of-the-art AUC in 174 of 176 unimodal comparisons, that gains hold for both temporal-bias and n-gram-bias datasets, and that in multimodal models the visual encoder's activations alone often suffice to detect membership. The paper also finds that membership information is not spread uniformly: temporal-bias datasets leak most in early layers, n-gram-overlap datasets in deeper layers, and multimodal models in middle layers.","pith_inferences":["If the per-layer activation signal is as separable as reported, defenders could inject noise or truncate activations at the specific high-leakage layers the paper identifies, reducing MIA success without full retraining.","The same layer-wise probing setup could be applied to other modalities, such as audio or video, and to copyright-violation detection, since the method does not depend on the model's output format.","The reported gains may overestimate deployable performance because the same random split is used both to choose the best layer and to compute the final AUC; an independent held-out split would likely lower the headline numbers.","Because the method requires white-box access to internal activations, its realistic users are model owners and auditors rather than remote API attackers, so the practical threat model differs from black-box membership inference."],"forward_implications":["LUMIA outperforms the best prior method in 174 of 176 unimodal configurations, with an average AUC gain of 15.71% and AUC above 0.6 in 65.33% of cases.","In multimodal models, LUMIA reaches AUC above 0.6 in 85.9% of experiments, and activations from the visual encoder alone are often sufficient to detect membership.","Membership signal is concentrated in specific layers rather than spread uniformly, so the choice of layer matters for detection.","AUC increases with model size in most configurations, meaning larger models are more vulnerable to this kind of probing.","Deduplication reduces attack effectiveness on temporal-bias datasets, while no clear difference appears on n-gram-bias datasets."],"supporting_citations":[{"why":"It introduces linear probes as classifiers over hidden states, the mechanism LUMIA generalizes to every layer.","marker":"[2]"},{"why":"It supplies the Mimir benchmark and n-gram-bias datasets whose baselines LUMIA is compared against.","marker":"[9]"},{"why":"It provides the WikiMIA dataset and the Min-k% probability attack that serves as a state-of-the-art baseline.","marker":"[25]"},{"why":"It defines temporal-bias datasets and the heuristic baselines used for comparison.","marker":"[7]"},{"why":"It proposes document-level membership inference, the black-box baseline for Gutenberg and ArXiv-1-month datasets.","marker":"[21]"},{"why":"It applies linear probes to a single layer of an LLM, the direct white-box predecessor LUMIA extends and improves.","marker":"[17]"},{"why":"It introduces a multimodal membership inference benchmark based on model output, the comparison for the multimodal results.","marker":"[16]"},{"why":"It provides the Pythia model family in deduplicated and non-deduplicated versions used for size and deduplication analysis.","marker":"[3]"},{"why":"It provides the LLaVA-OneVision multimodal models whose text and visual encoders are probed.","marker":"[15]"}],"fun_headline_variants":["Layer-wise probes spot LLM training data with 15.7% AUC boost","LUMIA: reading LLM layers beats state-of-the-art MIA by 15.7 AUC","Probing each LLM layer finds membership leaks that outputs miss","Multimodal leaks: visual inputs help detect training data in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 80/20 random split used both to pick the best layer and to report the final AUC is an unbiased evaluation; if that shared split inflates the peak, the reported gains shrink when measured on data never used for layer selection.","fun_headline_variants_meta":{"raw":{"variants":["Layer-wise probes spot LLM training data with 15.7% AUC boost","LUMIA: reading LLM layers beats state-of-the-art MIA by 15.7 AUC","Probing each LLM layer finds membership leaks that outputs miss","Multimodal leaks: visual inputs help detect training data in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1384,"prompt_tokens":968,"completion_tokens":416,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":339}},"tokens_in":584,"tokens_out":416,"duration_ms":4372,"temperature":1.0,"reasoning_tokens":339,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:43:17.885352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the paper's setups, such as Pythia 12B on a temporal-bias dataset, hold out a third partition never touched during training or layer selection, and recompute the AUC at the previously chosen layer $l^*$. If the average improvement over the best state-of-the-art method drops below the claimed 15.71% or the AUC>0.6 rate falls below 65.33%, the headline numbers are an artifact of layer selection on the test split.","supporting_citations":[],"review_version":1}