{"id":"2c0c02a4-78a0-4133-b0c9-734aba43c508","arxiv_id":"2412.01014","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Neuron activation patterns in Pythia 1B separate memorized from non-memorized tokens well enough to train probes with claimed 99.9% accuracy, and activation editing is said to disable memorization.","lead":"This paper claims that memorized tokens in a large language model can be detected almost perfectly by reading neuron activations, not output probabilities. The author applies this to Pythia 1B, reporting 99.9% probe accuracy and claiming that editing these activations suppresses memorization.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 99.9% detection claim rests on labels generated by thresholding the very activation used to train and validate the probes; unless that threshold is anchored to independent ground truth, the reported accuracy is self-consistency rather than memorization detection.","rationale":"The reader's weakest-assumption analysis correctly identifies the label-generation procedure as the load-bearing point. My stress test agrees: the automated labels are generated by thresholding activation 1857, and the probe validation is drawn from the same label-generating pipeline, so the 99.9% number is largely tautological. The manual 10k-token set is better, but it is small and its 'memorized' designation is based on the model's own confidence rather than on verified membership in Pythia's training data. Section 8's certainty activation strengthens the concern that the learned feature may be confidence rather than memorization. The intervention result is also fitted: a genetic algorithm optimizes the exact loss gap that is later reported as success, so it cannot independently confirm the causal claim. Given the lack of released code/data and the circular validation path, the reader's low-confidence REJECT is appropriate. My read does not move that verdict; it sharpens the reason.","tokens_in":14540,"tokens_out":4826,"duration_ms":47292,"concrete_test":"Construct an independent ground-truth benchmark for Pythia 1B from its actual training corpus: for a set of contexts, label a token as memorized iff the exact continuation appears in the training data and is extractable under a k-eidetic/membership criterion (e.g., Carlini et al., 2023), and label a distribution-matched, non-training continuation as not memorized, without using activation 1857. Run the trained probe on this benchmark; if balanced accuracy on distribution-matched pairs is far below 99.9%, or near chance, the threshold-derived labels are the source of the claimed accuracy and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—probes classify memorization with 99.9% accuracy—is not supported by the reported evaluation. In Part 2 (Section 3.1), the one million memorized/not-memorized labels used to train the probes are produced by thresholding activation 1857 in MLP layer 10, the same activation family the probes subsequently learn from. Test accuracy in Figures 7 and 8 is reported on a validation set 'derived from the same dataset used for training the probe,' so high accuracy only demonstrates that a probe can recover a thresholding rule already present in its labels. This is circular. The only non-circular evidence is the 10,000 manually labeled tokens from 100 curated samples, but those labels were selected by the author's judgment that outputs were 'very high confidence' on the correct prediction, which could encode certainty, source genre, or template structure rather than memorization. Section 8 confirms that a separate activation (1668) encodes certainty and separates memorization/repetition tokens, so activation separation is not by itself evidence of memorization. If the threshold labels track confidence or distribution rather than true memorization, every downstream result—including the Section 7 intervention, whose genetic algorithm optimizes exactly the loss gap used to report success—is undermined. The paper's own Section 11 admits the probes may capture additional internal processes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to detect memorization in LLMs by training linear and two-layer classification probes on neuron activations, after first identifying highly separable activations between manually labeled memorized and not-memorized tokens. The authors report 99.9% classification accuracy, extend the method to repetition detection, and show that intervening on probe directions can suppress memorization while leaving other behavior largely intact. The method is demonstrated on Pythia 1B, with large-scale labeling of SlimPajama tokens and a series of perturbation and interpretability analyses. The central claim is that activation 1857 in MLP layer 10, and the probes trained on it, both detect and causally control memorization.","tokens_in":14924,"tokens_out":2660,"duration_ms":25584,"significance":"If the central claims were valid, the paper would offer a practical and interpretable tool for detecting and intervening on memorization in LLMs, with potential applications to data curation and evaluation integrity. The idea of using activation-based separation to label large corpora is interesting and could be valuable. However, the paper does not establish that the detected signal is memorization rather than confidence, text genre, or template structure. The main accuracy number is computed against labels generated by thresholding the very activation used to train the probes, so the reported 99.9% accuracy is self-consistency rather than validation against independent ground truth. The intervention results are also measured against an objective that the genetic algorithm directly optimizes. Without independent ground-truth labels and out-of-distribution validation, the paper's contributions are not supported as stated. The paper also contains no code or data release, which limits reproducibility and further weakens the empirical claims.","major_comments":[{"comment":"The one million memorized/not-memorized labels used to train the probes are generated by thresholding activation 1857 in MLP layer 10, and the probes are then trained and evaluated on the same labeling procedure. The validation set is described as 'derived from the same dataset used for training the probe,' so the 99.9% accuracy reported in Figures 7 and 8 demonstrates only that a probe can recover the thresholding rule already present in its labels. This is a circular evaluation and does not establish detection of an independent ground truth. The authors should validate on a held-out set of tokens whose memorization status is determined by an independent criterion, such as membership in known training data or human-verified verbatim extraction.","section":"§3.1 (Part 2, step 2) and §4"},{"comment":"The initial 100 memorized samples are selected by the author's judgment that the model produces 'very high confidence' predictions on the correct token. Section 8 shows that a separate activation (1668) encodes certainty and separates memorization, repetition, and other high-confidence mechanisms. This raises the concern that the manually labeled ground truth tracks confidence, text genre, or template structure rather than memorization specifically. The authors should demonstrate that the high-confidence criterion is a valid proxy for memorization by showing, for example, that the selected samples appear verbatim in the training data and that low-confidence but memorized samples are rare or handled explicitly.","section":"§3.1 (Part 1, step 1) and §8"},{"comment":"The intervention experiment optimizes, via a genetic algorithm, the objective of raising the loss of memorized sequences to match that of not-memorized sequences while keeping not-memorized loss unchanged. Figure 17 then reports that this objective is achieved. This is circular: the intervention is tuned to produce exactly the reported effect. The claim that memorization is 'completely disabled while keeping all other mechanisms intact' is not supported by loss measurements alone, and the only qualitative example is a single anecdotal case. The authors should evaluate intervention effects on independent downstream tasks, perplexity on diverse corpora, and specific memorized samples not used during optimization, and they should report variability across samples.","section":"§7 and Figure 17"},{"comment":"The manual analysis of 1,000 high-loss sequences classified as memorized reports 10.6% 'Calls to action' and 7.4% 'Disclaimers' as misclassifications, plus 2.8% 'Others' where no reason could be ascertained. This directly contradicts the claim of near-perfect memorization detection, because these categories are not memorized in the traditional sense. The percentages are presented without counts, error bars, or inter-annotator agreement. The authors should either quantify the probe's precision on a balanced, independently labeled sample or revise the claim to acknowledge systematic false positives on templated language.","section":"§6 and Table 1"},{"comment":"The Limitations section concedes that 'the mechanisms identified by our probes might encompass more than just memorization or repetition' and that the probes may capture additional internal processes. This admission, combined with the circular labeling and the certainty mechanism documented in Section 8, means that the paper does not establish that the probes isolate memorization as opposed to a broader class of high-confidence or template-based predictions. The central conclusion should be tempered or the probes should be validated against a ground truth that distinguishes memorization from these other processes.","section":"§11"}],"minor_comments":[{"comment":"The notation is inconsistent between 'neuron 6181' and 'activation 6181' and between 'activation 1668' and 'neuron 1668'; the authors should choose one term and use it consistently.","section":"Throughout"},{"comment":"The equation for Cohen's d contains a typesetting artifact ('Cohen ′s d') that should be fixed, and the pooled standard deviation formula should be checked for correctness.","section":"Equation (1)"},{"comment":"The curated dataset evaluation is described as balanced and same-distribution, but no details are given about its size, composition, or how the same-distribution requirement was operationalized. This makes it difficult to interpret the near-100% accuracy claim.","section":"§4 and Figure 9"},{"comment":"The threshold for 'memorized' activation values is not specified; the paper should report the threshold value, the fraction of tokens that pass it, and the stability of the resulting labels to threshold choice.","section":"§3.1 (Part 2, step 2)"},{"comment":"The genetic algorithm for choosing intervention scales is described only briefly; the authors should specify the objective function, the search space, the number of generations, and whether the same intervention parameters generalize to held-out sequences.","section":"§7"},{"comment":"Several references are cited with incomplete bibliographic details (e.g., Carlini et al. 2020, Meng et al. 2022, Soboleva et al. 2023), and the Durmus et al. reference is not in a standard archival format; these should be completed.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's main empirical claim rests on a circular labeling procedure: the activation used to define memorization labels is the same activation family the probes learn from, so the 99.9% accuracy is not evidence of detecting an independent phenomenon. The intervention result is similarly circular because the genetic algorithm optimizes the exact loss gap that is reported as success. While the underlying idea of activation-based probing is interesting, the current manuscript would require a fundamentally new experimental design with independent ground-truth labels and out-of-distribution validation to support its claims. Given the scope of the rework needed, I do not see this as a minor or major revision of the present paper; I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the memorization probe paper. The headline: the 99.9% accuracy claim is circular. The one-million-token labels are generated by thresholding activation 1857 in MLP layer 10, and the probes are then trained on those labels and evaluated on a validation set derived from the same dataset. So the high accuracy mostly shows the probe can reproduce the thresholding rule, not that it detects memorization as an independent property. The stress-test note is right.\n\nThat said, there is a real empirical nugget here. A single MLP neuron (6181, layer 10) separates the author's curated memorized tokens from similar not-memorized tokens at about 99% accuracy, and the paper's exploration of neuron 1668 as a certainty signal is a nice observation. The perturbation tests in Section 6.0.1 are thoughtful and suggest the activation is not trivially gameable by surface prompts. The repetition extension is plausible.\n\nSoft spots beyond the circularity: the curated 10,000 labels are chosen by the author's judgment of high-confidence outputs, so they may encode certainty or genre rather than genuine memorization. The intervention in Section 7 uses a genetic algorithm that optimizes exactly the loss gap reported as success, so the intervention claim is also fitted. No code, data, or error bars are provided. The paper's own Section 11 concedes the probes may capture more than memorization.\n\nWho is this for? Someone interested in mechanistic interpretability of Pythia might find the activation observations worth following up. But as a detector of memorization, the paper doesn't establish its central claim. I would not cite it as evidence.\n\nVerdict: I'd send it to peer review, but with the expectation of heavy revision. A referee should ask for independent ground-truth labels (e.g., from known duplicated training data) and for the release of code and data. If the finding survives that check, it's a useful result. Right now it's an intriguing observation with a circular validation.","headline":"Circular labeling undermines the 99.9% memorization detection claim, but the underlying single-neuron observation is worth a closer look.","tokens_in":15339,"tokens_out":2312,"would_cite":false,"duration_ms":20413,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single neuron activation in a large language model marks memorized text with 99.9% accuracy, and suppressing that same direction disables the memorization.","keywords":["memorization detection","neuron activations","classification probes","intervention","repetition","certainty","interpretability","Pythia 1B"],"falsifier":"Run the trained probe on a held-out set where memorized and not-memorized texts are matched for genre, length, and style and labelled by an independent method such as known contamination of the training set; if accuracy falls well below 99.9%—or if intervening on the activation also suppresses ordinary high-confidence facts, quotes, and disclaimers—the feature is encoding certainty or genre, not memorization.","tokens_in":14354,"feed_emoji":"🧠","tokens_out":5997,"duration_ms":50344,"temperature":0.7,"pith_summary":"The paper claims that memorization in a large language model is not a diffuse output property but a specific, localizable internal mechanism: a single neuron activation in the MLP at layer 10 separates memorized from non-memorized tokens so cleanly that probes trained on it classify with 99.9% accuracy. The same analytical pipeline detects repetition at near-perfect accuracy, including non-verbatim repetition. Intervening along the discovered feature direction—subtracting a scaled, squared projection of the activations—reportedly disables memorization while leaving other abilities intact, and substantially weakens repetition. If true, this gives a practical, interpretable handle for evaluating models, cleaning training data, and controlling what a model recites.","feed_headline":"Probes spot memorized text in LLMs at 99.9%","feed_subtitle":"A single neuron activation flags memorized tokens and, when suppressed, stops the model from reciting them.","key_machinery":"The central object is a classification probe trained on neuron activations, together with a labeling pipeline that makes large-scale training possible. The pipeline starts with a small, balanced corpus of 100 memorized and 100 matched non-memorized samples, uses Cohen's d to rank activations, picks activation 1857 in the MLP at layer 10 as a reliable labeler, and thresholds its values over at least ten consecutive tokens to generate one million labels per class from SlimPajama. Probes are linear or two-layer ReLU classifiers over activations at each layer. For intervention, the same probe weights define a direction in activation space; subtracting $\\alpha$ times the squared projection of the activation vector onto that normalized direction attenuates the mechanism, with $\\alpha$ chosen per layer by a genetic algorithm.","core_discovery":"On its own terms, the paper demonstrates that the Pythia 1B model develops a dedicated internal feature for memorized text. Using 100 manually selected memorized samples and matched non-memorized samples, the author ranks neuron activations by Cohen's d and finds that activation 1857 in MLP layer 10 separates the two classes almost perfectly. That activation is then used to label one million memorized and one million non-memorized tokens from SlimPajama, and linear and two-layer probes trained on these labels reach 99.9% accuracy, including on a curated same-distribution test set. The same procedure yields near-perfect detection of repetition. Finally, subtracting a scaled squared projection of the activations onto the probe weight direction pushes the loss of memorized sequences up to the level of non-memorized ones, which the paper reads as evidence that the detected feature is causal, not merely correlational.","pith_inferences":["The paper only demonstrates the direction in Pythia 1B; if the feature is architecture-specific, each new model would need its own discovery pass rather than a transferable probe.","The 'tug-of-war' between memorization and repetition probes suggests both mechanisms compete for a shared certainty-related resource; an intervention on one may strengthen the other, which the paper observes but does not fully explain.","Because calls to action and disclaimers are flagged as memorized and then manually set aside, a practical data-cleaning use of the probe would need a second-stage filter to avoid removing boilerplate rather than memorized content.","A direct extension would be to test the same probe on factual-knowledge sentences: if suppressing memorization also degrades accurate fact recall, the line between memorized data and learned knowledge is not as clean as the binary labels assume."],"forward_implications":["Memorized tokens in Pythia 1B can be identified at the token level with 99.9% accuracy before the model outputs anything, using internal activations alone.","The same activation-based pipeline detects repetition with near-perfect accuracy, including repetition that is not word-for-word identical, such as URLs derived from titles.","Intervening on the memorization direction raises the loss of memorized sequences to match non-memorized ones while leaving non-memorized loss essentially unchanged; repetition is weakened but not fully removed.","The method supports labeling millions of tokens automatically, which the paper proposes as a route to better training-data selection and more reliable evaluation.","A separate activation, 1668, encodes certainty and correlates with top-1 prediction probability, suggesting a general axis linking memorization, repetition, and confidence."],"supporting_citations":[{"why":"Supplies the output-probability and loss-based detection approach that the paper argues is confounded and that it contrasts with activation-based detection.","marker":"Carlini et al. (2020)"},{"why":"Provides the Pythia 1B model whose activations are analyzed throughout the study.","marker":"Biderman et al. (2023b)"},{"why":"Supplies the SlimPajama dataset used to generate one million memorized and one million non-memorized token labels.","marker":"Soboleva et al. (2023)"},{"why":"Motivates choosing MLP activations by identifying mid-layer feed-forward networks as storage sites for factual associations.","marker":"Meng et al. (2022)"},{"why":"Supports the claim that memorization is distributed across many layers, which motivates the per-layer genetic-algorithm intervention.","marker":"Maini et al. (2023)"},{"why":"Provides evidence that verbatim memorization is intertwined with general language modeling, contextualizing the intervention results.","marker":"Huang et al. (2024)"},{"why":"Supplies the caution that feature detection need not imply feature intervention, which the paper's causal-intervention claim explicitly addresses.","marker":"Durmus et al. (2024)"}],"fun_headline_variants":["One neuron spots memorized text in LLMs at 99.9%","Neuron activation IDs memorized tokens with 99.9% accuracy","Detect and suppress LLM memorization via single neuron","One neuron knows when LLM recites memorized text","99.9% accurate: neuron activation flags memorized tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the manually chosen memorized examples and the one-million-token labels produced by thresholding one activation genuinely capture memorization rather than confidence, genre, or the author's selection bias.","fun_headline_variants_meta":{"raw":{"variants":["One neuron spots memorized text in LLMs at 99.9%","Neuron activation IDs memorized tokens with 99.9% accuracy","Detect and suppress LLM memorization via single neuron","One neuron knows when LLM recites memorized text","99.9% accurate: neuron activation flags memorized tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3453,"prompt_tokens":913,"completion_tokens":2540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":2451}},"tokens_in":529,"tokens_out":2540,"duration_ms":17736,"temperature":1.0,"reasoning_tokens":2451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:45:55.407867+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained probe on a held-out set where memorized and not-memorized texts are matched for genre, length, and style and labelled by an independent method such as known contamination of the training set; if accuracy falls well below 99.9%—or if intervening on the activation also suppresses ordinary high-confidence facts, quotes, and disclaimers—the feature is encoding certainty or genre, not memorization.","supporting_citations":[],"review_version":1}