{"id":"d2bedf13-2d0b-4e45-9aa3-44191897dc17","arxiv_id":"2505.03019","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM outputs fall sharply under small input perturbations when the input was memorized, and this sensitivity can be used to flag memorized content in black-box models.","lead":"This paper introduces PEARL, a black-box method that detects whether an LLM has memorized a piece of text by measuring how much its output degrades when the input is slightly corrupted. The approach could help identify training-data leakage and benchmark contamination in proprietary models whose training data is secret.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 3's sensitivity measure is internally inconsistent with Eq. 2's distance definition, and Table 4's reported values match neither the sign nor the index range of Eq. 3; every PEARL classification depends on this measure.","rationale":"The reader's weakest assumption is alpha calibration on presumed out-of-training data. I agree that this is a substantive limitation and the paper itself lists it in Section 5.2. However, the more load-bearing concern is that the sensitivity score itself, the quantity that alpha thresholds, is not well defined as written. Eq. 2 defines m as an average distance, with lower values meaning closer to the reference for NCD, while Eq. 3 takes max(m_j - m_{j+1}) and calls it a performance falloff. That is directionally wrong for a distance metric. Table 4 amplifies the problem: reported sensitivity values correspond to decreases in distance (e.g., #105, 0.36 = 0.68 - 0.32), which would mean perturbed outputs are closer to the reference than unperturbed ones, and some values (e.g., #105 again) use the m0 - m1 difference that Eq. 3's stated index range j in 1..k-1 excludes. This is not a mere typo in one row; the sensitivity measure is the core evidence for PSH and PEARL. Every classification in the paper reduces to sensitivity(X) > alpha, and the controlled Pythia experiment is an aggregate of exactly these per-sample decisions. If the implementation computes the literal Eq. 3, it may be selecting samples where perturbation improves fidelity, which is the opposite of the claimed memorization signal. The paper could still be right if the code uses a corrected sign and the equations are simply misprinted, but the current text does not permit the reader to verify this without accessing the repository. I therefore keep the reader's CONDITIONAL verdict rather than moving to REJECT, because a concrete re-derivation and code check could resolve the issue. This concern is partially aligned with the reader's stated 'ambiguous sensitivity implementation' in the rationale, but it differs from the alpha-calibration point emphasized as weakest_assumption, so agreement_with_reader is partial.","tokens_in":16980,"tokens_out":9536,"duration_ms":95916,"concrete_test":"Obtain the PEARL repository and re-implement Eq. 2 and Eq. 3 exactly; recompute the sensitivity column of Table 4 for all listed HumanEval and LBPP rows from the m columns under both sign conventions (m_j - m_{j+1} and m_{j+1} - m_j) and both index ranges (j=0..4 and j=1..4). If the reported Table 4 values match the literal Eq. 3 convention (as #105 does), the metric is mis-specified relative to the paper's prose. Then rerun the Pythia epoch experiment and the GPT-4o calibration with the corrected falloff direction; if the Pile/RefinedWeb separation disappears, the central claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central construct is the sensitivity score in Eq. 3: sensitivity(X) = max_{j in 1..k-1}(m(Y*_j) - m(Y*_{j+1})), where Eq. 2 defines m(Y*_k) as the mean distance between model outputs and the reference Y. For NCD, lower distance means higher fidelity. Under that definition, Eq. 3 is positive when more perturbation makes outputs closer to the reference, i.e., when performance improves, not when it falls off. The correct performance falloff would be m(Y*_{j+1}) - m(Y*_j). The paper's own Table 4 exposes the problem: for HumanEval sample #105, m = (0.68, 0.32, 0.42, 0.55, 0.32, 0.24) and the reported sensitivity is 0.36, which equals m0 - m1. That is a decrease in distance, i.e., an improvement under perturbation under Eq. 2's definition, and it also falls outside the j in 1..k-1 range specified in Eq. 3. Similar inconsistencies appear across the table: #103's reported 0.34 matches m3 - m4, not any value in the stated index range. Because the binary memorization decision is simply sensitivity(X) > alpha, any ambiguity or error in the sensitivity definition propagates into every Pythia and GPT-4o result, including the controlled validation in Section 4.2. The reader's alpha-calibration concern is real and is acknowledged in Section 5.2, but it is downstream of this issue: a threshold can only be calibrated and transferred once the sensitivity score itself is well-defined and directionally correct.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Perturbation Sensitivity Hypothesis (PSH): for a given model, task, and data point, if the model has memorized that data point, then its task performance will exhibit high sensitivity to small input perturbations. It then introduces PEARL, a black-box framework that bit-flips inputs, prompts the model, measures output fidelity to a reference via NCD or ROUGE, and flags a sample as memorized when a sensitivity score exceeds a threshold. The authors validate PSH on Pythia-410m by fine-tuning on Pile samples and comparing the proportion of flagged Pile vs. RefinedWeb samples, and present case studies on GPT-4o using HumanEval, Bible, NYT, and the LBPP dataset as a negative control for threshold calibration.","tokens_in":17352,"tokens_out":8158,"duration_ms":80011,"significance":"If the central claims held, PEARL would be a practically useful black-box memorization detector that requires neither model internals nor training-data knowledge. The paper has several strengths: it publicly releases artifacts, it includes a controlled Pythia experiment with fine-tuning epochs as a proxy for memorization, and it compares against the white-box ACR approach. However, the load-bearing sensitivity statistic is internally inconsistent with its own definition, the calibration reporting is self-contradictory, and the controlled experiment confounds membership with corpus identity and fine-tuning. As presented, the evidence does not support the strength of the claims, although the underlying hypothesis is plausible and the work is potentially salvageable.","major_comments":[{"comment":"The sensitivity definition is internally inconsistent. Eq. (2) defines m(Y*_k) as the mean distance between model outputs and the reference Y, so smaller m means closer to the reference and thus better fidelity. Under this definition, a performance falloff under perturbation should be m(Y*_{j+1}) - m(Y*_j), not m(Y*_j) - m(Y*_{j+1}). Eq. (3) therefore measures improvement (decreasing distance) rather than degradation. The reported values in Table 4 compound the problem: for HumanEval/105, m = (0.68, 0.32, 0.42, 0.55, 0.32, 0.24) and the reported sensitivity 0.36 equals m(Y*_0) - m(Y*_1), a term outside the stated index range j in {1, ..., k-1}, and it has the opposite sign of the correct falloff. Similar discrepancies appear for rows #103, #107, and others. Since the binary decision is simply sensitivity(X) > alpha, this error propagates into every Pythia and GPT-4o classification in the paper. If the authors instead intend m to be a similarity score, then Eq. (2) and the text describing NCD as a distance are mislabeled, and the table should be reconciled with a precise, self-consistent formula.","section":"Section 3.3, Eq. (3) and Table 4"},{"comment":"The calibration and the reporting are inconsistent. LBPP is used as the assumed-negative dataset for choosing alpha, yet Table 4, which lists cases 'identified as memorized' at alpha = 0.29, includes LBPP rows #17 (sensitivity 0.32) and #58 (sensitivity 0.29). Under the stated rule sensitivity > alpha, #58 should not be flagged at all, and #17 is a false positive on the calibration set. This indicates either that the threshold was not chosen to minimize false positives, that the table mistakenly includes calibration data, or that the reported sensitivities were computed with a different procedure. The authors must clarify and recompute the affected results.","section":"Section 4.4 and Annex 6, Table 4"},{"comment":"The positive and negative sets are not matched. The positive set consists of Pile samples on which the model is fine-tuned for up to 10 epochs, while the negative set is RefinedWeb, a different corpus that is never fine-tuned. The observed increase in flagged Pile samples over epochs could reflect overfitting to the exact input strings or distribution shift between corpora rather than memorization as defined by membership advantage. A stronger controlled experiment would use held-out samples from the same corpus (e.g., Pile samples not fine-tuned) as negatives, or at least a matched corpus, and would independently verify memorization via a white-box metric such as extraction likelihood or membership advantage. The ACR comparison in Annex 1 is a step in this direction but covers only 100 quotes and is not integrated into the main validation.","section":"Section 4.2, Pythia validation"},{"comment":"The paper acknowledges in Section 5.2 that PEARL's sensitivity threshold calibration requires access to data known to be outside the model's training set. For GPT-4o, alpha is chosen on LBPP, which is assumed to be post-release and therefore non-training data, and is then transferred to Bible, HumanEval, and NYT. This is a decision-boundary fit rather than a derived threshold, and all downstream membership claims inherit any error in that assumption or in the transfer across content types. The paper should report sensitivity of the conclusions to alpha values (Tables 2 and 3 partially do this) and should justify the transfer with an analysis of sensitivity distributions rather than only FPR curves. For the GPT-4o case studies, this limitation is load-bearing because the reported numbers of memorized instances depend directly on the calibrated threshold.","section":"Section 5.2 and Section 4.4, alpha calibration"}],"minor_comments":[{"comment":"Table 1 describes k as the 'percentage of token modified in the input', but Section 3.1 states that the perturbation function 'consists in flipping k bits'; these two descriptions are inconsistent and should be reconciled.","section":"Table 1"},{"comment":"The bit-flip perturbation operates on the binary encoding of the input text and decodes back to text; the manuscript should specify how invalid UTF-8 byte sequences are handled during decoding, since arbitrary bit flips can produce invalid sequences that may affect the results.","section":"Section 3.1"},{"comment":"The text claims 'a statistically significant difference between the medians of sensitivity values' but does not report the statistical test used, the test statistic, or the p-value; this should be added.","section":"Section 4.5"},{"comment":"The caption and legend of Figure 9 appear garbled, with unbalanced parentheses in '(Text/Code completion) (Text/Code summary'; this should be fixed.","section":"Figure 9"},{"comment":"Section 5.3 refers to 'our findings from GPT-4 and Pythia models', but the experiments use GPT-4o; the terminology should be aligned throughout the paper.","section":"Section 5.3"},{"comment":"Annexes 3 and 4 are referenced in the text, but their figures and content are missing from the manuscript as provided; the FPR-vs-alpha curves for LBPP on completion and summarization tasks should be included.","section":"Annexes 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The internal inconsistency in Eq. (3) and Table 4 is serious enough that the experimental results cannot be evaluated as currently presented. The core hypothesis is plausible and the Pythia experiment is a reasonable start, but the paper needs a corrected definition of sensitivity, a recomputation of results, and a more careful treatment of the calibration and confounding issues before it can be considered for publication. The framing of NYT as 'known to be part of the training set' should also be softened to 'assumed' unless external evidence is provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is the problem: Eq. 3 defines sensitivity as max(m(Y*_j) - m(Y*_{j+1})), but Eq. 2 defines m as mean distance, where lower is better. So a positive value means the model output moved closer to the reference after perturbation – performance improved. That is the opposite of the 'falloff' the text describes. Table 4 confirms it: for HumanEval sample #105, the reported sensitivity 0.36 is exactly m0 - m1, a decrease in distance. The index range also doesn't match Eq. 3's j in 1..k-1. This is an internal contradiction in the central statistic, and every classification in the paper – Pythia and GPT-4o – rests on it.\n\nThere is a real contribution here, though. The idea of detecting memorization through input perturbation is sensible, and the Pythia controlled experiment is well designed: fine-tuning on Pile raises the flag rate while RefinedWeb stays flat. That pattern is the right shape for a memorization signal, and the comparison to ACR and the effort to calibrate on presumed-negative data show a genuine attempt at validation. The paper is clearly written and the artifacts are promised.\n\nBut the sign error is load-bearing. The Pythia result may be recoverable after flipping the sign and re-running, but as written, the validation is hard to trust. The reader's alpha-calibration concern is real and the paper admits it in Section 5.2; a threshold on an assumed-negative set is a genuine limitation for black-box use. I'd view that as secondary, though – a correctly signed measure is a prerequisite before any threshold discussion.\n\nI also note the task dependence: sensitivity patterns largely vanish under summarization, which limits the scope. That's not a flaw per se, but it matters for the reported claims about NYT and Bible memorization.\n\nBottom line: the topic is important and the controlled Pythia setup is a good idea, but this paper needs major revision. The sensitivity statistic must be corrected and all experiments re-run. If the results survive, this could be a useful black-box detection tool. As written, I wouldn't cite it or rely on its claims. I'd still send it to peer review, because the flaw is identifiable and fixable, and the underlying question deserves attention.","headline":"The paper's sensitivity measure is sign-reversed relative to its own distance metric, so the core statistic likely measures improvement, not falloff; worth a serious look but needs major revision.","tokens_in":17899,"tokens_out":3851,"would_cite":false,"duration_ms":40074,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small input tweaks expose which texts a language model has memorized, and PEARL uses that fragility as a black-box detector.","keywords":["LLM memorization","input perturbation","black-box detection","membership inference","perturbation sensitivity hypothesis","data contamination","Pythia","GPT-4o"],"falsifier":"Run PEARL with the same alpha on a set of stylistically distinctive, provably unseen texts (for example, poems or legal opinions published after the model's release cutoff) alongside the datasets the paper flags; if the unseen texts are flagged at rates close to HumanEval or Pile, then the sensitivity signal is tracking style or uniqueness rather than membership in training data.","tokens_in":16805,"feed_emoji":"🔍","tokens_out":7553,"duration_ms":66543,"temperature":0.7,"pith_summary":"This paper sets out to prove the Perturbation Sensitivity Hypothesis: when a language model has memorized a specific piece of text, small perturbations to that text should cause a sharp drop in task performance, whereas genuinely learned patterns should degrade smoothly. The authors build PEARL, a black-box detector that flips a few bits in the input, prompts the model repeatedly, and measures the largest falloff in output similarity as perturbation intensity rises. On the open Pythia model they show that text from its training set (Pile) is flagged as memorized at far higher rates than text outside it (RefinedWeb), and that the gap widens as the model is fine-tuned on the training data. On GPT-4o they use the same detector to argue that HumanEval code and Bible verses are memorized in part, while New York Times articles show weaker and sparser signals. The paper's contribution is a practical, internals-free way to tell regurgitation from interpolation, with the caveat that the detection threshold must be calibrated on data known to be outside the model's training set.","feed_headline":"Tiny text tweaks reveal what an LLM memorized","feed_subtitle":"PEARL flags texts whose answers collapse under bit flips, separating memorization from true generalization without model access.","key_machinery":"The machinery is the perturbation-sensitivity measurement. PEARL encodes an input text to bits, flips k random bits for k in {0,1,2,3,4,5}, decodes back to text, and prompts the model with each perturbed version ten times. It scores each output against a reference using Normalized Compression Distance for completion tasks and ROUGE-L for summarization, then defines sensitivity(X) = max_j (m(Y*_j) - m(Y*_{j+1})), the largest drop in mean distance between consecutive perturbation intensities. A sample is called memorized when sensitivity(X) > alpha, where alpha is chosen from the false-positive curve on a dataset assumed to be outside the training set (RefinedWeb for Pythia, LBPP for GPT-4o). The sensitivity statistic is the load-bearing object: it converts the untestable notion of 'stored verbatim' into a single comparable number.","core_discovery":"PEARL's central claim is that memorization leaves a measurable behavioral fingerprint: if a model has memorized an input, its output quality collapses when the input is lightly perturbed, because the exact stored sequence is what triggers the correct answer. Formally, the paper defines sensitivity(X) as the maximum difference in mean output distance between consecutive perturbation levels, and classifies X as memorized when that maximum exceeds a threshold alpha. The Pythia experiments show the proportion of Pile samples flagged as memorized climbing from under 5% at epoch 0 to over 20% at epoch 10 at alpha=0.2, while RefinedWeb stays near 2%, and the gap persists at stricter thresholds. For GPT-4o, the same procedure flags 60 of 100 HumanEval samples at alpha=0.14, 42 of 100 Bible samples, and only 5 of 100 NYT samples, which the authors read as evidence that memorization concentrates in stylistically distinctive or unique content. The paper also reports that the signal largely disappears under summarization tasks, so the hypothesis is explicitly task-dependent: it reveals memorization best when the task demands exact reproduction.","pith_inferences":["Beyond the paper: if the sensitivity signature is a stable property of stored text, the same technique could be extended to images or audio by perturbing pixels or samples, but the threshold-calibration problem would become harder because 'outside training data' is even less knowable.","Beyond the paper: the bit-flip perturbation changes surface characters, so PEARL may be detecting memorization of exact surface form rather than memorized content; a testable extension is to compare bit flips against synonym substitutions to see whether semantically-equivalent perturbations also expose memorization.","Beyond the paper: the task-dependence result suggests PSH is really measuring 'retrieval fragility'—the model's reliance on an exact trigger—which could be used as a general probe for how template-like a model's behavior is, not just as a memorization label.","Beyond the paper: a practical implication the authors leave implicit is that PEARL's calibration requirement could be satisfied in production by continuously sampling dated, post-cutoff text as a live negative set, making the detector usable on models whose training data is updated."],"forward_implications":["Closed models can now be audited for memorization without parameter access or training-set disclosure: only prompts and generated outputs are needed.","The alpha threshold gives practitioners a dial: a loose threshold catches more candidate memorization but admits more false positives, so the method doubles as a triage tool rather than a binary verdict.","Fine-tuning on the same data makes the sensitivity signal stronger, so PEARL can track how memorization accumulates over training epochs.","Because the signal is task-dependent, evaluations that use completion-style benchmarks are more exposed to memorization artifacts than summarization-style benchmarks; benchmark designers should prefer tasks that do not reward exact reproduction.","In applied settings, the method can supply supporting evidence in data-contamination or copyright disputes, as the paper does for HumanEval and NYT, while stopping short of a standalone legal proof."],"supporting_citations":[{"why":"Supplies the open-source Pythia model suite and its deduplicated training-set variant used for controlled validation.","marker":"(Biderman et al., 2023)"},{"why":"Provides The Pile, the training corpus whose sampled texts form the positive set for Pythia experiments.","marker":"(Gao et al., 2020)"},{"why":"Provides RefinedWeb, the disjoint negative set used to calibrate alpha and estimate false positives.","marker":"(Penedo et al., 2023)"},{"why":"Provides LBPP, the post-release dataset used to calibrate the GPT-4o threshold and support the claim that it is outside training.","marker":"(Matton et al., 2024)"},{"why":"Provides HumanEval, the code-generation benchmark whose samples PEARL flags as memorized in GPT-4o.","marker":"(Chen et al., 2021)"},{"why":"Defines Adversarial Compression Ratio, the white-box comparison whose detection results PEARL partially overlaps on FamousQuotes.","marker":"(Schwarzschild et al., 2024)"},{"why":"Establishes the verbatim extraction phenomenon that motivates the memorization-detection problem.","marker":"(Carlini et al., 2021)"},{"why":"Supports the paper's task-dependency finding by showing memorization is localized and task-dependent.","marker":"(Dankers & Titov, 2024)"}],"fun_headline_variants":["Perturb inputs to catch LLMs regurgitating training data","Sensitivity to tweaks reveals LLM memorization","PEARL: bit flips expose memorized text in LLMs","Tiny tweaks separate LLM memorization from generalization","Input perturbation test spots LLM memorized snippets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything PEARL labels as memorized inherits the assumption that the threshold alpha calibrated on one out-of-training dataset (RefinedWeb for Pythia, LBPP for GPT-4o) transfers unchanged to the content being tested.","fun_headline_variants_meta":{"raw":{"variants":["Perturb inputs to catch LLMs regurgitating training data","Sensitivity to tweaks reveals LLM memorization","PEARL: bit flips expose memorized text in LLMs","Tiny tweaks separate LLM memorization from generalization","Input perturbation test spots LLM memorized snippets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1514,"prompt_tokens":985,"completion_tokens":529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":447}},"tokens_in":601,"tokens_out":529,"duration_ms":5049,"temperature":1.0,"reasoning_tokens":447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:01:29.411682+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PEARL with the same alpha on a set of stylistically distinctive, provably unseen texts (for example, poems or legal opinions published after the model's release cutoff) alongside the datasets the paper flags; if the unseen texts are flagged at rates close to HumanEval or Pile, then the sensitivity signal is tracking style or uniqueness rather than membership in training data.","supporting_citations":[],"review_version":1}