{"id":"aef4e454-48bb-4e55-aa23-8ed621b09d52","arxiv_id":"2506.00688","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Existing LLM unlearning evaluations are inconclusive: they can inject new information, depend heavily on task format, and rely on spurious correlations.","lead":"This paper argues that standard tests for whether large language models have truly unlearned sensitive data are unreliable. It shows evidence that some tests re-teach the model, that results flip depending on output format, and proposes two principles to make future evaluations conclusive.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The bit-capacity argument in §4.2 conflates per-question answer entropy with a single shared prefix; the information-injection claim for input-space attacks is not established, though task-format dependence still supports the broader conclusion.","rationale":"The reader's conditional verdict is appropriate. The strongest independent evidence for 'inconclusive' is the task-format sensitivity in §4.3: Enhanced GCG accuracy drops from 53.9% (letter) to 26.3% (text), finetuning attacks behave differently across MCQ scoring rules, and ACR success rates reorder unlearning methods across CHOOSE/OPTION/GENERATE. These results alone support the paper's central claim. However, the information-injection pillar in §4.2 rests on a bit-capacity comparison that is mathematically invalid: a single prefix shared across test questions cannot carry per-question answer information, and the comparison with N·log2(4)·accuracy conflates answer entropy with prefix information. The follow-up experiment is also consistent with elicitation. This does not overturn the central claim, but it does mean the paper should either repair or heavily qualify the injection argument before claiming that evaluations can re-teach the model during testing. Hence no change to the reader's CONDITIONAL verdict; our concern reinforces it.","tokens_in":13885,"tokens_out":9020,"duration_ms":95526,"concrete_test":"Run an Enhanced GCG label-permutation control: optimize the same 100-token prefix on the same seven WMDP-Bio training samples with the four answer choices randomly permuted, keeping the base model and all hyperparameters fixed, then evaluate the prefix on the held-out WMDP-Bio test set using maximum-letter decoding. If the permuted-label prefix no longer lifts accuracy from ~39.9% toward ~53.8% (or tracks the permuted key), the original gain is elicitation of latent knowledge, not injection; if it still transfers, the injection mechanism is confirmed. In either case, the paper should replace the §4.2 bit-count comparison with this controlled measure, since a single shared prefix cannot encode per-question test answers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the §4.2 information-injection argument. The paper compares the ~1500-bit capacity of a 100-token prefix with the ~1430 bits it claims are 'needed' for 55% accuracy on WMDP-Bio (1300×log2(4)×0.55). This is the wrong information measure: the prefix is a single string, optimized on fewer than ten samples and then fixed across all test questions. A fixed prefix cannot encode per-question answer identities, so per-question entropy of the test set is irrelevant; the mutual information between a fixed prefix and the answer labels is bounded by the information in the training samples, not by test-set entropy. The supporting experiment (optimizing a prefix on an overfit model raises base accuracy from 39.9% to 53.8%) is also ambiguous: the same gain would occur if the prefix elicited latent knowledge from the base model rather than injecting new facts. Consequently, the claim that input-space attacks 're-teach the model during testing' is not supported by the evidence, and the 'minimal information injection' principle loses one of its two empirical pillars. The broader 'inconclusive' conclusion still stands via task-format dependence in §4.3 and Figure 7.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that current adversarial LLM unlearning evaluations are inconclusive, identifying three failure modes: (i) finetuning and input-space attacks can inject new information into the model rather than elicit forgotten knowledge; (ii) evaluation outcomes depend heavily on the task format (MCQ letter probability, MCQ text probability, open-ended generation); and (iii) benchmarks such as TOFU contain spurious correlations that may allow relearning of the forget set from retain data. The authors propose two principles for future evaluations, minimal information injection and downstream task awareness, and support their critique with experiments on WMDP and TOFU using RMU and NPO unlearned models, Enhanced GCG attacks, and adversarial compression ratio (ACR) memorization detectors.","tokens_in":14158,"tokens_out":9392,"duration_ms":88009,"significance":"The task-format-dependence results, especially the ACR comparisons in Figure 7 and the Enhanced GCG open-ended versus MCQ contrast in Figure 5, are a useful and likely robust contribution to the unlearning evaluation literature. If the spurious-correlation claim about TOFU holds, it would complicate the interpretation of a widely used benchmark. The paper also proposes concrete reporting recommendations that could improve future evaluations. However, the information-injection claim for input-space attacks is not currently established: the bit-capacity argument in Section 4.2 is invalid as stated, and the supporting prefix-transfer experiment is ambiguous. The broader 'inconclusive' thesis is defensible through the task-dependence evidence alone, but the paper's first stated pitfall needs either a rigorous repair or a narrower restatement.","major_comments":[{"comment":"The computation 1300 × log2(4) × 0.55 ≈ 1430 bits is not the information required for a single shared 100-token prefix to reach 55% accuracy on WMDP-Bio. Because the prefix is fixed across all test questions, it cannot store per-question answer identities; the relevant quantity is the mutual information between the prefix and the labels achievable by the optimization procedure, which is bounded by the size of the optimization set (fewer than ten samples) and the complexity of the hypothesis class, not by the test-set answer entropy. The comparison with 100 × log2(32000) ≈ 1500 bits therefore does not establish that the observed accuracy gain can be explained by information injected through the prefix. This is load-bearing for the paper's first claimed pitfall, so the argument should be replaced or removed.","section":"§4.2, bit-capacity calculation"},{"comment":"The experiment that finetunes Llama-3.2-1B on WMDP-Bio to overfitting, optimizes an Enhanced GCG prefix on seven samples, and then reports improved accuracy on the original base model does not demonstrate that the prefix injects new knowledge. The base model's parameters are unchanged, so the accuracy gain from 39.9% to 53.8% must be explained either by latent knowledge already present in the base model or by a generic answer-selection heuristic induced by the prefix; both alternatives are consistent with the reported numbers, and the latter is exactly the possibility the paper intends to rule out. A control on questions whose answers were never present in any model used during optimization is needed, along with an analysis of what the optimized prefix encodes.","section":"§4.2, overfitting experiment"},{"comment":"The TOFU-MCQ finetuning experiment is presented as evidence that TOFU contains spurious correlations that let retain-set finetuning reintroduce forget-set knowledge. However, finetuning on a subset of TOFU-MCQ and evaluating on held-out questions about different fictitious authors may simply improve the model's ability to answer MCQs in the benchmark's format; it does not demonstrate that specific forget-set facts were relearned. The experiment should be embedded in an actual unlearning setup with disjoint forget and retain authors, and it should include a control that isolates format learning (e.g., finetuning on an unrelated MCQ dataset and measuring the same held-out accuracy gain). This does not invalidate the task-dependence findings in §4.3, but it weakens the first claimed pitfall.","section":"§4.1, Figure 2"},{"comment":"The success thresholds for the ACR evaluations in Figure 7 are selected after inspecting the optimization behavior on the same data: Appendix B states that thresholds are chosen such that further increasing them does not boost the success probability significantly. This data-dependent threshold choice makes the reported success rates difficult to interpret, and the qualitative comparison across tasks should be accompanied by a sensitivity analysis over threshold values or by a pre-specified thresholding rule defined on a held-out set. Without this, the conclusion that the relative effectiveness of RMU and NPO reverses across tasks rests on an arbitrary element of the evaluation protocol.","section":"§4.3 and Appendix B, ACR thresholds"}],"minor_comments":[{"comment":"These figures report point estimates without error bars or seed counts; adding error bars or stating that each curve is a single run would clarify the reliability of the comparisons, especially in Figure 4 where the curves are close.","section":"Figures 2, 4, 5, 6"},{"comment":"The notation Pr(x||y; M_U) is confusing; it appears to denote the probability that the model generates the target knowledge y given the prompt prefix x, but the conditional dependence is written backwards. Please clarify the notation.","section":"§3.1, Eq. (1)"},{"comment":"The definition of ACR says M(x)=y under greedy decoding, which is a deterministic equality, but §4.3 describes the ACR evaluation as finding a suffix that maximizes the probability of generating the correct answer. These two characterizations are inconsistent and should be reconciled.","section":"§3.1, Eq. (2)"},{"comment":"The abstract says the principles are 'validated' through experiments, but the paper does not implement a downstream-aware evaluation protocol; it demonstrates pitfalls and gives recommendations. Consider softening 'validated' to 'illustrated' or 'motivated'.","section":"Abstract and §5"},{"comment":"The proposed 'injection budget' is described only heuristically, and the paper's own bit-based measure is shown to be unreliable. A more concrete definition of how an injection budget would be computed and enforced would strengthen this recommendation.","section":"§5.1, Recommendation 1"},{"comment":"The table reports 40%, 50%, and 60% percentiles of ACR, but no explanation is given for why these particular percentiles are chosen; a brief justification would help the reader interpret the summary.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the venue and the task-dependence results are likely to be valuable to the unlearning community. The main obstacle is Section 4.2: the information-injection claim for input-space attacks is not supported by the current bit-capacity argument or the prefix-transfer experiment. If the authors cannot repair this argument, they should restrict the first pitfall to finetuning attacks and explicitly state that input-space attacks demonstrate task-format sensitivity but not information injection. The paper also relies heavily on ACR and TOFU from the same research group, but the central task-dependence findings do not depend on the validity of those tools, so I do not see this as a reason for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's main contribution is a set of demonstrations that unlearning evaluation outcomes flip when you change the downstream task format or the answer-scoring rule. Figure 7, especially the CHOOSE/OPTION/GENERATE comparison, is the strongest part: conclusions about RMU versus NPO change entirely depending on how you ask the question. Figure 2's TOFU-MCQ spurious generalization is also a nice, concrete finding. The two proposed principles—minimal information injection and downstream task awareness—are sensible organizing ideas, even if they are more like guidelines than formal criteria. I agree with the reader that the central claim is plausible and that the paper extends prior critiques rather than replacing them. The soft spot is Section 4.2. The bit-capacity argument is wrong as stated. The 100-token prefix is a single string, optimized on fewer than ten samples and then fixed across all test queries. It cannot encode per-question answer identities, so comparing ~1500 bits of prefix capacity with ~1430 bits of 'needed' test-set entropy is a category error. The supporting experiment—optimizing a prefix on an overfit model and seeing accuracy rise on the base model—is also ambiguous: it could be eliciting latent knowledge rather than injecting new facts. The stress-test note is correct that the information-injection pillar for input-space attacks is not established. I would not sink the paper on this, because the task-dependence evidence alone is enough to show that current evaluations are inconclusive, but the authors need to either fix the argument or change the claim to something like 'we cannot rule out information injection.' Missing error bars in Figures 2, 4, 5, and 6 are a minor issue; Figure 7 has them. No code or data release is mentioned, which matters because they introduce TOFU-MCQ. Who is this for? Anyone building or using LLM unlearning benchmarks, and safety evaluators who need to know when a benchmark result is format-dependent rather than knowledge-dependent. It is not a paradigm shift, but it is a useful cautionary result. I would send it to peer review with a request for major revision: rework or temper Section 4.2, add error bars, and release the TOFU-MCQ data and evaluation code. With those changes, it would be a solid, citable paper.","headline":"The task-format dependence evidence is solid enough to support 'inconclusive', but the paper's flagship bit-counting argument for information injection is unsound and should be reworked before publication.","tokens_in":728,"tokens_out":769,"would_cite":true,"duration_ms":25285,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that standard evaluations of LLM unlearning are inconclusive because they can inject new information, depend on question format, and rely on spurious correlations.","keywords":["machine unlearning","large language models","unlearning evaluation","adversarial attacks","information injection","multiple-choice accuracy","memorization detection","task sensitivity"],"falsifier":"Optimize an Enhanced GCG prefix on a handful of WMDP-Bio samples and evaluate it on held-out questions from the same dataset that were never used during optimization: if the prefix's accuracy gain largely disappears on those held-out questions, the injection account is supported, whereas if the gain persists, the prefix is eliciting latent knowledge and the information-injection violation is not established for input-space attacks.","tokens_in":13708,"feed_emoji":"🧹","tokens_out":7231,"duration_ms":63412,"temperature":0.7,"pith_summary":"The paper argues that current benchmarks for checking whether a large language model has truly unlearned sensitive knowledge do not support the conclusions drawn from them. It identifies three failure modes: some evaluation procedures feed new information into the model during testing, results swing with the task format, and datasets contain spurious correlations that let models appear to relearn knowledge they never saw. Because of these problems, the same unlearned model can look successfully erased under one protocol and completely unerased under another. The paper proposes two principles for future evaluations: minimal information injection and downstream task awareness. If the argument holds, existing claims about shallow unlearning, and also claims about successful unlearning, need to be re-examined.","feed_headline":"LLM unlearning tests both overstate and understate success","feed_subtitle":"Injected information, question format, and spurious correlations make current unlearning evaluations unreliable.","key_machinery":"The central machinery is the evaluation protocol itself, examined through three representative instruments: finetuning attacks, which modify weights on a small number of samples; input-space attacks, represented by Enhanced GCG, which optimizes a 100-token prefix prepended to forget-set prompts; and the adversarial compression ratio, the ratio of a string's length to the shortest prompt that makes the model generate it. A bit-capacity calculation compares the prefix's roughly 1,500 bits with the roughly 1,430 bits needed to encode 55 percent of the WMDP-Bio multiple-choice answers, motivating the charge that input-space attacks can inject, not merely elicit, information. These instruments are used to argue that no current evaluation cleanly separates pre-existing knowledge from evaluation artifacts.","core_discovery":"The paper's central claim is that current evaluation protocols may both overstate and understate unlearning success. It reaches this conclusion by showing three things: finetuning attacks and input-space attacks can inject substantial new information into the model during testing; evaluation outcomes shift with task format, such as multiple-choice letter probability, multiple-choice text probability, and open-ended generation; and benchmarks such as TOFU contain spurious correlations that allow models to generalize from retain data to unseen forget data. The paper therefore rejects the conclusiveness of standard adversarial unlearning evaluations and proposes two principles for future evaluations, minimal information injection and downstream task awareness, validated by targeted experiments showing how violations of each produce misleading conclusions.","pith_inferences":["The bit-capacity estimate is best read as a rough upper bound: a single shared 100-token prefix optimized on fewer than ten samples cannot store roughly 1,430 sample-specific answer bits, so the information-injection charge against input-space attacks would be stronger if paired with a per-sample transfer test on held-out questions.","The two principles imply that future benchmarks should report an explicit injection budget and a cross-format leakage matrix, which would make unlearning claims more actionable for regulators and deployment decisions.","If task-format sensitivity is as large as reported, unlearning success should be evaluated over a distribution of downstream tasks rather than a single multiple-choice score, and forget sets should be constructed to be independent of retain-set features to avoid spurious generalization."],"forward_implications":["If finetuning attacks can inject information, then a model that fails such an attack may simply have been re-taught the forget set during testing, so the failure does not prove the knowledge was retained.","If multiple-choice accuracy depends on whether answers are scored by letter probability or text probability, then benchmark numbers from different papers are not directly comparable, and reported unlearning success rates carry an unstated format assumption.","If TOFU-type benchmarks contain spurious correlations, then relearning on retain data improving forget-set accuracy does not establish that unlearning failed, since the improvement can come from benchmark artifacts rather than recovered knowledge.","If conclusions flip across tasks, then unlearning should be reported as a cross-format leakage matrix rather than a single accuracy number, and a model that leaks in any format has not truly forgotten.","If a memorization detector gives a positive result after unlearning, that is strong evidence the information is still present, making memorization detectors a useful high bar even though a negative result remains inconclusive."],"supporting_citations":[{"why":"Supplies the WMDP benchmark and the RMU unlearning algorithm used throughout the experiments.","marker":"[3]"},{"why":"Supplies the NPO unlearning algorithm used to generate unlearned models in the format-dependence experiments.","marker":"[4]"},{"why":"Defines the evaluation taxonomy and finetuning-attack setup that the paper builds on.","marker":"[5]"},{"why":"Supplies the Enhanced GCG attack whose information-injection and task-sensitivity the paper analyzes.","marker":"[6]"},{"why":"Introduces the adversarial compression ratio used to show that unlearning conclusions depend on the downstream task.","marker":"[7]"},{"why":"Supplies the TOFU dataset and the basis for the TOFU-MCQ variant used to demonstrate spurious generalization.","marker":"[28]"}],"fun_headline_variants":["LLM unlearning tests flunk reliability check","Unlearning evaluations: too flawed to trust","Current LLM unlearning scores are misleading","Unlearning benchmarks fail to measure true forgetting","Study: unlearning tests inject data, skew results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's information-injection critique of input-space attacks rests on the premise that the accuracy gained from an optimized 100-token prefix must be explained by information encoded in that prefix, measured by counting its bits, rather than by the prefix eliciting latent knowledge the model already possesses.","fun_headline_variants_meta":{"raw":{"variants":["LLM unlearning tests flunk reliability check","Unlearning evaluations: too flawed to trust","Current LLM unlearning scores are misleading","Unlearning benchmarks fail to measure true forgetting","Study: unlearning tests inject data, skew results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000123,"raw_usage":{"total_tokens":1050,"prompt_tokens":843,"completion_tokens":207,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":138}},"tokens_in":459,"tokens_out":207,"duration_ms":2358,"temperature":1.0,"reasoning_tokens":138,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:59:50.371006+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Optimize an Enhanced GCG prefix on a handful of WMDP-Bio samples and evaluate it on held-out questions from the same dataset that were never used during optimization: if the prefix's accuracy gain largely disappears on those held-out questions, the injection account is supported, whereas if the gain persists, the prefix is eliciting latent knowledge and the information-injection violation is not established for input-space attacks.","supporting_citations":[{"cited_title":"Model manipulation attacks enable more rigorous evaluations of llm capabilities","cited_arxiv_id":null,"evidence_quote":"Defines the evaluation taxonomy and finetuning-attack setup that the paper builds on."}],"review_version":1}