{"id":"aa007575-c397-4efb-82b3-fab3c9edd6db","arxiv_id":"2601.04275","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A projector plus PCA-based activation filter is claimed to unlearn LLM knowledge from anonymized data while preserving utility, but the reported metrics and efficiency claims are internally inconsistent.","lead":"This paper proposes a machine-unlearning method that operates on anonymized forget data, so PII never has to be shared with the model operator. It claims better forget-retain tradeoffs and 10x computational savings, but its own tables contradict the efficiency claim and its headline scores do not match the stated formulas.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's headline metrics are not derivable from Eqs. (4)-(13); e.g., Mistral-7B NSPU has GF=-0.084, CR=2.066 yet HCS=4.995, and the claimed 10x efficiency contradicts Table 3 (only ~1.7x).","rationale":"The reader's stated weakest assumption is about the Latent Representation Aligner's accuracy, which is a legitimate generalizability concern. However, the most load-bearing flaw is internal: the headline evaluation table is arithmetically inconsistent with the paper's own defining equations. This is not a matter of external consensus or tuning; it is a direct contradiction that can be checked with pencil and paper. If Table 2 cannot be reproduced, the central empirical claim that NSPU achieves superior unlearning performance is unsupported, regardless of aligner fidelity. The efficiency claim is similarly contradicted by Table 3, where NSPU is about 1.7x more efficient than GA, not 10x. These internal inconsistencies justify the reader's REJECT verdict. I partially agree with the reader because the metric issue was mentioned in the rationale, but it was not identified as the weakest assumption. A revised paper with corrected tables, released code, and a validation protocol for α might be reconsidered, but the current submission does not meet the bar for acceptance.","tokens_in":32681,"tokens_out":6340,"duration_ms":52857,"concrete_test":"Recompute every HCS, HRS, and HCNLL cell in Table 2 from the raw GF/CR, RR/FR, GFL/CRL values using Eqs. (4), (9), (13). If the entries do not match to within rounding (e.g., Mistral HCS would be −0.203 instead of 4.995; Llama-7B HCS 0.721 instead of 1.067), the central result is not reproducible and the superiority claim fails. Also independently derive the FLOPs ratio from Table 3; if NSPU is not ≥10× fewer FLOPs than every baseline, the abstract's efficiency claim is contradicted by the paper's own data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The primary evidence for 'NSPU consistently outperforms baselines' is Table 2, but the table cannot be reproduced from the paper's own equations. For example, Eq. (4) defines HPS = 2·GF/(GF·CR+1). For Mistral-7B NSPU, GF=−0.084 and CR=2.066 give HPS ≈ −0.203, yet Table 2 reports HCS=4.995. A negative GF (forget perplexity ratio below 1) cannot yield a large positive harmonic score. Similarly, Llama-7B NSPU has GF=1.388, CR=2.055 yielding HPS≈0.721, not the reported 1.067. HCNLL entries also fail Eq. (13): Mistral NSPU gives 0.741 vs reported 1.281. Some rows (e.g., GA and KLM in Llama-7B and Mistral) are numerically identical, suggesting a copy-paste error. If these numbers cannot be reproduced, the claim that NSPU achieves 'superior unlearning performance' is unsubstantiated. The efficiency claim is likewise unsupported: Table 3 lists NSPU FLOPs as 7.63e16 vs GA 1.29e17 for Llama-7B, a factor of ~1.7, not 'at least 10x' as stated in the abstract and §4.5.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Shadow Unlearning, a paradigm in which machine unlearning is performed on an anonymized forget set, and introduces NSPU, a three-stage method: (i) train an MLP to map anonymized activations back into the original activation space, (ii) construct a PCA-based forget subspace from the projected anonymized forget activations, and (iii) insert a non-trainable linear filter I−αUU^T into the frozen target model. The authors introduce a synthetic five-domain MuFU benchmark and evaluate four LLMs against GA, GD, KLM, DPO, and NPO using four harmonic trade-off metrics plus an SQS-based membership-inference analysis. The central claims are that NSPU consistently outperforms the baselines, preserves utility, enhances user privacy, and is at least 10 times more computationally efficient than standard unlearning approaches. I find that the headline evaluation is not self-consistent: the reported metric values in Table 2 cannot be reproduced from the paper's own equations, and the stated efficiency claim is contradicted by Table 3. As a result, the principal empirical claims are not currently supported.","tokens_in":33091,"tokens_out":13136,"duration_ms":121356,"significance":"The idea of performing unlearning on anonymized data is timely, and the frozen-target projection architecture is an interesting and nonstandard approach. The construction of the MuFU benchmark and the breadth of models and ablations are also useful if the underlying measurements are reliable. However, the current manuscript does not establish its central claims. The main evidence for 'NSPU consistently outperforms baselines' is a table whose values do not follow the paper's stated formulas, and the 'at least 10x' efficiency claim is contradicted by the paper's own FLOPs table. Because these are load-bearing issues in the abstract, Section 4.4, and Takeaway 2, the contribution cannot be assessed as presented. If the evaluation were corrected and independently reproducible, the paradigm could be a valuable addition to privacy-preserving unlearning; the present version does not provide that reliable basis.","major_comments":[{"comment":"The HCS/HPS and HCNLL values are not derivable from the definitions. For Mistral-7B NSPU, Table 2 gives GF=−0.084 and CR=2.066, but Eq. (4) yields HPS = 2(−0.084)/((−0.084)(2.066)+1) = −0.203, not 4.995. For Llama-7B NSPU, GF=1.388 and CR=2.055 yield HPS ≈ 0.721, not 1.067. Similarly, Eq. (13) with Mistral's GFL=0.955 and CRL=1.650 gives HCNLL ≈ 0.741, not 1.281. Since the aggregate scores and the conclusion that 'NSPU consistently outperforms baseline unlearning methods' are built on these columns, the central experimental result is unsupported as reported.","section":"§4.4, Table 2, Eqs. (4) and (13)"},{"comment":"The claim that NSPU is 'at least 10x more computationally efficient than standard unlearning approaches' is contradicted by the paper's own FLOPs table. For Llama-7B, NSPU is 7.63e16 FLOPs versus GA's 1.29e17, a ratio of about 1.7x; for Mistral-7B the ratio is about 1.7x; for Llama-13B about 1.7x; for OLMoE about 2.1x. Even compared with GD/DPO/NPO at 5.16e17 FLOPs, the Llama-7B ratio is about 6.8x, still below 10x. Only the comparison with retraining from scratch approaches the claimed 10^6x. Takeaway 2 and the abstract therefore materially overstate the efficiency result.","section":"Abstract, §4.5, Table 3"},{"comment":"The reported superiority of NSPU may be inflated by tuning and by construction. The forget subspace U is built from the same anonymized forget set that is later used to compute forget-set effectiveness, and α is selected separately for every model and overlap variant (values range from 0.01 to 0.315 in Table 12) with no described validation or selection criterion. Without a sensitivity analysis or an independent validation split, one cannot tell whether the head-to-head win reflects the method itself or favorable hyperparameter selection. The paper should report how α was chosen and show performance across a range of α values.","section":"§3.6, Table 12, §4.4"},{"comment":"The privacy claim is not currently established. The method requires training a projector to map anonymized activations back into the original activation space; Table 5 reports cosine similarities up to 0.87 (Llama-13B), and Appendix B refers to the projector as a 'deanonymization module.' Section 3.1 asserts that the anonymization function f is one-way in practice, but the projector appears to be a partial inversion tool. No attack on the projection is reported, and InvOptScore is only a heuristic training penalty rather than a demonstrated privacy guarantee. Thus 'enhances user privacy' remains a conjecture rather than a measured property.","section":"§3.4, §7, Table 5, Appendix B"},{"comment":"Several entries appear to be copy-paste artifacts rather than distinct measurements. In Table 2, the GA and KLM rows are numerically identical for Llama-7B (e.g., GF=104.730, CR=110.110) and again for Mistral-7B (GF=154.695, CR=202.520), even though KLM includes an additional retain-set KL regularizer and therefore should differ from GA. In Appendix B Table 9, the MLP reconstruction rows for Mistral-7B and OLMoE-1B-7B are identical across all overlap settings (e.g., 5%: MSE=1.044, R²=0.443), despite being different models. These duplicates further undermine the reliability of the reported numerical results.","section":"Table 2, Table 9"}],"minor_comments":[{"comment":"The notation is inconsistent: the text defines HPS, but Table 2 uses the column label HCS without defining it. Please harmonize the metric names.","section":"§4.2, Table 2"},{"comment":"The text says 'As depicted in Table 6' when referring to the LLM-as-a-judge results; the relevant artifact is Figure 6, while Table 6 is the PII attribute similarity table. The cross-reference should be corrected.","section":"§5.3"},{"comment":"Equation (31) writes UL_filter = I − α v_in U U^T, but v_in is an activation vector, not a d×d matrix, so the expression is dimensionally inconsistent. From the preceding definitions the filter should be I − α U U^T. Also, the section is titled 'Theoretical Guarantee' but contains only an algebraic re-derivation; no formal forgetting guarantee is provided.","section":"Appendix C, Eqs. (30)-(31)"},{"comment":"The phrase 'training-free unlearning procedure' is misleading because Stage 1 trains the latent representation aligner Pθ. It would be more accurate to say that NSPU performs no gradient updates to the target model.","section":"§3.6"},{"comment":"The abstract says 'Code is available at Github' without a URL, and the reproducibility statement does not provide a repository link or dataset access information. For a paper whose central evaluation is currently unreproducible, providing the actual artifacts is essential.","section":"Abstract, §8"}],"recommendation":"reject","confidential_remarks":"This manuscript requires a full experimental recheck before any further review. The issues are not presentation-level: the central Table 2 cannot be reproduced from the paper's own definitions, and the abstract's '10x' efficiency claim is contradicted by the paper's own FLOPs table. If the authors can recompute the metrics, correct the efficiency claim, clarify the α-selection protocol, and release the code and data, the underlying idea may be worth reconsideration in a future submission. As it stands, the load-bearing evidence does not support the stated conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the problem it targets is real: unlearning on anonymized forget data would fix a genuine privacy paradox in current unlearning practice. Second, the central evidence is not self-consistent. The headline numbers in Table 2 do not follow from the paper's own Equations (4) and (13). For example, Mistral-7B NSPU has GF=-0.084, CR=2.066, which yields HPS ≈ -0.20, but the table reports HCS = 4.995; the same pattern holds for Llama-7B and for HCNLL. Negative forget gain cannot produce a large positive harmonic score. That is load-bearing because the \"NSPU consistently outperforms\" claim rests on those aggregates.\n\nThe efficiency claim is also overstated. The abstract says \"at least 10x\" faster than standard unlearning, but Table 3 shows NSPU is ~1.7x cheaper than GA and ~6.8x cheaper than NPO for Llama-7B—real but not 10x. There are also duplicated rows (GA and KLM identical for two models, and Table 9 gives identical MLP metrics for Mistral and OLMoE), which looks like copy-paste error.\n\nWhat's genuinely new: the shadow-unlearning task framing is a useful contribution, and the method—training an MLP to map anonymized activations back to original space, then erasing the PCA subspace of the projected forget set—is a plausible recipe. The MuFU dataset with controlled overlap variants is a reasonable evaluation scaffold. That part deserves credit.\n\nSoft spots beyond the table issues: alpha is tuned per model and overlap variant, with no validation protocol or error bars, so we don't know if the reported superiority is a selection artifact. The privacy story is weak: the paper's own Table 5 shows the projector reconstructs original activations with cosine similarity up to 0.87. That is an effective deanonymizer, not \"inversion-resistant\" in a strong sense.\n\nIf the numbers can be corrected and the code actually released, this could be a solid paper. As it stands, I would not rely on the quantitative claims. The idea is worth a reading-group discussion, and I'd cite the task framing, not the results.\n\nRecommendation: send to peer review, but with a referee who can check the arithmetic. The core idea is worth refereeing; the current version needs major revision.","headline":"Real problem framing, but the main evaluation table doesn't reproduce from the paper's own formulas—verify before relying on any performance claim.","tokens_in":33604,"tokens_out":5766,"would_cite":false,"duration_ms":49563,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model can erase specific knowledge from itself even if the only data it sees is anonymized, by working in the model's internal activation geometry rather than on raw text.","keywords":["shadow unlearning","machine unlearning","privacy-preserving unlearning","anonymization","latent representation aligner","forget subspace","membership inference","LLM utility-fidelity tradeoff"],"falsifier":"Take a model after NSPU unlearning and ask it the original, non-anonymized forget questions: if it still reproduces the gold answers at the same confidence as the target model, the filter removed nothing relevant. A sharper test is out-of-distribution: feed the aligner anonymized forget text from a domain absent from its public training corpus and compare Pθ(φ_l(x_anon)) to the true φ_l(x_orig) via cosine similarity; if the similarity collapses toward zero, the forget subspace is built from noise and the reported separations on the synthetic set would not transfer.","tokens_in":32546,"feed_emoji":"🤖","tokens_out":4405,"duration_ms":44970,"temperature":0.7,"pith_summary":"The paper tries to resolve a privacy paradox in machine unlearning: to delete a user's data, the model operator usually has to be handed exactly that data, defeating the purpose. It proposes Shadow Unlearning, where unlearning runs on anonymized forget sets, and a concrete method, NSPU, that learns the semantic mapping between anonymized and original text inside the model's activation space, extracts the principal directions of the forget set from that space, and suppresses those directions during inference. If this works, a deletion request can be honored without anyone touching PII, while the model keeps most of its utility and the compute stays far below gradient-based unlearning. The experiments span four LLMs and five domains, and the paper's central claim is that this privacy--utility--efficiency balance is achieved by a training-free, projection-based filter.","feed_headline":"Forget data can stay anonymous during LLM unlearning","feed_subtitle":"Projecting anonymized text into the model's own activation space lets a simple filter erase forget-set knowledge at 10x lower compute.","key_machinery":"The Latent Representation Aligner Pθ, a lightweight MLP trained with ||Pθ(φ_l(x_anon)) - φ_l(x_orig)||² and an inversion-resistance term, maps anonymized activations into the original activation space. The Forget Subspace U is the top-k PCA basis of those projected forget activations. The Unlearning Filter UL_filter = I - α U U^T is a non-trainable adapter that projects each activation away from the forget subspace at inference, attenuating forget-related concepts while leaving the orthogonal complement intact.","core_discovery":"The central claim is that unlearning can be done on anonymized data alone by operating in the model's internal activation geometry. A lightweight MLP aligner, trained once on public (original, anonymized) text pairs, projects anonymized activations into the original activation space; PCA over the projected forget-set activations yields a forget subspace; and a non-trainable filter I - alpha*U*U^T, inserted at the final layer, subtracts the forget-subspace component from every activation during inference. The paper argues this simultaneously makes the forget set hard to detect via membership inference, preserves retain-set performance, and avoids any gradient update of the target model, givin","pith_inferences":["An extension the paper does not develop: because the filter is a linear projection independent of the input text, the same U could be built from any one-way anonymization function f(·), as long as the aligner is retrained on that f's outputs, making shadow unlearning a plug-in for pseudonymization schemes beyond NER tag replacement.","The paper tests paraphrased, context-augmented, and hard-token attacks on the filter, but it leaves open a sharper adversarial question: whether someone with access to the aligner Pθ can invert its outputs to reconstruct the original text. The InvOptScore term is designed to raise that cost, and a direct inversion attack on the aligner would be the natural stress test.","Because the evaluation rests on synthetic author profiles, the practical deployment claim—that a public-corpus-trained aligner maps real-world anonymized forget sets reliably—remains untested at scale. Running the pipeline on a real PII-bearing corpus with known ground-truth pairs would be the direct way to test transfer."],"forward_implications":["Unlearning pipelines can be restructured so the forget set is anonymized before it reaches the model operator, removing a major PII exposure point during deletion requests.","Because the target model stays frozen, a deletion request can be serviced with a single matrix multiply per token instead of gradient updates, with FLOP counts roughly 10x lower than gradient baselines and about 10^6 lower than retraining.","The harmonic evaluation stack (perplexity, truth ratio, ROUGE-L, and conditional NLL combined into retention-versus-forgetting ratios) gives a single score for the utility--efficacy tradeoff, making it possible to compare unlearning methods on how much knowledge survives rather than only on how much is forgotten.","The entity-level analysis suggests the method can erase specific PII types, such as person names, locations, and phone numbers, while leaving other domain knowledge in place."],"fun_headline_variants":["Shadow unlearning: anonymized forget data, no exposure","Activation-space unlearning cuts compute 10x","Unlearn with anonymized activations, keep model utility","Neuro-semantic unlearning: no PII, just geometry"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a small MLP trained on public (original, anonymized) text pairs can predict the original activation of an anonymized forget sample accurately enough that the principal directions of those predictions coincide with the true semantic content to be forgotten; the paper's own domain-wise similarity numbers (as low as 0.47 for one model and domain) and its stated dependence on overlap between forget and retain distributions mark this as the fragile","fun_headline_variants_meta":{"raw":{"variants":["Shadow unlearning: anonymized forget data, no exposure","Activation-space unlearning cuts compute 10x","Unlearn with anonymized activations, keep model utility","Neuro-semantic unlearning: no PII, just geometry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1526,"prompt_tokens":739,"completion_tokens":787,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":720}},"tokens_in":483,"tokens_out":787,"duration_ms":7628,"temperature":1.0,"reasoning_tokens":720,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T12:10:56.433445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model after NSPU unlearning and ask it the original, non-anonymized forget questions: if it still reproduces the gold answers at the same confidence as the target model, the filter removed nothing relevant. A sharper test is out-of-distribution: feed the aligner anonymized forget text from a domain absent from its public training corpus and compare Pθ(φ_l(x_anon)) to the true φ_l(x_orig) via cosine similarity; if the similarity collapses toward zero, the forget subspace is built from noise and the reported separations on the synthetic set would not transfer.","supporting_citations":[],"review_version":1}