{"id":"0f885f22-295b-4e7b-af87-abdc730b5dd2","arxiv_id":"2507.21084","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Sparse model diffing on task-agnostic text can flag which academic, safety, or behavioral categories a fine-tuned or unlearned LLM has unintentionally changed.","lead":"This paper introduces MNEME, a pipeline that compares the internal activations of an original and a modified LLM on generic text to find which knowledge areas or behaviors were unintentionally changed by unlearning or fine-tuning. A generalist reader might care because it promises an automated audit of model updates without access to the fine-tuning data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MNEME's headline accuracy metric measures semantic label agreement, not predictive side-effect detection; the sparse-diffing claim needs an item-level or null-diff test.","rationale":"The reader's verdict was CONDITIONAL, and I agree that the method is promising but not fully validated. However, the most load-bearing concern is not primarily the fixed layer or corpus coverage (the reader's weakest_assumption), but the validity of the evaluation metric itself. The reported 'accuracy' in the two headline scenarios is computed as agreement between LLM-generated semantic labels and benchmark-derived category labels, which is a form of semantic alignment rather than a demonstration that MNEME predicts actual side effects before they are observed. The paper takes care to avoid direct leakage by not consulting MMLU categories before mapping, but the semantic mapping LLM (LLaMA-3.1-70B-Instruct or Gemini 2.5-Pro) is itself a learned component trained on web text that includes MMLU-like taxonomies, and the evaluation categories are coarse (e.g., chemistry, biology), making high agreement easier to achieve. Section 7's finding that only 40% of latents have over 90% semantic overlap with target inputs further suggests that the sparse features are noisy. The proposed item-level AUC and null-diff tests would directly settle whether MNEME's features carry genuine predictive signal. My read does not change the CONDITIONAL verdict; it sharpens the condition: the method should be evaluated on held-out, item-level behavioral predictions, not on post-hoc semantic label alignment. This is a partial agreement with the reader because the reader flagged benchmark-aligned evaluation as a mild leakage risk, whereas I see it as the central unvalidated link in the argument.","tokens_in":13587,"tokens_out":4226,"duration_ms":46638,"concrete_test":"Re-run the WMDP experiment with an item-level predictive test: before computing category labels, use MNEME's amplified latents to score each MMLU item (e.g., by activation overlap or feature relevance), then measure AUC for predicting which individual items flip from correct to incorrect after unlearning. If item-level AUC is near chance while category-level accuracy remains above 90%, the headline metric overstates predictive power. In addition, run a null diff (base versus itself, or two seeds of the same fine-tune) and report the false-positive rate of side-effect categories; a nonzero false-positive rate on a null diff would show the method flags spurious side effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MNEME predicts side effects with up to 95% accuracy. The metric behind this claim is not predictive. In the WMDP study (Section 4.2), accuracy is defined as alignment between MNEME's LLM-generated semantic labels and the MMLU categories that the authors found to be degraded after unlearning. In the emergent misalignment study (Section 5.2 and Appendix B), the EM accuracy is the overlap between amplified features and features that Gemini 2.5-Pro judges to match misalignment instructions or responses. Both evaluations measure agreement among semantic labels produced by LLM-based components, not whether MNEME's features anticipate actual behavioral failures on held-out tasks. The high accuracy may therefore reflect the semantic mapping stage rather than the sparse diffing mechanism, and the framework could flag categories that are semantically related to the intervention even when no real side effect occurs. The Limitation section itself concedes that MNEME provides correlational rather than causal insights and that the interpretability pipeline depends on LLM-generated descriptions. Consequently, the 'up to 95% accuracy in predicting side effects' claim is not established by the reported metric; what is established is that MNEME's semantic labels align with benchmark-degraded categories in scenarios where those categories are known in advance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MNEME, a framework that applies sparse Cross-Coder model diffing to activations of a base model and its fine-tuned or unlearned counterpart, using task-agnostic corpora (The Pile, LMSYS-Chat-1M) so that fine-tuning data are not required. The pipeline generates latent features, attributes them via latent scaling, produces natural-language descriptions with an LLM, and maps those descriptions to semantic categories. The authors claim that this enables automated detection of side effects of unlearning and fine-tuning, reporting accuracies of up to 95% on WMDP unlearning, 85% on benign fine-tuning, and 50% on emergent misalignment (abstract/introduction), or 96%/98%/92% on WMDP, 92.9% on benign fine-tuning, and 68.2% on emergent misalignment (body). The paper also claims that retraining on high-activation samples can partially reverse side effects.","tokens_in":13805,"tokens_out":4623,"duration_ms":47846,"significance":"If the predictive claim were established, MNEME would be a practically valuable contribution: an automated, data-agnostic audit tool for post-training interventions, with applications to unlearning safety and alignment. The paper has real strengths: it builds on established Cross-Coder and BatchTopK machinery, evaluates across five models and three scenarios, includes baseline comparisons, and attempts to avoid label leakage when mapping generated categories to MMLU. However, the central claim is currently supported only by a semantic label-agreement metric, not by evidence that MNEME anticipates behavioral side effects on held-out tasks. The reported numbers are also internally inconsistent. The methodology is plausible, but the evaluation must be substantially reworked before the headline claim can be accepted.","major_comments":[{"comment":"The accuracy numbers are inconsistent between the abstract/introduction and the body. The abstract and §1 state up to 95% on WMDP, 85% on benign fine-tuning, and 50% on emergent misalignment, while §4.2 reports 96%, 98%, and 92% on WMDP, §5.2 reports 92.2% on MMLU-Pro and 68.2% on EM, and §6.2 reports 82.2% and 92.9% on the two fine-tuning scenarios. The reader cannot tell which set of numbers is authoritative, and the headline claims should be corrected to match the actual results.","section":"Abstract, §1 vs §4.2/§5.2/§6.2"},{"comment":"The reported accuracy does not establish predictive side-effect detection. In the WMDP study, accuracy is defined as alignment between MNEME's LLM-generated semantic categories and the MMLU categories that were already known to be degraded after unlearning. In the emergent misalignment study, accuracy is the overlap between amplified features and features that Gemini 2.5-Pro judged to correspond to misalignment instructions or responses. Both are measures of agreement among LLM-produced semantic labels, not evidence that the detected features anticipate behavioral failures on unseen prompts. The paper should add item-level held-out evaluations (e.g., do amplified features predict per-prompt degradation?) and a null-diff control in which the method is applied to a base model against itself or to a model pair with no intervention.","section":"§4.2, §5.2, Appendix B"},{"comment":"There is no control that isolates the contribution of sparse model diffing from the LLM semantic-mapping stage. The naive baseline in §6.2, which uses GPT-4o to infer side effects from domain knowledge, achieves 94.1% accuracy on benign fine-tuning, essentially tied with MNEME's 92.9%. This is consistent with the hypothesis that the semantic mapping stage, rather than the activation-diffing mechanism, drives the reported accuracy. The authors should compare MNEME against an activation-based null (e.g., diffing the base model against itself on different data splits, or against a random perturbation) and against a mapping-only baseline that is given the same feature descriptions but no diffing information.","section":"§3.4, §4.2"},{"comment":"The abstract claims that 'retraining on high-activation samples can partially reverse these effects,' but no experiment or section in the body reports such a retraining result. Section 7 discusses whether fine-tuning data activate the expected latents, and Section 8 is a conclusion; neither contains a retraining or reversal experiment. Either add the missing experiment and its results, or remove the claim from the abstract.","section":"Abstract, §7–§8"},{"comment":"The validation in Section 7 reports that only 40% of latents had over 90% semantic overlap when fine-tuning data were passed through the Cross-Coder, which the authors themselves describe as 'moderate correspondence.' This weakens the link between the detected features and the actual intervention, and it should be reconciled with the claim that MNEME achieves near-oracle accuracy in predicting side effects. The authors should clarify whether this validation is meant to support the predictive claim and, if so, why a 40% overlap is sufficient.","section":"§7"}],"minor_comments":[{"comment":"The fixed choices of layer 14, expansion factor 32, and top-k of 100 are justified only by reference to prior SAE work; a per-scenario sensitivity analysis would strengthen the paper, and if such an analysis is not feasible the limitation should be stated more explicitly.","section":"§3.1"},{"comment":"Model names are inconsistent: the text in §4.1 refers to 'LaMA 3.2-3B Instruct' while Figure 3 and §4.2 refer to 'LLaMA-3.3-3B-Instruct.' Please standardize the model names throughout.","section":"§4.1, Figure 3"},{"comment":"The AOA accuracy is reported as 82.2% in the text, while the figure caption and axis labels suggest a value around 0.82; the random baseline values are given as 48.7% and 49.9% in the text. Please verify all numbers in the figure against the text.","section":"§6.2, Figure 4"},{"comment":"The Limitations section is candid about correlational rather than causal insights and about dependence on LLM-generated descriptions, but it should also explicitly state that the reported accuracies are label-agreement measures, not out-of-sample behavioral predictions.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a plausible adaptation of Cross-Coder diffing to unlearning and fine-tuning auditing, but the headline claim of 'predicting side effects' is not supported by the current evaluation, which measures semantic label agreement. The internal accuracy inconsistencies and the missing retraining result should be fixed. I would encourage the authors to add an item-level predictive evaluation and an activation-based null control; if those experiments support the claim, the paper could be a solid contribution. The novelty relative to prior CrossCoder work is incremental, so the evaluation quality will be decisive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat's actually new here: MNEME is the first framework to combine BatchTopK Cross-Coders, latent scaling, and auto-interpretation specifically for detecting side effects of unlearning and fine-tuning, without access to the training data. That is a sensible and useful application of existing interpretability tools, and the authors deserve credit for assembling the pipeline and testing it on three safety-relevant scenarios.\n\nWhat the paper does well: the method is clearly described, the baselines are reasonable, and the limitations section is honest about correlational rather than causal claims and about the reliance on LLM-generated descriptions. The three case studies (WMDP unlearning, emergent misalignment, benign fine-tuning) are well chosen and cover distinct types of side effects.\n\nNow the soft spots. The central claim of \"up to 95% accuracy in predicting side effects\" is not supported by the metric used. In the WMDP study, the gold standard is the set of MMLU categories that the authors already know are degraded after unlearning; the accuracy measures whether MNEME's semantic labels match those categories. In the emergent misalignment study, the metric is overlap between amplified features and features that Gemini judges to match misalignment. That is label agreement, not prediction. The stress-test note lands: there is no held-out intervention, no item-level analysis, and no null-diff control (base vs base) to show the features are intervention-specific. The paper even concedes in the limitations that the method gives correlational insights, so the abstract's predictive claim is too strong.\n\nThere are also internal inconsistencies: the abstract reports 95% WMDP, 85% benign fine-tuning, and 50% emergent misalignment, while the body reports 96–98%, 92.9%, and 68.2% for the same numbers. That is the kind of discrepancy a referee will flag immediately. The retraining-reversal claim appears only in the abstract and has no experiment in the body. No error bars, no code release, and the layer/corpus choice is fixed without per-scenario validation.\n\nDespite these issues, I would send this to peer review. The idea is worth testing, the evaluation design can be fixed, and the paper gives the interpretability community a concrete direction. A serious referee should ask for a predictive evaluation: train on one intervention, test on another, include a null-diff control, and report consistent numbers with uncertainty.\n\nBottom line: promising but overclaimed. Worth engaging with, not worth accepting as-is.","headline":"Plausible pipeline, but the headline claim of predicting side effects is not supported by the current evaluation, which measures agreement with known degraded categories.","tokens_in":14387,"tokens_out":2102,"would_cite":false,"duration_ms":23374,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sparse model diffing predicts LLM fine-tuning side effects at 95 percent accuracy.","keywords":["LLM unlearning side effects","sparse model diffing","cross-coders","model interpretability","emergent misalignment","benign fine-tuning safety","machine unlearning","latent scaling"],"falsifier":"Take a fine-tuned model whose only measurable side effect is confined to a layer other than layer 14, or only manifests on tokens absent from generic corpora like The Pile or LMSYS-Chat-1M; if MNEME's layer-14 generic-text features still predict the side effect at benchmark-level accuracy, the claim is supported, and if they miss it while a data-rich baseline catches it, the fixed-layer and generic-corpus assumption is falsified.","tokens_in":13369,"feed_emoji":"🧠","tokens_out":7164,"duration_ms":60458,"temperature":0.7,"pith_summary":"MNEME is a lightweight, automated framework for spotting unintended side effects of fine-tuning or unlearning a large language model. Its central claim is that comparing the base and edited models on ordinary, task-independent text, without any access to the fine-tuning data, can expose the behavioral shifts that benchmarks typically miss. The method decomposes the difference between the two models' internal activations into sparse, interpretable features and labels those features in natural language, predicting which capabilities will degrade or which harmful behaviors will emerge. Across five models and three scenarios, the predictions match known side effects with up to 95 percent accuracy, and the paper shows that retraining on high-activation samples can partially reverse the detected effects.","feed_headline":"Sparse diffing predicts LLM side effects up to 95%","feed_subtitle":"MNEME compares base and edited models on generic text, flagging unintended shifts without seeing training data.","key_machinery":"The central mechanism is the BatchTopK Cross-Coder: a shared encoder that maps the concatenated activations of base and edited models to a sparse latent code, with two separate decoders reconstructing each model's activations. BatchTopK applies global competition across a batch rather than per input, which the paper finds yields more interpretable, monosemantic features. Feature attribution then uses closed-form latent-scaling coefficients to place each feature on an amplified-to-suppressed spectrum, and an instruction-tuned LLM generates descriptions and category labels for the top-activating contexts. The whole pipeline runs on roughly 200 million tokens from task-agnostic corpora, at layer 14 with an expansion factor of 32, so it needs only the two models and generic text, not the fine-tuning data.","core_discovery":"The paper's central discovery is that sparse model diffing turns an opaque intervention, fine-tuning or unlearning, into a readable list of 'what changed' at the feature level. A BatchTopK Cross-Coder learns a shared sparse latent space from paired activations of the base and edited models at a fixed layer, then a latent-scaling regression measures whether each feature is amplified, suppressed, or left unchanged. Auto-interpretation converts these features into natural-language descriptions and semantic categories. On hazardous-knowledge unlearning, emergent misalignment, and benign or implicit fine-tuning, the resulting feature set predicts observed side effects, such as lost chemistry knowledge, emergent deception, or increased compliance with harmful instructions, with accuracy up to 95 percent, outperforming random and heuristic baselines and approaching an oracle. The authors also report that retraining on the samples that most activate the changed features partially reverses the side effects.","pith_inferences":["Editorial inference: the fixed layer-14 choice means MNEME's recall is bounded by how much of an intervention's effect flows through that layer; a layer-adaptive version, selecting the layer with maximal cross-coder reconstruction shift, would likely close the gap on the lower emergent-misalignment accuracy.","Editorial inference: if amplified features are reliably detectable before they manifest behaviorally, MNEME could serve as a pre-deployment gate that triggers inspection or retraining before a harmful response is ever served.","Editorial inference: the same paired-activation diffing could be applied to multimodal or vision-language models, where side effects of unlearning might appear in non-text residual streams.","Editorial inference: the reported 40-percent semantic overlap between fine-tuning data and auto-interpreted features suggests a ceiling; combining MNEME with low-cost probing on a few labeled samples might push accuracy toward the oracle."],"forward_implications":["Model auditors can flag side effects of unlearning or fine-tuning using only the base model, the edited model, and generic text, without needing proprietary training data.","Emergent misalignment can be detected even when standard jailbreak-style benchmarks show no degradation, because the amplified harmful features appear on ordinary chat text.","Benign fine-tuning that silently erodes safety, including from as few as ten crafted harmless-looking examples, becomes visible as amplified compliance-related features.","Retraining on the samples that most strongly activate changed features offers a partial remedy, giving practitioners a targeted repair path rather than full retraining.","The method scales across model sizes and families, indicating the feature-level signature of an intervention is a general phenomenon rather than an artifact of one architecture."],"supporting_citations":[{"why":"It introduces Cross-Coders for model diffing, the architecture MNEME adapts.","marker":"Lindsey et al., 2024"},{"why":"It supplies BatchTopK sparsity, which the paper uses to obtain interpretable features.","marker":"Bussmann et al., 2024"},{"why":"It provides the latent-scaling attribution method and motivates the layer-14 choice.","marker":"Minder et al., 2025"},{"why":"It supplies the auto-interpretation pipeline that turns features into natural-language descriptions.","marker":"Paulo et al., 2024"},{"why":"It defines the hazardous-knowledge unlearning benchmark and RMU procedure used as the first scenario.","marker":"Li et al., 2024"},{"why":"It defines the emergent misalignment setup and fine-tuning procedure used as the second scenario.","marker":"Betley et al., 2025"},{"why":"It provides the implicit fine-tuning datasets and the finding that benign tuning compromises safety, forming the third scenario.","marker":"Qi et al., 2023"},{"why":"It supplies The Pile, the task-agnostic corpus used for raw fine-tuning scenarios.","marker":"Gao et al., 2020"},{"why":"It supplies LMSYS-Chat-1M, the task-agnostic corpus used for instruction-tuned scenarios.","marker":"Zheng et al., 2023a"}],"fun_headline_variants":["Sparse diffing reads LLM changes, predicts side effects 95%","MNEME: sparse diffing predicts LLM side effects with 95% accuracy","Unlearning side effects? Sparse diffing sees them coming (95%)","Model diffing exposes hidden shifts from fine-tuning (95% accuracy)","Predict LLM side effects via sparse diffing, up to 95% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the side effects of an intervention show up as changes in one fixed layer's activations on ordinary, task-independent text, so a side effect that lives in a different layer or only appears on rare, domain-specific tokens could pass through undetected.","fun_headline_variants_meta":{"raw":{"variants":["Sparse diffing reads LLM changes, predicts side effects 95%","MNEME: sparse diffing predicts LLM side effects with 95% accuracy","Unlearning side effects? Sparse diffing sees them coming (95%)","Model diffing exposes hidden shifts from fine-tuning (95% accuracy)","Predict LLM side effects via sparse diffing, up to 95% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1760,"prompt_tokens":954,"completion_tokens":806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":704}},"tokens_in":570,"tokens_out":806,"duration_ms":6951,"temperature":1.0,"reasoning_tokens":704,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:24:19.108283+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fine-tuned model whose only measurable side effect is confined to a layer other than layer 14, or only manifests on tokens absent from generic corpora like The Pile or LMSYS-Chat-1M; if MNEME's layer-14 generic-text features still predict the side effect at benchmark-level accuracy, the claim is supported, and if they miss it while a data-rich baseline catches it, the fixed-layer and generic-corpus assumption is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It introduces Cross-Coders for model diffing, the architecture MNEME adapts."}],"review_version":2}