{"id":"8b854615-528d-4365-a7a1-b7c7c7ea683d","arxiv_id":"2411.18797","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SEUF restricts MoE LLM unlearning to the top-1 most relevant expert and adds a router anchor loss, improving utility by up to 35% while keeping forget efficacy roughly comparable.","lead":"This paper studies machine unlearning for mixture-of-experts (MoE) language models, where many specialized subnetworks share the work. It shows that updating only the single most relevant expert, while holding the router's choice stable, removes target knowledge with far less damage to general ability than updating all parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim may conflate rerouting with erasure: SEUF freezes non-target experts that may still hold the forget-set knowledge, so 'unlearning one expert' could be a router-level workaround rather than true knowledge removal.","rationale":"Reader's weakest_assumption already identifies the attribution assumption, and I agree that it is the most load-bearing point. The paper's own evidence supports the concern: Fig. 3 shows the expert-selection overlap ratio dropping during unlearning, and the anchor loss is introduced precisely because non-target experts can be selected; but no experiment verifies that the frozen experts are free of the target knowledge after unlearning. The GCG attack only perturbs prompts, not the frozen parameter set. Thus the single most load-bearing question is whether SEUF erases knowledge or merely prevents its retrieval. The proposed restoration test directly answers it by asking whether the target knowledge reappears when the frozen experts are put back. I do not see an internal contradiction that forces rejection, and the empirical FE gains on RWKU are consistent with genuine suppression; hence the reader's CONDITIONAL verdict stands. The main addition would be to phrase the abstract in terms of 'forgetting behavior' rather than 'knowledge erasure' until the restoration test is run.","tokens_in":19176,"tokens_out":7264,"duration_ms":66906,"concrete_test":"Restore the frozen parameters after SEUF unlearning: load the SEUF checkpoint, then swap the original pretrained weights back into all non-target experts (and shared experts) while keeping the modified target expert and router. Evaluate FE on the forget test set. If FE returns toward the pretrained level, the frozen experts still encode the forgotten knowledge, so SEUF reroutes rather than erases. As a companion check, run the same restoration with only the modified router reverted (pretrained router weights, unlearned target expert) to separate routing effects from expert-content effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the expert identified by Eq. (2) is the sole or dominant storage site for the forget-set knowledge, so freezing every other expert leaves nothing to recover. The paper only tests the unlearned model's router-gated behavior (e.g., the GCG attack in Sec. 5); it never checks whether the frozen non-target experts or shared experts still contain the target knowledge. DeepSeek-V2-Lite's shared experts are activated for every token, and Table 2 explicitly shows that including them degrades utility, but the authors do not test whether target knowledge survives in those untouched shared experts. If the knowledge is distributed across many routed experts, or if the router anchor loss simply forces the model to stop routing to experts that still encode the knowledge, then SEUF achieves low FE on the benchmark by rerouting rather than erasing. That would make the central 'unlearning' claim misleading and the 0.06%-parameter efficiency less meaningful. The paper's own Insight 2 warns against unlearning 'shortcuts' that produce fake forgetting; the frozen-expert configuration is exactly the kind of shortcut that needs to be ruled out.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses machine unlearning for sparse Mixture-of-Experts LLMs, a setting which it argues is unexplored. It first documents via a pilot study that standard unlearning methods (GA, GDIFF, NPO, RMU) cause substantial utility drops on MoE LLMs, and traces the cause to a 'shortcut' whereby routers shift selection away from the experts most relevant to the forget set, leading to excessive updates of non-target experts. The paper then proposes SEUF: it computes per-expert gating affinities on a calibration subset of the forget set (Eq. 2), selects the top-M experts (default M=1), restricts unlearning updates to that expert and its router, and adds a router anchor loss (Eq. 3) to keep the target expert selected. Experiments on Qwen1.5-MoE-A2.7B-Chat, DeepSeek-V2-Lite (and Mixtral-8x7B in the appendix) across WMDP and RWKU report that SEUF improves utility by up to 35% while updating about 0.06% of parameters, with forget efficacy comparable or better in many cases, and that the unlearned model is robust to a GCG attack. The paper also compares SEUF against LoRA, ESFT, and random expert selection, and includes sensitivity analyses for alpha, M, and expert-selection robustness.","tokens_in":19439,"tokens_out":4704,"duration_ms":44878,"significance":"If the central claim holds, SEUF is a practically valuable contribution: it identifies a failure mode specific to MoE routing during unlearning and demonstrates a parameter-efficient fix that works in a plug-and-play manner with several existing unlearning losses. The paper is honest about limitations, and it provides useful control experiments such as random expert selection, sensitivity to alpha, analysis of shared-expert inclusion, and a white-box adversarial attack test. However, the central 'one expert is enough' claim rests on an attribution assumption that is not directly verified: that the selected expert is the dominant storage site for the target knowledge and that freezing all other experts neither preserves the knowledge nor leaves it recoverable. The current evidence is behavioral (router-gated outputs), not representational, so the paper does not fully rule out that SEUF achieves low forget efficacy by rerouting rather than erasing.","major_comments":[{"comment":"The abstract and contributions state that SEUF 'enhances forget quality,' but Table 3 does not show uniform improvements. For example, on DeepSeek WMDP, GA+SEUF has FE 0.2700 vs. GA 0.2457, and RMU+SEUF has FE 0.2859 vs. RMU 0.2530; on Qwen RWKU, RMU+SEUF has FE 0.0723 vs. RMU 0.0200, and NPO+SEUF has FE 0.0020 vs. NPO 0.0000. If the claim is instead that SEUF preserves or slightly trades off forget efficacy while recovering utility, the text should say so explicitly and quantify how many configurations improved versus worsened.","section":"Abstract; §5, Table 3"},{"comment":"The paper reports all results from a single run and selects the checkpoint that achieves the best balance between FE and UT on the evaluation metrics (Appendix A: 'All experiments were conducted in a single run without multiple trials' and 'we select the model checkpoint that achieves the best balance between FE and UT as the optimal checkpoint'). With no variance estimates and checkpoint selection performed on the test metrics, small differences in Table 3 (e.g., 0.2536 vs. 0.2612, or 0.2445 vs. 0.2457) cannot be distinguished from noise, and some of the reported gains may be inflated by selection. The authors should provide multiple independent runs or at least report seed variation, and either use a validation split for checkpoint selection or justify that the reported numbers are not selected post hoc.","section":"Appendix A; §5, Table 3"},{"comment":"The evidence that SEUF truly erases the target knowledge is incomplete because the GCG experiment in §5 tests only the router-gated behavior of the full model, not whether the frozen non-target experts or the shared experts still encode the forget-set knowledge. DeepSeek-V2-Lite's shared experts are activated for every token, and Table 2 shows that including shared experts degrades utility, but the paper does not test whether the target knowledge survives in those untouched shared experts. If SEUF merely makes the router avoid experts that still contain the knowledge, then low FE reflects rerouting rather than erasure, which is precisely the 'fake unlearning' shortcut warned about in Insight 2. The authors should add a direct test, such as probing frozen expert activations on forget-set inputs, forcing the router to select the target or non-target experts and measuring residual knowledge, or performing a representation-level similarity analysis between pre- and post-unlearning features.","section":"§5, 'Unlearning resilient to jailbreak attacks'; §4, Insight 2; Table 2"},{"comment":"The central attribution assumption is that the expert identified by Eq. (2) is the sole or dominant storage site for the forget-set knowledge, so freezing every other expert leaves nothing to recover. The paper shows that M=1 yields the best utility/FE trade-off under the given benchmarks, but this does not demonstrate that the knowledge is concentrated in a single expert; it could also mean that updating one expert plus the anchor loss is enough to block the router from exposing knowledge that remains distributed elsewhere. The authors should directly examine the distribution of forget-set knowledge across experts, for example by measuring the drop in forget-set accuracy when individual frozen experts are ablated or probed, before concluding that one expert is sufficient for erasure.","section":"§4, Eq. (2) and 'Selection of top M experts'"}],"minor_comments":[{"comment":"There is a grammatical error: 'have remain unexplored' should be 'have remained unexplored.'","section":"Abstract"},{"comment":"The limitation section contains a broken sentence: 'We have used two widely accepted LLM unlearning benchmarks: WMDP and RWKU. WMDP. We acknowledge...' The duplicated 'WMDP.' appears to be a citation or editing artifact and should be fixed.","section":"§7 Limitation"},{"comment":"The tunable parameter ratios contain formatting issues, including '0 .92%' and '0 .41%' with stray spaces; also, the table would benefit from a note clarifying whether the ratios refer to trainable parameters relative to total parameters for each model.","section":"Table 4"},{"comment":"The anchor loss uses an L2 distance to a one-hot target vector, which encourages the router to place all probability mass on the target expert, whereas the MoE only needs the target expert to remain in the top-K. The authors should add one sentence justifying why a one-hot target, rather than a top-K-preserving penalty, is the appropriate anchor objective.","section":"§4, Eq. (3)"},{"comment":"The sentence 'For LoRA, we apply low-rank adaptation to all layers of the model to enable full-layer fine-tuning' is imprecise: LoRA adds trainable adapters rather than performing full-layer fine-tuning. This wording should be corrected to avoid confusion about what is actually updated.","section":"Appendix A, 'Unlearning Baselines'"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful empirical contribution, but the two main blockers are (i) the lack of any variance or repeated-run information, combined with checkpoint selection on the evaluation metrics, and (ii) the unverified assumption that freezing all non-target experts does not leave the forget knowledge intact. Both are addressable with additional experiments rather than requiring a fundamentally new approach, so I recommend major revision rather than rejection. The reviewer should also check whether the authors' claim of 'enhanced forget quality' is revised to reflect the mixed FE results in Table 3."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Best read as a practical method paper rather than a settled claim about where knowledge lives. The new thing is real: nobody had systematically studied unlearning in MoE LLMs before, and the paper correctly identifies that standard unlearning methods make the router shift away from the experts that actually contain the forget knowledge, causing collateral damage. The fix—pick the top-1 expert by gate affinity, freeze everything else, and add an anchor loss that keeps the router selecting that expert—is simple, architecture-aware, and delivers large, consistent utility gains (all 16 SEUF rows in Table 3 beat the corresponding baseline on utility) with only 0.06% trainable parameters. That is a genuinely useful engineering contribution.\n\nThe soft spots are real but not fatal. First, the paper never verifies that the forget knowledge is actually erased from the frozen experts. Because the router is anchored to the target expert, a low forget score on the benchmark could just mean the model no longer routes to the experts that still hold the knowledge. The paper's own Insight 2 calls such routing-based behavior a 'shortcut,' and the GCG attack in Sec. 5 doesn't rule it out—it only shows that prompt optimization doesn't reroute the model. A simple probe or activation-patching experiment on the frozen experts would settle this. Without it, the 'unlearning' claim is stronger than the evidence.\n\nSecond, the evaluation is thin: one seed per setting, best checkpoint selected on the evaluation metrics, no error bars. Forget-efficacy gains are uneven—SEUF makes FE worse in five of the sixteen Table 3 rows, even though utility always improves. The abstract's 'enhances forget quality up to 5%' is misleading in that light. Third, no code release, which matters for a method whose main selling point is parameter efficiency.\n\nThe affinity metric is borrowed from ESFT, and the paper says so; the novelty is in applying it to unlearning and in the anchor loss. Citation pattern looks fine. The limitation section is honest about benchmark coverage but doesn't mention the erasure-vs-rerouting question.\n\nWho is this for? Anyone working on unlearning for MoE models, or on efficient fine-tuning for sparse architectures. It deserves a serious referee—the problem is well chosen and the method is likely to be useful—but the revision should add multi-seed runs, a fixed checkpoint rule, a code release, and an internal-knowledge check. I'd also soften the abstract's forget-quality claim.","headline":"The first systematic study of MoE unlearning, with a real diagnosis and a simple fix, but it never shows the knowledge is actually erased rather than rerouted, and the evaluation is single-run.","tokens_in":19955,"tokens_out":4153,"would_cite":true,"duration_ms":38827,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For mixture-of-experts language models, unlearning works best when it targets a single most-relevant expert and pins the router to keep using it.","keywords":["mixture-of-experts LLM","machine unlearning","expert attribution","router anchor loss","parameter-efficient unlearning","Qwen1.5-MoE","DeepSeek-V2-Lite","WMDP benchmark"],"falsifier":"After running SEUF on a MoE model, evaluate the forget set with the unlearned top-1 expert ablated or disabled; if accuracy on the forget set stays at the unlearned level, the expert was carrying the knowledge, and if it climbs back up, the knowledge lives elsewhere. Alternatively, unlearn the complement (every expert except the top-1) while keeping the anchor loss on the top-1; if this also achieves full forget efficacy, then the attribution step is not selecting a necessary expert.","tokens_in":18997,"feed_emoji":"🎯","tokens_out":11130,"duration_ms":83537,"temperature":0.7,"pith_summary":"Large language models that route each token through a few expert networks are hard to unlearn: standard unlearning methods cause the router to shift away from the experts that actually store the target knowledge, so the model forgets too much and harms unrelated abilities. This paper proposes SEUF, which first identifies the single expert most actively engaged on the forget set and then applies the unlearning loss only to that expert while a router anchor loss keeps the expert selected. On Qwen1.5-MoE and DeepSeek-V2-Lite, SEUF claims forget quality comparable to or better than full-model unlearning while raising model utility by up to 35% and updating about 0.06% of parameters. The paper argues this succeeds because topic-specific knowledge in a MoE model concentrates in a few long-tailed experts, so erasing the top expert is enough and freezing the rest protects utility.","feed_headline":"One expert is enough to unlearn a mixture-of-experts language model","feed_subtitle":"Pinning the router to the most relevant expert preserves utility while erasing target knowledge with 0.06% of parameters.","key_machinery":"The two load-bearing pieces are expert attribution and the router anchor loss. Expert attribution (Eq. 2) averages the router gate score $g^{(l)}_{i,t}$ over tokens in a calibration subset of the forget set to rank experts in each layer; after observing a long-tailed selection distribution, the paper takes the top-1 expert $\\mathbf{e}_M$ as the unlearning target. The router anchor loss (Eq. 3) computes the squared deviation between the current gate vector and the one-hot vector marking that target expert, and is added as $\\alpha L_{\\text{anchor}}$ to the unlearning objective. Together they keep the selected expert pinned active so the forget loss lands on the expert that holds the knowledge while no gradient touches non-target experts.","core_discovery":"Standard unlearning on MoE LLMs fails because routers invent shortcuts: instead of keeping the knowledge-bearing experts active for forget-set inputs, they switch to non-target experts, which then absorb the forgetting gradient and drag down utility. SEUF's central claim is that this is avoidable by unlearning only the expert with the highest average gate score on the forget set and adding an anchor loss that penalizes router movement away from that expert. With this recipe, GA, GDIFF, NPO, and RMU all achieve forget efficacy close to or better than their full-parameter versions, and utility on MMLU stays near the pretrained model instead of collapsing. The paper also reports that unlearning one expert outperforms unlearning three or six, and that including shared experts in the update hurts utility more than it helps.","pith_inferences":["If the long-tailed-concentration premise generalizes, SEUF should transfer to other token-level MoE architectures and topic-coherent forget sets; a testable extension is measuring how concentrated expert affinity is on a new domain and predicting when the single-expert premise breaks.","The paper's design leaves shared experts untouched; a natural probe is whether the erased knowledge is truly gone or merely dormant, by fine-tuning the frozen non-target experts or the router on the forget set and checking whether forget efficacy returns.","The anchor loss pins the router to pretrained behavior; an alternative design could let the router settle on a new expert after unlearning, which might maintain utility without holding expert selection fixed.","Part of the utility gain may come from the sheer smallness of the parameter update acting as a regularizer; an ablation matching gradient magnitude or parameter count between SEUF and full-model unlearning would separate selection effects from regularization effects."],"forward_implications":["Full-parameter unlearning is unnecessary for sparse MoE models: updating a single top-1 expert per layer can match or exceed the forget quality of updating everything.","Existing unlearning algorithms GA, GDIFF, NPO, and RMU become practical on MoE architectures when combined with SEUF, because utility loss drops to near-pretrained levels.","With only about 0.06% of parameters changed, unlearning becomes cheap enough to rerun for different forget sets while keeping the pretrained model intact as a starting point.","Pinning the router makes the unlearned model robust to GCG-style adversarial prompts: expert selection stays stable, and the attack does not recover forgotten knowledge.","Affinity-based selection is doing the real work: random expert selection preserves utility but leaves a higher forget-efficacy score (worse forgetting)."],"supporting_citations":[{"why":"Supplies the WMDP benchmark and the RMU unlearning method, both central to the paper's evaluation.","marker":"(Li et al., 2024)"},{"why":"Supplies the RWKU benchmark used for the second unlearning task.","marker":"(Jin et al., 2024)"},{"why":"Provides the gating-score affinity calculation that SEUF adopts for expert attribution, and the ESFT baseline.","marker":"(Wang et al., 2024b)"},{"why":"Introduces DeepSeek-V2-Lite, one of the two main MoE architectures unlearned.","marker":"(Dai et al., 2024)"},{"why":"Introduces Qwen1.5-MoE, the other main MoE architecture unlearned.","marker":"(Team, 2024)"},{"why":"Supplies Gradient Ascent (GA), one of the four base unlearning algorithms SEUF is tested with.","marker":"(Eldan and Russinovich, 2023)"},{"why":"Supplies Gradient Difference (GDIFF), one of the four base unlearning algorithms.","marker":"(Maini et al., 2024)"},{"why":"Supplies Negative Preference Optimization (NPO), one of the four base unlearning algorithms.","marker":"(Zhang et al., 2024)"},{"why":"Defines the forget-set/retain-set unlearning formulation in Eq. (1) that SEUF extends with the anchor loss.","marker":"(Liu et al., 2024c)"},{"why":"Supplies the GCG adversarial attack used to test whether the unlearned model stays robust.","marker":"(Zou et al., 2023)"}],"fun_headline_variants":["One expert is enough: unlearning MoE LLMs without utility collapse","MoE unlearning: target the top expert, pin the router","Unlearn 0.06% of params to erase knowledge, keep utility","Router shortcuts ruin MoE unlearning: SEUF fixes it","Pilot study: unlearn one expert, not many, in MoE LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the knowledge to be erased is concentrated mainly in a single expert per layer that the router's average gate score on a sample of the forget set can identify, and that the shared experts and all other routed experts do not, on their own, retain the target knowledge.","fun_headline_variants_meta":{"raw":{"variants":["One expert is enough: unlearning MoE LLMs without utility collapse","MoE unlearning: target the top expert, pin the router","Unlearn 0.06% of params to erase knowledge, keep utility","Router shortcuts ruin MoE unlearning: SEUF fixes it","Pilot study: unlearn one expert, not many, in MoE LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1277,"prompt_tokens":951,"completion_tokens":326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":229}},"tokens_in":567,"tokens_out":326,"duration_ms":3479,"temperature":1.0,"reasoning_tokens":229,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:53:02.503650+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"After running SEUF on a MoE model, evaluate the forget set with the unlearned top-1 expert ablated or disabled; if accuracy on the forget set stays at the unlearned level, the expert was carrying the knowledge, and if it climbs back up, the knowledge lives elsewhere. Alternatively, unlearn the complement (every expert except the top-1) while keeping the anchor loss on the top-1; if this also achieves full forget efficacy, then the attribution step is not selecting a necessary expert.","supporting_citations":[],"review_version":1}