Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For mixture-of-experts language models, unlearning works best when it targets a single most-relevant expert and pins the router to keep using it.

desk verdict The first systematic study of MoE unlearning, with a real diagnosis and a simple fix, but it never shows the knowledge is actually erased rather than rerouted, and the evaluation is single-run. read the letter →

arxiv 2411.18797 v2 pith:Y5WBTWEG submitted 2024-11-27 cs.LG cs.AIcs.CL

Haomin Zhuang , Yihua Zhang , Kehan Guo , Jinghan Jia , Gaowen Liu , Sijia Liu , Xiangliang Zhang This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords mixture-of-expertsLLMmachineunlearningexpertattributionrouteranchorlossparameter-efficientQwen1.5-MoEDeepSeek-V2-LiteWMDPbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models that route each token through a few expert networks are hard to unlearn: standard unlearning methods cause the router to shift away from the experts that actually store the target knowledge, so the model forgets too much and harms unrelated abilities. This paper proposes SEUF, which first identifies the single expert most actively engaged on the forget set and then applies the unlearning loss only to that expert while a router anchor loss keeps the expert selected. On Qwen1.5-MoE and DeepSeek-V2-Lite, SEUF claims forget quality comparable to or better than full-model unlearning while raising model utility by up to 35% and updating about 0.06% of parameters. The paper argues this succeeds because topic-specific knowledge in a MoE model concentrates in a few long-tailed experts, so erasing the top expert is enough and freezing the rest protects utility.

What carries the argument

The two load-bearing pieces are expert attribution and the router anchor loss. Expert attribution (Eq. 2) averages the router gate score $g^{(l)}_{i,t}$ over tokens in a calibration subset of the forget set to rank experts in each layer; after observing a long-tailed selection distribution, the paper takes the top-1 expert $\mathbf{e}_M$ as the unlearning target. The router anchor loss (Eq. 3) computes the squared deviation between the current gate vector and the one-hot vector marking that target expert, and is added as $\alpha L_{\text{anchor}}$ to the unlearning objective. Together they keep the selected expert pinned active so the forget loss lands on the expert that holds the knowledge while no gradient touches non-target experts.

What would settle it

After running SEUF on a MoE model, evaluate the forget set with the unlearned top-1 expert ablated or disabled; if accuracy on the forget set stays at the unlearned level, the expert was carrying the knowledge, and if it climbs back up, the knowledge lives elsewhere. Alternatively, unlearn the complement (every expert except the top-1) while keeping the anchor loss on the top-1; if this also achieves full forget efficacy, then the attribution step is not selecting a necessary expert.

Watch

Extended reading notes

Core claim

Standard unlearning on MoE LLMs fails because routers invent shortcuts: instead of keeping the knowledge-bearing experts active for forget-set inputs, they switch to non-target experts, which then absorb the forgetting gradient and drag down utility. SEUF's central claim is that this is avoidable by unlearning only the expert with the highest average gate score on the forget set and adding an anchor loss that penalizes router movement away from that expert. With this recipe, GA, GDIFF, NPO, and RMU all achieve forget efficacy close to or better than their full-parameter versions, and utility on MMLU stays near the pretrained model instead of collapsing. The paper also reports that unlearning one expert outperforms unlearning three or six, and that including shared experts in the update hurts utility more than it helps.

Load-bearing premise

The load-bearing premise is that the knowledge to be erased is concentrated mainly in a single expert per layer that the router's average gate score on a sample of the forget set can identify, and that the shared experts and all other routed experts do not, on their own, retain the target knowledge.

Editorial extensions

If this is right

  • Full-parameter unlearning is unnecessary for sparse MoE models: updating a single top-1 expert per layer can match or exceed the forget quality of updating everything.
  • Existing unlearning algorithms GA, GDIFF, NPO, and RMU become practical on MoE architectures when combined with SEUF, because utility loss drops to near-pretrained levels.
  • With only about 0.06% of parameters changed, unlearning becomes cheap enough to rerun for different forget sets while keeping the pretrained model intact as a starting point.
  • Pinning the router makes the unlearned model robust to GCG-style adversarial prompts: expert selection stays stable, and the attack does not recover forgotten knowledge.
  • Affinity-based selection is doing the real work: random expert selection preserves utility but leaves a higher forget-efficacy score (worse forgetting).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the long-tailed-concentration premise generalizes, SEUF should transfer to other token-level MoE architectures and topic-coherent forget sets; a testable extension is measuring how concentrated expert affinity is on a new domain and predicting when the single-expert premise breaks.
  • The paper's design leaves shared experts untouched; a natural probe is whether the erased knowledge is truly gone or merely dormant, by fine-tuning the frozen non-target experts or the router on the forget set and checking whether forget efficacy returns.
  • The anchor loss pins the router to pretrained behavior; an alternative design could let the router settle on a new expert after unlearning, which might maintain utility without holding expert selection fixed.
  • Part of the utility gain may come from the sheer smallness of the parameter update acting as a regularizer; an ablation matching gradient magnitude or parameter count between SEUF and full-model unlearning would separate selection effects from regularization effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses machine unlearning for sparse Mixture-of-Experts LLMs, a setting which it argues is unexplored. It first documents via a pilot study that standard unlearning methods (GA, GDIFF, NPO, RMU) cause substantial utility drops on MoE LLMs, and traces the cause to a 'shortcut' whereby routers shift selection away from the experts most relevant to the forget set, leading to excessive updates of non-target experts. The paper then proposes SEUF: it computes per-expert gating affinities on a calibration subset of the forget set (Eq. 2), selects the top-M experts (default M=1), restricts unlearning updates to that expert and its router, and adds a router anchor loss (Eq. 3) to keep the target expert selected. Experiments on Qwen1.5-MoE-A2.7B-Chat, DeepSeek-V2-Lite (and Mixtral-8x7B in the appendix) across WMDP and RWKU report that SEUF improves utility by up to 35% while updating about 0.06% of parameters, with forget efficacy comparable or better in many cases, and that the unlearned model is robust to a GCG attack. The paper also compares SEUF against LoRA, ESFT, and random expert selection, and includes sensitivity analyses for alpha, M, and expert-selection robustness.

Significance. If the central claim holds, SEUF is a practically valuable contribution: it identifies a failure mode specific to MoE routing during unlearning and demonstrates a parameter-efficient fix that works in a plug-and-play manner with several existing unlearning losses. The paper is honest about limitations, and it provides useful control experiments such as random expert selection, sensitivity to alpha, analysis of shared-expert inclusion, and a white-box adversarial attack test. However, the central 'one expert is enough' claim rests on an attribution assumption that is not directly verified: that the selected expert is the dominant storage site for the target knowledge and that freezing all other experts neither preserves the knowledge nor leaves it recoverable. The current evidence is behavioral (router-gated outputs), not representational, so the paper does not fully rule out that SEUF achieves low forget efficacy by rerouting rather than erasing.

major comments (4)
  1. [Abstract; §5, Table 3] The abstract and contributions state that SEUF 'enhances forget quality,' but Table 3 does not show uniform improvements. For example, on DeepSeek WMDP, GA+SEUF has FE 0.2700 vs. GA 0.2457, and RMU+SEUF has FE 0.2859 vs. RMU 0.2530; on Qwen RWKU, RMU+SEUF has FE 0.0723 vs. RMU 0.0200, and NPO+SEUF has FE 0.0020 vs. NPO 0.0000. If the claim is instead that SEUF preserves or slightly trades off forget efficacy while recovering utility, the text should say so explicitly and quantify how many configurations improved versus worsened.
  2. [Appendix A; §5, Table 3] The paper reports all results from a single run and selects the checkpoint that achieves the best balance between FE and UT on the evaluation metrics (Appendix A: 'All experiments were conducted in a single run without multiple trials' and 'we select the model checkpoint that achieves the best balance between FE and UT as the optimal checkpoint'). With no variance estimates and checkpoint selection performed on the test metrics, small differences in Table 3 (e.g., 0.2536 vs. 0.2612, or 0.2445 vs. 0.2457) cannot be distinguished from noise, and some of the reported gains may be inflated by selection. The authors should provide multiple independent runs or at least report seed variation, and either use a validation split for checkpoint selection or justify that the reported numbers are not selected post hoc.
  3. [§5, 'Unlearning resilient to jailbreak attacks'; §4, Insight 2; Table 2] The evidence that SEUF truly erases the target knowledge is incomplete because the GCG experiment in §5 tests only the router-gated behavior of the full model, not whether the frozen non-target experts or the shared experts still encode the forget-set knowledge. DeepSeek-V2-Lite's shared experts are activated for every token, and Table 2 shows that including shared experts degrades utility, but the paper does not test whether the target knowledge survives in those untouched shared experts. If SEUF merely makes the router avoid experts that still contain the knowledge, then low FE reflects rerouting rather than erasure, which is precisely the 'fake unlearning' shortcut warned about in Insight 2. The authors should add a direct test, such as probing frozen expert activations on forget-set inputs, forcing the router to select the target or non-target experts and measuring residual knowledge, or performing a representation-level similarity analysis between pre- and post-unlearning features.
  4. [§4, Eq. (2) and 'Selection of top M experts'] The central attribution assumption is that the expert identified by Eq. (2) is the sole or dominant storage site for the forget-set knowledge, so freezing every other expert leaves nothing to recover. The paper shows that M=1 yields the best utility/FE trade-off under the given benchmarks, but this does not demonstrate that the knowledge is concentrated in a single expert; it could also mean that updating one expert plus the anchor loss is enough to block the router from exposing knowledge that remains distributed elsewhere. The authors should directly examine the distribution of forget-set knowledge across experts, for example by measuring the drop in forget-set accuracy when individual frozen experts are ablated or probed, before concluding that one expert is sufficient for erasure.
minor comments (5)
  1. [Abstract] There is a grammatical error: 'have remain unexplored' should be 'have remained unexplored.'
  2. [§7 Limitation] The limitation section contains a broken sentence: 'We have used two widely accepted LLM unlearning benchmarks: WMDP and RWKU. WMDP. We acknowledge...' The duplicated 'WMDP.' appears to be a citation or editing artifact and should be fixed.
  3. [Table 4] The tunable parameter ratios contain formatting issues, including '0 .92%' and '0 .41%' with stray spaces; also, the table would benefit from a note clarifying whether the ratios refer to trainable parameters relative to total parameters for each model.
  4. [§4, Eq. (3)] The anchor loss uses an L2 distance to a one-hot target vector, which encourages the router to place all probability mass on the target expert, whereas the MoE only needs the target expert to remain in the top-K. The authors should add one sentence justifying why a one-hot target, rather than a top-K-preserving penalty, is the appropriate anchor objective.
  5. [Appendix A, 'Unlearning Baselines'] The sentence 'For LoRA, we apply low-rank adaptation to all layers of the model to enable full-layer fine-tuning' is imprecise: LoRA adds trainable adapters rather than performing full-layer fine-tuning. This wording should be corrected to avoid confusion about what is actually updated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: expert selection, unlearning, and evaluation are distinct stages with no definitional reduction.

full rationale

The paper's derivation chain is self-contained rather than circular. Expert attribution (Eq. 2) uses router gate scores on a calibration subset of the forget set to select the top expert, while forget efficacy is measured on held-out forget-test accuracy and utility on MMLU, so the selection criterion and the evaluation metrics are not the same quantity. The router anchor loss (Eq. 3) is an additional regularizer that stabilizes expert selection; it does not define the FE or UT metrics and is not fitted to them. The choice M=1 is justified empirically in Table 2 and Appendix C by comparing M=1, 3, 6 on the same benchmarks, rather than being imposed by the definition of the method. The 0.06% parameter-update figure is an arithmetic consequence of freezing all but one expert, which is a design choice rather than a fitted prediction disguised as a result. The paper's self-citations (e.g., Liu et al. 2024c, Jia et al. 2024a/b, Zhang et al. 2023b) are background and not load-bearing: the central claim does not reduce to any of these cited results. The limitation section explicitly restricts the evaluation to two benchmarks and notes the exclusion of larger models, which is an honest scope statement rather than evidence of circularity. No equation in the paper reduces, by construction, to its own inputs, and no fitted parameter is renamed as a prediction. The only substantive concern is whether unlearning a single routed expert truly erases knowledge that may also reside in frozen non-target or shared experts; that is a correctness and validity threat, not a circularity of the derivation chain.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim depends on three chosen quantities: M (number of unlearned experts), alpha (anchor strength), and the calibration subset size. It also depends on the inherited assumption that gradient-based unlearning objectives remove knowledge, that the ESFT affinity score correctly localizes knowledge, and that the anchor loss can pin the router without utility damage. No new physical or conceptual entities are introduced; the target expert is an existing expert selected by attribution.

free parameters (4)
  • M (number of selected target experts) = 1
    Chosen by comparing M = 1, 3, 6 in Tables 2 and 7; top-1 gives the best utility at matched forget efficacy.
  • alpha (anchor loss weight) = 1
    Set from the sensitivity analysis in Appendix B; alpha = 1 gives the best UT on DeepSeek with RWKU.
  • calibration token subset size = >=100,000
    Used for expert attribution; the authors state this is robust, supported by overlap analysis in Appendix D.
  • Baseline and unlearning hyperparameters = varies by model and method, e.g., lr 5e-5 vs 1e-4, batch size 4 vs 16, RMU c = 30000/3000
    Tuned per model and method in Appendix A; the comparison between SEUF and baselines depends on these choices.
assumptions (3)
  • domain assumption Gradient-based unlearning objectives (GA, GDIFF, NPO, RMU) remove target knowledge when applied to model weights
    Inherited from the cited unlearning literature; SEUF wraps these objectives rather than re-deriving them.
  • domain assumption The gate-affinity score in Eq. (2) ranks expert relevance to the forget topic
    Borrowed from ESFT (Wang et al., 2024b); Table 8 checks selection overlap but not whether the top expert actually stores the knowledge.
  • ad hoc to paper The anchor loss in Eq. (3) can pin router selection to the target expert without collapsing routing on retained data
    This is the core mechanism of SEUF; only an alpha-sensitivity study supports it, with no formal guarantee or isolating ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?." pith.science (2026). https://pith.science/paper/Y5WBTWEG

@misc{pith2026241118797,
  author       = {Pith},
  title        = {Pith review of: SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y5WBTWEG}},
  note         = {Machine review of arXiv:2411.18797}
}
read the original abstract

Recent advancements in LLMs unlearning have shown remarkable success in removing unwanted data-model influences while preserving the model's utility for legitimate knowledge. Despite these strides, sparse Mixture-of-Experts (MoE) LLMs--a key subset of the LLM family--have remained unexplored in the context of unlearning. As MoE LLMs are celebrated for their exceptional performance, we ask:How can unlearning be performed effectively and efficiently on MoE LLMs? Our pilot study shows that the dynamic routing nature of MoE LLMs introduces unique challenges, leading to excessive forgetting, uncontrolled knowledge erasure and substantial utility drops when existing unlearning methods are applied. To address this, we propose a novel Selected-Expert Unlearning Framework (SEUF). Through expert attribution, unlearning is concentrated on the most actively engaged experts for the specified knowledge. Concurrently, an anchor loss is applied to the router to stabilize the active state of this targeted expert, ensuring focused and controlled unlearning. SEUF is compatible with various standard unlearning algorithms. Extensive experiments demonstrate that SEUF enhances both forget quality up to 5% and model utility by 35% on MoE LLMs across various benchmarks and LLM architectures (compared to standard unlearning algorithms), while only unlearning 0.06% of the model parameters.

Figures

Figures reproduced from arXiv: 2411.18797 by the authors.

Figure 1
Figure 1. Overview of the key findings in this paper. (a) Illustration of the ineffectiveness of existing unlearning methods on MoE LLMs. Four unlearning algorithms—GA (Eldan and Russinovich, 2023), GDIFF (Maini et al., 2024), NPO (Zhang et al., 2024), and RMU (Li et al., 2024)—were applied to two MoE LLMs (DeepSeek-v2-Lite (Liu et al., 2024a) and Qwen1.5-MoE (Team, 2024)) and two dense LLMs (Phi3.5 (Abdin et al., 2024) and L… view at source ↗
Figure 2
Figure 2. Proportion of tokens assigned to each expert of the pre-trained DeepSeek-v2-Lite (K=6 in Topk) with samples from WMDP forget benchmark (Li et al., 2024), in different model layers. The dashed horizontal line marks 6/64, i.e., the proportion expected with uniform expert selection. The ex￾pert selection distribution clearly follows a long-tailed pattern when the input is sampled from a topic within a narrow scope. 0 1… view at source ↗
Figure 3
Figure 3. (left) Overlap ratio of selected experts between the original pretrained model and the unlearned model with different unlearning iterations using GA on WMDP benchmark. (right) Forget loss vs. the number of unlearning iterations, when controlling parameters to unlearn in MoE LLM. were assigned with the majority of the tokens in each layer, which was also confirmed in Wang et al. (2024b). Thus, we have the following i… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of affinity scores for all experts in the target layer of DeepSeek unlearned by SEUF + GA on the RWKU dataset, with and without the GCG attack. The target expert is marked as red. despite being one of the strongest prompt-level at￾tacks, GCG fails to recover…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A survey and framework that categorizes generative model unlearning by point-wise versus concept-wise objectives, parameter-based versus non-parametric methods, and completeness/utility/efficiency evaluation.

Reference graph

Works this paper leans on

73 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    George-Octavian Barbulescu and Peter Triantafillou. 2024. To each (textual sequence) its own: Improving memorized-data unlearning in large language models. arXiv preprint arXiv:2405.03097

    arXiv 2024

  5. [5]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  6. [6]

    Peizhuang Cong, Aomufei Yuan, Shimao Chen, Yuxuan Tian, Bowen Ye, and Tong Yang. 2024. Prediction is all moe needs: Expert load distribution goes from fluctuating to stabilizing. arXiv preprint arXiv:2404.16914

    arXiv 2024

  7. [7]

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066

    arXiv 2024

  8. [8]

    Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Stablemoe: Stable routing strategy for mixture of experts. arXiv preprint arXiv:2204.08396

    arXiv 2022

Show all 73 references
  1. [9]

    Databricks. 2024. Introducing dbrx: A new state-of-the-art open llm. https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm. Accessed: 2024-09-25

    2024

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Ronen Eldan and Mark Russinovich. 2023. https://arxiv.org/abs/2310.02238 Who's harry potter? approximate unlearning in llms . Preprint, arXiv:2310.02238

    2023 arXiv

  4. [12]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1--39

    2022

  5. [13]

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2023. Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems, 5:288--304

    2023

  6. [14]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  8. [16]

    Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. 2023. An overview of catastrophic ai risks. arXiv preprint arXiv:2306.12001

    2023 arXiv

  9. [17]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691

    2024 arXiv

  10. [18]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  11. [19]

    Xinshuo Hu, Dongfang Li, Baotian Hu, Zihao Zheng, Zhenyu Liu, and Min Zhang. 2024. Separate the wheat from the chaff: Model deficiency unlearning via parameter-efficient module operation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18252--18260

    2024

  12. [20]

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. 2023. Tutel: Adaptive mixture-of-experts at scale. Proceedings of Machine Learning and Systems, 5:269--287

    2023

  13. [21]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089

    2022 arXiv

  14. [22]

    Yoichi Ishibashi and Hidetoshi Shimodaira. 2023. Knowledge sanitization of large language models. arXiv preprint arXiv:2309.11852

    2023 arXiv

  15. [23]

    Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2022. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504

    2022 arXiv

  16. [24]

    Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalie Baracaldo, and Sijia Liu. 2024 a . Wagle: Strategic weight attribution for effective and modular unlearning in large language models. arXiv preprint arXiv:2410.17509

    2024 arXiv

  17. [25]

    Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2024 b . Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239

    2024 arXiv

  18. [26]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  19. [27]

    Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Rwku: Benchmarking real-world knowledge unlearning for large language models. arXiv preprint arXiv:2406.10890

    2024 arXiv

  20. [28]

    Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. 2021. Scalable and efficient moe training for multitask multilingual models. arXiv preprint arXiv:2109.10465

    2021 arXiv

  21. [29]

    Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2022. Sparse upcycling: Training mixture-of-experts from dense checkpoints. arXiv preprint arXiv:2212.05055

    2022 arXiv

  22. [30]

    Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah, and Dan Roth. 2022. Privacy adhering machine un-learning in nlp. arXiv preprint arXiv:2212.09573

    2022 arXiv

  23. [31]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668

    2020 arXiv

  24. [32]

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. 2024. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218

    2024 arXiv

  25. [33]

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. 2024. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887

    2024 arXiv

  26. [34]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024 a . Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434

    2024 arXiv

  27. [35]

    Bo Liu, Qiang Liu, and Peter Stone. 2022. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243--254. PMLR

    2022

  28. [36]

    Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. 2024 b . Large language model unlearning via embedding-corrupted prompts. arXiv preprint arXiv:2406.07933

    2024 arXiv

  29. [37]

    Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. 2024 c . Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787

    2024 arXiv

  30. [38]

    Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. 2024 d . Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058

    2024 arXiv

  31. [39]

    Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591--27609

    2022

  32. [40]

    Lipton, and J

    Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. 2024. https://arxiv.org/abs/2401.06121 Tofu: A task of fictitious unlearning for llms . Preprint, arXiv:2401.06121

    2024 arXiv

  33. [41]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359--17372

    2022

  34. [42]

    Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2023. More human than human: Measuring chatgpt political bias. Available at SSRN 4372349

    2023

  35. [43]

    Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. 2023. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579

    2023 arXiv

  36. [44]

    Joan Puigcerver, Rodolphe Jenatton, Carlos Riquelme, Pranjal Awasthi, and Srinadh Bhojanapalli. 2022. On the adversarial robustness of mixture of experts. arXiv preprint arXiv:2210.10253

    2022 arXiv

  37. [45]

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andr \'e Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583--8595

    2021

  38. [46]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538

    2017 arXiv

  39. [47]

    Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, and Chuang Gan. 2023. Moduleformer: Learning modular large language models from uncurated data. arXiv preprint arXiv:2306.04640

    2023 arXiv

  40. [48]

    Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. 2024. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460

    2024 arXiv

  41. [49]

    Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561

    2024 arXiv

  42. [50]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen-moe/ Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters"

    2024

  43. [51]

    Pratiksha Thaker, Yash Maurya, and Virginia Smith. 2024. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.03329

    2024 arXiv

  44. [52]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  45. [53]

    Yu Wang, Ruihan Wu, Zexue He, Xiusi Chen, and Julian McAuley. 2024 a . Large scale knowledge washing. arXiv preprint arXiv:2405.16720

    2024 arXiv

  46. [54]

    Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, and Y Wu. 2024 b . Let the expert stick to his last: Expert-specialized fine-tuning for sparse architectural large language models. arXiv preprint arXiv:2407.01906

    2024 arXiv

  47. [55]

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162

    2024 arXiv

  48. [56]

    Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. 2023. Unveiling the implicit toxicity in large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  49. [57]

    Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. 2023. Depn: Detecting and editing privacy neurons in pretrained language models. arXiv preprint arXiv:2310.20138

    2023 arXiv

  50. [58]

    xAI. 2024. Grok-1: Python library for interpretable machine learning with grok. https://github.com/xai-org/grok-1. Accessed: 2024-09-25

    2024

  51. [59]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  52. [60]

    Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. 2024. Machine unlearning of pre-trained large language models. arXiv preprint arXiv:2402.15159

    2024 arXiv

  53. [61]

    Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023. Large language model unlearning. arXiv preprint arXiv:2310.10683

    2023 arXiv

  54. [62]

    Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. 2023. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6032--6048

    2023

  55. [63]

    Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. 2023 a . Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303.17591

    2023 arXiv

  56. [64]

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868

    2024 arXiv

  57. [65]

    Yihua Zhang, Ruisi Cai, Tianlong Chen, Guanhua Zhang, Huan Zhang, Pin-Yu Chen, Shiyu Chang, Zhangyang Wang, and Sijia Liu. 2023 b . Robust mixture-of-expert training for convolutional neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ...

    2023

  58. [66]

    Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021. Moefication: Transformer feed-forward layers are mixtures of experts. arXiv preprint arXiv:2110.01786

    2021 arXiv

  59. [67]

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. 2022. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103--7114

    2022

  60. [68]

    Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. 2024. Llama-moe: Building mixture-of-experts from llama with continual pre-training. arXiv preprint arXiv:2406.16554

    2024 arXiv

  61. [70]

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022 b . St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906

    2022 arXiv

  62. [71]

    Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

  63. [72]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  64. [73]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  65. [74]

    for fill-in-the-blank tasks and ``Please briefly answer the following question. Question:

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1959

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.