Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Dropping experts and recombining their neurons preserves large MoE LLM accuracy better than whole-expert merging, without any retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:51 UTC pith:7VUWTB3T

load-bearing objection Legit new fine-grained MoE pruning method; the headline gains are real but alpha is tuned on the eval benchmarks, so treat the 5% margin as an upper bound. the 3 major comments →

arxiv 2509.10377 v1 pith:7VUWTB3T submitted 2025-09-12 cs.CL

Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs

Yixiao Zhou , Ziyu Zhao , Dongzhou Cheng , zhiliang wu , Jie Gui , Yi Yang , Fei Wu , Yu Cheng
show 1 more author
Hehe Fan
This is my paper
classification cs.CL
keywords mixture-of-expertsexpert pruningretraining-free compressionneuron-level mergingexpert segmentsspherical k-meanssparse MoE LLMmodel compression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that Mixtral-, Qwen-, and DeepSeek-style sparse mixture-of-experts LLMs can be compressed by cutting out whole experts and then redistributing the cut experts' neuron-level pieces into the experts that remain, without any retraining. The claim is that this neuron-level recombination beats prior expert-pruning and expert-merging baselines by more than five percentage points at 50% expert sparsity on commonsense reasoning and MMLU benchmarks, while also cutting memory and speeding up token generation. The paper identifies why prior whole-expert merging underperforms: experts are misaligned at the neuron level even when they look similar globally, so averaging whole experts blends conflicting internal representations. If correct, the result gives a training-free, task-agnostic recipe for shrinking MoE LLMs for deployment.

Core claim

DERN treats each expert as a set of expert segments—triplets formed by a gate row, an up-projection row, and a down-projection column—and reframes expert pruning as segment decomposition and recombination. After pruning low-importance experts based on router statistics, DERN pools the pruned experts' segments, reassigns each segment to the retained expert whose existing segments are most cosine-similar to it (filtered by a threshold), and then compresses each retained expert by spherical weighted k-means clustering of its original and newly assigned segments. The paper reports that on Mixtral-8x7B, Qwen2-57B-A14B, and DeepSeek-MoE-16B, DERN outperforms LLM-Pruner, M-SMoE, and NAEE, with the

What carries the argument

The central object is the expert segment: the minimal functional unit (w_g,i, w_u,i, w_d,i) of a gated MLP expert, whose output contribution is independent of other hidden dimensions. Flattening each segment to a fixed vector permits cosine similarity to decide which retained expert should inherit it, and spherical weighted k-means then merges compatible segments into a compact expert. The load-bearing mechanism is the 'less is more' reassignment threshold: reusing only a moderate fraction of structurally similar segments (alpha around 0.4–0.6) beats both indiscriminate merging (alpha=0) and no merging at all (alpha=1), because selective reuse avoids importing semantically conflicting neuron

Load-bearing premise

The whole pipeline leans on the assumption that cosine similarity between flattened parameter vectors of segments is a reliable proxy for whether two neurons are functionally compatible enough to share an expert.

What would settle it

Take a model whose experts are deliberately specialized (e.g., by domain or by synthetic task partition), so that parameter-similar segments are known to be functionally unrelated. If DERN's best reassignment threshold is applied and the pruned model's accuracy falls below the no-merge baseline (alpha=1), the parameter-space cosine proxy fails; conversely, replacing the similarity metric with a functional one (e.g., calibrated activation correlation of segments) and showing a large accuracy gap would settle the same question.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At 25% expert sparsity, Mixtral pruned with DERN scores 80.04 on the paper's benchmark average, within a point of the dense 80.88, while using 25% less memory and 18% higher throughput.
  • At 50% sparsity, DERN improves average accuracy over NAEE by about 5.1 points on Mixtral (70.79 vs 65.69), 12.9 points on Qwen2 (83.56 vs 70.71), and 8.8 points on DeepSeek (55.50 vs 46.67).
  • The routing contribution of a dropped expert is softly transferred to its retained host scaled by 1/n, preserving routing semantics without retraining.
  • The method is task-agnostic: calibration needs only 128 C4 sequences (2048 tokens each) to compute router statistics, and the resulting pruned model is evaluated zero-shot on commonsense benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because DERN operates per segment rather than per expert, its segment pool and reassignment can in principle be combined with intra-expert weight pruning or low-rank decomposition, potentially pushing effective compression below the expert-count ratio alone.
  • The threshold alpha is treated as a global constant in the paper; the ablation suggests a per-layer or per-expert adaptive threshold could squeeze more of the 'less is more' effect, especially in middle layers where inter-expert similarity peaks.
  • The paper's own DeepSeek results hint at a boundary condition: when experts are highly specialized and inter-expert similarity is low, segment reassignment has less compatible structure to work with, so models trained for extreme expert specialization may need an architecture-aware variant rather than a generic one.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DERN, a retraining-free, task-agnostic expert-pruning framework for sparse Mixture-of-Experts LLMs. DERN first selects retained experts using router activation statistics on a C4 calibration set, then decomposes pruned experts into neuron-level GLU triplets ('segments'), reassigns compatible segments to retained experts based on cosine similarity with threshold α, and finally compresses each retained expert's segment set via spherical weighted k-means. Experiments on Mixtral-8×7B-Instruct, Qwen2-57B-A14B-Instruct, and DeepSeek-MoE-16B-Chat at several sparsity levels report consistent gains over LLM-Pruner, M-SMoE, and NAEE, including more than 5-point average improvements at 50% expert sparsity, plus throughput and memory measurements. Ablations examine α, segment similarity components, clustering initialization, and importance weighting.

Significance. Neuron-level recombination of pruned MoE experts is a timely and practically useful idea, and the paper has concrete strengths: evaluations across three different SMoE architectures and multiple sparsity levels, ablations that isolate each design choice, a complete algorithmic pseudocode, and unusually detailed reproducibility information (OpenCompass dataset IDs, vLLM configurations, calibration settings). The authors also explicitly acknowledge that parameter-space cosine similarity may be insufficient for functional alignment. If the empirical claims withstand scrutiny, DERN would be a meaningful advance over whole-expert pruning/merging. The main weakness is that the headline gain depends on selecting the similarity threshold α on the evaluation benchmarks themselves, which is incompatible with the paper's task-agnostic framing and makes the reported >5% margin optimistic.

major comments (3)
  1. [§4.3, App. C.2, Eq. (7), Table 11] The central empirical claim is not protected against selection on the evaluation data. Table 11 shows Mixtral-4×7B average accuracy moving from 64.14 at α=0 to 70.87 at α=0.4, then to 69.62 at α=0.6, 70.06 at α=0.8, and 65.69 at α=1.0. The main-table DERN result (70.79) tracks the favorable end of this curve, and the paper recommends α∈[0.4,0.6] based on performance on these same benchmarks. The claimed >5-point margin over NAEE (65.69) exists only at the single best point; at α=0.6 the margin is only 3.93 points. Because α is a new hyperparameter introduced by the method and is effectively tuned on the test benchmarks, the reported advantage is an upper bound. For a task-agnostic method, α should be selected on a validation set disjoint from the reported benchmarks, or the main results should be reported across α without cherry-picking the best value.
  2. [§3.2, Eq. (8)] The routing-transfer rule is under-specified. The text states 'softly transfer the routing contribution from E_o to E_r, scaled by the number of segments in E_o: G_E_r ← G_E_r + 1/n · G_E_o'. If G denotes the soft routing weights defined in Eq. (1), then adding a constant fraction of the dropped expert's probability mass to every retained expert is not a probability-preserving operation; if G denotes pre-softmax logits, this needs to be stated and the renormalization made explicit. Since this step is applied in every layer and determines which experts are active after pruning, the exact update rule and its justification are load-bearing. Please provide the precise normalization and either a derivation that the updated distribution corresponds to the intended semantics or an empirical justification for the heuristic.
  3. [§3.3, Eq. (10)/(13)] The target hidden dimension k after clustering is a second free parameter, but its value in the main experiments is never stated. If k equals the original intermediate size h, the clustering is a re-parameterization rather than an additional compression; if k < h, the accuracy and memory numbers in Tables 2–4 depend on k. The appendix only says that the number of output segments is 'determined proportionally to the original expert size', which is vague. Please report k (or the effective segment-retention ratio) for every configuration in the main tables, and clarify whether any intra-expert compression contributes to the reported memory and latency savings beyond expert count reduction.
minor comments (6)
  1. [Algorithm 1, line 7] The pseudocode sets P ← ⋃_{E_i∈E_r} Decompose(E_i), but the text in §3.2 says the segment pool P is aggregated from pruned experts, not from retained experts. This looks like a typographical error (it should be the complement of E_r) and should be corrected.
  2. [Tables 2, 4, 5, 11] The reported DERN Mixtral-4×7B numbers are inconsistent across tables: Table 2 reports Avg. 70.79 and MMLU 54.91; Table 4 reports Avg. 70.60; Table 5 reports MMLU 55.01; Table 11 reports Avg. 70.87 and MMLU 54.74 at α=0.4. Please harmonize the values or explain the differences.
  3. [Fig. 4 caption] The caption says the bars show the ratio of segments retained 'in last layer'. It is unclear why only the last layer is used and how this ratio generalizes across layers; please specify whether this is representative or report per-layer statistics.
  4. [Table 6 header] The header 'Random samples seeds uniformly' appears to be a typo; it should likely read 'Random samples seeds uniformly'. Please fix the wording.
  5. [Abstract and §4.2] The abstract says the >5% improvement is demonstrated 'under 50% expert sparsity', but DeepSeek-MoE-16B is evaluated at 56/64 and 48/64 experts, i.e., 12.5% and 25% sparsity, respectively. Please clarify which configurations support the 50%-sparsity claim and avoid implying that the DeepSeek results are at 50% sparsity.
  6. [§4.1 Baselines] For NAEE on Qwen2 and DeepSeek, the paper approximates expert selection by sampling 10k combinations per layer. Please provide details on the computational cost and, if possible, report whether the approximation is a lower bound for NAEE; otherwise the comparison on these two models may be viewed as against an approximate baseline.

Circularity Check

0 steps flagged

No significant circularity: DERN's benchmark gains are measured outputs of a C4-calibrated pipeline; the α threshold is a hyperparameter selection risk, not a derivation loop.

full rationale

DERN's derivation chain does not reduce to its inputs. Expert importance (Eq. 1) is computed from router statistics on a C4 calibration set; segment decomposition (Eqs. 2–4) follows exactly from GLU algebra; reassignment (Eqs. 5–7) and clustering (Eqs. 10–13) operate purely on parameter vectors. None of these equations contains the evaluation targets (BoolQ, MMLU, etc.) or their accuracies, so the abstract's '>5% gain' is an empirical measurement reported in Tables 2–3, not a quantity derived from the method's definitions. The only benchmark-sensitive knob is the threshold α in Eq. 7: Table 11 shows a 6.7-point swing on Mixtral-4×7B, and App. C.2 recommends α∈[0.4,0.6] after ablation. That is a potential selection-on-test-set concern, but the paper does not claim to derive α from first principles, and the final accuracy is not equal to α by construction. The Limitations section candidly states that parameter-space cosine similarity 'may be insufficient for capturing functional alignment,' which is an acknowledged assumption, not a circular step. Self-citations in the motivation (e.g., Zhao et al. 2024/2025) are background references; the core pipeline and its evaluation are self-contained against public benchmarks. Therefore no circular step is exhibited.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

No new physical or architectural entities are introduced. 'Expert Segment' is a re-indexing of existing GLU weights into (gate, up, down) triplets, not a new module type. The ledger shows two free parameters (alpha and cluster count) and two domain assumptions that the empirical results depend on.

free parameters (2)
  • Similarity threshold alpha = 0.4 (recommended range 0.4-0.6)
    Controls whether a pruned-expert segment is reassigned to a retained expert (Eq. 7). Ablated on the evaluation benchmarks (Table 11); peak accuracy at alpha=0.4 for both Mixtral settings. This is a hyperparameter tuned on the same benchmarks used to report the 5% gain.
  • Target expert hidden dimension (number of clusters per expert) = not specified; 'proportionally to original expert size'
    The clustering step (Eq. 10) compresses each retained expert to k segments, but no formula is given for k. The final parameter counts (e.g., 45.1B for Qwen 48/64) depend on this choice, making it a free design parameter.
axioms (3)
  • standard math Each expert's output is an exact sum of independent hidden-neuron contributions (Eq. 3).
    The GLU decomposition into segment triplets (gate, up, down) is algebraically exact and used to define segments in Eq. 4.
  • domain assumption Neurons with high cosine similarity in parameter space can be merged without functional loss.
    Central to segment reassignment (Eqs. 6-7) and clustering (Eq. 10). The authors flag this as potentially insufficient for functional alignment in the Limitations section.
  • domain assumption Routing statistics measured on 128 C4 sequences identify experts that are redundant across all evaluation tasks.
    Stage 1 scores experts with Eq. 1 using C4 calibration; the task-agnostic claim rests on this generalization from calibration data to downstream benchmarks.

pith-pipeline@v1.3.0-alltime-deepseek · 18688 in / 7822 out tokens · 86508 ms · 2026-08-04T17:51:12.505259+00:00 · methodology

0 comments
read the original abstract

Sparse Mixture-of-Experts (SMoE) architectures are widely used in large language models (LLMs) due to their computational efficiency. However, though only a few experts are activated for each token, SMoE still requires loading all expert parameters, leading to high memory usage and challenges in deployment. Previous work has tried to reduce the overhead by pruning and merging experts, but primarily focused on expert-level operations, leaving neuron-level structure underexplored. We propose DERN (Dropping Experts, Recombining Neurons), a task-agnostic and retraining-free framework for expert pruning and reconstruction. We observe that experts are often misaligned and contain semantic conflicts at the neuron level, which poses challenges for direct merging. To solve this, DERN works in three steps: it first prunes redundant experts using router statistics; then it decomposes them into neuron-level expert segments, assigning each segment to its most compatible retained expert; and finally, it merges segments within each retained expert to build a compact representation. Experiments on Mixtral, Qwen, and DeepSeek SMoE models show that DERN improves performance by more than 5% on commonsense reasoning and MMLU benchmarks under 50% expert sparsity, without extra training. It also greatly reduces the number of experts and memory usage, making SMoE LLMs easier to deploy in practice.

Figures

Figures reproduced from arXiv: 2509.10377 by Dongzhou Cheng, Fei Wu, Hehe Fan, Jie Gui, Yixiao Zhou, Yi Yang, Yu Cheng, Zhiliang Wu, Ziyu Zhao.

Figure 1
Figure 1. Figure 1: Expert-level and neuron-level cosine similarity in layer 15 of Mixtral-8 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the DERN framework. (a) Redundant experts are pruned based on routing activation statistics. (b) Dropped experts are decomposed into neuron-level segments, each represented by a triplet of gate, up, and down projection vectors. These segments are then reassigned to retained experts based on local structural similarity. (c) Within each retained expert, reassigned and original segments are cluste… view at source ↗
Figure 3
Figure 3. Figure 3: Neuron-level similarity heatmap among ex [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of the similarity threshold α during segment reassignment. Line plots indicate average per￾formance, and bars show the ratio of segments retained in last layer. Results are averaged over all benchmarks for Mixtral-4×7B (left) and Mixtral-6×7B (right). regardless of compatibility. We observe a clear pattern across both Mixtral￾4×7B and Mixtral-6×7B: performance improves as α increases from 0 to aroun… view at source ↗
Figure 5
Figure 5. Figure 5: Neuron-level and expert-level similarity heatmaps across Layers 0, 7, 15, 23, and 31. Left: expert [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: ℓ∞ norm of row vectors Wg across Layers 10, 15, 20, and 25 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

    cs.LG 2026-03 conditional novelty 7.0

    EvoESAP uses evolutionary search guided by a speculative-decoding-inspired ESAP metric to discover non-uniform layer-wise sparsity allocations for MoE expert pruning, improving generation accuracy up to 19.6% at 50% sparsity.

  2. OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

    cs.CL 2026-02 conditional novelty 6.0

    OmniMoE's vector-level atomic experts + Cartesian-product router + expert-centric scheduling gives 50.9% zero-shot accuracy and a 10.9x inference speedup over PEER at matched 1.7B active parameters.

Reference graph

Works this paper leans on

52 extracted references · 30 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2022. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836

    Pith/arXiv arXiv 2022

  2. [2]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432--7439

    2020

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  4. [4]

    Mingyu Cao, Gen Li, Jie Ji, Jiaqi Zhang, Xiaolong Ma, Shiwei Liu, and Lu Yin. 2024. Condense, don't just prune: Enhancing efficiency and performance in moe layer pruning. arXiv preprint arXiv:2412.00069

    Pith/arXiv arXiv 2024

  5. [5]

    I Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, Chun-Yi Lee, and 1 others. 2024. Retraining-free merging of sparse mixture-of-experts via hierarchical clustering. arXiv preprint arXiv:2410.08589

    arXiv 2024

  6. [6]

    Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. 2022. Task-specific expert pruning for sparse mixture-of-experts. arXiv preprint arXiv:2206.00277

    Pith/arXiv arXiv 2022

  7. [7]

    Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, and 1 others. 2022. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 35:34600--34613

    2022

  8. [8]

    Mohammed Nowaz Rabbani Chowdhury, Meng Wang, Kaoutar El Maghraoui, Naigang Wang, Pin-Yu Chen, and Christopher Carothers. 2024. A provably effective method for pruning experts in fine-tuned sparse mixture-of-experts. arXiv preprint arXiv:2405.16646

    Pith/arXiv arXiv 2024

  9. [9]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044

    Pith/arXiv arXiv 2019

  10. [10]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    Pith/arXiv arXiv 2018

  11. [11]

    OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass

    2023

  12. [12]

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, and 1 others. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066

    Pith/arXiv arXiv 2024

  13. [13]

    Inderjit S Dhillon and Dharmendra S Modha. 2001. Concept decompositions for large sparse text data using clustering. Machine learning, 42:143--175

    2001

  14. [14]

    Yubo Dong and Hehe Fan. 2025. Enhancing large language models through structured reasoning. arXiv preprint arXiv:2506.20241

    Pith/arXiv arXiv 2025

  15. [15]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1--39

    2022

  16. [16]

    Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. 2025. Delta decompression for moe-based llms compression. arXiv preprint arXiv:2502.17298

    Pith/arXiv arXiv 2025

  17. [17]

    Shwai He, Daize Dong, Liang Ding, and Ang Li. 2024. Demystifying the compression of mixture-of-experts through a unified framework. arXiv preprint arXiv:2406.02500

    Pith/arXiv arXiv 2024

  18. [18]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    Pith/arXiv arXiv 2020

  19. [19]

    Zhaolin Hu, Yixiao Zhou, Zhongan Wang, Xin Li, Weimin Yang, Hehe Fan, and Yi Yang. 2025. Osda agent: Leveraging large language models for de novo design of organic structure directing agents. In The International Conference on Learning Representations

    2025

  20. [20]

    Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and Xiaojuan Qi. 2024. Mc-moe: Mixture compressor for mixture-of-experts llms gains more. arXiv preprint arXiv:2410.06270

    Pith/arXiv arXiv 2024

  21. [21]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    Pith/arXiv arXiv 2024

  22. [22]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  23. [23]

    Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. 2023. Merge, then compress: Demystify efficient smoe with hints from its routing policy. arXiv preprint arXiv:2310.01334

    Pith/arXiv arXiv 2023

  24. [24]

    Wei Li, Hehe Fan, Yongkang Wong, Mohan Kankanhalli, and Yi Yang. 2024 a . Cat-llm: Context-aware training enhanced large language models for multi-modal contextual image retrieval

    2024

  25. [25]

    Wei Li, Hehe Fan, Yongkang Wong, Mohan Kankanhalli, and Yi Yang. 2024 b . Topa: Extending large language models for video understanding via text-only pre-alignment. Advances in Neural Information Processing Systems, 37:5697--5738

    2024

  26. [26]

    Wei Li, Hehe Fan, Yongkang Wong, Yi Yang, and Mohan Kankanhalli. 2024 c . Improving context understanding in multimodal large language models via multimodal composition learning. In Forty-first International Conference on Machine Learning

    2024

  27. [27]

    Yuqi Li, Yao Lu, Zeyu Dong, Chuanguang Yang, Yihao Chen, and Jianping Gou. 2024 d . Sglp: A similarity guided fast layer partition pruning for compressing large deep models. arXiv preprint arXiv:2410.14720

    arXiv 2024

  28. [28]

    Xun Liang, Hanyu Wang, Huayi Lai, Simin Niu, Shichao Song, Jiawei Yang, Jihao Zhao, Feiyu Xiong, Bo Tang, and Zhiyu Li. 2025. Seap: Training-free sparse expert activation pruning unlock the brainpower of large language models. arXiv preprint arXiv:2503.07605

    Pith/arXiv arXiv 2025

  29. [29]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    Pith/arXiv arXiv 2024

  30. [30]

    Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. 2024 b . Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs. arXiv preprint arXiv:2407.00945

    Pith/arXiv arXiv 2024

  31. [31]

    Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. 2024. A closer look into mixture-of-experts in large language models. arXiv preprint arXiv:2406.18219

    Pith/arXiv arXiv 2024

  32. [32]

    Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. 2024 a . Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. arXiv preprint arXiv:2402.14800

    Pith/arXiv arXiv 2024

  33. [33]

    Yao Lu, Hao Cheng, Yujie Fang, Zeyu Wang, Jiaheng Wei, Dongwei Xu, Qi Xuan, Xiaoniu Yang, and Zhaowei Zhu. 2024 b . Reassessing layer pruning in llms: New insights and methods. arXiv preprint arXiv:2411.15558

    Pith/arXiv arXiv 2024

  34. [34]

    Yao Lu, Yutao Zhu, Yuqi Li, Dongwei Xu, Yun Lin, Qi Xuan, and Xiaoniu Yang. 2024 c . A generic layer pruning method for signal modulation recognition deep learning models. IEEE Transactions on Cognitive Communications and Networking

    2024

  35. [35]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702--21720

    2023

  36. [36]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789

    Pith/arXiv arXiv 2018

  37. [37]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  38. [38]

    Wei Ruan, Tianze Yang, Yifan Zhou, Tianming Liu, and Jin Lu. 2025. From task-specific models to unified systems: A review of model merging approaches. arXiv preprint arXiv:2503.08998

    Pith/arXiv arXiv 2025

  39. [39]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  40. [40]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538

    Pith/arXiv arXiv 2017

  41. [41]

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695

    Pith/arXiv arXiv 2023

  42. [42]

    Chao Wang, Hehe Fan, Ruijie Quan, Lina Yao, and Yi Yang. 2025. Protchatgpt: Towards understanding proteins with hybrid representation and large language models. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1076--1086

    2025

  43. [43]

    Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. 2024. Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router. arXiv preprint arXiv:2410.12013

    Pith/arXiv arXiv 2024

  44. [44]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024 a . https://arxiv.org/abs/2407.10671 Qwen2 technical report . Preprint, arXiv:2407.10671

    Pith/arXiv arXiv 2024

  45. [45]

    Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. 2024 b . Moe- i^2 : Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. arXiv preprint arXiv:2411.01016

    Pith/arXiv arXiv 2024

  46. [46]

    Jinluan Yang, Dingnan Jin, Anke Tang, Li Shen, Didi Zhu, Zhengyu Chen, Ziyu Zhao, Daixin Wang, Qing Cui, Zhiqiang Zhang, and 1 others. 2025. Mix data or merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging. arXiv preprint arXiv:2502.06876

    arXiv 2025

  47. [47]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    Pith/arXiv arXiv 2019

  48. [48]

    Yue Zhang, Hehe Fan, Wei Ji, Yongkang Wong, Roger Zimmermann, and Yi Yang. 2025. Prompt-aware adapter: Learning adaptive visual tokens for multimodal large language models. IEEE Transactions on Artificial Intelligence

    2025

  49. [49]

    Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu. 2024. Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering. arXiv preprint arXiv:2409.16167

    Pith/arXiv arXiv 2024

  50. [50]

    Ziyu Zhao, Yixiao Zhou, Zhi Zhang, Didi Zhu, Tao Shen, Zexi Li, Jinluan Yang, Xuwu Wang, Jing Su, Kun Kuang, and 1 others. 2025. Each rank could be an expert: Single-ranked mixture of experts lora for multi-task learning. arXiv preprint arXiv:2501.15103

    Pith/arXiv arXiv 2025

  51. [51]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  52. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...