Pith. sign in

REVIEW 12 cited by

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.14800 v2 pith:O22F5HK3 submitted 2024-02-22 cs.CL cs.AIcs.LG

Xudong Lu , Qi Liu , Yuhui Xu , Aojun Zhou , Siyuan Huang , Bo Zhang , Junchi Yan , Hongsheng Li This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords llmsexpertperformancepruningdeploymentefficiencylanguagelarge
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

A pivotal advancement in the progress of large language models (LLMs) is the emergence of the Mixture-of-Experts (MoE) LLMs. Compared to traditional LLMs, MoE LLMs can achieve higher performance with fewer parameters, but it is still hard to deploy them due to their immense parameter sizes. Different from previous weight pruning methods that rely on specifically designed hardware, this paper mainly aims to enhance the deployment efficiency of MoE LLMs by introducing plug-and-play expert-level sparsification techniques. Specifically, we propose, for the first time to our best knowledge, post-training approaches for task-agnostic and task-specific expert pruning and skipping of MoE LLMs, tailored to improve deployment efficiency while maintaining model performance across a wide range of tasks. Extensive experiments show that our proposed methods can simultaneously reduce model sizes and increase the inference speed, while maintaining satisfactory performance. Data and code will be available at https://github.com/Lucky-Lance/Expert_Sparsity.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

    cs.LG 2025-11 conditional novelty 7.0 of 10

    A training-free method that merges pairs of MoE experts at the individual-weight level and packs the required masks into unused exponent bits, cutting expert memory by 50% with minimal accuracy loss.

  2. MixNet: A Runtime Reconfigurable Optical-Electrical Fabric for Distributed Mixture-of-Experts Training

    cs.NI 2025-01 conditional novelty 7.0 of 10

    MixNet uses regionally reconfigurable optical switches to adapt the network topology during MoE training, reaching fat-tree-like performance with lower cost.

  3. Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference

    cs.DC 2026-07 conditional novelty 6.0 of 10

    Communication-aware expert placement plus device-level pruning yields 1.23–1.86× MoE inference throughput and better accuracy at equal speedup than load-balance or sequential baselines.

  4. Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs

    cs.CL 2025-09 conditional novelty 6.0 of 10

    DERN prunes SMoE LLMs by decomposing removed experts into neuron segments, reassigning the best-matching ones to kept experts, and clustering them into compact replacements, beating prior pruning baselines without retraining.

  5. Unified Start, Personalized End: Progressive Pruning for Efficient 3D Medical Image Segmentation

    cs.CV 2025-09 conditional novelty 6.0 of 10

    PSP-Seg prunes redundant modules during training to make 3D segmentation networks much smaller and faster without losing accuracy.

  6. Mixture-of-Experts for Personalized and Semantic-Aware Next Location Prediction

    cs.AI 2025-05 conditional novelty 6.0 of 10

    NextLocMoE combines a location-semantics expert mixer and a persona-based expert mixer inside a trimmed LLaMA backbone to set new accuracy marks on next-location prediction benchmarks.

  7. Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inference

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Occult co-locates frequently co-activated experts on the same device and optionally prunes routing to fewer devices, cutting all-to-all communication and speeding up MoE training and inference.

  8. DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference

    cs.DC 2024-12 conditional novelty 6.0 of 10

    DAOP accelerates Mixture-of-Experts inference by sequence-specific expert allocation between CPU and GPU plus one-layer-ahead predictive pre-calculation, achieving up to 1.35x over Fiddler and up to 8.20x over naive c...

  9. MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?

    cs.LG 2025-09 reject novelty 5.0 of 10

    In a 26-layer MoE model, injecting Gaussian weight errors into middle-layer experts hurts math accuracy most, while deep-layer errors can sometimes improve instruction compliance.

  10. LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference

    cs.LG 2025-09 conditional novelty 5.0 of 10

    LExI sets a different number of active experts per layer, found by weight-only sensitivity profiling and evolutionary search, improving MoE inference throughput with little accuracy loss.

  11. ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

    cs.LG 2024-12 conditional novelty 5.0 of 10

    ReLU-based routing with adaptive L1 regularization outperforms TopK routing in MoE language models across model sizes, expert counts, and granularity.

  12. A Survey on Inference Optimization Techniques for Mixture of Experts Models

    cs.LG 2024-12 conditional novelty 3.0 of 10

    A structured survey of MoE inference optimization that categorizes existing techniques into model, system, and hardware levels and summarizes reported speedups and memory savings.

Pith tools