Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

Pruning MoE experts by the stationary distribution of a routing Markov chain retains more task performance than local heuristics, with up to 10.61% higher average retention at 50% compression.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:50 UTC pith:IAB5XUMJ

load-bearing objection A genuinely new Markov-chain heuristic for MoE expert pruning with broad evaluation, but the theoretical claim overreaches and the empirical setup has fairness gaps. the 2 major comments →

arxiv 2607.08601 v2 pith:IAB5XUMJ submitted 2026-07-09 cs.CL

It Takes a MAESTRO To Prune Bad Experts

Palaash Goel , Ayush Maheshwari , Tanmoy Chakraborty This is my paper
classification cs.CL
keywords mixture-of-expertsstructured pruningMarkov chainstationary distributionexpert pruningmodel compressionLLM efficiencyrouting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the right way to decide which experts to prune from a mixture-of-experts language model is to ask how often routing visits each expert over the model's own autoregressive generations, not how important each expert looks in a single layer. It models the sequence of (layer, expert) selections as a Markov chain, estimates transition frequencies from a tiny calibration set, and uses the chain's stationary distribution as a global importance score. If the paper is right, the experts with the smallest long-run mass are the least load-bearing, and removing them uniformly across layers retains far more capability than existing per-layer heuristics — up to 10.61% higher average retention at 50% compression — while also making task performance more consistent across 17 benchmarks. A sympathetic reader would care because it offers a data-driven, theoretically grounded way to make MoE models fit in memory without sacrificing the capabilities that matter, including safety and bias behavior.

Core claim

MAESTRO's central claim is that expert importance in an MoE LLM can be read off the stationary distribution π of an ergodic Markov chain whose states are the (layer, expert) slots and whose transitions count which experts are selected at layer ℓ+1 given the selection at layer ℓ. Calibrated autoregressively on a small corpus, π gives each expert a long-run fraction of routing visits; the experts with the smallest π-mass are, by construction, the least frequently visited across all routing trajectories and are the natural candidates for removal (Section 3.2). Empirically, pruning those experts uniformly across layers and recovering with frozen-expert LoRA fine-tuning outperforms state-of-the-a

What carries the argument

The central object is an ergodic Markov chain over the LE expert slots of the model, with per-layer transition matrices C(ℓ) estimated from autoregressive rollout counts, assembled into a block-cyclic global matrix T, row-normalized to a stochastic kernel P, and made irreducible and aperiodic by cyclic closure plus an ε-self-loop smoothing (Pε = (1−ε)P + εI/Z). Its unique stationary distribution π (Perron–Frobenius) is the importance heuristic: experts with smallest π are pruned uniformly per layer. The chain is what carries the argument from raw routing traces to a global, cross-layer importance ranking.

Load-bearing premise

Expert importance is read off a first-order Markov chain over layer-to-layer routing, so the entire ranking rests on the assumption that the experts chosen at layer ℓ+1 depend only on the experts chosen at layer ℓ; if routing choices depend on longer histories, the stationary distribution mis-orders experts and the pruning decisions inherit the bias.

What would settle it

Zero out each expert individually on a held-out set, measure the performance drop, and compare with MAESTRO's ranking: if the experts with the smallest stationary mass are not among those whose removal least degrades performance, the central claim fails. A cheaper check replaces the first-order transition counts with second-order counts and asks whether the pruned expert set changes and whether the second-order pruned model outperforms the first-order one on the same benchmarks.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At 50% compression on GPT-OSS-20B, MAESTRO retains 92.59% average performance versus 83.92% for random pruning and 89.37% for the best baseline REAP, while halving cross-task standard deviation (6.30% vs 10.40%).
  • On Qwen3-30B (128 experts per layer), MAESTRO retains 93.40% at 50% compression, and its 25%-compressed model beats the unpruned base on generative tasks (log-perplexity 1.86 vs 1.97), suggesting that removing low-mass experts can act as a mild regularizer.
  • The gains are concentrated in generative and domain-specific tasks (e.g., MedQA: 51.93 vs 37.63 for REAP at 50% compression on GPT-OSS-20B), suggesting that domain-specialized experts are faithfully preserved by the stationary distribution.
  • Pruning decisions differ qualitatively from single-pass activation counting: in Layer 0 of GPT-OSS-20B, single-pass counting prunes a near-contiguous block of low-index experts (a positional bias), while the Markov chain selects a disjoint set, and early-layer errors are argued to compound downstream.
  • Safety, Bias, and Ethics scores remain broadly stable across compression ratios, with MAESTRO competitive or superior to baselines on Winogender, TruthfulQA, and Moral Stories.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A second-order Markov chain over two-layer histories is a natural test: if it produces a different pruned set with better retention, the first-order ranking was incomplete.
  • The cyclic closure block C(0) treats the transition from the last layer's experts to the first layer's experts on the next token as if it were another layer transition; this conflation of within-token and across-token dynamics may distort π for the first and last layers, and could be tested by ablating the closing block.
  • The calibration-size sensitivity (Table 3: cross-task standard deviation shifts by 15–21% when calibration ranges from 50 to 500 samples) suggests the importance scores inherit distribution shift from the calibration corpus; a testable extension is to calibrate on multiple domain-balanced subsets and check whether the pruned expert set is stable.
  • The method's logic is not specific to two MoE families; extending to other sparse architectures (e.g., different top-k or expert counts, or vision MoE models) would test whether the stationary-mass heuristic transfers beyond language.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper proposes MAESTRO, a structured MoE pruning method. It treats each (layer, expert) slot as a state in a Markov chain, estimates pairwise transition counts between consecutive layers from autoregressive rollouts on 250 SlimOrca samples (Tseq=100), row-normalizes and ε-smooths the resulting block-cyclic matrix, and computes its stationary distribution by power iteration. Experts with the smallest per-layer stationary mass are removed uniformly across layers (K=⌊ρE⌋ per layer), followed by LoRA recovery fine-tuning on attention projections with frozen experts/routers. Experiments on GPT-OSS-20B and Qwen3-30B at 25% and 50% compression across 17 tasks (including Safety, Bias, and Ethics) report higher average retention than random, HC-SMoE, MoP, and REAP, often with lower cross-task standard deviation. Ablations show that the uniform pruning constraint, autoregressive transition counting, and RFT each contribute; calibration-size sensitivity is small in mean RP but larger in std RP.

Significance. MAESTRO is a practically motivated, well-evaluated heuristic: the comparison includes two architecturally distinct MoE families, per-task results are reported, and the ablation suite (UNI, AUTO, –RFT, calibration size) is appropriate and informative. The empirical numbers appear internally consistent, and the consistently lower cross-task variance—especially at 50% compression on GPT-OSS—is a genuinely interesting property. However, the paper's theoretical framing, that π is by construction the long-run visit frequency of routing, is not established: Eq. (4) estimates a pairwise co-occurrence chain, not a chain over the actual k-subset routing states. The claimed uniqueness of π also lacks a complete ergodicity argument. These issues are correctable with revised claims and a validation experiment, but they are load-bearing because the stationary ranking is the entire basis for pruning decisions. With those revisions, the paper would be a solid empirical contribution to MoE compression.

major comments (2)
  1. [§3.2–3.4, Eq. (4)] The central theoretical claim that π identifies the least frequently visited expert slots is not justified by the construction. A token activates k>1 experts per layer, so the true trajectory is a sequence of k-subsets, not of single states. Eq. (4) counts every ordered pair (i,j) with i∈E_{ℓ−1} and j∈E_ℓ, i.e., k² counts per token-step; after row normalization this is a pairwise co-occurrence chain, not the empirical single-expert visit frequency. The stationary vector of this chain coincides with visit frequency only under an independence assumption over the k selected experts, which is not stated or tested. Since every pruning decision and all reported retention numbers inherit this ranking, the 'by construction' phrasing in §3.2 should be removed, replaced by a formal aggregate-chain derivation, or empirically validated—e.g., compare π against direct per-expert visit frequencies on t
  2. [§3.4, Eqs. (7)–(8)] The ergodicity argument is incomplete. Adding ε self-loops to a row-normalized count matrix does not create edges between disconnected components; if any expert has zero incoming or outgoing counts on the calibration corpus, it forms a closed communicating class and Perron–Frobenius does not guarantee a unique stationary distribution over all LE states. The sentence 'irreducible and aperiodic on the support of the calibration trajectories' does not resolve this, since the support may be disconnected. Also, Eq. (8) as written ('ε I/Z') is not well defined unless Z is specified; the denominator max(1, row-sum) in Eq. (7) only handles zero rows, not zero columns or multiple recurrent classes. Please state the exact normalization and either prove irreducibility on the observed support or add a uniform perturbation to every entry and report sensitivity to its magnitude.
minor comments (4)
  1. [§4/Table 1] The metrics 'Avg RP (%)' and 'Std RP (%)' are never formally defined. Please provide the formula, including how Log-PPL is converted into a retention value and how per-task retention is aggregated. A reader can reverse-engineer the numbers from Tables 4–5, but the main text should be self-contained.
  2. [§3.4, Eq. (8)] The notation 'I/Z' is ambiguous. If I is the identity matrix, it is already row-stochastic, and dividing by a scalar changes row sums; please clarify the intended normalization.
  3. [References] Reference [61] contains a malformed URL ('https://https://huggingface.co/...'). Please check all URLs and correct this typo.
  4. [§5] The comparison 'MAESTRO at 50% (93.40%) eclipses HC-SMoE at 25% (87.47%) by 6.78%' mixes relative and absolute language. Please state explicitly whether this is a relative or absolute percentage difference.

Circularity Check

1 steps flagged

No significant circularity: MAESTRO's benchmark gains are estimated on held-out tasks, not fitted by the Markov-chain construction; only the 'by construction' importance sentence is a definitional restatement.

specific steps
  1. self definitional [Introduction, echoed in Section 3.2]
    "Experts with the smallest stationary mass are, by construction, the least frequently visited across all routing trajectories and are the natural candidates for removal."

    The stationary mass π is defined (Eqs. 4–9) as the long-run frequency of the constructed Markov chain, so 'smallest stationary mass' and 'least frequently visited' are the same quantity by definition; the sentence adds no independent evidence that these experts are least important. Eq. 10 then selects exactly the bottom-Kℓ experts in π, making the importance claim a restatement of the pruning criterion rather than a derived result. The independent content of the paper is the held-out benchmark retention in Table 1, which is not used to fit π and does not reduce to the Markov-chain construction.

full rationale

MAESTRO's derivation chain is self-contained for the purposes of the empirical claim. Transition counts (Eq. 4), the global kernel (Eqs. 5–7), ε-smoothing (Eq. 8), and power iteration (Eq. 9) are estimated from 250 SlimOrca calibration rollouts, while all reported retention numbers come from 17 held-out benchmarks (Tables 1, 4, 5); no benchmark result is a fitted target of the Markov-chain construction. The self-citation [12] appears only as background related work and is not load-bearing. The k-subset/top-k collapse objection raised in the skeptical headline is a modeling-validity concern about whether the chain faithfully represents routing, not a case in which the output is equal to the input. Section 8 explicitly admits the first-order Markov assumption, and the choice of 250 calibration samples is a mild test-set-tuning concern rather than definitional circularity. The only definitional restatement is the 'by construction' sentence in the Introduction/Section 3.2, which merely equates π with visit frequency; this does not compromise the empirical comparison.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The central claim rests on: (a) standard Markov-chain theory (Perron-Frobenius), invoked correctly but with an irreducibility caveat — the empirical chain is only guaranteed irreducible 'on the support of the calibration trajectories'; (b) two unverified domain assumptions — the first-order Markov property of routing (the authors' own admitted limitation) and the representativeness of the 250-sample SlimOrca autoregressive calibration (Table 3 shows 15-21% Std-RP shifts with calibration size); (c) the heuristic premise that low stationary mass implies safely prunable experts; and (d) an ad hoc row-normalization/clipping scheme (Eq. 7) whose behavior on zero-count rows is glossed over. The free parameters are mostly benign (ε, τ, Nτ), except the calibration size, which was selected with evaluation outcomes in the loop.

free parameters (7)
  • ε (smoothing weight) = 1e-5
    Self-loop weight in Eq. (8) to induce aperiodicity; hand-set, no sensitivity analysis reported.
  • τ (power iteration tolerance) = 1e-6
    Stopping tolerance for stationary-distribution computation (Section 3.4); standard, low impact.
  • Nτ (max power iterations) = 1000
    Cap on power iteration (Section 3.4); computational, low impact.
  • Tseq (autoregressive steps) = 100
    Number of generated tokens per calibration context (Section 3.3); controls which routing transitions are observed; no sensitivity study.
  • calibration corpus size = 250 SlimOrca samples of 512 tokens
    Selected after comparing 50/100/500 samples on the evaluation benchmarks (Table 3) — a choice informed by target-task outcomes.
  • uniform per-layer pruning budget = K_ℓ = ⌊ρE⌋ per layer
    Design choice validated by the UNI ablation (Table 2), but it constrains the method to layer-uniform removal.
  • RFT hyperparameters = LoRA r=16, α=16, LR 2e-4, batch 2, 1 epoch, 2000 conversations
    Shared across all methods; recovery quality depends heavily on this stage (Table 2 shows ±20% relative swings without it).
axioms (5)
  • standard math Perron-Frobenius: an irreducible, aperiodic finite Markov chain has a unique stationary distribution
    Invoked in Section 3.4 to justify π; standard result, but the empirical chain is only claimed irreducible 'on the support of the calibration trajectories'.
  • domain assumption First-order Markov property of expert routing
    Section 3.2; admitted in Section 8 as potentially underestimating experts whose relevance depends on longer routing histories.
  • domain assumption Calibration transitions on 250 SlimOrca samples with 100 autoregressive steps approximate the deployment routing distribution
    Section 3.3/4; Table 3 shows cross-task Std RP varies by 15-21% with calibration size, indicating sensitivity to this assumption.
  • domain assumption Low stationary mass implies low contribution to task performance, so those experts are safe to prune
    Core premise of Section 3.5; asserted rather than derived, and evaluated empirically on the benchmark suite.
  • ad hoc to paper The max(1, row-sum) normalization (Eq. 7) plus ε-smoothing (Eq. 8) yields a well-defined stochastic kernel over all LE states
    Zero-count rows become all-zero in P; εI/Z adds a self-loop but states with no incoming counts can remain isolated, so the uniqueness claim does not strictly hold on the full state space.

pith-pipeline@v1.3.0-alltime-deepseek · 22177 in / 28966 out tokens · 280449 ms · 2026-08-02T07:50:01.511722+00:00 · methodology

0 comments
read the original abstract

Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.

Figures

Figures reproduced from arXiv: 2607.08601 by Ayush Maheshwari, Palaash Goel, Tanmoy Chakraborty.

Figure 1
Figure 1. Figure 1: Schematic of MAESTRO’s methodology. MAESTRO visualizes expert routing in MoE LLMs in the form of an Ergodic Markov chain and uses its stationary distribution as a global heuristic to remove redundant experts from each layer. et al., 2026; Lasby et al., 2026) that propose heuris￾tics such as activation variability scores (Hu et al., 2026) and router-weighted expert norms (Lasby et al., 2026) to better captu… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of uniform vs. non-uniform expert pruning strategies applied to GPT-OSS-20B at 25% [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of the run-times of different prun [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding

    cs.LG 2026-07 conditional novelty 7.0

    Half the experts in Qwen3.6-35B-A3B and Gemma-4-26B-A4B can be pruned without detectable HumanEval+ loss, but the winning selection criterion flips per family and pruning beats quantization only below ~3-bit weight precision.

Reference graph

Works this paper leans on

63 extracted references · 22 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P . Stock, S. Subramanian, S. Y ang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mixtral of experts,” 2024. [O...

    Pith/arXiv arXiv 2024

  2. [2]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [Online]. Available: https://arxiv.org/abs/2501.12948 12 P . Goel, A. Maheshwari and T. Chakraborty

    Pith/arXiv arXiv 2025

  3. [3]

    Qwen3 technical report,

    A. Y ang, A. Li, B. Y ang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Y ang, J. Tu, J. Zhang, J. Y ang, J. Y ang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Y ang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P . Zhang, P . Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. ...

    Pith/arXiv arXiv 2025

  4. [4]

    gpt-oss-120b & gpt-oss-20b model card,

    OpenAI, :, S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, E. Arbus, R. K. Arora, Y . Bai, B. Baker, H. Bao, B. Barak, A. Bennett, T. Bertao, N. Brett, E. Brevdo, G. Brockman, S. Bubeck, C. Chang, K. Chen, M. Chen, E. Cheung, A. Clark, D. Cook, M. Dukhan, C. Dvorak, K. Fives, V . Fomenko, T. Garipov, K. Georgiev, M. Glaese, T. Gogineni, A. Goucher, ...

    Pith/arXiv arXiv 2025

  5. [5]

    Nvidia nemotron 3: Efficient and open intelligence,

    NVIDIA, :, A. Blakeman, A. Grattafiori, A. Basant, A. Gupta, A. Khattar, A. Renduchintala, A. Vavre, A. Shukla, A. Bercovich, A. Ficek, A. Shaposhnikov, A. Kondratenko, A. Bukharin, A. Milesi, A. Taghibakhshi, A. Liu, A. Barton, A. S. Mahabaleshwarkar, A. Klein, A. Zuker, A. Geifman, A. Shen, A. Bhiwandiwalla, A. Tao, A. Agrusa, A. Verma, A. Guan, A. Mand...

    Pith/arXiv arXiv 2025

  6. [6]

    Llm-pruner: On the structural pruning of large language models,

    X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” in Advances in Neural Information Processing Systems, 2023

    2023

  7. [7]

    Slicegpt: Compress large language models by deleting rows and columns,

    S. Ashkboos, M. L. Croci, M. G. do Nascimento, T. Hoefler, and J. Hensman, “Slicegpt: Compress large language models by deleting rows and columns,” 2024. [Online]. Available: https://arxiv.org/abs/2401.15024

    Pith/arXiv arXiv 2024

  8. [8]

    ShortGPT: Layers in large language models are more redundant than you expect,

    X. Men, M. Xu, Q. Zhang, Q. Yuan, B. Wang, H. Lin, Y . Lu, X. Han, and W. Chen, “ShortGPT: Layers in large language models are more redundant than you expect,” in Findings of the Association for Computational Linguistics: ACL 2025 , W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational Linguistics, Jul. 20...

    2025

  9. [9]

    Slimllm: Accurate structured pruning for large language models,

    J. Guo, X. Chen, Y . Tang, and Y . Wang, “Slimllm: Accurate structured pruning for large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2505.22689

    Pith/arXiv arXiv 2025

  10. [10]

    Replaceme: Network simplification via depth pruning and transformer block linearization,

    D. Shopkhoev, A. Ali, M. Zhussip, V . Malykh, S. Lefkimmiatis, N. Komodakis, and S. Zagoruyko, “Replaceme: Network simplification via depth pruning and transformer block linearization,” 2026. [Online]. Available: https: //arxiv.org/abs/2505.02819

    arXiv 2026

  11. [11]

    Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks,

    J. Song, K. Oh, T. Kim, H. Kim, Y . Kim, and J.-J. Kim, “Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks,” in Proceedings of the 41st International Conference on Machine Learning , 2024

    2024

  12. [12]

    Y ou only prune once: Designing calibration-free model compression with policy learning,

    A. Sengupta, S. Chaudhary, and T. Chakraborty, “Y ou only prune once: Designing calibration-free model compression with policy learning,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=5RZoYIT3u6

    2025

  13. [13]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,

    S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” 2016. [Online]. Available: https://arxiv.org/abs/1510.00149

    Pith/arXiv arXiv 2016

  14. [14]

    A simple and effective pruning approach for large language models,

    M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A simple and effective pruning approach for large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2306.11695

    Pith/arXiv arXiv 2024

  15. [15]

    Optimal brain compression: a framework for accurate post-training quantization and pruning,

    E. Frantar, S. P . Singh, and D. Alistarh, “Optimal brain compression: a framework for accurate post-training quantization and pruning,” in Proceedings of the 36th International Conference on Neural Information Processing Systems , ser. NIPS ’22. Red Hook, NY, USA: Curran Associates Inc., 2022

    2022

  16. [16]

    Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,

    X. Lu, Q. Liu, Y . Xu, A. Zhou, S. Huang, B. Zhang, J. Y an, and H. Li, “Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , L.-W . Ku, A. Martins, and V . Srikumar, Eds. Bangkok, T...

    2024

  17. [17]

    Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router,

    Y . Xie, Z. Zhang, D. Zhou, C. Xie, Z. Song, X. Liu, Y . Wang, X. Lin, and A. Xu, “Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router,” 2024. [Online]. Available: https://arxiv.org/abs/2410.12013

    Pith/arXiv arXiv 2024

  18. [18]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks,

    J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” 2019. [Online]. Available: https://arxiv.org/abs/1803.03635

    Pith/arXiv arXiv 2019

  19. [19]

    Are sixteen heads really better than one?

    P . Michel, O. Levy, and G. Neubig, “Are sixteen heads really better than one?” 2019. [Online]. Available: https://arxiv.org/abs/1905.10650

    Pith/arXiv arXiv 2019

  20. [20]

    Large language models are overparameterized text encoders,

    T. D. K, T. Fischer, and C. Biemann, “Large language models are overparameterized text encoders,” in Proceedings of the 10th Workshop on Representation Learning for NLP (RepL4NLP-2025) , V . Adlakha, A. Chronopoulou, X. L. Li, B. P . Majumder, F. Shi, and G. Vernikos, Eds. Albuquerque, NM: Association for Computational Linguistics, May 2025, pp. 170–184. ...

    2025

  21. [21]

    Efficient 8-bit quantization of transformer neural machine language translation model,

    A. Bhandare, V . Sripathi, D. Karkada, V . Menon, S. Choi, K. Datta, and V . Saletore, “Efficient 8-bit quantization of transformer neural machine language translation model,” 2019. [Online]. Available: https://arxiv.org/abs/1906.00532 14 P . Goel, A. Maheshwari and T. Chakraborty

    Pith/arXiv arXiv 2019

  22. [22]

    Zeroquant: efficient and affordable post-training quantiza- tion for large-scale transformers,

    Z. Y ao, R. Y . Aminabadi, M. Zhang, X. Wu, C. Li, and Y . He, “Zeroquant: efficient and affordable post-training quantiza- tion for large-scale transformers,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY, USA: Curran Associates Inc., 2022

    2022

  23. [23]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” 2023. [Online]. Available: https://arxiv.org/abs/2210.17323

    Pith/arXiv arXiv 2023

  24. [24]

    QLLM: Accurate and efficient low-bitwidth quantization for large language models,

    J. Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang, “QLLM: Accurate and efficient low-bitwidth quantization for large language models,” in International Conference on Learning Representations (ICLR) , 2024

    2024

  25. [25]

    Training-free activation sparsity in large language models,

    J. Liu, P . Ponnusamy, T. Cai, H. Guo, Y . Kim, and B. Athiwaratkun, “Training-free activation sparsity in large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2408.14690

    Pith/arXiv arXiv 2025

  26. [26]

    Weight-aware activation sparsity with constrained Bayesian optimization scheduling for large language models,

    M. Wang, M. Zhang, X. Liu, and L. Nie, “Weight-aware activation sparsity with constrained Bayesian optimization scheduling for large language models,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, Eds. Suzhou, China: Association for Computational Lin...

    2025

  27. [27]

    Sparsing law: towards large language models with greater activation sparsity,

    Y . Luo, C. Song, X. Han, Y . Chen, C. Xiao, X. Meng, L. Deng, J. Wei, Z. Liu, and M. Sun, “Sparsing law: towards large language models with greater activation sparsity,” in Proceedings of the 42nd International Conference on Machine Learning, ser. ICML ’25. JMLR.org, 2025

    2025

  28. [28]

    Acttail: Global activation sparsity in large language models,

    W. Hou, X. Song, and S. Liu, “Acttail: Global activation sparsity in large language models,” 2026. [Online]. Available: https://arxiv.org/abs/2603.12272

    arXiv 2026

  29. [29]

    Sparsegpt: massive language models can be accurately pruned in one-shot,

    E. Frantar and D. Alistarh, “Sparsegpt: massive language models can be accurately pruned in one-shot,” in Proceedings of the 40th International Conference on Machine Learning , ser. ICML ’23. JMLR.org, 2023

    2023

  30. [30]

    BlockPruner: Fine-grained pruning for large language models,

    L. Zhong, F. Wan, R. Chen, X. Quan, and L. Li, “BlockPruner: Fine-grained pruning for large language models,” in Findings of the Association for Computational Linguistics: ACL 2025 , W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational Linguistics, Jul. 2025, pp. 5065–5080. [Online]. Available: https://ac...

    2025

  31. [31]

    Finercut: Finer-grained interpretable layer pruning for large language models,

    Y . Zhang, Y . Li, X. Wang, Q. Shen, B. Plank, B. Bischl, M. Rezaei, and K. Kawaguchi, “Finercut: Finer-grained interpretable layer pruning for large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2405.18218

    Pith/arXiv arXiv 2024

  32. [32]

    ALPS: Attention localization and pruning strategy for efficient adaptation of large language models,

    H. Chen, H. Li, Z. Xiao, L. Gao, Q. Zhang, X. Hu, N. Wang, X. Fu, and J. Zhao, “ALPS: Attention localization and pruning strategy for efficient adaptation of large language models,” in Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational ...

    2025

  33. [33]

    Fairness-aware structured pruning in transformers,

    A. Zayed, G. Mordido, S. Shabanian, I. Baldini, and S. Chandar, “Fairness-aware structured pruning in transformers,” in Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence , s...

    doi:10.1609/aaai.v38i20.30256 2024

  34. [34]

    STUN: Structured-then-unstructured pruning for scalable MoE pruning,

    J. Lee, S.-w. Hwang, A. Qiao, D. F. Campos, Z. Y ao, and Y . He, “STUN: Structured-then-unstructured pruning for scalable MoE pruning,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational L...

    2025

  35. [35]

    Mosaic pruning: A hierarchical framework for generalizable pruning of mixture-of-experts models,

    W. Hu, M. Zhao, S. Song, X. Zhu, X. Lai, and J. Wang, “Mosaic pruning: A hierarchical framework for generalizable pruning of mixture-of-experts models,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 40, no. 26, p. 21885–21893, Mar. 2026. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/39341

    2026

  36. [36]

    REAP the experts: Why pruning prevails for one-shot moe compression,

    M. Lasby, I. Lazarevich, N. Sinnadurai, S. Lie, Y . Ioannou, and V . Thangarasa, “REAP the experts: Why pruning prevails for one-shot moe compression,” in The Fourteenth International Conference on Learning Representations , 2026. [Online]. Available: https://openreview.net/forum?id=ukGxWd2aDG 15 It Takes a MAESTRO To Prune Bad Experts

    2026

  37. [37]

    Merging experts into one: Improving computational efficiency of mixture of experts,

    S. He, R.-Z. Fan, L. Ding, L. Shen, T. Zhou, and D. Tao, “Merging experts into one: Improving computational efficiency of mixture of experts,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 14 685–14 691. [...

    2023

  38. [38]

    Retraining-free merging of sparse moe via hierarchical clustering,

    I.-C. Chen, H.-S. Liu, W.-F. Sun, C.-H. Chao, Y .-C. Hsu, and C.- Y . Lee, “Retraining-free merging of sparse moe via hierarchical clustering,” 2025

    2025

  39. [39]

    Merge, then compress: Demystify efficient smoe with hints from its routing policy,

    P . Li, Z. Zhang, P . Y adav, Y .-L. Sung, Y . Cheng, M. Bansal, and T. Chen, “Merge, then compress: Demystify efficient smoe with hints from its routing policy,” 2024. [Online]. Available: https://arxiv.org/abs/2310.01334

    Pith/arXiv arXiv 2024

  40. [40]

    Soft merging of experts with adaptive routing,

    M. Muqeeth, H. Liu, and C. Raffel, “Soft merging of experts with adaptive routing,” 2024. [Online]. Available: https://arxiv.org/abs/2306.03745

    Pith/arXiv arXiv 2024

  41. [41]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P . Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2106.09685

    Pith/arXiv arXiv 2021

  42. [42]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” 2016. [Online]. Available: https://arxiv.org/abs/1609.07843

    Pith/arXiv arXiv 2016

  43. [43]

    The LAMBADA dataset: Word prediction requiring a broad discourse context,

    D. Paperno, G. Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández, “The LAMBADA dataset: Word prediction requiring a broad discourse context,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , K. Erk and N. A. Smith, Eds. Berlin, Germany:...

    2016

  44. [44]

    Piqa: Reasoning about physical commonsense in natural language,

    Y . Bisk, R. Zellers, R. L. Bras, J. Gao, and Y . Choi, “Piqa: Reasoning about physical commonsense in natural language,” in Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  45. [45]

    PROST: Physical reasoning about objects through space and time,

    S. Aroca-Ouellette, C. Paik, A. Roncone, and K. Kann, “PROST: Physical reasoning about objects through space and time,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , C. Zong, F. Xia, W. Li, and R. Navigli, Eds. Online: Association for Computational Linguistics, Aug. 2021, pp. 4597–4608. [Online]. Available: https://aclant...

    2021

  46. [46]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge,

    A. Talmor, J. Herzig, N. Lourie, and J. Berant, “CommonsenseQA: A question answering challenge targeting commonsense knowledge,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Minneapolis, Minnesota: Association for Comput...

    2019

  47. [47]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P . Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” 2018. [Online]. Available: https://arxiv.org/abs/1803.05457

    Pith/arXiv arXiv 2018

  48. [48]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams,

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P . Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” Applied Sciences , vol. 11, no. 14, 2021. [Online]. Available: https://www.mdpi.com/2076-3417/11/14/6421

    2021

  49. [49]

    Can a suit of armor conduct electricity? a new dataset for open book question answering,

    T. Mihaylov, P . Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, Eds. Brussels, Belgium: Association for Computational Linguistics, Oct....

    2018

  50. [50]

    Blimp: The benchmark of linguistic minimal pairs for english,

    A. Warstadt, A. Parrish, H. Liu, A. Mohananey, W. Peng, S.-F. Wang, and S. R. Bowman, “Blimp: The benchmark of linguistic minimal pairs for english,” Transactions of the Association for Computational Linguistics , vol. 8, pp. 377–392,

  51. [51]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions,

    C. Clark, K. Lee, M.-W . Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “BoolQ: Exploring the surprising difficulty of natural yes/no questions,” in Proceedings of the 2019 Conference of the No rth American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, a...

    2019

  52. [52]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,” Communications of the ACM, vol. 64, no. 9, pp. 99–106, 2021

    2021

  53. [53]

    CoQA: A conversational question answering challenge,

    S. Reddy, D. Chen, and C. D. Manning, “CoQA: A conversational question answering challenge,” Transactions of the Association for Computational Linguistics , vol. 7, pp. 249–266, 2019. [Online]. Available: https://aclanthology.org/ Q19-1016/

    2019

  54. [54]

    Gender bias in coreference resolution,

    R. Rudinger, J. Naradowsky, B. Leonard, and B. Van Durme, “Gender bias in coreference resolution,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), M. Walker, H. Ji, and A. Stent, Eds. New Orleans, Louisiana: Association for Computati...

    2018

  55. [55]

    TruthfulQA: Measuring how models mimic human falsehoods,

    S. Lin, J. Hilton, and O. Evans, “TruthfulQA: Measuring how models mimic human falsehoods,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , S. Muresan, P . Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 3214–3252. [Online]. Ava...

    2022

  56. [56]

    Moral stories: Situated reasoning about norms, intents, actions, and their consequences,

    D. Emelin, R. Le Bras, J. D. Hwang, M. Forbes, and Y . Choi, “Moral stories: Situated reasoning about norms, intents, actions, and their consequences,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Online and Punta Cana, Dominican Republic: Association fo...

    2021

  57. [57]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Y ang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” 2019. [Online]. Available: https://arxiv.org/...

    Pith/arXiv arXiv 2019

  58. [58]

    Transformers: State-of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P . Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P . von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Met...

    2020

  59. [59]

    PEFT: State-of-the-art parameter- efficient fine-tuning methods,

    S. Mangrulkar, S. Gugger, L. Debut, Y . Belkada, S. Paul, B. Bossan, and M. Tietz, “PEFT: State-of-the-art parameter- efficient fine-tuning methods,” https://github.com/huggingface/peft, 2022

    2022

  60. [60]

    The language model evaluation harness,

    L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “The language model evaluation harness,” 07 2024. [Online]. Available: https://zenodo.org/re...

    arXiv 2024

  61. [61]

    Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification,

    W. Lian, G. Wang, B. Goodson, E. Pentland, A. Cook, C. Vong, and ”Teknium”, “Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification,” 2023. [Online]. Available: https://https: //huggingface.co/Open-Orca/SlimOrca

    2023

  62. [62]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations, 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:53592270 17 It Takes a MAESTRO To Prune Bad Experts CR Method Generative World Understanding Domain-Specific Wikitext Lambada PIQA PROST CommonsenseQA OpenbookQA ...

    2017

  63. [2020]

    Available: https://doi.org/10.1162/tacl_a_00321

    [Online]. Available: https://doi.org/10.1162/tacl_a_00321

    doi:10.1162/tacl_a_00321