Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

LExI makes the case that the optimal number of active MoE experts differs per layer and can be set from weights alone, outperforming expert pruning on accuracy at equal throughput.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LExI sets a different number of active experts per layer, found by weight-only sensitivity profiling and evolutionary search, improving MoE inference throughput with little accuracy loss.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful data-free MoE optimization recipe, but the central claim about layer-adaptivity is unproven because there's no uniform-top-k control. the 4 major comments →

arxiv 2509.02753 v1 pith:IVYN7WDG submitted 2025-09-02 cs.LG

LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference

Krishna Teja Chitty-Venkata , Sandeep Madireddy , Murali Emani , Venkatram Vishwanath This is my paper
classification cs.LG
keywords Mixture-of-ExpertsMoE inferencelayer-adaptive top-kexpert pruningdata-free post-training optimizationperturbation-loss sensitivityevolutionary search
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Post-training expert pruning shrinks Mixture-of-Experts (MoE) models, but the paper argues it does not reliably speed up inference on optimized GPU stacks, because every token still hits the same number of experts and the remaining experts take on unbalanced token loads. LExI is a data-free alternative: it uses only the model's weights to estimate how much each layer's output changes when the number of active experts is reduced, then searches for a per-layer assignment of active-expert counts under a total compute budget. The central claim is that different layers have different tolerances for dropping experts, so allocating fewer experts to low-sensitivity layers preserves accuracy while cutting computation. On language and vision MoE benchmarks, the paper reports accuracy close to the unpruned model at throughput comparable to or better than inter-expert pruning (removing whole experts) and intra-expert pruning (thinning each expert's weights), with the headline example that Qwen1.5-MoE matches the throughput of traditional expert pruning with about 10% better accuracy. If right, this gives a plug-in inference optimization that needs no calibration data or retraining.

Core claim

LExI claims that the fixed top-k routing used in pretrained MoE models is suboptimal layer by layer: the number of experts that should be active per token is not uniform across depth. The paper's central discovery is that a per-layer, data-free sensitivity measure—the average Frobenius-norm change in a layer's output when its top-k is lowered, computed under random Gaussian inputs—predicts where expert capacity can be safely cut. A low-cost evolutionary search then assigns each layer a top-k value under a total active-expert budget, minimizing the summed perturbation loss. The authors report that this layer-adaptive schedule preserves task accuracy and perplexity near the unpruned baseline w

What carries the argument

The carrying mechanism is the per-layer top-k perturbation loss. For each MoE layer, LExI runs the layer on random Gaussian inputs at the baseline top-k and at each candidate top-k, and records the average Frobenius norm between the two outputs; a low value says the layer tolerates fewer active experts. These losses serve as a fitness proxy for an evolutionary search that picks one top-k per layer subject to a global sum-of-experts budget and per-layer min/max bounds, so the search never needs to load the full model. The same sensitivity values also reveal that sensitivity patterns differ by architecture—some models tolerate cuts in early layers, others in middle or late layers—which is why

Load-bearing premise

The load-bearing assumption is that a layer's output change on random Gaussian inputs predicts how much accuracy it will lose on real text or image tokens when its active-expert count is reduced; if those two rankings diverge, LExI's chosen top-k schedule will not be the accuracy-optimal one for its budget.

What would settle it

Run LExI on one of the evaluated MoE models to get its per-layer sensitivity ranking, then measure the actual accuracy loss of dropping one expert at a time in each layer on a held-out task set. If the weights-only ranking correlates poorly with the real per-layer accuracy loss, the proxy is the wrong signal; a stronger test is to compare two equal-budget allocations—one chosen by LExI and one chosen by the same search but with perturbation losses computed on real tokens—and show the data-free allocation is consistently worse on accuracy at matched throughput.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On six MoE models spanning language and vision-language tasks, LExI reports better accuracy-to-throughput trade-offs than inter-expert pruning and intra-expert pruning at matching budgets.
  • Because LExI computes fewer experts per token without deleting parameters, it cuts compute, memory-bandwidth use, and inter-GPU communication while leaving model size unchanged.
  • The method is data-free and requires no fine-tuning, so it can be applied to an already-trained MoE as a one-time post-training step.
  • LExI is complementary to pruning: pruning can still shrink the model in memory, while LExI reduces per-token compute.
  • The method cannot be applied when a model already uses the minimum possible top-k per layer (e.g., one active expert), because there is no room to reduce the count.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The core proxy assumption is testable: compute LExI's per-layer sensitivity ranking and compare it with per-layer accuracy loss measured on real tokens; if the rankings diverge, a calibration-free allocation is not necessarily accuracy-optimal.
  • The evolutionary objective sums independent per-layer losses, which implicitly treats layers as independent; in deep models, a capacity cut in an early layer may amplify or cancel a cut in a later layer, so the true optimal allocation might be non-separable.
  • Although the paper presents LExI for routing allocation, the same perturbation-loss profiles could double as a layer-sensitivity oracle for mixed-precision quantization or layer pruning, with the caveat that those use cases need their own validation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces LExI, a post-training, data-free method that assigns a static per-layer active-expert count (top-k) to a pretrained mixture-of-experts (MoE) model. Stage 1 estimates, for each layer and each candidate top-k, the average Frobenius-norm deviation of the layer output from the baseline output using random Gaussian inputs. Stage 2 runs an evolutionary search over allocations that satisfy a total active-expert budget B, minimizing the sum of these per-layer deviation proxies. The authors then compare accuracy versus throughput against inter- and intra-expert pruning baselines on five LLMs and one VLM, reporting that LExI improves the accuracy-throughput trade-off and, e.g., matches the throughput of 50% intra-pruning with 10% higher accuracy on OLMoE-1B-7B.

Significance. If the central claim held, LExI would be a practical and lightweight inference-time optimization: it requires no calibration data, no fine-tuning, and is simple to implement. The paper's strengths include evaluation across diverse MoE architectures (Mixtral, Qwen, OLMoE, MiniCPM, DeepSeek-V2, DeepSeek-VL2), multiple evaluation families (LM-Eval, LongBench, passkey retrieval, perplexity, and VLM tasks), and a clearly described algorithmic pipeline. However, the significance is currently limited by the absence of a direct test of the layer-adaptivity premise and by an unvalidated sensitivity proxy. The method is not circular, since the proxy is used for allocation and accuracy is measured independently, but the choice of proxy and the missing uniform-top-k baseline leave the core claim unsupported until additional experiments are supplied.

major comments (4)
  1. [Section 4 and Section 5, Figures 4-8] The central claim is that layer-adaptive top-k assignment is superior to the conventional fixed uniform top-k. Yet no uniform-top-k baseline is reported anywhere. All comparisons are against inter- and intra-expert pruning, which keep the original top-k and instead remove experts or weights. A natural control is a uniform allocation with the same total active-expert budget B as each LExI run; for example, Qwen1.5-MoE with B=48 should be compared against a uniform k=2 in all 24 layers, and Mixtral with B=40 against the closest feasible uniform integer allocation. Without such a control, the observed gains could be explained entirely by reducing the number of active experts rather than by adaptively choosing where to reduce. This is load-bearing because the paper's motivation in Section 1 explicitly argues that fixed uniform top-k is suboptimal. Please add the uniform-reduction baseline fo
  2. [Section 3, 'Inter and Intra Expert Pruning'] The pruning baselines are underspecified. The text defines 12.5%, 25%, and 50% pruning but does not describe which experts are removed, what importance criterion is used, whether calibration data is employed, or how the pruned model is deployed after pruning. The citations to NAEE and MoE-I2 are insufficient if the implementation is not identical to those methods. Without this information, the comparison is not reproducible, and the reported superiority of LExI may reflect suboptimally chosen pruning baselines rather than a genuine advantage. Please specify the exact pruning procedure, the selection criterion, and any hyperparameters used.
  3. [Algorithm 1, Section 4] The random-Gaussian sensitivity proxy is not validated. The perturbation loss is computed on inputs drawn from N(0,1), but MoE hidden-state distributions are not Gaussian and vary across layers and models. The paper states that the Monte Carlo procedure gives a 'statistically robust estimate,' but it does not show that the proxy ranking correlates with actual task accuracy or perplexity changes when top-k is reduced. A concrete validation would be to compare the proxy ranking against the true loss change on a few layers, or to compare the allocation found by the proxy with an allocation found using a small real-data calibration set. If the proxy is misaligned with real token statistics, the resulting allocation may not maximize accuracy under the budget, and the reported trade-offs may not hold on real workloads.
  4. [Section 5, Figures 4-8] No error bars or statistical significance are reported for throughput or accuracy. Some claimed gains are small, e.g., '+0.5% accuracy' on Qwen1.5-MoE in the LM-Eval results, and vLLM throughput can vary between runs. The paper should report multiple runs with standard deviations or confidence intervals, or at least describe the variance across runs. Without this information, the reader cannot determine whether the reported improvements are within experimental noise.
minor comments (6)
  1. [Algorithm 2, Section 4] Evolutionary-search hyperparameters Npop, Gmax, eta_mut, kmin, and kmax are not specified. Report the values used for each model and the computational cost of the search.
  2. [Algorithm 2, Section 4] The output is described as 'Optimal topk allocation,' but the evolutionary search is a heuristic and does not guarantee global optimality. The separable resource-allocation problem can be solved exactly by dynamic programming for the small search spaces considered. At minimum, acknowledge this or compare with an exact solver.
  3. [Section 4, text after Figure 3] Typo: 'latency can be can be optimized' should read 'latency can be optimized.'
  4. [Abstract] Minor grammar issue: 'improve only the memory footprint but provide limited gains' reads awkwardly; suggest 'improve the memory footprint but provide limited gains in inference-time compute efficiency.'
  5. [Section 4, Algorithm 1] The paper mentions 'millions of random input samples' but does not report Niter. Please provide the exact number of iterations and the profiling time per model.
  6. [Section 6, Limitations] The limitations section already notes that LExI does not reduce memory and is inapplicable when the baseline top-k is 1. It should also explicitly state that the total active-expert budget B is a user-specified input, not automatically determined by the method.

Circularity Check

0 steps flagged

No significant circularity; LExI's layer-adaptive top-k allocation is derived from an independently evaluated proxy and benchmarked against external accuracy/throughput metrics.

full rationale

The paper's derivation chain is not circular. Stage 1 (Algorithm 1) defines a perturbation proxy: for each layer and candidate top-k, it computes the Frobenius norm between the baseline MoE output (at default top-k) and the output under the reduced top-k, using random Gaussian inputs. Stage 2 (Algorithm 2) minimizes the sum of these per-layer proxy losses under a fixed active-expert budget B, yielding a layer-wise allocation k*. This allocation is then deployed on real models and evaluated on LM-Eval, LongBench, passkey retrieval, perplexity, and VLM benchmarks. The proxy is not defined in terms of those evaluation metrics, and the reported accuracy/throughput results are measured independently of the proxy. There is no equation in which the predicted outcome (e.g., accuracy gain over pruning) is identical by construction to the fitted objective; the evolutionary search only minimizes the perturbation proxy, and whether that proxy transfers to task accuracy is an empirical question, not a tautology. The paper also does not rely on load-bearing self-citation: its novelty claim is based on its own weight-only perturbation profiling, and existing pruning methods are cited as external baselines. The absence of a uniform-top-k baseline is a substantive experimental gap regarding the paper's claim that layer-adaptivity itself is superior to a uniform reduction of active experts, but that is a comparison-design issue, not circularity. The hand-selected budgets B are chosen by the authors, but choosing a budget and then optimizing within it is not circular; the optimization does not presuppose which layers should be reduced. Therefore, no circular step meeting the required evidentiary standard is present, and the paper's central empirical comparisons are self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

No new physical or architectural entities are introduced; LExI is a method, not an entity. The load-bearing assumptions are the transferability of the Gaussian-input proxy, the feasibility of variable top-k in vLLM, and the reuse of pretrained routers at reduced top-k.

free parameters (4)
  • Active expert budget B per model = OLMoE: 64, 72, 100; Qwen1.5-MoE: 48, 56, 72, 80; MiniCPM: 50, 60, 70; Mixtral: 40, 48, 56; DeepSeek-V2-Lite: 78, 104, 13
    Budgets are chosen per model to define the total number of active experts across layers; the paper does not justify why these specific values are reported, so the Pareto front may be cherry-picked.
  • Monte Carlo iterations Niter for sensitivity profiling = not specified (text says 'millions')
    Niter affects the stability of the perturbation-loss estimates; without a value, the profiling is not exactly reproducible.
  • Evolutionary search hyperparameters (Npop, Gmax, eta_mut, kmin, kmax) = not specified
    Algorithm 2 defines these parameters but the paper gives no values, so the search cannot be reproduced.
  • Synthetic input distribution for profiling = N(0,1)
    Choosing Gaussian inputs is a modeling choice; the paper provides no validation that this distribution matches real token embeddings, which is load-bearing for the proxy.
axioms (3)
  • domain assumption The Frobenius-norm output perturbation measured under random Gaussian inputs is a valid proxy for a layer's sensitivity to reducing top-k on real data.
    Stage 1 (Algorithm 1) uses only synthetic inputs; the paper assumes this transfers to actual token distributions without evidence.
  • domain assumption The inference framework (vLLM/FusedMoE) can be modified to execute per-layer variable top-k without extra overhead that would offset the compute savings.
    All throughput claims rely on this implementation capability, but no patch or overhead analysis is provided.
  • domain assumption A pretrained MoE router can be used with reduced top-k values per layer without retraining or calibration and still produce useful gating decisions.
    The method assumes the existing gating weights remain valid when fewer experts are selected; the paper does not analyze routing distortion from masking out top experts.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference." pith.science (2026). https://pith.science/paper/IVYN7WDG

@misc{pith2026250902753,
  author       = {Pith},
  title        = {Pith review of: LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IVYN7WDG}},
  note         = {Machine review of arXiv:2509.02753}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Mixture-of-Experts (MoE) models scale efficiently by activating only a subset of experts per token, offering a computationally sparse alternative to dense architectures. While prior post-training optimizations, such as inter- and intra-expert pruning, reduce memory usage they provide limited gains in inference-time compute efficiency. Moreover, existing MoE architectures typically activate a fixed number of experts uniformly across all layers, resulting in redundant computation and suboptimal performance. In this work, we first demonstrate that MoE pruning strategies improve only the memory footprint but do not significantly improve inference performance on GPU using optimized frameworks such as vLLM. To address this, we introduce LExI, a data-free optimization technique that determines the optimal number of active experts per layer in a pretrained MoE model. LExI leverages only the model weights to estimate the relative importance of each layer and adaptively assigns the number of active experts accordingly per layer. Experiments on state-of-the-art language and vision MoE benchmarks demonstrate that LExI significantly outperforms traditional MoE pruning approaches in terms of inference efficiency with negligible accuracy loss. For example, using LExI, Qwen1.5-MoE achieves the same throughput on Nvidia H100 GPU with 10% better accuracy than traditional expert pruning.

Figures

Figures reproduced from arXiv: 2509.02753 by Krishna Teja Chitty-Venkata, Murali Emani, Sandeep Madireddy, Venkatram Vishwanath.

Figure 1
Figure 1. Figure 1: Overview of MoE Optimization Methods. (a) Baseline Trained Model (b) Unstructured [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Throughput vs. Active Experts under Inter and Intra Expert Pruning [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Top-k sensitivity analysis. The heatmap plots depict the layer-wise output deviation with respect to changing the top-k. The initial layers in Mixtral model are less sensitive to top-k perturbation than deeper layers, while OLMoE exhibits a bell curve pattern where initial and last layers are more sensitive. Heatmaps for MiniCPM and DeepSeekV2 are shown in Appendix A.2. model displays a bell-shaped sensiti… view at source ↗
Figure 4
Figure 4. Figure 4: Average Accuracy (↑) vs Throughput (↑) on 9 LM-Eval Tasks (ARC-c, ARC-e, BoolQ, HellaSwag, MMLU, OBQA, RTE, WinoGrande). B: Active Expert Budget sum of TopK Perturbed Frobenius Norm losses) across P L layers, subject to the budget constraint j = 1L kj = B and per-layer limits kmin ≤ kj ≤ kmax for all j. We initialize a population of Npop allocations (each satisfying the constraints) and then evolve this po… view at source ↗
Figure 5
Figure 5. Figure 5: F1 Score (↑) vs Throughput (↑) on Qasper Dataset in LongBench. B: Active Expert Budget Base Model Inter Pruning (12.5%, 25%, 50%) Intra Pruning (12.5%, 25%, 50%) LExI (Ours) 0 25 50 75 100 Average Accuracy 6.8k 7.0k 7.1k 7.3k 7.4k Throughput OLMoE-1B-7B (a) OlmoE-1B-7B-Instruct (B = 64, 72, 100) 0 25 50 75 100 Average Accuracy 3.7k 3.9k 4.0k 4.2k 4.4k Throughput Qwen1.5-MoE (b) Qwen1.5-MoE-A2.7B (B = 48, 5… view at source ↗
Figure 6
Figure 6. Figure 6: Passkey Retrieval Average (↑) vs Throughput (↑) Comparison. B: Active Expert Budget achieves a score of 35.5 at ∼ 4.1k throughput, outperforming inter pruning (F1 34 at ∼ 3.9k) and intra pruning (F1 30 at ∼ 3.75k) with a +0.5–5.5 F1 gain and +5.1%–9.3% higher throughput. Passkey Retrieval Task [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Perplexity (↓) vs Throughput (↑) on C4, PTB & WikiText(WT) B: Active Expert Budget Base Model Inter Pruning (12.5%, 25%, 50%) Intra Pruning (12.5%, 25%, 50%) LExI (Ours) 868 1048122814081587 Score 14 15 16 17 Throughput MME Perception (a) DeepSeekVL2-Tiny (B = 44, 48, 55) 182 231 280 329 378 Score 14 15 16 17 Throughput MME Reasoning (b) DeepSeekVL2-Tiny (B = 44, 48, 55) 14 21 28 35 42 Score 14 15 16 17 Th… view at source ↗
Figure 8
Figure 8. Figure 8: DeepSeekVL2-Tiny: Average Accuracy (↑) vs Throughput (↑) on MME, MMMU, Sci￾enceQA Tasks in VLMEvalKit. B: Active Expert Budget inference. This means that while our approach can significantly speed up inference by reducing the number of expert computations, it does not reduce model size. As a result, it is less effective in memory-constrained deployment scenarios. Nevertheless, our method can be effectively… view at source ↗
Figure 9
Figure 9. Figure 9: illustrates the topk sensitivity heatmaps for MiniCPM-MoE and DeepSeekV2 Lite Chat model based on Algorithm 1. 0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536373839 Layer Index 1 TopK 2 MiniCPM-MoE-8x2B 0.0 0.5 1.0 Loss 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Layer Index 1 2 3 4 5 6 Topk DeepSeek-V2-Lite 0.00 0.25 0.50 0.75 Loss [PITH_FULL_IMAGE:figu… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference

    cs.LG 2026-04 unverdicted novelty 6.0

    Alloc-MoE allocates a fixed expert activation budget using layer-level dynamic programming based on sensitivity and token-level score-based redistribution, delivering 1.15x prefill and 1.34x decode speedups on DeepSee...

  2. Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference

    cs.DC 2025-10 conditional novelty 6.0

    Comprehensive profiling of expert selection in frontier MoE models reveals temporal and spatial patterns that enable 6.6x speedup on wafer-scale GPUs and 1.25x on existing systems via targeted optimizations.

Reference graph

Works this paper leans on

20 extracted references · 1 canonical work pages · cited by 2 Pith papers

  1. [5]

    Documenting large webtext corpora: A case study on the colossal clean crawled corpus

    Jesse Dodge, Maarten Sap, Ana Marasovi ´c, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758,

    Pith/arXiv arXiv

  2. [6]

    Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin

    URL https://zenodo.org/records/12608602. Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin. Dynamic mixture of ex- perts: An auto-tuning approach for efficient transformer models. arXiv preprint arXiv:2405.14297,

    arXiv

  3. [7]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    10 Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395,

    Pith/arXiv arXiv

  4. [8]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088,

    Pith/arXiv arXiv

  5. [11]

    Olmoe: Open mixture-of-experts language models

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060,

    Pith/arXiv arXiv

  6. [12]

    Yarn: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071,

    Pith/arXiv arXiv

  7. [13]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641,

    Pith/arXiv arXiv 1907

  8. [14]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695,

    Pith/arXiv arXiv

  9. [15]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman

    URL https://qwenlm.github.io/blog/qwen-moe/. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium...

    work page 2018

  10. [16]

    doi: 10.18653/v1/W18-5446

    Association for Computational Linguistics. doi: 10.18653/v1/W18-5446. URL https://aclanthology.org/W18-5446. Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:24...

    Pith/arXiv arXiv

  11. [17]

    Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router

    11 Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router. arXiv preprint arXiv:2410.12013,

    Pith/arXiv arXiv

  12. [18]

    Moe-i2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition

    Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-i2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. arXiv preprint arXiv:2411.01016,

    Pith/arXiv arXiv

  13. [19]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549,

    Pith/arXiv arXiv

  14. [20]

    Learning n: m fine-grained structured sparse neural networks from scratch

    Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: m fine-grained structured sparse neural networks from scratch. arXiv preprint arXiv:2102.04010,

    Pith/arXiv arXiv

  15. [2018]

    A dataset of information-seeking questions and answers anchored in research papers

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011,

    Pith/arXiv arXiv

  16. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457,

    Pith/arXiv arXiv

  17. [2021]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146,

    Pith/arXiv arXiv

  18. [2022]

    Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models

    Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models. arXiv preprint arXiv:2402.14800,

    Pith/arXiv arXiv

  19. [2023]

    Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model. arXiv preprint arXiv:2405.04434, 2024a. Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, a...

    Pith/arXiv arXiv

  20. [2024]

    doi: 10.18653/v1/ 2024.acl-long.172

    Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long.172. URL https://aclanthology.org/2024.acl-long.172. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.