Pith. sign in

REVIEW 3 major objections 5 minor 31 references

R&Q claims that inference-time MoE load imbalance can be cut by up to 1.4x — without retraining or touching the router — by replicating the busiest expert as a quantized copy and quantizing a low-importance one.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:27 UTC pith:HUR5XFER

load-bearing objection The diagnostic observations are worth a look, but the claimed method is missing its central mechanism: with the router unchanged, the replicated expert receives no tokens and LIS cannot drop. the 3 major comments →

arxiv 2602.19938 v2 pith:HUR5XFER submitted 2026-02-23 cs.LG

A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs

Zijie Liu , Jie Peng , Jinhao Duan , Zirui Liu , Kaixiong Zhou , Mingfu Liang , Luke Simon , Xi Liu
show 2 more authors
Zhaozhuo Xu Tianlong Chen
This is my paper
classification cs.LG
keywords mixture-of-expertsload balancingexpert replicationquantizationinference efficiencyload imbalance scorelarge language modelstraining-free adaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Pre-trained sparse mixture-of-experts models route tokens unevenly: a small set of experts absorbs most of the load, and the skew grows with batch size. The paper proposes Replicate-and-Quantize (R&Q), a training-free, plug-and-play fix: in each MoE layer, add a quantized copy of the heaviest-loaded expert and aggressively quantize the least important expert, so memory stays roughly constant. To measure the effect, it defines the Load Imbalance Score (LIS), the ratio of the busiest expert's token count to the fair-share count; across Switch Transformer, LLaMA-MoE, DeepSeek-MoE, and DeepSeek-V2 Lite, R&Q lowers LIS by up to about 1.4x while keeping accuracy within ±0.6%. The core bet is that a busy expert need not be an important expert, which lets the method relieve the bottleneck without protecting the wrong parameters.

Core claim

R&Q identifies, per MoE layer, two experts: the heavy-hitter, found by counting tokens in a small calibration pass, and the least important expert, found by a pruning-based score that averages pruned weight-activation magnitudes. It appends a quantized replica of the heavy-hitter and quantizes the least important expert, leaving the router and all other weights untouched. The paper reports that, under this operation, LIS drops consistently — for example from 4.35 to 3.29 on PIQA for DeepSeek-V2 Lite, and from 1.97 to 1.39 on GSM8K for the 8-expert Switch Transformer — with per-task accuracy changes between -1.2 and +3.0 percentage points.

What carries the argument

The Load Imbalance Score (LIS), defined per layer as m·max_j n_{i,j}/(n k), converts token counts into a scalar skew measure: 1 is perfectly balanced, larger is more skewed. The operational pair is replication plus selective quantization — a quantized copy of the heavy-hitter is inserted to absorb extra tokens, while a pruning-scored low-importance expert is quantized to offset the memory cost. The router is not modified; the claim is that the replica itself changes the effective load distribution.

Load-bearing premise

R&Q assumes that adding a quantized replica of the heavy-hitter changes which tokens go where — but the paper leaves the router untouched and never says how tokens are split to the replica; if the router keeps sending every heavy-hitter token to the original expert, the reported LIS reduction cannot happen.

What would settle it

On a fixed batch, trace per-expert token counts before and after applying R&Q with the router frozen; if the quantized replica receives zero tokens while the original heavy-hitter keeps all of its tokens, then the measured LIS cannot have moved, contradicting Table 4.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If R&Q holds, serving engineers can apply it to existing MoE checkpoints without retraining or custom runtimes, and get more even expert utilization and lower latency at large batch sizes.
  • Because the paper shows imbalance worsens with batch size, the method's benefit is largest in exactly the high-throughput serving regime where tokens are batched heavily.
  • The decoupling of load from importance means load balancing and accuracy preservation are not in conflict at inference time: the busy expert is replicated (not sacrificed) and the low-importance expert absorbs the compression cost.
  • The method is architecture-agnostic, applying to top-1, top-2, and shared-expert routing schemes without changing the router, so it plugs into several deployed MoE families.
  • LIS gives a cheap per-layer diagnostic that operators could monitor online to detect routing skew as inputs drift.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A real system must define how the unchanged router dispatches to the replica — the paper does not specify the split rule. A natural test is to route heavy-hitter tokens alternately (or by score threshold) between the original and the replica and verify that LIS drops as reported.
  • The calibration set can likely be replaced by a sliding window of recent tokens, enabling fully online hot-swapping of replicas as the input distribution drifts; the paper's streaming experiment is a step in that direction but still uses pre-identified quantized experts.
  • Because LIS only tracks the maximum load, R&Q could shift the hotspot rather than flatten the distribution. A stronger check, not in the paper, is to measure the full load profile (e.g., top-3 load or entropy) before and after replication.
  • The small accuracy gains on some tasks (e.g., +2–3% on MMLU) hint that quantizing a low-importance expert acts as structured noise that recalibrates routing; an ablation that quantizes a random expert instead would test whether the gain depends on the importance criterion.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript studies load imbalance in sparse Mixture-of-Experts (SMoE) LLMs at inference time. It defines a Load-Imbalance Score (LIS) as m·max_j n_{i,j}/(nk) and uses routing traces to argue that imbalance persists and grows with batch size, and that selection frequency is not a reliable proxy for expert importance. The proposed Replicate-and-Quantize (R&Q) framework identifies a heavy-hitter expert per layer (Algorithm 1), identifies a least-important expert via a Wanda-style score (Algorithm 2), appends a quantized replica of the heavy-hitter and quantizes the least-important expert (Algorithm 3), while leaving the router unchanged. Experiments on Switch Transformer, LLaMA-MoE, DeepSeek-MoE, and DeepSeek-V2 Lite report LIS reductions of up to about 1.4×, with accuracy claimed to be maintained within ±0.6%, plus streaming and ablation studies.

Significance. The diagnostic parts of the paper are potentially useful: LIS is a simple, interpretable statistic computed from routing traces, and the observation that heavily routed experts need not be the most important could inform post-hoc optimization. If R&Q worked as described, it would be a practical training-free intervention for deployment. However, the central result is not supported as written: the paper never specifies how an unchanged router can dispatch tokens to the added replica, so the reported LIS reductions are not derivable from Algorithm 3. Additionally, the accuracy claim in the abstract is contradicted by the paper's own Table 4. These are load-bearing problems, not presentation issues.

major comments (3)
  1. [§2.5, Algorithm 3; Definition 1] Algorithm 3 appends quant(layer[re]) to each MoE layer and quantizes layer[qe], while the text states that 'this replication does not require any modification to the model's routing mechanism.' Definition 1 defines LIS using n_{i,j}, the number of tokens routed to expert j. With an unchanged router, token-to-expert assignments are exactly the raw model's, the replica is never selected, and LIS cannot decrease. Yet Table 4 reports reductions such as 1.9709→1.3937 (Switch Transformer GSM8K) and 4.3504→3.2925 (DeepSeek V2 Lite PIQA). The paper never defines a post-R&Q LIS or a dispatch rule that splits the heavy-hitter's tokens between original and replica. Without such a mechanism, the reported reductions are not derivable from the described algorithm. If serving-level token splitting is intended, it must be specified and the LIS recomputation formalized.
  2. [Abstract; Section 1; Table 4] The abstract and Section 1 claim that accuracy is maintained within ±0.6% across all tasks, and Section 3.2 states that R&Q 'maintains or slightly improves' accuracy. Table 4 shows deviations far outside this bound: Switch Transformer (8 experts) MMLU +2.2, DeepSeek V2 Lite MMLU +3.0, and DeepSeek V2 Lite GSM8K -1.2. These entries are directly inconsistent with the headline claim. The bound must be corrected, or the table entries explained/recomputed; as written, the 'near-lossless' claim is contradicted by the paper's own data.
  3. [§2.4, Algorithm 2] The importance-score procedure is internally inconsistent. Section 2.4 says Wanda scores are sorted in ascending order and the bottom-s fraction is pruned, with 'lower means imply higher importance.' Algorithm 2 sorts scores in descending order, takes sorted_idx[1:int(Cin×s)], and sets IE_l = argmax_e IS[l][e]. These two descriptions select different experts, and the argmax conflicts with the stated interpretation of the score. Because Algorithm 2 determines which expert is quantized, the selective-quantization step and the validation in Table 2 are non-reproducible as stated.
minor comments (5)
  1. [Figure 5] The caption refers to a 'normalized gap value (Definition 1)', but Definition 1 defines LIS, not the expert activation gap described in the text. Define the gap metric explicitly.
  2. [Algorithm 1] The input 's' (sparsity factor) is declared but never used in the algorithm. Either remove it or explain its role.
  3. [§3.1 vs. Figure 3] The setup states that LIS is computed with batch size 1 and max_new_tokens=1, but Figure 3 reports LIS for batch size 32. Clarify how the batch-32 LIS is computed and whether it uses the same generation settings.
  4. [Table 6] The row 'Replicate Heavy-Hitter Experts + Quantize' appears to describe the same combination as 'R&Q (Ours)'. Clarify the difference, e.g., whether the less-important expert is also quantized in the former.
  5. [References] The citation for Dhasade et al. (2025) is titled 'Practical Federated Learning Without a Server', which does not match the 'MoEShard' work cited in Section 1. Verify and correct this reference.

Circularity Check

1 steps flagged

R&Q's LIS reductions rely on an unstated token-splitting convention: with the router unchanged, Definition 1's counts are invariant, so the reported drops are imposed by construction.

specific steps
  1. self definitional [Definition 1; Section 2.5 (Algorithm 3); Table 4]
    "Definition 1: 'For block i, let ni,j be the number of tokens routed to expert j... li = m·max j∈[m] ni,j / nk.' Algorithm 3: 'layer←layer+quant(layer[re])... layer[qe]←quant(layer[qe])' and 'Importantly, this replication does not require any modification to the model’s routing mechanism.' Yet Table 4 reports 'LIS on GSM8K drops from 1.9709 to 1.3937.'"

    LIS is defined solely on router-produced token counts. If the router is untouched, adding quant(layer[re]) does not alter any ni,j (the replica receives 0 tokens), so LIS must remain identical to the raw value. The reductions in Table 4 can only be obtained by an unstated rule that splits the heavy-hitter's tokens between the original and the replica (or by changing the router). That split is exactly the claimed imbalance reduction; hence the 'prediction' is a counting convention imposed on Definition 1, not a consequence of Algorithm 3 as written.

full rationale

The raw-model diagnostics (LIS on unmodified routing traces, Wanda-based importance scores) are external and not circular. The circular issue is confined to the central effectiveness claim: the paper never defines LIS for the replicated model, and Algorithm 3 contains no dispatch mechanism. Given the explicit 'no modification to the model’s routing mechanism' statement, Definition 1's n_{i,j} are unchanged by the algorithm; the large LIS drops reported for R&Q are therefore not derivable from the described method. The only coherent way to reproduce them is to redefine post-R&Q token counts so that the replica absorbs a share of the heavy-hitter's tokens, which makes the measured reduction equivalent to the assumption embedded in the counting. This is a definitional gap in the evaluation metric rather than a self-citation issue. If the authors intended a system-level scheduler to split batches, that mechanism is absent and the reported metric is not yet defined; under the paper's own equations, the result reduces by construction. Accuracy inconsistencies (e.g., DeepSeek V2 Lite MMLU +3.0% vs the claimed ±0.6%) are correctness concerns, not circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The paper's headline contribution rests on several unstated choices: the Wanda pruning fraction s, the 10% calibration subset, one replica per layer, and one quantized low-importance expert per layer. The largest conceptual assumption is that an unchanged router can send tokens to a replica; this is the unmeasured 'graviton' of the paper. The Wanda-based importance proxy and the single-token LIS protocol are domain assumptions that lack validation in deployment-like settings.

free parameters (5)
  • Wanda sparsity factor s
    Algorithm 2 prunes the bottom s fraction of weights to score expertise; s is never given, so the importance ranking depends on an unspecified hand-chosen constant.
  • Calibration fraction = 10%
    Heavy-hitter and importance estimates use 10% of each benchmark (Sections 2.3, 3.1); no sensitivity analysis is provided, and calibration and evaluation sets overlap.
  • Replicated experts per layer = 1
    Only the single most-loaded expert per layer is replicated; no search or ablations over replication count, although this choice directly controls the LIS reduction.
  • Quantized low-importance experts per layer = 1
    Only one expert per layer is aggressively quantized; the effect on accuracy and memory depends on this choice, which is not varied.
  • Quantization precision = fp16 (Switch), 8-bit (LLaMA-MoE, DeepSeek)
    Precision is chosen per model based on its default dtype; no ablation on the precision/cost trade-off is reported.
axioms (5)
  • domain assumption Wanda score provides a valid ordering of expert importance for selecting which expert to quantize.
    Section 2.4 treats pruning robustness as importance; validation is only one-expert-per-layer removal, and Table 2 shows inconsistent gains.
  • domain assumption A 10% calibration subset is representative of the inference workload.
    Used in Sections 2.3 and 3.2; no distribution-shift test is performed, and calibration and evaluation come from the same benchmarks.
  • ad hoc to paper Replicated experts can receive tokens without modifying the router.
    Algorithm 3 copies quantized heavy-hitter weights, but the router is explicitly unchanged; the paper never specifies the dispatch mechanism that sends tokens to replicas, so post-R&Q token counts are undefined.
  • domain assumption Quantizing replicas and low-importance experts keeps the model within the original memory budget.
    Memory equivalence is asserted in Sections 2.5 and 3, but no memory measurements or parameter-count arithmetic are provided.
  • domain assumption LIS computed with max_new_tokens=1 measures inference-time imbalance relevant to full-generation deployment.
    Section 3.1 restricts generation to one token for LIS but uses unconstrained generation for accuracy; the relationship between these settings is unexamined.
invented entities (1)
  • Quantized replica expert no independent evidence
    purpose: Absorb tokens from the heavy-hitter expert and increase parallel capacity without retraining or router change.
    The replica is a copy of an existing expert at lower precision; because the router is unchanged, it is not clear how any token is ever dispatched to it. No system trace, no device-level measurement, and no token-splitting rule is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 22860 in / 14336 out tokens · 128190 ms · 2026-08-02T21:27:28.025660+00:00 · methodology

0 comments
read the original abstract

Sparse Mixture-of-Experts (SMoE) architectures are increasingly used to scale large language models efficiently, delivering strong accuracy under fixed compute budgets. However, SMoE models often suffer from severe load imbalance across experts, where a small subset of experts receives most tokens while others are underutilized. Prior work has focused mainly on training-time solutions such as routing regularization or auxiliary losses, leaving inference-time behavior, which is critical for deployment, less explored. We present a systematic analysis of expert routing during inference and identify three findings: (i) load imbalance persists and worsens with larger batch sizes, (ii) selection frequency does not reliably reflect expert importance, and (iii) overall expert workload and importance can be estimated using a small calibration set. These insights motivate inference-time mechanisms that rebalance workloads without retraining or router modification. We propose Replicate-and-Quantize (R&Q), a training-free and near-lossless framework for dynamic workload rebalancing. In each layer, heavy-hitter experts are replicated to increase parallel capacity, while less critical experts and replicas are quantized to remain within the original memory budget. We also introduce a Load-Imbalance Score (LIS) to measure routing skew by comparing heavy-hitter load to an equal allocation baseline. Experiments across representative SMoE models and benchmarks show up to 1.4x reduction in imbalance with accuracy maintained within +/-0.6%, enabling more predictable and efficient inference.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 27 linked inside Pith

  1. [1]

    Deep rewiring: Training very sparse deep networks.arXiv preprint arXiv:1711.05136,

    Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks.arXiv preprint arXiv:1711.05136,

    Pith/arXiv arXiv

  2. [6]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635,

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635,

    Pith/arXiv arXiv

  3. [9]

    Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee

    Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Hsien-Hsin S. Lee, Anjali Sridhar, Shruti Bhosale, Carole-Jean Wu, and Benjamin Lee. Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference, 2023.https://arxiv.org/abs/2303.06182. Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey.arXi...

    Pith/arXiv arXiv 2023

  4. [10]

    Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference, 2024.https://arxiv.org/abs/2308

    Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-expert inference, 2024.https://arxiv.org/abs/2308. 12066. Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXi...

    Pith/arXiv arXiv 2024

  5. [13]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.https://arxiv.org/abs/ 2001.08361. Young Jin Kim, Raffy Fahim, and Hany Hassan Awadalla. Mixture of quantized experts (moqe): Complementary effect of low-bit quantizati...

    Pith/arXiv arXiv 2020

  6. [14]

    Raghuraman Krishnamoorthi

    https: //arxiv.org/abs/2308.15030. Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper.arXiv preprint arXiv:1806.08342,

    Pith/arXiv arXiv

  7. [15]

    Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020a

    D Lepikhin, H Lee, Y Xu, D Chen, O Firat, Y Huang, M Krikun, N Shazeer, and Z Gshard. Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020a. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant mod...

    Pith/arXiv arXiv 2006

  8. [16]

    19 Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W

    https://arxiv.org/abs/2109.07958. 19 Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models,

    Pith/arXiv arXiv

  9. [17]

    https://arxiv.org/abs/2410.10912. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmar...

    Pith/arXiv arXiv 2021

  10. [18]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher

    https://arxiv.org/abs/2305.11627. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models,

    Pith/arXiv arXiv

  11. [19]

    Pruning convolutional neural networks for resource efficient inference.arXiv preprint arXiv:1611.06440,

    Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference.arXiv preprint arXiv:1611.06440,

    Pith/arXiv arXiv

  12. [21]

    https://aclanthology.org/Q19-1016/

    doi: 10.1162/tacl_a_00266. https://aclanthology.org/Q19-1016/. Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts.Advances in Neural Information Processing Systems, 34:8583–8595,

    doi:10.1162/tacl_a_00266

  13. [23]

    Systematic characterization of llm quantization: A performance, energy, and quality perspective, 2025.https://arxiv.org/abs/2508.16712

    Tianyao Shi and Yi Ding. Systematic characterization of llm quantization: A performance, energy, and quality perspective, 2025.https://arxiv.org/abs/2508.16712. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024.https://arxiv.org/abs/2306.11695. LLaMA-MoE Team. Llama-moe: Buildin...

    Pith/arXiv arXiv 2025

  14. [24]

    Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li

    https://github.com/pjlab-sys4nlp/llama-moe. Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning.arXiv preprint arXiv:2310.03731,

    Pith/arXiv arXiv

  15. [25]

    Auxiliary-loss-free load balancing strategy for mixture-of-experts, 2024.https://arxiv.org/abs/2408.15664

    Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-free load balancing strategy for mixture-of-experts, 2024.https://arxiv.org/abs/2408.15664. Huanrui Yang, Lin Duan, Yiran Chen, and Hai Li. Bsq: Exploring bit-level sparsity for mixed-precision neural network quantization, 2021.https://arxiv.org/abs/2102.10462. 20 Yi Yang, Wen-ta...

    Pith/arXiv arXiv 2024

  16. [26]

    doi: 10.18653/v1/D15-1237.https://aclanthology.org/D15-1237/

    Association for Computational Linguistics. doi: 10.18653/v1/D15-1237.https://aclanthology.org/D15-1237/. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825,

    Pith/arXiv arXiv

  17. [29]

    Incremental network quantization: Towards lossless cnns with low-precision weights.arXiv preprint arXiv:1702.03044, 2017a

    Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights.arXiv preprint arXiv:1702.03044, 2017a. Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights, 2017b.https://arxiv.org/abs/17...

    Pith/arXiv arXiv

  18. [30]

    Exploring sparse moe in gans for text-conditioned image synthesis.arXiv preprint arXiv:2309.03904,

    Jiapeng Zhu, Ceyuan Yang, Kecheng Zheng, Yinghao Xu, Zifan Shi, and Yujun Shen. Exploring sparse moe in gans for text-conditioned image synthesis.arXiv preprint arXiv:2309.03904,

    Pith/arXiv arXiv

  19. [31]

    St-moe: Designing stable and transferable sparse expert models, 2022.https://arxiv.org/abs/2202.08906

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022.https://arxiv.org/abs/2202.08906. Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer wi...

    Pith/arXiv arXiv 2022

  20. [1991]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024a

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024a. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Sin...

    Pith/arXiv arXiv

  21. [2015]

    Grace-moe: Grouping and replication with locality-aware routing for efficient distributed moe inference, 2025.https://arxiv.org/abs/2509.25041

    Yu Han, Lehan Pan, Jie Peng, Ziyang Tao, Wuyang Zhang, and Yanyong Zhang. Grace-moe: Grouping and replication with locality-aware routing for efficient distributed moe inference, 2025.https://arxiv.org/abs/2509.25041. Shwai He, Weilin Cai, Jiayi Huang, and Ang Li. Capacity-aware inference: Mitigating the straggler effect in mixture of experts, 2025.https:...

    Pith/arXiv arXiv 2025

  22. [2016]

    Hellaswag: Can a machine really finish your sentence?, 2019.https://arxiv.org/abs/1905.07830

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019.https://arxiv.org/abs/1905.07830. Hao Zhang, Aining Jia, Weifeng Bu, Yushu Cai, Kai Sheng, Hao Chen, and Xin He. Flexq: Efficient post-training int6 quantization for llm serving via algorithm-system co-design, 2025.https://ar...

    Pith/arXiv arXiv 2019

  23. [2017]

    Piqa: Reasoning about physical commonsense in natural language, 2019.https://arxiv.org/abs/1911.11641

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.https://arxiv.org/abs/1911.11641. Tianlong Chen, Zhenyu Zhang, Ajay Jaiswal, Shiwei Liu, and Zhangyang Wang. Sparse moe as the new dropout: Scaling dense and self-slimmable transformers.arXiv preprint arXiv:2303.01610,

    Pith/arXiv arXiv 2019

  24. [2018]

    Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023.https://arxiv.org/abs/2210.17323

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023.https://arxiv.org/abs/2210.17323. Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts, 2022.https://arxiv.org/abs/2211.15841. Shan...

    Pith/arXiv arXiv 2023

  25. [2019]

    Load balancing mixture of experts with similarity preserving routers, 2025.https://arxiv.org/abs/2506.14038

    Nabil Omi, Siddhartha Sen, and Ali Farhadi. Load balancing mixture of experts with similarity preserving routers, 2025.https://arxiv.org/abs/2506.14038. Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization.arXiv preprint arXiv:1802.05668,

    arXiv 2025

  26. [2020]

    Sergey Zagoruyko and Nikos Komodakis

    doi: 10.1109/micro50266.2020.00071.http: //dx.doi.org/10.1109/MICRO50266.2020.00071. Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.arXiv preprint arXiv:1612.03928,

    arXiv 2020

  27. [2021]

    Winogrande: An adversarial winograd schema challenge at scale, 2019.https://arxiv.org/abs/1907.10641

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.https://arxiv.org/abs/1907.10641. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer...

    Pith/arXiv arXiv 2019

  28. [2022]

    https://arxiv.org/abs/2204.02311. Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifr...

    Pith/arXiv arXiv 2022

  29. [2023]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference, 2017.https://arxiv.org/abs/1712.05877

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference, 2017.https://arxiv.org/abs/1712.05877. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, a...

    Pith/arXiv arXiv 2017

  30. [2024]

    Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022.https://arxiv.org/abs/2208.07339

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022.https://arxiv.org/abs/2208.07339. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023.https://arxiv.org/abs/2305.14314. Akash Dhasade, Anne-Marie Kermarr...

    Pith/arXiv arXiv 2022

  31. [2025]

    Zachary Doucet, Rishi Sharma, Martijn de Vos, Rafael Pires, Anne-Marie Kermarrec, and Oana Balmau

    doi: 10.1145/3721146.3721938.http://dx.doi.org/10.1145/3721146.3721938. Zachary Doucet, Rishi Sharma, Martijn de Vos, Rafael Pires, Anne-Marie Kermarrec, and Oana Balmau. Harmoeny: Efficient multi-gpu inference of moe models, 2025.https://arxiv.org/abs/2506.12417. David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a com...

    arXiv 2025