Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Klotski claims that MoE inference under tight memory is bottlenecked by pipeline bubbles, not by expert-prediction accuracy, and that scheduling a group of batches around hot experts makes the pipeline almost bubble-free, raising…

desk verdict A plausible MoE offloading engine whose core multi-batch idea is inherited from FlexGen; the expert-aware additions are real but modest, and the evaluation as written does not support the headline 85x or 'almost bubble-free' claims. read the letter →

arxiv 2502.06888 v1 pith:7ZABSE4O submitted 2025-02-09 cs.LG cs.AI

Zhiyuan Fang , Yuegui Huang , Zicong Hong , Yufeng Lyu , Wuhui Chen , Yue Yu , Fan Yu , Zibin Zheng This is my paper · ORCID
classification cs.LGcs.AI
keywords MixtureofExpertsinferenceoffloadingpipelinebubblesmulti-batchschedulingexpertprefetchinghotthroughput-latencytrade-off
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Klotski is a system paper claiming that the biggest inefficiency in offloaded Mixture-of-Experts (MoE) inference is neither expert prefetching accuracy nor raw I/O volume, but the bubbles in the compute pipeline: the GPU waits while expert weights arrive, and each expert's computation is too short to hide the transfer. The proposed fix, called an expert-aware multi-batch pipeline, processes several batches together, prefetches only the gate and the few hot experts most tokens will use, and reorders expert computation so that the long-running hot experts overlap the slow I/O of cold experts. The authors argue that when the batch-group size $n$ is chosen to satisfy a set of timing inequalities, the pipeline becomes almost bubble-free. Their measurements on two large MoE models in two hardware environments show a better throughput-latency trade-off than five offloading baselines, with throughput up to 85.12x higher in the most favorable setting. If the claim holds, large MoE models become practical on a single consumer GPU.

What carries the argument

The load-bearing mechanism is the expert-aware multi-batch pipeline: multiple batches are fused into a batch group, the MoE layer is decomposed into gate and experts, and computations are scheduled by expert rather than by batch. The planner chooses the smallest integer $n$ such that the combined compute time of $n$ attention layers, and later gate and hot-expert layers, covers the transfer time of the gate, the $K$ hot experts, the remaining cold experts, and the next attention layer, which the paper writes as inequalities (4)--(7). A correlation-aware prefetcher estimates which experts are hot from a warm-up run and updates the estimate online; the adaptive tensor placement spreads weights across VRAM, DRAM, and disk. Together these carry the claim that the pipeline is almost bubble-free.

What would settle it

Run a serving workload whose gate outputs are forced to be uniformly random across all experts, or anti-correlated with the warm-up table, and compare Klotski against a baseline that prefetches the entire MoE layer; if the throughput advantage falls to near parity, the almost-bubble-free claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that hot-expert imbalance is not a problem to be solved by better prediction but a scheduling resource to be exploited. During MoE inference, a few experts receive most of the tokens, so across a batch group they have long total compute time; cold experts have the opposite profile, expensive to move and cheap to run. Klotski partitions the multi-batch computational graph by experts rather than by batches, prefetches the gate and the K predicted hot experts while attention runs, then computes hot experts first and cold experts behind them in transfer order. The authors state that this orchestration, together with a constraint-sensitive planner that picks the smallest batch-group size satisfying inequalities (4)--(7), yields an almost bubble-free pipeline, and they attribute the measured throughput gains to the elimination of inter-layer and intra-layer bubbles.

Load-bearing premise

The whole speedup rests on hot experts staying hot: the prefetcher learns who is hot from a warm-up run, and if real inputs route tokens to different experts, the prefetching loses its value; the paper reports only 58.89% hot-expert prediction accuracy and admits a worst case where all tokens pick cold experts would make the strategy ineffective.

Editorial extensions

If this is right

  • If Klotski is right, MoE inference under offloading is schedule-bound rather than I/O-bound, so further gains come from better ordering rather than only faster storage or bigger batches.
  • A single consumer GPU can serve MoE models that currently need server-class or multi-GPU memory.
  • The same timing-inequality planner can be reused for new hardware and new models by measuring per-layer compute and transfer times, without re-tuning.
  • Techniques that shrink expert payloads, such as low-bit quantization, become compounding wins because they lower the batch-group size $n$ required for a bubble-free pipeline and reduce KV-cache pressure.
  • The better throughput-latency frontier means that under a fixed latency budget, the engine can serve several times more requests than the baselines, not just push peak throughput.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper does not run: force the gate to activate experts uniformly at random, defeating the correlation table, and check whether the throughput advantage over whole-layer prefetching collapses; the paper's own worst-case analysis predicts it would.
  • Because the correlation table is learned from a general text corpus, a domain-shift benchmark such as code or structured data would quantify how much of the gain depends on the warm-up distribution matching the serving distribution.
  • The planner returns the smallest $n$ that satisfies the inequalities, yet the evaluation uses the largest feasible $n$ for top throughput; comparing the planner's $n$ with the empirically optimal $n$ across batch sizes would test whether the inequality model captures real overheads.
  • If the hot-expert assumption is the real bottleneck, an obvious extension is to make the prefetcher adaptive to distribution shift within a session by re-ranking hot experts from recent gate decisions rather than from a fixed table.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. Klotski proposes an MoE inference engine that groups multiple batches and schedules expert computations so that the compute time of hot experts overlaps the I/O of cold experts. The system includes a constraint-sensitive planner, adaptive tensor placement, a correlation-aware expert prefetcher, and an execution pipeline built on four CUDA streams. The paper evaluates Klotski on Mixtral-8x7B and Mixtral-8x22B in two hardware environments, reporting throughput improvements over existing offloading systems of up to 85.12x, and includes ablations, memory-usage measurements, prefetch-accuracy analysis, and a bubble-reduction study.

Significance. The core idea is appealing: exploiting the skewed distribution of expert activation to overlap I/O with the compute of hot experts is a plausible and potentially valuable technique for single-GPU MoE inference. If the quantitative results are reproducible, Klotski would be a strong addition to the offloading literature. The paper also contributes a clean decomposition of inter-layer and intra-layer bubbles and an ablation study that isolates the effect of each design choice. However, the evaluation as presented has load-bearing weaknesses: the main throughput comparison uses a hand-picked batch-group size rather than the planner's own output, the FlexGen baseline is forced to use the same size without tuning, no variance information is reported, and the prefetcher is tested only in a single data domain. These issues currently prevent verification of the headline claims.

major comments (4)
  1. [Section 9.2] The main throughput results use n = 15, taken as the maximum of the parameter sweep in Figure 14, instead of the n produced by the constraint-sensitive planner described in Section 7. For Mixtral-8x22B in Environment 1, n = 10 is used because the computed n causes OOM. This means the planner's central prediction is not validated, and the reported advantage over FlexGen is tied to an n that may not be selected by Klotski's own algorithm. Furthermore, FlexGen is 'set to use the same n as us' without tuning, which can arbitrarily disadvantage it. Please re-run the end-to-end comparison with the planner-selected n for Klotski and with FlexGen tuned on its own, and report throughput as a function of n for all systems.
  2. [Section 9.6] The claim of an almost bubble-free pipeline depends on the prefetched hot experts accounting for a large fraction of the compute in the batch group. Figure 13 shows only 58.89% average accuracy for identifying the true hot experts, and the 100% 'participate in computation' line merely indicates that each prefetched expert was selected by at least one token in the batch. An expert selected by one token in a batch of 64 provides negligible overlap. The authors should report the distribution of token counts handled by prefetched hot experts, and evaluate the pipeline under conditions of low hot-expert accuracy to show that the throughput gain degrades gracefully.
  3. [Section 7 and Section 6.2] The worst-case analysis in Section 7 states that when all tokens select cold experts the prefetching strategy is ineffective, and dismisses this as 'intuitively' improbable. Yet Section 6.2 acknowledges that MoE is data-sensitive and hot experts may vary with different inputs. The evaluation warms up the correlation table on wikitext-2 and tests on wikitext-103, which are both Wikipedia-style corpora, so the data-dependent behavior is never stress-tested. Add experiments on a clearly different distribution (e.g., code or legal text) and report prefetch accuracy and throughput on that distribution.
  4. [Section 9.1] All throughput numbers are reported as averages 'from multiple trials' without error bars or the number of trials. Given the headline factor of 85.12x, the absence of variance information makes it impossible to assess whether differences between systems are significant. Please include standard deviations and, where relevant, statistical tests across repeated runs.
minor comments (5)
  1. [Section 9.2] The claim that 'even if we increase the batch size to 128, Klotski can still achieve a 15% throughput improvement over FlexGen' is not backed by any plotted data point, since Figure 10 goes only to batch size 64. Please add the data or remove the claim.
  2. [Section 8] The phrase 'over 3k LOC of Python' suggests a substantial implementation, but no code or repository link is provided. If the system is available, please include the URL; otherwise state that the implementation is not released.
  3. [Figure 13] The two accuracy curves would benefit from a precise definition in the caption, since 'really hot' and 'participate in comp' are ambiguous.
  4. [Section 7] In the inequality group, the set Q and its length len(Q) are used in inequality (7) but defined only after the equations; move the definition before the inequality.
  5. [General] There are minor typographical issues, such as inconsistent use of 'IO-compute' versus 'I/O-compute' across the abstract and Section 7; a careful proofread would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the bubble-free claim is a planner constraint, and the prefetcher accuracy is empirically reported rather than derived from the throughput claim.

full rationale

The paper's load-bearing claims are experimentally measured system results, not first-principles derivations. The planner's inequalities (4)-(7) are scheduling constraints: if the measured per-layer compute and I/O times satisfy them, the pipeline has no bubbles by the schedule's own dependency definition. That is a design condition, not a circular prediction. The correlation-aware expert prefetcher is an empirical heuristic: its table is built in a warm-up run on wikitext-2, updated online, and its quality is reported as measured 58.89% hot-expert accuracy and 42.24% single-sequence accuracy, not as a consequence of the throughput claim. The use of n=15 taken from the same sweep in Figure 14 is a configuration-selection concern, not a logical reduction. There are no self-citations carrying the argument, no imported uniqueness theorem, and no ansatz disguised as an external result. The failure modes the paper admits (cold-expert worst case, data sensitivity) are limitations and correctness risks, not circularity. Therefore the derivation chain is self-contained and no circular step is exhibited.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

Klotski introduces no new physical entities. The engineering parameters are n, correlation path length l, quantization group sizes, and warm-up settings; the hot-expert stability and the transferability of the correlation table are domain assumptions that the central claim depends on. The 'worst case is unlikely' remark is an explicit ad hoc assumption in Section 7.

free parameters (4)
  • Batch-group size n (evaluation override) = 15 (Mixtral-8x7B Env1), 10 (Mixtral-8x22B Env1)
    Chosen post hoc from Figure 14 to maximize reported throughput instead of using the planner-computed default (Section 9.2).
  • Correlation path length l = 1
    Small value chosen deliberately; increasing l would add table complexity (Section 8).
  • Quantization group sizes = 4-bit weights, group size 64, zero-scale group size 128
    Preset values in Section 7 Compression section; affect I/O reduction and reported throughput for Klotski(q).
  • Warm-up inference batch size and sequence length for correlation table = batch 8, sequence length 512
    Small batches chosen deliberately to keep correlation statistics meaningful (Section 8).
assumptions (5)
  • domain assumption A small set of hot experts consistently handles the majority of tokens and remains predictable during inference
    The prefetcher and hot-first ordering depend on this; the paper provides empirical heatmaps (Figure 5) but the phenomenon is data-dependent (Section 6.2).
  • domain assumption The measured per-layer computation and I/O times are stable and additive, so inequalities (4)-(7) guarantee bubble reduction
    Planning relies on cached measurements and a linear time model; no formal proof or sensitivity analysis is provided (Section 7).
  • ad hoc to paper The worst case in which all tokens select cold experts is very unlikely
    Stated in Section 7: 'intuitively, the probability of encountering such a worst-case scenario is very low.' No quantitative bound is given, and the planner offers no fallback.
  • domain assumption The expert correlation table built on wikitext-2 transfers to the test distribution (wikitext-103)
    Warm-up data and evaluation data differ; the paper notes data sensitivity but does not evaluate across datasets (Sections 6.2, 8, 9.1).
  • domain assumption Top-k gate activation patterns remain similar when the batch-group size n is increased
    The motivation observes that more batches activate more experts, but the planner assumes K hot experts still dominate; sensitivity is not measured (Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline." pith.science (2026). https://pith.science/paper/7ZABSE4O

@misc{pith2026250206888,
  author       = {Pith},
  title        = {Pith review of: Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZABSE4O}},
  note         = {Machine review of arXiv:2502.06888}
}
read the original abstract

Mixture of Experts (MoE), with its distinctive sparse structure, enables the scaling of language models up to trillions of parameters without significantly increasing computational costs. However, the substantial parameter size presents a challenge for inference, as the expansion in GPU memory cannot keep pace with the growth in parameters. Although offloading techniques utilise memory from the CPU and disk and parallelise the I/O and computation for efficiency, the computation for each expert in MoE models is often less than the I/O, resulting in numerous bubbles in the pipeline. Therefore, we propose Klotski, an efficient MoE inference engine that significantly reduces pipeline bubbles through a novel expert-aware multi-batch pipeline paradigm. The proposed paradigm uses batch processing to extend the computation time of the current layer to overlap with the loading time of the next layer. Although this idea has been effectively applied to dense models, more batches may activate more experts in the MoE, leading to longer loading times and more bubbles. Thus, unlike traditional approaches, we balance computation and I/O time and minimise bubbles by orchestrating their inference orders based on their heterogeneous computation and I/O requirements and activation patterns under different batch numbers. Moreover, to adapt to different hardware environments and models, we design a constraint-sensitive I/O-compute planner and a correlation-aware expert prefetcher for a schedule that minimises pipeline bubbles. Experimental results demonstrate that Klotski achieves a superior throughput-latency trade-off compared to state-of-the-art techniques, with throughput improvements of up to 85.12x.

Figures

Figures reproduced from arXiv: 2502.06888 by the authors.

Figure 1
Figure 1. Comparison of three kinds of pipeline. We use multiple computations of the current layer to overlap the I/O of the next layer to reduce inter-layer bubbles and adjust the experts’ computation order to reduce intra-layer bubbles. experts are activated, resulting in more intra-layer bubbles in the pipeline, due to the long I/O time for these experts. To tackle this challenge, we propose an expert-aware multi￾batch pip… view at source ↗
Figure 2
Figure 2. Architecture and inference process of MoE models. Some recent literature [15, 23, 30, 45] has focused on the optimization of MoE. DeepSpeed-MoE [30] introduces a spe￾cialized MoE architecture called Pyramid-Residual MoE and employs staged knowledge distillation to obtain the Mixture￾of-Students. This approach not only accelerates MoE train￾ing but also reduces inference latency and cost. Lina [23], an extension of D… view at source ↗
Figure 3
Figure 3. Illustration of offloading an LLM in a multi-level storage system. Only a few layers of parameters can be placed in VRAM, and the rest are placed in DRAM and disk. param. refers to parameters. solving linear programming problems within the computa￾tional graph. HeteGen [41] leverages heterogeneous parallel computing between CPU and GPU, reducing the need for pa￾rameter I/O and achieving better resource allocation. S… view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: The expert heatmaps in Mixtral-8×7B, decoder part of switch-base-8 and switch-base-16. The darker the color, the higher the frequency of selection. other layers. Therefore, while performing multiple compu￾tations of the attention layer, we prefetch the gate and K exper…
Figure 6
Figure 6. Figure 6: System overview of Klotski. Comp. refers to Compute, Corr. refers to Correlation, and Infer. refers to Inference. The blue, purple, and yellow graphics represent the expert, gate, and attention layers, respectively. The dotted line in the pipeline indicates that the co…
Figure 7
Figure 7. Figure 7: Expert-aware multi-batch computational graph. Algorithm 1: Schedule Algorithm of the Paradigm. Init :Generate length 𝑙, number of layers 𝑛_𝑙𝑎𝑦𝑒𝑟, number of batches 𝑛_𝑏𝑎𝑡𝑐ℎ, hidden state ℎ, KV cache 𝑐. The indices 𝑖, 𝑗, 𝑘 indicate that the it is processing the 𝑖-th toke…
Figure 8
Figure 8. Figure 8: An example of the expert correlation table. Each expert layer has four experts. The gate selects the top-1 expert. The correlation path length 𝑙 is 1. memory. Specifically, we dynamically manage tensors for a fixed number of layers 𝐿 within the limited CPU memory. As t…
Figure 9
Figure 9. Figure 9: Multi-batch pipeline of Klotski. of the next attention layer’s computations. These arrows col￾lectively suggest that the corresponding tensor transmission must be completed before these points to ensure that I/O and computation are fully overlapped. We list the four ke…
Figure 10
Figure 10. Figure 10: Throughput comparison between Klotski and baselines in different scenarios. (q) means that quantization and dequantization are used. 0 5 10 15 20 Throughput (token/s) 2 9 2 10 2 11 2 12 Latency (s) Mixtral-8x7B in Env1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Throughput (token…
Figure 11
Figure 11. Figure 11 [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: GPU memory usage over the prefill. Each step represents one computation of a layer or an expert, i.e. a block in the computation graph as shown in [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 15
Figure 15. Figure 15: Compare the actual pipelines of different methods for performing inference in Environment 1 with Mixtral￾8×7B. Simple overlap means prefetching the next layer when executing the current layer. Comp. means computation. 9.8 Bubble Reduction As shown in [PITH_FULL_IMAGE…
Figure 14
Figure 14. Figure 14: The impacts of 𝑛 and batch size on throughput. 9.7 Impacts of 𝑛 and Batch Size We present detailed end-to-end throughput data, as shown in [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    PagedWeight pages quantized MoE expert weights on and off the GPU at runtime, releasing memory to the KV cache while using sensitivity, routing, and prompt signals to choose which experts to shrink.

  2. SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models

    cs.AI 2026-06 conditional novelty 5.0 of 10

    SpecPrefetch trains lightweight adapters to prefetch next-layer experts during offloaded MoE inference while keeping the native router authoritative, improving decoding throughput by up to ~20% on a mobile device.

Reference graph

Works this paper leans on

51 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al . 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al . 2022. Deepspeed-inference: enabling ef- ficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analy...

    work page 2022

  3. [3]

    Hicham Badri and Appu Shaji. 2023. Half-Quadratic Quantization of Large Machine Learning Models. https://mobiusml.github.io/hqq_ blog/

    work page 2023

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  5. [5]

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of- experts language models. arXiv preprint arXiv:2401.06066 (2024)

    arXiv 2024

  6. [6]

    DeepSeek-AI. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434 [cs.CL]

    arXiv 2024

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

  8. [8]

    Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai Li, Yiran Chen, et al . 2023. SiDA: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models. arXiv preprint arXiv:2310.18859 (2023)

    work page Pith review arXiv 2023

Show all 51 references
  1. [9]

    Artyom Eliseev and Denis Mazur. 2023. Fast inference of mixture- of-experts language models with offloading. arXiv preprint arXiv:2312.17238 (2023)

    2023 arXiv

  2. [10]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch trans- formers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39

    2022

  3. [11]

    Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive lan- guage models can be accurately pruned in one-shot. In International Conference on Machine Learning. PMLR, 10323–10337

    2023

  4. [12]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 (2022)

    2022 arXiv

  5. [13]

    Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan

  6. [14]

    Liwei Guo, Wonkyo Choe, and Felix Xiaozhu Lin. 2023. Sti: Turbocharge nlp inference at the edge via elastic pipelining. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 791–803

    2023

  7. [15]

    Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. 2022. Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Program...

    2022

  8. [16]

    Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, et al. 2024. DeepSpeed- FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. arXi...

    2024 arXiv

  9. [17]

    Chien-Chin Huang, Gu Jin, and Jinyang Li. 2020. Swapadvisor: Push- ing deep learning beyond the gpu memory limit via smart swap- ping. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. 1341–1355

    2020

  10. [18]

    Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. 2024. Pre-gated moe: An algorithm- system co-design for fast and scalable mixture-of-expert inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)...

    2024

  11. [19]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    2024 arXiv

  12. [20]

    Keisuke Kamahori, Yile Gu, Kan Zhu, and Baris Kasikci. 2024. Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models. arXiv preprint arXiv:2402.07033 (2024)

    2024 arXiv

  13. [21]

    Young Jin Kim, Raffy Fahim, and Hany Hassan Awadalla. 2023. Mix- ture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness.arXiv preprint arXiv:2310.02410 (2023)

    2023 arXiv

  14. [22]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668 (2020)

    2020 arXiv

  15. [23]

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. 2023. Accelerating distributed MoE training and inference with lina. In2023 USENIX Annual Technical Conference (USENIX ATC 23). 945–959

    2023

  16. [24]

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. 2024. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947 (2024)

    2024 arXiv

  17. [25]

    Hou-I Liu, Marco Galindo, Hongxia Xie, Lai-Kuan Wong, Hong-Han Shuai, Yung-Hui Li, and Wen-Huang Cheng. 2024. Lightweight Deep Learning for Resource-Constrained Environments: A Survey.Comput. Surveys (2024)

    2024

  18. [26]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems 36 (2023), 21702–21720

    2023

  19. [27]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher

  20. [28]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al . 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing syste...

    2019

  21. [29]

    Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. 2020. Capuchin: Tensor-based gpu memory management for deep learning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages ...

    2020

  22. [30]

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yux- iong He. 2022. Deepspeed-moe: Advancing mixture-of-experts infer- ence and training to power next-generation ai scale. In International conference on mach...

    2022

  23. [31]

    Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the international conference for high performance computing, networking, storage and analysis. 1–14

    2021

  24. [32]

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He

  25. [33]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017)

    2017 arXiv

  26. [34]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning...

    2023

  27. [35]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al . 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  28. [36]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  29. [37]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)

    2019 arXiv

  30. [38]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning. PMLR, 38087–38099

    2023

  31. [39]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023)

    2023 arXiv

  32. [40]

    Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. 2024. A survey of resource-efficient llm and multimodal founda- tion models. arXiv preprint arXiv:2401.08092 (2024)

    2024 arXiv

  33. [41]

    ZHAO XUANLEI, Bin Jia, Haotian Zhou, Ziming Liu, Shenggan Cheng, and Yang You. 2024. HeteGen: Efficient Heterogeneous Parallel Infer- ence for Large Language Models on Resource-Constrained Devices. Proceedings of Machine Learning and Systems 6 (2024), 162–172

    2024

  34. [42]

    Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchun- shu Zhou, and Yang You. 2024. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739 (2024)

    2024 arXiv

  35. [43]

    Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. 2024. MoE-Infinity: Activation-Aware Expert Offloading for Efficient MoE Serving. arXiv preprint arXiv:2401.14361 (2024)

    2024 arXiv

  36. [44]

    Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. 2024. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems 36 (2024)

    2024

  37. [45]

    Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, and Ji- dong Zhai. 2023. SmartMoE: Efficiently Training{Sparsely-Activated} Models through Combining Offline and Online Parallelization. In2023 USENIX Annual Technical Conference (USENIX ATC 23). 961–975

    2023

  38. [46]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Vic- toria Lin, et al . 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022)

    2022 arXiv

  39. [47]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing...

    2023

  40. [2016]

    arXiv:1609.07843 [cs.CL]

    Pointer Sentinel Mixture Models. arXiv:1609.07843 [cs.CL]

    arXiv

  41. [2018]

    arXiv preprint arXiv:1810.04805 (2018)

    Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  42. [2021]

    In 2021 USENIX Annual Technical Conference (USENIX ATC 21)

    Zero-offload: Democratizing{billion-scale} model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). 551– 564

    2021

  43. [2022]

    https://github.com/huggingface/accelerate

    Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.