Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MoE-Lightning achieves up to 10.3x higher throughput than prior offloading systems for Mixtral 8x7B on a single 16GB T4 GPU, and reaches the memory-bound throughput ceiling with 2-3x less CPU memory.

desk verdict Credible systems work with real scheduling and policy-search ideas; the headline comparisons are too narrow and the model behind the upper-bound claim is unvalidated. read the letter →

arxiv 2411.11217 v1 pith:REHZO4IT submitted 2024-11-18 cs.DC cs.AIcs.LG

Shiyi Cao , Shu Liu , Tyler Griggs , Peter Schafhalter , Xiaoxuan Liu , Ying Sheng , Joseph E. Gonzalez , Matei Zaharia
show 1 more author
Ion Stoica
This is my paper · ORCID
classification cs.DCcs.AIcs.LG
keywords MixtureofExpertsLLMinferenceweightoffloadingCPU-GPUpipeliningrooflinemodelmemory-constrainedGPUstensorparallelism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MoE-Lightning aims to show that large Mixture-of-Experts language models can be served at high batch throughput on GPUs with far less memory than the model requires, by keeping the GPU, CPU, and I/O path busy at the same time. The paper's claim is that its two mechanisms—a CPU-GPU-I/O pipeline schedule called CGOPipe and a Hierarchical Roofline Model (HRM) that searches for the best batch and placement policy—move offloaded MoE inference from low utilization to near the hardware bandwidth ceiling. If true, a single 16GB T4 GPU can run Mixtral 8x7B up to 10.3x faster than prior offloading systems, and memory-bound workloads reach their throughput upper bound while using 2-3x less CPU memory. This matters because it would make current open MoE models practical on commodity hardware, not just on high-end datacenter GPUs.

What carries the argument

The central machinery is CGOPipe, a CPU-GPU-I/O pipeline schedule that overlaps pre-attention GPU work, CPU-side attention, paged weight transfers, and hidden-state transfers across micro-batches. HRM, the Hierarchical Roofline Model, extends the classical roofline model with memory roofs at multiple hierarchy levels and with turning points and a balance point that identify which resource becomes the bottleneck; the policy optimizer uses it to minimize per-layer decode latency $T = \max(\mathrm{comm}_{\mathrm{cpu\to gpu}}, T_{\mathrm{cpu}}, T_{\mathrm{gpu}})$ over the search space of batch size, micro-batch size, attention placement, FFN placement, and weight/KV ratios. Weight paging with a double buffer lets next-layer weights transfer while current-layer compute proceeds, and a custom CPU grouped-query-attention kernel makes CPU attention faster than transferring KV cache to the GPU.

What would settle it

Run the decode stage for a single layer under the policy HRM selects and compare measured per-layer latency to $\max(\mathrm{comm}_{\mathrm{cpu\to gpu}}, T_{\mathrm{cpu}}, T_{\mathrm{gpu}})$ computed from profiled peak rates; if measured latency consistently exceeds the model by a large margin, the optimizer misses the balance point and the throughput-upper-bound claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in memory-constrained MoE inference is not raw GPU compute but the idle time caused by serial weight loads, and that this idle time can be scheduled away. MoE-Lightning performs attention on the CPU rather than transferring KV cache to the GPU, executes post-attention and pre-attention GPU work on different micro-batches back to back, and transfers next-layer expert weights in pages while the current layer computes. HRM then picks the batch size, micro-batch size, attention placement, FFN placement, and static weight ratio that balances CPU-to-GPU bandwidth against CPU and GPU compute, modeled per layer as $T = \max(\mathrm{comm}_{\mathrm{cpu\to gpu}}, T_{\mathrm{cpu}}, T_{\mathrm{gpu}})$. The reported result is up to 10.3x higher generation throughput than the best offloading baseline on a single T4 for Mixtral 8x7B, 3.5x under request padding, and, in the GPU-memory-bound regime, the theoretical throughput ceiling with 2-3x less CPU memory than prior systems.

Load-bearing premise

The optimizer trusts a model that treats per-layer latency as the maximum of three independently overlappable components—CPU-to-GPU transfer time, CPU compute time, and GPU compute time—derived from theoretical FLOP counts and profiled peak rates; if real kernels or bidirectional transfer contention violate that independence, the policy search can pick the wrong batch size, micro-batch size, or placement.

Editorial extensions

If this is right

  • A single 16GB T4 GPU can serve Mixtral 8x7B batch workloads at up to 10.3x the throughput of earlier offloading systems, with 3.5x improvement even when requests are padded to equal length.
  • When GPU memory is the binding constraint, the system can reach the theoretical throughput ceiling while using 2-3x less CPU memory, meaning cheaper CPU configurations become viable.
  • Scaling to 2-4 T4 GPUs with tensor parallelism raises the throughput ceiling, and the paper reports super-linear scaling for Mixtral 8x22B as GPU memory grows.
  • The policy optimizer runs offline in under a minute, so the same system can adapt to different hardware, models, and workloads without extensive per-deployment profiling or data fitting.
  • Under CGOPipe, throughput stays GPU-memory-bound even as generation length grows, avoiding the throughput decline that prior offloading systems show.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of HRM's core assumption—that CPU-to-GPU transfers in opposite directions overlap without contention—could be done by comparing predicted per-layer latency against measured per-layer latency; the paper does not report such a validation.
  • The same hierarchical-roofline view could be pushed one level deeper to include disk or multi-node transfers, potentially extending CGOPipe-style scheduling to models too large for CPU memory, a direction the paper leaves as future work.
  • The reported 2-3x CPU-memory savings suggest the balance point is driven more by I/O bandwidth than by buffer capacity, which could be tested by sweeping CPU memory limits while holding bandwidth fixed and observing when throughput saturates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. MoE-Lightning proposes a system for high-throughput batched inference of Mixture-of-Experts models on GPUs whose memory is insufficient to hold the model. The two main contributions are CGOPipe, a CPU-GPU-I/O pipeline schedule with paged weight transfers that overlaps CPU attention, GPU computation, and transfers, and HRM, a hierarchical roofline model that estimates per-layer decode latency as a max of CPU-to-GPU communication, CPU compute, and GPU compute, and is used to search for the batch size, micro-batch size, and placement policy. The paper reports up to 10.3x higher throughput than FlexGen/DeepSpeed on Mixtral 8x7B, results on Mixtral 8x22B and DBRX with up to four T4 GPUs, and claims that the system can reach the throughput upper bound with 2-3x less CPU memory. The evaluation includes end-to-end throughput measurements, request-padded and unpadded comparisons, ablations of the optimizer policy, and case studies of policy changes under different hardware configurations.

Significance. If the claims are fully substantiated, this is a useful contribution to memory-constrained MoE serving: a principled roofline-based policy search plus a pipeline schedule that packs CPU compute, GPU compute, and I/O better than the dense-model-oriented baselines. The paper's strengths are its end-to-end experiments against two existing offloading systems, its ablation in §6.1 showing the policy optimizer gives a 1.77x improvement on FlexGen, and the breadth of model/hardware settings (Mixtral 8x7B/8x22B, DBRX, T4/L4, 1-4 GPUs). The main gaps are that the headlined 'reaches the throughput upper bound with 2-3x less CPU memory' claim is not supported by a direct measurement, the performance model that defines the upper bound is not validated against measured per-layer latencies, and the 'state-of-the-art' comparison excludes MoE-specific offloading systems cited in the paper's own related work. These issues are addressable, but they affect the central claims.

major comments (3)
  1. [§4.2, Eq. (12), Algorithm 1] The per-layer latency model in Eq. (12) takes a max of three aggregated durations (CPU-to-GPU communication, CPU compute, GPU compute), which presumes that these three resource streams can be perfectly overlapped and that opposite-direction transfers proceed simultaneously as stated in §4.1. Algorithm 1's dependencies (for example, CPUAttn(i,j+2) requires OffloadQKV(i,j+2), and same-direction transfers D1/D2/D3 are serialized) introduce bubbles and queueing effects that the max formula cannot represent. The paper acknowledges 'discrepancies between the theoretical performance estimation and the kernel's real performance' in §4.2, but it reports no measured-versus-predicted comparison. Because the abstract's claim that MoE-Lightning 'can reach the throughput upper bound' is defined relative to the same HRM that generates the policy, this part of the headline claim is currently unsupported. Please add a validation experiment that compares HRM-predicted per-layer decode latency with measured latency across a range of policies and hardware settings, and report the prediction error and the policy-ranking accuracy.
  2. [§5.1, §5.2, §7] The evaluation compares against FlexGen and DeepSpeed Zero-Inference only, yet §7 cites MoE-specific offloading systems (MoE-Infinity, PowerInfer, Fiddler, LLM in a Flash) that directly address running MoE models on memory-constrained GPUs. Therefore the abstract's 'state-of-the-art offloading-enabled LLM inference systems' is not established by the experiments presented. Either add one or more of these systems as baselines, or provide concrete arguments for exclusion (for example, that they target latency-oriented single-request serving, rely on activation sparsity, or do not support batched throughput evaluation), and then adjust the state-of-the-art phrasing accordingly.
  3. [Abstract, Fig. 1, §5] The claim that MoE-Lightning 'can reach the throughput upper bound with 2-3x less CPU memory' is not supported by any measurement in the evaluation section. Fig. 1 is a schematic, and the throughput results in §5 do not include a sweep of CPU memory capacity. Please add an experiment that, for a fixed GPU and model, plots achieved throughput versus CPU memory capacity for MoE-Lightning and at least one baseline, and shows how the measured achievable throughput compares with the computed upper bound (for example, the throughput when the model and KV cache fit entirely in GPU memory, or the HRM-computed bound). Without this, the '2-3x less CPU memory' efficiency claim remains an unverified extrapolation.
minor comments (5)
  1. [§7 heading] The heading 'Memory-constriant LLM Inference' contains a typo; it should be 'Memory-constrained LLM Inference'.
  2. [§4.1] The sentence 'FastDecode [17] dose not consider weights offloading' contains a typo; 'dose' should be 'does'.
  3. [§3.3, footnote 4] Footnote 4 states that the case study uses theoretical operational intensity instead of numbers from real profiling; this should be stated more prominently in the main text, since Figs. 4 and 5 are otherwise presented as empirical roofline plots.
  4. [Appendix A.2, Algorithm 2] The pseudocode has a suspicious condition at line 6 ('if partitions == ∅' after partitions has been initialized with n_ub entries), and line 8 references the minimizer of partitions_sums without checking whether idx is valid; please clarify the intended batching logic.
  5. [Abstract, §5.2] The abstract reports 'up to 10.3x higher throughput' without stating that this is the unpadded comparison; the padded comparison (MoE-Lightning(p)) reaches at most 3.5x. Please state the padding protocol in the abstract so the headline number is not misleading.

Circularity Check

1 steps flagged · score 2.0 of 10

The only self-referential element is the HRM-defined 'throughput upper bound'; the measured throughput comparisons to baselines are independent.

  1. self definitional [Abstract; Section 4.2, Eq. (12)]
    "we search for the optimal policy P that minimizes per-layer latency T(M,H,W,P) for the pipeline schedule in Section 4.1 ... in order to reach the optimal balance point (Eq. (11)). ... T(M,H,W,P) = max(comm_cpu_to_gpu,T_cpu,T_gpu) ... When the theoretical system throughput is bounded by the GPU memory, MoE-Lightning can reach the throughput upper bound with 2-3x less CPU memory."

    The claimed 'throughput upper bound' is not independently measured; it is defined by the HRM balance point that the optimizer is designed to reach. Section 4.2 defines per-layer latency as T = max(comm_cpu_to_gpu, T_cpu, T_gpu) and searches for the policy that minimizes this exact T. Therefore, reaching the 'upper bound' is equivalent to finding the minimum of the model's own objective function, making the achievement partly true by construction. The paper acknowledges discrepancies between the model and real kernel performance and provides no measured-vs-predicted validation, so the claim remains internal to the model rather than an empirical prediction. The measured speedups over FlexGen/DeepSpeed are external benchmarks and unaffected by this circularity.

full rationale

The paper is largely self-contained: CGOPipe is a new pipeline schedule, and the 10.3x throughput improvements are measured against external baselines using real workloads. The central non-circular evidence is the experimental comparison to FlexGen and DeepSpeed, which does not depend on the authors' own definitions. However, a portion of the headline claim ('can reach the throughput upper bound with 2-3x less CPU memory') is circular in a mild sense: the 'upper bound' is defined by the same performance model (HRM, Eq. (11)) that selects the policy, and the optimizer minimizes Eq. (12) to achieve it. No validation against measured per-layer latencies is provided, and the assumptions of simultaneous bidirectional transfers and full overlap in Eq. (12) are acknowledged as potentially discrepant. This self-referential bound inflates the reported achievement but does not invalidate the empirical comparison. Given that the main performance claims rest on external measurements, the circularity score is 2 rather than higher.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a small set of idealizations: peak-rate roofline accounting, contention-free bidirectional transfers, CPU attention being preferable in the target regime, and sufficient CPU memory. No novel physical entity or fitted constant is introduced; policy variables such as batch size, micro-batch size, and placement ratios are search outputs, not parameters fitted to the reported throughput.

assumptions (6)
  • standard math Achievable performance is bounded by min(peak compute, local memory bandwidth, cross-level bandwidth) as in Eq. (7).
    Invoked in Section 3.2 to define HRM; assumes peak FLOPS and peak bandwidth are the only limits, excluding kernel efficiency, latency, and contention.
  • domain assumption Host-device transfers in opposite directions can proceed simultaneously without mutual interference.
    Stated in Section 4.1: 'Due to independent data paths, data transfers in opposite directions can happen simultaneously.' CGOPipe's overlap relies on this.
  • domain assumption For the memory-constrained settings targeted, CPU attention is consistently better than GPU attention with KV offload.
    Concluded in Section 3.3 and used in Section 4: 'for the memory-constrained scenarios we target in this paper, CPU attention is consistently better than GPU attention, according to our performance model.'
  • domain assumption CPU memory is large enough to hold the full model and KV cache, so disk offloading is unnecessary.
    Section 4 states 'we do not consider disk offloading in this work'; Future Work C says the system 'focuses on scenarios where GPU memory is limited but sufficient CPU memory is available to hold the model.'
  • domain assumption Policy ranking can be based on theoretical FLOPs/bytes and profiled peak hardware rates, despite kernel-level discrepancies.
    Section 4.2: 'While there are discrepancies between the theoretical performance estimation and the kernel's real performance, such modeling can provide a reasonable estimation of the relative effectiveness of any two policies.' No direct validation is provided.
  • domain assumption Tensor parallelism multiplies GPU memory and bandwidth with no modeling of GPU-GPU communication cost.
    Section 4.3: 'we have tp_size times more GPU memory capacity and GPU memory bandwidth'; Future Work C admits the model is single-node and does not account for GPU-GPU communication.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs." pith.science (2026). https://pith.science/paper/REHZO4IT

@misc{pith2026241111217,
  author       = {Pith},
  title        = {Pith review of: MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/REHZO4IT}},
  note         = {Machine review of arXiv:2411.11217}
}
read the original abstract

Efficient deployment of large language models, particularly Mixture of Experts (MoE), on resource-constrained platforms presents significant challenges, especially in terms of computational efficiency and memory utilization. The MoE architecture, renowned for its ability to increase model capacity without a proportional increase in inference cost, greatly reduces the token generation latency compared with dense models. However, the large model size makes MoE models inaccessible to individuals without high-end GPUs. In this paper, we propose a high-throughput MoE batch inference system, that significantly outperforms past work. MoE-Lightning introduces a novel CPU-GPU-I/O pipelining schedule, CGOPipe, with paged weights to achieve high resource utilization, and a performance model, HRM, based on a Hierarchical Roofline Model we introduce to help find policies with higher throughput than existing systems. MoE-Lightning can achieve up to 10.3x higher throughput than state-of-the-art offloading-enabled LLM inference systems for Mixtral 8x7B on a single T4 GPU (16GB). When the theoretical system throughput is bounded by the GPU memory, MoE-Lightning can reach the throughput upper bound with 2-3x less CPU memory, significantly increasing resource utilization. MoE-Lightning also supports efficient batch inference for much larger MoEs (e.g., Mixtral 8x22B and DBRX) on multiple low-cost GPUs (e.g., 2-4 T4).

Figures

Figures reproduced from arXiv: 2411.11217 by the authors.

Figure 1
Figure 1. MoE-Lightning achieves higher throughput with far less CPU memory, enabled by CGOPipe and HRM [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of a Mixture of Experts in Large Language Models. Despite their advantages, the widespread use of MoE mod￾els faces challenges due to the difficulties in managing and deploying models with extremely high parameter counts that demand substantial memory. Thus, our work aims to make MoE models more accessible to those lacking extensive high-end GPU resources. 2.2 LLM Inference LLMs are trained to predict t… view at source ↗
Figure 3
Figure 3. Hardware Configurations for the L4 Instance. 3.3 Case Study To visualize the turning points and balance points discussed in the preceding sections, we conduct a case study with real HRM plots for computations2 in a single layer of the Mixtral 8x7B model on a Google Cloud Platform L4 instance. The hardware setting is as detailed in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Hierarchical Roofline Model for Mixtral 8x7B’s Grouped Query Attention Block in Decode Stage on L4 In￾stance. (Context Length = 512) 2We only discuss the attention and feed-forward blocks since they account for the majority of computation time and represent quite diffe…
Figure 5
Figure 5. Figure 5: Hierarchical Roofline Model for Mixtral 8x7B’s MoE Feed-Forward Block in Decode Stage on L4 Instance. strategies can increase the operational intensity for the MoE FFN computation regarding the data on the CPU. If the batch size can be continually increased, then when …
Figure 6
Figure 6. Figure 6: Different Scheduling Strategies: Square sizes vary with workloads and policies. For example, larger 𝜇 or longer sequences lengthen the orange (attention) and the green (KV cache transfer from CPU to GPU) squares. Squares with red zigzag lines indicate the unnecessary G…
Figure 7
Figure 7. Figure 7: End-to-end Results for MTBench on Different Model-Hardware Configurations. Normally, MoE-Lightning’s perfor￾mance will be much higher than MoE-Lightning (p) since padding will lead to higher memory consumption and attention computation overhead8 . Here MoE-Lightning ac…
Figure 8
Figure 8. Figure 8: MoE-Lightning with Tensor-Parallelism for MT￾Bench @ S8 & S9. S6 and S7 in [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Latency Comparison for a single layer’s KV cache transfer, CPU Attention Kernel and the MoE FFN Kernel wrt. 𝜇 and Context Length in Decode Stage. CPU to GPU memory bandwidth, the trade-offs will also change. Then the question becomes: when we have enough GPU memory (e.…
Figure 10
Figure 10. Figure 10: Policy changes with different hardware config￾urations (prompt length=512, generation length=32). Red points denote performing attention on the CPU. We can see that when running Mixtral 8x7B on two A100 GPUs, as CPU-to-GPU memory bandwidth increases, more weight will …
Figure 11
Figure 11. Figure 11: Simplified Demonstration of MoE-Lightning’s Memory Management. A.2 Request Batching For a given workload, the optimizer introduced in §4.2 takes the average prompt length to search for an optimal policy. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FloE: On-the-Fly MoE Inference on Memory-constrained GPU

    cs.LG 2025-05 conditional novelty 6.0 of 10

    FloE compresses and prefetches only the active parts of MoE experts, running Mixtral-8x7B on an 11 GB GPU with about 5 to 8 percent average accuracy loss.

Reference graph

Works this paper leans on

57 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Flashinfer: Kernel library for llm serving

    Flashinfer AI. Flashinfer: Kernel library for llm serving. https://github. com/flashinfer-ai/flashinfer , 2024. Accessed: 2024-05-20

    work page 2024

  2. [2]

    Gqa: Training generalized multi- query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi- query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

  3. [3]

    Llm in a flash: Efficient large language model inference with limited memory, 2024

    Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llm in a flash: Efficient large language model inference with limited memory, 2024

    work page 2024

  4. [4]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, page...

    2022

  5. [5]

    Accelerating large language model decoding with speculative sampling, 2023

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023

    2023

  6. [6]

    Lifelong language pretraining with distribution-specialized experts

    Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. Lifelong language pretraining with distribution-specialized experts. In International Conference on Ma- chine Learning, pages 5383–5395. PMLR, 2023. 13

    work page 2023

  7. [7]

    Spreadsheetcoder: Formula prediction from semi-structured context

    Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. Spreadsheetcoder: Formula prediction from semi-structured context. In International Conference on Machine Learning, pages 1661–1672. PMLR, 2021

    work page 2021

  8. [8]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas An- gelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024

    arXiv 2024

Show all 57 references
  1. [9]

    Generating long sequences with sparse transformers, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019

    2019

  2. [10]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-ex...

    2024 arXiv

  3. [11]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Repre- sentations (ICLR), 2024

    2024

  4. [12]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  5. [13]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lep- ikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning , pages 5547–5569. ...

    2022

  6. [14]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Ka- dian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Fast inference of mixture-of-experts language models with offloading, 2023

    Artyom Eliseev and Denis Mazur. Fast inference of mixture-of-experts language models with offloading, 2023

    2023

  8. [16]

    Dap- ple: A pipelined data parallel approach for training large models

    Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al. Dap- ple: A pipelined data parallel approach for training large models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Paral...

    2021

  9. [17]

    Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines, 2024

    Jiaao He and Jidong Zhai. Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines, 2024

    2024

  10. [18]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  11. [19]

    Hugging face accelerate

    HuggingFace. Hugging face accelerate. https://huggingface.co/docs/ accelerate/index, 2022

    2022

  12. [20]

    Intel(r) oneapi math kernel library (onemkl)

    Intel. Intel(r) oneapi math kernel library (onemkl). https://www.intel. com/content/www/us/en/developer/tools/oneapi/onemkl.html, 2024

    2024

  13. [21]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):79– 87, 1991

    1991

  14. [22]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Men- sch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-An...

    2024 arXiv

  15. [23]

    Hierarchical mixtures of experts and the em algorithm

    Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994

    1994

  16. [24]

    Fu, Christo- pher Ré, and Azalia Mirhoseini

    Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christo- pher Ré, and Azalia Mirhoseini. Hydragen: High-throughput llm inference with shared prefixes, 2024

    2024

  17. [25]

    Fiddler: Cpu- gpu orchestration for fast inference of mixture-of-experts models, 2024

    Keisuke Kamahori, Yile Gu, Kan Zhu, and Baris Kasikci. Fiddler: Cpu- gpu orchestration for fast inference of mixture-of-experts models, 2024

    2024

  18. [26]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  19. [27]

    Gshard: Scaling giant models with conditional com- putation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional com- putation and automatic sharding. arXiv preprint arXiv:2006.16668 , 2020

    2006 arXiv

  20. [28]

    Fast inference from transformers via speculative decoding, 2023

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding, 2023

    2023

  21. [29]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  22. [30]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving, 2024

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving, 2024

    2024

  23. [31]

    Gonzalez, Ion Stoica, and Matei Zaharia

    Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E. Gonzalez, Ion Stoica, and Matei Zaharia. Optimizing llm queries in relational workloads, 2024

    2024

  24. [32]

    https://mistral.ai/news/mixtral-8x22b/, April 2024

    MistralAI. https://mistral.ai/news/mixtral-8x22b/, April 2024

    2024

  25. [33]

    Can foundation models wrangle your data?arXiv preprint arXiv:2205.09911, 2022

    Avanika Narayan, Ines Chami, Laurel Orr, and Christopher Ré. Can foundation models wrangle your data?arXiv preprint arXiv:2205.09911, 2022

    2022 arXiv

  26. [34]

    Pipedream: generalized pipeline parallelism for dnn train- ing

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: generalized pipeline parallelism for dnn train- ing. In Proceedings of the 27th ACM symposium on operating systems principles,...

    2019

  27. [35]

    Efficient large- scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large- scale language model training on gpu clusters using megatron-lm. In Procee...

    2021

  28. [36]

    Pytorch: An imperative style, high- performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  29. [37]

    Efficiently scaling transformer inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems , 5, 2023

    2023

  30. [38]

    Int4 decoding gqa cuda optimizations for llm inference

    Sarunya Pumma, Jongsoo Park, Jianyu Huang, Amy Yang, Jaewon Lee, Daniel Haziza, Grigory Sizov, Jeremy Reizenstein, Jeff Johnson, and Ying Zhang. Int4 decoding gqa cuda optimizations for llm inference. https://pytorch.org/blog/int4-decoding/, 2024

    2024

  31. [39]

    Accelerating transformer inference for translation via parallel decod- ing

    Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodola. Accelerating transformer inference for translation via parallel decod- ing. In Proceedings of the 61st Annual Meeting of the Association for Computati...

    2023

  32. [40]

    Bloom: A 176b-parameter open- access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, 14 François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open- access multilingual language model. arXiv preprint arXiv:2211.05100, 2022

    2022 arXiv

  33. [41]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  34. [42]

    Flexgen: High-throughput generative inference of large language models with a single gpu

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, page...

    2023

  35. [43]

    Powerinfer: Fast large language model serving with a consumer-grade gpu, 2023

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with a consumer-grade gpu, 2023

    2023

  36. [44]

    Blockwise parallel decoding for deep autoregressive models, 2018

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models, 2018

    2018

  37. [45]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774, 2024

    2024 arXiv

  38. [46]

    Introducing dbrx: A new state-of-the-art open llm, 2024

    Mosaic Research Team. Introducing dbrx: A new state-of-the-art open llm, 2024. https://www.databricks.com/blog/introducing-dbrx-new- state-art-open-llm , March 2024. Accessed 2024-06-20

    2024

  39. [47]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  40. [48]

    Patterson

    Samuel Williams, Andrew Waterman, and David A. Patterson. Roofline: an insightful visual performance model for multicore ar- chitectures. Commun. ACM, 52(4):65–76, 2009

    2009

  41. [49]

    Hetegen: Efficient heterogeneous parallel inference for large language models on resource-constrained devices

    ZHAO XUANLEI, Bin Jia, Haotian Zhou, Ziming Liu, Shenggan Cheng, and Yang You. Hetegen: Efficient heterogeneous parallel inference for large language models on resource-constrained devices. Proceedings of Machine Learning and Systems , 6:162–172, 2024

    2024

  42. [50]

    Moe- infinity: Activation-aware expert offloading for efficient moe serving, 2024

    Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, and Mahesh Marina. Moe- infinity: Activation-aware expert offloading for efficient moe serving, 2024

    2024

  43. [51]

    Orca: A distributed serving system for {Transformer-Based} generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) , pages 521–538, 2022

    2022

  44. [52]

    Llm inference unveiled: Survey and roofline model insights, 2024

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. Llm inference unveiled: Survey and roofline model insights, 2024

    2024

  45. [53]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Vic- toria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  46. [54]

    H2o: Heavy-hitter oracle for efficient generative in- ference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative in- ference of large language models. Advances in Neural Information Processing Sys...

    2024

  47. [55]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems , 36, 2024

    2024

  48. [56]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024

    2024

  49. [57]

    Mixture-of- experts with expert choice routing

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of- experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114, 2022. 15

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.