Pith. sign in

REVIEW 3 major objections 5 minor 41 references

HarMoEny: Efficient Multi-GPU Inference of MoE Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HarMoEny claims that moving tokens to underused GPUs and prefetching experts behind compute removes nearly all waiting in multi-GPU MoE inference, lifting throughput by 37%-70% under skew.

desk verdict Solid systems paper with a convincing ablation, but the undisclosed token threshold q makes the headline gains hard to verify. read the letter →

arxiv 2506.12417 v2 pith:LZND4V2K submitted 2025-06-14 cs.DC

Zachary Doucet , Rishi Sharma , Martijn de Vos , Rafael Pires , Anne-Marie Kermarrec , Oana Balmau This is my paper
classification cs.DC
keywords Mixture-of-ExpertsMoEinferencemulti-GPUservingexpertparallelismloadbalancingtokenrebalancingprefetchingtime-to-first-token
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

During multi-GPU inference of Mixture-of-Experts (MoE) models, a handful of experts receive most tokens, so the GPUs holding those popular experts work while the others wait at the all-to-all synchronization barrier; the paper measures this waiting at up to 86% of GPU time. HarMoEny claims that two lightweight techniques remove most of that waiting: a greedy per-batch scheduler that moves tokens from overloaded GPUs to underloaded ones, and asynchronous prefetching that loads needed expert weights into GPU memory while another expert is still computing. Because both techniques adapt every batch from a small metadata exchange (about 4 kB) rather than from offline profiling, the system tracks shifting expert popularity. On models with 128 and 60 experts, the paper reports throughput gains of 37%-70% and time-to-first-token reductions of 34%-41% over the next-best baseline under heavy skew, with GPU idle time reduced by up to 84.7%.

What carries the argument

The load-bearing object is the pair formed by a greedy token-rebalancing schedule (Algorithm 2) and an asynchronous expert-fetching protocol, tied together by the transfer threshold $q$. Each batch, GPUs exchange a few kilobytes of routing metadata so every GPU can deterministically recompute the same schedule: find the most overloaded GPU, find the largest block of tokens headed to it for one expert, and move as many of those tokens as fit to the least loaded GPU without pushing it past the average load, stopping when the block is below $q$. The threshold is set by the inequality $q > \frac{\phi \cdot d_{\mathrm{type}}}{2\beta}$, derived from a two-layer feed-forward network FLOPs model, which expresses the condition that computing on $q$ tokens takes longer than fetching an expert's weights from system memory. The prefetch protocol then loads the needed weights on a dedicated GPU stream, overwriting the memory of a just-finished expert, so the transfer is hidden behind computation.

What would settle it

Run the same two models on an eight-GPU system while sweeping $q$ from below the Eq. (4) bound to well above it, and also rerun with the interconnect bandwidth artificially halved. If the lowest-latency $q$ is not in the predicted range, or if halving the bandwidth does not shift the optimal $q$, the masking model behind Eq. (4) is wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is that load imbalance, not all-to-all communication, dominates MoE inference latency, and that it can be nearly eliminated by changing where tokens are executed rather than where experts are placed. With balanced loads the synchronization step accounts for only 2% of execution time, whereas static expert placement leaves GPUs idle up to 86% of the time. HarMoEny's scheduler computes a global token-to-GPU assignment each batch from exchanged token counts, then greedily moves the largest token blocks from the most loaded GPU to the least loaded GPU, provided the block is large enough to justify loading an expert on the receiving GPU. The asynchronous prefetch protocol overwrites a finished expert's memory with the next needed expert so the weight transfer runs concurrently with computation. In experiments, this combination cuts mean waiting time from about 82%-86% of GPU time to about 1%-2.6%, raises throughput by 37%-70%, and lowers time-to-first-token by 34%-41% relative to the next-best of four baselines under heavy skew.

Load-bearing premise

The whole speedup rests on the token threshold $q$ being chosen so that loading an expert into GPU memory is hidden behind the compute of another expert, yet the paper never reports the $q$ values used or tests how sensitive results are to that choice.

Editorial extensions

If this is right

  • Servers can drop offline profiling and integer-program placement: a per-batch metadata exchange of a few kilobytes and a greedy pass is enough to keep GPUs balanced as expert popularity changes between batches.
  • The gains grow with expert size: for the 33 MB experts of Qwen, HarMoEny is 15%-28% faster than shadowing-based baselines, whose replication is limited by GPU memory.
  • The technique is complementary to kernel-level optimizations: because it works at the MoE-layer scheduling level, other optimized serving stacks could adopt the same rebalancing and prefetching logic.
  • Under workloads with little skew, the overhead is small but not zero: on the 128-expert Switch model HarMoEny trails FastMoE and FasterMoE by about 8% at zero skew, so the benefit is conditional on imbalance being present.
  • A near-perfect token balance turns the all-to-all barrier into a minor cost (about 2% of execution time), meaning future optimization effort can focus on expert compute rather than communication.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $q$ formula should be re-examined for heterogeneous MoE layers, where expert compute is not a uniform function of token count; a per-expert or measured-FLOPs threshold would make the scheduler robust to layers with different shapes.
  • The same prefetch-and-rebalance pattern could be applied to other sparsely activated components, such as attention heads or retrieval-augmented memory, whenever a small router decides which parameters to load.
  • If $q$ were reported per experiment, the results would be directly portable to other hardware; the paper's formula gives a starting point but leaves validation of $q$'s sensitivity as future work.
  • On clusters where GPUs share interconnect bandwidth asymmetrically, the single bandwidth $\beta$ in Eq. (4) would need to become per-GPU or topology-aware; otherwise prefetching on one GPU can steal bandwidth from another.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. HarMoEny proposes two techniques for multi-GPU MoE inference: (i) dynamic token redistribution across GPUs according to a greedy load-balancing schedule, and (ii) asynchronous expert prefetching that overwrites finished experts' GPU memory while other experts compute. The authors claim that these techniques achieve near-perfect GPU load balance under skewed, dynamically changing expert popularity, increasing throughput by 37%–70% and reducing time-to-first-token by 34%–41% compared with the next-best baseline. The paper includes a derivation of a token threshold q (Eq. 4) that is intended to make expert prefetch time fully masked by expert computation, an implementation in PyTorch, and an evaluation on two MoE models (Switch128 and Qwen), three real datasets plus synthetic workloads, and four baselines (DeepSpeed, FastMoE, FasterMoE, ExFlow). An ablation isolates the effects of token rebalancing and asynchronous prefetching on GPU idle time and layer latency.

Significance. If the reported results hold, HarMoEny is a useful systems contribution: it addresses dynamic load imbalance without offline profiling, which is a genuine limitation of placement-based systems like ExFlow and static round-robin schemes. The paper makes a strong empirical case that token count is an adequate proxy for compute load on the evaluated models, and the ablation study cleanly separates the two mechanisms. The open-source implementation and the comparison against four external baselines are also strengths. However, the central mechanism depends on an unreported q threshold and unreported batch sizes, and the reported scheduling overhead is high enough to conflict with the paper's 'minimal overhead' narrative. These issues are fixable within the manuscript's scope and do not require reimagining the approach.

major comments (3)
  1. [§4.4, §5.2 (Figs. 7–10), Alg. 2] The token threshold q is load-bearing but never reported. Eq. (4) gives q > phi*d_type/(2*beta); for the DGX-1 V100s and PCIe system memory this is on the order of thousands of tokens, yet Section 5 never states the q values used or the batch size (tokens per GPU per MoE layer). Because Algorithm 2 aborts rebalancing when t_move < q or t_g[gmin] + q > tavg, a large q would leave the mechanism inactive and a small q would violate the prefetch-masking criterion of Eq. (1). The paper's statement that HarMoEny is 'not extremely sensitive to q' is unsupported by data. Please report q for each model/skew/dataset configuration, the batch sizes, and a sensitivity sweep around Eq. (4) (e.g., 0.5x, 1x, and 2x of the estimated q) for at least the 90% skew and real-dataset configurations.
  2. [§5.3.1, Fig. 11, §1] The paper claims in Section 1 and Section 4.1 that HarMoEny's metadata exchange is 'small (4kB)' and 'negligible', but Figure 11 and the text in Section 5.3.1 report that the scheduling/rebalancing step takes 30.8% and 20.3% of mean layer latency for Switch128 and Qwen, respectively. These statements are in tension: a 30.8% share of a 149.5 ms layer is not a minimal overhead. The authors should decompose the metadata exchange, scheduler compute, and rebalancing costs, and state whether the 8% throughput loss observed for FastMoE/FasterMoE at zero skew (Section 5.2) is attributable to this overhead. This does not invalidate the main result but is necessary for the claimed lightweight design.
  3. [§5.2, Figs. 7–10] The headline quantitative claims (37%–70% throughput gain, 34%–41% TTFT reduction) are presented without error bars, repeated seeds, or confidence intervals. Given that Figures 9 and 10 show large batch-to-batch variance and that HarMoEny's advantage varies by dataset, a single run is insufficient to support the stated ranges. Please add at least three seeds with mean ± std (or box plots) for the main comparisons, and report the exact workload sizes (batch tokens per GPU) used in Figures 7–10.
minor comments (5)
  1. [Figs. 7–8] The x-axis labels 'bookcorpusrandom wikitextwmt19' are missing separators between dataset names; please add spaces or distinct tick labels for readability.
  2. [§5.2] The sentence 'FastMoE and FasterMoE obtain virtually identical results, within 92% to 98% the throughput of HarMoEny' should read 'within 92% to 98% of the throughput of HarMoEny'.
  3. [Appendix B, Eq. (9)] The simplification from Eq. (7) to Eq. (9) is notationally abrupt: the step from q*p*(2m-1) to 2*q*p*m is not shown, and the equality '2qpm + 2qpm' is confusing because the two terms represent different operations. Please spell out the simplification.
  4. [§5.1.2] The description of the Random synthetic dataset says it is constructed by 'stringing random tokens in a sequence of a desired length', but it does not state the vocabulary size or whether the same random sequence is used across GPUs and baselines; please clarify.
  5. [§4.3] The reported expert loading times (11 ms reduced to 2 ms for V100 GPUs) seem inconsistent with the PCIe bandwidth used in Eq. (4): an 18 MB expert at 16 GB/s transfers in about 1.1 ms. Please explain what the 11 ms and 2 ms numbers include (e.g., offload plus load, allocation overhead, kernel launch overhead).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: HarMoEny's gains are evaluated against external baselines and the q threshold is derived from hardware parameters rather than fitted to the reported outcomes.

full rationale

The paper's central claim (37%-70% throughput increase, 34%-41% TTFT reduction under heavy skew) is supported by experiments comparing against four external baselines (DeepSpeed, FastMoE, FasterMoE, ExFlow), not by a self-referential derivation. The token rebalancing algorithm (Algorithm 2) explicitly equalizes token counts, so the 'near-perfect load balance' shown in Figure 2 is a direct description of the algorithm's behavior rather than a prediction smuggled in as a finding; this is presented as design, not as an independent result. The token threshold q is derived in Section 4.4 / Appendix B from Eq. (1) (compute time > transfer time) using hardware FLOPS phi, expert size, and PCIe bandwidth beta, yielding Eq. (4) q > phi*d_type/(2*beta). This is a first-principles design rule, not fitted to the latency/throughput numbers, and the paper does not rename any fitted parameter as a prediction. There are no load-bearing self-citations: the references are to external systems and prior scheduling work (e.g., Quasar, Borg), and no 'uniqueness theorem' from the authors' own prior work is invoked. The undisclosed q value and missing sensitivity analysis are reproducibility/correctness concerns, not circularity: an unspecified operating point does not make the evaluation equal to its inputs. Therefore the derivation chain is self-contained and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The system introduces no new physical entities. Its main tunable is q, whose actual value is undisclosed; the compute-load proxy and overwrite assumptions are standard for uniform MoE inference but should be stated.

free parameters (2)
  • q (token transfer threshold)
    Lower bound derived in Eq. (4) from GPU FLOPS, PCIe bandwidth, and dtype size, but the actual value used in Section 5 experiments is not reported. It controls when token offloading is profitable.
  • expert_cache_size
    Appears in MoEConfig (Appendix C); determines how many experts are resident per GPU but its value and effect are not discussed in the evaluation.
assumptions (4)
  • domain assumption Token count is a valid proxy for GPU compute load; all experts impose equal per-token FLOPs.
    The scheduler balances token counts (Algorithm 2) and the evaluation interprets balanced token counts as balanced time. This holds for the uniform Switch128 and Qwen experts tested, but not for heterogeneous expert sizes.
  • domain assumption Expert weights in GPU memory can be overwritten without offloading during inference.
    Section 4.3 states offloading is not needed; this presumes inference does not require the old expert state (no backward pass), which is true for this setting.
  • standard math The greedy rebalancing algorithm (Algorithm 2) terminates quickly and produces the same schedule on all GPUs without further synchronization.
    The algorithm is deterministic given the global metadata exchanged in Step 2; its runtime is measured empirically but not bounded analytically.
  • ad hoc to paper The q estimation model ignores activation memory, kernel launch overhead, and NVLink vs PCIe bandwidth differences.
    Section 4.4 derives q from a simplified FLOPs/bandwidth inequality; the paper acknowledges 'ignoring negligible factors' but provides no validation of the approximation on the testbed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HarMoEny: Efficient Multi-GPU Inference of MoE Models." pith.science (2026). https://pith.science/paper/LZND4V2K

@misc{pith2026250612417,
  author       = {Pith},
  title        = {Pith review of: HarMoEny: Efficient Multi-GPU Inference of MoE Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZND4V2K}},
  note         = {Machine review of arXiv:2506.12417}
}
read the original abstract

Mixture-of-Experts (MoE) models offer computational efficiency during inference by activating only a subset of specialized experts for a given input. This enables efficient model scaling on multi-GPU systems that use expert parallelism without compromising performance. However, load imbalance among experts and GPUs introduces waiting times, which can significantly increase inference latency. To address this challenge, we propose HarMoEny, a novel solution to address MoE load imbalance through two simple techniques: (i) dynamic token redistribution to underutilized GPUs and (ii) asynchronous prefetching of experts from the system to GPU memory. These techniques achieve a near-perfect load balance among experts and GPUs and mitigate delays caused by overloaded GPUs. We implement HarMoEny and compare its latency and throughput with four MoE baselines using real-world and synthetic datasets. Under heavy load imbalance, HarMoEny increases throughput by 37%-70% and reduces time-to-first-token by 34%-41%, compared to the next-best baseline. Moreover, our ablation study demonstrates that HarMoEny's scheduling policy reduces the GPU idling time by up to 84% compared to the baseline policies.

Figures

Figures reproduced from arXiv: 2506.12417 by the authors.

Figure 1
Figure 1. The ECDF of token placement across all 128 experts [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The ECDF of token placement across GPUs in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. A transformer block (left) and a MoE block, contain￾ing the MoE logic (right). MoE-based models replace the FFN with a router and multiple experts implemented as FFNs. costs while offering diminishing returns [8, 16]. After processing the expert FFN, the token’s output is combined, normalized, and passed to the next layer. Expert parallelism (EP). In multi-GPU scenarios, transformers typically use three types of par… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Token scattering (step 1) and gathering (step 2) when [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: (a) Fixed expert placement causes long waiting due to load imbalance. (b) Throughput fluctuates with static place￾ment, given the nature of requests. from the minibatches to experts. Since the tokens and their assigned experts can potentially be on different GPUs, an a…
Figure 6
Figure 6. Figure 6: An example of the token rebalancing process by [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Throughput (↑ is better) for different systems and different skews (left) and datasets (right) when using the Switch128 (top) and Qwen models (bottom). 0.0 0.5 0.9 100 101 Mean TTFT (log scale) [ms] Switch128 Deepspeed Fast MoE Faster MoE Exflow HarMoEny random bookcor…
Figure 8
Figure 8. Figure 8: Mean TTFT (↓ is better) for different systems and different skews (left) and datasets (right) when using the Switch128 (top) and Qwen (bottom) models. Similar to the baselines, there is a deterioration in the throughput of HarMoEny as the skew increases. This happens b…
Figure 9
Figure 9. Figure 9: Throughput with 0%–50% skew randomly chosen [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: The throughput of HarMoEny and baselines across iterations (top) and skew in expert popularity (bot￾tom), for the Switch128 (left) and Qwen (right) models. Har￾MoEny maintains consistent throughput while skew varies from 0% to 95% per batch. perform equal computation.…
Figure 13
Figure 13. Figure 13: Mean TTFT (↓ is better) for different token load policies and different skews (left) and datasets (right) when using the Switch128 (top) and Qwen models (bottom). affinity between tokens and experts, formulating a placement opti￾mization problem to minimize the commun…
Figure 11
Figure 11. Figure 11: Time breakdown of HarMoEny and baselines for the Switch128 (left) and Qwen (right) models. 0.0 0.5 0.9 0 100 200 Throughput [tokens/s] Switch128 Round-robin Exflow Even split HarMoEny random bookcorpus wmt19 wikitext 0 100 200 0.0 0.5 0.9 0 20 40 Skew Qwen random book…
Figure 12
Figure 12. Figure 12: Throughput (↑ is better) for different token load policies and different skews (left) and datasets (right) when using the Switch128 (top) and Qwen models (bottom). is employed by DeepSpeed, FastMoE, and FasterMoE; (2) The ExFlow policy utilizes an integer programming-…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 30 canonical work pages

  1. [1]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Amazon EC2 update – inf1 instances with AWS inferentia chips for high performance cost-effective inferencing, 2019

    Jeff Barr. Amazon EC2 update – inf1 instances with AWS inferentia chips for high performance cost-effective inferencing, 2019. Accessed: January 2025

    work page 2019

  3. [3]

    A neural probabilistic language model

    Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. In NeurIPS, 2000

    work page 2000

  4. [4]

    Datacenter power and energy management: past, present, and future

    Ricardo Bianchini, Christian Belady, and Anand Sivasubramaniam. Datacenter power and energy management: past, present, and future. IEEE Micro, 2024

    work page 2024

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020

    2020

  6. [6]

    Large scale distributed deep networks

    Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc Le, and Andrew Ng. Large scale distributed deep networks. In NeurIPS, volume 25, 2012

    work page 2012

  7. [7]

    Quasar: Resource-efficient and qos-aware cluster management

    Christina Delimitrou and Christos Kozyrakis. Quasar: Resource-efficient and qos-aware cluster management. ACM Sigplan Notices, 49(4), 2014

    work page 2014

  8. [8]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120), 2022

    work page 2022

Show all 41 references
  1. [9]

    Acl 2019 fourth conference on machine translation (wmt19), shared task: Machine translation of news

    Wikimedia Foundation. Acl 2019 fourth conference on machine translation (wmt19), shared task: Machine translation of news

    2019

  2. [10]

    Megablocks: Efficient sparse training with mixture-of-experts

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts. In MLSys, 2023

    2023

  3. [11]

    Character- based NMT with transformer

    Rohit Gupta, Laurent Besacier, Marc Dymetman, and Matthias Gallé. Character- based NMT with transformer. arXiv:1911.04997, 2019

    1911 arXiv

  4. [12]

    Fastmoe: A fast mixture-of-expert training system

    Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. Fastmoe: A fast mixture-of-expert training system. arXiv:2103.13262, 2021

    2021 arXiv

  5. [13]

    Fastermoe: Modeling and optimizing training of large-scale dynamic pre-trained models

    Jiaao He, Jidong Zhai, Tiago Antunes, Haojie Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. Fastermoe: Modeling and optimizing training of large-scale dynamic pre-trained models. In PPoPP, 2022

    2022

  6. [14]

    Tutel: Adaptive mixture-of- experts at scale

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive mixture-of- experts at scale. MLSys, 2023

    2023

  7. [15]

    Adaptive mixtures of local experts

    Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1), 1991

    1991

  8. [16]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  9. [17]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.08361, 2020

    2001 arXiv

  10. [18]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  11. [19]

    Subword regularization: Improving neural network translation models with multiple subword candidates

    Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In ACL, 2018

    2018

  12. [20]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In SOSP, 2023

    2023

  13. [21]

    AWS to offer nvidia’s t4 GPUs for AI inferencing, 2019

    George Leopold. AWS to offer nvidia’s t4 GPUs for AI inferencing, 2019. Accessed: January 2025

    2019

  14. [22]

    Gshard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In ICLR, 2021

    2021

  15. [23]

    Accelerating distributed MoE training and inference with lina

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. Accelerating distributed MoE training and inference with lina. In USENIX ATC, 2023

    2023

  16. [24]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv:2412.19437, 2024

    2024 arXiv

  17. [25]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  18. [26]

    Deepspeed-mii: Mii makes low-latency and high-throughput inference possible, powered by deepspeed

    Microsoft. Deepspeed-mii: Mii makes low-latency and high-throughput inference possible, powered by deepspeed. https://github.com/microsoft/DeepSpeed-MII,

  19. [27]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. InNeurIPS, 2019

    2019

  20. [28]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140), 2020

    2020

  21. [29]

    DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation ai scale

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation ai scale. In ICML, 2022

    2022

  22. [30]

    Outrageously large neural networks: The sparsely- gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Ge- offrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of-experts layer. In ICLR, 2017

    2017

  23. [31]

    Megatron-LM: Training multi-billion parameter language models using model parallelism, 2020

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism, 2020

    2020

  24. [32]

    Borg: the next generation

    Muhammad Tirmazi, Adam Barker, Nan Deng, Md E Haque, Zhijing Gene Qin, Steven Hand, Mor Harchol-Balter, and John Wilkes. Borg: the next generation. In EuroSys, 2020

    2020

  25. [33]

    Attention is all you need

    A Vaswani et al. Attention is all you need. NeurIPS, 2017

    2017

  26. [34]

    Prophet: Fine-grained load balancing for parallel training of large-scale moe models

    Wei Wang, Zhiquan Lai, Shengwei Li, Weijie Liu, Keshi Ge, Yujie Liu, Ao Shen, and Dongsheng Li. Prophet: Fine-grained load balancing for parallel training of large-scale moe models. In IEEE International Conference on Cluster Computing (CLUSTER), 2023

    2023

  27. [35]

    Resource-efficient algorithms and systems of foundation models: A survey

    Mengwei Xu, Dongqi Cai, Wangsong Yin, Shangguang Wang, Xin Jin, and Xu- anzhe Liu. Resource-efficient algorithms and systems of foundation models: A survey. ACM Computing Surveys, 2024

    2024

  28. [36]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv:2412.15115, 2024

    2024 arXiv

  29. [37]

    Harnessing the power of LLMs in practice: A survey on chatgpt and beyond

    Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. Harnessing the power of LLMs in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, 18(6), 2024

    2024

  30. [38]

    Exploiting inter-layer expert affinity for accelerating mixture-of- experts model inference

    Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, and Dhabaleswar K DK Panda. Exploiting inter-layer expert affinity for accelerating mixture-of- experts model inference. In IEEE IPDPS, 2024

    2024

  31. [39]

    SmartMoE: Efficiently training Sparsely-Activated models through combining offline and online parallelization

    Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, and Jidong Zhai. SmartMoE: Efficiently training Sparsely-Activated models through combining offline and online parallelization. In USENIX ATC, 2023

    2023

  32. [40]

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books

    Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urta- sun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv:1506.06724, 2015

    2015 arXiv

  33. [2022]

    Accessed: 2025-01-13

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.