Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read gLLM claims that independently throttling prefill and decode token counts across pipeline micro-batches removes most inter-batch pipeline bubbles, giving 11% to 398% higher maximum throughput than state-of-the-art pipeline or tensor…

desk verdict Decoupled prefill/decode token throttling is a real contribution, but the paper never validates that token counts equal compute times, and the abstract oversells the latency claim. read the letter →

arxiv 2504.14775 v2 pith:K3LOUZZR submitted 2025-04-21 cs.DC

Tianyu Guo , Xianwei Zhang , Jiangsu Du , Zhiguang Chen , Nong Xiao , Yutong Lu This is my paper · ORCID
classification cs.DC
keywords pipelineparallelismLLMservingtokenthrottlingbubblesprefill-decodeschedulingKVcachemanagementdistributedinferencethroughputoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models deployed across GPU nodes often use pipeline parallelism because it moves less data between GPUs than tensor parallelism, but the pipeline stalls whenever consecutive micro-batches take different amounts of time. gLLM argues that in LLM serving these stalls come mostly from token-count imbalance: prefill batches burst with many tokens while decode batches trickle, so pipeline stages sit idle. Its Token Throttling mechanism regulates prefill and decode tokens independently, using pending token counts and KV-cache pressure for prefill and pipeline depth for decode, to make micro-batches roughly equal. The paper reports that this raises maximum throughput by 11% to 398% over current pipeline or tensor parallelism systems while lowering latency. If the claim holds, serving systems can get more requests through the same GPUs without changing pipeline topology or paying tensor parallelism's communication cost.

What carries the argument

Token Throttling is the load-bearing mechanism: a scheduling policy that regulates prefill and decode token counts independently rather than under one shared budget. For prefill, a throttle formula combines the number of tokens waiting, a target number of iterations over which to drain them, and the KV-cache idle rate, with a threshold that halts prefill when memory headroom is too low; for decode, the throttle sets each micro-batch to the total running decode tokens divided by pipeline depth. A second component, the asynchronous runtime, gives each pipeline stage its own worker process, has a driver worker broadcast metadata ahead of activations, and keeps the frontend separate from backend compute. Together they aim to make every micro-batch take nearly the same wall-clock time, which is what keeps all pipeline stages busy.

What would settle it

Instrument gLLM to record the start and end wall-clock time of every micro-batch at every pipeline stage while serving a fixed workload, and compare micro-batches that received equal token counts: if their durations differ by more than a small margin under attention or KV-cache pressure, token count is not the right balancing unit. A sharper version is to run a variant that throttles by measured compute time rather than token count; if that variant beats gLLM, the token-count premise fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that decoupled, globally informed token throttling turns an unbalanced LLM serving pipeline into a near-balanced one. For decode, it schedules $\#D = \#R_D / \#PP_{depth}$ tokens per micro-batch, spreading running decode tokens as evenly as pipeline depth allows. For prefill, it sets batch size from waiting-token count and KV-cache idle rate, clamps it between minimum and maximum bounds, and pauses prefill when idle KV-cache space falls below a threshold. The paper argues that these two throttles, plus an asynchronous runtime that separates scheduling metadata from activation transfer, eliminate the inter-batch bubbles left by fixed token-budget hybrid scheduling.

Load-bearing premise

The balancing rules assume that a micro-batch's compute time tracks its token count for both prefill and decode; if attention cost or memory bandwidth makes per-token time vary by sequence length or cache state, equalizing token counts will not equalize stage durations and the bubble reduction will not materialize.

Editorial extensions

If this is right

  • Serving clusters can absorb 2-6x higher request arrival rates before time-to-first-token starts rising steeply, because balanced micro-batches keep the pipeline full.
  • Maximum throughput rises 11% to 398% over the tested pipeline and tensor parallelism systems, with the largest gains in cross-node deployments where communication cost is high.
  • At 80% SLO attainment, the cross-node 100B-model deployment sustains 79% higher request rate than the pipeline-parallel baseline, while SLO coverage improves by 64%.
  • The KV-cache-aware prefill throttle is the larger contributor: omitting it raises TTFT by 22%, TPOT by 91%, and end-to-end latency by 38% in the ablation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The token-count balancing logic treats all tokens as equal work; weighting by sequence length or attention span would matter for long-context workloads, where the paper's own Azure results show smaller gains.
  • The paper's statement that gLLM can back prefill-decode disaggregated clusters implies the throttles could be reused as a scheduling layer inside disaggregated architectures, not only in monolithic pipelines.
  • The fixed iteration-count hyperparameter could be made adaptive: the ablation shows it trades TTFT for TPOT, so an online controller reading queue depth and arrival rate could tune it continuously to meet SLOs.
  • A clean decomposition test would isolate runtime gains from scheduling gains by running the prior fixed-budget scheduler inside gLLM's runtime across all evaluated models, not just the single ablation configuration.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents gLLM, a distributed LLM serving system based on pipeline parallelism. The key idea, Token Throttling, independently regulates the number of prefill tokens (Eqs. 1-3) and decode tokens (Eq. 4) scheduled into each micro-batch, using the number of waiting prefill tokens, KV cache utilization, and the number of running decode tokens, with the goal of balancing computation across micro-batches and reducing pipeline bubbles. The system also introduces an asynchronous runtime with dedicated worker processes, non-blocking communication, and decoupled metadata/activation paths. Evaluation on Qwen2.5-14B/32B and Llama3.1-100B across 4-GPU intra-node and cross-node configurations, on ShareGPT and Azure workloads, reports 11-398% higher maximum throughput than vLLM and SGLang, generally lower latency, better SLO attainment, and includes ablations and hyperparameter sensitivity. The code is open-sourced.

Significance. If the results hold, gLLM provides a practical, open-source pipeline-parallel serving system with a simple and apparently effective scheduling heuristic. The evaluation is broad: two model families (plus a 100B variant), three GPU platforms, intra/cross-node deployments, two real-world workloads, ablations, and a sensitivity study. The ablation isolating the runtime contribution (gLLM w/CK) is a good practice. The main limitation is that the paper does not validate the core premise that token-count balancing implies compute-time balancing, so the causal explanation of the throughput gains remains incomplete; and the abstract's 'lower latency' claim is stronger than the evidence. With additional validation, the contribution would be a solid systems result.

major comments (3)
  1. [§3.1.3, Eq. (3)] The text states that when the KV cache idle rate KV_free is below KV_thresh, 'the system automatically suspends prefill token processing to prevent KV cache overflow.' However, Eq. (3) computes #P = max(min(#WP/#T, #MaxP·(KV_free−KV_thresh)/(1−KV_thresh)), #MinP). When KV_free < KV_thresh, the middle term is negative, so the expression evaluates to #MinP, not to zero. With the default #MinP = 32, prefill is not suspended at all. Please correct either the equation (e.g., introduce an explicit conditional that sets #P = 0) or the description of the safeguard.
  2. [§3.1–§3.2, Eqs. (1)–(4)] The central claim of the paper is that token-count balancing across micro-batches balances computation and thereby reduces pipeline bubbles. This premise is not validated. Prefill execution time is dominated by attention, whose cost scales with the sum of squared sequence lengths in the batch (FlashAttention reduces constant factors but not the quadratic-in-length dependence), and decode cost per token grows with the KV cache length of each request. Two batches with equal token counts can therefore have very different durations. The evaluation reports only end-to-end throughput and latency, never per-stage or per-micro-batch compute times, so the causal chain from Token Throttling to reduced bubbles is not established. Please add measurements of per-micro-batch execution times or GPU utilization across pipeline stages, or compare against a compute-time-aware balancing policy, or otherwise demonstrate that token count is a good proxy for compute time on the tested workloads.
  3. [Abstract and §4.2] The abstract claims gLLM achieves higher maximum throughput 'while simultaneously maintaining lower latency.' Section 4.2 explicitly reports that gLLM performs slightly worse than vLLM when serving Llama3.1-100B on the Azure dataset at request rate 4, and Section 4.4 notes that at low request rates gLLM's TTFT is marginally increased. The latency claim should be qualified to reflect the actual scope (e.g., 'in most tested scenarios').
minor comments (5)
  1. [Section 4, generally] No repeated runs, error bars, or statistical significance tests are reported; given the variability of GPU serving workloads, at least a few repeats for the main throughput/latency points would strengthen the claims.
  2. [Section 4.1, footnote 3] The paper justifies omitting prefill-decode disaggregated systems (Splitwise, DistServe) by saying gLLM could serve as their backend. A quantitative comparison, even a limited one, would make the 'state-of-the-art' claim more convincing.
  3. [Eqs. (1) and (3)] The formulas do not specify behavior when the number of waiting prefill tokens is smaller than #MinP (or #MinP·#T); presumably the scheduler uses all available tokens, but this should be stated explicitly.
  4. [Section 4.6, Figure 16] The normalization used for the sensitivity study is not described; please state the baseline to which each metric is normalized.
  5. [Text quality] There are several typos and grammatical errors, e.g., 'deep nerual networks' in Section 1, 'pipelilelism' in Section 5, 'evaulated' in the Table 1 caption, and a missing 'than' in the sentence '64% higher ... than vLLM' in Section 4.4. A careful proofread is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; throughput claims are measured against external baselines, with scheduling equations as design choices.

full rationale

gLLM's scheduling equations (Eqs. 1-4) define the throttling policy rather than derive the reported performance numbers. The throughput, latency, and SLO results in Section 4 are measured against vLLM, SGLang, and the gLLM ablations, so the central claim is an experimental outcome, not a formal consequence of the equations. The hyperparameters (#T=8, #MaxP=2048, #MinP=32, KV_thresh=0.05) are fixed settings with a sensitivity study, not fitted to the reported curves, so no fitted input is renamed as a prediction. The only same-author citation (Liger, ref [56]) appears in a related-work survey of tensor-parallelism optimizations and is not load-bearing; no uniqueness theorem is invoked. The unverified assumption that per-batch execution time tracks token count is a correctness or validity concern, not a circular one, because the paper does not define 'balanced computation' as 'equal token count' and then claim throughput follows analytically. No specific equation reduces to its own input, so the paper is not circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on four hand-picked hyperparameters plus three domain assumptions about compute-time proportionality, KV-cache consistency, and pipeline micro-batch concurrency. No new physical or conceptual entities are introduced.

free parameters (4)
  • #T = 8
    Number of iterations to spread pending prefill tokens; selected by the authors, not derived. Sensitivity study shows it trades TTFT against TPOT.
  • #MaxP = 2048
    Maximum batched prefill tokens per iteration; set to match baseline token budget, chosen by authors.
  • #MinP = 32
    Minimum batched prefill tokens to avoid empty batches.
  • KV_thresh = 0.05
    KV cache idle rate threshold below which prefill is suspended; authors' choice.
assumptions (4)
  • domain assumption Compute time per micro-batch is proportional to the number of tokens in the batch, for both prefill and decode.
    Underlies Eqs. (1)-(4) which balance token counts rather than measured compute times. Not validated with a compute-time model.
  • domain assumption KV cache usage is consistent across all pipeline stages because they share unified page tables.
    Stated in Section 3.1.4; needed for throttling based on a single KV_free value.
  • domain assumption The maximum number of concurrently active micro-batches equals the pipeline depth.
    Standard pipeline parallelism property used in Eq. (4).
  • domain assumption Poisson arrivals with fixed rates adequately model cloud serving workloads.
    Workload generation in Section 4.1 relies on this to synthesize request streams.

how reviews work

0 comments
Cite this review

Pith. "Pith review of gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling." pith.science (2026). https://pith.science/paper/K3LOUZZR

@misc{pith2026250414775,
  author       = {Pith},
  title        = {Pith review of: gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3LOUZZR}},
  note         = {Machine review of arXiv:2504.14775}
}
read the original abstract

Pipeline parallelism has emerged as a predominant approach for deploying large language models (LLMs) across distributed nodes, owing to its lower communication overhead compared to tensor parallelism. While demonstrating high throughput in request serving, pipeline parallelism often suffers from performance limitations caused by pipeline bubbles, which are primarily resulted from imbalanced computation delays across batches. Existing methods like Sarathi-Serve attempt to address this through hybrid scheduling of chunked prefill and decode tokens using a fixed token budget. However, such methods may experience significant fluctuations due to either insufficient prefill tokens or uneven distribution of decode tokens, ultimately leading to computational imbalance. To overcome these inefficiencies, we present gLLM, a globally balanced pipeline parallelism system incorporating Token Throttling to effectively mitigate the pipeline bubbles. Our Token Throttling mechanism is a fine-grained scheduling policy that independently regulates the quantities of prefill and decode tokens, thus enabling balanced computation by leveraging global information from the inference system. Specifically, for decode tokens, gLLM maintains near-consistent token count across processing batches. For prefill tokens, it dynamically adjusts batch sizes based on both total pending tokens and the memory utilization rates of key-value cache (KV cache). Furthermore, gLLM runtime adopts an asynchronous execution and message passing architecture specifically optimized for pipeline parallelism characteristics. Experimental evaluations with representative LLMs show that gLLM achieves significant performance improvements, delivering 11% to 398% higher maximum throughput compared to state-of-the-art pipeline or tensor parallelism systems, while simultaneously maintaining lower latency.

Figures

Figures reproduced from arXiv: 2504.14775 by the authors.

Figure 1
Figure 1. A comparison of the scheduled token counts of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of data parallelism, tensor parallelism [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Under-utilized GPU caused by unbalanced schedul [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Comparison between coupled and decoupled sched [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 7
Figure 7. Figure 7: Case study (the pipeline depth is 2) of prefill [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Case study (the pipeline depth is 4) of decode [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Architecture of gLLM runtime. new requests from the front-end, scheduling micro-batches, broad￾casting metadata for each schedule and streaming output back to the front-end. Meanwhile, all the workers focus on model execution, receiving activations from the previous st…
Figure 10
Figure 10. Figure 10: The latency and throughput comparison between [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Distribution of input and output length of the [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 13
Figure 13. Figure 13: The latency and throughput comparison between [PITH_FULL_IMAGE:figures/full_fig_p009_13.png]
Figure 15
Figure 15. Figure 15: An ablation study on the design choices of gLLM. [PITH_FULL_IMAGE:figures/full_fig_p009_15.png]
Figure 16
Figure 16. Figure 16: Normalized metric under different setting of hy [PITH_FULL_IMAGE:figures/full_fig_p010_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference

    cs.DC 2025-06 conditional novelty 6.0 of 10

    SiPipe claims 1.6x to 2.1x throughput gains over vLLM for pipeline-parallel LLM inference by using idle CPUs for sampling, double-buffered CUDA graphs, and structure-aware transmission.

  2. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

Reference graph

Works this paper leans on

61 extracted references · 51 canonical work pages · cited by 2 Pith papers

  1. [1]

    Determlr: Augmenting llm-based logical reasoning from indeterminacy to determinacy,

    H. Sun, W. Xu, W. Liu, J. Luan, B. Wang, S. Shang, J. Wen, and R. Yan, “Determlr: Augmenting llm-based logical reasoning from indeterminacy to determinacy, ” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 (L. Ku, A. Martins, and V. Srikumar...

    work page 2024

  2. [2]

    Boosting logical fallacy reasoning in llms via logical structure tree,

    Y. Lei and R. Huang, “Boosting logical fallacy reasoning in llms via logical structure tree, ” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024 (Y. Al-Onaizan, M. Bansal, and Y. Chen, eds.), pp. 13157–13173, Association for Computational Linguistics, 2024

    work page 2024

  3. [3]

    Verifiable, debuggable, and repairable commonsense logical reasoning via llm-based theory resolution,

    A. Toroghi, W. Guo, A. Pesaranghader, and S. Sanner, “Verifiable, debuggable, and repairable commonsense logical reasoning via llm-based theory resolution, ” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024 (Y. Al-Onaizan, M. Bansal, and Y. Chen, eds.), pp. 6634–665...

    work page 2024

  4. [4]

    Docmath-eval: Evaluating math reasoning capabilities of llms in understanding financial documents,

    Y. Zhao, Y. Long, H. Liu, R. Kamoi, L. Nan, L. Chen, Y. Liu, X. Tang, R. Zhang, and A. Cohan, “Docmath-eval: Evaluating math reasoning capabilities of llms in understanding financial documents, ” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 202...

    work page 2024

  5. [5]

    Gsm-plus: A comprehensive bench- mark for evaluating the robustness of llms as mathematical problem solvers,

    Q. Li, L. Cui, X. Zhao, L. Kong, and W. Bi, “Gsm-plus: A comprehensive bench- mark for evaluating the robustness of llms as mathematical problem solvers, ” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 (L. Ku, A. Martins, and V. Srikumar, ...

    work page 2024

  6. [6]

    Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark,

    H. Liu, Z. Zheng, Y. Qiao, H. Duan, Z. Fei, F. Zhou, W. Zhang, S. Zhang, D. Lin, and K. Chen, “Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark, ” inFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (L. Ku, A. Martins,...

    work page 2024

  7. [7]

    Mathge- nie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms,

    Z. Lu, A. Zhou, H. Ren, K. Wang, W. Shi, J. Pan, M. Zhan, and H. Li, “Mathge- nie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms, ” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 (L. Ku...

    work page 2024

  8. [8]

    Metacognitive capabilities of llms: An exploration in mathematical problem solving,

    A. Didolkar, A. Goyal, N. R. Ke, S. Guo, M. Valko, T. P. Lillicrap, D. J. Rezende, Y. Bengio, M. C. Mozer, and S. Arora, “Metacognitive capabilities of llms: An exploration in mathematical problem solving, ” inAdvances in Neural Informa- tion Processing Systems 38: Annual Conference on Neural Information Process- ing Systems 2024, NeurIPS 2024, Vancouver,...

    work page 2024

Show all 61 references
  1. [9]

    RL on incorrect synthetic data scales the efficiency of LLM math reasoning by eight-fold,

    A. Setlur, S. Garg, X. Geng, N. Garg, V. Smith, and A. Kumar, “RL on incorrect synthetic data scales the efficiency of LLM math reasoning by eight-fold, ” in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, ...

    2024

  2. [10]

    VELMA: verbalization embodiment of LLM agents for vision and language navigation in street view,

    R. Schumann, W. Zhu, W. Feng, T. Fu, S. Riezler, and W. Y. Wang, “VELMA: verbalization embodiment of LLM agents for vision and language navigation in street view, ” inThirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative App...

    2024

  3. [11]

    Chedbot: Designing a domain- specific conversational agent in a simulational learning environment using llms,

    A. Martin, C. Pande, H. F. Witschel, and J. Mathez, “Chedbot: Designing a domain- specific conversational agent in a simulational learning environment using llms, ” in Proceedings of the AAAI 2024 Spring Symposium Series, Stanford, CA, USA, March 25-27, 2024 (R. P. A. Petrick ...

    2024

  4. [12]

    Retrieval-augmented generation and LLM agents for biomimicry design solutions,

    C. Toukmaji and A. Tee, “Retrieval-augmented generation and LLM agents for biomimicry design solutions, ” inProceedings of the AAAI 2024 Spring Symposium Series, Stanford, CA, USA, March 25-27, 2024 (R. P. A. Petrick and C. W. Geib, eds.), pp. 273–278, AAAI Press, 2024

    2024

  5. [13]

    Mobile-bench: An evaluation benchmark for llm-based mobile agents,

    S. Deng, W. Xu, H. Sun, W. Liu, T. Tan, J. Liu, A. Li, J. Luan, B. Wang, R. Yan, and S. Shang, “Mobile-bench: An evaluation benchmark for llm-based mobile agents, ” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2024

  6. [14]

    Opex: A component-wise analysis of llm-centric agents in embodied instruction following,

    H. Shi, Z. Sun, X. Yuan, M. Côté, and B. Liu, “Opex: A component-wise analysis of llm-centric agents in embodied instruction following, ” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thail...

    2024

  7. [15]

    Psychogat: A novel psychological measurement paradigm through interactive fiction games with LLM agents,

    Q. Yang, Z. Wang, H. Chen, S. Wang, Y. Pu, X. Gao, W. Huang, S. Song, and G. Huang, “Psychogat: A novel psychological measurement paradigm through interactive fiction games with LLM agents, ” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    2024

  8. [16]

    Deepseek-v3 technical report,

    DeepSeek-AI, A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, ...

    2024 arXiv

  9. [17]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sra- vankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Rozière, B. Biron, B. ...

    2024 arXiv

  10. [18]

    The falcon series of open language models,

    E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Deb- bah, É. Goffinet, D. Hesslow, J. Launay, Q. Malartic, D. Mazzotta, B. Noune, B. Pannier, and G. Penedo, “The falcon series of open language models, ”CoRR, vol. abs/2311.16867, 2023

    2023 arXiv

  11. [19]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report, ” CoRR, vol. abs/2303.08774, 2023

    2023 arXiv

  12. [20]

    Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context,

    M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. P. Lillicrap, J. Alayrac, R. Sori- cut, A. Lazaridou, O. Firat, J. Schrittwieser, I. Antonoglou, R. Anil, S. Borgeaud, 11 Tianyu Guo, Xianwei Zhang#, Jiangsu Du, Zhiguang Chen, Nong Xiao, and Yutong Lu A. M. Dai, K. Millican,...

    2024 arXiv

  13. [21]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, ”J. Mach. Learn. Res., vol. 23, pp. 120:1–120:39, 2022

    2022

  14. [22]

    Dynamollm: De- signing LLM inference clusters for performance and energy efficiency,

    J. Stojkovic, C. Zhang, Í. Goiri, J. Torrellas, and E. Choukse, “Dynamollm: De- signing LLM inference clusters for performance and energy efficiency, ”CoRR, vol. abs/2408.00741, 2024

    2024

  15. [23]

    Helix: Distributed serving of large language models via max-flow on heterogeneous gpus,

    Y. Mei, Y. Zhuang, X. Miao, J. Yang, Z. Jia, and R. Vinayak, “Helix: Distributed serving of large language models via max-flow on heterogeneous gpus, ”CoRR, vol. abs/2406.01566, 2024

    2024 arXiv

  16. [24]

    Distributed inference and fine-tuning of large language models over the internet,

    A. Borzunov, M. Ryabinin, A. Chumachenko, D. Baranchuk, T. Dettmers, Y. Belkada, P. Samygin, and C. A. Raffel, “Distributed inference and fine-tuning of large language models over the internet, ” inAdvances in Neural Information Processing Systems 36: Annual Conference on Neur...

    2023

  17. [25]

    Moirai: Towards op- timal placement for distributed inference on heterogeneous devices,

    B. Zhang, H. Zhu, F. Gao, Z. Yang, and X. S. Wang, “Moirai: Towards op- timal placement for distributed inference on heterogeneous devices, ” CoRR, vol. abs/2312.04025, 2023

    2023 arXiv

  18. [26]

    Pipedream: Fast and efficient pipeline parallel DNN training,

    A. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, and P. B. Gibbons, “Pipedream: Fast and efficient pipeline parallel DNN training, ”CoRR, vol. abs/1806.03377, 2018

    2018 arXiv

  19. [27]

    Memory- efficient pipeline-parallel DNN training,

    D. Narayanan, A. Phanishayee, K. Shi, X. Chen, and M. Zaharia, “Memory- efficient pipeline-parallel DNN training, ” in Proceedings of the 38th Interna- tional Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event (M. Meila and T. Zhang, eds.), vol. 139 ofPr...

    2021

  20. [28]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism,

    Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. X. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen, “Gpipe: Efficient training of giant neural networks using pipeline parallelism, ” inAdvances in Neural Information Processing Systems 32: Annual Conference on Neural In...

    2019

  21. [29]

    Adapipe: Optimizing pipeline parallelism with adaptive recomputation and partitioning,

    Z. Sun, H. Cao, Y. Wang, G. Feng, S. Chen, H. Wang, and W. Chen, “Adapipe: Optimizing pipeline parallelism with adaptive recomputation and partitioning, ” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating S...

    2024

  22. [30]

    Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency,

    Z. Liu, S. Cheng, H. Zhou, and Y. You, “Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency, ” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2023, Denver, CO, USA, N...

    2023

  23. [31]

    Weipipe: Weight pipeline parallelism for communication-effective long-context large model training,

    J. Lin, Z. Liu, Y. You, J. Wang, W. Zhang, and R. Zhao, “Weipipe: Weight pipeline parallelism for communication-effective long-context large model training, ” in Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoPP 2025...

    2025

  24. [32]

    Mario: Near zero-cost activation checkpointing in pipeline parallelism,

    W. Liu, M. Li, G. Tan, and W. Jia, “Mario: Near zero-cost activation checkpointing in pipeline parallelism, ” inProceedings of the 30th ACM SIGPLAN Annual Sympo- sium on Principles and Practice of Parallel Programming, PPoPP 2025, Las Vegas, NV, USA, March 1-5, 2025 , pp. 197–...

    2025

  25. [33]

    Zero bubble (almost) pipeline parallelism,

    P. Qi, X. Wan, G. Huang, and M. Lin, “Zero bubble (almost) pipeline parallelism, ” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 , OpenReview.net, 2024

    2024

  26. [34]

    Bpipe: Memory-balanced pipeline paral- lelism for training large language models,

    T. Kim, H. Kim, G. Yu, and B. Chun, “Bpipe: Memory-balanced pipeline paral- lelism for training large language models, ” inInternational Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA (A. Krause, E. Brun- skill, K. Cho, B. Engelhardt, S. Saba...

    2023

  27. [35]

    Efficient large-scale language model training on GPU clusters using megatron-lm,

    D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on GPU clusters using megatron-lm, ” inInternational Conferenc...

    2021

  28. [36]

    Taming throughput-latency tradeoff in LLM inference with sarathi-serve,

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in LLM inference with sarathi-serve, ” in18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, ...

    2024

  29. [37]

    Splitwise: Efficient generative LLM inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative LLM inference using phase splitting, ” in 51st ACM/IEEE Annual International Symposium on Computer Architecture, ISCA 2024, Buenos Aires, Argentina, June 29 - July ...

    2024

  30. [38]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, “Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving, ” in18th USENIX Symposium on Operating Systems Design and Implemen- tation, OSDI 2024, Santa Clara, CA, U...

    2024

  31. [39]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need, ” inAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Sys- tems 2017, December 4-9, 2017, Lo...

    2017

  32. [40]

    Orca: A distributed serving system for transformer-based generative models,

    G. Yu, J. S. Jeong, G. Kim, S. Kim, and B. Chun, “Orca: A distributed serving system for transformer-based generative models, ” in16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 2022 (M. K. Aguilera and H. Weather...

    2022

  33. [41]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention, ” inProceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany,...

    2023

  34. [42]

    Faster Transformer

    “Faster Transformer. ” https://github.com/NVIDIA/FasterTransformer

  35. [43]

    Flashattention: Fast and memory- efficient exact attention with io-awareness,

    T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory- efficient exact attention with io-awareness, ” inAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Sys- tems 2022, NeurIPS 2022, New Orlean...

    2022

  36. [44]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning, ” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 , OpenReview.net, 2024

    2024

  37. [45]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision,

    J. Shah, G. Bikshandi, Y. Zhang, V. Thakkar, P. Ramani, and T. Dao, “Flashattention-3: Fast and accurate attention with asynchrony and low-precision, ” in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, Neu...

    2024

  38. [46]

    Sglang: Efficient execution of struc- tured language model programs,

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. W. Barrett, and Y. Sheng, “Sglang: Efficient execution of struc- tured language model programs, ” inAdvances in Neural Information Processing Systems 38: Annual Conference...

    2024

  39. [47]

    Qwen2.5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y....

    2024 arXiv

  40. [48]

    Chatglm: A family of large language models from GLM-130B to GLM-4 all tools,

    A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Rojas, G. Feng, H. Zhao, H. Lai, H. Yu, H. Wang, J. Sun, J. Zhang, J. Cheng, J. Gui, J. Tang, J. Zhang, J. Li, L. Zhao, L. Wu, L. Zhong, M. Liu, M. Huang, P. Zhang, Q. Zheng, R. Lu, S. Duan, S. Zhang, S. Cao, S. Yang, W. L. Tam, W....

    2024 arXiv

  41. [49]

    Mmlu- pro: A more robust and challenging multi-task language understanding bench- mark,

    Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen, “Mmlu- pro: A more robust and challenging multi-task language understanding bench- mark, ” inAdvances in Neural Information ...

    2024

  42. [50]

    ShareGPT

    “ShareGPT. ” https://sharegpt.com/

  43. [51]

    Flexgen: High-throughput generative inference of large language models with a single GPU,

    Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single GPU, ” in International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolul...

    2023

  44. [52]

    Infinigen: Efficient generative inference of large language models with dynamic KV cache management,

    W. Lee, J. Lee, J. Seo, and J. Sim, “Infinigen: Efficient generative inference of large language models with dynamic KV cache management, ” in18th USENIX 12 gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling Symposium on Operati...

    2024

  45. [53]

    Breaking the computation and communication abstraction barrier in distributed machine learning workloads,

    A. Jangda, J. Huang, G. Liu, A. H. N. Sabet, S. Maleki, Y. Miao, M. Musuvathi, T. Mytkowicz, and O. Saarikivi, “Breaking the computation and communication abstraction barrier in distributed machine learning workloads, ” inASPLOS ’22: 27th ACM International Conference on Archit...

    2022

  46. [54]

    Overlap communication with dependent computation via decomposition in large deep learning models,

    S. Wang, J. Wei, A. Sabne, A. Davis, B. Ilbeyi, B. Hechtman, D. Chen, K. S. Murthy, M. Maggioni, Q. Zhang, S. Kumar, T. Guo, Y. Xu, and Z. Zhou, “Overlap communication with dependent computation via decomposition in large deep learning models, ” inProceedings of the 28th ACM I...

    2023

  47. [55]

    Centauri: En- abling efficient scheduling for communication-computation overlap in large model training via communication partitioning,

    C. Chen, X. Li, Q. Zhu, J. Duan, P. Sun, X. Zhang, and C. Yang, “Centauri: En- abling efficient scheduling for communication-computation overlap in large model training via communication partitioning, ” inProceedings of the 29th ACM International Conference on Architectural Su...

    2024

  48. [56]

    Liger: Interleaving intra- and inter-operator parallelism for distributed large model inference,

    J. Du, J. Wei, J. Jiang, S. Cheng, D. Huang, Z. Chen, and Y. Lu, “Liger: Interleaving intra- and inter-operator parallelism for distributed large model inference, ” in Proceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoP...

    2024

  49. [57]

    Alpa: Automating inter- and intra- operator parallelism for distributed deep learning,

    L. Zheng, Z. Li, H. Zhang, Y. Zhuang, Z. Chen, Y. Huang, Y. Wang, Y. Xu, D. Zhuo, E. P. Xing, J. E. Gonzalez, and I. Stoica, “Alpa: Automating inter- and intra- operator parallelism for distributed deep learning, ” in16th USENIX Symposium on Operating Systems Design and Implem...

    2022

  50. [58]

    HAP: SPMD DNN training on heterogeneous GPU clusters with automated program synthesis,

    S. Zhang, L. Diao, C. Wu, Z. Cao, S. Wang, and W. Lin, “HAP: SPMD DNN training on heterogeneous GPU clusters with automated program synthesis, ” in Proceedings of the Nineteenth European Conference on Computer Systems, EuroSys 2024, Athens, Greece, April 22-25, 2024 , pp. 524–...

    2024

  51. [59]

    Metis: Fast automatic distributed training on heterogeneous gpus,

    T. Um, B. Oh, M. Kang, W. Lee, G. Kim, D. Kim, Y. Kim, M. Muzzammil, and M. Jeon, “Metis: Fast automatic distributed training on heterogeneous gpus, ” in Proceedings of the 2024 USENIX Annual Technical Conference, USENIX ATC 2024, Santa Clara, CA, USA, July 10-12, 2024 (S. Bag...

    2024

  52. [60]

    Whale: Efficient giant model training over heterogeneous gpus,

    X. Jia, L. Jiang, A. Wang, W. Xiao, Z. Shi, J. Zhang, X. Li, L. Chen, Y. Li, Z. Zheng, X. Liu, and W. Lin, “Whale: Efficient giant model training over heterogeneous gpus, ” inProceedings of the 2022 USENIX Annual Technical Conference, USENIX ATC 2022, Carlsbad, CA, USA, July 1...

    2022

  53. [61]

    SWARM parallelism: Training large models can be surprisingly communication-efficient,

    M. Ryabinin, T. Dettmers, M. Diskin, and A. Borzunov, “SWARM parallelism: Training large models can be surprisingly communication-efficient, ” inInterna- tional Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA (A. Krause, E. Brunskill, K. Cho, ...

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.