Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Characterizing Communication Patterns in Distributed Large Language Model Inference

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distributed LLM inference communication can be predicted by closed-form volume formulas that reveal when tensor, pipeline, or hybrid parallelism best meets latency targets.

desk verdict The communication-volume formulas are solid and citable; the unexplained TP=4/PP=2 SLO outlier undermines the hybrid-parallelism advice and needs hard scrutiny. read the letter →

arxiv 2507.14392 v1 pith:CMVJHJ2P submitted 2025-07-18 cs.DC

Lang Xu , Kaushik Kandadi Suresh , Quentin Anthony , Nawras Alnaasan , Dhabaleswar K. Panda This is my paper
classification cs.DC
keywords distributedLLMinferencecommunicationcharacterizationtensorparallelismpipelinehybridvolumemodelservicelevelobjectivesv
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the communication cost of running a large language model across many GPUs can be written down in closed form, not just measured. It derives equations that predict the total bytes moved for tensor, pipeline, and hybrid parallelism from model dimensions, sequence lengths, and parallelism degree, and it validates those equations against profiler traces from the vLLM serving engine. The validated formulas support a practical ranking: tensor parallelism moves the most data but gives the fastest responses for short prompts, pipeline parallelism moves the least data but adds latency, and hybrid layouts need balanced tuning to avoid severe slowdowns. This matters because it gives deployment engineers a quantitative way to choose a parallelism strategy before committing hardware.

What carries the argument

The load-bearing objects are Equations (1)–(7), closed-form formulas for communication volume written in terms of architecture constants (hidden size $h$, layer count $L$, vocabulary size $v$, bytes per element $b$), request lengths ($S_p$, $S_d$), and parallelism degrees ($t$, $p$). The formulas encode the NCCL collective correction factors—$2(t-1)/t$ for Allreduce and $(t-1)/t$ for Allgather—so that the predicted volumes count real network payload rather than naive message sizes. The same derivation produces operation counts (such as the $(2L+1)$ Allreduces per token in TP and the point-to-point transfers in PP), which the paper checks against PyTorch profiler traces of each configuration.

What would settle it

Profile the same Llama-3.1-8B workload on vLLM with production defaults (custom AllReduce enabled, torch.compile on) and a small batch of concurrent requests, then compare the measured Allreduce counts, communication volumes, and TPOT values against Equations (1)–(3) and the claim that decode dominates; if the custom AllReduce eliminates most of the Allreduce cost or batching changes the volume profile, the paper's central predictive claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the network traffic of distributed transformer inference obeys explicit, architecture-dependent counting rules. For pure tensor parallelism of degree $t$, the total bytes moved for a request with prefill length $S_p$ and decode length $S_d$ is $V_{tp} = (2L+1)(S_p+S_d-1)hb \cdot 2(t-1)/t + S_d (v/t)b$, where the first term counts the $(2L+1)$ Allreduces per token (two per transformer layer plus one for the embedding) with the collective's volume correction factor, and the second counts the vocabulary Gather. Pure pipeline parallelism with $p$ stages moves $V_{pp} = (p-1)\cdot 2(S_p+S_d-1)hb$ through point-to-point links, and hybrid parallelism adds separate Allgather and Gather terms. Profiling of Llama-3.2-3B, Llama-3.1-8B, and Llama-2-13B in vLLM shows operation counts and message sizes matching the formulas, and reveals that the decode stage generates roughly two orders of magnitude more communication operations than prefill (e.g., 8255 vs 65 Allreduces at 128 tokens). The SLO measurements then tie the formulas to user experience: intra-node tensor parallelism improves all metrics up to TP=4, inter-node TP=8 collapses time-per-output-token from about 0.86 ms to 11.56 ms, pipeline parallelism degrades latency as depth grows, and an unbalanced hybrid such as TP=4 PP=2 reaches 15.15 s end-to-end on Llama-2-13B.

Load-bearing premise

The results assume single-request inference with vLLM's custom AllReduce and torch.compile disabled, so the reported volumes and SLO rankings may not hold in production systems that use batching or those optimizations.

Editorial extensions

If this is right

  • Deployment engineers can compute expected per-request communication volume from model-card numbers (hidden size, layers) and expected sequence lengths, before renting hardware.
  • Because decode, not prefill, drives communication, optimizing high-frequency Allreduce operations in the decode loop should yield the largest serving-speedup per unit of engineering effort.
  • Pure tensor parallelism should be reserved for short-sequence interactive serving on a single node with high-bandwidth interconnects; splitting TP across nodes is likely to hurt time-per-output-token.
  • Pure pipeline parallelism, despite higher latency, is the safe choice for long-generation or bandwidth-limited deployments because its traffic is minimal and scales cleanly.
  • Balanced hybrids such as TP=2 PP=4 are viable middle grounds, while unbalanced ratios such as TP=4 PP=2 can produce order-of-magnitude latency regressions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Adding batch size to the formulas as a multiplier on the $S_d$ term would make them testable against production continuous-batching servers, and would likely show that decode dominance persists but the SLO ranking between TP and PP narrows.
  • Editorial inference: The correction-factor methodology transfers naturally to Mixture-of-Experts and sequence parallelism, where analogous AlltoAll and reduced-scatter terms could yield a unified model for next-generation serving stacks.
  • Editorial inference: The inter-node TP collapse suggests a concrete optimization: place TP groups strictly within a node and use PP or expert parallelism across nodes, then re-run the SLO measurements to check whether the paper's recommended TP=8 for short prompts survives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies inter-GPU communication in distributed LLM inference serving (vLLM) for tensor, pipeline, and hybrid parallelism. It derives closed-form expressions (Eqs. 1-7) for communication volume as a function of model architecture (L, h, v) and sequence lengths (Sp, Sd), with no fitted parameters. Profiling experiments on Llama-3.2-3B, Llama-3.1-8B, and Llama-2-13B under TP, PP, and TP×PP configurations are used to validate predicted operation counts and message shapes. The paper also reports SLO metrics (end-to-end latency, TTFT, TPOT) for different parallelism degrees and configurations across one and two nodes, and concludes that TP gives the best latency for short sequences, PP minimizes communication but increases latency, and hybrid configurations require careful tuning.

Significance. If the validation is taken at face value, the paper provides a useful, parameter-free analytical basis for estimating communication volumes in dense transformer inference, and the exact matching of operation counts and shapes across multiple model sizes is a concrete strength. However, the SLO-based recommendations are weakened by the unexplained hybrid outlier and lack of reported variance; the volume-model validation is also indirect. With additional diagnostics and statistical rigor, the paper could be a valuable reference for practitioners selecting parallelism configurations.

major comments (4)
  1. [Section V.C, Figure 10] The reported 6.4x end-to-end slowdown of TP=4/PP=2 (15.15s) versus TP=8/PP=1 (2.37s) on Llama-2-13B is unexplained by the paper's own model: Eqs. (3)-(7) with L=40, h=5120, v=32000, t=4, p=2, Sp=Sd=128 give roughly 165 MB of communication, less than the roughly 371 MB predicted by Eq. (1) for TP=8/PP=1, and TP=4/PP=2 has only two pipeline stages. Since this single point is the empirical basis for the abstract's claim that 'combined approaches demand careful tuning,' the manuscript must either provide a mechanistic explanation (e.g., NCCL communicator setup, node placement, per-stage bubbles), repeat the run with variance reporting, or substantially soften the claim.
  2. [Sections IV.B and V.C] All SLO experiments appear to be single-request runs with no reported number of repetitions or error bars. Figures 8-10 compare point estimates (e.g., TP=2 vs TP=4 TTFT 150ms vs 90ms; PP=2 vs PP=4 E2E 0.69s vs 1.36s), and the hybrid outlier in Figure 10 could be a one-off artifact. Without variance information the paper's deployment recommendations, which rest on these rankings, are not statistically supported; please report mean plus/minus standard deviation over multiple runs and, ideally, the number of runs.
  3. [Section III.C, Eq. (4) and Table VI] Eq. (4) defines V_allreduce = (2L/p) * (Sp+Sd-1) * h * b * 2(t-1)/t, but the text immediately states that the initial pipeline rank incurs an additional embedding-layer Allreduce, and Table VI's count of 33 for L=32, p=2 includes that extra operation. The formula as written omits the embedding term; please either include it explicitly in Eq. (4) or introduce a separate term, so the equation matches the validation data and the code path described.
  4. [Section V.A and Figures 4-5] The empirical validation of the volume formulas is indirect. Tables III-VI report per-operation counts and tensor shapes, and Figures 4-5 plot 'total message size' computed as count times shape, not measured network bytes after the ring-allreduce correction factors (e.g., 2(t-1)/t). Because the paper's headline contribution is predictive communication volume, the validation should report actual profiler-measured sent and received bytes per collective, or explicitly state that the validation covers operation counts and shapes only and that the correction factors are taken from Ref. [16].
minor comments (4)
  1. [Section V.C] The phrase 'Time-Ter-Output-Token' should be 'Time-Per-Output-Token'.
  2. [Section IV.A] The manuscript disables vLLM's custom AllReduce and torch.compile to isolate NCCL-level communication; the conclusions should restate this as an explicit limitation so that the reported SLO numbers are not interpreted as default vLLM production behavior.
  3. [Section IV.B] The text says rank-0 profiles are excluded from analysis; please clarify whether all collective counts are collected from a single non-zero rank and how symmetric operations such as Send/Recv are aggregated (e.g., whether the counts in Table V are per rank or summed over ranks).
  4. [Section III.C] The embedding-layer Allreduce contribution is described in prose but not in the equations; consider adding a short derivation or a note that Eq. (4) applies to non-initial ranks, for self-containedness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the communication-volume formulas are deterministic functions of architecture constants and sequence lengths, with no fitted parameters; validation is a bookkeeping identity and self-citations are not load-bearing.

full rationale

The paper's central analytical models, Equations (1)-(7), take only architecture constants (L, h, v, t, p) and sequence lengths (Sp, Sd) as inputs; no parameter is fitted to profiling data. The empirical validation in Section V.A confirms that profiler counts equal deterministic structural counts, for example 65 = 2L+1 Allreduce operations for L=32 and 8255 = 65 x (Sd - 1) decode operations, which is a bookkeeping identity rather than an independent risky prediction, but it is not circular because the model inputs do not include the measured outputs. The correction factor 2(t-1)/t is an NCCL semantic convention cited to external documentation [16], and its use in both the formulas and the volume definition is a modeling choice, not a fitted parameter. The self-citations to Anthony et al. ([10], [22]) appear only as related work on training-communication characterization and are not load-bearing for the inference-specific claims. The paper explicitly notes its platform-specific validation in Future Work, which is a generalizability limitation rather than circularity. The unexplained TP=4/PP=2 SLO outlier in Section V.C (15.15s vs 2.37s for TP=8) is a correctness and robustness concern, not evidence of circular reasoning, since it is not derived from the analytical model. Overall, the derivation chain is self-contained and contains no reduction of a prediction to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The analytical models introduce no free parameters or new entities. They rest on the framework-specific operation counts, on the NCCL volume correction factors, and on the representative-ness of the profiled configuration. The models are deterministic bookkeeping over architecture constants and sequence lengths.

assumptions (4)
  • domain assumption The collective operation patterns in the vLLM V0 engine follow the described sequence of AllReduce, Gather, AllGather, and Send/Recv per transformer layer.
    Equations (1)-(7) count exactly these operations; Tables III-VI show profiler counts matching.
  • domain assumption NCCL AllReduce and AllGather cost can be represented by volume correction factors 2(t-1)/t and (t-1)/t respectively.
    Cites the NCCL test performance guide [16]; no independent validation of this model on the testbed is provided.
  • domain assumption FP16/BF16 half-precision (b=2 bytes per element) and batch size 1 capture inference communication volume.
    All experiments use FP16/BF16 and single requests; the paper states this isolates batching effects.
  • domain assumption Excluding rank-0 profiles does not remove significant inference-time communication.
    Used to drop server initialization overhead (Section IV.B), but the impact on measured counts and volumes is not verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Characterizing Communication Patterns in Distributed Large Language Model Inference." pith.science (2026). https://pith.science/paper/CMVJHJ2P

@misc{pith2026250714392,
  author       = {Pith},
  title        = {Pith review of: Characterizing Communication Patterns in Distributed Large Language Model Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMVJHJ2P}},
  note         = {Machine review of arXiv:2507.14392}
}
read the original abstract

Large Language Models (LLMs) built on transformer architectures have transformed natural language processing, achieving remarkable performance across diverse applications. While distributed inference frameworks enable practical deployment of these models, inter-GPU communication creates significant performance constraints that limit service quality in real-world systems. This paper investigates communication dynamics in distributed LLM serving-analyzing how various parallelization approaches coordinate data exchange between GPU workers during inference. We study dense transformer-based models as representative examples of contemporary architectures widely used in operational deployments. Our work combines detailed profiling measurements with predictive analytical models to characterize communication behavior across different parallelization configurations. Results show that tensor parallelism incurs substantial network overhead but delivers superior response times for brief sequences, pipeline parallelism minimizes data transfer requirements while increasing total latency, and combined approaches demand careful tuning to achieve balanced performance. These insights offer practical recommendations for selecting appropriate parallelization schemes in production LLM services and identify key opportunities for optimizing inference frameworks and communication infrastructure.

Figures

Figures reproduced from arXiv: 2507.14392 by the authors.

Figure 1
Figure 1. Communication-computation breakdown for Llama [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of general LLM inference A. Large Language Model Inference LLM inference is typically an autoregressive process that involves two distinct phases. The prefill phase processes the input prompt in parallel, computing initial Key-Value (KV) states for attention that are stored in a KV cache. This is followed by a sequential decoding phase, where tokens are generated one by one, with each step utilizing the… view at source ↗
Figure 3
Figure 3. Illustration of Tensor and Pipeline Parallelism [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Tensor Parallelism: Validation with observed data (Count & Total Message Size), TP=4, across models [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Pipeline Parallelism: Validation with observed data (Count & Total Message Size), Llama-3.1-8B, across PP Degrees [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Communication volume comparison across parallelism [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Communication volume scaling with decode sequence length across parallelism strategies for LLM inference with [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Llama-3.2-3B SLO metrics comparison across tensor parallelism degrees for LLM inference with [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Llama-3.2-3B SLO metrics comparison across pipeline parallelism degrees for LLM inference with [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Llama-2-13B SLO metrics comparison across hybrid parallelism strategies for LLM inference with [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Understanding and Improving Communication Performance in Multi-node LLM Inference

    cs.DC 2025-11 conditional novelty 5.0 of 10

    Performance analysis of multi-node LLM inference identifies all-reduce bottlenecks and introduces NVRAR hierarchical all-reduce achieving 1.9-3.6x lower latency than NCCL and up to 1.72x end-to-end batch latency reduc...

Reference graph

Works this paper leans on

30 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [22]

    Demystifying the communication characteristics for distributed transformer models,

    Q. Anthony, B. Michalowicz et al. , “Demystifying the communication characteristics for distributed transformer models,” 2024. [Online]. Available: https://arxiv.org/abs/2408.10197

    arXiv 2024

  2. [16]

    NCCL Test Performance Measure- ment Guide,

    NVIDIA, “NCCL Test Performance Measure- ment Guide,” https://github.com/NVIDIA/nccl- tests/blob/master/doc/PERFORMANCE.md, 2024, accessed: September 21, 2025

    work page 2024

  3. [1]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey et al. , “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    arXiv 2024

  4. [2]

    The claude 3 model family: Opus, sonnet, haiku

    “The claude 3 model family: Opus, sonnet, haiku.” [Online]. Available: https://api.semanticscholar.org/CorpusID:268232499

    work page

  5. [3]

    GPT-4 Technical Report,

    OpenAI and Josh Achiam and Steven Adler and Sandhini Agarwal and Lama Ahmad and Ilge Akkaya and et. al., “GPT-4 Technical Report,” 2024

    work page 2024

  6. [4]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu et al. , “Training language models to follow instructions with human feedback,” 2022. [Online]. Available: https://arxiv.org/abs/2203.02155

    arXiv 2022

  7. [5]

    A survey on test-time scaling in large language models: What, how, where, and how well?

    Q. Zhang, F. Lyu et al. , “A survey on test-time scaling in large language models: What, how, where, and how well?” 2025. [Online]. Available: https://arxiv.org/abs/2503.24235

    arXiv 2025

  8. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, D. Guo et al. , “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [Online]. Available: https://arxiv.org/abs/2501.12948

    arXiv 2025

Show all 30 references
  1. [7]

    Openai o1 system card,

    OpenAI, : et al. , “Openai o1 system card,” 2024. [Online]. Available: https://arxiv.org/abs/2412.16720

    2024 arXiv

  2. [8]

    Gemini 2.5: Our most intelligent AI model,

    “Gemini 2.5: Our most intelligent AI model,” https://blog.google/technology/google-deepmind/gemini-model- thinking-updates-march-2025/#gemini-2-5-thinking, [Accessed 25- 05-2025]

    2025

  3. [9]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang et al. , “Chain-of-thought prompting elicits reasoning in large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2201.11903

    2023 arXiv

  4. [10]

    Demystifying the communication characteristics for distributed transformer models,

    Q. Anthony, B. Michalowicz et al. , “Demystifying the communication characteristics for distributed transformer models,” in 2024 IEEE Sym- posium on High-Performance Interconnects (HOTI) , 2024, pp. 57–65

    2024

  5. [13]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li et al. , “Efficient memory management for large language model serving with pagedattention,” 2023. [Online]. Available: https://arxiv.org/abs/2309.06180

    2023 arXiv

  6. [14]

    Megatron-LM: Training Multi-Billion Parameter Language Models using GPU Model Parallelism,

    M. Shoeybi, M. Patwary et al., “Megatron-LM: Training Multi-Billion Parameter Language Models using GPU Model Parallelism,” arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  7. [15]

    GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism,

    Y . Huang, Y . Cheng et al. , “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism,” 2019

    2019

  8. [17]

    An In-depth Performance Characterization of CPU- and GPU-based DNN Training on Modern Architectures,

    A. A. Awan, H. Subramoni, and D. K. Panda, “An In-depth Performance Characterization of CPU- and GPU-based DNN Training on Modern Architectures,” in Proceedings of the Machine Learning on HPC Environments , ser. MLHPC’17. New York, NY , USA: Association for Computing Machinery...

    2017

  9. [18]

    Performance Characterization of DNN Training using TensorFlow and PyTorch on Modern Clusters,

    A. Jain, A. A. Awan et al. , “Performance Characterization of DNN Training using TensorFlow and PyTorch on Modern Clusters,” in 2019 IEEE International Conference on Cluster Computing (CLUSTER) , 2019, pp. 1–11. 10

    2019

  10. [19]

    Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation,

    A. A. Awan, J. B ´edorf et al. , “Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation,” in 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID) , 2019, pp. 498–507

    2019

  11. [20]

    Comparative Study of Large Language Model Architectures on Frontier ,

    J. Yin, A. Bose et al. , “Comparative Study of Large Language Model Architectures on Frontier ,” 5 2024, accepted, to be presented at IPDPS 2024

    2024

  12. [21]

    Characterization of Large Language Model Development in the Datacenter,

    Q. Hu, Z. Ye et al. , “Characterization of Large Language Model Development in the Datacenter,” in 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) . Santa Clara, CA: USENIX Association, 4 2024, pp. 709–729. [Online]. Available: https://www.usenix...

    2024

  13. [23]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li et al., “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    2023

  14. [24]

    Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale,

    R. Y . Aminabadi, S. Rajbhandari et al., “Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale,” 2022

    2022

  15. [25]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu et al., “Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,” 2024. [Online]. Available: https://arxiv.org/abs/2401.09670

    2024 arXiv

  16. [26]

    Orca: A distributed serving system for Transformer-Based generative models,

    G.-I. Yu, J. S. Jeong et al. , “Orca: A distributed serving system for Transformer-Based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . Carlsbad, CA: USENIX Association, Jul. 2022, pp. 521–538. [Online]. Available: https...

    2022

  17. [27]

    Alpa: Automating inter- and intra-operator parallelism for distributed deep learning,

    L. Zheng, Z. Li et al. , “Alpa: Automating inter- and intra-operator parallelism for distributed deep learning,” 2022. [Online]. Available: https://arxiv.org/abs/2201.12023

    2022 arXiv

  18. [28]

    Efficiently scaling transformer inference,

    R. Pope, S. Douglas et al. , “Efficiently scaling transformer inference,”

  19. [29]

    Thorough characterization and analysis of large transformer model training at-scale,

    S. Cheng, J.-L. Lin et al. , “Thorough characterization and analysis of large transformer model training at-scale,” Proc. ACM Meas. Anal. Comput. Syst. , vol. 8, no. 1, feb 2024. [Online]. Available: https://doi.org/10.1145/3639034

    2024 doi

  20. [30]

    Available: https://arxiv.org/abs/2211.05102

    [Online]. Available: https://arxiv.org/abs/2211.05102

    arXiv

  21. [31]

    Welcome to TensorRT-LLM Documentation! ; TensorRT-LLM — nvidia.github.io,

    “Welcome to TensorRT-LLM Documentation! ; TensorRT-LLM — nvidia.github.io,” https://nvidia.github.io/TensorRT-LLM/, [Accessed 14- 07-2025]. 11

    2025

  22. [32]

    Sglang: Efficient execution of structured language model programs,

    L. Zheng, L. Yin et al. , “Sglang: Efficient execution of structured language model programs,” 2024. [Online]. Available: https://arxiv.org/abs/2312.07104

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.