Pith. sign in

REVIEW 4 major objections 4 minor 65 references

Glinthawk: A Two-Tiered Architecture for Offline LLM Inference

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By moving the attention step to a cluster of cheap machines, Glinthawk reports 5.9x throughput at 2.8x lower cost for offline LLM inference.

desk verdict Measured throughput gains are real and worth engaging; the headline cost claim is tied to retail hardware prices, and the abstract overstates the baseline. read the letter →

arxiv 2501.11779 v2 pith:JPZITHN4 submitted 2025-01-20 cs.LG cs.DCcs.PF

Pouya Hamadanian , Sadjad Fouladi This is my paper · ORCID
classification cs.LGcs.DCcs.PF
keywords LLMinferencetwo-tierarchitectureattentionoffloadingKVcachethroughputoptimizationpipelineparallelismcommoditynetworkingbatchprocessing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Glinthawk sets out to prove that the most expensive part of an LLM inference cluster—the high-end accelerator—does not need to run attention at all. The paper argues that attention is compute-light, embarrassingly parallel, and the only stateful operation in the Transformer, since it depends on a per-prompt key-value cache. By offloading that cache and its arithmetic to a second tier of low-end, memory-rich nodes connected by ordinary Ethernet, the cache can grow independently of the model weights. That lets the first tier run far larger batches, and in the paper's prototype it reports 5.9x higher throughput and 2.8x lower cost than paged-cache baselines, rising to 16.3x throughput on long sequences. If the claim holds, batch offline inference could be built on commodity hardware instead of scaling scarce accelerators.

What carries the argument

The load-bearing mechanism is the two-tier split itself, formalized as a scheduling analysis of in-flight batches. The quantity $IF_{\mathrm{gh}}(B,K')$—the number of batches that must be in flight to keep Tier-1 busy while batches travel to and from Tier-2—stays roughly constant as the number of Tier-2 nodes $K'$ grows, unlike single-tier pipeline parallelism where the required in-flight count grows with the number of GPUs. The analysis is supported by a discrete-event simulator that models compute kernels, FIFO network queues, and link latency, and is used to choose the configuration (number of Tier-1 nodes, Tier-2 nodes, and batch size) that maximizes throughput or minimizes cost per token.

What would settle it

Run the paper's optimal configuration (16 Tier-1 GPUs with 48 Tier-2 CPU nodes) with the inter-tier link replaced by a real high-bandwidth-delay network, say 100 Gbps with 20 ms round-trip time, and measure end-to-end token throughput under TCP-style congestion control; if throughput drops below the single-tier baseline at comparable cost, the claim that ordinary Ethernet suffices is falsified. A second check: resample kernel durations from measured per-run variance in the simulator; if the 5.9x advantage disappears, the result is an artifact of the stability assumption.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that attention and non-attention work in a Transformer are separable in a way that changes the economics of inference. Non-attention operations are stateless weight-matrix multiplications whose weight-loading cost is amortized by batching; attention is a low-compute, memory-bound, embarrassingly parallel operation that is the only piece carrying per-prompt state. Glinthawk therefore assigns non-attention layers to Tier 1 (high-end accelerators) and attention, with its key-value cache, to Tier 2 (cheap CPU or low-end nodes), with a central dispatcher routing state objects between tiers over commodity Ethernet. Because the key-value cache no longer occupies accelerator memory, batch size scales with the number of Tier-2 nodes rather than with GPU memory. The prototype—mid-range GPUs for Tier 1 and CPU virtual machines for Tier 2—achieves 5.9x higher token throughput and 2.8x lower cost per unit throughput than single-tier pipeline baselines, and 16.3x throughput at 2.4x lower cost for long sequences, while needing less than 50 Gbps of inter-tier bandwidth and tolerating tens of milliseconds of latency.

Load-bearing premise

The entire configuration and the headline numbers rest on a simulator that assumes kernel times are stable, network links behave as simple FIFO queues with negligible dispatcher overhead, and that a hand-tuned oversubscription factor captures real memory usage; if real-world congestion or straggler variance is significant, the required number of in-flight batches grows and the measured gains shrink.

Editorial extensions

If this is right

  • Offline, latency-tolerant LLM workloads can be served on a heterogeneous mix of accelerators and commodity CPU nodes over Ethernet, without InfiniBand-class interconnects.
  • Long-context processing, which inflates the key-value cache, benefits most: the paper reports 16.3x throughput at 2.4x lower cost at long sequence lengths.
  • Scaling throughput means adding Tier-2 nodes, not expensive GPUs; inter-tier bandwidth needs only grow linearly with token throughput and stay below roughly 50 Gbps for the tested cluster.
  • The configuration can be chosen automatically: a simulation-based search over Tier-1 count, Tier-2 count, and batch size predicts measured throughput with high fidelity for the prototype.
  • The architecture generalizes beyond the prototype: profiled simulations suggest other pairings, such as high-end GPUs with mid-range GPUs as Tier-2, also improve throughput and cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the two-tier split is as effective as reported, the same decomposition could be applied to other stateful components in sequence models—retrieval caches, recurrent states, or external memory—turning memory-bound state into a separately scalable service.
  • The cost analysis counts hardware only; production economics would need to account for network infrastructure, power, and the utilization effect of Tier-2 DRAM bandwidth, which could narrow or widen the reported advantage depending on deployment.
  • A testable extension: applying the same dispatcher-based split to a tensor-parallel Tier-1 (the paper notes this is trivial) would let existing tensor-parallel serving stacks realise the same batching gains without switching to pipeline parallelism.
  • The paper anticipates but does not build custom Tier-2 hardware; if attention proves to be memory-bandwidth-bound as argued, purpose-built low-cost memory engines could intensify the cost advantage beyond the CPU prototype.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. Glinthawk is a two-tier architecture for offline LLM inference in which high-end GPUs (Tier 1) execute only the non-attention GEMM operations while attention computation and the KV cache are offloaded to a second tier of cheaper CPU nodes (Tier 2). The paper presents an analytical model of KV-cache dynamics and in-flight batching, a discrete-event simulator for configuration search, and a prototype built on NVIDIA T4 GPUs and AMD EPYC CPU VMs. The authors report a 5.9x end-to-end throughput improvement and 2.8x cost reduction for their main configuration, and 16.3x throughput at long sequence lengths, against single-tier or paged-attention baselines. The simulator matches the measured end-to-end configurations with high accuracy (Figure 6), and the prototype is publicly available.

Significance. If the headline results withstand scrutiny, Glinthawk is a useful contribution to the growing line of disaggregated LLM-inference designs: it demonstrates on a real prototype that attention can be separated from non-attention computation over commodity Ethernet with modest bandwidth requirements, and that the KV-cache bottleneck can be addressed by scaling a cheap memory tier. The paper provides a public implementation, a simulator validated against real runs, and a clear analytical framing of why attention is a good offload candidate. The measured throughput results are credible and the simulator fidelity is a notable strength. However, the significance is tempered by two load-bearing issues: the cost-reduction claim is tied to retail equipment prices that do not represent the cloud-rental scenario targeted by the paper, and the long-sequence (16.3x/2.4x) claims are presented without specifying whether they are measured or simulated. The analytical model also contains a factor-of-two error in the KV-cache formula.

major comments (4)
  1. [§5.5, Table 6, Abstract] The headline 2.8x cost reduction is computed from September-2024 retail equipment prices (Table 6) and excludes network, power, and operational costs. For the paper's target use case—offline, throughput-oriented inference on rented cloud infrastructure—the relevant unit cost is the cloud instance price, and the relative cost of CPU VMs versus T4 GPUs is very different. At common on-demand rates (e.g., an r5.4xlarge-style 16-vCPU/128 GiB instance at roughly $1.0/hr versus a g4dn.xlarge-style T4 instance at roughly $0.5/hr), the 48-CPU + 16-T4 configuration costs roughly 6–7x as much per hour as the 16-T4 baseline, while delivering a 5.9x throughput gain; the cost per token is then comparable to or worse than the baseline, not 2.8x better. Since the 2.8x figure appears in the abstract and motivates the architecture, the cost analysis needs either a cloud-price sensitivity study or a clear scope restriction to owned hardware, and the abstract should be qualified accordingly.
  2. [§5.7, Figures 9b and 9c, Abstract] The 16.3x throughput and 2.4x cost improvements for long sequence lengths are presented as results, but the section does not state whether these numbers come from end-to-end measurements or from the simulator of §4.2. Given the testbed's scale (e.g., 80-GPU single-tier baselines at 16K sequences), these appear to be simulation predictions. If so, the paper should label them as simulated, because the Appendix C limitations—stable kernel times, FIFO network queues, and a hand-tuned KV oversubscription factor—bear directly on the reliability of these long-sequence predictions. The abstract's phrasing should also be corrected to distinguish simulated from measured results.
  3. [Abstract, §5.4, Figure 5a] The abstract claims the 5.9x throughput and 2.8x cost improvements are "compared to paged attention baselines," but §5.4 computes these factors relative to the authors' own single-tier (pipeline-parallel) implementation, not directly to vLLM or TensorRT. Figure 5a shows that the single-tier baseline is at least as fast as the vLLM and TensorRT configurations, but the 5.9x figure is not the measured ratio against vLLM. The wording should be changed to refer to the single-tier baseline, consistent with the Introduction, or the comparison should be recomputed against vLLM.
  4. [§3.1, KV-cache formula] The KV-cache memory formula is internally inconsistent: the text states that a single token and layer requires 4D_kv bytes, but then gives M := 2 N S D_kv. The correct expression is M = 4 N S D_kv, which matches the quoted 640 MiB for Llama2-70B at 2048 tokens (N=80, D_kv=1024). The factor-of-two error propagates to the analytical examples in §3.2 and to the max-batch-size expressions in Table 3. While the measured end-to-end results do not rely on this formula, the analytical model should be corrected because readers will use it to compute KV-cache requirements for their own configurations.
minor comments (4)
  1. [§3.2, pipeline-parallel example] The numbers in the Llama2-70B example do not follow from the stated formulas: with K=10, N=80, t_n/t_c = 1/5.6, Eq. (1) gives IFpp ≈ 11, not 20, and the claimed free context of C_max/M = 32 prompts would require more than 16 GiB per T4 GPU for the 70B model split 10 ways. Please rework the example with consistent parameters or state clearly that the numbers are illustrative and not exact.
  2. [§3.1 and elsewhere] There are several typos and spacing errors, e.g., "rival the the size" in §3.1 and "outpacing... by16×" in the Introduction; a careful proofread is needed.
  3. [Figure 7 caption] The caption "Glinthawk's throughput compared to running each Tier as a separate pipeline" is unclear; it should state explicitly that the comparison is against the sum of the throughputs of the two tiers when run independently.
  4. [§5.5] The sentence "We consider computing and memory equipment as the bulk cost of our setups" should be expanded to justify why network, power, and operational costs are excluded, and to note that the cost conclusion is highly sensitive to the assumed hardware prices and to whether the hardware is owned or rented.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: headline throughput and cost results are measured end-to-end; the simulator is validated against measurements, not used to generate them.

full rationale

Glinthawk's central claims are end-to-end measurements on a real prototype, not predictions derived from the same data used to test them. The configuration optimizer uses a discrete-event simulator with profiled kernel times, but the reported 5.9x and 16.3x throughput improvements and the cost figures come from running the actual system and comparing with baselines. Figure 6 validates the simulator against end-to-end measurements; this is a fairness check rather than a circular prediction. The hand-tuned oversubscription factor in Appendix C and the simulation assumptions for hypothetical H100/A100 configurations affect external validity and risk of overfitting, but they do not make any stated result equivalent to its input by construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no renamed known result. The internal inconsistency in the KV-cache formula M := 2NSD_kv versus the stated 4D_kv per token-layer is a correctness or exposition issue, not a circularity. The retail-price-based cost analysis may be sensitive to market assumptions, but that is an economic-validity concern, not a circularity concern.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces a system architecture, not new physical entities. The main postulates are hardware capability and cost assumptions listed above.

free parameters (1)
  • KV cache oversubscription factor = 3x (2-2.5x for single-tier)
    Appendix C: used to estimate maximum in-flight batches from context memory; tuned by experiment ('We have found a factor of 3x to work well'). Affects simulator throughput predictions and hence chosen configurations.
assumptions (5)
  • domain assumption Attention is compute-light relative to non-attention GEMMs and can be spread across many low-end nodes without throughput loss.
    Section 2.2 and Table 2; the entire feasibility of Tier-2 CPUs rests on this complexity analysis.
  • domain assumption Network transfer delays are accurately modeled by FIFO link queues plus a static RTT (alpha-beta model).
    Appendix C; no validation against real congested or high-bandwidth-delay-product links.
  • domain assumption Dispatcher, Tier Router, and Network Controller overheads are negligible, on the order of tens of microseconds.
    Appendix C limitation (2); if this fails, in-flight batch requirements increase.
  • domain assumption KV cache oversubscription by 3x will not cause memory exhaustion in practice.
    Appendix C; single-tier needed a lower 2-2.5x factor to avoid OOM, which shows sensitivity.
  • domain assumption Retail hardware purchase price is the dominant cost component for comparing systems.
    Section 5.5; power, networking, and operational costs are excluded from the 2.8x cost claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Glinthawk: A Two-Tiered Architecture for Offline LLM Inference." pith.science (2026). https://pith.science/paper/JPZITHN4

@misc{pith2026250111779,
  author       = {Pith},
  title        = {Pith review of: Glinthawk: A Two-Tiered Architecture for Offline LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JPZITHN4}},
  note         = {Machine review of arXiv:2501.11779}
}
abstract

We introduce Glinthawk, an architecture for offline Large Language Model (LLM) inference. By leveraging a two-tiered structure, Glinthawk optimizes the utilization of the high-end accelerators ("Tier 1") by offloading the attention mechanism to lower-end compute tier ("Tier 2"). This separation allows the memory demand of the attention, known as the key-value cache, to scale independently from the model weights, enabling larger batch sizes and more efficient accelerator usage. Prototyped with NVIDIA T4 GPUs and standard CPU VMs, Glinthawk improves throughput by $5.9\times$ and reduces cost of generation by $2.8\times$, compared to paged attention baselines. For long sequence lengths, it achieves $16.3\times$ throughput improvement at $2.4\times$ less cost. Our evaluation shows that this architecture can tolerate moderate network latency with minimal performance degradation, making it highly effective for latency-tolerant, throughput-focused applications such as batch processing. The prototype is publicly available at https://github.com/microsoft/glinthawk.

Figures

Figures reproduced from arXiv: 2501.11779 by the authors.

Figure 1
Figure 1. Glinthawk dissects attention memory and compute [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Throughput gain vs. batch size, for the Llama2-70B [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Glinthawk’s batching schedule. Glinthawk hides the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Inference throughput vs. setup cost for various [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: (a) Inference throughput for various schemes using 16 NVIDIA T4 GPUs. Glinthawk extracts more throughput from [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 7
Figure 7. Figure 7: Glinthawk’s throughput compared to running [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Glinthawk’s cost per throughput unit compared to [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: (a) Glinthawk’s inference throughput with various inter-tier RTT values, normalized to throughput without induced [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: (a) Synchronous vs. (b) asynchronous parallelism. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: We simulate (a) Glinthawk and (b) single-tier [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 14
Figure 14. Figure 14: Box plots of Time to first token. Black lines denote [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 13
Figure 13. Figure 13: Box plots of Time per output token. Black lines [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 21 canonical work pages

  1. [1]

    [n. d.]. cuBLAS — developer.nvidia.com. https://developer.nvidia.c om/cublas. ([n. d.]). [Accessed 18-09-2024]

    work page 2024

  2. [2]

    [n. d.]. GitHub - flashinfer-ai/flashinfer: FlashInfer: Kernel Library for LLM Serving — github.com. https://github.com/flashinfer-ai/flashinfer. ([n. d.]). [Accessed 18-09-2024]

    work page 2024

  3. [3]

    [n. d.]. GitHub - NVIDIA/TensorRT-LLM at release/0.5.0 — github.com. https://github.com/NVIDIA/TensorRT-LLM/tree/release/0.5.0?tab= readme-ov-file. ([n. d.]). [Accessed 07-02-2024]

    work page 2024

  4. [4]

    [n. d.]. GitHub - vllm-project/vllm: A high-throughput and memory- efficient inference and serving engine for LLMs — github.com. https://github.com/vllm-project/vllm. ([n. d.]). [Accessed 18-09-2024]

    work page 2024

  5. [5]

    [n. d.]. Introducing Low-Level GPU Virtual Memory Man- agement | NVIDIA Technical Blog — developer.nvidia.com. https://developer.nvidia.com/blog/introducing- low- level- g pu-virtual-memory-management/. ([n. d.]). [Accessed 18-09-2024]

    work page 2024

  6. [6]

    [n. d.]. Long context | Generative AI on Vertex AI | Google Cloud — cloud.google.com. https://cloud.google.com/vertex-ai/generative-a i/docs/long-context. ([n. d.]). [Accessed 18-09-2024]

    work page 2024

  7. [7]

    [n. d.]. NVIDIA Collective Communications Library (NCCL) — developer.nvidia.com. https://developer.nvidia.com/nccl. ([n. d.]). [Accessed 18-09-2024]

    work page 2024

  8. [8]

    [n. d.]. openmp.org. https://www.openmp.org/wp-content/uploads /openmp-4.5.pdf. ([n. d.]). [Accessed 18-09-2024]

    work page 2024

Show all 65 references
  1. [9]

    [n. d.]. ShareGPT: Share your wildest ChatGPT conversations with one click. — sharegpt.com. https://sharegpt.com/. ([n. d.]). [Accessed 18-09-2024]

    2024

  2. [10]

    IEEE Standard for Floating-Point Arithmetic

    2008. IEEE Standard for Floating-Point Arithmetic. IEEE Std 754-2008 (2008), 1–70. https://doi.org/10.1109/IEEESTD.2008.4610935

    2008

  3. [11]

    Gulavani, and Ramachandran Ramjee

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhar- gav S. Gulavani, and Ramachandran Ramjee. 2023. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. (2023). arXiv:cs.LG/2308.16369 https://arxiv.org/abs/2308.16369

    2023 arXiv

  4. [12]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. (2023). arXiv:cs.CL/2305.13245

    2023 arXiv

  5. [13]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. 2022. DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. (2022). a...

    2022 arXiv

  6. [14]

    Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel

  7. [15]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020 arXiv

  8. [16]

    Zixian Cai, Zhengyang Liu, Saeed Maleki, Madanlal Musuvathi, Todd Mytkowicz, Jacob Nelson, and Olli Saarikivi. 2021. Synthesizing optimal collective algorithms. InProceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP ’21). ACM,...

    2021

  9. [17]

    Shaoyuan Chen, Yutong Lin, Mingxing Zhang, and Yongwei Wu. 2024. Efficient and Economic Large Language Model Infer- ence with Attention Offloading. (2024). arXiv:cs.LG/2405.01814 https://arxiv.org/abs/2405.01814

    2024 arXiv

  10. [18]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. (March 2023). https://lmsys....

    2023

  11. [19]

    Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. (2023). arXiv:cs.LG/2307.08691

    2023 arXiv

  12. [20]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

  13. [21]

    Abhimanyu Dubey et al. 2024. The Llama 3 Herd of Models. (2024). arXiv:cs.AI/2407.21783 https://arxiv.org/abs/2407.21783

    2024 arXiv

  14. [22]

    FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. (2022). arXiv:cs.LG/2205.14135

    2022 arXiv

  15. [23]

    Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, Jean Steiner, Itay Laish, and Amir Feder. 2023. LLMs Accelerate Annotation for Medical Information Extraction. InProceed- ings of the 3rd M...

    2023

  16. [24]

    William Fedus, Jeff Dean, and Barret Zoph. 2022. A Review of Sparse Expert Models in Deep Learning. (2022). arXiv:cs.LG/2209.01667 https://arxiv.org/abs/2209.01667

    2022 arXiv

  17. [25]

    Ke Wen James Reed, Pavel Belevich. 2022. PiPPy: Pipeline Parallelism for PyTorch. https://github.com/pytorch/PiPPy. (2022)

    2022

  18. [26]

    Roger W. Hockney. 1994. The communication challenge for MPP: Intel Paragon and Meiko CS-2. Parallel Comput. 20, 3 (1994), 389–398. https://doi.org/10.1016/S0167-8191(06)80021-9

    1994 doi

  19. [27]

    Fu, Christo- pher Ré, and Azalia Mirhoseini

    Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christo- pher Ré, and Azalia Mirhoseini. 2024. Hydragen: High-Throughput LLM Inference with Shared Prefixes. (2024). arXiv:cs.LG/2402.05099

    2024 arXiv

  20. [28]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  21. [29]

    Gonzalez, and Ion Stoica

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. (2023). arXiv:cs.LG/2302.11665

    2023 arXiv

  22. [30]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica

  23. [31]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. (2024). arXiv:cs.CL/2306.00978 14 A Two-Tiered Architectur...

    2024 arXiv

  24. [32]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. (2023). arXiv:cs.CL/2310.01889 https://arxiv.org/abs/2310.01889

    2023 arXiv

  25. [33]

    Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, and Wei Lin. 2024. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache. (2024...

    2024 arXiv

  26. [34]

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Anan- thanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. 2024. CacheGen: KV Cache Compression and Streaming for Fast Large Language M...

    2024

  27. [35]

    Avanika Narayan, Ines Chami, Laurel Orr, Simran Arora, and Christo- pher Ré. 2022. Can Foundation Models Wrangle Your Data? (2022). arXiv:cs.LG/2205.09911 https://arxiv.org/abs/2205.09911

    2022 arXiv

  28. [36]

    Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. 2020. Very Deep Transformers for Neural Machine Translation. (2020). arXiv:cs.CL/2008.07772

    2020 arXiv

  29. [37]

    John Nickolls, Ian Buck, Michael Garland, and Kevin Skadron. 2008. Scalable Parallel Programming with CUDA: Is CUDA the parallel programming model that application developers have been waiting for? Queue 6, 2 (mar 2008), 40–53. https://doi.org/10.1145/1365490.1365500

    2008

  30. [38]

    OpenAI. [n. d.]. OpenAI Batch API. https://platform.openai.com/do cs/guides/batch. ([n. d.]). [Accessed 18-09-2024]

    2024

  31. [39]

    Newegg. [n. d.]. Newegg Electronic Store. https://www.newegg.com. ([n. d.]). [Accessed 19-09-2024]

    2024

  32. [40]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently Scaling Transformer Inference. InProceedings of Machine Learning and Systems, D. Song, M. Carbin, and T. Chen (Eds.), Vo...

    2023

  33. [41]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9

    2019

  34. [42]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative LLM inference using phase splitting. (2024). arXiv:cs.AR/2311.18677 https://arxiv.org/abs/2311.18677

    2024 arXiv

  35. [43]

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He

  36. [44]

    Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. 2023. TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches. In 20th USENIX Symposium on Networked Systems Design ...

    2023

  37. [45]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yux- iong He. 2020. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. (2020). arXiv:cs.LG/1910.02054 https://arxiv.org/abs/1910.02054

    2020 arXiv

  38. [46]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. (2017). arXiv:cs.LG/1701.06538 https://arxiv.org/abs/1701.06538

    2017 arXiv

  39. [47]

    Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models wit...

    2023 arXiv

  40. [48]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGres- ley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. CoRR abs/1909.08053 (2019). arXiv:1909.08053 http://arxiv.org/abs/1909.08053

    2019 arXiv

  41. [49]

    Noam Shazeer. 2019. Fast Transformer Decoding: One Write- Head is All You Need. (2019). arXiv:cs.NE/1911.02150 https://arxiv.org/abs/1911.02150

    2019 arXiv

  42. [50]

    Gemma Team. 2024. Gemma: Open Models Based on Gem- ini Research and Technology. (2024). arXiv:cs.CL/2403.08295 https://arxiv.org/abs/2403.08295

    2024 arXiv

  43. [51]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation La...

    2023 arXiv

  44. [52]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  45. [53]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto

  46. [54]

    https://github.com/tatsu-lab/stanford_alpaca

    Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca. (2023)

    2023

  47. [55]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. (2024). arXiv:cs.CL/2309.17453 https://arxiv.org/abs/2309.17453

    2024 arXiv

  48. [56]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serv- ing System for Transformer-Based Generative Models. In 16th USENIX Symposium on Operating Systems Design and Implemen- tation (OSDI 22) . USENIX Association, Carlsbad...

    2022

  49. [57]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences...

    2023 arXiv

  50. [58]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. (2023). arXiv:cs.CL/1706.03762

    2023 arXiv

  51. [59]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learnin...

    2023

  52. [60]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2023. Efficiently Programming Large Language Models using SGLang. (2023). arXiv:cs.AI/2312.07104 16 A...

    2023 arXiv

  53. [63]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. (2023). arXiv:cs.CL/2306.05685 http...

    2023 arXiv

  54. [64]

    Xing, Joseph E

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. (2022). arXiv...

    2022 arXiv

  55. [2021]

    ZeRO-Offload: Democratizing Billion-Scale Model Training. (2021). arXiv:cs.DC/2101.06840 https://arxiv.org/abs/2101.06840

    2021 arXiv

  56. [2022]

    arXiv preprint arXiv:2209.01188 (2022)

    Petals: Collaborative Inference and Fine-tuning of Large Models. arXiv preprint arXiv:2209.01188 (2022). https://arxiv.org/abs/2209.01188

    2022 arXiv

  57. [2023]

    Efficient Memory Management for Large Language Model Serving with PagedAttention. (2023). arXiv:cs.LG/2309.06180

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.