Pith. sign in

REVIEW 2 major objections 5 minor 52 references

SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Offloading sampling and inter-stage communication to idle CPUs removes the three main pipeline bubbles of LLM decoding, SiPipe claims, lifting throughput up to 2.1x.

desk verdict A thoughtful PP inference design with a plausible mechanism, but the headline gains rest on a static-batch assumption that isn't tested under the churn that production serving actually sees. read the letter →

arxiv 2506.22033 v1 pith:RB763DXV submitted 2025-06-27 cs.DC

Yongchao He , Bohan Zhao , Zheng Cao This is my paper
classification cs.DC
keywords pipelineparallelismLLMinferenceCPUoffloadingsamplingexecutionbubblesstaticgraphsasynchronouscommunicationthroughput-latencytradeoff
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pipeline parallelism (PP) splits a large language model across GPUs, but each stage pays three overheads—extra sampling work on the last stage, CPU-side input preparation before every forward, and synchronized metadata-laden communication between stages—so GPU time is wasted in what the paper calls load-imbalance, intra-stage, and inter-stage bubbles. SiPipe's central claim is that these bubbles can be largely eliminated by putting otherwise-idle host CPUs to work: a pool of CPU samplers takes over the final stage's sampling, a token-safe execution model lets the CPU prepare the next batch while the GPU runs the current forward without violating the fixed buffer bindings of statically captured execution graphs, and structure-aware transmission learns the hidden-state layout once so stages can pre-allocate buffers and exchange data asynchronously. Under the same PP configuration as the leading open-source serving engine, the paper reports 1.4–2.1x higher throughput on eight- and sixteen-GPU testbeds, up to 42.7% lower per-token latency, and up to 23 percentage points higher average GPU utilization. If the results hold, PP becomes a substantially cheaper way to scale LLM serving across nodes, because the system converts otherwise-wasted CPU cycles into pipeline saturation rather than leaving them idle.

What carries the argument

Three coordination mechanisms carry the argument. (1) Column-wise CPU sampling: the final stage's logits are stored transposed as $Z^\top\in\mathbb{R}^{V\times B}$, so penalties and outputs accumulate incrementally across iterations with only $p$ replicas (one per pipeline stage) and no per-iteration allocation of a $\sim$300 MB penalty tensor; this makes sampling fast enough to run on host cores inside the 1–2 ms decoding slack. (2) Token-safe execution model (TSEM): for each batch size, two statically captured execution graphs are bound to two shared input buffers and alternated every iteration, coordinated by a finite-state machine with separate CPU and GPU progress counters, so input preparation for iteration $i+1$ overlaps the forward pass of iteration $i$ without race conditions. (3) Structure-aware transmission (SAT): the receiver learns the static structure—keys, shapes, dtypes—of the inter-stage hidden-state dictionary on the first iteration, then reads the batch size from the scheduler's broadcast, pre-allocates memory, and posts asynchronous receives, removing the multi-round metadata serialization/deserialization that dominated the 2–5 ms inter-stage handoff and leaving only the sub-500-$\mu$s data transfer, which hides under computation.

What would settle it

Measure end-to-end throughput while continuously varying the composition of each scheduled microbatch—for example, by injecting short requests that finish and leave the batch between consecutive iterations—and check whether CPU sampling time per iteration grows beyond the 1–2 ms decoding slack; if it does, the claimed 16–38% throughput contribution of CPU sampling should shrink or reverse.

Watch

Extended reading notes

Core claim

The paper establishes that the three pipeline-bubble types are separable and can be removed by rebalancing work onto the host CPU rather than re-partitioning the model. The final stage's 22–40% extra load is eliminated by moving sampling to a CPU process pool that works on transposed logits ($Z^\top\in\mathbb{R}^{V\times B}$) and updates penalty tensors incrementally—only the entries corresponding to newly sampled tokens change each iteration—so a $\sim$300 MB penalty matrix never has to be rebuilt. The 12–19% intra-stage gap disappears under a token-safe execution model that pre-captures two versions of the static execution graph per batch size, each bound to a separate input buffer, and alternates versions under a message-driven finite-state machine; the CPU executor fills one buffer while the GPU reads the other, making the write-after-read hazard structurally impossible. Inter-stage latency falls from 2–5 ms to under 500 $\mu$s because structure-aware transmission captures the fixed dictionary layout of hidden states on the first iteration and then derives the only dynamic field—batch size—from the scheduler's output, enabling pre-allocated asynchronous communication with no metadata handshakes. The measured consequence is that per-stage timelines show essentially no bubbles, and with the same PP degree and TP degree as the baseline engine, throughput rises 1.6–2.1x on 8 GPUs and 1.4–1.7x on 16 GPUs, with up to 42.7% lower time-per-output-token and up to 23% higher average GPU utilization.

Load-bearing premise

The speedup of the whole design rests on the CPU sampling optimization, which assumes that the sequences in consecutively scheduled microbatches are identical or nearly identical so that penalty tensors can be updated incrementally; if real request dynamics change the batch composition or size every iteration, the incremental updates would require recomputation and the CPU sampler could become the new straggler.

Editorial extensions

If this is right

  • If the bubbles are genuinely gone, pipeline-parallel inference becomes attractive at higher PP degrees, because the previous cost of deeper pipelines (more stages, more bubbles) no longer applies; the 16-GPU experiments show 1.8–2.0x scaling when doubling GPUs, versus 1.4–1.7x for the baseline.
  • Removing the final stage's sampling load means the slowest stage is no longer the one that samples, so the steady-state iteration time is set by the forward pass itself; the reported per-stage timeline shows the last stage is no longer the bottleneck.
  • The 12–19% intra-stage and 2–5 ms inter-stage stalls, once overlapped, translate directly into lower time-per-output-token (up to 42.7% on the 16-GPU setups), which shortens the tail of the latency CDF that user-facing latency SLOs are drawn from.
  • On faster GPUs the relative gain is larger—the H100 testbed shows 1.6–2.1x versus 1.4–1.7x on the A100-class testbed—because forward time shrinks relative to bubble overheads, a trend the paper expects to continue on future accelerators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The column-wise, incrementally updated penalty trick is not specific to sampling: the same transposed-layout idea could accelerate other per-token CPU-side operations in serving stacks (logits post-processing for speculative decoding, reward-model scoring, or safety classifiers), since they also process a stable batch of sequences iteration after iteration.
  • A natural stress test is running SiPipe under a chunked-prefill or continuous-batching scheduler where microbatches change size every iteration; the paper's constant-batch assumption would be violated, and the degradation curve would tell how much headroom CPU sampling actually has.
  • TSEM's versioned-buffer scheme and SAT's structure capture could carry over to disaggregated prefill-decode architectures, where the hand-off of prompts and KV caches between pools has the same metadata-handshake and buffer-binding problems the paper solves inside a single pipeline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents SiPipe, a pipeline-parallel LLM inference engine that offloads sampling to underutilized CPU cores, uses a token-safe execution model (TSEM) to overlap CPU-side input preparation with GPU forward passes, and uses structure-aware transmission (SAT) to reduce inter-stage synchronization overhead. The evaluation compares SiPipe against vLLM and SGLang on six LLMs and three testbeds, reporting 1.6–2.1× throughput gains, up to 42.7% lower per-token latency, and higher GPU utilization, with ablations attributing 16–38% of the gain to CPU sampling. The paper argues that three pipeline bubble types—load-imbalance, intra-stage, and inter-stage—are largely eliminated, as illustrated by per-stage timelines.

Significance. If the reported results hold beyond the evaluated conditions, SiPipe would be a useful addition to the LLM serving systems literature: it identifies a real and measurable source of pipeline inefficiency, and the CPU-offload direction is well motivated by the observed CPU idle time. The evaluation breadth is a genuine strength: six models, three testbeds, multiple PP/TP configurations, batch-size sweeps, and per-component ablations. The per-stage timeline in Figure 11 provides direct visual evidence that the three bubble types are reduced in the tested static-batch regime. However, the central CPU-sampling mechanism rests on a static-batch assumption that conflicts with the paper's own description of continuous batching, and no dynamic workload is evaluated. The headline gains are therefore demonstrated only in an idealized regime. No code or data are released, and no run-to-run variance is reported, which limits reproducibility.

major comments (2)
  1. [§5.1 and §7.1/§7.5] The incremental CPU-sampling scheme in §5.1 assumes that consecutively scheduled microbatches are identical or highly similar, so that only B entries of the penalty tensor f need be updated per iteration and p column-wise replicas suffice. This assumption is contradicted by the paper's own description of LLM serving in §2.1, which states that completed sequences are removed and new ones are added to maintain batch occupancy, and by the Figure 2 caption, which acknowledges preemption. In any continuous-batching workload, a microbatch slot revisited p iterations later will contain different sequences; their penalty columns must be zeroed and recomputed, and the corresponding TSEM BatchMetadata replicas in §5.2 lose their reuse. With realistic completion and preemption rates, penalty maintenance reverts to O(B·V) memory traffic, the CPU sampler can no longer fit within the 1–2 ms decoding slack, and the 16–38% CPU-sampling contribution attributed in §7.5 (Figure 16) collapses. The workload described in §7.1 reuses a fixed ShareGPT prompt set with no arrivals, completions, or preemptions, so the 1.6–2.1× throughput and 42.7% TPOT gains are not demonstrated for the dynamic serving scenario the paper claims to serve. The authors should either extend the design to handle batch churn (e.g., dirty-row tracking with amortized reset) and evaluate it with a continuous-batching workload, or explicitly scope the claims to static offline workloads.
  2. [§5.1 and §7 (sampling correctness)] Because the penalty tensor f is reused across iterations, sequence churn without row reset would produce sampling distributions that do not correspond to the specified penalties: stale entries from a departed sequence would affect the sampling of subsequent sequences in the same row block. The paper reports no generation-quality or distributional equivalence check (e.g., comparing the output token distribution or generated text against a reference sampling implementation that recomputes penalties from scratch) in §7. Such a check is necessary even for the static workload, both to validate the incremental update semantics and to establish a baseline for any future dynamic-batching extension.
minor comments (5)
  1. [Figure 12 and Figure 13 captions] Both captions contain the typo 'different enginess'; this should read 'different engines'.
  2. [§7.2, batch-size sweep paragraph] The text reports '1.4-.17×' for DeepSeek V3; this should read '1.4–1.7×'.
  3. [Figure 14 and Figure 15 captions] The caption sentence 'vLLM (TP) and SiPipe adopt the default configuration, while vLLM (TP) using pure TP configuration' is confusing; the first reference should presumably be to vLLM (PP), and the second should read 'uses pure TP configuration'.
  4. [§1 vs Appendix A] The simplified scaling formula in §1, T(p,t) ∝ 1/(k1/(pt) + k2 log t / p + b), does not match the Appendix A expression in Eq. (7), where a p-dependent term 4L·s·b·h/(p·B1) appears. The notation for k1, k2, and b should be reconciled with the appendix derivation.
  5. [§7.1–§7.4] All throughput, latency, and utilization results appear to be single measurements; reporting at least three runs and standard deviations for the headline configurations would help the reader assess the stability of the 1.4–2.1× speedups. A reproducibility appendix with CUDA graph capture policies, CPU core assignments, and penalty buffer sizes would also strengthen the artifact, especially since the code is not yet released.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation chain: headline gains are external measurements; the only self-citation (TSEM shadow buffering) is minor and non-load-bearing.

full rationale

SiPipe's principal claims—1.6–2.1× throughput, up to 42.7% lower TPOT, and 23% higher GPU utilization—are empirical comparisons against vLLM and SGLang under identical PP configurations (Figs. 8, 12–13), not outputs of the paper's analytic model or of its own assumptions. The Appendix A model is a standard TP/PP scaling model and is not used to compute the reported speedups, so there is no self-definitional reduction. The incremental CPU-sampling scheme does rest on the explicit §5.1 assumption that 'consecutively scheduled batches are identical or exhibit high similarity (i.e., the microbatch size B remains constant across iterations)'; this is a real limitation for dynamic continuous-batching workloads (and §2.1 itself notes that completed sequences are removed and new ones added, with Figure 2's caption acknowledging preemption), but it is an unvalidated precondition, not a circular derivation—the evaluation's fixed ShareGPT, no-churn workload means the 16–38% CPU-sampling contribution may not transfer, yet the gain is measured, not derived from the assumption. The one self-citation, [22] in §5.2 for the 'shadow buffering-based scheme' in TSEM, is minor and not load-bearing: TSEM's correctness is independently specified through the versioned-buffer FSM and message-driven communicator, and the overall speedups come from external benchmarks. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled in via self-citation. Overall circularity is therefore low; score 2 reflects the one minor self-citation, not any reduction of the claims to their inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 3 invented entities

The design depends on several explicit assumptions: batch similarity for incremental CPU sampling, static hidden-state structure for structure-aware transmission, availability of idle CPU capacity, and a particular characterization of vLLM's execution model. None of these are validated with external data or artifacts in the preprint. The Appendix A performance model uses standard analytic approximations and is not fitted to the reported speedups.

assumptions (5)
  • domain assumption Consecutively scheduled microbatches are identical or highly similar, keeping microbatch size constant across iterations, enabling incremental CPU-side penalty updates.
    Stated explicitly in Section 5.1. The CPU sampling optimization, contributing 16 to 38 percent of the claimed speedup, depends on this; if batches vary in composition or size, incremental updates must be recomputed from scratch, raising CPU latency and potentially re-creating the load-imbalance bubble.
  • domain assumption Hidden-state dictionaries passed between pipeline stages have static structure: fixed keys, dtypes, and devices; only the batch dimension varies.
    Section 5.3. Structure-aware transmission leverages this to eliminate metadata exchange; if the structure changed, the receiver would need the multi-round metadata protocol and latency would return to the structure-unaware case.
  • domain assumption CPU resources on the serving node are underutilized during LLM inference, providing spare capacity for offloaded sampling and communication.
    Motivating premise from Sections 1 and 7.4; the paper asserts CPUs are typically below 10 percent utilized. The measured vLLM CPU usage in Figure 15 appears higher than this, and no external data is cited, so the reserve capacity is asserted rather than demonstrated.
  • domain assumption vLLM's execution model serializes CPU input preparation and GPU forward via synchronous copies into fixed CUDA graph buffers, creating WAR hazards if overlapped.
    Observation 2 in Section 3.1. This characterization of the baseline is load-bearing for TSEM's design, but it is not independently verified in the paper.
  • standard math The analytic throughput/latency model in Appendix A adequately captures TP/PP scaling behavior (e.g., two all-reduces per layer, start-up latency model).
    Appendix A. Used to argue that PP reduces communication volume for p^2 t < 4N. It is a simplified model and does not directly produce the measured speedups.
invented entities (3)
  • Buffered IPC Channel (BIC) with BIC-I, BIC-L, and BIC-O
    purpose: Low-latency shared-memory ring buffers and a TCP-based combine channel connecting scheduler, workers, and CPU samplers.
    New software component; its performance is only evaluated as part of the whole system; no standalone artifact, benchmark, or code is provided.
  • CPU sampler pool with column-wise layout
    purpose: Offloads sampling from the final GPU stage to idle CPU cores using transposed logits and incremental penalty buffers.
    Software mechanism; the paper claims 16 to 38 percent throughput gains, but no isolated profiling of CPU sampling latency versus GPU sampling is reported and no code is released.
  • Token-safe execution model (TSEM) with versioned buffers
    purpose: Double-buffered static CUDA graphs to overlap CPU input preparation with GPU forward without WAR hazards.
    Technique largely reuses the known shadow-buffering idea from the authors' prior paper (reference [22]); no independent verification artifact is provided here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference." pith.science (2026). https://pith.science/paper/RB763DXV

@misc{pith2026250622033,
  author       = {Pith},
  title        = {Pith review of: SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RB763DXV}},
  note         = {Machine review of arXiv:2506.22033}
}
read the original abstract

As inference workloads for large language models (LLMs) scale to meet growing user demand, pipeline parallelism (PP) has become a widely adopted strategy for multi-GPU deployment, particularly in cross-node setups, to improve key-value (KV) cache capacity and inference throughput. However, PP suffers from inherent inefficiencies caused by three types of execution bubbles-load-imbalance, intra-stage, and inter-stage-which limit pipeline saturation. We present SiPipe, a heterogeneous pipeline design that improves throughput by leveraging underutilized CPU resources to offload auxiliary computation and communication. SiPipe incorporates three key techniques-CPU sampling, a token-safe execution model, and structure-aware transmission-to mitigate pipeline bubbles and improve execution efficiency. Across diverse LLMs, SiPipe achieves up to 2.1 times higher throughput, 43% lower per-token latency, and up to 23% higher average GPU utilization compared to the state-of-the-art vLLM under the same PP configuration, demonstrating its generality across LLMs and deployment scenarios.

Figures

Figures reproduced from arXiv: 2506.22033 by the authors.

Figure 1
Figure 1. Throughput comparison of different engines under various parallel configurations on 16 H100 GPUs. Each con￾figuration is denoted as 𝑃 𝑖 𝑗 , where the PP degree 𝑝 = 𝑖 and the TP degree 𝑡 = 𝑗. See §7.1 for detailed experimental settings. is used to further parallelize the forward of that stage—i.e., a portion of the model—across the assigned GPUs. This hy￾brid strategy reduces inter-GPU communication and enables effic… view at source ↗
Figure 2
Figure 2. Example of iterative autoregressive generation in LLM serving. Note that the scheduled batch may changed from one iteration to the next—for instance, certain sequences may be preempted by others. The diagram omits these dy￾namics for clarity. with each stage responsible for a subset of layers. During in￾ference, unfinished sequences are grouped into microbatches and processed in a pipelined fashion. For example, wit… view at source ↗
Figure 3
Figure 3. Per-iteration execution breakdown using vLLM on 8 H100 GPUs with Qwen-2.5-72B (𝑡 = 2, 𝑝 = 4). Bars denote iteration time; filled regions represent GPU forward time. 0 100 200 300 400 500 600 Time (milliseconds) Stage 0 Stage 1 Stage 2 Stage 3 End-to-End Iteration Forward on GPU [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Per-iteration execution breakdown using vLLM on 16 H100 GPUs with DeepSeek V3 (𝑡 = 4, 𝑝 = 4). 3 Motivation This section presents key observations on PP and identifies core challenges in achieving high throughput. These observa￾tions reveal fundamental inefficiencies in…
Figure 5
Figure 5. Figure 5: Architecture and workflow of SiPipe. 4 Overview 4.1 System Architecture [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Illustration of the token-safe execution model for conflict-free GPU execution. the CPU executor, and asynchronously transmits GPU execu￾tor outputs to the next stage while receiving hidden states from the previous one. To drive progress, it maintains two indica￾tors—t…
Figure 7
Figure 7. Figure 7: (b) illustrates SiPipe’s structure-aware transmis￾sion mechanism, which avoids redundant metadata commu￾nication by separating static and dynamic components of the hidden states. The mechanism consists of two key compo￾nents: static structure capture and dynamic batch …
Figure 8
Figure 8. Figure 8: Throughput comparison of different engines under various parallel configurations, evaluated on multiple testbeds. Each configuration is denoted as 𝑃 𝑖 𝑗 , where PP degree 𝑝 = 𝑖 and the TP degree 𝑡 = 𝑗. Specifically, 𝑃 1 𝑡 denotes the pure TP mode. 64 128 256 512 Batch …
Figure 10
Figure 10. Figure 10: Effect of the number of pipeline stages (i.e., PP degree 𝑝) on throughput. In the first three subfigures, each stage uses a TP degree of 𝑡=2, while the last subfigure uses 𝑡=4. The dashed line denotes the performance of SiPipe under the default configuration. large LL…
Figure 11
Figure 11. Figure 11: Per-iteration execution breakdown of each pipeline stage using SiPipe (𝑃 4 4 ) on 16 H100 GPUs in the H100-NVL testbed with the DeepSeek V3. We therefore compare one-instance scaling directly. large LLMs use 𝑡 = 2 while ultra-large models use 𝑡 = 4. As shown in [PITH…
Figure 13
Figure 13. Figure 13: TPOT comparison of different enginess, evaluated with 16×H100 on H100-NVL testbed. can delay the entire pipeline by up to 𝑝Δ𝑡, as all 𝑝 stages are sequentially blocked. Notably, for Mixtral-8×7B, SiPipe’s TPOT is comparable to that of the pure TP baseline. This model …
Figure 14
Figure 14. Figure 14: GPU utilization over time on H100-NVL testbed. vLLM (TP) and SiPipe adopt the default configuration, while vLLM (TP) using pure TP configuration. 0 30 60 Time (seconds) (a) Qwen-2.5-72B (8×H100) 0 20 40 60 80 100 CPU Utilization (%) 0 30 60 Time (seconds) (b) DeepSeek…
Figure 15
Figure 15. Figure 15: CPU utilization over time on H100-NVL testbed. vLLM (TP) and SiPipe adopt the default configuration, while vLLM (TP) using pure TP configuration. approximately 60%, confirming that the CPU now contributes meaningfully across the inference process. We also observe diff…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 28 canonical work pages

  1. [1]

    [n. d.]. Llama 2. https://infohub.delltechnologies.com/ja-jp/l/llama- 2-inferencing-on-a-single-gpu/introduction-3976/

    work page

  2. [2]

    Llama 2 follow-up: too much RLHF, GPU sizing, technical details

    2024. Llama 2 follow-up: too much RLHF, GPU sizing, technical details. https://www.interconnects.ai/p/llama-2-part-2

    work page 2024

  3. [3]

    AI Inference Market Size, Share and Trends Report

    2025. AI Inference Market Size, Share and Trends Report. https: //www.grandviewresearch.com/industry-analysis/artificial- intelligence-ai-inference-market-report

    work page 2025

  4. [4]

    OpenAI API Documentation

    2025. OpenAI API Documentation. https://platform.openai.com/docs

    work page 2025

  5. [5]

    ShareGPT Datasets

    2025. ShareGPT Datasets. https://huggingface.co/collections/bun nycore/sharegpt-datasets-66fa831dcee14c587f1e6d1c

    work page 2025

  6. [6]

    vLLM–Optimization and Tuning

    2025. vLLM–Optimization and Tuning. https://docs.vllm.ai/en/latest/configuration/optimization.html

    work page 2025

  7. [7]

    David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. 1985. A learning algorithm for Boltzmann machines. Cognitive science 9, 1 (1985), 147–169

    work page 1985

  8. [8]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ram- jee. 2024. Taming{Throughput-Latency} tradeoff in{LLM} inference with{Sarathi-Serve}. In 18th USENIX Symposium on Operating Sys- tems Design and Implementation (OSDI 24). 117–134

    work page 2024

Show all 52 references
  1. [9]

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ram- jee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implemen...

    2024

  2. [10]

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255 (2022)

    2022 arXiv

  3. [11]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems33 (2020), 1877–1901

    2020

  4. [12]

    Branden Butler, Sixing Yu, Arya Mazaheri, and Ali Jannesari. 2024. PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC) . IEEE, 1–19...

    2024 arXiv

  5. [13]

    Xin Chen, Hengheng Zhang, Xiaotao Gu, Kaifeng Bi, Lingxi Xie, and Qi Tian. 2023. Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism. arXiv preprint arXiv:2304.11414 (April 2023)

    2023 arXiv

  6. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  7. [15]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833 (2018)

    2018 arXiv

  8. [16]

    Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, and Luo Mai. 2024. ServerlessLLM: Low- latency serverless inference for large language models. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Asso...

    2024

  9. [17]

    Philip Gage. 1994. A new algorithm for data compression. The C Users Journal 12, 2 (1994), 23–38

    1994

  10. [18]

    Design Guide. 2020. Cuda c++ programming guide. NVIDIA, July (2020)

    2020

  11. [19]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  12. [20]

    Tianyu Guo, Xianwei Zhang, Jiangsu Du, Zhiguang Chen, Nong Xiao, and Yutong Lu. 2025. gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling. arXiv preprint arXiv:2504.14775 (April 2025)

    2025 arXiv

  13. [21]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778

    2016

  14. [22]

    Yongchao He, Wenfei Wu, Yanfang Le, Ming Liu, and ChonLam Lao. 2023. A generic service to provide in-network aggregation for key-value streams. In Proceedings of the 28th ACM International Con- ference on Architectural Support for Programming Languages and Operating Systems, V...

    2023

  15. [23]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi

  16. [24]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    2024 arXiv

  17. [25]

    Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , Eduardo Blanco...

    2018 doi

  18. [26]

    Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston

  19. [27]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  20. [28]

    Yanying Lin, Yanbo Li, Shijie Peng, Yingfei Tang, Shutian Luo, Haiy- ing Shen, Cheng-Zhong Xu, and Kejiang Ye. 2024. Quart: Latency- Aware FaaS System for Pipelining Large Model Inference. In Proceed- ings of the 44th IEEE International Conference on Distributed Com- puting Sy...

    2024

  21. [29]

    Yanying Lin, Shijie Peng, Shuaipeng Wu, Yanbo Li, Chengzhi Lu, Chengzhong Xu, and Kejiang Ye. 2024. Planck: Optimizing LLM Inference Performance in Pipeline Parallelism with Fine-Grained SLO Constraint. In 2024 IEEE International Conference on Web Services (ICWS). IEEE, 1306–1313

    2024

  22. [30]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434 (2024)

    2024 arXiv

  23. [31]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al . 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  24. [32]

    Ruilong Ma, Xiang Yang, Jingyu Wang, Qi Qi, Haifeng Sun, Jing Wang, Zirui Zhuang, and Jianxin Liao. 2024. HPipe: Large Language Model Pipeline Parallelism for Long Context on Heterogeneous Cost-Effective Devices. In Proceedings of the Conference of the North American Chapter o...

    2024 doi

  25. [33]

    Microsoft. 2024. DeepSpeed. https://github.com/microsoft/DeepS peed

    2024

  26. [34]

    OpenAI. 2023. ChatGPT application. https://chat.openai.com/

    2023

  27. [35]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wain- wright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina 13 Y ongchao He, Bohan Zhao, and Zheng Cao Slama, Alex Ray, et al . 2022. Training language models to follow instructions with human feedback. Advanc...

    2022

  28. [36]

    Pitch Patarasuk and Xin Yuan. 2009. Bandwidth optimal all-reduce algorithms for clusters of workstations. J. Parallel and Distrib. Comput. 69, 2 (2009), 117–124

    2009

  29. [37]

    David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2022. The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink. arXiv:2204.05149 [cs.LG] https: //arxiv.org/abs/2...

    2022 arXiv

  30. [38]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation—A{KVCache-centric} Architecture for Serving{LLM} Chatbot. In 23rd USENIX Conference on File and Storage Te...

    2025

  31. [39]

    Maryam Rahimi Jafari et al. 2024. PISeL: Pipelining DNN Inference for Serverless Computing. In Proceedings of the CIKM Conference. ACM

    2024

  32. [40]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. InProceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Katrin Erk and Noah A. Smith (Eds.). Associ...

    2016 doi

  33. [41]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi- billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

    2019 arXiv

  34. [42]

    Yifan Tan, Cheng Tan, Zeyu Mi, and Haibo Chen. 2025. Pipellm: Fast and confidential large language model services with speculative pipelined encryption. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Sys...

    2025

  35. [43]

    Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. https: //qwenlm.github.io/blog/qwen2.5/

    2024

  36. [44]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  37. [45]

    Jiannan Wang, Jinyang Fang, Aoyu Li, and Pengcheng Yang. 2024. PipeFusion: Displaced Patch Pipeline Parallelism for Inference of Dif- fusion Transformer Models. arXiv preprint arXiv:2405.14430 (May 2024)

    2024 arXiv

  38. [46]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521–538

    2022

  39. [47]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of struc- tured language model programs. arXiv preprint arXiv:2312.07104 (2024)

    2024 arXiv

  40. [48]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (...

    2024

  41. [49]

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, and Xin Liu. 2025. Table 1. Notations ...

    2025 arXiv

  42. [2018]

    arXiv preprint arXiv:1811.00907 2 (2018)

    Importance of a search strategy in neural dialogue modelling. arXiv preprint arXiv:1811.00907 2 (2018)

    2018 arXiv

  43. [2019]

    arXiv preprint arXiv:1904.09751 (2019)

    The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019)

    2019 arXiv

  44. [2023]

    In Proceedings of the 29th Symposium on Operating Systems Principles

    Efficient memory management for large language model serv- ing with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.