Pith. sign in

REVIEW 3 major objections 4 minor 44 references

DeltaServe claims that the idle GPU capacity of production LLM serving can be converted into LoRA fine-tuning throughput at full SLO compliance, by treating fine-tuning forward passes as prefill-like requests that join inference batches onl

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:29 UTC pith:7JEZ42BK

load-bearing objection A genuinely useful host-agnostic co-serving system with three real integrations; the SLO guarantee is only as strong as the unvalidated fitted latency model, so ask for prediction-error validation before trusting it beyond the evaluated traces. the 3 major comments →

arxiv 2607.28848 v1 pith:7JEZ42BK submitted 2026-07-30 cs.DC cs.LG

DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

Jiaxuan Chen , Jianshu She , Ye Yuan , Rajat Ghosh , Karan Gupta , Qirong Ho , Xue Liu , Oana Balmau This is my paper
classification cs.DC cs.LG
keywords LLM inference servingLoRA fine-tuningco-servingSLO-aware admission controlCUDA graphsmulti-LoRA batchinglatency modelingGPU utilization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Production LLM serving is provisioned for peak traffic, so between bursts most GPU capacity sits idle. DeltaServe claims that this idle capacity can be harvested for LoRA fine-tuning—updating a small adapter on a frozen base model—without violating inference latency targets. It does so by folding fine-tuning forward passes into inference batches as prefill-like single-step requests, admitting them only when a fitted latency model predicts enough SLO headroom, and running the backward pass in a separate subprocess that yields to inference. If correct, this means continual model adaptation can run in the background of existing serving, on the same hardware, rather than requiring dedicated training GPUs; on a 20-minute production trace the authors report 2.9x the fine-tuning throughput of the closest co-serving system while meeting SLOs for all requests.

Core claim

The paper's central claim is that LoRA fine-tuning and inference prefill are structurally the same forward computation over a frozen base model, so a fine-tuning step can ride inside the serving engine's existing multi-LoRA batch rather than running as a separate job. DeltaServe turns each fine-tuning sample into a single-step, prefill-only request bound to a reserved training adapter, captures the activations the backward pass will need, and admits these samples into inference batches only when an analytical latency model predicts the step will still finish within the earliest request's TTFT and TPOT budgets. The model, fitted offline and refined online, prices step time from prefill length

What carries the argument

The load-bearing mechanism is a per-step admission controller built around an analytical latency model. For a mixed batch, predicted step time is T ≈ α Σ(n_i+B_d)² + β(T_in+B_d) + γT_ft + εK + c, pricing quadratic attention cost, linear feed-forward cost, the cost of saving fine-tuning activations, KV-cache memory traffic, and fixed overhead, with a separate decode-only form. Coefficients are profiled offline and refined online, and are chosen separately for CUDA-graph replay versus eager execution, since mixed batches with activation hooks cannot be replayed from a graph. The scheduler converts each request's SLO into a per-step time budget (TTFT slack and TPOT slack), rejects admission if

Load-bearing premise

The SLO-preservation guarantee rests on the accuracy of the fitted analytical latency model; if the model under-predicts the cost of a mixed inference-plus-fine-tuning batch, the scheduler will admit fine-tuning that pushes requests past their TTFT and TPOT deadlines, and the paper does not report the model's prediction error against measured step times.

What would settle it

Measure the scheduler's predicted versus actual step latency on a workload whose batch composition lies outside the profiling grid—for example, many short prompts interleaved with long decodes and continuous fine-tuning admission. If the median absolute prediction error at typical batch sizes exceeds one TTFT/TPOT budget (e.g., 400 ms TTFT and 120 ms TPOT on the A100 setup), the 100% SLO-compliance claim for untested traces would not survive.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • LLM serving clusters can use already-idle GPU capacity for adapter training, reducing or eliminating the need for dedicated fine-tuning hardware.
  • Any inference engine that already supports multi-LoRA batching can adopt co-serving through a compact hook interface, without rearchitecting its scheduler or execution pipeline.
  • Co-serving can hold 100% SLO compliance on bursty production traffic, while systems that tie fine-tuning to decode-phase headroom miss SLOs under dense bursts.
  • Fine-tuning throughput automatically throttles during load spikes and expands during lulls, so background training tracks inference headroom at sub-second timescales.
  • Because fine-tuning forward passes are retired before generating output tokens, co-serving remains invisible to inference clients.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper: the same headroom-budgeting logic should apply to any batch-compatible background task whose cost can be modeled, such as data-prep passes or calibration runs; the key is only that the cost be predictable and preemptible.
  • Going beyond the paper: the design harvests prefill-phase headroom but does not also exploit decode-phase memory-bandwidth slack the way LLMStation does; combining both mechanisms could raise fine-tuning throughput further than either alone.
  • Going beyond the paper: a direct extension would be to larger models or multi-GPU tensor-parallel serving, where the latency model's coefficients are architecture-specific but the admission logic is not—offline calibration would need re-running, not the algorithm.
  • Going beyond the paper: the 100% SLO-compliance claim is only as strong as the latency model's fit; logging predicted versus measured step times on an untested workload would reveal how much conservative margin the scheduler actually needs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. DeltaServe is a host-agnostic co-serving design that runs LoRA fine-tuning inside existing LLM inference engines (vLLM, SGLang, S-LoRA) by treating each fine-tuning forward pass as a prefill-like request. It uses a fitted analytical latency model (Eqs. I–II) to admit fine-tuning only when the predicted mixed-batch execution time remains within TTFT/TPOT budgets, and it executes backward passes in a separate subprocess that yields to inference at layer boundaries. On a 20-minute Nutanix production trace, DeltaServe-vLLM reports 2.9x higher fine-tuning throughput than LLMStation (1418 vs 489 tok/s) and 39% higher than a split-pool vLLM+torchtune baseline (1418 vs 1014 tok/s), while maintaining 100% SLO compliance versus 85% for LLMStation. The paper claims portability through a compact hook interface and evaluates the design on two hardware setups and three workloads.

Significance. If the results hold, DeltaServe makes a valuable practical contribution: idle inference capacity can be converted into fine-tuning throughput without violating latency objectives, and the host-agnostic design is a genuine step beyond single-engine co-serving systems. The paper provides measured results on two GPU platforms, three host engines, and a production trace, and it makes a reasonable comparison against LLMStation and a split-pool baseline. The main risk is that the SLO-preservation guarantee is entirely gated by the latency model of §3.3, which is fitted but not validated against held-out batch compositions. This is a load-bearing gap, but it can be addressed with additional experiments, so the paper merits a major revision rather than rejection.

major comments (3)
  1. [§3.3, Eq. (I), Algorithm 1 line 17] Additional text to ensure completeness. The same functional-form concern applies to Eq. II, which drops the O(B_d·K) decode attention compute and only retains a linear dependence on K. This is not fatal if the regression is empirically accurate, but that accuracy is not demonstrated.
  2. [§3.4 'Estimator calibration'] This is a load-bearing issue because the model is both the admission controller and the quantity being calibrated.
  3. [§4.2, Figure 5] Even a few repeated runs would substantially increase confidence in the measured claims.
minor comments (4)
  1. [§4.2, Figure 5] The split-pool baseline vLLM+torchtune dedicates one GPU to fine-tuning and three to inference, whereas DeltaServe uses all four GPUs for both. The comparison is fair in total hardware, but the baseline's inference-only capacity is lower, which explains some of its latency advantage. Please discuss this explicitly or add a 4-GPU inference-only reference to separate co-serving effects from hardware allocation.
  2. [§3.3, Eq. (I)] The variable K is not precisely defined. Is it total KV-cache length in tokens, the number of KV-cache entries, or the memory footprint? Please define it and clarify whether it is per-layer or aggregated.
  3. [§4.3, Figure 6] For SGLang and S-LoRA, the paper reports fine-tuning throughput but not the SLO compliance rate. Please include the measured SLO satisfaction for these integration points, as it is the central metric for the design.
  4. [§4.4, Figure 8] The interruption experiment compares DeltaServe-Temp with and without interruption, but it is not clear from the figure whether the 5% tail numbers are computed over all requests or only over requests that arrived during an in-flight fine-tuning step. Please clarify the statistical basis of the tail comparison.

Circularity Check

0 steps flagged

No significant circularity: the paper's headline results are measured end-to-end, and the latency model is a fitted control input rather than a premise that defines the claimed outcome.

full rationale

I walked the paper's claimed derivation chain and found no circular step that can be exhibited as an equation-level reduction or as a fitted parameter renamed as a prediction. The central claims are empirical: fine-tuning throughput (1418 tok/s on the Nutanix trace), SLO satisfaction rates, and latency comparisons are measured from actual request timelines, as reported in Figure 5, not produced by the latency model. Equation I and Equation II are analytical latency models whose coefficients are seeded offline and refined online; they are used by Algorithm 1 to admit or reject fine-tuning samples. This is a standard closed-loop controller: a fitted surrogate model gates admission, while the paper's SLO-compliance numbers are measured against real per-request TTFT/TPOT behavior. The model is not defined in terms of the claimed SLO outcome, nor is any SLO number computed from the model's own predictions. There are also no load-bearing self-citations: the prior co-serving systems cited (LLMStation, FlexLLM) have disjoint author sets, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The weakest point in the paper is the lack of validation of Equation I's functional form against held-out measured step times, and the reader's concern about the (n_i+B_d)^2 attention term is a legitimate correctness/robustness risk, not a circularity. The paper would be stronger with a prediction-error study, but that does not make the derivation circular. The throughput and SLO claims stand independently as measurements on the evaluated workloads.

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 0 invented entities

The central claim depends on the fitted latency model and several hardware/software domain assumptions. No new physical entities are introduced.

free parameters (3)
  • Latency model coefficients (α, β, γ, ε, c) for Eq. I = Fitted offline; refined online
    The model predicts mixed-batch step time; coefficients are regression parameters fit to measured latencies during profiling and serving (Section 3.3-3.4).
  • Latency model coefficients (β', ε', c') for Eq. II = Fitted offline; refined online
    Decode-only step time model (Section 3.3).
  • Backward batch size = 256 tokens
    Activations are accumulated until 256 fine-tuning tokens before a backward pass; this hand-chosen value affects training throughput and activation buffer usage (Section 4.1).
axioms (7)
  • domain assumption Multi-LoRA batching is available in the host engine
    DeltaServe's hook interface requires the host to support multi-LoRA batching; this is true for vLLM, SGLang, S-LoRA but not for all serving systems (Section 3.1).
  • domain assumption The forward pass of LoRA fine-tuning is structurally identical to inference prefill
    The paper relies on this to fold fine-tuning into inference batches (Section 3.2). This is a property of the transformer forward computation, not proven by the paper.
  • domain assumption CUDA MPS allows concurrent execution of inference and backward kernels with distinct CUDA contexts and preemption at layer boundaries
    The backward subprocess design depends on this (Section 3.2). The paper does not measure interference in isolation.
  • ad hoc to paper The latency model forms in Eq. I and II accurately describe step time
    The functional forms are asserted as analytical approximations; their coefficients are fitted to data and are not derived or validated against measured step times (Section 3.3).
  • ad hoc to paper Greedy admission based on the latency model preserves SLOs
    Algorithm 1 admits fine-tuning while predicted batch time <= budget; there is no proof that the model's predictions are error-bounded, so the SLO guarantee is an empirical property, not a formal one (Section 3.4).
  • domain assumption The offline profiling sweeps cover the online workload distribution
    Coefficients are initialized by profiling; if online workloads differ from the profiled shapes, the model may be inaccurate (Section 3.4).
  • domain assumption The Nutanix trace is representative of production inference load
    The headline result is demonstrated on a single production trace; generalizability to other workloads is assumed.

pith-pipeline@v1.3.0-alltime-deepseek · 19608 in / 15972 out tokens · 159879 ms · 2026-08-03T01:29:54.003716+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs." pith.science (2026). https://pith.science/paper/7JEZ42BK

@misc{pith2026260728848,
  author       = {Pith},
  title        = {Pith review of: DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7JEZ42BK}},
  note         = {Machine review of arXiv:2607.28848}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak. We present DeltaServe, a host-agnostic co-serving design that converts this idle inference capacity into LoRA fine-tuning throughput while preserving inference service-level objectives (SLOs). DeltaServe integrates with existing inference engines through a compact hook interface that requires only multi-LoRA batching support. It exploits the shared execution structure of inference prefill and LoRA fine-tuning forward passes, and uses an SLO-aware scheduler to admit and execute fine-tuning only when sufficient inference headroom is available. The scheduler is driven by a CUDA-graph-aware latency model calibrated offline and refined online. We integrate DeltaServe with vLLM, SGLang, and S-LoRA. On a production trace from Company X, DeltaServe on vLLM delivers 2.9x higher fine-tuning throughput than LLMStation at 100% inference SLO compliance, versus 85% for LLMStation. It also achieves 39% higher fine-tuning throughput than a baseline running vLLM+torchtune, using no additional hardware and maintaining full SLO compliance.

Figures

Figures reproduced from arXiv: 2607.28848 by Jianshu She, Jiaxuan Chen, Karan Gupta, Oana Balmau, Qirong Ho, Rajat Ghosh, Xue Liu, Ye Yuan.

Figure 1
Figure 1. Figure 1: Inference fluctuation in a production workload (Nutanix). On average, 40% of the GPU is unused. a substantial fraction of GPU compute idle when inference is the only workload running. Prior work has approached this underutilization from two angles. The first improves the inference pipeline itself: Split￾wise [26] disaggregates prefill and decode across machines and Orca [38] mixes prefill and decode reques… view at source ↗
Figure 2
Figure 2. Figure 2: DeltaServe architecture and host interface. White boxes are generic inference serving system components; blue boxes are DeltaServe add-ons. Numbered markers indicate the attachment points and their relation described in Sec￾tion 3: CPU-side admission control, GPU-side activation cap￾ture, shared GPU memory, and the backward subprocess. outside the host engine’s inference request queue. It loads and tokeniz… view at source ↗
Figure 3
Figure 3. Figure 3: Co-serving workflow in DeltaServe. The vertical axis shows the serving pipeline, and time advances from left to right. Each box represents one request or fine-tuning sample: yellow boxes are new arrivals, white boxes are inference prefill, orange boxes are inference decode, dashed gray boxes are completed requests, and blue boxes are fine-tuning samples. Dashed blue outlines mark DeltaServe add-on componen… view at source ↗
Figure 4
Figure 4. Figure 4: Inference workloads for evaluation: the synthetic burst-light and burst-dense patterns (top) and the 20-minute Nutanix production trace (bottom). In each panel the grey shaded bars report the incoming request rate and the blue line reports the resulting output-token rate. Inference Workloads [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: End-to-end comparison of DeltaServe-vLLM, LLMStation, and vLLM+torchtune on the 4-GPU deployment, one row per workload (top: burst-light; middle: burst-dense; bottom: Nutanix). The left column overlays per-request end-to-end latency against arrival time, with per-system averages annotated. The three right columns show each system’s per-second token throughput as a stacked band: fine-tuning (orange hatched)… view at source ↗
Figure 6
Figure 6. Figure 6: Portability of DeltaServe to SGLang and S-LoRA on a single RTX 5090, replaying the scaled-down Nutanix trace. Top: request timeline. Middle and bottom: per-request end-to-end latency as dots (left axis) for the unmodified host engine and its DeltaServe-enabled version, with fine-tuning throughput as the filled curve (right axis). includes optimizations that reduce inference overhead and leave more usable h… view at source ↗
Figure 7
Figure 7. Figure 7: Forward batch co-serving on a single RTX 5090 using the 600–800s window of the Nutanix trace, scaled to one RTX 5090. Top: scheduled request timeline. Bottom: per￾request end-to-end latency for vLLM, DeltaServe-Temp, and DeltaServe-vLLM, with fine-tuning throughput for the two DeltaServe variants. DeltaServe-Temp restricts fine￾tuning to inference-free steps. The orange curve in the lower panel shows that … view at source ↗
Figure 8
Figure 8. Figure 8: Effect of fine-tuning-only step interruption on one RTX 5090 over a scaled-down burst-light trace. Dots show per-request end-to-end latency; filled curves show fine￾tuning throughput. Disabling interruption forces inference requests arriving during a fine-tuning-only step to wait until that step completes. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 6 linked inside Pith

  1. [1]

    Gulavani, and Ramachandran Ramjee

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. 2023. SARATHI: Ef- ficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arXiv:2308.16369 [cs.LG]

    Pith/arXiv arXiv 2023

  2. [2]

    Zhihao Bai, Zhen Zhang, Yibo Zhu, and Xin Jin. 2020. PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications. In14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association

    2020

  3. [3]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerat- ing Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318 [cs.CL]

    Pith/arXiv arXiv 2023

  4. [4]

    Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. 2024. Punica: Multi-Tenant LoRA Serving. In Proceedings of Machine Learning and Systems, P. Gibbons, G. Pekhi- menko, and C. De Sa (Eds.), Vol. 6

    2024

  5. [5]

    Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. 2022. Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing. In2022 USENIX Annual Technical Conference (USENIX ATC 22). USENIX Association, Carlsbad, CA

    2022

  6. [6]

    Yujeong Choi and Minsoo Rhu. 2020. PREMA: A Predictive Multi-Task Scheduling Algorithm For Preemptible Neural Processing Units. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)

    2020

  7. [7]

    Franklin, Joseph E

    Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: a low-latency online prediction serving system. InProceedings of the 14th USENIX Conference on Networked Systems Design and Implementation(Boston, MA, USA). USENIX Association, Usa

    2017

  8. [8]

    Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Paral- lelism and Work Partitioning. arXiv:2307.08691 [cs.LG]

    Pith/arXiv arXiv 2023

  9. [9]

    Aditya Dhakal, Sameer G Kulkarni, and K. K. Ramakrishnan. 2020. GSLICE: controlled spatial sharing of GPUs for a scalable inference platform. InProceedings of the 11th ACM Symposium on Cloud Com- puting(Virtual Event, USA). Association for Computing Machinery, New York, NY, USA

    2020

  10. [10]

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kauf- mann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. In14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association

    2020

  11. [11]

    Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thi- nakaran, Bikash Sharma, Mahmut Taylan Kandemir, and Chita R. Das

  12. [12]

    Yongjun He, Haofeng Yang, Yao Lu, Ana Klimović, and Gustavo Alonso

  13. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations

    2022

  14. [14]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  15. [15]

    Yunseong Lee, Alberto Scolari, Byung-Gon Chun, Marco Domenico Santambrogio, Markus Weimer, and Matteo Interlandi. 2018. PRET- ZEL: Opening the Black Box of Machine Learning Prediction Serving Systems. In13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA

    2018

  16. [16]

    Jiamin Li, Hong Xu, Yibo Zhu, Zherui Liu, Chuanxiong Guo, and Cong Wang. 2023. Lyra: Elastic Scheduling for Deep Learning Clusters. InProceedings of the Eighteenth European Conference on Computer Systems(Rome, Italy). Association for Computing Machinery, New York, NY, USA

    2023

  17. [17]

    Zikun Li, Zhuofu Chen, Remi Delacourt, Gabriele Oliaro, Zeyu Wang, Qinghan Chen, Shuhuai Lin, April Yang, Zhihao Zhang, Zhuoming Chen, Sean Lai, Xinhao Cheng, Xupeng Miao, and Zhihao Jia. 2025. AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding. arXiv:2501.12162 [cs.CL]

    Pith/arXiv arXiv 2025

  18. [18]

    Gonzalez, and Ion Stoica

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. In17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). USENIX Association, Boston, MA

    2023

  19. [19]

    Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021. TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models. InProceedings of the 38th International Conference on Machine Learning, Marina Meila and Tong Zhang (Eds.), Vol. 139. Pmlr

    2021

  20. [20]

    Xiaoxuan Liu, Jongseok Park, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Chen Zhang, Kuntai Du, Xiangxi Mo, Kaichao You, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. 2025. TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput. arXiv:2406.14066 [cs.AI]

    Pith/arXiv arXiv 2025

  21. [21]

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz. 2022. PEFT: State-of- the-art Parameter-Efficient Fine-Tuning methods.https://github.com/ huggingface/peft

    2022

  22. [22]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna 13 Jiaxuan Chen, Jianshu She, Ye Yuan, Rajat Ghosh, Karan Gupta, Qirong Ho, Xue Liu, and Oana Balmau Abhyankar, and Zhihao Jia. 2024. SpecInfer: Accelerating Large Lan...

    2024

  23. [23]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on GPU clusters using megatron-LM. InProceedings of the International Conference for High...

    2021

  24. [24]

    Gabriele Oliaro, Zhihao Jia, Daniel Campos, and Aurick Qiao. 2025. SuffixDecoding: Extreme Speculative Decoding for Emerging AI Appli- cations. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

    2025

  25. [25]

    Gabriele Oliaro, Xupeng Miao, Xinhao Cheng, Vineeth Kada, Mengdi Wu, Ruohan Gao, Yingyi Huang, Remi Delacourt, April Yang, Yingcheng Wang, Colin Unger, and Zhihao Jia. 2026. FlexLLM: Token- Level Co-Serving of LLM Inference and Finetuning with SLO Guar- antees. InThe 23rd USENIX Symposium on Networked Systems Design and Implementation

    2026

  26. [26]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2025. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. InProceedings of the 51st Annual International Symposium on Computer Architecture (Buenos Aires, Argentina). IEEE Press

    2025

  27. [27]

    Yadwadkar, and Christos Kozyrakis

    Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2021. INFaaS: Automated Model-less Inference Serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association

    2021

  28. [28]

    Haichen Shen, Lequn Chen, Yuchen Jin, Liangyu Zhao, Bingyu Kong, Matthai Philipose, Arvind Krishnamurthy, and Ravi Sundaram. 2019. Nexus: a GPU cluster engine for accelerating DNN-based video analy- sis. InProceedings of the 27th ACM Symposium on Operating Systems Principles(Huntsville, Ontario, Canada). Association for Computing Machinery, New York, NY, USA

    2019

  29. [29]

    Gonzalez, and Ion Stoica

    Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. 2024. SLoRA: Scalable Serving of Thousands of LoRA Adapters. InProceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6

    2024

  30. [30]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: high-throughput generative inference of large language models with a single GPU. InProceedings of the 40th Inter- national Conference on Machine Learning(Honolulu, Hawaii, USA). JMLR.org

    2023

  31. [31]

    Foteini Strati, Xianzhe Ma, and Ana Klimovic. 2024. Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications. InProceedings of the Nineteenth European Conference on Computer Sys- tems(Athens, Greece). Association for Computing Machinery, New York, NY, USA

    2024

  32. [32]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model.https:// github.com/tatsu-lab/stanford_alpaca

    2023

  33. [33]

    2024.torchtune: PyTorch’s finetuning library.https://github.com/pytorch/torchtune

    torchtune maintainers and contributors. 2024.torchtune: PyTorch’s finetuning library.https://github.com/pytorch/torchtune

    2024

  34. [34]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL]

    Pith/arXiv arXiv 2023

  35. [35]

    Abhishek Vijaya Kumar, Gianni Antichi, and Rachee Singh. 2025. Aqua: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. Association for Computing Machinery, New York, NY, USA

    2025

  36. [36]

    Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou. 2018. Gan- diva: Introspective Cluster Scheduling for Deep Learning. In13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA

    2018

  37. [37]

    Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, Zhi Li, Yihui Feng, Wei Lin, and Yangqing Jia. 2020. AntMan: Dynamic Scaling on GPU Clusters for Deep Learning. In14th USENIX Sym- posium on Operating Systems Design and Implementation (OSDI 20). USENIX Association

    2020

  38. [38]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA

    2022

  39. [39]

    Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. 2019. MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving. In2019 USENIX Annual Technical Confer- ence (USENIX ATC 19). USENIX Association, Renton, WA

    2019

  40. [40]

    Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. SHEPHERD: Serving DNNs in the Wild. In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA

    2023

  41. [41]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. InThe Thirty-eighth Annual Conference on Neural Information Processing Sys- tems. 14

    2024

  42. [2022]

    In19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22)

    Cocktail: A Multidimensional Optimization for Model Serving in Cloud. In19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX Association, Renton, WA

  43. [2023]

    InProceedings of the 29th Symposium on Operating Systems Principles(Koblenz, Germany)

    Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles(Koblenz, Germany). Association for Computing Machinery, New York, NY, USA

  44. [2025]

    InProceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference(Boston, MA, USA)

    Resource multiplexing in tuning and serving large language models. InProceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference(Boston, MA, USA). USENIX Association, Usa

    2025