Pith. sign in

REVIEW 5 major objections 5 minor 55 references

Adaptive Request Scheduling for CodeLLM Serving with SLA Guarantees

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SABER shows that replacing a static batch-size setting with an SLA-aware admission controller improves goodput by up to 26% and cuts latency variability by up to 45% in single-GPU CodeLLM serving.

desk verdict SABER is a solid engineering study of adaptive admission control for CodeLLM serving, but the headline numbers overshoot the data and the SLA setup is self-referential; with a corrected abstract and held-out evaluation it would be a reasonable systems paper. read the letter →

arxiv 2506.19677 v2 pith:RC4DBL6A submitted 2025-06-24 cs.SE

Shi Chang , Boyuan Chen , Kishanthan Thangarajah , Hanan Lutfiyya , Ahmed E. Hassan This is my paper · ORCID
classification cs.SE
keywords largelanguagemodelsrequestschedulingSLAcomplianceadaptivesystemscontinuousbatchingadmissioncontrolcodegenerationgoodput
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Continuous batching in LLM serving engines relies on a fixed maximum batch size, but no single fixed size stays optimal as the mix of coding tasks and arrival rate changes. SABER replaces that static knob with an admission-control layer that predicts, for each incoming request, whether it can finish before its SLA deadline without slowing already-running requests past theirs. On a single-GPU CodeLLM workload, the paper reports that SABER beats the best static configuration in goodput — the fraction of requests completed within their SLA — by up to 26%, with up to 45% lower latency variability, and does so without restarts or manual tuning. The paper's case is that SLA-aware adaptive scheduling, not better static tuning, is the path to reliable self-hosted CodeLLM serving.

What carries the argument

The load-bearing object is the estimation function $\hat{v}=f(L)$, a Universal Scalability Law curve fitted offline to measured pairs of concurrency and token-generation speed; it predicts the per-request generation speed when $L$ requests run together. The admission control loop uses $f(\text{current load} + 1)$ to test two conditions before letting a request join the batch: the predicted speed must cover the incoming request's required speed (max tokens divided by remaining time to deadline), and it must not push any already-executing request below its own required speed. A two-tier queue supports the loop: requests whose required speed already exceeds the engine's fastest speed are demoted to a low-priority best-effort queue, and a random window over the high-priority queue prevents head-of-line blocking.

What would settle it

Feed SABER a workload whose output-token distribution is deliberately shifted after the offline profile is fixed, for example all code-translation requests with double the profiled output length, or a mix that alternates between heavy and light tasks every few minutes. If goodput then falls below the best static configuration chosen for each phase, or SLA violations concentrate on requests the controller predicted would succeed, the claim that the concurrency-only estimator generalizes is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the failure of static continuous batching under heterogeneous CodeLLM loads is avoidable: a lightweight admission controller can match or beat the best static configuration. The authors show experimentally that the optimal maximum batch size shifts with workload composition (30 for heavy tasks, 70 for light tasks, 80 for a balanced mix) and with request rate, so no fixed setting can serve a production workload well. SABER instead fits a Universal Scalability Law curve to offline measurements of token-generation speed at different concurrency levels, then online admits a request only if the predicted generation speed at current load plus one is enough to meet both that request's required speed and the required speeds of requests already executing. Requests whose deadlines have become impossible are demoted to a best-effort low-priority queue. Under contention the policy becomes a deliberate triage: short tasks are accelerated, hopeless long tasks are postponed, and aggregate goodput rises while completion-time variability falls.

Load-bearing premise

Everything rests on the fitted speed estimator $f(L)$, which predicts token-generation rate from the number of concurrent requests alone; if a live workload mix behaves differently from the offline profile, the admission decisions will either admit requests that bust their SLAs or reject requests that would have made it.

Editorial extensions

If this is right

  • Operators no longer need to restart the serving engine to retune batch size when the workload shifts, because SABER adjusts admission decisions continuously.
  • Under heavy load, light interactive tasks such as short Q&A and summaries are protected at the expense of heavy long-generation tasks, matching what users tolerate waiting for.
  • The goodput advantage over the best static configuration grows with load, from near zero when capacity is ample to as much as 26 percentage points under saturation.
  • SABER can be added to an existing continuous-batching engine as an admission layer plus an offline profile, without changing the inference engine's internals.
  • The quality of the speed estimator determines the gain: replacing the fitted USL curve with less accurate linear or logistic regressors eroded or reversed the goodput advantage in light workloads.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's estimator uses only concurrency as input; feeding it additional features such as average sequence length or prefill/decode split could let the same admission rule survive workload drift outside the offline profile.
  • SABER's triage appears to optimize perceived responsiveness rather than raw SLA count; a user study could test whether deferring heavy tasks is experienced as better service, not just measured as higher goodput.
  • The single-GPU framing suggests the method could lift to multi-GPU serving by replacing $f(L)$ with a distributed cost model, a setting the paper explicitly leaves unexplored.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SABER, an adaptive request-scheduling layer for CodeLLM serving systems that use continuous batching. In an offline phase, SABER fits a Universal Scalability Law (USL) curve to measured per-request token-generation speed as a function of the number of concurrently executing requests. Online, a two-tier queue classifies requests by their required generation speed, and an admission-control loop admits a request only if the predicted speed at the increased concurrency is sufficient for both the incoming request and all active requests. The evaluation uses Qwen-Coder-2.5B on vLLM with three workload mixes, compares against the best static batch-size configuration for each request rate, and reports goodput, completion-time ratios, and coefficient-of-variation metrics. The paper claims up to 26% goodput improvement and up to 45% latency-variability reduction over the best static configurations.

Significance. If the claims hold, SABER is a useful and practical contribution: it is a lightweight, non-intrusive admission-control layer that can be added to existing serving engines, and it directly targets the realistic problem that a static batch-size configuration cannot adapt to changing request rates and workload mixes. The study has clear strengths: it performs a systematic sweep over 360 static configurations, uses the per-RPS best static configuration as a conservative oracle baseline, and includes a sensitivity analysis of the estimation function. The empirical comparison is the core of the paper and is directionally reasonable. However, several load-bearing points need attention before the headline claims can be accepted: the reported 45% variability-reduction figure is not supported by the CV data in the paper, the admission decision rests entirely on a single-variable speed estimator whose residual spread and out-of-distribution behavior are not validated, and the SLA thresholds are defined from the system's own mean completion times rather than from external requirements.

major comments (5)
  1. [Abstract; Section VI-C, Figure 6] The abstract states that SABER "reduces latency variability by up to 45%", but the CV values reported in Figure 6 do not support this number. The largest relative reduction is in Workload 1, from 52.4% to 33.7%, which is a 35.7% reduction; Workload 2 is 20.8% to 17.5% (15.9%) and Workload 3 is 36.4% to 25.1% (31.0%). The claim should be corrected to the largest observed reduction or an explicit computation of the 45% figure should be provided.
  2. [Section IV-B; Algorithm 1, lines 8-13] Every admission decision is based on f(L), a single-variable USL curve fitted to mean per-request token-generation speed as a function of concurrency. The paper reports R^2=0.99 for the fit but does not report residual spread, per-request speed variance at fixed L, or validation on held-out workload mixes. Since the cited work of Sun et al. [35] documents up to 2.6x speed variation with batch composition, a high R^2 on the mean curve does not establish that the admission predicate is safe for the specific prefill/decode mix being admitted. This is load-bearing because Section VII shows that replacing the estimator changes the Workload 2 average advantage from +1.2% to -5.3% (logistic) and -6.7% (linear); the headline goodput gains are therefore not established outside the calibration distribution.
  3. [Section IV-C, Step 2; Algorithm 1, line 10] The admission check compares the predicted speed against each active request's reqSpd, but reqSpd is recorded once at admission as maxTokens/(deadline - currentTime) and is never recomputed from the remaining number of tokens. An active request that has generated fewer tokens than its schedule requires can therefore have an actual remaining speed requirement higher than the stored reqSpd, and the algorithm will admit new requests that push it past its SLA without detecting the violation. The algorithm should update active-request requirements on each admission decision, or check the remaining tokens, for the SLA-guarantee claim to hold.
  4. [Section III-B] The SLA thresholds in Table I are not external requirements but the system's own mean completion times measured at 10 RPS on the same hardware and model. Because SABER is evaluated against thresholds derived from the same serving stack it controls, the goodput numbers measure compliance with a self-referential target. The paper should either anchor the SLA values to independent, user-defined requirements or explicitly state that the evaluation demonstrates relative improvement under a simulated SLA regime, not an absolute SLA guarantee.
  5. [Section V-B; Section VI-B] The evaluation appears to use the same task types, datasets, and SLA thresholds from which the offline profile of Section IV-B is built; the paper does not state that evaluation workloads are disjoint from calibration data. Without a held-out workload mix or an explicit statement of overlap, the goodput improvements cannot be separated from in-distribution calibration effects. A validation on workload compositions or request-length distributions not seen during profiling is needed.
minor comments (5)
  1. [Abstract] The abstract contains a grammatical error: "systems employs" should be "systems employ"; also, "We propose" is capitalized mid-sentence and should be "we propose" for consistency.
  2. [Figure 4] The Figure 4 caption says the CDF analysis is at RPS=10, but Section V-C reports comparisons at RPS=5 for Workload 1 and RPS=20 for Workload 2; the caption should match the text.
  3. [Throughout] The name is written inconsistently as "SABER" in the text and "Saber" in figures and some figure captions; one capitalization convention should be used throughout.
  4. [Section VI-B, Equation (1)] Equation (1) defines the coefficient of variation, but the text does not explicitly state that sigma and mu are computed over the per-request completion-time-to-SLA ratios across the different RPS values; this should be clarified.
  5. [Section III-B] Each configuration is evaluated with only 100 requests and three repetitions; reporting only the mean hides run-to-run variability, which is especially relevant for the goodput differences of a few percentage points in Figure 5. Confidence intervals or per-run points should be reported.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SABER's goodput and latency claims are empirically measured end-to-end; the fitted USL speed estimator is an intermediate design input, not the source of the headline numbers.

full rationale

Walking the derivation chain: SABER's only fitted component is the offline USL estimator f(L) (Section IV-B), which maps concurrency L to mean per-request token-generation speed. That estimator is used inside the online admission predicate (Algorithm 1, line 8) to decide whether admitting a request would violate SLAs. The paper's headline outcomes, goodput and latency coefficient of variation, are not algebraic consequences of f(L); they are measured end-to-end on a vLLM serving system in Sections V and VI. The claimed 26% goodput improvement is an observed comparison against static configurations, not a value derived from the fitted curve. The paper's own sensitivity analysis in Section VII replaces USL with logistic and linear fits and reports different goodput, which confirms that the estimator influences the result but does not show that the result is equivalent to the fit. The only self-referential design choice is the SLA calibration in Section III-B, where SLA thresholds are set to the system's own mean completion time at 10 RPS; this is a benchmark-construction limitation, not a circular derivation, because SABER's admission rule neither defines nor predicts those thresholds from the fitted curve. No load-bearing self-citations appear, and no uniqueness theorem or prior-work ansatz is invoked to force the design. I find no step in which a claimed prediction reduces to its fitting input by construction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central mechanism rests on a fitted USL curve, an assumed relationship between concurrency and per-request speed, and self-derived SLA thresholds. No new physical entities are introduced; SABER is a software policy layer over existing serving infrastructure.

free parameters (5)
  • USL curve parameters (sigma, kappa) = not reported
    Fitted to offline profiling data in Section IV-B; the admission decision uses f(L+1), so the central mechanism's accuracy depends on this fit.
  • windowSize (n in Algorithm 1) = not reported
    Used in Step 2 to sample from the queue head; value never specified in the paper.
  • sleep interval delta in Algorithm 1 = not reported
    Controls the admission control loop's frequency; value never specified.
  • SLA thresholds per task = 1s, 8s, 1s, 12s for the four tasks
    Derived as the mean completion time at 10 RPS for 100 requests in Section III-B, not from external requirements.
  • Max generated tokens per request = estimated from workload statistics
    Used to compute required speed in Step 1; the paper cites average token estimation but does not evaluate the impact of estimation error on admission decisions.
assumptions (5)
  • domain assumption Universal Scalability Law (USL) is an appropriate model for per-request token generation speed under concurrency
    Adopted in Section IV-B to fit f(L); selected over linear and logistic by R2, but no theoretical justification is given for its applicability to LLM decode speed.
  • domain assumption Per-request token generation speed depends only on the number of concurrent requests (L), not on batch composition
    The fitted f(L) is a single scalar function; admission decisions in Algorithm 1 use only f(L+1) and per-request required speed, ignoring batch composition and request types.
  • domain assumption The three synthetic workload mixes (W1-W3) and Poisson arrivals are representative of real CodeLLM usage
    Stated in Sections III-B and V-B; the threats section acknowledges they may not reflect actual user traces.
  • domain assumption The maximum number of generated tokens per request can be estimated in advance
    Used in Step 1 (Queue Placement) to compute required speed; the paper cites ShareGPT for average token estimation but does not evaluate estimation error.
  • standard math Standard statistics and curve fitting (scipy, R2) are used correctly
    Section IV-B Step 2; no formal proof of the fit quality beyond R2 values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Request Scheduling for CodeLLM Serving with SLA Guarantees." pith.science (2026). https://pith.science/paper/RC4DBL6A

@misc{pith2026250619677,
  author       = {Pith},
  title        = {Pith review of: Adaptive Request Scheduling for CodeLLM Serving with SLA Guarantees},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RC4DBL6A}},
  note         = {Machine review of arXiv:2506.19677}
}
read the original abstract

Code Large Language Models (CodeLLMs) are increasingly integrated into modern software development workflows, yet efficiently serving them in resource-constrained, self-hosted environments remains a significant challenge. Existing LLM serving systems employs Continuous Batching for throughput improvement. However, they rely on static batch size configurations that cannot adapt to fluctuating request rates or heterogeneous workloads, leading to frequent SLA (Service Level Agreement) violations and unstable performance. In this study, We propose SABER, a dynamic batching strategy that predicts per-request SLA feasibility and adjusts decisions in real time. SABER improves goodput by up to 26% over the best static configurations and reduces latency variability by up to 45%, all without manual tuning or service restarts. Our results demonstrate that SLA-aware, adaptive scheduling is key to robust, high-performance CodeLLM serving.

Figures

Figures reproduced from arXiv: 2506.19677 by the authors.

Figure 1
Figure 1. Goodput across different batch sizes and request rates. Each heatmap shows the percentage of requests meeting their [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The SABER workflow. Offline: profiles workloads and trains estimation function f(L) for token generation speed prediction. Online: uses f(L) to admit requests that won’t violate existing SLAs, deferring others to maintain system performance. IV. OUR APPROACH This section presents SABER, our approach towards SLA￾aware continuous batching. Section IV-A provides an overview of the approach. Section IV-B details the Off… view at source ↗
Figure 3
Figure 3. Goodput comparison between static concurrency configurations and Saber across different request rates. Each subfigure [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Cumulative Distribution Function (CDF) analysis of request completion times at RPS=10. Each subplot shows [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Goodput comparison between SABER and best static configurations across varying loads. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance stability analysis across varying request loads. Lines show mean completion time normalized by SLA [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Comparative analysis of goodput across four settings: SABER-USL, SABER-Logistic, SABER-Linear, and the best [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 31 canonical work pages

  1. [35]

    Llumnix: Dynamic scheduling for large language model serving,

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y . Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,”arXiv preprint arXiv:2406.03243, 2024

    arXiv 2024

  2. [1]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    arXiv 2023

  3. [2]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Liet al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,”arXiv preprint arXiv:2401.14196, 2024

    arXiv 2024

  4. [3]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Danget al., “Qwen2. 5-coder technical report,”arXiv preprint arXiv:2409.12186, 2024

    arXiv 2024

  5. [4]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Weiet al., “Starcoder 2 and the stack v2: The next generation,”arXiv preprint arXiv:2402.19173, 2024

    arXiv 2024

  6. [5]

    Fine tuning large language model for secure code generation,

    J. Li, A. Sangalay, C. Cheng, Y . Tian, and J. Yang, “Fine tuning large language model for secure code generation,” inProceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering, 2024, pp. 86–90

    work page 2024

  7. [6]

    Repohyper: Search-expand-refine on semantic graphs for repository-level code com- pletion,

    H. N. Phan, H. N. Phan, T. N. Nguyen, and N. D. Bui, “Repohyper: Search-expand-refine on semantic graphs for repository-level code com- pletion,”arXiv preprint arXiv:2403.06095, 2024

    arXiv 2024

  8. [7]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,”arXiv preprint arXiv:2406.00515, 2024

    arXiv 2024

Show all 55 references
  1. [8]

    Language models for code completion: A practical evaluation,

    M. Izadi, J. Katzy, T. Van Dam, M. Otten, R. M. Popescu, and A. Van Deursen, “Language models for code completion: A practical evaluation,” inProceedings of the IEEE/ACM 46th International Con- ference on Software Engineering, 2024, pp. 1–13

    2024

  2. [9]

    Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation,

    V . Murali, C. Maddila, I. Ahmad, M. Bolin, D. Cheng, N. Ghorbani, R. Fernandez, N. Nagappan, and P. C. Rigby, “Ai-assisted code authoring at scale: Fine-tuning, deploying, and mixed methods evaluation,” 2024. [Online]. Available: https://arxiv.org/abs/2305.12050

    2024 arXiv

  3. [10]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,

    Y . Yao, J. Duan, K. Xu, Y . Cai, Z. Sun, and Y . Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,”High-Confidence Computing, p. 100211, 2024

    2024

  4. [11]

    Security and privacy challenges of large language models: A survey,

    B. C. Das, M. H. Amini, and Y . Wu, “Security and privacy challenges of large language models: A survey,”ACM Computing Surveys, vol. 57, no. 6, pp. 1–39, 2025

    2025

  5. [12]

    Hierarchical repository-level code summarization for business applications using local llms,

    N. Dhulshette, S. Shah, and V . Kulkarni, “Hierarchical repository-level code summarization for business applications using local llms,”arXiv preprint arXiv:2501.07857, 2025

    2025 arXiv

  6. [13]

    Using ollama,

    F. S. Marcondes, A. Gala, R. Magalh ˜aes, F. Perez de Britto, D. Dur ˜aes, and P. Novais, “Using ollama,” inNatural Language Analytics with Generative Large-Language Models: A Practical Approach with Ollama and Open-Source LLMs. Springer, 2025, pp. 23–35

    2025

  7. [14]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 611–626

    2023

  8. [15]

    Efficiently programming large language models using sglang

    L. Zheng, L. Yin, Z. Xie, J. Huang, C. Sun, C. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalezet al., “Efficiently programming large language models using sglang.” 2023

    2023

  9. [16]

    C. Daniel. (2023, 6) Achieve 23x llm inference throughput & reduce p50 latency. [Online]. Available: https://www.anyscale.com/ blog/continuous-batching-llm-inference

    2023

  10. [17]

    Orca: A distributed serving system for{Transformer-Based}generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for{Transformer-Based}generative models,” in16th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 22), 2022, pp. 521–538

    2022

  11. [18]

    Llumnix: Dynamic scheduling for large language model serving,

    B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y . Li, and W. Lin, “Llumnix: Dynamic scheduling for large language model serving,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 173–191. [Onl...

    2024

  12. [19]

    Batch: Machine learn- ing inference serving on serverless platforms with adaptive batching,

    A. Ali, R. Pinciroli, F. Yan, and E. Smirni, “Batch: Machine learn- ing inference serving on serverless platforms with adaptive batching,” inSC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2020, pp. 1–15

    2020

  13. [20]

    W. Cui, H. Zhao, Q. Chen, H. Wei, Z. Li, D. Zeng, C. Li, and M. Guo, in 2022 USENIX Annual Technical Conference (USENIX ATC 22), 2022, pp. 183–198

    2022

  14. [21]

    Efficient deep neural network serving: Fast and furious,

    F. Yan, Y . He, O. Ruwase, and E. Smirni, “Efficient deep neural network serving: Fast and furious,”IEEE Transactions on Network and Service Management, vol. 15, no. 1, pp. 112–126, 2018

    2018

  15. [22]

    A comprehensive review of yolo architectures in computer vision: From yolov1 to yolov8 and yolo-nas,

    J. Terven, D.-M. C ´ordova-Esparza, and J.-A. Romero-Gonz ´alez, “A comprehensive review of yolo architectures in computer vision: From yolov1 to yolov8 and yolo-nas,”Machine learning and knowledge extraction, vol. 5, no. 4, pp. 1680–1716, 2023

    2023

  16. [23]

    Survey of uncertainty estimation in large language models-sources, methods, applications, and challenge,

    J. He, L. Yu, C. Li, R. Yang, F. Chen, K. Li, M. Zhang, S. Lei, X. Zhang, M. Beigiet al., “Survey of uncertainty estimation in large language models-sources, methods, applications, and challenge,” 2025

    2025

  17. [24]

    Enabling efficient batch serving for lmaas via generation length prediction,

    K. Cheng, W. Hu, Z. Wang, P. Du, J. Li, and S. Zhang, “Enabling efficient batch serving for lmaas via generation length prediction,” in 2024 IEEE International Conference on Web Services (ICWS). IEEE, 2024, pp. 853–864

    2024

  18. [25]

    [performance]: [v1] increasing the request batch size causes a significant drop in performance,

    vllm project, “[performance]: [v1] increasing the request batch size causes a significant drop in performance,” GitHub issue #10709, 2025, accessed: 2025-05-07. [Online]. Available: https: //github.com/vllm-project/vllm/issues/10709

    2025

  19. [26]

    [performance]: Added request take too much time, and the model will not run untill all the request are added into the cache,

    ——, “[performance]: Added request take too much time, and the model will not run untill all the request are added into the cache,” GitHub issue #13259, 2025, accessed: 2025-05-07. [Online]. Available: https://github.com/vllm-project/vllm/issues/13259

    2025

  20. [27]

    [performance]: Why does the tpot increase with the request rate increase?

    ——, “[performance]: Why does the tpot increase with the request rate increase?” GitHub issue #13084, 2025, accessed: 2025-05-07. [Online]. Available: https://github.com/vllm-project/vllm/issues/13084

    2025

  21. [28]

    [performance]: Ttft spikes when qps increases during deepseek- r1 testing with tp8 and pp2,

    ——, “[performance]: Ttft spikes when qps increases during deepseek- r1 testing with tp8 and pp2,” GitHub issue #13610, 2025, accessed: 2025-05-07. [Online]. Available: https://github.com/vllm-project/vllm/ issues/13610

    2025

  22. [29]

    [performance]: poor performance in pipeline parallesm when batch-size is large,

    ——, “[performance]: poor performance in pipeline parallesm when batch-size is large,” GitHub issue #15330, 2025, accessed: 2025-05-07. [Online]. Available: https://github.com/vllm-project/vllm/issues/15330

    2025

  23. [30]

    [performance]: How to improve performance under concurrency,

    ——, “[performance]: How to improve performance under concurrency,” GitHub issue #9722, 2025, accessed: 2025-05-07. [Online]. Available: https://github.com/vllm-project/vllm/issues/9722

    2025

  24. [31]

    {SHEPHERD}: Serv- ing{DNNs}in the wild,

    H. Zhang, Y . Tang, A. Khandelwal, and I. Stoica, “{SHEPHERD}: Serv- ing{DNNs}in the wild,” in20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023, pp. 787–808

    2023

  25. [32]

    Serving{DNNs}like clockwork: Performance predictability from the bottom up,

    A. Gujarati, R. Karimi, S. Alzayat, W. Hao, A. Kaufmann, Y . Vig- fusson, and J. Mace, “Serving{DNNs}like clockwork: Performance predictability from the bottom up,” in14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), 2020, pp. 443–462

    2020

  26. [33]

    {INFaaS}: Automated model-less inference serving,

    F. Romero, Q. Li, N. J. Yadwadkar, and C. Kozyrakis, “{INFaaS}: Automated model-less inference serving,” in2021 USENIX Annual Technical Conference (USENIX ATC 21), 2021, pp. 397–411

    2021

  27. [34]

    Llama: A heterogeneous & serverless framework for auto-tuning video analytics pipelines,

    F. Romero, M. Zhao, N. J. Yadwadkar, and C. Kozyrakis, “Llama: A heterogeneous & serverless framework for auto-tuning video analytics pipelines,” inProceedings of the ACM symposium on cloud computing, 2021, pp. 1–17

    2021

  28. [36]

    {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 193–210

    2024

  29. [37]

    Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills,

    A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee, “Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills,”arXiv preprint arXiv:2308.16369, 2023

    2023 arXiv

  30. [38]

    Inference without interference: Disaggre- gate llm inference for mixed downstream workloads,

    C. Hu, H. Huang, L. Xu, X. Chen, J. Xu, S. Chen, H. Feng, C. Wang, S. Wang, Y . Baoet al., “Inference without interference: Disaggre- gate llm inference for mixed downstream workloads,”arXiv preprint arXiv:2401.11181, 2024

    2024 arXiv

  31. [39]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, ´I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 118–132

    2024

  32. [40]

    Exegpt: Constraint-aware resource scheduling for llm inference,

    H. Oh, K. Kim, J. Kim, S. Kim, J. Lee, D. seong Chang, and J. Seo, “Exegpt: Constraint-aware resource scheduling for llm inference,” 2024. [Online]. Available: https://arxiv.org/abs/2404.07947

    2024 arXiv

  33. [41]

    Niyama : Breaking the silos of llm inference serving,

    K. Goel, J. Mohan, N. Kwatra, R. S. Anupindi, and R. Ramjee, “Niyama : Breaking the silos of llm inference serving,” 2025. [Online]. Available: https://arxiv.org/abs/2503.22562

    2025 arXiv

  34. [42]

    Hl-codellama-chat-response dataset,

    Magnifi, “Hl-codellama-chat-response dataset,” 2025, accessed: 2025- 01-12. [Online]. Available: https://huggingface.co/datasets/magnifi/ hl-codellama-chat-response

    2025

  35. [43]

    Synthetic code generations dataset,

    VishaalY , “Synthetic code generations dataset,” 2025, accessed: 2025- 01-12. [Online]. Available: https://huggingface.co/datasets/VishaalY/ synthetic-code-generations

    2025

  36. [44]

    Code summary java dataset,

    Samaxr, “Code summary java dataset,” 2025, accessed: 2025- 01-12. [Online]. Available: https://huggingface.co/datasets/samaxr/ code-summary-java/tree/main

    2025

  37. [45]

    Code translation dataset,

    CodeTranslatorLLM, “Code translation dataset,” 2025, accessed: 2025-01-12. [Online]. Available: https://huggingface.co/datasets/ CodeTranslatorLLM/Code-Translation

    2025

  38. [46]

    Learned best-effort llm serving,

    S. Jha, C. Hooper, X. Liu, S. Kim, and K. Keutzer, “Learned best-effort llm serving,” 2024. [Online]. Available: https://arxiv.org/abs/2401.07886

    2024 arXiv

  39. [47]

    Preble: Efficient distributed prompt scheduling for llm serving,

    V . Srivatsa, Z. He, R. Abhyankar, D. Li, and Y . Zhang, “Preble: Efficient distributed prompt scheduling for llm serving,” 2024. [Online]. Available: https://arxiv.org/abs/2407.00023

    2024 arXiv

  40. [48]

    Past-future scheduler for llm serving under sla guarantees,

    R. Gong, S. Bai, S. Wu, Y . Fan, Z. Wang, X. Li, H. Yang, and X. Liu, “Past-future scheduler for llm serving under sla guarantees,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2025,...

    2025

  41. [49]

    ShareGPT Dataset: A Collection of ChatGPT Conversations,

    ShareGPT Community, “ShareGPT Dataset: A Collection of ChatGPT Conversations,” https://sharegpt.com, 2023, online platform for sharing ChatGPT conversations

    2023

  42. [50]

    Integrating concurrency control in n-tier application scaling management in the cloud,

    Q. Wang, H. Chen, S. Zhang, L. Hu, and B. Palanisamy, “Integrating concurrency control in n-tier application scaling management in the cloud,”IEEE Transactions on Parallel and Distributed Systems, vol. 30, no. 4, pp. 855–869, 2018

    2018

  43. [51]

    An r-square coefficient based on final prediction error,

    V . Rousson and N. F. Gos ¸oniu, “An r-square coefficient based on final prediction error,”Statistical Methodology, vol. 4, no. 3, pp. 331–340, 2007

    2007

  44. [52]

    Scipy 1.0: fundamental algorithms for scientific computing in python,

    P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Brightet al., “Scipy 1.0: fundamental algorithms for scientific computing in python,” Nature methods, vol. 17, no. 3, pp. 261–272, 2020

    2020

  45. [53]

    Multi-dimensional sla-based resource allocation for multi-tier cloud computing systems,

    H. Goudarzi and M. Pedram, “Multi-dimensional sla-based resource allocation for multi-tier cloud computing systems,” in2011 IEEE 4th International Conference on Cloud Computing, 2011, pp. 324–331

    2011

  46. [54]

    Decision model for cloud comput- ing under sla constraints,

    A. Andrzejak, D. Kondo, and S. Yi, “Decision model for cloud comput- ing under sla constraints,” in2010 IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, 2010, pp. 257–266

    2010

  47. [55]

    When average is not average: large response time fluctuations in n-tier systems,

    Q. Wang, Y . Kanemasa, M. Kawaba, and C. Pu, “When average is not average: large response time fluctuations in n-tier systems,” inProceed- ings of the 9th international conference on Autonomic computing, 2012, pp. 33–42

    2012

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.