Clairvoyant: Predictive Shortest-Job-First Admission for Serial LLM Inference

Aravind Sundaresan

REVIEW 3 major objections 3 minor 20 references

Clairvoyant: Predictive Shortest-Job-First Admission for Serial LLM Inference

T0 review · 3 major / 3 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Clairvoyant shows that a sidecar proxy using lexical length prediction can impose shortest-job-first admission on serial LLM backends, cutting short-request P50 latency by 70–76% in bursts while adding only 0.029 ms per request.

desk verdict The dataset-bias finding and the sidecar design are real, but the headline latency numbers do not survive contact with the paper's own predictor measurements, and the benchmark's ordering source has to be verified before any of the P50 claims can be trusted. read the letter →

arxiv 2606.07248 v2 pith:7VRENOHD submitted 2026-06-05 cs.DC

Aravind Sundaresan This is my paper

classification cs.DC

keywords head-of-lineblockingshortest-job-firstschedulingserialLLMinferenceoutputlengthpredictionlexicalfeaturesrankingfidelitystarvationtimeoutqueueingtheory

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Clairvoyant sets out to prove that serial LLM inference backends need not stay with first-come-first-served admission: a proxy that predicts response length from lexical features and dispatches shortest jobs first can remove most head-of-line blocking without the memory cost of continuous batching. The paper reports 70–76% short-request P50 latency reductions under a 100-request burst, 17% at 0.74 utilization, an 83.6% time-to-first-token cut in a trace replay, and 0.029 ms predictor overhead. The second claimed discovery is that curated instruction datasets are structurally unusable for training length predictors because under 0.02% of their responses are long enough to learn from; only natural conversation logs supply the needed diversity. The contribution is an admission-layer patch for the large class of memory-constrained deployments where batched serving does not fit.

What carries the argument

The load-bearing object is the per-request predicted probability of a long response, produced by a 19-feature tree-based classifier (prompt-length proxy, code/format/length-constraint flags, clause count, and a one-hot instruction verb) exported to a portable runtime format. Requests sit in a min-heap keyed on this score, so lower predicted length dispatches first. Because only relative order matters, the paper optimizes pairwise ranking accuracy—the fraction of short/long pairs correctly ordered—rather than 3-class classification. A starvation timer set to three times the empirically measured mean short-request sojourn time on the target hardware promotes long-waiting requests to the head o

What would settle it

Re-run the 100-request burst benchmark while logging the predicted score and dispatch order for every request; if the order tracks the dataset's short/long labels but not the predictor's scores, compute latency under the predictor's own ordering and compare.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that a serial backend's admission queue can be converted into a shortest-job-first queue using nothing more than 19 lightweight lexical features. The scheduler keys on predicted probability of a long response rather than an exact token count, because admission only needs correct pairwise ordering. Trained on natural conversation logs, the predictor reaches 62–96% in-distribution and 52–66% cross-distribution pairwise ranking accuracy, and the end-to-end benchmarks show the reported latency reductions under closed-loop bursts and steady-state arrivals. The paper also claims that curated instruction datasets are degenerate for this task: they cont

Load-bearing premise

The headline latency reductions presuppose that the dispatcher actually ordered the queue by the predictor's scores; the paper reports only 52.7–58.4% ranking accuracy on the benchmark dataset and never reports accuracy on the exact subset used, so if the benchmark's known short/long labels drove the ordering, the central latency claim collapses.

Editorial extensions

If this is right

Memory-constrained edge and local deployments can eliminate most admission-layer head-of-line blocking without the large KV-cache memory that concurrent batching requires.
A drop-in proxy, not a backend rewrite, is sufficient: the inference engine sees the same requests, just in a different order.
The scheduling layer is nearly free: 0.029 ms of predictor overhead versus multi-second generation times, so it can run on the same host without measurable throughput cost.
The operational envelope is narrow: measurable benefit appears mainly in the 0.55–0.80 utilization band, with gains below 3% at low load and with the starvation timeout eroding gains above 0.85.
Any team training a length-predictive scheduler must use natural conversation logs; instruction datasets will not supply enough long-response examples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

I infer that the reported burst gains are an upper bound: cross-distribution ranking accuracy on the benchmark's instruction-dataset subset is only 52.7–58.4% in the paper's own table, so a cold deployment outside the training distribution would see much smaller gains unless the dispatcher benefited from label-derived ordering.
The dataset-starvation finding is separable from the scheduling claim: even if the latency numbers are soft, the discovery that curated instruction corpora contain under 0.02% long responses constrains any length-predictive scheduler, lexical or embedding-based.
The same sidecar architecture could be extended to per-tenant fairness by adding tenant weights to the priority key; the starvation timer already bounds worst-case delay, so it would not break the core mechanism.
A testable extension is to drive the queue directly from the continuous predicted score rather than class-boundary keying; the paper's own ranking metric assumes the continuous score is the right signal.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing to know: this paper has a genuinely useful negative result and a plausible lightweight architecture, but the central burst-latency claims are internally inconsistent. The abstract promises TTFT and M1 reductions that the body explicitly says were not measured, and Table 8's SJF gains require near-perfect short/long ordering while the paper's own Dolly ranking accuracy is 52–58%. Read the body, not the abstract.

What is actually new: the packaging. A zero-modification sidecar proxy that uses 19 lexical features and an ONNX-exported XGBoost ranker to do SJF admission for serial OpenAI-compatible backends is a real niche, and the 0.029 ms/request overhead is well below generation time. The dataset-starvation finding—Alpaca and CodeAlpaca have under 0.02% Long-class examples—is solid and independently useful for anyone working on length-predictive scheduling. The paper is also honest in its limitations section and ships reproducible code at a pinned commit. The ranking-accuracy metric (Eq. 2) is the right one for scheduling, and the ablation study is thorough.

The soft spots are load-bearing, though. First, the abstract's 83.6% TTFT reduction is contradicted by Section 7 (“Time-to-First-Token is not separately instrumented”) and by Section 5.6 listing TTFT instrumentation as future work. The 69.7% M1 figure is an n=8 dispatch-logic test, not a latency benchmark. Those are not minor slip-ups; they are headline claims that the paper itself says were never measured.

Second, and more serious, is the Table 8 ordering problem. Under a 50/50 burst, short P50 dropping from 229.5 s to 69.1 s is exactly what perfect short-before-long dispatch produces. But the trained predictors achieve only 52.7–58.4% cross-distribution ranking accuracy on Dolly (Table 6). A scheduler driven by a ~55%-accurate ranker would leave many Long jobs ahead of the median Short, so the observed short P50 should be far worse. The paper never states whether the benchmark's dispatch order came from the predictor or from the harness's known Short/Long labels. If the latter, Table 8 validates an oracle SJF, not Clairvoyant. The stress-test note is correct to flag this; it does not hold up only if the predictor magically performed far better on that specific Dolly subset than on the Dolly test set in Table 6, and no such evidence is provided.

Minor but real: tau is stated as 10.5 s in Figure 3 and 15 s in Section 3.4 for the same hardware and rule, and the steady-state simulation uses tau chosen as that same simulation's Pareto elbow—a small calibration loop.

The core idea is plausible and worth pursuing. The dataset finding stands on its own. But as submitted, the paper asserts results it explicitly says it did not measure, and the central burst result has an unverified mechanism. I would not trust the latency numbers until the authors confirm that Table 8's dispatch order was produced by the predictor and report predictor accuracy on the exact benchmark subset, and they must reconcile the abstract with the body.

Who is this for? Anyone building lightweight admission control for edge/local LLM serving. It deserves a serious referee, not because the claims are credible but because the niche is real and the flaws are fixable with a benchmark rerun and a rewritten abstract. My advice: send it to peer review with a strong request to verify the dispatch-order source and to correct the abstract, but treat the current version as a promising preprint, not a measured result.

Referee Report

3 major / 3 minor

Summary. The paper presents Clairvoyant, a sidecar proxy that applies non-preemptive Shortest-Job-First admission to serial OpenAI-compatible LLM backends. Output length is predicted from 19 lexical features by an ONNX-exported XGBoost classifier, and requests are dispatched in ascending order of predicted P(Long), with a starvation timeout tau = 3 x mu_short. The paper reports in-distribution ranking accuracy of 62-96%, cross-distribution ranking accuracy of 52-66%, and end-to-end GPU measurements on an RTX 4090 showing 70-76% short-request P50 latency reduction under burst conditions and 17% reduction under steady-state Poisson arrivals at rho=0.74. The abstract additionally claims an 83.6% TTFT reduction on a GCP L4 trace replay and a 69.7% reduction on Apple M1. The authors also report a dataset-bias finding: curated instruction datasets contain almost no Long-class examples, making natural conversation logs the only viable training signal.

Significance. If substantiated, the system would be a valuable low-overhead mechanism for a real, underserved regime: serial edge deployments where continuous batching is infeasible. The 0.029 ms per-request predictor latency, the open-source artifact, and the systematic documentation of Long-class starvation in common instruction datasets are genuine strengths. However, the central end-to-end latency claims are not currently supported: the abstract reports measurements the body explicitly disclaims, and the burst benchmark is only consistent with near-perfect Short/Long separation, which is contradicted by the paper's own cross-distribution ranking accuracy on the benchmark dataset. As written, the paper does not establish that Clairvoyant's predictor, rather than the benchmark harness's known labels, produced the dispatch order.

major comments (3)

[Abstract, Section 7, Section 5.6] The abstract claims an 83.6% TTFT reduction on a GCP L4 trace replay (rho=0.80) and a 69.7% reduction on Apple M1. These are contradicted by the body. Section 7 states 'Time-to-First-Token is not separately instrumented,' and Section 5.6 lists TTFT instrumentation as a 'planned extension' for a future v2. The Apple M1 evidence is explicitly an n=8 dispatch-logic test (Sections 1 and 5.4), not a queue-level latency measurement. The GCP L4 trace replay appears nowhere in the experimental sections of the manuscript. These abstract claims report results the paper itself says were not measured and must be removed or substantiated with data.
[Section 5.4, Table 8, Table 6] The 100-request burst result is internally inconsistent with the paper's own predictor measurements unless the dispatch order was produced by the benchmark harness's known Short/Long labels. With 50 Short and 50 Long concurrent requests, a Short P50 of 69.1 s (Gemma3:4b) or 38.0 s (Llama3.1:8b) is what near-perfect Short-before-Long ordering produces. But Table 6 reports Dolly ranking accuracy of only 52.7% (Model A), 58.4% (Model B), and 57.7% (Model C) — barely above the 50% random baseline defined by Eq. 2. At approximately 55% pair ranking, roughly half the Long jobs would precede the median Short, giving a far larger P50 than Table 8 reports. Section 5.4 says only that the requests were 'drawn from the Dolly 15K test split' and dispatched via profiler/benchmark.py; it does not state whether the dispatch order was produced by the ONNX predictor's P(Long) scores or by the script's kno
[Section 5.5, Section 3.4] The steady-state 17% improvement claim is the output of a calibration loop. Section 5.5 selects tau = 3 x mu_short as 'the elbow' of the same discrete-event simulation whose service-time distributions are calibrated to the same RTX 4090 measurements, and then the same DES reports the 17% short-P50 improvement and the 17% long-P95 bound. This is not an out-of-sample evaluation of the starvation-timeout policy; it is a fitted result. The DES also uses normally distributed service times, a simplifying assumption acknowledged in Section 7. The steady-state number should be presented as a calibration result, and its robustness should be checked with a held-out arrival trace or a sensitivity analysis over service-time distributions rather than only over tau.

minor comments (3)

[Section 3.4 and Figure 3] Section 3.4 states 'mu_short approximately 3.5s yields tau = 15s' for the RTX 4090, but 3 x 3.5 = 10.5 s, and the Figure 3 caption says tau = 3 x mu_short = 10.5 s. The inconsistency should be corrected.
[Section 7] The limitation paragraph says 'The token-length proxy (len(response) // 4) diverges from true BPE counts,' but the feature defined in Section 3.2 is prompt_token_len = len(prompt) // 4. The manuscript should clarify whether the proxy applies to prompts, responses, or both.
[Table 2] OASST1 row mixes exact and estimated counts, with the footnote saying some counts are derived from the 6.3% Long rate. Since the paper otherwise emphasizes reproducibility via commit 39ad9a64, reporting the exact filtered counts would be preferable.

Circularity Check

1 steps flagged · score 3.0 of 10

Steady-state 17% gain is selected from the same simulation used to tune τ; burst latency and ranking results remain independent, so circularity is partial.

fitted input called prediction [Sections 3.4 and 5.5, Table 9, Figure 3]
"We select τ = 3×µ short based on Pareto analysis of the short/long latency tradeoff (Section 5.5). At this threshold, short-request P50 latency improves by 17% over FCFS while bounding long-request P95 inflation to 17%—the elbow of the Pareto frontier."

The same discrete-event simulation (Poisson arrivals at ρ≈0.74, service times N(3.5s,0.8s)/N(8.9s,2.0s) fit to the RTX 4090) is used both to pick τ=3×µ_short as the Pareto elbow and to report the headline 17% short-P50 steady-state improvement and the 0.55–0.80 deployment window. The reported operating point is the result of the selection rule, not an out-of-sample prediction; the steady-state benefit is partly constructed by the calibration loop. This does not affect Table 8's separate end-to-end burst measurements or the held-out ranking evaluations.

full rationale

Most of the derivation chain is self-contained. Held-out ranking accuracy (Tables 5-6) and the burst latency reductions (Table 8) do not reduce by definition to the paper's inputs. I found no load-bearing self-citations or imported uniqueness/ansatz. The one partial circularity is the τ=3×µ_short calibration: the steady-state 17% gain and deployment boundary are read off the same DES that was used to select τ as the 'elbow', making that result a selected operating point. I do not treat the Dolly dispatch-order ambiguity as circularity because the paper never states whether benchmark.py ordered the 100 requests by the ONNX predictor or by ground-truth Short/Long labels; that is a verification gap. Abstract claims of 83.6% TTFT and 69.7% M1 reductions are contradicted by Section 7 ('Time-to-First-Token is not separately instrumented') and Section 5.6/3.4 (M1 evidence is an n=8 dispatch-logic test), and the internal mismatch between Table 6's 52.7-58.4% Dolly ranking accuracy and Table 8's near-optimal SJF behavior is a correctness risk; neither is a demonstrated circular step. Overall score 3 reflects one supporting prediction fitted to its tuning input while the central hardware benchmark remains independent.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

No new physical entities, particles, or mediators are introduced; the 3-class length taxonomy, min-heap scheduler, and sidecar proxy are engineering artifacts. The paper's empirical claims rest on several hand-set constants: class boundaries, XGBoost hyperparameters, the 19-feature taxonomy, the tau=3x mu_short rule calibrated on the same hardware where gains are measured, and the DES service-time distributions fitted to RTX 4090 measurements. The only external anchors are standard M/G/1 formulas. That is a lot of calibration for a paper whose headline is an empirical latency claim.

free parameters (6)

Class boundaries (Short<200, Medium 200-800, Long>=800 tokens) = 200 / 800 token thresholds
Ad hoc thresholds defining the 3-class length taxonomy that the scheduler's priority key and the benchmark classes depend on (Section 4.1).
XGBoost hyperparameters = 300 estimators, max depth 6, lr 0.1, seed 42
Fixed across all training runs without a tuning analysis (Section 4.3).
Starvation timeout tau = 3 x mu_short = M1: 120s; RTX 4090: 10.5s (Fig 3) or 15s (Sec 3.4)
Calibrated empirically from mu_short measured under burst conditions on the target hardware; the paper uses both 10.5s and 15s for the same hardware, and the 17% steady-state result is reported at this calibrated value (Sections 3.4, 5.5).
DES service-time distributions = Short N(3.5s, 0.8s); Long N(8.9s, 2.0s); lambda=0.12/s
Fitted to RTX 4090 Gemma3:4b measurements; the rho=0.74 steady-state curves and the 0.55-0.80 deployment boundary derive from these fitted inputs (Section 5.5, Figure 3).
prompt_token_len = len(prompt)//4 = n/a
Hand-chosen BPE proxy; the paper acknowledges in Section 7 that it diverges for code-heavy, multilingual, or symbol-dense inputs.
19-feature set including 13 verb categories = n/a
Hand-designed feature and verb taxonomy; ablation (Section 4.4) shows has_format_keyword and clause_count are net-harmful yet retained.

assumptions (6)

standard math M/G/1 FCFS mean-waiting formula (P-K): W = rho E[S](1+Cs^2) / 2(1-rho) (Eq. 1)
Used in Section 2.4 to motivate SJF; standard queueing theory.
standard math Non-preemptive SJF minimizes mean waiting time among non-preemptive work-conserving policies
Invoked in Section 2.2, citing Kleinrock [8].
standard math SRPT is optimal for mean sojourn time in M/G/1
Motivates the SJF choice in Section 2.2, citing Schrage [14].
domain assumption Approximate ranking suffices: cross-class (Short/Long) separation captures the dominant HOLB cost even with within-class misordering
Structural argument in Section 2.4; not proven and the basis for accepting 62-96% ranking accuracy as sufficient.
domain assumption Serial backends (Ollama/llama.cpp) dispatch FCFS with NUM_PARALLEL=1 on edge hardware and cannot afford concurrent KV-caches
Scope premise of Section 2.3; defines the deployment regime the whole paper targets.
domain assumption The deployed predictor's ranking accuracy on the live/benchmark workload is high enough to drive the reported gains
Cross-distribution accuracy is only 52-66% (Section 5.2); the end-to-end gain assumes production-level fidelity, which Table 6's Dolly column contradicts for the benchmark dataset.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Clairvoyant: Predictive Shortest-Job-First Admission for Serial LLM Inference." pith.science (2026). https://pith.science/paper/7VRENOHD

@misc{pith2026260607248,
  author       = {Pith},
  title        = {Pith review of: Clairvoyant: Predictive Shortest-Job-First Admission for Serial LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VRENOHD}},
  note         = {Machine review of arXiv:2606.07248}
}

read the original abstract

Serial LLM inference backends process requests sequentially under First-Come-First-Served (FCFS) admission, causing Head-of-Line Blocking (HOLB) under mixed workloads: short factual queries can be delayed by minutes behind long generation jobs. While cloud-scale deployments mitigate HOLB via continuous batching (e.g., vLLM, Orca), these solutions require tens of gigabytes of VRAM for concurrent KV-caches, rendering them infeasible for memory-constrained edge and local deployments that rely on serial request dispatch. We present Clairvoyant, a drop-in sidecar proxy for serial OpenAI-compatible backends (e.g., Ollama, llama.cpp) that implements predictive Shortest-Job-First (SJF) admission. Clairvoyant predicts response length using 19 lightweight lexical features via an ONNX-exported XGBoost classifier, achieving 0.029 ms per-request latency. Because admission scheduling relies on relative ranking rather than exact token prediction, Clairvoyant captures over 95% of the ranking fidelity of fine-tuned transformers at a fraction of the computational cost. We also uncover a critical dataset bias: curated instruction datasets are degenerate training sources for length prediction, as GPT-imposed brevity constraints reduce Long-class representation to under 0.02% of examples, establishing natural conversation logs as the only viable training signal. End-to-end evaluations demonstrate substantial latency reductions across diverse hardware regimes: a 70-76% short-request P50 latency reduction on an RTX 4090, a 69.7% reduction on Apple M1 edge hardware, and an 83.6% reduction in Time-To-First-Token (TTFT) on a GCP NVIDIA L4 real-world trace replay (rho = 0.80). Clairvoyant is open-source, requires zero modifications to the inference backend, and provides a low-overhead mechanism to eliminate HOLB in edge LLM environments.

Figures

Figures reproduced from arXiv: 2606.07248 by the authors.

**Figure 2.** Clairvoyant intercepts API requests, predicts output length in 0.029 ms via lexical feature extraction and ONNX inference, and dispatches to the backend in SJF order. The response path is a transparent pass-through. The starvation guard promotes any request waiting longer than τ = 3 × µshort regardless of predicted P(Long). Evaluated on Apple M1 (Ollama, Gemma3:4b) and RTX 4090 (Ollama, Gemma3:4b, Llama3.1:8b). serv… view at source ↗

**Figure 3.** SJF latency reduction for short requests vs. queue utilisation [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 2 linked inside Pith

[1]

Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve

Amey Agrawal, Niket Kedia, Ashish Panwar, Jayashree Mohan, Jinwoo Kwak, Gregory R Ganger, Anton Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 117–134, 2024

2024
[2]

CodeAlpaca: An instruction-following LLaMA model for code generation.https: //github.com/sahil280114/codealpaca, 2023

Sahil Chaudhary. CodeAlpaca: An instruction-following LLaMA model for code generation.https: //github.com/sahil280114/codealpaca, 2023. Accessed: 2026-06-02

2023
[3]

XGBoost: A scalable tree boosting system

Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. ACM, 2016

2016
[4]

Databricks Dolly 15K: An open-source dataset for instruction-following large language models.https://github.com/databrickslabs/dolly, 2023

Databricks. Databricks Dolly 15K: An open-source dataset for instruction-following large language models.https://github.com/databrickslabs/dolly, 2023. Accessed: 2026-06-02

2023
[5]

Efficient LLM scheduling by learning to rank

Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. Efficient LLM scheduling by learning to rank. InAdvances in Neural Information Processing Systems (NeurIPS 37), 2024

2024
[6]

Cambridge University Press, 1st edition, 2013

Mor Harchol-Balter.Performance Modeling and Design of Computer Systems: Queueing Theory in Action. Cambridge University Press, 1st edition, 2013

2013
[7]

S3: Increasing GPU utilization during generative inference for higher throughput

Yunho Jin, Xiaoxuan Pan, Alvin Wang, Guohao Yu, Zhihao Zhu, Ion Stoica, and Hao Zhang. S3: Increasing GPU utilization during generative inference for higher throughput. InAdvances in Neural Information Processing Systems (NeurIPS 36), 2023

2023
[8]

Volume 1: Theory

Leonard Kleinrock.Queueing Systems. Volume 1: Theory. Wiley-Interscience, 1975

1975

Show all 20 references

[9]

OpenAssistant conversations: Democratizing large language model alignment

Andreas Köpf et al. OpenAssistant conversations: Democratizing large language model alignment. https://huggingface.co/datasets/OpenAssistant/oasst1, 2023. Accessed: 2026-06-02

2023
[10]

Efficient memory management for large language model serving with PagedAttention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th ACM Symposium on Operating Systems Principles...

2023
[11]

LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset

LMSYS Org. LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset. https:// huggingface.co/datasets/lmsys/lmsys-chat-1m, 2023. Accessed: 2026-06-02

2023
[12]

ONNX Runtime.https://onnxruntime.ai, 2021

ONNX Runtime Developers. ONNX Runtime.https://onnxruntime.ai, 2021. Accessed: 2026-06-02

2021
[13]

Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Nedeljko Vujic, Zhenhua Liu, Chenyang Wang, Ioannis Stavrakakis, Stratis Ioannidis, and David A. Wood. Efficient interactive LLM serving with proxy model-based sequence length prediction.arXiv preprint arXiv:240...

2024 arXiv
[14]

Linus E. Schrage. A proof of the optimality of the shortest remaining processing time discipline. Operations Research, 16(3):687–690, 1968. 16

1968
[15]

Liu, and Christopher D

Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer- generator networks. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1073–1083. Association for Computational Linguistics, 2017

2017
[16]

ShareGPT dataset.https://sharegpt.com, 2023

ShareGPT. ShareGPT dataset.https://sharegpt.com, 2023. Accessed: 2026-06-02

2023
[17]

Hashimoto

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model.https: //github.com/tatsu-lab/stanford_alpaca, 2023. Accessed: 2026-06-02

2023
[18]

Fast distributed inference serving for large language models.arXiv preprint arXiv:2305.05920, 2023

Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models.arXiv preprint arXiv:2305.05920, 2023

2023 arXiv
[19]

Predicting LLM output length via entropy-guided representations

Huanyi Xie, Yubin Chen, Liangyu Wang, Lijie Hu, and Di Wang. Predicting LLM output length via entropy-guided representations. InThe Fourteenth International Conference on Learning Representations (ICLR 2026), 2026

2026
[20]

Orca: A distributed serving system for transformer-based generative models

Gyeong-In Yu, Jeongmin Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538. USENIX Association, 2022. 17

2022

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve

Amey Agrawal, Niket Kedia, Ashish Panwar, Jayashree Mohan, Jinwoo Kwak, Gregory R Ganger, Anton Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 117–134, 2024

2024

[2] [2]

CodeAlpaca: An instruction-following LLaMA model for code generation.https: //github.com/sahil280114/codealpaca, 2023

Sahil Chaudhary. CodeAlpaca: An instruction-following LLaMA model for code generation.https: //github.com/sahil280114/codealpaca, 2023. Accessed: 2026-06-02

2023

[3] [3]

XGBoost: A scalable tree boosting system

Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. ACM, 2016

2016

[4] [4]

Databricks Dolly 15K: An open-source dataset for instruction-following large language models.https://github.com/databrickslabs/dolly, 2023

Databricks. Databricks Dolly 15K: An open-source dataset for instruction-following large language models.https://github.com/databrickslabs/dolly, 2023. Accessed: 2026-06-02

2023

[5] [5]

Efficient LLM scheduling by learning to rank

Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. Efficient LLM scheduling by learning to rank. InAdvances in Neural Information Processing Systems (NeurIPS 37), 2024

2024

[6] [6]

Cambridge University Press, 1st edition, 2013

Mor Harchol-Balter.Performance Modeling and Design of Computer Systems: Queueing Theory in Action. Cambridge University Press, 1st edition, 2013

2013

[7] [7]

S3: Increasing GPU utilization during generative inference for higher throughput

Yunho Jin, Xiaoxuan Pan, Alvin Wang, Guohao Yu, Zhihao Zhu, Ion Stoica, and Hao Zhang. S3: Increasing GPU utilization during generative inference for higher throughput. InAdvances in Neural Information Processing Systems (NeurIPS 36), 2023

2023

[8] [8]

Volume 1: Theory

Leonard Kleinrock.Queueing Systems. Volume 1: Theory. Wiley-Interscience, 1975

1975