Pith. sign in

REVIEW 18 cited by

Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.08509 v2 pith:VSQFCPLS submitted 2024-04-12 cs.DC cs.CLcs.LG

Haoran Qiu , Weichao Mao , Archit Patke , Shengkun Cui , Saurabh Jha , Chen Wang , Hubertus Franke , Zbigniew T. Kalbarczyk
show 2 more authors
Tamer Başar Ravishankar K. Iyer
This is my paper
classification cs.DCcs.CLcs.LG
keywords batchingservinginteractivessjfacrossefficientfcfsllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths. Our open-source SSJF implementation does not require changes to memory management or batching strategies. Evaluations on real-world datasets and production workload traces show that SSJF reduces average job completion times by 30.5-39.6% and increases throughput by 2.2-3.6x compared to FCFS schedulers, across no batching, dynamic batching, and continuous batching settings.

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. General Non-Clairvoyant KV-Cache Scheduling via Regime-Aware Routing

    cs.DS 2026-07 accept novelty 7.0 of 10

    A regime-aware routing meta-scheduler is O(1)-competitive for total completion time on general non-clairvoyant KV-cache batch scheduling with arbitrary prompts and responses.

  2. Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    SVF scheduling algorithm achieves a competitive ratio of 3 for LLM serving and integrates into vLLM to reduce average and tail latency.

  3. Beyond Prediction: Tail-Aware Scheduling for LLM Inference

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Presents a distribution-aware scheduling framework for LLM inference that reduces P99 TTLT by 35-50% and TTFT by 34-47% versus SRPT with perfect length knowledge using statistical signals instead of predictions.

  4. Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamics

    cs.DC 2026-04 unverdicted novelty 7.0 of 10

    Autopoiesis uses LLM-driven program synthesis to evolve serving policies online during deployment, delivering up to 53% and average 34% gains over prior LLM serving systems under runtime dynamics.

  5. SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips

    cs.DC 2026-01 conditional novelty 7.0 of 10

    SuperInfer improves TTFT SLO attainment by up to 74.7% on GH200 Superchips via SLO-aware rotary scheduling (RotaSched) and full-duplex KV cache rotation (DuplexKV) over NVLink-C2C while preserving TBT and throughput.

  6. Make a Video Call with LLM: A Measurement Campaign over Six Mainstream Apps

    cs.NI 2025-10 conditional novelty 7.0 of 10

    Commercial AI video chat apps differ by 4× in video bitrate, 10× in framerate, and from zero to 10+ minutes of visual memory, with none replying in under 1.5 seconds.

  7. Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving

    cs.AR 2026-07 conditional novelty 6.0 of 10

    NeuScale routes LLM inference requests to the most energy/cost-efficient configuration of heterogeneous NPU chips using roofline allocation and runtime auto-scaling.

  8. Online Linear Programming for Multi-Objective Routing in LLM Serving

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Online LP bid-price routing with warm-started dual gradient updates substantially improves multi-SLO LLM decode routing over heuristics in simulation.

  9. Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale

    cs.DC 2026-05 unverdicted novelty 6.0 of 10

    BalanceRoute reduces data-parallel imbalance in LLM inference via F-score routing and lookahead, yielding higher end-to-end throughput on 144-NPU clusters versus vLLM baselines.

  10. Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale

    cs.DC 2026-05 unverdicted novelty 6.0 of 10

    BalanceRoute uses a piecewise-linear F-score (with optional short lookahead) for sticky request routing in LLM serving, reducing DP imbalance and raising end-to-end throughput versus vLLM baselines on production and A...

  11. A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    A queueing model derives stability conditions for LLM inference services under combined compute and KV cache memory limits, with experimental validation showing typical deviations under 10%.

  12. Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    LLM output lengths conditioned on a prompt form heavy-tailed distributions, so robust estimation from multiple samples outperforms single-sample labels for prediction.

  13. Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving

    cs.DC 2026-03 conditional novelty 6.0 of 10

    Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.

  14. CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing

    cs.DC 2025-12 conditional novelty 6.0 of 10

    CascadeInfer partitions LLM instances into length-specialized groups, uses dynamic programming for stage partitioning, and applies runtime refinement plus decentralized load balancing to cut latency and raise throughput.

  15. Clairvoyant: Predictive Shortest-Job-First Admission for Serial LLM Inference

    cs.DC 2026-06 reject novelty 5.0 of 10

    Predictive shortest-job-first admission using an XGBoost classifier on 19 lexical prompt features cuts short-request P50 latency by 70-76% in serial LLM backends under burst load, per the paper's RTX 4090 benchmarks.

  16. Clairvoyant: Predictive Shortest-Job-First Admission for Serial LLM Inference

    cs.DC 2026-06 unverdicted novelty 5.0 of 10

    Clairvoyant predicts LLM response lengths from 19 lexical features with an XGBoost classifier to enable SJF scheduling in serial backends, reporting 70-76% P50 latency reduction for short requests under high load.

  17. STAR: Decode-Phase Rescheduling for LLM Inference

    cs.DC 2025-10 unverdicted novelty 5.0 of 10

    STAR cuts P99 TPOT by 75.1% and raises goodput 2.63x via a lightweight hidden-state length predictor and dynamic decode rescheduling that combines current and predicted loads.

  18. Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs

    cs.DC 2026-06 unverdicted novelty 4.0 of 10

    Festina reduces energy consumption by up to 56% for serverless LLM inference on shared GPUs while keeping TTFT/TBT SLO attainment within 2% of four state-of-the-art baselines.

Pith tools