REVIEW 4 cited by
SCORPIO: Serving the Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
SCORPIO: Serving the Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
read the original abstract
Existing Large Language Model (LLM) serving systems prioritize maximum throughput. They often neglect Service Level Objectives (SLOs) such as Time to First Token (TTFT) and Time Per Output Token (TPOT), which leads to suboptimal SLO attainment. This paper introduces SCORPIO, an SLO-oriented LLM serving system designed to maximize system goodput and SLO attainment for workloads with heterogeneous SLOs. Our core insight is to exploit SLO heterogeneity for adaptive scheduling across admission control, queue management, and batch selection. SCORPIO features a TTFT Guard, which employs least-deadline-first reordering and rejects unattainable requests, and a TPOT Guard, which utilizes a VBS-based admission control and a novel credit-based batching mechanism. Both guards are supported by a predictive module. Evaluations demonstrate that SCORPIO improves system goodput by up to 14.4X and SLO adherence by up to 46.5% compared to state-of-the-art baselines.
Forward citations
Cited by 4 Pith papers
-
Execution Envelopes: A Shared Admission Contract for Backend AI Execution Requests
Execution envelopes define a shared admission contract for backend AI execution requests to attach unified logging, governance, and policy hooks at entry.
-
Trusted Floors Under Untrusted Learners: A Runtime Assured-SLO Guard for ML Serving
A Simplex-style guard around untrusted learned admission controllers structurally enforces an assured tenant floor, holding miss 0.0 in real 2xV100 tests where unguarded learners miss 0.86-0.94.
-
Trusted Floors Under Untrusted Learners: A Runtime Assured-SLO Guard for ML Serving
Reservation plus assured-first priority holds admitted assured-class miss at 0.0 on real 2xV100 under every miscalibration of a learned admitter, while GAIE Flow Control fails under label swap.
-
Demonstrating chart-plot: Closing the Last Mile of Academic Chart Generation
chart-plot is an agentic harness using style-aware code generation from venue figures, a LaTeX-aware render-and-revise loop, and structured edit handles to produce top-venue-ready academic charts.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.