Balancing only first-turn agent requests while keeping follow-ups cache-aware raises LLM cluster TPS 10-16% (colocation) and prefill TPS 2-34% (disaggregation) over SOTA schedulers on production agent traces.
Flashinfer: Efficient and customizable attention engine for LLM inference serving
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.DC 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Balancing only first-turn agent requests while keeping follow-ups cache-aware raises LLM cluster TPS 10-16% (colocation) and prefill TPS 2-34% (disaggregation) over SOTA schedulers on production agent traces.