REVIEW 13 cited by
Autellix: An Efficient Serving Engine for LLM Agents as General Programs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language model (LLM) applications are evolving beyond simple chatbots into dynamic, general-purpose agentic programs, which scale LLM calls and output tokens to help AI agents reason, explore, and solve complex tasks. However, existing LLM serving systems ignore dependencies between programs and calls, missing significant opportunities for optimization. Our analysis reveals that programs submitted to LLM serving engines experience long cumulative wait times, primarily due to head-of-line blocking at both the individual LLM request and the program. To address this, we introduce Autellix, an LLM serving system that treats programs as first-class citizens to minimize their end-to-end latencies. Autellix intercepts LLM calls submitted by programs, enriching schedulers with program-level context. We propose two scheduling algorithms-for single-threaded and distributed programs-that preempt and prioritize LLM calls based on their programs' previously completed calls. Our evaluation demonstrates that across diverse LLMs and agentic workloads, Autellix improves throughput of programs by 4-15x at the same latency compared to state-of-the-art systems, such as vLLM.
Forward citations
Cited by 13 Pith papers
-
Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
Across 13.5M production GitHub Copilot sessions, coding-agent work is dominated by autonomous LLM–tool loops, with KV-cache reuse at ~90% inside a turn but 55% at turn boundaries and 8% after model switches.
-
A Policy-Driven Runtime Layer for Agentic LLM Serving
Introduces a three-tier architecture with an agent runtime layer and four primitives for agent-aware policies in LLM serving, validated on KV caching via CacheSage showing 13-37pp hit-rate gains on five workloads.
-
Low-Scaling Many-Body Green's Function Calculations for Molecular Systems via Interacting-Bath Dynamical Embedding Theory
Molecular ibDET assembles full-system GW/EOM-CCSD Green's functions from small atom-centered impurity problems and recovers IPs/EAs within about 0.1 eV of full-system results.
-
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Balancing only first-turn agent requests while keeping follow-ups cache-aware raises LLM cluster TPS 10-16% (colocation) and prefill TPS 2-34% (disaggregation) over SOTA schedulers on production agent traces.
-
A Workflow-Aware Serving Layer for Agentic Applications
A workflow-aware serving layer compiles per-node model-verifier-backend plans with an ILP and adapts only uncommitted work via pre-solved pressure rungs and residual re-solves.
-
Coordinated Networking for On-Device Agent-Augmented Real-Time Communication
HAFS coordinates video and AI-agent context flows in real-time communication, raising video quality 1.5x and cutting agent response time 31% in testbed trials.
-
TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI
Task-consistent sticky routing with a contextual bandit improves accuracy-latency trade-offs on agentic benchmarks by learning from delayed task-level reward.
-
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.
-
ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
A program-aware scheduler and tool-lifecycle manager for LLM agent workflows raises serving throughput by 1.5–3.6x and RL rollout throughput by 1.8–3.9x over vLLM/Continuum baselines.
-
Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
TTL-based KV-cache pinning that uses predicted tool-call durations and queueing-delay costs cuts agent job completion time by up to 8x.
-
Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms
Murakkab uses declarative workflow specs and a profile-guided MILP optimizer to reduce GPU, energy, and cost for agentic workflow serving while meeting percentile-defined SLOs.
-
KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows
KVFlow uses workflow-aware eviction priorities and overlapped KV prefetching to cut cache-miss latency in LLM multi-agent serving.
-
Scalable LLM Agent Tool Access in the Cloud
A cloud-scale MCP gateway with hybrid dense-sparse retrieval lets LLM agents work with 3,000+ tools at 98% Top-15 recall, cutting tool-selection time 8.9× and token use 23.8×.
Discussion (0). Continue with ORCID to comment.