REVIEW 3 cited by
Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The complexity of large language model (LLM) serving workloads has substantially increased due to the integration with external tool invocations, such as ChatGPT plugins. In this paper, we identify a new opportunity for efficient LLM serving for requests that trigger tools: tool partial execution alongside LLM decoding. To this end, we design Conveyor, an efficient LLM serving system optimized for handling requests involving external tools. We introduce a novel interface for tool developers to expose partial execution opportunities to the LLM serving system and a request scheduler that facilitates partial tool execution. Our results demonstrate that tool partial execution can improve request completion latency by up to 38.8%.
Forward citations
Cited by 3 Pith papers
-
Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
Across 13.5M production GitHub Copilot sessions, coding-agent work is dominated by autonomous LLM–tool loops, with KV-cache reuse at ~90% inside a turn but 55% at turn boundaries and 8% after model switches.
-
Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents
SearchAgent-X speeds up LLM-based search agents by prioritizing requests that benefit most from cached computations and by stopping approximate retrieval early when the model is ready, with no measured loss in answer quality.
-
PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networks
PICE uses progressive inference, cloud sketches plus parallel edge expansion, to raise LLM serving throughput by 1.5-2x and cut latency by up to 43 percent in testbed experiments.
Discussion (0). Continue with ORCID to comment.