Pith. sign in

REVIEW 3 major objections 6 minor 12 cited by

LLM agents can run tools while the model is still thinking by predicting recurring call patterns, cutting average task time by nearly half.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.

T0 review reviewed 2026-07-13 challenge →

load-bearing objection Solid agent-serving systems paper: Pattern Tuples plus slack-aware speculative tools give real measured latency wins; transferability of mined patterns is the main open risk, not a broken design. the 3 major comments →

arxiv 2603.18897 v3 pith:KSEEQ52X submitted 2026-03-19 cs.DC cs.AI

Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving

Yifan Sui , Han Zhao , Rui Ma , Zhiyuan He , Hao Wang , Jianxun Li , Kaiqiang Xu , Kai Chen
show 1 more author
Yuqing Yang
This is my paper
classification cs.DC cs.AI
keywords LLM agentsspeculative tool executionagent servingtool latencypattern miningopportunistic schedulingdistributed systems
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern LLM agents solve tasks in a strict loop: the model thinks, then waits for an external tool, then thinks again. That waiting time often dominates the whole job. This paper shows that agent tool use is far more regular than it looks: the same short sequences of tools and the same ways of passing arguments between them keep recurring. PASTE mines those patterns offline, predicts the next concrete tool call (including arguments) from recent events, and starts the tool while the model is still generating. Speculative results stay isolated until the model confirms them, and a scheduler only runs speculation on spare capacity so it cannot slow the real path. On deep-research, coding, and scientific agents the system reports large end-to-end speedups without changing final answers.

Core claim

Agent workloads expose a serial LLM–tool bottleneck that standard serving and serverless systems do not hide. PASTE claims that stable application-level control-flow patterns plus simple symbolic argument mappings are accurate enough to launch future tool calls early, overlapping them with generation and cutting average task completion time by up to 48.5% and observed tool latency by about 1.8× versus strong baselines, while preserving correctness through isolation and confirmation.

What carries the argument

The Pattern Tuple (context, target tool, value-mapping function, probability): a payload-agnostic event-signature context plus a late-binding function that derives arguments from prior tool outputs. At runtime an opportunistic scheduler runs only those predictions on slack resources, preempts them under contention, and promotes a hit to authoritative so no work is repeated.

Load-bearing premise

Tool sequences and simple argument-derivation rules mined from past traces still hold on new tasks often enough that speculative hits outweigh wasted work under a fixed spare-resource budget.

What would settle it

On a held-out set of agent tasks with the same tools and resource limits, if overall speculative hit rate collapses near zero and end-to-end latency shows no improvement over serial or non-pattern baselines, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Agent serving systems can treat tool latency as hideable overlap rather than an unavoidable serial stall.
  • Operators can trade a small, budgeted amount of spare CPU, memory, and network for large reductions in user-visible task time.
  • Correctness for side-effecting tools becomes a policy and sandbox problem (dry-run, staging, confirm-before-commit) rather than a reason to forbid all speculation.
  • Pattern libraries can be mined once per domain (coding, research, science) and reused across sessions without changing the agent’s tool API.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • As agents gain more tools and longer multi-step plans, the value of pattern-based look-ahead likely grows faster than pure LLM-serving optimizations, which ignore tool time.
  • The same control-flow-plus-data-flow abstraction could apply to non-LLM orchestrators that emit tools online (for example, planner–executor loops) if their event signatures are similarly regular.
  • Hit-rate gaps between structured coding loops and open-ended research suggest domain-specific pattern packs or online pattern updates would be a natural next measurement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents PASTE, a middleware for LLM-agent serving that hides tool latency by predicting near-future tool invocations from offline-mined Pattern Tuples (context C as event-signature subsequences, target tool T, symbolic value-mapping function f, confidence p) and executing them speculatively while the LLM is still generating. Speculative work is confined to a slack budget, preemptible, and only committed after authoritative confirmation (promotion). On gemini-cli, Qwen-DR, and Virtual-Lab over DeepResearchBench, SWE-bench, and ScholarQA, against ORION and SpecFaaS under fixed hardware and tool interfaces, PASTE reports up to ~48.5% lower average E2E latency (abstract: 43.5%), ~1.8× lower average tool latency, improved throughput and concurrency scaling, a side-effect audit with no final-result divergences, and modest resource overhead.

Significance. If the empirical gains hold under realistic pattern transfer, this is a solid systems contribution for agent serving: it targets the serial LLM–tool loop that general LLM serving and static-DAG serverless optimizers leave exposed. The Pattern Tuple cleanly separates control-flow signatures from late-bound data-flow maps, and the opportunistic scheduler with promotion/preemption and an explicit eligibility policy is a careful correctness design. Strengths include multi-agent/multi-benchmark evaluation, CDFs and concurrency sweeps, prediction metrics, a side-effect audit (602 potentially unsafe speculative actions blocked; no result divergence), and overhead accounting. The work is practically relevant as agents become a dominant serving workload.

major comments (3)
  1. [§7.5, §4.2, Fig. 4, Fig. 13] §2.3 Insights 1–2, §4.2 / Alg. 1, and §7.5: The headline E2E and tool speedups rest on offline-mined patterns and simple symbolic maps f (field/path lookup, index fallback, light string normalization) transferring to a disjoint evaluation set well enough that multi-candidate hits dominate waste under budget B. Reported Top-1 accuracy is only up to 27.8% and Top-3 recall 43.9%; the 93.8% overall hit rate is achieved by breadth of speculation when spare resources allow. There is no controlled transfer-failure experiment (same agent/tool surface, tasks whose control flow or producer–consumer argument paths diverge from the mining corpus—new domains, different edit–verify loops, or tools whose arguments are LLM-generated rather than derived as in Fig. 4). Without that, it is unclear whether Fig. 13’s overlap and the ~48.5%/1.8× claims survive distribution shift, or whether gains are confined
  2. [§5.1 Eq. (1)–(2), Alg. 1, §7] Free parameters of the system (min support σ, confidence τ, max context length k, speculative budget B, and the utility U(j)=p·T/(c·d) in Eq. (2) / §5.1) are load-bearing for both hit rate and non-interference, yet the evaluation does not report sensitivity or ablations over them. A systems claim of this form needs at least a budget/threshold sweep showing that E2E gains remain positive and that authoritative latency is not degraded when B is large or τ is low. Without it, the reported speedups cannot be separated from a favorable operating point.
  3. [Abstract, §5, §6] Abstract vs. body: the abstract claims PASTE “jointly schedules tool execution and returning LLM sessions to avoid shifting bottlenecks to the GPU,” but §5–§6 develop an opportunistic tool scheduler (authoritative vs. speculative tool jobs, slack budget, preemption) as a tool-serving proxy; co-scheduling of returning LLM/GPU sessions is not specified or measured. Either develop and evaluate that joint GPU/session policy, or narrow the abstract claim to what is implemented so the central systems contribution is accurately scoped.
minor comments (6)
  1. [Abstract, §1, §7.2] Headline numbers disagree: abstract states 43.5% average task completion time reduction; the paper body and §7.2 state up to 48.5% (and average speedups of 1.25×/1.32× over ORION/SpecFaaS). Align abstract, intro, and evaluation text.
  2. [Title page] Title inconsistency: arXiv/metadata title is “Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving”; the manuscript title is “Act While Thinking: Accelerating LLM Agents via Pattern-Aware Speculative Tool Execution.” Use one title throughout.
  3. [§7.1] §7.1 baselines: briefly justify why ORION and SpecFaaS (static DAG / serverless speculation) are the strongest available comparisons for agent loops, and state any agent-native speculative baselines that were considered and rejected.
  4. [Fig. 13, §7.3] Fig. 13 time-breakdown: define units and how “overlap region,” “tool stall,” and “spec. overhead” are measured (wall-clock attribution rules) so the 67% stall reduction and 10× overlap claims are reproducible.
  5. [§8] §8 Related work has a duplicated sentence (“Yet, most of these techniques assume a largely static workflow… which PASTE is designed to handle” appears twice). Clean up.
  6. [§4.2, §6] Clarify whether pattern mining is purely offline or periodically refreshed online (§4.2 vs. §6 “updated online and checkpointed”), and how cold start of the pattern pool is handled for a new agent deployment.

Circularity Check

0 steps flagged

No circular derivation: PASTE is an empirical systems design whose latency claims are measured against external baselines on disjoint tasks, not forced by construction from fitted inputs.

full rationale

This paper does not present a first-principles derivation whose target reduces to its inputs. Pattern Tuples (C, T, f, p) are mined offline from historical execution traces (Alg. 1, §4.2) via structural subsequence mining plus simple symbolic value maps, then applied at runtime as best-effort speculative hints under a slack budget and eligibility policy (§5). Probabilities p are empirical frequencies on the mining corpus, not parameters fitted to the evaluation latency numbers. Evaluation (§7.1) uses a disjoint task set with no train/test overlap and reports measured E2E/tool latency against ORION and SpecFaaS under fixed hardware, models, and resource budgets. Hit rates and speedups are experimental outcomes, not identities. Self-citations (e.g., prior serverless pre-loading work by overlapping authors) appear only in related work and do not load-bear the central claim. Transferability of mined patterns is a validity/assumption risk, not circularity: nothing in the paper equates the claimed 43.5–48.5% latency reduction to the mining inputs by definition. Score 0 is the correct honest finding.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central empirical claim rests on domain assumptions that agent tool use is pattern-rich and that simple symbolic argument maps suffice, plus several operator-chosen thresholds that gate mining and speculation. No new physical entities are postulated; the Pattern Tuple is an engineering abstraction. Free parameters control which patterns fire and how much slack speculation may consume, so reported speedups are conditional on those settings and on the mined corpus.

free parameters (5)
  • min support σ (pattern mining)
    Algorithm 1 drops tools/contexts with insufficient historical support; changes which patterns enter the pool and thus hit rate.
  • confidence threshold τ
    Algorithm 1 retains only patterns with empirical p ≥ τ; directly controls speculation aggressiveness vs waste.
  • max context length k
    Bounds preceding signature sequences mined as contexts; affects pattern specificity and transfer.
  • speculative resource budget B
    Algorithm 3 caps resources for speculative jobs; primary knob trading latency reduction for overhead.
  • utility score weights in U(j)=p·T/(c·d)
    Greedy speculative priority depends on estimated latency reduction, cost, and duration; estimates are system-chosen.
axioms (5)
  • domain assumption Agent workloads exhibit stable short-horizon tool-type transition patterns (e.g., edit→test, search→visit) with high enough probability to be useful for speculation.
    Core of §2.3 Insight 1 and the Pattern Analyzer; without it, prediction cannot beat waiting for the LLM.
  • domain assumption Many tool arguments are recoverable from prior tool payloads via simple symbolic maps rather than free-form LLM generation.
    §2.3 Insight 2 and Phase II of mining; required for concrete speculative invocations, not just tool-type warmup.
  • domain assumption Speculative work can be confined to slack resources and preempted without delaying authoritative LLM/tool paths under the stated isolation policy.
    §5 scheduling principles; load-bearing for the non-interference claim.
  • standard math Standard sequential pattern mining over event signatures plus empirical conditional probability is a valid estimator of near-future tool calls.
    Algorithm 1 uses PrefixSpan-style mining and p = |M★|/|C|; ordinary frequent-sequence statistics.
  • ad hoc to paper User/operator speculation eligibility policy correctly labels side-effect risk and allowed speculation depth per tool.
    §5.2 and Figure 6; safety results depend on this external policy rather than automatic side-effect inference.
invented entities (2)
  • Pattern Tuple (C, T, f, p) independent evidence
    purpose: Formalize decoupled control-flow context, predicted tool, symbolic argument mapping, and confidence for speculative tool execution.
    Central abstraction introduced in §3–4; engineering construct with independent operational meaning (can be mined, matched, and scored), not a physical postulate.
  • Opportunistic speculative scheduler with promotion/preemption independent evidence
    purpose: Run predicted tools on slack, promote matches to authoritative, preempt on contention.
    System component in §5; falsifiable via latency/isolation measurements in the evaluation.

reviewed 2026-07-13 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving." pith.science (2026). https://pith.science/paper/KSEEQ52X

@misc{pith2026260318897,
  author       = {Pith},
  title        = {Pith review of: Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KSEEQ52X}},
  note         = {Machine review of arXiv:2603.18897}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

LLM-powered agents execute tasks through a sequential loop of model generation and tool execution. Today's serving systems serialize this loop, leaving tool latency exposed on the task critical path. This paper presents PASTE, a tool-aware agent-serving system that predicts concrete future tool invocations from recurring agent patterns and executes them speculatively while the LLM is still generating. PASTE isolates speculative results until confirmed by the LLM and jointly schedules tool execution and returning LLM sessions to avoid shifting bottlenecks to the GPU. Across deep research, coding, and scientific-agent workloads, PASTE reduces average task completion time by 43.5% and lowers observed tool latency by 1.8x.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

    cs.DC 2026-07 accept novelty 7.0

    A training-free self-fork of an agentic LLM predicts its own next tool call early enough to hide 16–37% tool-wait latency, cutting Qwen3-32B GAIA P95 by 18% with accuracy preserved.

  2. A Policy-Driven Runtime Layer for Agentic LLM Serving

    cs.AI 2026-05 unverdicted novelty 7.0

    Introduces a three-tier architecture with an agent runtime layer and four primitives for agent-aware policies in LLM serving, validated on KV caching via CacheSage showing 13-37pp hit-rate gains on five workloads.

  3. Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Tools

    cs.CR 2026-06 unverdicted novelty 6.0

    Ghost tool calls from speculative dispatch create persistent intent leaks that only issue-time policies changing or suppressing call arguments or destinations can reduce, per evaluations of twelve policies on three corpora.

  4. Idleness is Relative: Exploiting Tool-Call Idle Windows for Offloading in Agentic Systems with MORI

    cs.OS 2026-05 unverdicted novelty 6.0

    MORI improves throughput 20-71% and TTFT 18-43% over baselines by ranking programs on a continuous idleness spectrum and shifting the GPU-CPU boundary to match capacity in agentic LLM serving.

  5. A Policy-Driven Runtime Layer for Agentic LLM Serving

    cs.AI 2026-05 conditional novelty 6.0

    A four-primitive agent runtime layer between agent frameworks and LLM serving engines raises cross-session KV-cache hit rate by 13–37 percentage points in the paper's experiments.

  6. HARBOR: Automated Harness Optimization

    cs.LG 2026-04 unverdicted novelty 6.0

    HARBOR formalizes harness optimization as constrained noisy Bayesian optimization over mixed-variable spaces and reports a case study where it outperforms manual tuning on a production coding agent.

  7. Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems

    cs.SE 2026-04 conditional novelty 6.0

    Claude Code answers recurring agent design questions with a thin model loop wrapped in dense safety, context, extensibility, and persistence harnesses, and those same questions get different answers in OpenClaw and Hermes.

  8. SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents

    cs.CL 2026-05 unverdicted novelty 5.0

    SpecHop accelerates multi-hop LLM tool use via continuous multi-threaded speculation with asynchronous verification, approaching oracle latency gains and reducing latency up to 40% on retrieval tasks.

  9. Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management

    cs.LG 2026-05 unverdicted novelty 5.0

    PBKV predicts agent invocations in dynamic LLM workflows to manage KV-cache reuse, delivering up to 1.85x speedup over LRU and 1.26x over KVFlow.

  10. Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems

    cs.SE 2026-04 unverdicted novelty 5.0

    Claude Code centers on a model-tool while-loop surrounded by permission systems, context compaction, extensibility hooks, subagent delegation, and session storage; the same design questions yield different answers in ...

  11. Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering

    cs.SE 2026-04 accept novelty 5.0

    LLM agent progress depends on externalizing cognitive functions into memory, skills, protocols, and harness engineering that coordinates them reliably.

  12. B-PASTE: Beam-Aware Pattern-Guided Speculative Execution for Resource-Constrained LLM Agents

    cs.DC 2026-04 unverdicted novelty 5.0

    B-PASTE uses beam-aware speculation of tool-call branches ranked by critical-path reduction to deliver up to 1.4x end-to-end speedup in resource-constrained LLM agents.

Reference graph

Works this paper leans on

60 extracted references · 1 canonical work pages · cited by 10 Pith papers

  1. [1]

    Agent Skills

    2025. Agent Skills. https://agentskills.io/home. Accessed: 2026-01-25

    2025

  2. [2]

    Build, Debug & Deploy with AI

    2025. Build, Debug & Deploy with AI. https://geminicli.com/

    2025

  3. [3]

    Claude Code | Claude

    2025. Claude Code | Claude. https://www.claude.com/product/claude-code

    2025

  4. [4]

    GitHub Copilot

    2025. GitHub Copilot. https://github.com/features/copilot

    2025

  5. [5]

    Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang

  6. [6]

    InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol

    InferCept: Efficient Intercept Support for Augmented Large Language Model Inference. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 81–95

  7. [7]

    Gulavani, Alexey Tumanov, and Ramachandran Ramjee

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association

    2024

  8. [8]

    2025.DeepResearch: Tongyi Deep Research, the Leading Open-source Deep Research Agent

    Alibaba-NLP. 2025.DeepResearch: Tongyi Deep Research, the Leading Open-source Deep Research Agent. https://github.com/Alibaba-NLP/DeepResearch

    2025

  9. [9]

    Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen tau Yih, Pang Wei Koh, and Hannaneh Hajishirzi. 2024. OpenSch...

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.14199 2024

  10. [10]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. InProceedings of the 41st International Conference on Machine Learning (ICML), Vol. 235. PMLR

    2024

  11. [11]

    Zefan Cai, Wen Xiao, Hanshi Sun, Cheng Luo, Yikai Zhang, Ke Wan, Yucheng Li, Yeyang Zhou, Li-Wen Chang, Jiuxiang Gu, Zhen Dong, Anima Anandkumar, Abedelkadir Asi, and Junjie Hu. 2025. R-KV: Redundancy-aware KV Cache Compression for Training-Free Reasoning Models Acceleration.Advances in Neural Information Processing Systems(2025)

    2025

  12. [12]

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao

  13. [13]

    DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents.arXiv preprint(2025)

    2025

  14. [14]

    Jiangfei Duan, Runyu Lu, Haojie Duanmu, et al. 2024. MuxServe: Flexible Spatial- Temporal Multiplexing for Multiple LLM Serving. InForty-first International Conference on Machine Learning (ICML)

    2024

  15. [15]

    Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yu- vraj Patel, and Luo Mai. 2024. ServerlessLLM: Low-Latency Serverless Inference for Large Language Models. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, 135–153

    2024

  16. [16]

    Lorenzo Giusti, Ole Anton Werner, Riccardo Taiello, Matilde Carvalho Costa, Emre Tosun, Andrea Protani, Marc Molina, Rodrigo Lopes de Almeida, Paolo Cacace, Diogo Reis Santos, and Luigi Serio. 2025. Federation of Agents: A Semantics-Aware Communication Fabric for Large-Scale Agentic AI.arXiv preprint arXiv:2509.20175(2025)

    arXiv 2025

  17. [17]

    Liangxuan Guo, Bin Zhu, Qingqian Tao, Kangning Liu, Xun Zhao, Xianzhe Qin, Jin Gao, and Guangfu Hao. 2025. Agentic Lybic: Multi-Agent Execution System with Tiered Reasoning and Orchestration.arXiv preprint arXiv:2509.11067(2025)

    arXiv 2025

  18. [18]

    Junhao Hu, Wenrui Huang, Haoyi Wang, Weidong Wang, Tiancheng Hu, Qin Zhang, Hao Feng, Xusheng Chen, Yizhou Shan, and Tao Xie. 2024. EPIC: Efficient Position-Independent Context Caching for Serving Large Language Models. arXiv preprint arXiv:2410.15332(2024)

    Pith/arXiv arXiv 2024

  19. [19]

    Zixiao Huang, Wen Zeng, Tianyu Fu, Tengxuan Liu, Yizhou Sun, Ke Hong, Xinhao Yang, Chengchun Liu, Yan Li, Quanlu Zhang, Guohao Dai, Zhenhua Zhu, and Yu Wang. 2025. Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design.arXiv preprint arXiv:2511.20048(2025)

    arXiv 2025

  20. [20]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations (ICLR)

    2024

  21. [21]

    Abad, Gregory Van Seghbroeck, Sam Deckers, Alexander Lemmens, and Mohammad Shahrad

    Artjom Joosen, Arjun Agarwal, Cristina L. Abad, Gregory Van Seghbroeck, Sam Deckers, Alexander Lemmens, and Mohammad Shahrad. 2025. Serverless Cold Starts and Where to Find Them. InProceedings of the Twentieth European Conference on Computer Systems (EuroSys). doi:10.1145/3689031.3696073

    doi:10.1145/3689031.3696073 2025

  22. [22]

    Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W Lee, Sangdoo Yun, and Hyun Oh Song. 2025. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction.Advances in Neural Information Processing Systems (2025)

    2025

  23. [23]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAtten- tion. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  24. [24]

    Yueying Li, Jim Dai, and Tianyi Peng. 2025. Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents.arXiv preprint arXiv:2504.07347 (2025)

    Pith/arXiv arXiv 2025

  25. [25]

    Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: efficient serving of LLM-based applications with semantic variable. InProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation(Santa Clara, CA, USA)(OSDI). USENIX Association, USA, Article 50, 17 pages

    2024

  26. [26]

    Liu, Amit Levy, Shadi Noghabi, and Sebastian Burckhardt

    David H. Liu, Amit Levy, Shadi Noghabi, and Sebastian Burckhardt. 2023. Doing More with Less: Orchestrating Serverless Applications without an Orchestrator. In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI). USENIX Association, 1505–1519

    2023

  27. [27]

    Larus, and Haibo Chen

    Qingyuan Liu, Yanning Yang, Dong Du, Yubin Xia, Ping Zhang, Jia Feng, James R. Larus, and Haibo Chen. 2024. Harmonizing Efficiency and Practicability: Opti- mizing Resource Utilization in Serverless Computing with Jiagu. In2024 USENIX Annual Technical Conference (ATC). USENIX Association, 1–17

    2024

  28. [28]

    Gonzalez, and Ion Stoica

    Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. Gonzalez, and Ion Stoica. 2025. Autellix: An Efficient Serving Engine for LLM Agents as General Programs.arXiv preprint arXiv:2502.13965(2025)

    Pith/arXiv arXiv 2025

  29. [29]

    Ashraf Mahgoub, Edgardo Barsallo Yi, Karthick Shankar, Sameh Elnikety, So- mali Chaterji, and Saurabh Bagchi. 2022. ORION and the Three Rights: Sizing, Bundling, and Prewarming for Serverless DAGs. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Carlsbad, CA, 303–320

    2022

  30. [30]

    Manus. 2025. Manus: Hands On AI. https://manus.im/

    2025

  31. [31]

    Cynthia Marcelino, Leonard Guelmino, Thomas Pusztai, and Stefan Nastic. 2025. Databelt: A Continuous Data Path for Serverless Workflows in the 3D Compute Continuum.Journal of Systems Architecture(2025). arXiv:2508.15351

    Pith/arXiv arXiv 2025

  32. [32]

    Cynthia Marcelino and Stefan Nastic. 2024. Truffle: Efficient Data Passing for Data-Intensive Serverless Workflows in the Edge-Cloud Continuum. In2024 IEEE/ACM 17th International Conference on Utility and Cloud Computing (UCC). 53–62. doi:10.1109/UCC63386.2024.00017

    work page doi:10.1109/ucc63386.2024.00017 2024

  33. [33]

    2025.Kimi-Researcher: End-to-End RL Training for Emerging Agentic Capabilities

    Moonshot AI. 2025.Kimi-Researcher: End-to-End RL Training for Emerging Agentic Capabilities. https://moonshotai.github.io/Kimi-Researcher/

    2025

  34. [34]

    Daniel Nichols, Prajwal Singhania, Charles Jekel, Abhinav Bhatele, and Harshitha Menon. 2025. Optimizing Agentic Language Model Inference via Speculative Tool Calls.arXiv preprint arXiv:2512.15834(2025). 12 Act While Thinking: Accelerating LLM Agents via Pattern-Aware Speculative Tool Execution

    arXiv 2025

  35. [35]

    OpenAI. 2025. Introducing Deep Research. https://openai.com/index/ introducing-deep-research/

    2025

  36. [36]

    Zaifeng Pan, Wan-Lu Li, Lianhui Qin, Yida Wang, and Yufei Ding. 2025. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows. arXiv preprint arXiv:2507.07400(2025)

    Pith/arXiv arXiv 2025

  37. [37]

    Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. 2024. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention. arXiv:2405.04437 [cs.LG]

    Pith/arXiv arXiv 2024

  38. [38]

    Yifan Qiao, Shu Anzai, Shan Yu, Haoran Ma, Yang Wang, Miryung Kim, and Harry Xu. 2024. ConServe: Harvesting GPUs for Low-Latency and High-Throughput Large Language Model Serving.arXiv preprint arXiv:2410.01228(2024)

    Pith/arXiv arXiv 2024

  39. [39]

    Kalbarczyk, Tamer Baçar, and Ravishankar K

    Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbigniew T. Kalbarczyk, Tamer Baçar, and Ravishankar K. Iyer

  40. [40]

    Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction.arXiv preprint arXiv:2404.08509(2024)

    Pith/arXiv arXiv 2024

  41. [41]

    Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. 2020. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider. In2020 USENIX Annual Technical Conference (USENIX ATC). USENIX Association, 205–218

    2020

  42. [42]

    Jovan Stojkovic, Tianyin Xu, Hubertus Franke, and Josep Torrellas. 2023. Spec- FaaS: Accelerating Serverless Applications with Speculative Function Execution. In2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA). 814–827. doi:10.1109/HPCA56546.2023.10071120

    doi:10.1109/hpca56546.2023.10071120 2023

  43. [43]

    Yifan Sui, Hanfei Yu, Yitao Hu, Jianxun Li, and Hao Wang. 2024. Pre-Warming is Not Enough: Accelerating Serverless Inference With Opportunistic Pre-Loading. InProceedings of the 2024 ACM Symposium on Cloud Computing(Redmond, WA, USA)(SoCC). Association for Computing Machinery, New York, NY, USA, 178–195

    2024

  44. [44]

    Yifan Sui, Hanfei Yu, Yitao Hu, Jianxun Li, and Hao Wang. 2026. Accelerat- ing ML Inference via Opportunistic Pre-Loading on Serverless Clusters.IEEE Transactions on Parallel and Distributed Systems37, 2 (2026), 472–488

    2026

  45. [45]

    Bulaong, John E

    Kyle Swanson, Wesley Wu, Nash L. Bulaong, John E. Pak, and James Zou. 2025. The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies.Nature646, 8085 (Oct. 2025), 716–723. doi:10.1038/s41586-025-09442-9 Epub 2025-07-29

    doi:10.1038/s41586-025-09442-9 2025

  46. [46]

    Tongyi DeepResearch Team. 2025. Tongyi DeepResearch: A New Era of Open- Source AI Researchers. https://github.com/Alibaba-NLP/DeepResearch

    2025

  47. [47]

    Ying Wang, Zhen Jin, Jiexiong Xu, Wenhai Lin, Yiquan Chen, and Wenzhi Chen

  48. [48]

    AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving.arXiv preprint arXiv:2512.04013(2025)

    Pith/arXiv arXiv 2025

  49. [49]

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Wang, Xuanzhe Liu, and Xin Jin. 2023. Fast Distributed Inference Serving for Large Language Models.arXiv preprint arXiv:2305.05920(2023)

    Pith/arXiv arXiv 2023

  50. [50]

    Hao Wu, Junxiao Deng, Minchen Yu, Yue Yu, Yaochen Liu, Hao Fan, Song Wu, and Wei Wang. 2024. FaaSTube: Optimizing GPU-oriented Data Transfer for Serverless Computing.arXiv preprint arXiv:2411.01830(2024)

    Pith/arXiv arXiv 2024

  51. [51]

    Zhiqiang Xie, Ziyi Xu, Mark Zhao, Yuwei An, Vikram Sharma Mailthody, Scott Mahlke, Michael Garland, and Christos Kozyrakis. 2025. Strata: Hierarchical Context Caching for Long Context Language Model Serving.arXiv preprint arXiv:2508.18572(2025)

    Pith/arXiv arXiv 2025

  52. [52]

    Hongshen Xu, Zihan Wang, Zichen Zhu, Lei Pan, Xingyu Chen, Shuai Fan, Lu Chen, and Kai Yu. 2025. Alignment for Efficient Tool Calling of Large Language Models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 17776–17792

    2025

  53. [53]

    Jingbo Yang, Bairu Hou, Wei Wei, Yujia Bao, and Shiyu Chang. 2025. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse.arXiv preprint arXiv:2502.16002(2025)

    arXiv 2025

  54. [54]

    Naimeng Ye, Arnav Ahuja, Georgios Liargkovas, Yunan Lu, Kostis Kaffes, and Tianyi Peng. 2025. Speculative Actions: A Lossless Framework for Faster Agentic Systems.arXiv preprint arXiv:2510.04371(2025)

    Pith/arXiv arXiv 2025

  55. [55]

    Yi Zhai, Dian Shen, Junzhou Luo, and Bin Yang. 2026. ToolCaching: Towards Efficient Caching for LLM Tool-calling.arXiv preprint arXiv:2601.15335(2026)

    arXiv 2026

  56. [56]

    Chaoyun Zhang, Liqun Li, He Huang, Chiming Ni, Bo Qiao, Si Qin, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2025. UFO3: Weaving the Digital Agent Galaxy.arXiv preprint arXiv:2511.11332(2025)

    arXiv 2025

  57. [57]

    Haoran Zhang, Konstantinos Kallas, Spyros Pavlatos, Rajeev Alur, Sebastian Angel, and Vincent Liu. 2024. MuCache: A General Framework for Caching in Microservice Graphs. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI). USENIX Association, 221–238

    2024

  58. [58]

    Haoran Zhang, Shuai Mu, Sebastian Angel, and Vincent Liu. 2025. CausalMesh: A Causal Cache for Stateful Serverless Computing.Proceedings of the VLDB Endowment(2025)

    2025

  59. [59]

    Lei Zhang, Mouxiang Chen, Ruisheng Cao, Jiawei Chen, Fan Zhou, Yiheng Xu, Jiaxi Yang, Zeyao Ma, Liang Chen, Changwei Luo, Kai Zhang, Fan Yan, KaShun Shum, Jiajun Zhang, Zeyu Cui, Feng Hu, Junyang Lin, Binyuan Hui, and Min Yang. 2026. MegaFlow: Large-Scale Distributed Orchestration System for the Agentic Era.arXiv preprint arXiv:2601.07526(2026)

    arXiv 2026

  60. [60]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Lan- guage Model Programs. InAdvances in Neural Information Processing Systems 37 (NeurIPS). 13

    2024

This paper was first reviewed by grok-4.5 on July 13, 2026.