Pith. sign in

REVIEW 3 major objections 4 minor 55 references

Agent serving bottlenecks lie in tool execution and context retention, not just token generation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:10 UTC pith:EBP3PUVX

load-bearing objection A worthwhile characterization paper on agent serving with a genuinely useful trajectory-level framework; the cost-model section needs parameter disclosure before its headline numbers are trusted. the 3 major comments →

arxiv 2607.29069 v1 pith:EBP3PUVX submitted 2026-07-31 cs.DC

Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework

Leonid Kondrashov , Hongrui Liu , JooYoung Park , Boxi Zhou , Zonghao Liu , Chengzhi Lu , Riccardo Mancini , Esha Choukse
show 9 more authors
Haris Javaid German Sviridov Tao Peng Chen Zhao Anastasia Avdeeva Aleksei Gusev Marios Kogias Luo Mai Dmitrii Ustiugov
This is my paper
classification cs.DC
keywords agentic servingLLM serving systemsagent trajectoriestool sandboxesKV cache managementcontext retentionserverless snapshotsserving benchmarks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that agentic serving cannot be adequately optimized with conventional LLM serving metrics and request-centric infrastructure. It introduces Aries, an experimentation framework that reconstructs end-to-end agent trajectories with correlated telemetry, and uses it to show that harness and tool execution dominate latency, that long context gives diminishing accuracy while cutting capacity, and that tool sandboxes are mostly idle but bursty, making current snapshot-based suspension unprofitable. A sympathetic reader should care because cloud AI infrastructure is being repurposed for agents, and these claims identify concrete, previously hidden bottlenecks.

Core claim

The paper's central claim is that agentic serving shifts the unit of work from a single model request to a multi-step trajectory, and that current request-centric infrastructure—which optimizes token generation and treats context and tool sandboxes incidentally—misallocates resources. Using the Aries framework, the authors show that harness and tool execution can occupy 13–48% of end-to-end latency; that retaining context beyond workload-specific thresholds plateaus task accuracy while cutting resident serving capacity 4.4x; and that tool sandboxes are idle more than 80% of the time but bursty during tool calls, so snapshot-based scale-to-zero is uneconomical (4.9x cost amplification) and fo

What carries the argument

Aries: an experimentation framework that makes the agent trajectory the first-class unit of observation. It separates task specification (workload and success semantics) from execution configuration (harness, model backend, sandbox), propagates trajectory and event identifiers across model and tool boundaries, and exposes stateful tool execution through a common interface over heterogeneous sandbox substrates. This lets the authors reconstruct cross-component trajectories, correlate telemetry, and attribute latency and capacity effects to individual stages.

Load-bearing premise

The economic conclusion that snapshot-based suspension is a false choice rests on assumed snapshot sizes, capture/restore durations, and per-snapshot prices in the AWS Lambda MicroVM pricing model; if those assumptions are off by even 2–3x, the claimed 4.9x cost amplification and the 3x ideal-system reduction could shrink or invert.

What would settle it

Re-run the §4.3 trace replay with publicly documented Lambda MicroVM snapshot capture/restore prices for measured sandbox sizes (e.g., 1-4 GB memory and a few GB disk) and vary the keepalive timeout—if the total-cost curve no longer shows a 4.9x peak at short keepalives, or the persistent baseline is within noise of the optimum, the paper's false-choice claim is not supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Serving stacks should expose harness and tool phases as first-class, instrumented stages, not hidden between requests.
  • Context and KV-cache management should be driven by task-progress utility, not by maximum context size.
  • Sandbox runtimes require elasticity mechanisms that avoid full snapshot capture-restore on every tool call.
  • Cost and performance evaluation for agent serving should compare tasks solved per token and tasks solved per hour, not raw token throughput.
  • Sandbox isolation should be workload-tailored to shrink the attack surface as agents grow more capable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the context-sufficiency thresholds generalize beyond the three benchmarks tested, a runtime could learn per-task or per-trajectory thresholds online and trigger compaction before capacity degrades—something the paper proposes as a direction but does not implement.
  • The trajectory-correlation approach Aries applies to agent serving could extend to other long-horizon, multi-resource workloads, such as robotics pipelines or multi-agent orchestration, where local logs also fail to reconstruct end-to-end behavior.
  • The 3x cost-reduction target implied by the analysis suggests a concrete design point: keep sandbox OS state and files externalized and reconstruct on demand instead of snapshotting whole microVMs, though the economic win depends on snapshot pricing assumptions the paper does not fully parameterize.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents Aries, a full-stack experimentation framework for agentic serving systems. Aries separates task specifications from execution configurations, reconstructs cross-component agent trajectories with correlated telemetry, and provides a uniform interface for stateful tool execution across heterogeneous sandbox substrates. Using Aries, the authors run controlled experiments on three open agent harnesses (OpenClaw, Hermes, OpenHands) and three benchmarks (SWE-Bench Pro, Terminal-Bench 2, DeepResearch Bench), complementing them with production traces from a commercial platform. The main empirical findings are: (1) harness and tool execution contribute 13--48% of end-to-end latency, so token-centric metrics are insufficient; (2) retaining extra context beyond a workload-dependent threshold gives little accuracy benefit while increasing KV-cache pressure and reducing serving capacity (from 176 to 40 resident requests, a 4.4x reduction); and (3) tool sandboxes are mostly idle but bursty, while snapshot-based scale-to-zero approaches are economically unattractive, creating a 'false choice' between idle capacity and snapshot overhead. The paper also includes a security analysis and a vision for agent-native serving systems. The Aries code, toolchain, and production traces are released open source.

Significance. If the measurements are credible, this is a timely and useful empirical contribution to systems research on agentic workloads. The strengths of the paper include the open-source framework and traces, the controlled comparison across three harnesses and benchmarks, and the direct measurement of latency breakdowns and capacity effects. The latency and capacity findings in §4.1 and §4.2 are largely grounded in reproducible experiments and are valuable even if the absolute numbers depend on the specific model/hardware configuration. However, the cost analysis in §4.3, which underpins Takeaway 3, rests on an underspecified pricing model, and the limited sample size with no error bars weakens the accuracy-plateau claims. The paper's vision and proposed research directions are sensible and will likely influence future work, but the load-bearing quantitative claims need to be made fully verifiable before the paper can be accepted as-is.

major comments (3)
  1. [§4.3, Fig. 10] The analytical replay of tool-call traces against the AWS Lambda MicroVMs pricing model does not disclose the model parameters or cost formula. In particular, the snapshot size, capture/restore duration, per-snapshot price, and the exact billing equations are missing. For a fixed trace, the total C/R cost is linear in the product of these quantities, so the reported 4.9x total-cost amplification, the 65% instance-cost reduction, and the 3x ideal-system reduction cannot be independently checked. Under plausible alternative parameter values, the claimed amplification could invert, which would change the 'false choice' conclusion. Please provide the full cost model, all parameter values with sources, and a sensitivity analysis over a realistic parameter range.
  2. [§4, Figs. 3, 4, 6, 7, 9, 10] The experimental section states that 20 tasks per benchmark were sampled and each repeated five times, but the figures report only point estimates without error bars or confidence intervals. The accuracy plateaus in Fig. 6 are read off single-percentage differences (e.g., 95% vs 90% vs 80% across context sizes for OpenClaw on DeepResearch Bench), and the latency breakdowns in Fig. 3 are averages without variance. With n=20 independent tasks per condition, these differences may be within sampling noise, which undermines the strength of Takeaway 2. Please report per-task variance, bootstrap confidence intervals, or a statistical test for the key comparisons.
  3. [§4.3, 'ideal system' claim] The text states that 'the total cost can be reduced by 3× in an ideal system compared to the persistent-sandbox baseline,' but no definition of this ideal system is given. Without specifying the assumed characteristics (e.g., zero-cost capture/restore, zero idle waste, perfect demand prediction) and showing the derivation, the 3x claim is not reproducible or falsifiable. Please define the ideal system and show the arithmetic that leads to the factor of 3.
minor comments (4)
  1. [Fig. 5] The x-axis labels 'Conv Agent' are ambiguous. Please spell out 'Conversational' and 'Agentic' in the figure or caption.
  2. [Fig. 8] The caption says values are normalized for trace anonymization, but the normalization procedure is not described. Please state what normalization was applied.
  3. [§4.4] The use of 'share of Linux kernel CVEs identified by AI tools' as a proxy for agentic vulnerability-discovery capability depends on commit-message matching, which could be noisy. Please acknowledge this limitation and ideally provide a sensitivity check.
  4. [§4.3] The C/R cost methodology is attributed to prior work [43], but the paper does not summarize it. A short description or the relevant equations from [43] would help readers assess the analysis without looking up the reference.

Circularity Check

0 steps flagged

No significant circularity: the paper's findings are empirical measurements and an external-pricing replay; the only self-citation is a related-work pointer and does not carry the derivation.

full rationale

I walked the paper's derivation chain. §4.1's latency breakdown (13–48% tool/harness share) is a direct measurement from Aries telemetry on open harnesses; nothing is defined in terms of the conclusion. §4.2's context-capacity reduction (176 to 40 resident requests) is arithmetic from observed production context lengths under a stated 25 GB budget, and the accuracy plateau is an empirical sweep result. §4.3 replays recorded tool-call traces against the external AWS Lambda MicroVM pricing model [2]; although the snapshot size and C/R price parameters are not enumerated (a reproducibility/robustness concern, not circularity), the 4.9× and 3× figures emerge from that external model under stated keepalive policies and are not fitted to validate Takeaway 3. §4.4 uses external CVE and exploit-cost data. The paper's only discernible self-citation is [43] in a related-work sentence about systems benchmarks; it is not load-bearing for any central claim, and it points to an externally published ASPLOS paper. No prediction or first-principles result reduces by construction to its own input.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claims rest on measured traces and an analytical cost replay. The main assumptions are that Aries instrumentation is non-perturbative, that the sampled tasks and production traces represent real agent workloads, that the AWS Lambda pricing model applies to the hypothetical sandbox scenario, and that the CVE proxy reflects agent offensive capability. No new physical or conceptual entities are introduced.

free parameters (4)
  • Context window size sweep points = 64K, 96K, 128K, 160K, 192K, 224K, 256K
    Independent variables in the accuracy-capacity sweep (§4.2, Fig 6); the 'sufficiency threshold' is read off these points post hoc.
  • KV cache memory budget = 25GB
    Fixed budget used to derive the 4.4x resident capacity reduction (§4.2, Fig 5).
  • Target task success rate = 50%
    Tasks are sampled to hit a 50% success rate, biasing the trajectory distribution toward error-prone tasks (§4).
  • Snapshot C/R cost model parameters = not stated in text
    The 4.9x/3x cost figures depend on assumed snapshot size, capture/restore duration, and per-snapshot price; the paper cites AWS Lambda MicroVM pricing but does not state the replay parameters in the text (§4.3, Fig 10).
axioms (4)
  • domain assumption Aries instrumentation does not materially perturb the measured agent-serving behavior
    The framework adds telemetry hooks to harnesses and sandboxes (§3, Fig 2); if the overhead changes scheduling or KV-cache residency, the measured timings and capacities would be biased.
  • domain assumption The 20 sampled tasks per benchmark are representative of real agentic serving workloads
    Controlled results (§4) generalize from a small, success-balanced sample to production behavior; the production trace cannot be cross-correlated with these events.
  • domain assumption AWS Lambda MicroVM pricing and snapshot C/R costs apply to the hypothetical agent-serving sandbox scenario
    The cost analysis (§4.3) replays traces against a public pricing model that may not match the commercial platform's actual internal costs.
  • domain assumption Share of Linux kernel CVEs mentioning AI tools is a valid proxy for autonomous offensive capability
    The security argument in §4.4 relies on this proxy; AI-mention in CVE text is noisy and may reflect reporting bias rather than actual agent capability.

pith-pipeline@v1.3.0-daily-deepseek · 13544 in / 11391 out tokens · 110746 ms · 2026-08-03T14:10:28.880173+00:00 · methodology

0 comments
read the original abstract

Autonomous agents challenge conventional LLM serving by coupling repeated inference with persistent context and sandboxed tool execution. We present Aries, a full-stack experimentation framework that separates task semantics from execution configurations, reconstructs cross-component agent trajectories with correlated system telemetry, and exposes stateful tool execution through a consistent interface across heterogeneous sandbox substrates. We use Aries to conduct reproducible experiments on open agent harnesses and benchmarks. We complement these experiments with production traces from a commercial platform, grounding low-level systems research in observed production behavior. Our results show that (1) token-centric metrics miss non-inference bottlenecks, (2) retaining additional context yields diminishing accuracy benefits while reducing serving capacity, and (3) tool sandboxes alternate between long idle periods and short resource bursts, while current snapshot-based state management makes aggressive suspension costly. A complementary security analysis further highlights the need to reduce the sandbox attack surface. We then discuss the vision for agent-native serving systems designed around trajectory-level metrics, adaptive context management, elastic sandbox resource management, and sandboxes with minimized attack surface.

Figures

Figures reproduced from arXiv: 2607.29069 by Aleksei Gusev, Anastasia Avdeeva, Boxi Zhou, Chengzhi Lu, Chen Zhao, Dmitrii Ustiugov, Esha Choukse, German Sviridov, Haris Javaid, Hongrui Liu, JooYoung Park, Leonid Kondrashov, Luo Mai, Marios Kogias, Riccardo Mancini, Tao Peng, Zonghao Liu.

Figure 1
Figure 1. Figure 1: Overview of conventional LLM-serving and emerg [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Aries architecture. The blue-colored components [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Context length and resident capacity comparison [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Latency distributions for LLM inference vs. tool [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: CDFs over the one-hour window of the agent serv [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗
Figure 12
Figure 12. Figure 12: Agent-native AI serving system vision. TB2 SWE DR 0 50 100 Generated tokens/s TB2 SWE DR 0 50 Solved / 1M gen. tok. TB2 SWE DR 0 20 Solved / hour (a) token throughput (b) solved per 1M generated tokens (c) solved per hour OpenHands Hermes OpenClaw [PITH_FULL_IMAGE:figures/full_fig_p006_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 17 linked inside Pith

  1. [1]

    Amazon Web Services. 2026. AWS Lambda MicroVMs. https://aws.amazon.com/ lambda/lambda-microvms/. Accessed: 2026-07-03

    2026

  2. [2]

    Amazon Web Services. 2026. AWS Lambda Pricing. https://aws.amazon.com/ lambda/pricing/. Accessed: 2026-07-03

    2026

  3. [3]

    Nicholas Carlini, Newton Cheng, Keane Lucas, Michael Moore, Milad Nasr, Vinay Prabhushankar, Winnie Xiao, Hakeem Angulu, Evyatar Ben Asher, Jackie Bow, Keir Bradwell, Ben Buchanan, David Forsythe, Daniel Freeman, Alex Gaynor, Xinyang Ge, Logan Graham, Kyla Guru, Hasnain Lakhani, Matt McNiece, Mo- jtaba Mehrara, Renee Nichol, Adnan Pirzada, Sophia Porter, ...

    2026

  4. [4]

    Jinyuan Chen, Jiuchen Shi, Quan Chen, and Minyi Guo. 2025. Kairos: Low-latency multi-agent serving with shared llms and excessive loads in the public cloud. arXiv preprint arXiv:2508.06948 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Ya- dav. 2025. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413 (2025)

    Pith/arXiv arXiv 2025

  6. [6]

    Kexin Chu, Zecheng Lin, Dawei Xiang, Zixu Shen, Jianchang Su, Cheng Chu, Yiwei Yang, Wenhui Zhang, Wenfei Wu, and Wei Zhang. 2025. Selective kv- cache sharing to mitigate timing side-channels in llm inference. arXiv preprint arXiv:2508.08438 (2025)

    arXiv 2025

  7. [7]

    CryptoBriefing. 2026. Codex surges to 6 million active users, overtaking Claude Code’s 2 million. https://cryptobriefing.com/codex-surges-6m-users-overtakes- claude-code/

    2026

  8. [8]

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. 2025. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941 (2025)

    Pith/arXiv arXiv 2025

  9. [9]

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. arXiv preprint (2025)

    2025

  10. [10]

    Luke Emberson. 2026. Disclosure of serious cyber vulnerabilities spiked around the release of Claude Mythos Preview. Epoch AI. https://epoch.ai/data-insights/ cve-severity-spike Accessed: 2026-07-07

    2026

  11. [11]

    Michael Ferdman, Almutaz Adileh, Onur Kocberber, Stavros Volos, Mohammad Alisafaee, Djordje Jevdjic, Cansu Kaynak, Adrian Daniel Popescu, Anastasia Ailamaki, and Babak Falsafi. 2012. Clearing the clouds: a study of emerging scale-out workloads on modern hardware. In Proceedings of the Seventeenth International Conference on Architectural Support for Progr...

    arXiv 2012

  12. [13]

    Yu Gan, Yanqi Zhang, Dailun Cheng, Ankitha Shetty, Priyal Rathi, Nayan Katarki, Ariana Bruno, Justin Hu, Brian Ritchken, Brendon Jackson, et al . 2019. An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems. In Proceedings of the twenty-fourth international conference on architectural support for...

    2019

  13. [14]

    Google Cloud. 2026. About GKE Agent Sandbox. https://docs.cloud.google.com/ kubernetes-engine/docs/concepts/machine-learning/agent-sandbox. Accessed: 2026-07-03

    2026

  14. [15]

    Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2024. A real-world webagent with planning, long context understanding, and program synthesis. InInternational Conference on Learning Representations, Vol. 2024. 52690–52717

    2024

  15. [16]

    Roman Gushchin. 2026. Sashiko: An Agentic Linux Kernel Code Review System. https://sashiko.dev/

    2026

  16. [17]

    Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, et al . 2024. Memserve: Context caching for disaggregated llm serving with elastic memory pool. arXiv preprint arXiv:2406.17565 (2024)

    Pith/arXiv arXiv 2024

  17. [18]

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. 2024. Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads. arXiv:2401.11181 [cs.DC] doi:10.48550/arXiv. 2401.11181

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.11181 2024

  18. [19]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InThe TwelfthInternational Conference on Learning Representations (ICLR 2024). https://openreview.net/forum?id=VTF8yNQM66

    2024

  19. [20]

    Minki Kang, Wei-Ning Chen, Dongge Han, Huseyin A Inan, Lukas Wutschitz, Yanzhi Chen, Robert Sim, and Saravan Rajmohan. 2025. Acon: Optimizing context Conference’17, July 2017, Washington, DC, USA Kondrashov et al. compression for long-horizon llm agents.arXiv preprint arXiv:2510.00615 (2025)

    Pith/arXiv arXiv 2025

  20. [21]

    Sanjay Kariyappa and G Edward Suh. 2026. SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning. arXiv preprint arXiv:2602.22603 (2026)

    arXiv 2026

  21. [22]

    Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, et al. 2009. seL4: Formal verification of an OS kernel. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. 207–220

    2009

  22. [23]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAtten- tion. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23). Association for Computing Machinery, New York, NY, USA, 611...

    arXiv 2023

  23. [24]

    Hanchen Li, Runyuan He, Qiuyang Mang, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Hangrui Zhou, Alvin Cheung, Joseph Gonzalez, and Ion Stoica. 2025. Continuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live. arXiv preprint arXiv:2511.02230 (2025)

    Pith/arXiv arXiv 2025

  24. [25]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al . 2024. Agentbench: Evalu- ating llms as agents. In International Conference on Learning Representations, Vol. 2024. 52989–53046

    2024

  25. [26]

    Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, and Junchen Jiang. 2025. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. (2025). arXiv:2510.09665 [cs.LG] https://arxiv.org/abs/2510.09665

    arXiv 2025

  26. [27]

    Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. 2025. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460 (2025)

    Pith/arXiv arXiv 2025

  27. [28]

    Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E Gonzalez, et al. 2026. Agentix: An Efficient Serving Engine for{LLM} Agents as General Programs. In 23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). 2443–2459

    2026

  28. [29]

    Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al

  29. [30]

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. Gaia: a benchmark for general ai assistants. In International Conference on Learning Representations, Vol. 2024. 9025–9049

    2024

  30. [31]

    Nous Research. 2026. Hermes Agent: The Agent That Grows with You. https: //github.com/NousResearch/hermes-agent. Open-source software repository; accessed July 10, 2026

    2026

  31. [32]

    OpenClaw Contributors. 2026. OpenClaw: Personal AI Assistant. https://github. com/openclaw/openclaw. Open-source software repository; accessed July 10, 2026

    2026

  32. [33]

    Harshith Padigela, Chintan Shah, and Dinkar Juyal. 2025. Ml-dev-bench: Com- parative analysis of ai agents on ml development workflows. arXiv preprint arXiv:2502.00964 (2025)

    Pith/arXiv arXiv 2025

  33. [34]

    Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, and Yufei Ding. 2025. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows. arXiv preprint arXiv:2507.07400 (2025). arXiv:2507.07400 https://arxiv.org/abs/2507.07400

    Pith/arXiv arXiv 2025

  34. [35]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM In- ference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). 118–132. doi:10.1109/ISCA59077. 2024.00019

    arXiv 2024

  35. [36]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. MOONCAKE: trading more storage for less computation — a KVCache-centric architecture for serving LLM chatbot. , Article 10 (2025), 16 pages

    2025

  36. [37]

    Qwen Team. 2026. Qwen3.6-35B-A3B: Agentic Coding Power, Now Open to All. https://qwen.ai/blog?id=qwen3.6-35b-a3b

    2026

  37. [38]

    Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. 2025. Zep: a temporal knowledge graph architecture for agent memory. arXiv preprint arXiv:2501.13956 (2025)

    Pith/arXiv arXiv 2025

  38. [39]

    Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, et al . 2020. Mlperf inference benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 446–459

    2020

  39. [40]

    Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. 2020. Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider. In2020 USENIX Annual Technical Conference (USENIX ATC 20). USENIX Association, 205...

    2020

  40. [41]

    The Terminal-Bench Team. 2025. Terminal-Bench: A Benchmark for AI Agents in Terminal Environments. https://github.com/laude-institute/terminal-bench

    2025

  41. [42]

    Saad Ullah, Praneeth Balasubramanian, Wenbo Guo, Amanda Burnett, Hammond Pearce, Christopher Kruegel, Giovanni Vigna, and Gianluca Stringhini. 2025. From cve entries to verifiable exploits: An automated multi-agent framework for reproducing cves. arXiv preprint arXiv:2509.01835 (2025)

    arXiv 2025

  42. [43]

    Dmitrii Ustiugov, Plamen Petrov, Marios Kogias, Edouard Bugnion, and Boris Grot. 2021. Benchmarking, Analysis, and Optimization of Serverless Func- tion Snapshots. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’21). ACM. doi:10.1145/3445814.3446714

    arXiv 2021

  43. [44]

    Xingyao Wang, Simon Rosenberg, Juan Michelini, Calvin Smith, Hoang Tran, Engel Nyst, Rohit Malhotra, Xuhui Zhou, Valerie Chen, Robert Brennan, and Graham Neubig. 2026. The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents. In Proceedings of the 9th Conference on Machine Learning and Systems (MLSys ’26). arXiv:2511....

    Pith/arXiv arXiv 2026

  44. [45]

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al . 2024. Os- world: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems 37 (2024), 52040–52094

    2024

  45. [46]

    Minrui Xu, Jiani Fan, Xinyu Huang, Conghao Zhou, Jiawen Kang, Dusit Niyato, Shiwen Mao, Zhu Han, Kwok-Yan Lam, et al. 2025. Forewarned is forearmed: A survey on large language model-based agents in autonomous cyberattacks. arXiv preprint arXiv:2505.12786 (2025)

    Pith/arXiv arXiv 2025

  46. [47]

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang

  47. [48]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. 𝑡𝑎𝑢- bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv preprint arXiv:2406.12045 (2024)

    Pith/arXiv arXiv 2024

  48. [49]

    Advances in Neural Information Processing Systems 38 (2026), 17577–17604

    A-mem: Agentic memory for llm agents. Advances in Neural Information Processing Systems 38 (2026), 17577–17604

    2026

  49. [50]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. 2022. Orca: A distributed serving system for{Transformer-Based} generative models. In16th USENIX symposium on operating systems design and implementation (OSDI 22). 521–538

    2022

  50. [51]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022)

    Pith/arXiv arXiv 2022

  51. [52]

    Gonzalez, Clark W

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Struc- tured Language Model Programs. In Advances in Neural Information Processing Systems (NeurIPS ’24). https://arxiv.org/abs/2312.07104

    Pith/arXiv arXiv 2024

  52. [53]

    Andy Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Wang, Junrong Wu, Kyleen Liao, Jiliang Li, Jinghan Hu, et al. 2026. Bountybench: Dollar impact of ai agent attackers and defenders on real-world cybersecurity systems. Advances in Neural Information Processing Systems 38 (2026)

    2026

  53. [54]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://openreview.net/forum?id=oKn9c6ytLx

    2024

  54. [55]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Associ- ation, Santa Clara, CA, 193–210. https://www.usenix.org/co...

    2024

  55. [2026]

    arXiv preprint arXiv:2601.11868 (2026)

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868 (2026)

    Pith/arXiv arXiv 2026