{"id":"ddc7b478-5a03-4eab-80f2-168ff98c3ba1","arxiv_id":"2607.29069","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Agent serving faces non-inference bottlenecks (up to 48% of latency), a 4.4x serving-capacity loss from long context, and 4.9x cost amplification from snapshot-based sandbox suspension.","lead":"This paper presents Aries, a framework for running and measuring AI-agent workloads, and uses it to show that agent serving is bottlenecked by tool execution and context memory as much as by the LLM itself. It quantifies how retaining context trades accuracy against server capacity and why snapshot-based sandbox suspension is expensive, findings that could reshape cloud infrastructure for autonomous agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sandbox snapshot cost model in §4.3 rests on undisclosed parameters; the 4.9x and 3x claims could invert under plausible pricing.","rationale":"The reader's weakest assumption identifies the §4.3 cost model as the most fragile point. I agree that this is the single most load-bearing concern because Takeaway 3's quantitative claims (the 4.9x amplification and the 3x ideal-system reduction) are not reproducible from the information in the paper. The AWS Lambda MicroVM pricing model [2] is referenced, but the paper does not disclose the per-snapshot cost parameters or the calculation. Since the conclusion that 'aggressive suspension is uneconomical' is central to one of the three main takeaways, and since a moderate change in parameters can invert it, this is more serious than the missing error bars or the security proxy. Nevertheless, the rest of the paper stands on independent measurements (latency breakdown, context capacity, utilization CDFs) that are plausible and reproducible with Aries. The paper's contribution — a trajectory-level experimentation framework and qualitative observations about agent serving — remains valuable. The cost model needs to be disclosed and stress-tested, which fits a conditional acceptance. The reader's verdict of CONDITIONAL is appropriate; I do not see a reason to move to ACCEPT or REJECT. I therefore mark verdict_should_be as UNCHANGED, meaning no change to the reader's verdict.","tokens_in":13819,"tokens_out":6038,"duration_ms":60681,"concrete_test":"Re-run the §4.3 cost model with explicit AWS Lambda MicroVM parameters. Obtain published prices for snapshot storage and capture/restore operations, set snapshot size to the actual sandbox memory footprint distribution from Fig. 9 (e.g., 1–10 GB), use C/R duration values measured from Lambda SnapStart or similar systems (e.g., 50–500 ms), and recompute total cost for the recorded tool-call traces under different keepalive policies. Then perform a sensitivity sweep over each parameter across a plausible range and check whether the 4.9x amplification and 3x ideal-system reduction persist. If any plausible parameter set yields amplification below 1x, the takeaway must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Takeaway 3 — that snapshot-based state management makes aggressive suspension uneconomical, forcing a 'false choice' between idle capacity and snapshot overhead — depends entirely on the analytical cost replay in §4.3. The text states that recorded tool-call traces are replayed against the AWS Lambda MicroVM pricing model [2], but it never specifies the snapshot size, capture/restore duration, per-snapshot price, or the cost formula. These are not incidental details: with a fixed set of tool-call traces, the number of C/R cycles is constant, so the total C/R cost scales linearly with the product of snapshot size, C/R duration, and price. If any of these values differs from the authors' implicit assumptions by 2–3x, the claimed 4.9x total-cost amplification relative to the persistent baseline could drop below 1x — meaning aggressive suspension becomes cheaper than keeping sandboxes resident, and the 'false choice' inverts. The paper also does not define the 'ideal system' that achieves the 3x reduction, so that claim is not independently checkable. This is the most load-bearing concern because it is a quantitative finding that directly supports a key takeaway and can reverse under plausible parameter values, unlike the latency breakdown (§4.1) which is reproducible from open harnesses and the context-capacity ratio (§4.2) which is a straightforward calculation from observed context lengths.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Aries, a full-stack experimentation framework for agentic serving systems. Aries separates task specifications from execution configurations, reconstructs cross-component agent trajectories with correlated telemetry, and provides a uniform interface for stateful tool execution across heterogeneous sandbox substrates. Using Aries, the authors run controlled experiments on three open agent harnesses (OpenClaw, Hermes, OpenHands) and three benchmarks (SWE-Bench Pro, Terminal-Bench 2, DeepResearch Bench), complementing them with production traces from a commercial platform. The main empirical findings are: (1) harness and tool execution contribute 13--48% of end-to-end latency, so token-centric metrics are insufficient; (2) retaining extra context beyond a workload-dependent threshold gives little accuracy benefit while increasing KV-cache pressure and reducing serving capacity (from 176 to 40 resident requests, a 4.4x reduction); and (3) tool sandboxes are mostly idle but bursty, while snapshot-based scale-to-zero approaches are economically unattractive, creating a 'false choice' between idle capacity and snapshot overhead. The paper also includes a security analysis and a vision for agent-native serving systems. The Aries code, toolchain, and production traces are released open source.","tokens_in":14107,"tokens_out":4159,"duration_ms":42098,"significance":"If the measurements are credible, this is a timely and useful empirical contribution to systems research on agentic workloads. The strengths of the paper include the open-source framework and traces, the controlled comparison across three harnesses and benchmarks, and the direct measurement of latency breakdowns and capacity effects. The latency and capacity findings in §4.1 and §4.2 are largely grounded in reproducible experiments and are valuable even if the absolute numbers depend on the specific model/hardware configuration. However, the cost analysis in §4.3, which underpins Takeaway 3, rests on an underspecified pricing model, and the limited sample size with no error bars weakens the accuracy-plateau claims. The paper's vision and proposed research directions are sensible and will likely influence future work, but the load-bearing quantitative claims need to be made fully verifiable before the paper can be accepted as-is.","major_comments":[{"comment":"The analytical replay of tool-call traces against the AWS Lambda MicroVMs pricing model does not disclose the model parameters or cost formula. In particular, the snapshot size, capture/restore duration, per-snapshot price, and the exact billing equations are missing. For a fixed trace, the total C/R cost is linear in the product of these quantities, so the reported 4.9x total-cost amplification, the 65% instance-cost reduction, and the 3x ideal-system reduction cannot be independently checked. Under plausible alternative parameter values, the claimed amplification could invert, which would change the 'false choice' conclusion. Please provide the full cost model, all parameter values with sources, and a sensitivity analysis over a realistic parameter range.","section":"§4.3, Fig. 10"},{"comment":"The experimental section states that 20 tasks per benchmark were sampled and each repeated five times, but the figures report only point estimates without error bars or confidence intervals. The accuracy plateaus in Fig. 6 are read off single-percentage differences (e.g., 95% vs 90% vs 80% across context sizes for OpenClaw on DeepResearch Bench), and the latency breakdowns in Fig. 3 are averages without variance. With n=20 independent tasks per condition, these differences may be within sampling noise, which undermines the strength of Takeaway 2. Please report per-task variance, bootstrap confidence intervals, or a statistical test for the key comparisons.","section":"§4, Figs. 3, 4, 6, 7, 9, 10"},{"comment":"The text states that 'the total cost can be reduced by 3× in an ideal system compared to the persistent-sandbox baseline,' but no definition of this ideal system is given. Without specifying the assumed characteristics (e.g., zero-cost capture/restore, zero idle waste, perfect demand prediction) and showing the derivation, the 3x claim is not reproducible or falsifiable. Please define the ideal system and show the arithmetic that leads to the factor of 3.","section":"§4.3, 'ideal system' claim"}],"minor_comments":[{"comment":"The x-axis labels 'Conv Agent' are ambiguous. Please spell out 'Conversational' and 'Agentic' in the figure or caption.","section":"Fig. 5"},{"comment":"The caption says values are normalized for trace anonymization, but the normalization procedure is not described. Please state what normalization was applied.","section":"Fig. 8"},{"comment":"The use of 'share of Linux kernel CVEs identified by AI tools' as a proxy for agentic vulnerability-discovery capability depends on commit-message matching, which could be noisy. Please acknowledge this limitation and ideally provide a sensitivity check.","section":"§4.4"},{"comment":"The C/R cost methodology is attributed to prior work [43], but the paper does not summarize it. A short description or the relevant equations from [43] would help readers assess the analysis without looking up the reference.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a solid, useful paper. The Aries framework is a real contribution—separating task semantics from execution configuration, correlating telemetry across harness, model, and sandbox, and it's open-sourced. The main findings—harness and tool execution account for 13–48% of end-to-end latency, long context amplifies KV footprint and cuts resident capacity from 176 to 40 concurrent requests, and sandboxes are idle-bursty—are new measurements, and they support a sensible push toward trajectory-aware serving.\n\nThe paper does well to pair production traces with controlled reproduction, and it is honest about the limits of production trace fidelity. The latency-breakdown and capacity results are directly measured and reproducible from the open harnesses; I'd trust those. The context-sufficiency threshold in §4.2 is plausible, though read off single-percentage changes across a sweep; that is a moderate concern, not a fatal one.\n\nThe weak spot is §4.3. The cost analysis replays tool-call traces against AWS Lambda pricing but never states the snapshot size, capture/restore duration, per-snapshot cost, or the cost formula. The stress-test note is right: the 4.9x and 3x claims scale linearly with those parameters and could reverse if the assumed overhead is off by 2–3x. That is not a minor nit—Takeaway 3 rests on it. The paper should disclose the parameters and run a sensitivity sweep, or soften the 'false choice' claim. I'd ask for that in revision.\n\nOther soft spots are secondary: the task sample is 20 tasks x 5 repeats, there are no error bars in the figures, and the security proxy (AI-found CVEs) is a rough indicator—the paper does call it a proxy, so that is fine for what it is.\n\nSo: this is a paper for systems/ML people seriously looking at agent infrastructure. It deserves a thorough referee, not a desk reject. I'd recommend sending it out, with a clear request for the cost-model parameters and sensitivity analysis. The core measurements and the Aries framework are worth keeping.","headline":"A worthwhile characterization paper on agent serving with a genuinely useful trajectory-level framework; the cost-model section needs parameter disclosure before its headline numbers are trusted.","tokens_in":14665,"tokens_out":3767,"would_cite":true,"duration_ms":34252,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent serving bottlenecks lie in tool execution and context retention, not just token generation.","keywords":["agentic serving","LLM serving systems","agent trajectories","tool sandboxes","KV cache management","context retention","serverless snapshots","serving benchmarks"],"falsifier":"Re-run the §4.3 trace replay with publicly documented Lambda MicroVM snapshot capture/restore prices for measured sandbox sizes (e.g., 1-4 GB memory and a few GB disk) and vary the keepalive timeout—if the total-cost curve no longer shows a 4.9x peak at short keepalives, or the persistent baseline is within noise of the optimum, the paper's false-choice claim is not supported.","tokens_in":13696,"feed_emoji":"⚙️","tokens_out":3556,"duration_ms":35553,"temperature":0.7,"pith_summary":"The paper tries to establish that agentic serving cannot be adequately optimized with conventional LLM serving metrics and request-centric infrastructure. It introduces Aries, an experimentation framework that reconstructs end-to-end agent trajectories with correlated telemetry, and uses it to show that harness and tool execution dominate latency, that long context gives diminishing accuracy while cutting capacity, and that tool sandboxes are mostly idle but bursty, making current snapshot-based suspension unprofitable. A sympathetic reader should care because cloud AI infrastructure is being repurposed for agents, and these claims identify concrete, previously hidden bottlenecks.","feed_headline":"Tool execution drives up to 48% of agent task latency","feed_subtitle":"Aries experiments show context retention also cuts serving capacity 4.4x, pointing to trajectory-aware systems.","key_machinery":"Aries: an experimentation framework that makes the agent trajectory the first-class unit of observation. It separates task specification (workload and success semantics) from execution configuration (harness, model backend, sandbox), propagates trajectory and event identifiers across model and tool boundaries, and exposes stateful tool execution through a common interface over heterogeneous sandbox substrates. This lets the authors reconstruct cross-component trajectories, correlate telemetry, and attribute latency and capacity effects to individual stages.","core_discovery":"The paper's central claim is that agentic serving shifts the unit of work from a single model request to a multi-step trajectory, and that current request-centric infrastructure—which optimizes token generation and treats context and tool sandboxes incidentally—misallocates resources. Using the Aries framework, the authors show that harness and tool execution can occupy 13–48% of end-to-end latency; that retaining context beyond workload-specific thresholds plateaus task accuracy while cutting resident serving capacity 4.4x; and that tool sandboxes are idle more than 80% of the time but bursty during tool calls, so snapshot-based scale-to-zero is uneconomical (4.9x cost amplification) and fo","pith_inferences":["If the context-sufficiency thresholds generalize beyond the three benchmarks tested, a runtime could learn per-task or per-trajectory thresholds online and trigger compaction before capacity degrades—something the paper proposes as a direction but does not implement.","The trajectory-correlation approach Aries applies to agent serving could extend to other long-horizon, multi-resource workloads, such as robotics pipelines or multi-agent orchestration, where local logs also fail to reconstruct end-to-end behavior.","The 3x cost-reduction target implied by the analysis suggests a concrete design point: keep sandbox OS state and files externalized and reconstruct on demand instead of snapshotting whole microVMs, though the economic win depends on snapshot pricing assumptions the paper does not fully parameterize."],"forward_implications":["Serving stacks should expose harness and tool phases as first-class, instrumented stages, not hidden between requests.","Context and KV-cache management should be driven by task-progress utility, not by maximum context size.","Sandbox runtimes require elasticity mechanisms that avoid full snapshot capture-restore on every tool call.","Cost and performance evaluation for agent serving should compare tasks solved per token and tasks solved per hour, not raw token throughput.","Sandbox isolation should be workload-tailored to shrink the attack surface as agents grow more capable."],"fun_headline_variants":["Tool execution makes up to 48% of agent latency","Retaining extra context cuts serving capacity 4.4x","Tool sandboxes idle 80% of time, scale-to-zero costly","Agent serving needs trajectory-level metrics"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The economic conclusion that snapshot-based suspension is a false choice rests on assumed snapshot sizes, capture/restore durations, and per-snapshot prices in the AWS Lambda MicroVM pricing model; if those assumptions are off by even 2–3x, the claimed 4.9x cost amplification and the 3x ideal-system reduction could shrink or invert.","fun_headline_variants_meta":{"raw":{"variants":["Tool execution makes up to 48% of agent latency","Retaining extra context cuts serving capacity 4.4x","Tool sandboxes idle 80% of time, scale-to-zero costly","Agent serving needs trajectory-level metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000732,"raw_usage":{"total_tokens":3102,"prompt_tokens":721,"completion_tokens":2381,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":2315}},"tokens_in":465,"tokens_out":2381,"duration_ms":16445,"temperature":1.0,"reasoning_tokens":2315,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T14:10:28.880173+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the §4.3 trace replay with publicly documented Lambda MicroVM snapshot capture/restore prices for measured sandbox sizes (e.g., 1-4 GB memory and a few GB disk) and vary the keepalive timeout—if the total-cost curve no longer shows a 4.9x peak at short keepalives, or the persistent baseline is within noise of the optimum, the paper's false-choice claim is not supported.","supporting_citations":[],"review_version":1}