{"id":"3a288bdb-b4e1-46a1-82b9-a97c71d3e793","arxiv_id":"2608.00881","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AOSpec co-speculates actions and observations in LLM agents, using expected-value decoding and joint action-state verification to hide tool execution latency, achieving 11.8-32.5% end-to-end latency savings in trace replay.","lead":"This paper introduces AOSpec, a system that reduces the time agents wait for tools by speculatively predicting both the agent's next actions and the environment's next observations. It reports 11.8-32.5% lower mean end-to-end latency and up to 42.8% lower p99 latency on agent tasks, with gains growing as model decoding gets faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lossless claim is scoped to filesystem state: JASV verifies only a CoW filesystem root hash (Section 4.3), and non-filesystem inputs/effects are excluded, so a reused fork can diverge from serial execution.","rationale":"I agree with the reader that the weakest load-bearing assumption is the scope of the environment state used by JASV. This is the right focus because 'lossless' is the headline property, and Eq. (4) plus Section 4.3 show the equality is filesystem-only. The rest of the paper is internally consistent: EVD's objective is well-motivated, the latency-concentration measurements are concrete, and the ablations support the claim that joint speculation adds value. The trace-replay evaluation is a reasonable methodology for comparing serving regimes, though it does not resolve the separate question of stochastic actor outputs under live speculation; I regard that as secondary because the filesystem gap affects even deterministic actors and is explicitly admitted. The concern is addressable: narrow the claim to filesystem-managed environments, or extend the sandbox contract to non-filesystem effects. This is a CONDITIONAL outcome rather than a rejection, since the contribution is substantial and the limitation is stated. The proposed test would show whether the exclusion actually causes divergence on the evaluated workloads; if it does not, the conditional can be upgraded.","tokens_in":10946,"tokens_out":7704,"duration_ms":72691,"concrete_test":"Instrument the sandbox to log, for every fork accepted by JASV, the filesystem root hash and a compact hash of non-filesystem state (open network connections and peer counters, wall-clock time, process table) at fork creation and at JASV verification. Run a controlled task where a target action (e.g., `curl -s http://127.0.0.1:PORT/counter`) both writes a file and increments/reads an external counter, plus run the full Terminal-Bench replay suite. If any accepted fork has non-filesystem state at verification that differs from the committed environment, or if AOSpec's final trajectory differs from serial on the controlled task, the lossless guarantee is falsified. If no such case occurs, the exclusion is not practically load-bearing for these workloads.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that AOSpec is lossless and preserves serial semantics. The mechanism that is supposed to guarantee this is JASV, but JASV's environment check in Eq. (4) is ν(S_f)=ν(S_i), and Section 4.3 (Runtime Support) defines ν as a filesystem root hash and states 'Non-filesystem inputs and effects are excluded.' Therefore AOSpec treats two environments as equivalent whenever their filesystem images match, even if network, process, clock, or other non-filesystem state differs. A speculative fork launched from an earlier boundary can execute a target action against a different external state, and the fork's non-filesystem effects are not contained by the sandbox. If the observation produced in the fork differs from what a serial execution would return from the committed environment, AOSpec will commit that observation (Algorithm 1, lines 21-23) and the trajectory diverges. The paper identifies this exclusion itself, but the abstract and Section 4.3 state losslessness without this qualification. On Terminal-Bench workloads, many costly calls are package-install or server-style commands that touch network or service state, so the exclusion is practically relevant rather than a corner case. The claim must be scoped to filesystem-managed state, or the runtime must contain and verify non-filesystem effects.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AOSpec proposes a framework for reducing end-to-end latency in LLM-agent serving by speculating both observations and actions. Observation speculation uses Expected Value Decoding (EVD) to rank candidate observations by expected hidden tool time, while action speculation executes latency-critical target actions in isolated sandboxed forks; Joint Action-State Verification (JASV) accepts a fork only if the speculative action equals the emitted action and the fork's origin environment version equals the committed environment version. The paper claims AOSpec is lossless, breaks the lookahead-accuracy tradeoff, and reduces mean end-to-end latency by 11.8-32.5% with p99 reductions up to 42.8% across Terminal-Bench harness/model configurations, with the observation model transferring to SWE-bench Verified without retraining.","tokens_in":11223,"tokens_out":4649,"duration_ms":43300,"significance":"If the stated results hold, AOSpec addresses a practically important bottleneck in agent serving: tool execution time increasingly dominates as decoding accelerates. The paper's strengths include a concrete measurement of latency concentration, a clear two-level speculation design, an objective (EVD) that targets expected time hidden rather than hit rate, and a broad trace-replay evaluation spanning multiple harnesses, models, and TPOT settings. The transfer experiment to SWE-bench is a useful generalization check. However, the central losslessness guarantee is established only for filesystem-managed state, and the evaluation uses fixed actor trajectories, so the contribution should be framed as latency optimization under a constrained semantics rather than a universally lossless serving system. With appropriate scoping and additional validation, the framework is a meaningful step for low-latency agent serving.","major_comments":[{"comment":"The losslessness claim in the abstract and in Section 4.3 is not supported as stated. The environment-equality test in Eq. (4) is ν(S_f)=ν(S_i), and the text immediately defines ν as a copy-on-write filesystem root hash, stating 'Non-filesystem inputs and effects are excluded.' A speculative fork can therefore be accepted even when network, process, clock, or other non-filesystem state differs from serial execution. If the observation produced in the fork differs from what serial execution would return, Algorithm 1 (lines 21-23) commits that observation and the trajectory diverges. The losslessness claim must be explicitly scoped to filesystem-managed state, or the runtime must isolate and verify non-filesystem effects before the abstract's unqualified claim can stand.","section":"Section 4.3, Eq. (4)"},{"comment":"The claim that JASV 'breaks the lookahead--accuracy tradeoff' is contradicted by the paper's own data. Figure 8(b) shows target-action accuracy declining from 23.7% at distance one to 7.7% at distance ten. JASV removes the multiplicative failure of full-chain prediction, but the accuracy of predicting even a single distant target still degrades with distance. The paper should reframe this as mitigating or reducing the cost of the tradeoff, and it should quantify the net end-to-end latency benefit at the lookahead distances where accuracy has already decayed.","section":"Section 5.5, Figure 8(b)"},{"comment":"The evaluation replays all methods on identical actor trajectories, as the authors acknowledge. Because AOSpec's benefit depends on continuing generation from speculative observations and then discarding mismatched branches, this protocol cannot measure the effect of wrong speculations on the actor's future trajectory. The reported 11.8-32.5% savings are therefore conditional on trajectories not diverging, and the lossless claim is not validated end-to-end under live scheduling. A live or partially-live evaluation, or at minimum an analysis of how often the commit logic produces a different trajectory than serial execution, is needed before the serving-latency claim can be taken at face value.","section":"Section 5.1"}],"minor_comments":[{"comment":"The word 'lossless' appears in the abstract before the filesystem-only qualification introduced in Section 4.3; consider qualifying it at first use to avoid overclaiming.","section":"Abstract"},{"comment":"The sentence 'Our implementation adopts a CoW agent runtime from concurrent work under anonymous review' and the Judy harness footnote make the evaluation depend on artifacts that cannot be inspected; please provide a public implementation or a detailed runtime interface specification.","section":"Section 4.3"},{"comment":"The text says the accuracy of intervening action chains is 'effectively zero at this distance,' but Figure 8(b) reports 7.7% at distance ten; the wording should match the reported data.","section":"Section 5.5, Figure 8(b)"},{"comment":"The rows in Table 2 are misaligned (for example, 'Task query28.7 33.6'); fix the formatting so each removed-input condition and its two metrics are readable.","section":"Table 2"},{"comment":"The similarity kernel K(c,o_j) is not defined; specify how Pattern-EVD mines regular-expression patterns and what representation Embedding-EVD uses, since both are compared in Figure 6(d).","section":"Equation (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper depends on an anonymous CoW runtime and on 'Judy,' an anonymous benchmark submission, for two of the nine configurations. The editor may wish to verify that these anonymous artifacts do not introduce a circularity or disclosure problem, and that the authors' reliance on them is disclosed clearly in the final version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: AOSpec is worth taking seriously. It is the first agent-serving paper I know that co-speculates observations and actions in one loop, and the JASV state-version check is a real mechanism: instead of predicting every intervening action, it launches a target action in a sandboxed fork and later validates both the predicted action and the origin-environment version. That is a clean way to buy long-range runway, and the 11.8–32.5% mean savings on Terminal-Bench trace replay are consistent with the latency-concentration measurements in Section 3.1. EVD is also a sensible objective: rank drafts by probability times estimated tool time, not raw hit rate, and the ablation shows it hides more seconds per trial.\n\nCredit where due: the paper quantifies the three barriers with data, uses disjoint training/test splits, reports SWE-bench transfer without retraining the observation model, and includes ablations for candidate width, component contributions, and input context. The authors are also upfront that trace replay is the community convention because live scheduling can change trajectories; they do not hide that the action drafter remains imperfect at long lookahead.\n\nSoft spots. The headline 'lossless' and 'breaking the lookahead–accuracy tradeoff' are both too strong. Section 4.3 explicitly says non-filesystem inputs and effects are excluded, and JASV's Eq. (4) compares only a filesystem root-hash version. If a speculative fork runs a network or service-state call, its environment may differ from serial execution while the root hash is unchanged; committing that fork's observation (Algorithm 1 lines 21–23) can then diverge from serial semantics. On Terminal-Bench, package-install and server-style commands make this a practical issue, not a corner case. And Figure 8(b) shows target-action accuracy dropping from 23.7% at distance one to 7.7% at distance ten, so JASV does not 'break' the tradeoff; it shifts the dependency from a full chain to a single state-verified target, which is still increasingly hard to predict. The paper would be stronger if the abstract and conclusion said 'lossless with respect to filesystem-managed state' and described JASV as 'mitigating' rather than 'breaking' the tradeoff.\n\nAlso minor: no code or data are released, which matters for a systems paper whose central numbers come from trace replay. The pattern-based EVD similarity function has a few free parameters, but the paper reports sensitivity to the similarity function, which is enough. Citation pattern looks fine; concurrent work is cited where relevant, including the anonymous sandbox runtime.\n\nWho this is for: people building agent-serving stacks or doing speculative execution for LLMs. It deserves a serious referee. I would send it out, with a request to fix the lossless scope claim and release artifacts.","headline":"Solid systems paper with a genuinely new co-speculation mechanism; the main caveat is that 'lossless' is scoped to filesystem-observable state, and the 'breaking the lookahead-accuracy tradeoff' claim is stronger than Figure 8 supports.","tokens_in":11762,"tokens_out":2334,"would_cite":true,"duration_ms":20829,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims AOSpec is a lossless framework that co-speculates observations and actions, cutting mean end-to-end latency by 11.8-32.5% and p99 latency by up to 42.8% without changing agent outputs.","keywords":["LLM agents","speculative execution","latency reduction","tool execution","observation speculation","action speculation","agent serving","lossless speculation"],"falsifier":"Run an agent task where a speculated tool call mutates an external service, for example a request that increments a counter on a remote server, and a later action depends on that counter. In serial execution the external change is visible; under AOSpec the speculative fork's filesystem version is unchanged, so JASV can accept the fork even though the external state differs, producing a different later action. A direct check is to compare the environment version function before and after such an external mutation while the filesystem is untouched; the paper's root-hash design implies the version would not change, which would falsify the losslessness claim.","tokens_in":10747,"feed_emoji":"⚡","tokens_out":15279,"duration_ms":113154,"temperature":0.7,"pith_summary":"This paper tries to establish that the latency of stateful tool execution in LLM agents can be largely hidden by speculating on both sides of the agent-environment loop at once, without changing the final outputs. It introduces AOSpec, which drafts observations ranked by expected tool time saved rather than by likelihood of being correct, and launches latency-critical actions in isolated sandboxed forks whose effects are verified against the committed environment before reuse. The verification step, called Joint Action-State Verification (JASV), accepts a speculative result only when the predicted action matches the emitted action and the fork started from an identical environment version, which the paper argues makes the speculation lossless. On Terminal-Bench traces across four harnesses, five actor models, and five serving speeds, AOSpec reports 11.8-32.5% mean end-to-end latency reduction and up to 42.8% p99 reduction, with the observation model transferring to SWE-bench Verified without retraining. A sympathetic reader would care because as token generation gets faster, tool execution becomes the dominant cost in agent serving, and this is a route to hiding that cost instead of paying it serially.","feed_headline":"Co-speculating actions and observations cuts agent latency up to 32.5%","feed_subtitle":"As decoding accelerates, tool calls dominate agent latency; AOSpec hides that time without changing outputs.","key_machinery":"Three mechanisms carry the argument. Expected Value Decoding (EVD) is a ranking rule $V_o(c) = p_\\theta(c | H_t, a_t) \\hat{T}(c)$, where $\\hat{T}(c)$ estimates a candidate observation's tool time from historical executions weighted by outcome similarity; it selects drafts to maximize expected hidden time rather than hit rate. Joint Action-State Verification (JASV) is the acceptance rule $[\\hat{a}_f = a_i] \\wedge [\\nu(S_f) = \\nu(S_i)]$: a speculative fork $f$ is reused only if its predicted action equals the actor's emitted action and the fork's origin environment version equals the committed environment version. The runtime support is a copy-on-write sandbox whose filesystem image has an immutable root-hash version, making JASV a cheap version comparison rather than a recursive state scan; non-filesystem inputs and effects are explicitly excluded. Together these mechanisms turn long-chain action prediction into target-action verification, and observation prediction into a latency-weighted draft race against execution.","core_discovery":"The central claim, stated on the paper's own terms, is that action-only or observation-only speculation is insufficient for stateful tool workloads, and that jointly speculating both with verification is lossless and faster. AOSpec treats a speculative tool call as a fork of a versioned environment: the fork runs the predicted action in isolation, and JASV accepts it only when the action string and the environment version both equal the committed ones at the moment the actor emits the action. This converts a multi-step lookahead problem, which would require the entire intervening action chain to be predicted correctly, into a target action-state verification problem. Observations that can be predicted from context are handled by Expected Value Decoding, which scores candidate observations by $p_\\theta \\cdot \\hat{T}$, the product of predicted probability and estimated tool time, so the system drafts the outcomes whose correct prediction hides the most wall-clock time rather than the most likely ones. Because mismatched forks are discarded and the action is re-executed from the committed state, the paper argues the committed trajectory is identical to serial execution, and the trace-replay experiments across 45 harness-model-TPOT configurations report latency savings that grow as decoding accelerates.","pith_inferences":["The paper reports that its copy-on-write runtime comes from concurrent work under anonymous review and will be cited later; until that runtime is public, the isolation contract cannot be re-implemented or audited from the paper alone.","Because EVD produces a per-step expected-time score, a natural extension is an adaptive draft budget that spends more observation branches on steps with high predicted tool time and uncertain outcomes, rather than the fixed five-branch budget used in the experiments.","The byte-identical observation verification is conservative; canonicalizing inconsequential fields such as process IDs or timestamps, which the paper mentions as a possibility, would raise the acceptance rate without changing application semantics and is a direct testable modification."],"forward_implications":["If the claims hold, agent workloads whose tool time is concentrated in a few long calls can hide most of that time: the paper reports that 17% of calls lasting at least one second account for 97% of total tool time, and EVD aims drafts specifically at those calls.","Faster decoding makes the method more valuable, not less: AOSpec's mean end-to-end saving rises from 11.8% at 20 ms/token to 32.5% at 1 ms/token, so it complements rather than competes with inference accelerators.","The lookahead ceiling is no longer the multiplicative accuracy of a predicted action chain: JASV can accept a fork launched several steps earlier as long as the target action and origin environment match, and the ablations report target-action accuracy of 7.7% at distance ten, a regime where full-chain prediction is effectively zero.","Tail latency improves more than median latency, with p99 reduction up to 42.8% versus about 10.7% median reduction at 10 ms/token, so the benefit is strongest for the slowest, most user-visible trials.","The observation model transfers across benchmarks: trained only on Terminal-Bench, it still yields 18.9% end-to-end savings at 1 ms/token on SWE-bench Verified against 4.2% for the strongest baseline, suggesting the value structure of tool observations is not benchmark-specific."],"supporting_citations":[{"why":"Terminal-Bench 2.0 traces and harness/model configurations supply the dataset on which all main latency comparisons are measured.","marker":"Merrill et al. 2026"},{"why":"Defines the prior action-speculation baseline (Speculative Actions) whose lossless framework AOSpec extends and against which it is compared.","marker":"Ye et al. 2025"},{"why":"Defines the SpecHop observation-speculation baselines (cache and LLM variants) used as the observation-speculation comparison points.","marker":"Saberi, Rezaei, and Feizi 2026"},{"why":"Represents prior multi-step tool-execution speculation whose full action chain must be predicted; JASV is designed to avoid that lookahead-accuracy collapse.","marker":"Sui et al. 2026"},{"why":"Provides CubeSandbox, demonstrating the snapshot/fork/rollback primitives that AOSpec's isolation contract builds on.","marker":"Tencent Cloud 2026"},{"why":"Provides DeltaBox, a millisecond-level checkpoint/rollback sandbox that supplies another isolation primitive for speculative actions.","marker":"Dong et al. 2026"},{"why":"Supplies SWE-bench Verified, the unseen benchmark used to test cross-benchmark transfer of the observation model.","marker":"Jimenez et al. 2024"},{"why":"Supplies vLLM, the serving backend used to run the drafter models in the implementation.","marker":"Kwon et al. 2023"}],"fun_headline_variants":["Co-speculating actions and observations trims agent latency by up to 32.5%","AOSpec hides tool-call latency with joint action-observation speculation","Joint action-observation speculation hides tool-execution latency","Co-speculating tool actions and outcomes slashes p99 latency 42.8%","Lossless co-speculation trims agent latency by up to 32.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The lossless claim rests on the premise that a sandboxed fork contains every effect of a speculative tool call, because the paper versions only the filesystem and states in Section 4.3 that non-filesystem inputs and effects are excluded; if a tool call changes something outside the filesystem, such as a network service, an external database, or a process outside the sandbox, the version check will not see the divergence.","fun_headline_variants_meta":{"raw":{"variants":["Co-speculating actions and observations trims agent latency by up to 32.5%","AOSpec hides tool-call latency with joint action-observation speculation","Joint action-observation speculation hides tool-execution latency","Co-speculating tool actions and outcomes slashes p99 latency 42.8%","Lossless co-speculation trims agent latency by up to 32.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000804,"raw_usage":{"total_tokens":3594,"prompt_tokens":1067,"completion_tokens":2527,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":2422}},"tokens_in":683,"tokens_out":2527,"duration_ms":16403,"temperature":1.0,"reasoning_tokens":2422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:14:53.670415+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an agent task where a speculated tool call mutates an external service, for example a request that increments a counter on a remote server, and a later action depends on that counter. In serial execution the external change is visible; under AOSpec the speculative fork's filesystem version is unchanged, so JASV can accept the fork even though the external state differs, producing a different later action. A direct check is to compare the environment version function before and after such an external mutation while the filesystem is untouched; the paper's root-hash design implies the version would not change, which would falsify the losslessness claim.","supporting_citations":[],"review_version":2}