Pith. sign in

REVIEW 2 major objections 5 minor 46 references

A forked probe from the running model can start tool calls early, cutting agent tail latency without extra predictors or training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 03:12 UTC pith:N46EIBTI

load-bearing objection Training-free self-fork for agent tool wait is real, useful, and bounded; the 18% GAIA P95 is directionally solid but not cleanly isolated from D3 and single-run API noise. the 2 major comments →

arxiv 2607.03333 v1 pith:N46EIBTI submitted 2026-07-03 cs.DC cs.AIcs.LG

SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

Huajun Bai , Weiwei Lv , Huichuan Zheng , Youyou Lu , Jiwu Shu This is my paper
classification cs.DC cs.AIcs.LG
keywords LLM agentstool usespeculative executioninference accelerationself-speculationprefix cachechain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Agent systems often wait on external tools after the model finishes reasoning, and that wait eats a large share of end-to-end time. SPORK claims the same model already knows its next tool early: a short forced probe forked after the first token predicts the tool name with high accuracy, so the controller can dispatch the tool while the remaining chain-of-thought still decodes. A cost model says speculation pays when accepted overlap beats probe overhead; three mechanisms push that balance—prefix-cache sharing cuts probe cost, a logprob confidence gate filters bad guesses, and rejected probes recycle as draft tokens for the main stream. On real web-tool GAIA with Qwen3-32B, P95 falls 18% while exact-match accuracy stays within one point of the serial baseline, with the same idea holding across model sizes and dense versus mixture-of-experts architectures when thinking-mode CoT is long enough.

Core claim

Instruction-tuned models in thinking mode already expose their next tool call early enough that a training-free self-fork can hide tool latency behind remaining decode: a start-of-generation probe predicts Qwen3-32B tool names at 74.6–99.6% across five benchmarks, and a strict name-and-arguments gate plus fallback keeps the agent lossless while cutting GAIA P95 from 131.9 s to 108.1 s.

What carries the argument

SPORK’s self-speculative fork: after the main stream’s first token, a prefix-cache-sharing probe with a forced tool-call opener emits a candidate call; a min-span logprob gate decides whether to dispatch the tool early; on exact match the result is ready when reasoning ends, otherwise serial fallback runs and the probe’s verified prefix is reused as speculative-decoding draft tokens (D1–D3 against the EQ1 break-even).

Load-bearing premise

The model must already reveal its next tool early enough, with separable confidence and a long enough thinking-mode reasoning window, so that accepted overlap beats the cost of running the probe.

What would settle it

Run the same real-tool agent suite in no-think mode or with a native tool format that diverges from the forced probe: if name accuracy collapses or speedup falls below 1× as the paper’s own tau2 no-think and XML boundary cases predict, the self-speculation claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. SPORK is a training-free controller that accelerates agentic LLM inference by forking a forced tool-call probe from the running model’s own prefix-cached state, dispatching the predicted tool early, and overlapping tool latency with remaining chain-of-thought. Three mechanisms map to a simple cost model (EQ1): D1 prefix-cache fork (lowers T_oh), D2 logprob confidence gate (raises α·t_overlap), and D3 partial-token accept of rejected probes as speculative-decoding drafts (lowers T*_base on misses). On real tools, the full system cuts Qwen3-32B GAIA P95 from 131.9 s to 108.1 s (−18%) with EM within 1 pp of baseline; similar directional gains appear on HotpotQA and a tau2 latency sweep, and across 4B–32B dense and a MoE model. Correctness is preserved by a strict name+arguments match gate and serial fallback.

Significance. If the result holds, SPORK fills a practical gap between token-level speculative decoding (which does not issue tools earlier) and prior action-level speculation systems that need auxiliary predictors, verifiers, or historical traces. The day-one, API-compatible design, open-source controller, falsifiable EQ1 break-even condition (validated within ~2% residual on tau2/GAIA), and explicit operating envelope (no-think mode, format divergence, short tools) are genuine systems contributions. Real-tool evaluation (GAIA web search, Wikipedia API) and multi-architecture generalization strengthen the claim beyond synthetic stalls. Orthogonality to ngram speculative decoding is demonstrated by stacking, which is useful for production stacks.

major comments (2)
  1. [§6.4, Figure 12; Abstract; Appendix D] §6.4 and Figure 12 attribute best P95 to D1+D2+D3 and mean/P50 to D1, but do not report GAIA P95 (or the accepted-turn t_overlap distribution) for D1+D2 with D3 off. The abstract and §1 frame the 18% primarily as accept-path tool overlap (“on acceptance, the tool result is ready when reasoning ends”), while Appendix D’s BrowseComp decomposition shows mean realized t_overlap of only 1.03 s despite large mean T_tool. Without the missing D1+D2 P95 arm on the same GAIA N=165 run, the causal weight of tool-overlap versus D3 draft recovery (and single-run variance) for the headline 131.9→108.1 s figure remains under-isolated. Please add that ablation (or an equivalent per-turn breakdown of saved tool wait vs. saved tool-call decode) so the central latency claim is attributed cleanly.
  2. [§6.2; Abstract] §6.2 reports a single-seed P95 on N=165 real-API queries (~8 queries at the 95th percentile) under the same search-API and vLLM-batching nondeterminism the paper cites for EM noise (§6.3–6.4). EQ1 is well calibrated on mean speedup, but the abstract’s precise 18% P95 number is a tail statistic on a thin sample. Either multi-seed P95 (or bootstrap CIs) on GAIA, or a clearer statement that the robust claim is directional tail reduction under the EQ1 envelope rather than a single-run 18% point estimate, would make the load-bearing latency claim proportionate to the evidence.
minor comments (5)
  1. [§2.2, Appendix A, EQ1] §2.2 / Appendix A: EQ1 is presented in two slightly different forms (uniform T_oh vs. accept/reject-specific overhead). A single canonical equation in the main text with the Appendix A expansion would reduce reader friction when checking the tau2 residual claims.
  2. [Abstract; §4.3; §5] §4.3 and §5: D3 is correctly described as an engine-side prototype, but the abstract’s “thin controller over standard completion APIs” phrasing can be read as covering full D1+D2+D3. A one-sentence clarification that the open HTTP path is D1+D2 and D3 needs the SporkProposer integration would avoid overclaim.
  3. [Figure 10; Table 3] Figure 10 uses per-panel y-scales and mixed HTTP/engine baselines (Table 3). A short caption note that each model is compared only to its own baseline under the same serving mode would prevent cross-panel misreading of absolute seconds.
  4. [§6.5, Figure 13] §6.5 cross-model case study is valuable; stating GPU count and whether the 4B drafter was co-located or on a second GPU more prominently in the figure caption would make the “avoids a second served model” claim easier to audit.
  5. [Throughout; §3.2] Typos/style: “Spork” vs “SPORK” capitalization is inconsistent between title/abstract and body; “think-end” / “no-think” could be defined once in §3.2 for readers outside the Qwen3 ecosystem.

Circularity Check

1 steps flagged

Empirical systems paper with an accounting cost model and measured speedups; no derivation that redefines its target by construction.

specific steps
  1. fitted input called prediction [§4.2 Threshold selection / Figure 8]
    "At θ=0.90, the gate achieves 88% precision with 100% recall (F1=0.937), filtering 77% of all probes while retaining every correct one. We select θ=0.90 as the operating point that maximizes F1."

    θ is chosen by maximizing F1 on the same GAIA probe logprob distribution that the gate later uses. This is ordinary hyperparameter selection, not a derivation that redefines latency. It does not force the headline P95 claim (which is measured end-to-end against external tools), so it is only a minor fitted-input step and does not raise the score above 1.

full rationale

SPORK is a training-free systems controller evaluated on external agent benchmarks (GAIA, HotpotQA, tau2) against serial/ngram baselines. EQ1 is an accounting identity over measurable quantities (α, t_overlap, T_oh, T*_base), not a fitted law that forces the headline latency claim. The break-even condition is validated on held-out operating points (tau2 latency sweep residual ≤1.84%; real-tool GAIA/HotpotQA placement). Threshold θ=0.90 is a standard hyperparameter chosen by F1 on probe logprobs, not a circular proof of the 18% P95 result. Insights 1–3 are empirical measurements of fork accuracy, confidence separability, and prefix overlap; D1–D3 are engineering mechanisms that target EQ1 terms. Self-citations are absent as load-bearing uniqueness theorems. The only minor circular-adjacent element is ordinary threshold selection on the same probe distribution later used for gating, which does not force the end-to-end wall-time claim. Score 1 reflects that minor hyperparameter fit, not a self-definitional derivation.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The central claim rests on measurable serving assumptions and one fitted gate threshold, not on new physical entities. Load-bearing background is standard LLM serving (prefix KV cache, logprobs, speculative-decoding verification) plus the empirical regularity that thinking-mode models reveal tool intent early. The main free parameter is the confidence threshold θ; invented machinery is the controller/proposer, not an unfalsifiable object.

free parameters (3)
  • confidence threshold θ = 0.90
    Min top-1 probability over the tool-name span; selected at 0.90 by F1 sweep on 997 GAIA probes (precision 88%, recall 100%). Directly controls α and wasted dispatches.
  • retry budget R and CoT token step s
    Control how often D2 re-probes as more chain-of-thought arrives; affect probe traffic and late supersession behavior (Appendix H).
  • first-token / probe timeout = 3 s (BrowseComp)
    BrowseComp used a 3 s first-token threshold; timeouts abort probes (3.4% of turns) and force serial fallback.
axioms (5)
  • domain assumption Prefix KV-cache sharing between concurrent main and fork requests makes probe prefill near-zero after the main’s first token.
    D1’s overhead claim (1.6 s → 0.35 s) depends on vLLM/SGLang-style prefix caching (§4.1).
  • domain assumption Thinking-mode chain-of-thought is long enough (≥~2 s in reported workloads) to host probe decode and create a positive overlap window.
    Stated operating envelope; no-think mode collapses the window (§6.6).
  • domain assumption Strict exact match of tool name and serialized arguments is necessary and sufficient to preserve serial agent semantics on accepted turns.
    Correctness argument for the gate; name-only acceptance is deliberately rejected (§4.2).
  • ad hoc to paper Only read-only tools may be speculated; write/non-idempotent tools always take the serial path.
    Safety restriction in §5; without sandbox/checkpoint rollback, mis-speculation would corrupt state.
  • domain assumption Speculative-decoding verification preserves the target model’s greedy output distribution when recycling rejected probe prefixes (D3).
    Inherited from Leviathan et al. speculative decoding; used to claim D3 does not change correctness (§6.3).
invented entities (2)
  • SPORK self-speculative fork controller (D1+D2) and SporkProposer (D3) independent evidence
    purpose: Issue forced tool-call probes from the running model, gate dispatch on min-span logprob, and inject rejected probe tokens as per-request drafts.
    The system is the contribution; independent evidence is the open-source implementation and reported benchmarks, not an external physical detection channel.
  • EQ1 speculative-overlap cost model (α, t_overlap, T_oh, T*_base) independent evidence
    purpose: Predict when self-speculation breaks even and map D1–D3 to specific latency terms.
    Accounting model derived in Appendix A; validated against tau2/GAIA operating points rather than postulated as a new force.

pith-pipeline@v1.1.0-grok45 · 26086 in / 3677 out tokens · 36476 ms · 2026-07-12T03:12:24.595660+00:00 · methodology

0 comments
read the original abstract

LLM agents are becoming a common interface for research, coding, and question answering, yet their Thought-Action-Observation loop is often serial: the model reasons, emits a tool call, then idles the GPU until the result returns. This wait consumes 16-37% of wall time in our workloads and 35-61% in prior reports. Speculative tool execution can hide this wait, but existing systems need auxiliary predictors, historical traces, or static workflow graphs, leaving a gap for training-free, day-one deployment. We observe that the model can be its own predictor: a probe forked at the start of generation predicts Qwen3-32B's upcoming tool name with 74.6-99.6% accuracy across five benchmarks. We present SPORK (Self-sPeculative fORKing), a training-free controller that dispatches the speculated tool call early, overlapping its execution with the remaining chain-of-thought decode. A cost model captures when speculation breaks even, and each component improves one of its terms: a prefix-cache fork cuts probe cost, a confidence gate filters mispredictions, and partial-token accept turns rejected probes into speculative-decoding drafts. On acceptance, the tool result is ready when reasoning ends; on rejection, SPORK falls back to serial execution with no correctness penalty. On real-tool benchmarks, SPORK cuts Qwen3-32B's GAIA P95 by 18% (131.9 to 108.1 s); the mechanism holds across model sizes from 4B to 32B and across dense and mixture-of-experts models, with task accuracy within 1 pp of baseline or better wherever measured. SPORK deploys as a thin controller over standard completion APIs (no retraining, no auxiliary models, no offline traces) and is orthogonal to token-level speculative decoding. SPORK is open source at https://github.com/baihuajun24/spork.

Figures

Figures reproduced from arXiv: 2607.03333 by Huajun Bai, Huichuan Zheng, Jiwu Shu, Weiwei Lv, Youyou Lu.

Figure 1
Figure 1. Figure 1: Tool intent is visible early. A speculative thread, forked at the main thread’s first token with a forced tool-call prefix, predicts the exact tool call that the main thread emits ∼2,000 CoT tokens later. tool-assisted question answering [3, 9, 20, 25, 29, 39]. In these systems, the LLM acts as the controller: it reasons over the user request, emits a structured tool call, waits for the external tool to fi… view at source ↗
Figure 2
Figure 2. Figure 2: Tool time is significant across agent tasks. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Fork accuracy rises monotonically with CoT context. On GAIA (Qwen3-32B), args-exact accuracy climbs from 7.6% at fork-at-start to 61.0% at 80% CoT and 97.5% at think-end. Later probes are more accurate but leave a smaller overlap budget. confidence gating. The remaining challenge is argument ac￾curacy, which Insight 2 addresses. 3.2 Insight 2: Confidence Improves with Chain-of-Thought The useful operating … view at source ↗
Figure 5
Figure 5. Figure 5: Most rejected probes share recoverable pre [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Spork method overview. Top: the baseline turn is strictly serial: (re)prefill, CoT decode, tool-call decode, then a stall while the tool executes. Bottom: Spork forks the KV cache after the main’s first token and launches a forced tool-call probe (D1); once the probe’s logprob confidence clears the gate (D2), the predicted tool executes during the remaining decode, producing the overlap 𝑡overlap. A strict … view at source ↗
Figure 8
Figure 8. Figure 8: D2 confidence gate threshold selection (GAIA N=127, 997 probes). (a) Correct probes cluster above 𝜃=0.90; incorrect probes spread below. The threshold cleanly sepa￾rates them. (b) Precision/Recall/F1 vs. 𝜃: the operating point 𝜃=0.90 maximizes F1 (0.937) with 88% precision and 100% re￾call. D2 filters 77% of probes (saving wasted tool executions) while committing every probe that would have been correct. a… view at source ↗
Figure 9
Figure 9. Figure 9: D3 treats the rejected fork as a draft for [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Spork reduces 𝑃95 tail latency across models and benchmarks. Teal: baseline; orange: the paper-facing Spork configuration for each run (engine D1+D2+D3; HTTP D1+D2). 𝑃95 latency on GAIA, HotpotQA, and tau2 (2 s tool floor) for Qwen3.5-35B-A3B, Qwen3-32B, and Qwen3-4B (note the per-panel 𝑦-scales); the percentage above each pair is the 𝑃95 reduction. Spork generalizes across model scale (4B→32B) and archit… view at source ↗
Figure 11
Figure 11. Figure 11: Spork preserves task accuracy. Paired com￾parison of baseline (teal) vs. Spork (orange) exact-match scores across benchmarks and model sizes. Quality stays within 1 pp of baseline on all settings and sometimes im￾proves (+1.5 pp EM on HotpotQA with Qwen3-32B). The strict gate prevents speculative side effects from corrupting conversation history. 6.2 End-to-End Speedup [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
Figure 13
Figure 13. Figure 13: Self-speculation trades a slower probe for [PITH_FULL_IMAGE:figures/full_fig_p011_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Spork stays at or above break-even across the operating envelope. Mean per-task speedup vs. tool latency on the tau2 sweep (N=155). Full Spork (D1+D2+D3) stays above break-even and rises with tool latency; the gated D1+D2 line approaches break-even on the shortest floors, matching EQ1’s 𝛼 · 𝑡overlap ≥ 𝑇oh condition. Stars mark the two real-tool operating points (Qwen3-32B mean speedup): near-neutral Hotpo… view at source ↗
Figure 15
Figure 15. Figure 15: Qwen3-4B 𝑃95 latency across benchmarks (scaling boundary). Spork vs. baseline (GAIA/HotpotQA: engine D1+D2+D3 vs. ngram spec-dec; tau2: HTTP D1+D2 vs. serial). tau2 sees a modest gain, GAIA is marginally posi￾tive, and HotpotQA is neutral. Quality is preserved or im￾proved (GAIA EM 29 → 36/165; HotpotQA EM 40 → 39/200, within −1 pp). Benchmark N Baseline 𝑃95 Spork 𝑃95 Speedup GAIA 165 29.95 s 28.67 s 1.03… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 14 linked inside Pith

  1. [1]

    Agrawal, N

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In18th USENIX Symposium on Operating Systems Design and Implementation. USENIX Association, 2024

    2024

  2. [2]

    Z. An, H. Bai, Z. Liu, D. Li, and E. Barsoum. PARD: Accelerating LLM inference with low-cost PARallel draft model adaptation. InIn- ternational Conference on Learning Representations (ICLR), 2026. URL https://arxiv.org/abs/2504.18583

    arXiv 2026

  3. [3]

    Claude code.https://www.claude.com/product/claude- code, 2025

    Anthropic. Claude code.https://www.claude.com/product/claude- code, 2025

    2025

  4. [4]

    Barres, H

    V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan.𝜏 2-bench: Evalu- ating conversational agents in a dual-control environment, 2025. URL https://arxiv.org/abs/2506.07982

    Pith/arXiv arXiv 2025

  5. [5]

    T. Cai, Y. Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao. Medusa: Simple LLM inference acceleration framework with multiple decod- ing heads.International Conference on Machine Learning, 2024

    2024

  6. [6]

    D. Choi, K. Park, W. Song, S. Dingliwal, S. M. Jayanthi, J. Shin, and A. Galstyan. IdleSpec: Exploiting idle time via speculative planning for LLM agents, 2026. URLhttps://arxiv.org/abs/2605.22154

    Pith/arXiv arXiv 2026

  7. [7]

    Y. Fu, J. Chen, S. Zhu, Z. Fu, Z. Dai, Y. Zhuang, Y. Ma, A. Qiao, T. Ros- ing, I. Stoica, and H. Zhang. Efficiently scaling LLM reasoning with certaindex, 2024. URLhttp://arxiv.org/abs/2412.20993

    Pith/arXiv arXiv 2024

  8. [8]

    Y. Fu, L. Xue, Y. Huang, A.-O. Brabete, D. Ustiugov, Y. Patel, and L. Mai. ServerlessLLM: Low-latency serverless inference for large language models. In18th USENIX Symposium on Operating Systems Design and Implementation, pages 135–153. USENIX Association, 2024

    2024

  9. [9]

    GitHub Copilot.https://github.com/features/copilot, 2025

    GitHub. GitHub Copilot.https://github.com/features/copilot, 2025

    2025

  10. [10]

    J. Hu, J. Xu, Z. Liu, Y. He, Y. Chen, H. Xu, et al. DeepServe: Serverless large language model serving at scale, 2025. URLhttps://arxiv.org/ abs/2501.14417

    Pith/arXiv arXiv 2025

  11. [11]

    Huang, W

    Z. Huang, W. Zeng, T. Fu, T. Liu, Y. Sun, K. Hong, X. Yang, C. Liu, Y. Li, Q. Zhang, G. Dai, Z. Zhu, and Y. Wang. Reducing latency of LLM search agent via speculation-based algorithm-system co-design,

  12. [12]

    URLhttp://arxiv.org/abs/2511.20048

    arXiv

  13. [13]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues?International Conference on Learning Representations, 2024

    2024

  14. [14]

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large lan- guage model serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023

    2023

  15. [15]

    Leviathan, M

    Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from trans- formers via speculative decoding. InInternational Conference on Ma- chine Learning, pages 19274–19286, 2023

    2023

  16. [16]

    Y. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees, 2024. URLhttps://arxiv. org/abs/2406.16858

    Pith/arXiv arXiv 2024

  17. [17]

    Y. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty.International Conference on Machine Learning, 2024

    2024

  18. [18]

    Y. Li, F. Wei, C. Zhang, and H. Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test, 2025. URLhttps://arxiv.org/abs/2503.01840

    Pith/arXiv arXiv 2025

  19. [19]

    C. Lin, Z. Han, C. Zhang, Y. Yang, F. Yang, C. Chen, and L. Qiu. Par- rot: Efficient serving of llm-based applications with semantic variable. InProceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation. USENIX Association, 2024

    2024

  20. [20]

    Mahgoub, E

    A. Mahgoub, E. B. Yi, K. Shankar, S. Elnikety, S. Chaterji, and S. Bagchi. ORION and the three rights: Sizing, bundling, and prewarming for serverless DAGs. In16th USENIX Symposium on Operating Sys- tems Design and Implementation, pages 303–320. USENIX Association, 2022

    2022

  21. [21]

    Manus: Hands on ai.https://manus.im/, 2025

    Manus. Manus: Hands on ai.https://manus.im/, 2025

    2025

  22. [22]

    Mialon, C

    G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom. GAIA: A benchmark for general AI assistants. InInternational Con- ference on Learning Representations (ICLR), 2024. URLhttps://arxiv. org/abs/2311.12983

    Pith/arXiv arXiv 2024

  23. [23]

    Mohammadi, N

    B. Mohammadi, N. Potamitis, L. Klein, A. Arora, and L. Bindschaedler. Atomix: Timely, transactional tool use for reliable agentic workflows,

  24. [24]

    URLhttp://arxiv.org/abs/2602.14849

    Pith/arXiv arXiv

  25. [25]

    Nichols, P

    D. Nichols, P. Singhania, C. Jekel, A. Bhatele, and H. Menon. Optimiz- ing agentic language model inference via speculative tool calls, 2025. URLhttp://arxiv.org/abs/2512.15834

    arXiv 2025

  26. [26]

    Oliaro, Z

    G. Oliaro, Z. Jia, D. Campos, and A. Qiao. SuffixDecoding: Extreme speculative decoding for emerging AI applications, 2024. URLhttps: //arxiv.org/abs/2411.04975

    arXiv 2024

  27. [27]

    Introducing deep research.https://openai.com/index/ introducing-deep-research/, Feb

    OpenAI. Introducing deep research.https://openai.com/index/ introducing-deep-research/, Feb. 2025

    2025

  28. [28]

    S. G. Patil, H. Mao, F. Yan, C. C.-J. Ji, V. Suresh, I. Stoica, and J. E. Gon- zalez. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. InInternational Conference on Machine Learning (ICML), 2025

    2025

  29. [29]

    Qwen3.5: Towards native multimodal agents, 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, 2026. URL https://qwen.ai/blog?id=qwen3.5

    2026

  30. [30]

    A. Saxena. Prompt lookup decoding, 2023. URLhttps://github.com/ apoorvumang/prompt-lookup-decoding. Model-free n-gram draft- ing; basis of vLLM’s ngram speculative decoding

    2023

  31. [31]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools.Advances in Neural Infor- mation Processing Systems, 36, 2024

    2024

  32. [32]

    Y. Song, Z. Mi, H. Xie, and H. Chen. PowerInfer: Fast large language model serving with a consumer-grade GPU. InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (SOSP),

  33. [33]

    URLhttps://arxiv.org/abs/2312.12456

    Pith/arXiv arXiv

  34. [34]

    Stojkovic, T

    J. Stojkovic, T. Xu, H. Franke, and J. Torrellas. SpecFaaS: Accelerating serverless applications with speculative function execution. In2023 IEEE International Symposium on High-Performance Computer Archi- tecture, pages 814–827, 2023. doi: 10.1109/HPCA56546.2023.10071120

    doi:10.1109/hpca56546.2023.10071120 2023

  35. [35]

    Y. Sui, H. Zhao, R. Ma, Z. He, H. Wang, J. Li, and Y. Yang. Act while thinking: Accelerating LLM agents via pattern-aware speculative tool 13 Huajun Bai, Weiwei Lv, Huichuan Zheng, Youyou Lu, and Jiwu Shu execution, 2026. URLhttp://arxiv.org/abs/2603.18897

    Pith/arXiv arXiv 2026

  36. [36]

    C.-E. Sun, L. Liu, G. Yan, Z. Wang, and T.-W. Weng. LLM agents al- ready know when to call tools – even without reasoning, 2026. URL https://arxiv.org/abs/2605.09252

    Pith/arXiv arXiv 2026

  37. [37]

    J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese. BrowseComp: A simple yet challenging benchmark for browsing agents, 2025. URLhttps://arxiv. org/abs/2504.12516

    Pith/arXiv arXiv 2025

  38. [38]

    Yang et al

    A. Yang et al. Qwen3 technical report, 2025. URLhttps://arxiv.org/ abs/2505.09388

    Pith/arXiv arXiv 2025

  39. [39]

    C. Yang, Q. Si, Y. Duan, Z. Zhu, C. Zhu, Q. Li, M. Chen, Z. Lin, and W. Wang. Dynamic early exit in reasoning models. InInternational Conference on Learning Representations (ICLR), 2026. doi: 10.48550/ arXiv.2504.15895. URLhttps://arxiv.org/abs/2504.15895

    arXiv 2026

  40. [40]

    R. Yang, H. Bai, S. Liu, G. Yu, et al. SpecExit: Accelerating large rea- soning model via speculative exit, 2025. URLhttps://arxiv.org/abs/ 2509.24248

    arXiv 2025

  41. [41]

    Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InConference on Empirical Methods in Natural Language Processing, 2018

    2018

  42. [42]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing reasoning and acting in language models. InIn- ternational Conference on Learning Representations, 2023

    2023

  43. [43]

    N. Ye, A. Ahuja, G. Liargkovas, Y. Lu, K. Kaffes, and T. Peng. Specu- lative actions: A lossless framework for faster agentic systems. 2026. doi: 10.48550/arXiv.2510.04371. URLhttp://arxiv.org/abs/2510.04371

    Pith/arXiv arXiv doi:10.48550/arxiv.2510.04371 2026

  44. [44]

    Zheng, L

    L. Zheng, L. Yin, Z. Xie, J. Huang, C. Sun, C. H. Yu, S. Cao, C. Kober, Y. Sheng, J. E. Gonzalez, I. Stoica, and H. Zhang. SGLang: Efficient execution of structured language model programs. InAdvances in Neural Information Processing Systems, 2024

    2024

  45. [45]

    Zhong, B

    S. Zhong, B. Lu, Q. Chen, C. Liu, F. Yang, and M. Li. DualSpec: Ac- celerating deep research agents via dual-process action speculation,

  46. [46]

    URLhttp://arxiv.org/abs/2603.07416. A Full EQ1 Derivation Let a single agent turn have: •𝑇 dec: main generation wall time (post-prefill, includes CoT and tool-call decode) •𝑇 tool: tool execution wall time •𝑇 base =𝑇 dec +𝑇 tool: serial baseline cost UnderSpork, let𝛼be the gate acceptance rate and𝑡 overlap be the mean realized overlap on accepted turns. O...

    arXiv 2061