Pith. sign in

REVIEW 4 major objections 5 minor 90 references

Simulated tool execution feedback improves LLM agent tool calls consistently on two benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:40 UTC pith:J4H33JMX

load-bearing objection A practical simulation environment for test-time refinement of LLM tool calls; gains are consistent but the sim-to-real fidelity premise needs a direct test. the 4 major comments →

arxiv 2602.19218 v2 pith:J4H33JMX submitted 2026-02-22 cs.SE cs.MA

Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Calls

Zeyu Zhang , Guohao Li , Zhenchang Xing , Alexandros Apostolopoulos , Yu Lin Lee , Liang Zheng This is my paper
classification cs.SE cs.MA
keywords LLM agentstool usefunction callingtest-time scalingsimulation environmentstateful feedbacktool-call refinementGATS
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that LLM agents can improve their tool-calling ability without executing real tools, by refining calls inside a stateful simulation environment named Gecko. Gecko returns three kinds of feedback: argument validation, synthesized tool responses, and task-level completion checks, and the refinement loop built on it (GATS) improves several LLMs on two tool-use benchmarks. The concrete gains include GPT-4o rising from 76.93% to 84.62% overall accuracy on BFCLv3. If this holds, it gives a safe and cheaper alternative to test-time scaling that relies on real tool execution.

Core claim

The central claim is that grounding agent tool calls in a simulator is enough to drive meaningful correction: Gecko combines rule-based syntactic checks with an LLM semantic validator, generates schema-conforming and state-consistent responses through an LLM response generator, tracks an evolving task state, and uses a judge LLM to score task completion against a checklist. Feeding these signals back into the planning LLM in an iterative loop—GATS—consistently lifts performance across GPT-4o, GPT-5, Gemini-3.0-pro, and other models on BFCLv3 and τ2-bench, including new best reported overall accuracy on BFCLv3.

What carries the argument

The carrying mechanism is the three-channel feedback loop: (1) an argument validator that rejects malformed or semantically implausible tool calls; (2) an LLM-based response generator that synthesizes tool outputs conditioned on the current task state to preserve cross-call consistency; (3) a task-state estimator plus judge LLM that tracks cumulative effects and emits task-level feedback identifying whether objectives are complete or what remains. The session-based task state is what gives the feedback its stateful, multi-turn character.

Load-bearing premise

The load-bearing premise is that LLM-synthesized tool responses and task states are faithful enough to real execution that a plan refined against them transfers to real tools; the response generator is explicitly instructed to invent realistic values rather than report missing data.

What would settle it

Run GATS on a benchmark whose final state depends on a hidden real database (e.g., τ2-retail write-only tools), refine calls in Gecko, then execute the refined sequences against the real tools and compare final database states; if the simulated-refined calls frequently invent identifiers or data that do not exist in the real DB, the transfer claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • LLM agents can be improved at test time without touching real tools, removing API fees and side effects such as accidental posts or irreversible actions.
  • The refinement loop is model-agnostic: the same Gecko feedback improves GPT-4o, GPT-5, Gemini-3.0-pro, Deepseek-V3, Qwen-3-14B and others on both benchmarks.
  • Accuracy scales with the retry budget: on BFCL-Non-Live-Simple, GPT-4o goes from 92.25% with no refinement to 96.50% with three retries, with most gain in the first retry and rising latency and cost.
  • Gecko can be reused as a verifier for tool-call data synthesis and as an environment for reinforcement learning, turning existing SFT datasets into RL trajectories.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The simulator's read responses are partly invented (Appendix C instructs synthesizing plausible values when data is absent), so gains measured on BFCLv3 may overstate transfer to real database-backed tools; the paper's own hybrid suggestion—real read-only tools, simulated write tools—is the natural safeguard.
  • If the judge checklist is policy-aware, the same loop could enforce safety or scope constraints at test time, e.g., refusing to mark an action that violates policy as completed; the paper's judge prompt already rules against policy violations but does not develop this as a safety mechanism.
  • The observation that different planning LLMs converge to similar accuracy on some BFCLv3 subsets after GATS hints at a shared ceiling from the benchmark or the simulator, not from the model; the paper leaves this unexplained.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Gecko, a stateful simulation environment for LLM tool use, and GATS, a test-time scaling method that iteratively refines a planning LLM's tool calls using three kinds of Gecko feedback: argument validation, simulated tool responses, and task-level feedback from a judge LLM. The method is evaluated on BFCLv3 and τ2-bench across multiple planning LLMs (GPT-4o, GPT-5 variants, Gemini-3.0-pro, DeepSeek-V3, Qwen-3-14B, etc.), reporting consistent accuracy improvements over baselines, with ablations on Gecko's components, a comparison against Reflexion/Self-refine/Best-of-N, and a retry-budget scaling study. The paper argues that GATS improves tool calling without relying on real tool executions during refinement.

Significance. If the empirical claims hold, GATS is a practically important test-time scaling method: it converts a simulation environment into a reusable training/inference-time feedback mechanism, avoids the cost and side-effect risks of real tool execution, and is shown to yield consistent gains across many LLMs and two benchmarks. The paper also contains genuinely useful engineering details (argument validation rules, prompts for response generation and task-state tracking, OpenAPI conversion) and its ablations and scaling analysis are informative. However, the central premise — that simulated tool responses are faithful enough for refinements to transfer to real execution — is only indirectly validated, and the reported headline numbers lack error bars or significance tests. These are the main gaps between the current evidence and the paper's claims.

major comments (4)
  1. [§3.2, Appendix C, Table 2] The load-bearing assumption is that LLM-synthesized tool responses and task states are faithful enough to real execution that refining against them improves real performance. Appendix C's response-generator prompt (rule 3) explicitly instructs the generator to fabricate read/query results when data are absent, which is the opposite of realistic API behavior. Table 2 only measures the accuracy of task success/failure judgment when using real vs simulated tools (91.3% vs 96.7% true positive on BFCL-Multi-Turn-Base); it does not compare the final tool-call sequences produced by GATS when the feedback uses simulated vs real tool responses. I ask the authors to add a direct experiment: run GATS on a subset of BFCLv3 and τ2-bench with (a) all tool responses simulated, (b) read-only tools executed against real tools/databases (as they already do for τ2), and compare final benchmark scores. With
  2. [§6.2, Tables 3-4] No variance or significance information is reported for the main results. BFCL accuracy appears to be from a single pass, and τ2-bench reports only the average pass@1 over 4 runs without standard deviations or per-run values. Some gains, e.g., GPT-5-thinking on BFCLv3 overall (+4.14) and τ2-airline (+4.0), could be within run-to-run noise given the small subset sizes (τ2-airline has only 50 tasks). Please report confidence intervals or standard errors, ideally with multiple independent GATS runs (the τ2 protocol already has 4), and perform a paired test across tasks/models to support the claim of 'consistent' improvement.
  3. [§6.1 and §5] The abstract and introduction state that GATS avoids real tool execution during refinement, but §6.1 discloses that for τ2-bench, read-only tools (6 of 13 retail tools, 6 of 12 airline tools) are executed against real tools and real databases. This is a meaningful caveat: the method still requires read-only real API access for database-dependent domains, and the 'no real execution' claim is only about state-changing/write tools. Additionally, §5 says 'for multi-turn tasks, real tools are used at the end of each turn' — this is inconsistent with the τ2-bench description (read-only real tools are used when the planning LLM calls them) and is not clearly explained for BFCLv3 multi-turn. Please clarify the exact setup and qualify the abstract accordingly.
  4. [Table 5 / Fig. 5] The comparison with existing test-time methods (Reflexion, Self-refine, Best-of-N) and the retry scaling study are informative, but they do not include statistical uncertainty. For example, GATS outperforms Reflexion by 5% on τ2-airline (65.0% vs 60.0%) with a single 50-task benchmark; without variance, it is unclear whether this ordering is stable. Reporting confidence intervals for Table 5 would also strengthen the cost-performance claims.
minor comments (5)
  1. [Abstract / §6.1] The phrase 'Gecko has automatically synthesized 8,578 and 13 tools' is ambiguous: does 13 refer only to τ2-retail, or to both retail and airline subsets? τ2-airline also has 12 APIs; please clarify how many tools were synthesized for each benchmark.
  2. [§5 Limitations] Typo: 'get_temprature' should be 'get_temperature'. Also in §6.1, 'significantly in accurate' should be 'significantly inaccurate'.
  3. [Fig. 2] The figure contains 'reprot.pdf' and 'temp' directory diagrams; the spelling 'reprot.pdf' appears to be a typo. Also, the red/rightmost panel says 'Pass' but the earlier attempts are labeled 'Failed' — the color scheme is not explained in the caption, which may confuse readers.
  4. [Table 3 caption] The caption says 'We select eight most important metrics from BFCL website', but the table appears to contain nine columns of metrics (Overall plus four non-live, three live, and one multi-turn). Please verify the count and either list the metrics explicitly or fix the caption.
  5. [§6.3] The token usage column in Table 5 is labeled 'Tokens (k)' but the numbers (e.g., 238.2) suggest thousands of tokens. If they are in thousands, the label should be 'Tokens (k)' and the values should be consistent with that; please clarify. Also specify how tokens from simulated tool calls are counted.

Circularity Check

0 steps flagged

No circularity: headline improvements are graded by external benchmarks (BFCLv3 exact-match, τ2-bench final database state); simulated feedback is an input to refinement, not to the score.

full rationale

The derivation chain is: a planning LLM proposes tool calls, Gecko returns validation, simulated responses, and task feedback, GATS iterates on that feedback, and the final tool-call sequence is then scored externally. The paper states: 'For single-turn tasks, a prediction is counted as correct only if the tool calls produced by planning LLM exactly match the reference solution' and 'for τ2-bench ... a run is successful only if the agent responses all required information and the final database matches the annotation.' These external graders do not use Gecko's simulated responses, task states, or judge outputs as the score, so the reported accuracy gains cannot reduce to Gecko's own feedback by construction. The only free budget, max_retry_times, is hand-chosen and is swept in Fig. 5, not fitted to the test sets. The paper's own limitation that 'simulation outputs ... can be very different from real-world results' for external-database tools is a fidelity/validity threat, not a circularity, because the benchmark scores are computed independently of the simulation. Self-citations such as CAMEL [13] are used only as an implementation framework and are not load-bearing for the central claim. No uniqueness theorem, ansatz, or fitted parameter is imported from prior work to force the result. Thus, the central claim is self-contained with respect to the external benchmark evaluation.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

No new physical entities (particles, forces, dimensions) are introduced. 'Task state' and 'session isolation' are software constructs, not new entities with independent falsifiable handles. The free-parameter count is minimal: only the retry budget. The main assumptions are about simulation fidelity and the reliability of LLM-based judging, both domain assumptions the paper partially tests.

free parameters (1)
  • max_retry_times = 3
    Maximum number of GATS refinement attempts, chosen by hand. Fig. 5 shows accuracy improves with more retries (92.25% to 96.50% at 0 to 3 retries) while latency and cost grow, so the budget is a tunable knob, not fitted to a test set.
axioms (5)
  • domain assumption LLM-synthesized tool responses are sufficiently faithful to real tools that refinement transfers
    The response generator prompt (Appendix C) instructs open-world reads to 'MUST synthesize realistic, schema-compliant values instead of returning not-found or error responses', meaning simulated data can be invented. The central claim assumes refinement against such responses improves real benchmark scores.
  • domain assumption The judge LLM's checklist-based completion assessment approximates true task completion
    Section 3.4 relies on a judge LLM to generate a checklist and verify it against simulated state. If the judge is systematically lenient or wrong, GATS could stop early or chase false objectives.
  • domain assumption Task state estimator tracks cumulative effects without drift
    Section 3.3 updates state from tool responses; errors in state propagate into response generation and judging.
  • domain assumption Benchmark exact-match and final-database scoring are valid external measures
    The paper's only non-circular evidence is improvement on BFCLv3 and τ2-bench external scoring, so the validity of these benchmarks is load-bearing.
  • domain assumption Planning LLM can act on textual feedback to fix tool calls
    GATS assumes 'feedback from Gecko is sent to the planning LLM to refine the tool calls' (Section 4) is actionable; the consistent gains suggest this holds for the tested models.

pith-pipeline@v1.3.0-alltime-deepseek · 20863 in / 11236 out tokens · 97493 ms · 2026-08-02T21:40:28.620136+00:00 · methodology

0 comments
read the original abstract

The ability to use tools is fundamental for large language model (LLM) agents. Given a task, existing systems use LLMs to plan and generate tool calls, which are executed by real-world tools to complete the task. However, tool calls are prone to errors because they are generated primarily from the intrinsic capabilities of LLMs. Moreover, while it is useful to let LLMs iteratively refine the tool-call sequence using execution results from real tools, this process can be expensive and may cause unsafe side effects. To improve LLM tool calls and address issues caused by using real tools for refinement, we introduce Gecko, a stateful simulation environment that provides informative feedback for refining LLM tool calls before real execution. Specifically, Gecko combines rules and LLMs to check the validity of tool names and arguments, synthesize schema-conforming and state-consistent responses, and judge task completion against the user objective. These three types of feedback allow LLMs to refine their tool calls in simulation, forming a simple yet effective test-time scaling method named GATS. On BFCLv3 and $\tau^2$-bench, GATS consistently improves the performance of various LLMs.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

90 extracted references · 22 linked inside Pith

  1. [1]

    Claude opus 4 / opus 4.1

    Anthropic. Claude opus 4 / opus 4.1. An- thropic model page / announcement, 2025. URL https://www.anthropic.com/claude /opus

    2025

  2. [2]

    τ2-bench: Evaluating conver- sational agents in a dual-control environment,

    Barres, V ., Dong, H., Ray, S., Si, X., and Narasimhan, K. τ2-bench: Evaluating conver- sational agents in a dual-control environment,

  3. [3]

    Acebench: Who wins the match point in tool usage?, 2025

    Chen, C., Hao, X., Liu, W., Huang, X., Zeng, X., Yu, S., Li, D., Wang, S., Gan, W., Huang, Y., Liu, W., Wang, X., Lian, D., Yin, B., Wang, Y., and Liu, W. Acebench: Who wins the match point in tool usage?, 2025. URL https://arxiv.or g/abs/2501.12851

    arXiv 2025

  4. [4]

    Learning evolving tools for large language models, 2025

    Chen, G., Zhang, Z., Cong, X., Guo, F., Wu, Y., Lin, Y., Feng, W., and Wang, Y. Learning evolving tools for large language models, 2025. URLhttps://arxiv.org/abs/2410.06617

    Pith/arXiv arXiv 2025

  5. [5]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., and Others. Deepseek-v3 technical report, 2025. URLhttps...

    Pith/arXiv arXiv 2025

  6. [6]

    Anytool: Self- reflective, hierarchical agents for large-scale api calls, 2024

    Du, Y., Wei, F., and Zhang, H. Anytool: Self- reflective, hierarchical agents for large-scale api calls, 2024. URL https://arxiv.org/abs/24 02.04253

    2024

  7. [7]

    Gemini 3 pro model card

    Google DeepMind. Gemini 3 pro model card. Technical report, Google DeepMind, December

  8. [8]

    Stabletool- bench: Towards stable large-scale benchmark- ing on tool learning of large language models,

    Guo, Z., Cheng, S., Wang, H., Liang, S., Qin, Y., Li, P ., Liu, Z., Sun, M., and Liu, Y. Stabletool- bench: Towards stable large-scale benchmark- ing on tool learning of large language models,

  9. [9]

    URL https://storage.googleapis.c om/deepmind-media/Model-Cards/Gemini-3 -Pro-Model-Card.pdf

  10. [10]

    T1: Tool- integrated self-verification for test-time com- pute scaling in small language models, 2025

    Kang, M., Jeong, J., and Cho, J. T1: Tool- integrated self-verification for test-time com- pute scaling in small language models, 2025. URLhttps://arxiv.org/abs/2504.04718

    Pith/arXiv arXiv 2025

  11. [11]

    URL https://arxiv.org/abs/2403.0 7714

  12. [12]

    Z., and Sun, L

    Huang, Y., Shi, J., Li, Y., Fan, C., Wu, S., Zhang, Q., Liu, Y., Zhou, P ., Wan, Y., Gong, N. Z., and Sun, L. Metatool benchmark for large language models: Deciding whether to use tools and which to use, 2024. URL https: //arxiv.org/abs/2310.03128

    Pith/arXiv arXiv 2024

  13. [13]

    Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., and Ghanem, B. Camel: Com- municative agents for "mind" exploration of large language model society, 2023. URLhttps: //arxiv.org/abs/2303.17760

    Pith/arXiv arXiv 2023

  14. [14]

    Longfunceval: Measuring the effectiveness of long context models for func- 12 tion calling, 2025

    Kate, K., Pedapati, T., Basu, K., Rizk, Y., Chen- thamarakshan, V ., Chaudhury, S., Agarwal, M., and Abdelaziz, I. Longfunceval: Measuring the effectiveness of long context models for func- 12 tion calling, 2025. URL https://arxiv.org/ abs/2505.10570

    Pith/arXiv arXiv 2025

  15. [15]

    Start: Self-taught reasoner with tools, 2025

    Li, C., Xue, M., Zhang, Z., Yang, J., Zhang, B., Wang, X., Yu, B., Hui, B., Lin, J., and Liu, D. Start: Self-taught reasoner with tools, 2025. URLhttps://arxiv.org/abs/2503.04625

    Pith/arXiv arXiv 2025

  16. [16]

    Agentbench: Evaluating llms as agents, 2023

    Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., Zhang, S., Deng, X., Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. Agentbench: Evaluating llms as agents, 2023. URL https://arxiv.org/ab s/2308.03688

    Pith/arXiv arXiv 2023

  17. [17]

    Li, M. Q. and Fung, B. C. M. Security concerns for large language models: A survey, 2025. URL https://arxiv.org/abs/2505.18889

    Pith/arXiv arXiv 2025

  18. [18]

    Toolace: Winning the points of llm function calling, 2024

    Liu, W., Huang, X., Zeng, X., Hao, X., Yu, S., Li, D., Wang, S., Gan, W., Liu, Z., Yu, Y., Wang, Z., Wang, Y., Ning, W., Hou, Y., Wang, B., Wu, C., Wang, X., Liu, Y., Wang, Y., Tang, D., Tu, D., Shang, L., Jiang, X., Tang, R., Lian, D., Liu, Q., and Chen, E. Toolace: Winning the points of llm function calling, 2024. URL https://arxi v.org/abs/2409.00920

    Pith/arXiv arXiv 2024

  19. [19]

    P ., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P

    Madaan, A., Tandon, N., Gupta, P ., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P ., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P . Self-refine: Iterative refine- ment with self-feedback. In Oh, A., Nau- mann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advan...

    2023

  20. [20]

    C., Wang, H., Heinecke, S., and Xiong, C

    Liu, Z., Hoang, T., Zhang, J., Zhu, M., Lan, T., Kokane, S., Tan, J., Yao, W., Liu, Z., Feng, Y., Murthy, R., Yang, L., Savarese, S., Niebles, J. C., Wang, H., Heinecke, S., and Xiong, C. Api- gen: Automated pipeline for generating verifi- able and diverse function-calling datasets, 2024. URLhttps://arxiv.org/abs/2406.18518

    Pith/arXiv arXiv 2024

  21. [21]

    Toolsandbox: A stateful, conversational, interactive evaluation bench- mark for llm tool use capabilities, 2025

    Lu, J., Holleis, T., Zhang, Y., Aumayer, B., Nan, F., Bai, F., Ma, S., Ma, S., Li, M., Yin, G., Wang, Z., and Pang, R. Toolsandbox: A stateful, conversational, interactive evaluation bench- mark for llm tool use capabilities, 2025. URL https://arxiv.org/abs/2408.04682

    Pith/arXiv arXiv 2025

  22. [22]

    G., Mao, H., Yan, F., Ji, C

    Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V ., Stoica, I., and Gonzalez, J. E. The berke- ley function calling leaderboard (BFCL): From tool use to agentic evaluation of large lan- guage models. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=2GmDdh BdDk

    2025

  23. [23]

    Introducing gpt-4.1 in the api

    OpenAI. Introducing gpt-4.1 in the api. OpenAI blog / API docs, 2025. URL https://openai .com/index/gpt-4-1/

    2025

  24. [24]

    Gpt-5 system card

    OpenAI. Gpt-5 system card. Technical report, OpenAI, 2025. URL https://cdn.openai.com /gpt-5-system-card.pdf

    2025

  25. [25]

    Toolllm: Facilitating large language models to master 16000+ real- world apis, 2023

    Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Hong, L., Tian, R., Xie, R., Zhou, J., Gerstein, M., Li, D., Liu, Z., and Sun, M. Toolllm: Facilitating large language models to master 16000+ real- world apis, 2023. URL https://arxiv.org/ab s/2307.16789

    Pith/arXiv arXiv 2023

  26. [26]

    C., Heinecke, S., Yao, W., Wang, H., Savarese, S., and Xiong, C

    Prabhakar, A., Liu, Z., Zhu, M., Zhang, J., Awal- gaonkar, T., Wang, S., Liu, Z., Chen, H., Hoang, T., Niebles, J. C., Heinecke, S., Yao, W., Wang, H., Savarese, S., and Xiong, C. Apigen-mt: Agentic pipeline for multi-turn data genera- tion via simulated agent-human interplay, 2025. URLhttps://arxiv.org/abs/2504.03601

    Pith/arXiv arXiv 2025

  27. [27]

    Making language models better tool learners with execution feedback, 2024

    Qiao, S., Gui, H., Lv, C., Jia, Q., Chen, H., and Zhang, N. Making language models better tool learners with execution feedback, 2024. URL https://arxiv.org/abs/2305.13068

    Pith/arXiv arXiv 2024

  28. [28]

    Reflexion: language agents with verbal reinforcement learning

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: language agents with verbal reinforcement learning. In Oh, A., Naumann, T., Glober- son, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 8634–

  29. [29]

    Learning to use tools via coopera- tive and interactive agents, 2024

    Shi, Z., Gao, S., Chen, X., Feng, Y., Yan, L., 13 Shi, H., Yin, D., Ren, P ., Verberne, S., and Ren, Z. Learning to use tools via coopera- tive and interactive agents, 2024. URL https: //arxiv.org/abs/2403.03031

    Pith/arXiv arXiv 2024

  30. [30]

    Tool learn- ing in the wild: Empowering language models as automatic tool agents

    Shi, Z., Gao, S., Yan, L., Feng, Y., Chen, X., Chen, Z., Yin, D., Verberne, S., and Ren, Z. Tool learn- ing in the wild: Empowering language models as automatic tool agents. In Proceedings of the ACM on Web Conference 2025, pp. 2222–2237, 2025

    2025

  31. [31]

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases, 2023

    Tang, Q., Deng, Z., Lin, H., Han, X., Liang, Q., Cao, B., and Sun, L. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases, 2023. URL https://arxiv.or g/abs/2306.05301

    Pith/arXiv arXiv 2023

  32. [32]

    Kimi k2: Open agentic intelligence, 2025

    Team, K., Bai, Y., Bao, Y., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y., Chen, Y., Chen, Y., Chen, Z., Cui, J., Ding, H., Dong, M., Du, A., Du, C., Du, D., Du, Y., Fan, Y., Feng, Y., Fu, K., Gao, B., Gao, H., Gao, P ., Gao, T., Gu, X., Guan, L., Guo, H., Guo, J., Hu, H., Hao, X., He, T., He, W., He, W., Hong, C., Hu, Y., and Others. Kimi k2: Open age...

    Pith/arXiv arXiv 2025

  33. [33]

    Agentic reasoning and tool integration for llms via reinforcement learning, 2025

    Singh, J., Magazine, R., Pandya, Y., and Nambi, A. Agentic reasoning and tool integration for llms via reinforcement learning, 2025. URL ht tps://arxiv.org/abs/2505.01441

    Pith/arXiv arXiv 2025

  34. [34]

    Workbench: a benchmark dataset for agents in a realistic workplace setting, 2024

    Styles, O., Miller, S., Cerda-Mardini, P ., Guha, T., Sanchez, V ., and Vidgen, B. Workbench: a benchmark dataset for agents in a realistic workplace setting, 2024. URL https://arxiv. org/abs/2405.00823

    Pith/arXiv arXiv 2024

  35. [35]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024

    Yao, S., Shinn, N., Razavi, P ., and Narasimhan, K. τ-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https://arxiv.org/abs/2406.12045

    Pith/arXiv arXiv 2024

  36. [36]

    H., Wang, Z., Lee, K

    Zhou, H., Chen, Y., Guo, S., Yan, X., Lee, K. H., Wang, Z., Lee, K. Y., Zhang, G., Shao, K., Yang, L., and Wang, J. Memento: Fine-tuning llm agents without fine-tuning llms, 2025. URL https://arxiv.org/abs/2508.16153. 14 A Design Principles of Gecko Gecko is a simulated tool execution environment built on modern web-service principles and accessible over ...

    Pith/arXiv arXiv 2025

  37. [37]

    watt-tool-70B model card

    watt-ai. watt-tool-70B model card. Hugging Face model hub, 2025. URL https://huggin gface.co/watt-ai/watt-tool-70B

    2025

  38. [38]

    Qwen3 technical report,

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P ., Wang, P ., Zhu,...

  39. [39]

    URL https://arxiv.org/abs/2505.0 9388

  40. [42]

    − Ignore arguments not present in the schema (do not treat them as errors)

    **Scope** − Only validate arguments defined in the provided schemas. − Ignore arguments not present in the schema (do not treat them as errors). − Type validation has already been handled elsewhere. Just skip type checking

  41. [43]

    full−time, part−time

    **Semantic Checks** − Validate according to the parameter description, examples, enums, or format ,→requirements. − If examples are provided (e.g. "full−time, part−time"), treat them as semantic categories. ,→Any value in the same category (e.g. "internship", "contract") is valid. − If the description specifies a format (e.g. ‘YYYY−MM−DD‘), enforce that e...

  42. [44]

    \<if false, list each invalid argument and reason>

    **Error Messages** − Concise, precise, and human−readable. − Do not include or suggest correct values. − Only state which argument is invalid and why. **Output Format**: ‘‘‘ valid=\<true|false> error\_message="\<if false, list each invalid argument and reason>" ‘‘‘ 16 C Response Generator Implementation Details The response generator synthesizes a tool re...

  43. [45]

    Always match the schema exactly (structure, names, types, formats, required ,→fields)

    Schema first. Always match the schema exactly (structure, names, types, formats, required ,→fields)

  44. [46]

    Do not contradict any provided state or any prior successful ,→responses in this session

    Entity−level consistency. Do not contradict any provided state or any prior successful ,→responses in this session

  45. [47]

    Open−world reads. For read/query/search operations, if requested entities/data are ,→absent in the provided state, you MUST synthesize realistic, schema−compliant values ,→instead of returning not−found or error responses

  46. [48]

    For create/modify/delete operations, produce a success result ,→consistent with the schema unless it would contradict previously returned state; do not ,→invent conflicts

    Writes remain consistent. For create/modify/delete operations, produce a success result ,→consistent with the schema unless it would contradict previously returned state; do not ,→invent conflicts

  47. [49]

    Do not invent constraints beyond the tool definition and the provided state

    No extra rules. Do not invent constraints beyond the tool definition and the provided state. Realism & uniqueness guidelines (domain−agnostic): − Deterministic diversity: derive identifier−like fields using stable transforms of input ,→arguments (e.g., incorporating parts of arguments or their hashes) so that different ,→arguments yield different values w...

  48. [50]

    **Domain Data (Databases)**: The actual data that tools operate on * FileSystem toolkit: files, directories structure * Airline toolkit: users, flights, tickets, bookings * Message toolkit: messages, inbox items * These are stored at appropriate top−level or domain−specific keys

  49. [51]

    **Runtime Variables**: Execution context and session state * Store these DIRECTLY under ’runtime\_state’ (flat structure) * Examples: current\_working\_directory, current\_user, is\_logged\_in, session\_token * IMPORTANT: Read toolkit descriptions carefully for initialization requirements CRITICAL: * NO ’runtime\_state.toolkits’ structure − keep runtime\_...

  50. [52]

    Preserve all existing structures in the backgound information

  51. [53]

    Add runtime variables DIRECTLY under ’runtime\_state’ (flat structure)

  52. [54]

    Add domain data at appropriate keys (not in runtime\_state)

  53. [55]

    NEVER create ’runtime\_state.toolkits’ or any similar nesting

  54. [56]

    Avoid duplicating the same concept

  55. [57]

    18 The main prompt for the task state updater is as below

    Output valid JSON only Background information: {background_information} Toolkits summary: {json.dumps(toolkits\_summary, indent=2)} Return the UPDATED config JSON with the necessary domain data and runtime state. 18 The main prompt for the task state updater is as below. You are an expert at tracking the execution state of a task. Update the system state ...

  56. [58]

    In synthesis mode Store ALL synthesized data from read operations as ground truth ,→state

    **State Tracking Principles** − Update the system state to reflect ALL persistent state changes caused by tool calls − Operations that create, modify, or delete resources MUST update the corresponding ,→structures − {"In synthesis mode Store ALL synthesized data from read operations as ground truth ,→state" if synthesis_mode else "Operations that just que...

  57. [59]

    runtime_state

    **System state Organization** − When tool operations modify existing structures, update them directly (e.g., adding a ,→new directory should add it to the directory tree) − For execution context that doesn’t fit existing domain structures, use the root−level ,→"runtime_state" − The "runtime_state" section is ONLY for execution context and ephemeral teleme...

  58. [60]

    **Value Formatting** − When recording locations, positions, or identifiers, use complete, unambiguous values − Avoid partial or relative references that could be misinterpreted − Preserve the format conventions used in the original system state

  59. [61]

    19 E Task Feedback Generation Implementation Details Task feedback generation has two steps: checklist generation and judgement generation

    **What Changes to Track** − Resource creation/deletion/modification (files, directories, database records, etc.) − State transitions (status changes, position changes, mode switches) − Context updates (current location, active selections, session data) − DO NOT track query results, search results, temporary computations, or read−only ,→operation outputs O...

  60. [62]

    Restate user intent and constraints neutrally

  61. [63]

    Extract policy constraints relevant to this task

  62. [64]

    Output a small set of verifiable checks covering both user request and policy

  63. [65]

    If current message has no actionable request (thanks/closing/small talk), return []

  64. [66]

    Be objective; do not add your own goals

  65. [67]

    Describe verifiable outcomes/evidence, not procedures; do not require specific ,→tools/APIs/operations

  66. [68]

    Prefer 4−8 items; merge closely related policy checks when possible

  67. [69]

    State−changing actions affected only the user−requested scope, and nothing else

    For state−changing tasks (cancel/modify/delete/book/refund/transfer), include: "State−changing actions affected only the user−requested scope, and nothing else."

  68. [70]

    Agent asked the user for the missing eligibility detail

    Do not infer eligibility from assumptions. If key eligibility fact is unknown, require ,→clarification, e.g.: "Agent asked the user for the missing eligibility detail."

  69. [71]

    Only require facts/fields explicitly required by user request or policy

  70. [72]

    If multiple entity IDs appear in history, prefer one grouped item listing all required IDs

  71. [73]

    results obtained / previous results / three values

    If task says "results obtained / previous results / three values", treat this as output from ,→the most recent relevant prior step. Output format: − Return JSON only. − Return a JSON array of objects. − Each object must contain "description": string. − Optional: "kind" (e.g., user_intent, policy_gate, scope_guard, state_check, ,→clarify_if_needed). 20 E.2...

  72. [74]

    Current system state (primary source of truth)

  73. [75]

    Tool calls with arguments and results

  74. [76]

    Conversation history (multi−turn context){history_text}

  75. [77]

    completed

    Tool definitions{tool_defs_text} Policy−critical rule: − Judge correctness against policy, not checklist wording alone. − If a checklist item asks to deny an action, but policy allows a compliant workaround, denial ,→is not "completed". Evaluate each checklist item and assign one status: − completed: requirement satisfied with evidence − in_progress: reas...

  76. [78]

    Recompute numeric/time conclusions independently; do not trust agent math

  77. [79]

    For eligibility/claims, require tool−based verification when tools exist

  78. [80]

    Irreversible actions need explicit user consent before execution

  79. [81]

    Asking user for data that tools can retrieve (when parameters are already available) is ,→failed

  80. [82]

    If workaround exists in policy but agent rejects directly, mark failed (not rejected)

Showing first 80 references.