LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback

Caiming Xiong; Chien-Sheng Wu; Huan Wang; Jake Grigsby; Jianguo Zhang; Juan Carlos Niebles; Kung-Hsiang Huang; Michael S Ryoo; Ming Zhu; Shelby Heinecke

REVIEW 3 major objections 6 minor 1 cited by

LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM agent can improve its own tool-use performance by training on trajectories it generates through online exploration, with pass-rate gains up to 49.3% in the paper's experiments.

desk verdict Useful data-generation framework, but the headline claim that exploration plus feedback drives the gains is not actually tested against the obvious gold-solution-path control. read the letter →

arxiv 2506.02298 v1 pith:E7BQHOV3 submitted 2025-06-02 cs.CL cs.AIcs.LG

Thai Hoang , Kung-Hsiang Huang , Shirley Kokane , Jianguo Zhang , Zuxin Liu , Ming Zhu , Jake Grigsby , Tian Lan

show 7 more authors

Michael S Ryoo Chien-Sheng Wu Shelby Heinecke Huan Wang Silvio Savarese Caiming Xiong Juan Carlos Niebles

This is my paper

classification cs.CLcs.AIcs.LG

keywords largeactionmodelsdatagenerationonlineexplorationtrajectoryfeedbacktooluseself-improvementagenticAIprogrammaticevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LAM SIMULATOR, a framework for generating training data for large action models (LAMs) — AI agents that plan, call tools, and respond to feedback. The central idea is to let an LLM agent autonomously explore tasks in an environment, collect the resulting action trajectories, filter them against programmatically computed ground-truth answers, and fine-tune the same model on the surviving trajectories. The paper reports that this self-improvement loop yields pass-rate gains of up to 49.3% on CRMArena and 19.3% on ToolBench over baseline models, and that the generated data beats data produced by ToolLLM. If these results hold, high-quality agent training data can be produced with minimal human labeling, which would speed up the development of tool-using AI agents.

What carries the argument

The key machinery is the LAM SIMULATOR pipeline, which combines three components: a template-filling query generator that creates novel tasks with pre-computed ground-truth answers, an interactive environment whose action handler validates every tool call and returns error messages for malformed or hallucinated calls, and a trajectory filter that keeps only trajectories whose final answer matches the ground truth and that either contain no errors or recover from errors in the next step. This machinery allows the agent to explore with a tool set that can differ from the solution path used to compute the ground truth, receive real-time feedback, and generate training trajectories that include error-recovery behavior. The framework's programmatic evaluation avoids LLM-based judgment, which the paper argues makes the resulting training data more reliable than previous approaches.

What would settle it

Run the fine-tuning procedure with exploration tasks and tools that are provably disjoint from the evaluation tasks (for example, generate exploration data from the two held-out CRMArena tasks TCU and TII and evaluate on the other four, or swap the tool sets between exploration and evaluation on ToolBench). If the pass-rate gains shrink or disappear on such disjoint test sets, the claimed improvements would be shown to come from in-distribution fine-tuning rather than from the framework's exploration-and-feedback mechanism.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that an LLM agent can improve its own tool-use performance by training on trajectories it collected through online exploration, provided the exploration is supervised by a programmatic action handler and the trajectories are filtered by matching their final answers to ground truths. The authors demonstrate this by fine-tuning five models on self-generated data from LAM SIMULATOR and showing consistent pass-rate improvements over each baseline: for example, gpt-4o rises from 47.4% to 51.5% on ToolBench, and gpt-4o-mini rises from 18.5% to 67.8% on CRMArena. They further claim that the gains extend to out-of-domain evaluations, including unseen tools and categories on ToolBench and two held-out CRM tasks, and that LAM SIMULATOR's fully programmatic evaluation produces higher-quality training data than ToolLLM's LLM-based generation and evaluation.

Load-bearing premise

The load-bearing premise is that the evaluation test sets are genuinely independent of the exploration-generated training data, so that the reported pass-rate gains reflect improved agent ability rather than in-distribution fine-tuning on near-duplicate tasks or tools.

Editorial extensions

If this is right

Models fine-tuned on their own exploration trajectories can improve tool-use pass rates without new human-labeled data, reducing the cost of agent training.
Programmatic evaluation (action handler plus ground-truth filtering) produces training data that outperforms LLM-based generation and evaluation pipelines like ToolLLM in direct comparison on CRMArena.
The measured gains on unseen tools, unseen categories, and held-out CRM tasks suggest that self-generated trajectories teach generalizable tool-use behavior, not just answers to the exploration tasks.
Lower-performing models benefit disproportionately more from self-generated data (mixtral-8x7b-inst rises from 11.7% to 31.0% on ToolBench; gpt-4o-mini rises from 18.5% to 67.8% on CRMArena), indicating the approach can bootstrap weaker agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the independence assumption holds, the same exploration-and-filter loop could be applied to other agent benchmarks and real-world tool environments, but its dependence on hand-designed query templates and solution paths may limit scaling to open-ended tasks — a limitation the paper itself acknowledges.
The ablation shows that monitoring actions and filtering trajectories are each essential; a natural next experiment would ablate the error-recovery criterion specifically to see whether the correction trajectories are the main driver of the gains.
The paper compares against ToolLLM only on CRMArena; a stronger test would generate data from both methods on the same task pool and evaluate on multiple benchmarks to separate data-quality from task-distribution effects.
The claim of 'minimal human input' applies to the exploration phase; the initial creation of query templates, solution paths, and tool cleanup still requires substantial human design, so the framework's universality would be tested by automating template creation or transferring templates across domains.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Briefly: this is a legitimately useful integration—multi-turn query generation with programmatic ground-truth answers, an interactive tool environment, and trajectory filtering—and the authors report large pass-rate gains on ToolBench and CRMArena after fine-tuning on self-generated data. The ToolLLM comparisons are a fair touch, and the ablation showing that removing the action handler or trajectory filter hurts performance is real evidence those components matter.

The soft spot is the one the stress-test note flags, and it lands on reading the paper. Section 3.4 argues that self-explored trajectories teach error recovery and generalize better, but the paper never runs the natural control: fine-tuning on the gold solution paths defined in Section 3.2. Those paths are programmatic, require no exploration, and exist for all 800 query instances. Without that control, the reported gains (e.g., mixtral from 11.7% to 31.0% on ToolBench; gpt-4o-mini from 18.5% to 67.8% on CRMArena) could just be the effect of adding more in-distribution SFT data. The ToolLLM baselines don't resolve this because they use a different trajectory-generation pipeline, not the gold paths.

There's also a distribution-overlap issue: ToolBench templates are 'based on instances from the ToolBench training dataset,' and CRMArena explores four of the six tasks it evaluates on. The out-of-domain sets (G1_tool, G1_cat, TCU, TII) help, but the in-domain gains are suspect, and there are no confidence intervals or significance tests. The 'minimal human input' claim is overstated: designing 36 high-level tasks, dozens of query templates, placeholder descriptions, and solution paths for 800 instances is substantial human engineering. That doesn't sink the paper, but it should be described honestly.

Bottom line: for people working on data generation for tool-using agents, this is a useful reference point, and the programmatic-evaluation piece is a genuine improvement over LLM-as-judge pipelines. But the central mechanistic claim—that online exploration and trajectory feedback, not just additional SFT data, drive the gains—is untested. A serious referee should ask for the gold-solution-path control, clearer in-domain/out-of-domain separation, and code/data release. I'd send it to review; it's a solid empirical contribution that needs one more experiment to nail the interpretation.

Referee Report

3 major / 6 minor

Summary. The paper introduces LAM Simulator, a framework for generating agentic training data through online exploration. Queries are produced by filling manually designed templates with LLM-sampled parameters; ground-truth answers are computed programmatically via predefined tool sequences; agents explore the resulting tasks in an interactive environment with an action handler and trajectory filtering; the filtered trajectories are then used to fine-tune the base model. Experiments on ToolBench and CRMArena report pass-rate gains for gpt-4o, gpt-4o-mini, xlam-8x7b, xlam-7b, and mixtral-8x7b, with the largest gain being 49.3 percentage points for gpt-4o-mini on CRMArena. The paper also compares against ToolLLM variants on CRMArena and includes ablations of the action-handler and trajectory-filtering components. Section 8 appropriately acknowledges that the current implementation focuses on predefined tasks and tools, but several load-bearing empirical claims in Sections 4-6 need additional controls before the main conclusions are supported.

Significance. If the central empirical claim is robust, the framework would be a useful contribution to agent data generation: it replaces LLM-based evaluation with programmatic answer verification, assembles a large cleaned tool library, and demonstrates across multiple base models that self-explored trajectories can improve fine-tuned agent performance. The paper's strengths include its fully programmatic trajectory filtering, dynamic answer computation, and the breadth of models tested in Section 6. However, the significance is currently conditional because the paper does not isolate the contribution of online exploration and trajectory feedback from simply adding extra supervised trajectories, and because the training data generation overlaps with the evaluation benchmarks in ways that may inflate the reported gains. With additional controls, the result would be valuable for the agent-data-generation community; as written, the empirical contribution is not yet causally identified.

major comments (3)

[§3.2, §3.4, §5.3, Tables 2 and 3] The paper defines gold solution paths in §3.2 (and Appendix A.1) and argues in §3.4 that self-explored trajectories teach error recovery and expose the model to a wider range of scenarios, but it never runs the natural control of fine-tuning on the gold solution paths. Since all reported gains are relative to base models that receive no additional SFT data, the improvements in Tables 2 and 3 could come from adding 500 valid in-distribution trajectories rather than from online exploration, error recovery, or trajectory feedback. I request a control where each base model is fine-tuned on the same number of gold solution-path trajectories (and ideally on randomly sampled successful non-explored trajectories) and evaluated on the same test sets; without this control, the central claim that the exploration and feedback mechanisms drive the improvement is not supported.
[§4.1, §4.2, §5.1, Tables 2 and 3] The evaluation benchmarks overlap with the data-generation setup in ways that may inflate the reported gains. For ToolBench, §4.1 says the 30 query templates are 'based on instances from the ToolBench training dataset', and the evaluation in §5.1 uses ToolBench test sets whose tools and categories may overlap with the exploration tasks; the assertion that the overlap is 'exceedingly low' is not supported by any measurement. For CRMArena, §4.2 conducts exploration on NCR, HTU, MTA, and BRI while §5.1 evaluates on all six tasks, so the four in-domain tasks share scenarios and tools with training; the large in-domain gains (e.g., gpt-4o on CRMArena ALL from 62.3% to 86.4%) are therefore at least partly attributable to in-distribution fine-tuning. The paper should quantify the overlap in instructions, tools, and data distributions between the generated queries and each test set, and should evaluate on tasks or environments with no template or tool overlap to support the generalization claims. The out-of-domain TCU and TII results are encouraging but cover only two of six tasks.
[Tables 2 and 3, §5.1, §5.3] No confidence intervals or significance tests are reported anywhere in the results. Evaluation temperature is set to 0.0, but fine-tuning initialization and trajectory sampling are stochastic, so a single run cannot establish small gains. For example, xlam-7b on ToolBench ALL moves from 39.2% to 41.3% on 600 test instances, which is within roughly two standard errors of the baseline, and gpt-4o on ToolBench ALL moves from 47.4% to 51.5%, which is a marginal difference. The paper should report variance over multiple fine-tuning seeds or bootstrap confidence intervals, and should identify which of the reported gains are statistically distinguishable from the baselines.

minor comments (6)

[Abstract and §6.2] The '49.3% improvement' and 'over 31% higher' figures are percentage-point differences, not relative improvements; the paper should state this explicitly to avoid overstating the gains (18.5% to 67.8% is a 49.3-point increase but a roughly 267% relative increase).
[Table 3] The column header 'MTI' appears to be a typo for 'MTA' (Monthly Trend Analysis), which is the task named in §4.2.
[Abstract, §1, §3.2, §3.5] The claim of 'minimal human input' conflicts with the detailed, per-template human effort described in §3.2 and Appendix A.1: humans must write query templates, placeholder metadata, and solution-path tool sequences for every template. The authors should either quantify the one-time human effort or temper the claim to 'minimal per-instance human annotation'.
[§3.5 and §8] Section 3.5 says generalizing to a new environment requires only query templates and tool-call mappings, while Section 8 acknowledges that the implementation is limited to predefined tasks and tools; these statements should be reconciled, and the limitations section should be reflected in the main text's generalizability claims.
[Appendix A.1] The appendix contains language errors such as 'An example is showned at 6' and the sentence 'The arguments would be searched among 1) placeholder values and 2) objects generated during execution' is grammatically incomplete.
[Figure 1 caption] The caption contains a typo: 'emphasizing the its ability' should be 'emphasizing its ability'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an empirical self-training pipeline with no fitted-parameter predictions or derivation that reduces to its own inputs.

full rationale

LAM SIMULATOR is an empirical data-generation framework, not a mathematical derivation. Training trajectories are produced by letting an LLM agent explore tools, then filtering them by string-matching the agent's final answer against a programmatically computed ground truth; the filtered trajectories are used to fine-tune the same base model. Gains on ToolBench and CRMArena are measured by pass rates on held-out test sets. There is no equation in which a predicted quantity is defined as the fitted input, and no parameter is fitted to a subset and then reported as a prediction. The framework's own ablations (removing action monitoring or trajectory filtering) provide independent evidence that the monitoring components, rather than the mere addition of data, drive the reported gains. The main caveat is data independence: CRMArena's NCR, HTU, MTA, and BRI test tasks are acknowledged to share scenarios or tools with the exploration tasks, and ToolBench query templates are built from the ToolBench training split, so part of the improvement may reflect in-distribution fine-tuning rather than a generalizable self-improvement mechanism. That is an experimental confound, not a circular derivation, and the paper does present genuinely out-of-domain test tasks (TCU, TII, G1_tool, G1_cat) where nontrivial gains are also reported. Self-citations to xLAM, CRMArena, APIGen, and related prior work are used to source models and benchmarks, but the central empirical claim does not rest on an unverified self-cited theorem or on a uniqueness argument. Thus no circular step meeting the required evidentiary standard is present; the score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard ML training assumptions plus dataset-independence assumptions that are not demonstrated. No new physical entities are introduced. The main unstated burdens are the reliability of the sandbox and string-matching filter, and the assumption that benchmark test splits do not overlap with the templated training tasks.

free parameters (3)

Exploration temperature = 1.0
Set for trajectory sampling during exploration; affects diversity and success rate of self-generated data.
Trajectory collection cap = 500
Exploration stops after 500 filtered trajectories per model; dataset size and training outcome depend on this cap.
Trajectory filtering criteria = no errors, or errors corrected in next action
Hand-designed quality threshold that determines which trajectories enter the fine-tuning dataset.

assumptions (4)

domain assumption Supervised fine-tuning on filtered trajectories improves agent performance
Standard ML assumption used throughout Section 3.4; not independently verified here.
domain assumption The sandbox environment faithfully executes tool calls and returns realistic observations
Needed for the exploration trajectories to be meaningful; no validation of the sandbox's fidelity is provided.
domain assumption String matching between final answer and precomputed ground truth correctly identifies task success
Used in Trajectory Filtering (Section 3.3); brittle because correct answers phrased differently would be rejected.
domain assumption ToolBench and CRMArena test sets are independent of the templated exploration tasks
Required for the reported gains to reflect generalization; contradicted by template construction based on ToolBench training instances (Section 4.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback." pith.science (2026). https://pith.science/paper/E7BQHOV3

@misc{pith2026250602298,
  author       = {Pith},
  title        = {Pith review of: LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E7BQHOV3}},
  note         = {Machine review of arXiv:2506.02298}
}

read the original abstract

Large Action Models (LAMs) for AI Agents offer incredible potential but face challenges due to the need for high-quality training data, especially for multi-steps tasks that involve planning, executing tool calls, and responding to feedback. To address these issues, we present LAM SIMULATOR, a comprehensive framework designed for online exploration of agentic tasks with high-quality feedback. Our framework features a dynamic task query generator, an extensive collection of tools, and an interactive environment where Large Language Model (LLM) Agents can call tools and receive real-time feedback. This setup enables LLM Agents to explore and solve tasks autonomously, facilitating the discovery of multiple approaches to tackle any given task. The resulting action trajectory data are then used to create high-quality training datasets for LAMs. Our experiments on popular agentic benchmarks, ToolBench and CRMArena, highlight the effectiveness of LAM SIMULATOR: models trained with self-generated datasets using our framework achieve significant performance gains, up to a 49.3\% improvement over their original baselines. LAM SIMULATOR requires minimal human input during dataset creation, highlighting LAM SIMULATOR's efficiency and effectiveness in speeding up development of AI agents.

Figures

Figures reproduced from arXiv: 2506.02298 by the authors.

**Figure 1.** Overview of the LAM SIMULATOR, illustrating the framework’s main components, their interactions, and emphasizing the its ability to generate tool-use data, execute functions, and evaluate results. Query Instance Generation is responsible for generating tasks, including creating user queries, preparing available tools for the LLM Agent, and dynamically computing ground-truth answers. Trajectory Synthesis manages the … view at source ↗

**Figure 2.** Ablation study on monitoring components across three ToolBench test sets. mixtral-8x7b-inst shows baseline model performance. "Not monitor Action" indicates performance when fine-tuning with a self-exploration dataset via LAM SIMULATOR without action monitoring, while "Not monitor Trajectory" shows results without trajectory monitoring. mixtral-8x7b-ls demonstrates performance with both action and trajectory moni… view at source ↗

**Figure 6.** Example of Filled-in Query with Query tem [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

Figures from the paper (6 more)

**Figure 3.** Figure 3: , a query template contains natural text portions and placeholders to-be-filled. " query_template ": "I've been looking up { movie_detail } about the movie { movie_name }. Fun fact : the set of { movie_name } was built inside a massive warehouse to create a surreal atm…

**Figure 7.** Figure 7: Example of a solution path for the task 6. The arguments would be searched among 1) placeholder values and 2) objects generated during execution. In this example, movie_name can be extracted directly from the placeholder value (The Dark Night), from 5, while id is a ne…

**Figure 5.** Figure 5: Example of generated Placeholders Filled-in query With Query Template (3), and Generated Placeholders (5), we can fill in the value of the placeholder into the query template to create Filled-in query. An example is showned at 6. Answer computation We also give an exam…

**Figure 8.** Figure 8: Example set of available tools to be provided for LLM Agent for exploration. Note that this tools set does not include get_search_movie_for_movie_tools, but instead include an alternative version search_movie_for_imdb, which does the similar objective with get_search…

**Figure 9.** Figure 9: Tools distribution for ToolBench environment. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

**Figure 10.** Figure 10: Example of system prompt for LLM Agent D Others When constructing this paper, we used gpt-4o (Achiam et al., 2023) for several paraphrasing [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models
cs.AI 2025-07 conditional novelty 6.0 of 10

MCPEval is an automated MCP-based framework that generates, verifies, and scores LLM agent tool-use tasks; its experiments reveal a consistent gap between how well agents execute tool calls and how well they synthesiz...

Reference graph

Works this paper leans on

30 extracted references · 4 canonical work pages · cited by 1 Pith paper

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card

2024
[3]

Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915

arXiv 2023
[4]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024
[5]

Nicholas Farn and Richard Shin. 2023. Tooltalk: Evaluating tool-usage in a conversational setting. arXiv preprint arXiv:2311.10775

arXiv 2023
[6]

Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. 2025. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associatio...

work page 2025
[7]

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

arXiv 2024
[8]

Shirley Kokane, Ming Zhu, Tulika Awalgaonkar, Jianguo Zhang, Thai Hoang, Akshara Prabhakar, Zuxin Liu, Tian Lan, Liangwei Yang, Juntao Tan, Rithesh Murthy, Weiran Yao, Zhiwei Liu, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, Caiming Xiong, and Silivo Savarese. 2024. https://arxiv.org/abs/2411.13547 Spectool: A benchmark for characterizing errors in to...

arXiv 2024

Show all 30 references

[9]

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688

2023 arXiv
[10]

Zhiwei Liu, Weiran Yao, Jianguo Zhang, Liangwei Yang, Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, et al. 2024 a . Agentlite: A lightweight library for building and advancing task-oriented llm agent system. arXiv preprint arXiv:2402.15538

2024 arXiv
[11]

Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. 2024 b . Apigen: Automated pipeline for...

2024
[12]

Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024 a . Agentboard: An analytical evaluation board of multi-turn llm agents. arXiv preprint arXiv:2401.13178

2024 arXiv
[13]

Zixian Ma, Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Caiming Xiong, Ranjay Krishna, and Silvio Savarese. 2024 b . https://arxiv.org/abs/2412.05479 Taco: Learning multi-modal action models with synthetic chai...

2024
[14]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

2022
[15]

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789

2023 arXiv
[16]

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36

2024
[17]

Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. https://arxiv.org/abs/2403.02502 Trial and error: Exploration-based trajectory optimization for llm agents . Preprint, arXiv:2403.02502

2024 arXiv
[18]

Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Arık. 2025. https://arxiv.org/abs/2501.10893 Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments . Preprint, arXiv:2501.10893

2025 arXiv
[19]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[20]

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972

2024 arXiv
[21]

Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, et al. 2024. Lemur: Harmonizing natural language and code for language agents. ICLR

2024
[22]

Patil, Ion Stoica, and Joseph E

Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html

2024
[23]

Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. tau -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045

2024 arXiv
[24]

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct : Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

2023
[25]

Jianguo Zhang, Thai Hoang, Ming Zhu, Zuxin Liu, Shiyu Wang, Tulika Awalgaonkar, Akshara Prabhakar, Haolin Chen, Weiran Yao, Zhiwei Liu, et al. 2025. Actionstudio: A lightweight framework for data and training of action models. arXiv preprint arXiv:2503.22673

2025 arXiv
[26]

Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng, Zuxin Liu, et al. 2024 a . Agentohana: Design unified data and training pipeline for effective agent learning. arXiv preprint arXiv:2402.15506

2024 arXiv
[27]

Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, et al. 2024 b . xlam: A family of large action models to empower ai agent systems. arXiv preprint arXiv:2409.03215

2024 arXiv
[28]

Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854

2023 arXiv
[29]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[30]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card

2024

[3] [3]

Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915

arXiv 2023

[4] [4]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024

[5] [5]

Nicholas Farn and Richard Shin. 2023. Tooltalk: Evaluating tool-usage in a conversational setting. arXiv preprint arXiv:2311.10775

arXiv 2023

[6] [6]

Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. 2025. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associatio...

work page 2025

[7] [7]

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

arXiv 2024

[8] [8]

Shirley Kokane, Ming Zhu, Tulika Awalgaonkar, Jianguo Zhang, Thai Hoang, Akshara Prabhakar, Zuxin Liu, Tian Lan, Liangwei Yang, Juntao Tan, Rithesh Murthy, Weiran Yao, Zhiwei Liu, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, Caiming Xiong, and Silivo Savarese. 2024. https://arxiv.org/abs/2411.13547 Spectool: A benchmark for characterizing errors in to...

arXiv 2024