{"id":"d2c3a9b2-fa13-4936-a0a7-f3bf757c4f3b","arxiv_id":"2506.02298","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Self-generated agent trajectories from LAM SIMULATOR improved fine-tuned model pass rates by up to 49.3% on ToolBench and CRMArena.","lead":"LAM SIMULATOR is a framework that lets AI agents explore tasks by calling tools in an interactive environment, then uses the successful action trajectories as training data. Fine-tuning models on this self-generated data improved their pass rates by up to 49.3% on two agent benchmarks, ToolBench and CRMArena.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing gold-solution-path control: reported gains may come from added SFT data, not from online exploration and trajectory feedback.","rationale":"I read the paper in good faith. The framework is plausible, the evaluation is on standard benchmarks, and the reported out-of-domain gains (e.g., CRMArena TCU/TII, ToolBench G1_cat) are real evidence of generalization beyond the immediate exploration tasks. The reader's weakest assumption was test-set independence; that concern is substantial but partially mitigated by the existence of unseen-tool and unseen-category evaluations and by the out-of-domain CRMArena tasks. The more load-bearing gap is the absence of a gold-trajectory training control. The paper itself, in Section 3.4, contrasts self-explored trajectories with gold-standard trajectories and claims two advantages, yet it never reports that comparison. Without it, the empirical improvement could be explained by simply having 500 additional correct demonstrations for in-distribution tasks, regardless of whether they were discovered by exploration or taken from the precomputed solution paths. This directly targets the mechanism the title and abstract emphasize: 'online exploration and trajectory feedback.' The ablation in Section 6.3.2 tests monitoring components, but it does not isolate exploration from trajectory content. The comparison with ToolLLM variants tests a different pipeline, not the gold-solution baseline. Therefore the central claim is conditional: it requires showing that exploration-derived trajectories outperform the cheaper gold trajectories that the framework already computes. The supplied concrete test would settle this with minimal additional computation, reusing the existing query instances and training protocol. I therefore keep the reader's CONDITIONAL verdict, with this missing control as the primary condition for acceptance.","tokens_in":14787,"tokens_out":6716,"duration_ms":62138,"concrete_test":"Fine-tune the same base models (mixtral-8x7b-inst, xlam-8x7b-r, xlam-7b-r, gpt-4o, gpt-4o-mini) on 500 trajectories constructed directly from the Section 3.2 solution paths for the same query instances, using identical hyperparameters (5e-6, 3 epochs; OpenAI defaults for GPT models) and temperature-0.0 evaluation. Compare per-task pass rates on ToolBench G1_inst/G1_tool/G1_cat and CRMArena NCR/HTU/MTA/BRI/TCU/TII against Tables 2 and 3. If gold-trajectory models match or exceed LAM Simulator models, the online-exploration component is not responsible for the reported gains; if they fall short, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that online exploration with trajectory feedback produces training data that improves agent performance. The paper never runs the natural control: fine-tuning on the gold solution paths defined in Section 3.2. Those paths are the programmatic tool sequences used to compute ground-truth answers; they exist for all 400 ToolBench and 400 CRMArena query instances and require no exploration, no action handler, and no trajectory filtering. Section 3.4 explicitly argues that self-explored trajectories teach error recovery and expose the model to a wider range of scenarios, but no experiment isolates this variable. If fine-tuning on 500 gold trajectories achieves the same or larger pass-rate gains, then the reported improvements (e.g., mixtral-8x7b-inst from 11.7% to 31.0% on ToolBench; gpt-4o-mini from 18.5% to 67.8% on CRMArena) are attributable to additional in-distribution supervised data, not to the exploration/feedback mechanism. The ToolLLM_Full and ToolLLM_Partial comparisons do not resolve this, because those baselines generate trajectories through a different pipeline rather than using the gold solution paths as a control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LAM Simulator, a framework for generating agentic training data through online exploration. Queries are produced by filling manually designed templates with LLM-sampled parameters; ground-truth answers are computed programmatically via predefined tool sequences; agents explore the resulting tasks in an interactive environment with an action handler and trajectory filtering; the filtered trajectories are then used to fine-tune the base model. Experiments on ToolBench and CRMArena report pass-rate gains for gpt-4o, gpt-4o-mini, xlam-8x7b, xlam-7b, and mixtral-8x7b, with the largest gain being 49.3 percentage points for gpt-4o-mini on CRMArena. The paper also compares against ToolLLM variants on CRMArena and includes ablations of the action-handler and trajectory-filtering components. Section 8 appropriately acknowledges that the current implementation focuses on predefined tasks and tools, but several load-bearing empirical claims in Sections 4-6 need additional controls before the main conclusions are supported.","tokens_in":14987,"tokens_out":6133,"duration_ms":56116,"significance":"If the central empirical claim is robust, the framework would be a useful contribution to agent data generation: it replaces LLM-based evaluation with programmatic answer verification, assembles a large cleaned tool library, and demonstrates across multiple base models that self-explored trajectories can improve fine-tuned agent performance. The paper's strengths include its fully programmatic trajectory filtering, dynamic answer computation, and the breadth of models tested in Section 6. However, the significance is currently conditional because the paper does not isolate the contribution of online exploration and trajectory feedback from simply adding extra supervised trajectories, and because the training data generation overlaps with the evaluation benchmarks in ways that may inflate the reported gains. With additional controls, the result would be valuable for the agent-data-generation community; as written, the empirical contribution is not yet causally identified.","major_comments":[{"comment":"The paper defines gold solution paths in §3.2 (and Appendix A.1) and argues in §3.4 that self-explored trajectories teach error recovery and expose the model to a wider range of scenarios, but it never runs the natural control of fine-tuning on the gold solution paths. Since all reported gains are relative to base models that receive no additional SFT data, the improvements in Tables 2 and 3 could come from adding 500 valid in-distribution trajectories rather than from online exploration, error recovery, or trajectory feedback. I request a control where each base model is fine-tuned on the same number of gold solution-path trajectories (and ideally on randomly sampled successful non-explored trajectories) and evaluated on the same test sets; without this control, the central claim that the exploration and feedback mechanisms drive the improvement is not supported.","section":"§3.2, §3.4, §5.3, Tables 2 and 3"},{"comment":"The evaluation benchmarks overlap with the data-generation setup in ways that may inflate the reported gains. For ToolBench, §4.1 says the 30 query templates are 'based on instances from the ToolBench training dataset', and the evaluation in §5.1 uses ToolBench test sets whose tools and categories may overlap with the exploration tasks; the assertion that the overlap is 'exceedingly low' is not supported by any measurement. For CRMArena, §4.2 conducts exploration on NCR, HTU, MTA, and BRI while §5.1 evaluates on all six tasks, so the four in-domain tasks share scenarios and tools with training; the large in-domain gains (e.g., gpt-4o on CRMArena ALL from 62.3% to 86.4%) are therefore at least partly attributable to in-distribution fine-tuning. The paper should quantify the overlap in instructions, tools, and data distributions between the generated queries and each test set, and should evaluate on tasks or environments with no template or tool overlap to support the generalization claims. The out-of-domain TCU and TII results are encouraging but cover only two of six tasks.","section":"§4.1, §4.2, §5.1, Tables 2 and 3"},{"comment":"No confidence intervals or significance tests are reported anywhere in the results. Evaluation temperature is set to 0.0, but fine-tuning initialization and trajectory sampling are stochastic, so a single run cannot establish small gains. For example, xlam-7b on ToolBench ALL moves from 39.2% to 41.3% on 600 test instances, which is within roughly two standard errors of the baseline, and gpt-4o on ToolBench ALL moves from 47.4% to 51.5%, which is a marginal difference. The paper should report variance over multiple fine-tuning seeds or bootstrap confidence intervals, and should identify which of the reported gains are statistically distinguishable from the baselines.","section":"Tables 2 and 3, §5.1, §5.3"}],"minor_comments":[{"comment":"The '49.3% improvement' and 'over 31% higher' figures are percentage-point differences, not relative improvements; the paper should state this explicitly to avoid overstating the gains (18.5% to 67.8% is a 49.3-point increase but a roughly 267% relative increase).","section":"Abstract and §6.2"},{"comment":"The column header 'MTI' appears to be a typo for 'MTA' (Monthly Trend Analysis), which is the task named in §4.2.","section":"Table 3"},{"comment":"The claim of 'minimal human input' conflicts with the detailed, per-template human effort described in §3.2 and Appendix A.1: humans must write query templates, placeholder metadata, and solution-path tool sequences for every template. The authors should either quantify the one-time human effort or temper the claim to 'minimal per-instance human annotation'.","section":"Abstract, §1, §3.2, §3.5"},{"comment":"Section 3.5 says generalizing to a new environment requires only query templates and tool-call mappings, while Section 8 acknowledges that the implementation is limited to predefined tasks and tools; these statements should be reconciled, and the limitations section should be reflected in the main text's generalizability claims.","section":"§3.5 and §8"},{"comment":"The appendix contains language errors such as 'An example is showned at 6' and the sentence 'The arguments would be searched among 1) placeholder values and 2) objects generated during execution' is grammatically incomplete.","section":"Appendix A.1"},{"comment":"The caption contains a typo: 'emphasizing the its ability' should be 'emphasizing its ability'.","section":"Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for CS.CL, and the framework is a plausible contribution to agent data generation. My main concern is that the headline results are not yet causally identified: the missing gold-trajectory control and the training/evaluation overlap leave the empirical contribution ambiguous. If the authors can supply those experiments, the paper would likely be acceptable. I would also ask the editor to encourage artifact release or at least detailed release of query templates and tool implementations, since the current manuscript does not provide enough detail to reproduce the pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Briefly: this is a legitimately useful integration—multi-turn query generation with programmatic ground-truth answers, an interactive tool environment, and trajectory filtering—and the authors report large pass-rate gains on ToolBench and CRMArena after fine-tuning on self-generated data. The ToolLLM comparisons are a fair touch, and the ablation showing that removing the action handler or trajectory filter hurts performance is real evidence those components matter.\n\nThe soft spot is the one the stress-test note flags, and it lands on reading the paper. Section 3.4 argues that self-explored trajectories teach error recovery and generalize better, but the paper never runs the natural control: fine-tuning on the gold solution paths defined in Section 3.2. Those paths are programmatic, require no exploration, and exist for all 800 query instances. Without that control, the reported gains (e.g., mixtral from 11.7% to 31.0% on ToolBench; gpt-4o-mini from 18.5% to 67.8% on CRMArena) could just be the effect of adding more in-distribution SFT data. The ToolLLM baselines don't resolve this because they use a different trajectory-generation pipeline, not the gold paths.\n\nThere's also a distribution-overlap issue: ToolBench templates are 'based on instances from the ToolBench training dataset,' and CRMArena explores four of the six tasks it evaluates on. The out-of-domain sets (G1_tool, G1_cat, TCU, TII) help, but the in-domain gains are suspect, and there are no confidence intervals or significance tests. The 'minimal human input' claim is overstated: designing 36 high-level tasks, dozens of query templates, placeholder descriptions, and solution paths for 800 instances is substantial human engineering. That doesn't sink the paper, but it should be described honestly.\n\nBottom line: for people working on data generation for tool-using agents, this is a useful reference point, and the programmatic-evaluation piece is a genuine improvement over LLM-as-judge pipelines. But the central mechanistic claim—that online exploration and trajectory feedback, not just additional SFT data, drive the gains—is untested. A serious referee should ask for the gold-solution-path control, clearer in-domain/out-of-domain separation, and code/data release. I'd send it to review; it's a solid empirical contribution that needs one more experiment to nail the interpretation.","headline":"Useful data-generation framework, but the headline claim that exploration plus feedback drives the gains is not actually tested against the obvious gold-solution-path control.","tokens_in":15578,"tokens_out":3028,"would_cite":true,"duration_ms":25174,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM agent can improve its own tool-use performance by training on trajectories it generates through online exploration, with pass-rate gains up to 49.3% in the paper's experiments.","keywords":["large action models","data generation","online exploration","trajectory feedback","tool use","self-improvement","agentic AI","programmatic evaluation"],"falsifier":"Run the fine-tuning procedure with exploration tasks and tools that are provably disjoint from the evaluation tasks (for example, generate exploration data from the two held-out CRMArena tasks TCU and TII and evaluate on the other four, or swap the tool sets between exploration and evaluation on ToolBench). If the pass-rate gains shrink or disappear on such disjoint test sets, the claimed improvements would be shown to come from in-distribution fine-tuning rather than from the framework's exploration-and-feedback mechanism.","tokens_in":14597,"feed_emoji":"🤖","tokens_out":9577,"duration_ms":70521,"temperature":0.7,"pith_summary":"The paper introduces LAM SIMULATOR, a framework for generating training data for large action models (LAMs) — AI agents that plan, call tools, and respond to feedback. The central idea is to let an LLM agent autonomously explore tasks in an environment, collect the resulting action trajectories, filter them against programmatically computed ground-truth answers, and fine-tune the same model on the surviving trajectories. The paper reports that this self-improvement loop yields pass-rate gains of up to 49.3% on CRMArena and 19.3% on ToolBench over baseline models, and that the generated data beats data produced by ToolLLM. If these results hold, high-quality agent training data can be produced with minimal human labeling, which would speed up the development of tool-using AI agents.","feed_headline":"Agent self-exploration lifts tool-use pass rates by up to 49.3%","feed_subtitle":"Fine-tuning on self-generated trajectories beats base models on ToolBench and CRMArena.","key_machinery":"The key machinery is the LAM SIMULATOR pipeline, which combines three components: a template-filling query generator that creates novel tasks with pre-computed ground-truth answers, an interactive environment whose action handler validates every tool call and returns error messages for malformed or hallucinated calls, and a trajectory filter that keeps only trajectories whose final answer matches the ground truth and that either contain no errors or recover from errors in the next step. This machinery allows the agent to explore with a tool set that can differ from the solution path used to compute the ground truth, receive real-time feedback, and generate training trajectories that include error-recovery behavior. The framework's programmatic evaluation avoids LLM-based judgment, which the paper argues makes the resulting training data more reliable than previous approaches.","core_discovery":"On its own terms, the paper's central claim is that an LLM agent can improve its own tool-use performance by training on trajectories it collected through online exploration, provided the exploration is supervised by a programmatic action handler and the trajectories are filtered by matching their final answers to ground truths. The authors demonstrate this by fine-tuning five models on self-generated data from LAM SIMULATOR and showing consistent pass-rate improvements over each baseline: for example, gpt-4o rises from 47.4% to 51.5% on ToolBench, and gpt-4o-mini rises from 18.5% to 67.8% on CRMArena. They further claim that the gains extend to out-of-domain evaluations, including unseen tools and categories on ToolBench and two held-out CRM tasks, and that LAM SIMULATOR's fully programmatic evaluation produces higher-quality training data than ToolLLM's LLM-based generation and evaluation.","pith_inferences":["If the independence assumption holds, the same exploration-and-filter loop could be applied to other agent benchmarks and real-world tool environments, but its dependence on hand-designed query templates and solution paths may limit scaling to open-ended tasks — a limitation the paper itself acknowledges.","The ablation shows that monitoring actions and filtering trajectories are each essential; a natural next experiment would ablate the error-recovery criterion specifically to see whether the correction trajectories are the main driver of the gains.","The paper compares against ToolLLM only on CRMArena; a stronger test would generate data from both methods on the same task pool and evaluate on multiple benchmarks to separate data-quality from task-distribution effects.","The claim of 'minimal human input' applies to the exploration phase; the initial creation of query templates, solution paths, and tool cleanup still requires substantial human design, so the framework's universality would be tested by automating template creation or transferring templates across domains."],"forward_implications":["Models fine-tuned on their own exploration trajectories can improve tool-use pass rates without new human-labeled data, reducing the cost of agent training.","Programmatic evaluation (action handler plus ground-truth filtering) produces training data that outperforms LLM-based generation and evaluation pipelines like ToolLLM in direct comparison on CRMArena.","The measured gains on unseen tools, unseen categories, and held-out CRM tasks suggest that self-generated trajectories teach generalizable tool-use behavior, not just answers to the exploration tasks.","Lower-performing models benefit disproportionately more from self-generated data (mixtral-8x7b-inst rises from 11.7% to 31.0% on ToolBench; gpt-4o-mini rises from 18.5% to 67.8% on CRMArena), indicating the approach can bootstrap weaker agents."],"supporting_citations":[{"why":"It supplies the ToolBench benchmark, the 16,464-API collection that is cleaned into 3,420 tools, and the ToolLLM baseline against which LAM SIMULATOR is compared.","marker":"(Qin et al., 2023)"},{"why":"It supplies the CRMArena environment, its 25 CRM tools and six tasks (four used for exploration, two held out), and the evaluation test sets.","marker":"(Huang et al., 2025)"},{"why":"APIGen is the prior automated function-calling data pipeline that LAM SIMULATOR extends to multi-turn settings and programmatic evaluation.","marker":"(Liu et al., 2024b)"},{"why":"Learn-by-Interact is the prior self-adaptation framework whose LLM-based evaluation and documentation dependence LAM SIMULATOR aims to avoid.","marker":"(Su et al., 2025)"},{"why":"It provides the xLAM family of action models (xlam-8x7b-r, xlam-7b-r) that are fine-tuned on LAM SIMULATOR-generated trajectories.","marker":"(Zhang et al., 2024b)"},{"why":"It provides the GPT-4o and GPT-4o-mini models used both as base models for fine-tuning and as the LLM agents that explore tasks.","marker":"(Achiam et al., 2023)"}],"fun_headline_variants":["Self-exploration boosts agent tool-use accuracy by 49.3%","Agents learn from own trajectories, gain up to 49.3%","LLM agents improve tool-use 49.3% via self-generated data","Online exploration yields 49.3% better agent tool-use","Train on own actions: agent tool-use up 49.3%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the evaluation test sets are genuinely independent of the exploration-generated training data, so that the reported pass-rate gains reflect improved agent ability rather than in-distribution fine-tuning on near-duplicate tasks or tools.","fun_headline_variants_meta":{"raw":{"variants":["Self-exploration boosts agent tool-use accuracy by 49.3%","Agents learn from own trajectories, gain up to 49.3%","LLM agents improve tool-use 49.3% via self-generated data","Online exploration yields 49.3% better agent tool-use","Train on own actions: agent tool-use up 49.3%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000742,"raw_usage":{"total_tokens":3315,"prompt_tokens":957,"completion_tokens":2358,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2262}},"tokens_in":573,"tokens_out":2358,"duration_ms":15495,"temperature":1.0,"reasoning_tokens":2262,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:26:39.204130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the fine-tuning procedure with exploration tasks and tools that are provably disjoint from the evaluation tasks (for example, generate exploration data from the two held-out CRMArena tasks TCU and TII and evaluate on the other four, or swap the tool sets between exploration and evaluation on ToolBench). If the pass-rate gains shrink or disappear on such disjoint test sets, the claimed improvements would be shown to come from in-distribution fine-tuning rather than from the framework's exploration-and-feedback mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the CRMArena environment, its 25 CRM tools and six tasks (four used for exploration, two held out), and the evaluation test sets."}],"review_version":1}