{"id":"6ae0db42-b1c7-40ba-b3b8-8689cda4b5e9","arxiv_id":"2501.01702","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Training LLM agents on synthetic trajectories that include mistake-and-refinement steps improves generalization to unseen agent environments.","lead":"This paper trains open-source LLM agents to correct their own mistakes during tasks, using synthetic environments and refinement trajectories generated by GPT-4o. It claims this improves generalization to new, unseen agent environments compared to existing agent-tuning methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The refinement-specific benefit is confounded by data regeneration: the 'w/o refinement data' ablation changes the entire training distribution, so it cannot isolate refinement; a matched-pair removal of refinement turns is required.","rationale":"The reader's weakest assumption focused on the fidelity of GPT-4o's synthetic environments and feedback, and on the small reliability sample in Section 8. I agree those are real concerns, but the more load-bearing issue is internal to the paper's ablations: the claimed mechanism (self-refinement) is not isolated by the reported experiments. The paper wants to conclude that refinement tuning explains held-out gains. Table 2's 'w/o refinement data' regenerates a new corpus rather than editing the same corpus, so the comparison conflates refinement content with dataset diversity, trajectory quality, and verifier selection. This is not a pedantic point: the paper itself argues in §4.4 that environment and thought diversity contribute to generalization, so a correct-only dataset generated by the same pipeline could easily be better or worse for reasons unrelated to refinement. Appendix I, Table 11 strengthens the worry, since the 'w/o refinement data' variant also shows low perturbation variance, suggesting the pipeline's diversity, not refinement, drives robustness. Section 8's human check validates turn-level error labels on 50 trajectories, which is useful but does not establish that the synthetic environment feedback resembles real environment signals in Alfworld, ScienceWorld, PDDL, BabyAI, Jericho, or HotpotQA. The central claim is plausible and the direction of the result may be right, but the causal attribution to refinement requires a matched-pair ablation and release of the data. I do not see any evidence of bad faith; the issue is experimental design. My recommendation therefore leaves the reader's CONDITIONAL verdict unchanged, with the condition sharpened to require a controlled refinement-removal ablation and data release.","tokens_in":19877,"tokens_out":3819,"duration_ms":34268,"concrete_test":"Release the AgentRefine dataset (all 32k trajectories) and the verification code. Then run the essential control: take the same set of AgentRefine trajectories, remove the erroneous turns and the immediately following refinement turns (or mask them in the loss), and train on the remaining correct turns only. If performance drops to the same degree as the current 'w/o refinement data' row, the refinement content is causal. If the drop is small, the reported gains come from data regeneration rather than refinement. This control isolates the refinement signal because environment, task distribution, and trajectory lengths are held fixed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that refinement tuning drives generalization gains. Table 2 compares AgentRefine against a variant trained on re-generated data without errors/refinement ('w/o refinement data') and against loss-masking ablations. This design cannot separate the effect of refinement from the effect of regenerating a different training corpus. The 'w/o refinement data' set is synthesized separately, so it differs in environment coverage, task difficulty, trajectory length, and thought diversity—factors the paper itself credits for generalization (§4.4). This is not an ablation of refinement content; it is a comparison across datasets. The observed gains attributed to refinement may instead be caused by the regeneration producing a different distribution, or by verifier selection. The loss-masking variant ('w/o refinement loss') shows erroneous tokens interfere, but does not test whether positive refinement steps teach generalizable correction behavior. Moreover, Appendix I, Table 11 shows that 'w/o refinement data' retains perturbation robustness, undercutting the paper's attribution of robustness to refinement. The paper's causal statement that refinement tuning drives improvement therefore rests on a confound. Section 8's reliability check covers only 50 trajectories and validates GPT-4o turn-level error labels, not whether refinement steps resemble real environment feedback. Without released data this confound cannot be inspected.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AgentRefine, a framework for instruction-tuning LLM agents on synthetic trajectories that deliberately include error-and-refinement turns, with a loss function that masks erroneous turns. The authors claim that this refinement tuning significantly improves generalization to held-out agent tasks compared to prior agent-tuning methods such as Agent-FLAN and AgentGym, and that it also improves robustness to environment perturbations and performance on a reasoning task (HotpotQA). The method is evaluated on five AgentBoard environments and on HotpotQA using LLaMA-3 and Mistral models.","tokens_in":20121,"tokens_out":3702,"duration_ms":36825,"significance":"The paper addresses an important problem: open-source LLM agents trained by instruction tuning on existing agent corpora tend to overfit to training environments and fail to generalize. If the central claim is correct, adding synthetic error-and-refinement trajectories is a simple and potentially widely applicable recipe for improving held-out agent performance, and the connection between self-refinement and generalization would be a useful conceptual contribution. The paper includes several analyses (diversity, perturbation robustness, Best-of-N, open-source synthesis) that go beyond a single benchmark comparison. However, the strength of the causal claims about refinement is not fully supported by the experiments as designed.","major_comments":[{"comment":"The 'w/o refinement data' ablation is confounded. The text states that this condition uses a re-generated training set without error and refinement trajectories; because the data are regenerated, the condition differs from AgentRefine along many dimensions other than the presence of refinement, including environment coverage, trajectory length, task difficulty, and thought diversity. The paper itself credits such diversity for generalization (§4.4), so the observed performance drop cannot be uniquely attributed to refinement tuning. A matched-pair ablation that removes only the refinement turns from the same trajectories (or otherwise controls for trajectory-level attributes) is needed to support the paper's central causal claim. The 'w/o refinement loss' condition is more controlled, but it only tests the effect of loss masking, not the effect of the refinement content on learned behavior.","section":"§4.2, Table 2"},{"comment":"The reported results are obtained by selecting the checkpoint with the best average results on the evaluation benchmarks after training for 10 epochs. This is a form of test-set model selection and can inflate reported performance and mask variance. For a paper whose central claim is that AgentRefine 'significantly outperforms' prior methods, the selection procedure should be reported transparently, and the main results should either be averaged over checkpoints or selected using a validation set that is disjoint from the reported test tasks. Otherwise, the improvements over baselines may be partly due to cherry-picking a favorable epoch.","section":"Appendix D (and §4.1, Table 1)"},{"comment":"The robustness claim is undercut by the 'w/o refinement data' condition. Table 11 shows that this condition achieves essentially the same average success rate (48.78 versus 48.48) and a comparable standard deviation (5.47 versus 5.78) under the same perturbations as AgentRefine. This contradicts the attribution of perturbation robustness to refinement tuning. The paper should either temper the robustness claim or provide an explanation of why the 'no refinement' model retains robustness; the data diversity explanation is a plausible alternative that is not ruled out by the current experiments.","section":"§4.3 and Appendix I, Table 11"},{"comment":"The reliability check of the synthetic data uses only 50 trajectories and covers only whether GPT-4's turn-level error labels agree with human annotations. It does not validate whether the refinement steps (the corrected actions and their surrounding thoughts) faithfully resemble behaviors that would recover from real environment feedback in the target benchmarks. Since the transfer of the learned refinement behavior to AgentBoard and HotpotQA rests on the fidelity of the synthetic feedback, a larger and more direct fidelity check is necessary. At minimum, the paper should acknowledge this limitation explicitly in the main text rather than only presenting the 94%/82% agreement numbers.","section":"§8, Figure 10"}],"minor_comments":[{"comment":"The loss equation is not written correctly: the indicator 1(Aj) appears inside the logarithm, which would produce log(0) for erroneous turns. The intended operation is clearly to multiply the log-likelihood by the indicator (i.e., 1(Aj) * log(...)), and the equation should be corrected to avoid confusion about the masking scheme.","section":"§3.3, Eq. (1)"},{"comment":"The cross-reference 'In Appendix 5, we use deepseek-v2.5' is incorrect; Section 5 is in the main text, not an appendix. The reference should be fixed.","section":"§3.2"},{"comment":"The text says 'As shown in Figure 3' when referring to the environment similarity heatmap, but the heatmap is in Figure 7. The figure reference is wrong.","section":"§4.4"},{"comment":"The paper says it changes AgentBoard's prompts from Act-only to ReAct. This is a modification of the evaluation setup, and although it is applied consistently across all methods, the deviation from the standard AgentBoard protocol should be described more precisely (e.g., exact prompt format, which example prompts were adjusted) so that readers can replicate or compare with other published results.","section":"§4.1"},{"comment":"Perturbation 5 is described as removing all in-distribution training data and retraining the model, which is not a perturbation of the test environment but a change to the trained model. Reporting this as a perturbation alongside four genuine environment perturbations can be misleading; it should be presented as a separate analysis of training-data dependence.","section":"Appendix K, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is already published at ICLR 2025, so the technical novelty is established. For a journal version, the main concern is that the causal attribution of the gains to refinement tuning is not supported by the current ablation design, and the checkpoint-selection procedure inflates the reported results. These issues are fixable within the scope of a revision, provided the authors either add a matched-pair ablation, report results without test-set checkpoint selection, or substantially weaken the causal claims in the conclusion. The paper would also benefit from releasing the synthetic dataset to allow independent inspection of the confound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: AgentRefine is a useful empirical contribution, and the held-out gains are probably real. But the headline attribution—refinement tuning causes the generalization jump—is not established by the paper's ablation design.\n\nWhat's new and good: the combination is genuinely original: persona-based environment synthesis, trajectories with deliberately injected errors and subsequent refinement steps, plus loss masking over erroneous turns. The results across five AgentBoard tasks and HotpotQA are consistent, and the scaling curve (4k to 64k) is a nice touch. The perturbation study is also informative: Agent-FLAN and AgentGym collapse under small action-description changes, while AgentRefine holds steady. That is a real behavioral difference, and it matches the story about memorization vs. learning.\n\nSoft spots, in rough order of importance. First, the central ablation is confounded. 'w/o refinement data' is not a matched removal of refinement tokens; it is a separately generated training corpus with no errors or refinement. That corpus differs in environment coverage, difficulty, trajectory length, and thought diversity—factors the paper itself credits for generalization (§4.4). So the comparison cannot isolate refinement. A matched-pair ablation that removes or masks refinement turns from the same trajectories would be needed. The loss-masking result ('w/o refinement loss') shows that erroneous tokens hurt, which is an internally coherent finding, but it doesn't show that the positive refinement steps are what generalizes. Appendix I, Table 11 makes this worse: 'w/o refinement data' retains perturbation robustness, which suggests the robustness may come from data diversity, not refinement.\n\nSecond, Appendix D says they report the checkpoint with the best average test results. That is test-set selection, and it inflates the absolute numbers. It doesn't invalidate the relative comparisons, but it should be disclosed in the main text.\n\nThird, the AgentGym baseline provenance is shaky. The paper says they reproduce AgentGym under LLaMA3, then later says AgentGym is not open-sourced and they only report the result from the AgentGen paper. Those two statements don't square.\n\nFourth, Section 8 checks only 50 trajectories and validates GPT-4o's turn-level error labels, not whether the simulated environment feedback resembles real environments. Small sample, and it addresses the wrong question.\n\nNo code or data is released, which makes the confound hard to inspect.\n\nWho is this for? Anyone working on agent instruction tuning or synthetic trajectory data. The recipe is concrete and likely reusable even if the causal claim needs qualification. As a referee, I'd ask for the matched-pair ablation, a non-test-based checkpoint selection, and code/data release. The paper deserves a serious referee, and if it came to me I'd send it back with these requests.","headline":"Plausible recipe, solid held-out gains, but the causal claim that refinement drives the gains is confounded by a regenerated comparison corpus.","tokens_in":20663,"tokens_out":4571,"would_cite":true,"duration_ms":38310,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training LLM agents on synthetic trajectories that include deliberate mistakes and subsequent refinements makes them generalize to new agent tasks better than prior agent-tuning methods.","keywords":["agent generalization","refinement tuning","synthetic agent data","instruction tuning","LLM agents","self-refinement","ReAct","AgentBoard"],"falsifier":"Take the AgentRefine synthetic corpus and flip or corrupt a fraction of the error labels, for example marking 20% of correct turns as errors and vice versa, while keeping trajectory diversity identical; if held-out success does not fall as corruption increases, then refinement labels are not what drives the gains, and the argument would instead point to trajectory diversity or formatting.","tokens_in":19686,"feed_emoji":"🤖","tokens_out":5521,"duration_ms":51615,"temperature":0.7,"pith_summary":"The paper argues that open-weights LLM agents fail to generalize across agent environments because standard agent-tuning data only teaches them to memorize observation-action associations; once the environment changes, they repeat errors and never recover. AgentRefine is proposed as a fix: generate a large, persona-driven variety of synthetic environments and tasks, have a strong LLM act out trajectories that contain deliberate mistakes followed by refinements, and fine-tune the agent on those trajectories while masking the loss on the erroneous turns. The resulting model, trained only on synthetic data, improves held-out success and progress rates on five decision-making benchmarks compared to prior agent-tuning methods and stays stable when the action vocabulary of a training environment is perturbed. The paper claims this shows a causal connection between self-refinement during training and generalization at inference.","feed_headline":"Training agents to fix their own mistakes boosts generalization","feed_subtitle":"Synthetic error-and-refine trajectories let open LLM agents beat prior tuning methods on unseen tasks.","key_machinery":"The core mechanism is the constructed refinement trajectory plus a modified training loss. Data is built in three stages: an LLM generates an environment script (locations, items, goal, available actions with regex validation) from a sampled persona; the same LLM plays both Dungeon Master and player to produce multi-turn ReAct-style trajectories in which the player sometimes acts wrongly and the DM returns feedback; and a verifier checks the script and trajectory, requiring at least two error-refine pairs before a successful finish. Training then uses Equation 1, which computes the log-likelihood of each turn's thought and action conditioned on prior turns but multiplies by an indicator $\\mathbb{1}(A_j)$ that is 1 only when the action is correct, so erroneous tokens are never learned. This is what makes refinement tuning distinct from ordinary supervised fine-tuning on agent trajectories.","core_discovery":"AgentRefine's central claim is that refinement tuning, not just more or more diverse agent data, is what carries generalization. On trajectories synthesized by GPT-4o, each player turn is followed by a Dungeon Master observation, and a verifier keeps only trajectories containing at least two error-refinement cycles before a successful finish. Training on these trajectories with the erroneous turns' tokens masked out yields models (LLaMA-3-8B/70B, Mistral-7B) that outperform Agent-FLAN, AgentGym, and AgentGen on held-out AgentBoard tasks, for example leading Agent-FLAN by 13.3% success on SciWorld, and that also transfer to the HotpotQA reasoning task. Ablations show that removing or masking the refinement loss cuts held-out performance sharply, while including loss on erroneous tokens can cost roughly 75% of SciWorld success. The paper positions this as establishing a link between step-level self-refinement and agent generalization.","pith_inferences":["If refinement is the true driver, then data diversity alone is not enough; this suggests a testable recipe of deliberately injecting controlled failures into any synthetic or collected agent corpus.","The loss masking turns each error-refine cycle into something close to implicit preference data; one could extract pairs of erroneous versus corrected turns and train with direct preference optimization or reinforcement learning, which the paper does not do.","The fidelity of GPT-4o's simulated feedback is the critical transfer assumption; an extension would corrupt a fraction of error labels during synthesis and measure how held-out performance degrades, separating refinement learning from mere trajectory diversity.","Because gains appear on text-based simulation benchmarks, testing on web navigation or tool use would show whether the refinement habit transfers across environment types."],"forward_implications":["Trained on synthetic data with no overlap with the five test environments, AgentRefine improves held-out success and progress rates on SciWorld, BabyAI, PDDL, and Jericho, and on Alfworld when no training data came from it.","Because refinement trajectories are the key ingredient, other agent-tuning pipelines could be upgraded by inserting error-and-refine cycles into their data and masking erroneous turns, without changing the model architecture.","Models become resistant to small environment perturbations: while Agent-FLAN and AgentGym drop sharply in success when action wording is reordered, AgentRefine's average success rate stays flat or slightly increases.","The same recipe transfers to a reasoning task such as HotpotQA, suggesting refinement tuning teaches general recovery behavior rather than task-specific exploration."],"supporting_citations":[{"why":"Supplies the billion-scale persona pool used to sample diverse environments and tasks for synthetic data construction.","marker":"(Chan et al., 2024)"},{"why":"Provides Agent-FLAN, the primary baseline whose held-in versus held-out gap and error-stuck behavior motivate the paper.","marker":"(Chen et al., 2024)"},{"why":"Provides AgentGen, the synthetic-environment baseline that AgentRefine must beat on held-out generalization.","marker":"(Hu et al., 2024)"},{"why":"Supplies the self-refine concept that the paper adapts into step-level refinement training.","marker":"(Madaan et al., 2024)"},{"why":"Offers Reflexion, the verbal-reinforcement method used in an additional experiment and an inspiration for refinement.","marker":"(Shinn et al., 2024)"},{"why":"Defines the ReAct thought-action-observation format used for trajectory generation, training, and inference.","marker":"(Yao et al., 2022)"},{"why":"Provides AgentBoard, the evaluation framework and the five tasks that define held-out generalization and progress rates.","marker":"(Ma et al., 2024)"},{"why":"Supplies the LLaMA-3 base models on which most refinement-tuning experiments are run.","marker":"(Dubey et al., 2024)"},{"why":"Provides HotpotQA, the reasoning task used to test whether refinement tuning transfers beyond decision-making environments.","marker":"(Yang et al., 2018)"}],"fun_headline_variants":["Self-correcting agents beat prior tuning on unseen tasks","Training agents to fix their own mistakes boosts generalization","Refinement tuning: agents that correct themselves generalize","AgentRefine: teach agents to self-correct for better transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training signal assumes that GPT-4o's simulated environment feedback and error labels are faithful proxies for real environment feedback, so that a model learning to refine on synthetic feedback will also refine well in real environments; this fidelity is checked on only 50 judged trajectories in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Self-correcting agents beat prior tuning on unseen tasks","Training agents to fix their own mistakes boosts generalization","Refinement tuning: agents that correct themselves generalize","AgentRefine: teach agents to self-correct for better transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1303,"prompt_tokens":991,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":247}},"tokens_in":607,"tokens_out":312,"duration_ms":3574,"temperature":1.0,"reasoning_tokens":247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:21:21.174243+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the AgentRefine synthetic corpus and flip or corrupt a fraction of the error labels, for example marking 20% of correct turns as errors and vice versa, while keeping trajectory diversity identical; if held-out success does not fall as corruption increases, then refinement labels are not what drives the gains, and the argument would instead point to trajectory diversity or formatting.","supporting_citations":[],"review_version":1}