{"id":"0b4afb9a-c593-4bcb-93d4-2cfc886a75f8","arxiv_id":"2507.14897","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AgentFly enables multi-turn RL training for LM agents with token masking, asynchronous tool calls, and pooled environment management, demonstrated on six tasks.","lead":"AgentFly is a reinforcement learning framework for training language-model agents that call tools and interact with environments, built on top of the verl RL stack. It masks environment-provided tokens during training, runs tool calls and rewards asynchronously, and manages pools of isolated environment instances.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training reward curves alone do not demonstrate genuine agent learning; final task success and baseline comparisons are missing.","rationale":"The Reader's verdict is CONDITIONAL and already flags the lack of final task success metrics, baselines, and error bars. I agree with that overall assessment. However, the Reader's designated weakest assumption is the token-level masking mechanism in Section 2.2. In my reading, the mask is a standard way to restrict the policy gradient to LM-generated tokens, and while it deserves ablative validation, it is not the main load-bearing risk. The more decisive gap is empirical: the paper's effectiveness claim rests entirely on training reward curves, which can increase due to reward hacking, format rewards, partial subgoal rewards, or simple memorization. A concrete held-out evaluation would settle whether the proposed training actually improves task competence. If such evaluation shows large gains, the central claim would hold; if it shows no gains, the framework's value reduces to an engineering contribution. Thus the verdict remains CONDITIONAL pending this evidence, so no adjustment to the Reader's verdict is needed.","tokens_in":622,"tokens_out":4211,"duration_ms":139780,"concrete_test":"Run the released AgentFly code to train one environment (e.g., Code Interpreter or ALFWorld) to completion; then evaluate the final checkpoint and the initial Instruct checkpoint on the standard held-out/test split of that environment, reporting exact task success or answer accuracy. If final success is not substantially above the initial checkpoint and above a no-training baseline, the reward curves in Figures 3 and 4 do not establish 'successful agent training.' For a stronger test, also report per-task success curves and compare against a single-turn or no-mask variant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AgentFly enables effective Agent-RL training, but the only empirical support is training reward curves in Figures 3, 4, and 6. These curves are computed on the training rollouts and can rise without improving task competence. For example, the Code Interpreter reward includes a 0.1 format reward for calling a tool, so the agent can increase reward simply by learning to call the interpreter more often even if final answers remain wrong. For ScienceWorld the trajectory reward is defined as the highest subgoal reward achieved at any point, so curves can reflect partial subgoal progress rather than successful task completion. Moreover, no final accuracy, task-success rate, held-out evaluation, or baseline comparison is reported; even the initial checkpoint's reward is not shown. Therefore the crucial condition for the central claim, namely that the reported improvements reflect genuine agent learning, is unverified. This is a missing-support problem, not an internal inconsistency, but it is load-bearing because every downstream claim about scalability and extensibility rests on this evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AgentFly, a reinforcement learning (RL) framework for training language model (LM) agents, built on top of the verl training infrastructure. The framework introduces token-level masking to adapt single-turn RL losses (PPO, REINFORCE++, GRPO, RLOO) to multi-turn agent trajectories, a tool abstraction with stateful and non-stateful tools, asynchronous chain-based rollouts, and a centralized environment pool for scalable environment coordination. The authors implement decorator-based tool and reward definitions and provide prebuilt tools/environments including a code interpreter, Search, Retrieve, ScienceWorld, WebShop, and ALFWorld. Experiments train Qwen2.5-Instruct 3B and 7B models on these tasks, reporting reward curves that increase during training. The paper claims that the framework is scalable, extensible, and effective for agent training across diverse tasks.","tokens_in":11307,"tokens_out":4796,"duration_ms":49689,"significance":"If the empirical claims are substantiated, AgentFly would be a useful open-source infrastructure contribution to the emerging area of Agent-RL, addressing real engineering challenges: multi-turn optimization, asynchronous tool execution, environment pooling, and modular tool/reward definitions. The code release and the breadth of integrated environments are concrete strengths. However, the current evidence falls short of demonstrating that the framework 'successfully trains' agents: the only empirical support is training reward curves, with no final task-success metrics, no held-out evaluation, no baseline comparisons (e.g., prompt-based ReAct, SFT, or existing agent RL frameworks), and no variance or seed information. Because several reward definitions (e.g., a format reward of 0.1 for any tool call, and using the highest subgoal reward in ScienceWorld) can increase without improving task completion, the training curves do not by themselves establish genuine agent learning. The central claim therefore rests on a load-bearing but unverified empirical foundation.","major_comments":[{"comment":"The only empirical evidence for the central claim of 'successful agent training' is training reward curves computed on training rollouts. No final task-success rates, held-out evaluations, or comparisons against untrained/initial checkpoints or baseline methods are reported. This is particularly concerning because several reward designs can increase without improving task competence: in B.1, the code-interpreter reward gives 0.1 just for calling a tool, so an agent can raise reward by learning to invoke the interpreter even if final answers remain wrong; in B.5, the ScienceWorld reward is defined as the highest subgoal reward achieved at any point, so curves reflect partial progress rather than successful task completion. Additionally, Figure 4 shows the ALFWorld reward for the 3B model 'keeps around zero', yet §3.2 states that 'both the 3B and 7B models can be effectively trained'. The abstract's claim of demonstrated effectiveness is therefore not supported by the presented metrics.","section":"§3, §3.2, Figures 4 and 6, Appendix B"},{"comment":"The comparison of PPO, REINFORCE++, GRPO, and RLOO reports only reward curves with no variance, no number of seeds, no final reward values, and no statistical significance. The observation that 'REINFORCE++ shows slightly lower performance' is followed by a speculation that masking the advantage function is the cause, but no ablation isolates the effect of masking. Without controlling for hyperparameters or reporting run-to-run variance, the algorithm comparison is not informative, and the masking hypothesis remains untested.","section":"§3.1, Figure 3"},{"comment":"The multi-turn masking adaptation is the core algorithmic contribution, but its correctness is asserted rather than analyzed or ablated. The paper does not specify how advantages are computed over multi-turn trajectories (e.g., whether a value network sees the full context including observations, and whether GAE is computed over all tokens or only LM-generated tokens). The claim that masking 'ensures that the model learns only from its own outputs' is plausible, but the potential effect on credit assignment across turns is not examined. Given that the paper itself attributes the REINFORCE++ result to this masking, an ablation or at least a formal description of the advantage computation is needed to establish that the update optimizes the intended objective.","section":"§2.2, Eq. (1)"}],"minor_comments":[{"comment":"The notation in Eq. (1) is ambiguous: 'Mt = 1 if at ∈ r1' uses set membership for a token in a response, but it is not defined how token indices map to turns, nor whether the context st includes prior observations. Please clarify the notation and define the multi-turn context.","section":"§2.2, Eq. (1)"},{"comment":"The sentence 'We evaluate four RL algorithms: [11], REINFORCE++ [6], GRPO [12], and RLOO [1]' should spell out 'PPO' instead of citing the reference number alone.","section":"§3.1"},{"comment":"The legend contains a typo: 'REINFOCE++' should be 'REINFORCE++'.","section":"Figure 3"},{"comment":"The four subplots have no y-axis labels; the caption lists the metrics (avg_turns, hallucination, alfworld_admissible_commands, alfworld_step), but the plots themselves should be labeled for readability.","section":"Figure 5"},{"comment":"The reward description 'if the model calls at least one tool, we give it a format reward 0.1, if it further gets the answer correct, we give it a reward of 1.0' is unclear about whether the rewards are additive or whether 1.0 replaces 0.1; please specify the exact reward-scale logic.","section":"Appendix B.1"},{"comment":"The code snippet '*, reward_val, *, * = await env.step(\"\")' is not valid Python; please provide a correct unpacking or illustrate the intended interface in pseudocode.","section":"Appendix B.3"},{"comment":"Several references are incomplete or garbled, notably [6] (the REINFORCE++ citation) and [15] (RL-Factory, listed as a bare GitHub URL with no author or year); these should be cleaned up before publication.","section":"References"},{"comment":"The queueing behavior when no environment instance is available is described, but it is unclear whether requests can deadlock if all environments are occupied and a chain times out; please clarify the timeout and release policy.","section":"§2.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems contribution, and the open-source code plus the broad suite of integrated environments are valuable. The main risk is the thin empirical validation: training reward curves without baselines, final metrics, or variance cannot support the central claim of successful agent training. This is fixable within the manuscript's scope if the authors add held-out success rates, baseline comparisons, and seed information. I would also encourage them to address the ambiguity around the multi-turn masking and its effect on credit assignment, as that is central to the method's correctness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey [Colleague],\n\nRead AgentFly. Bottom line: decent systems paper for RL training of LM agents, with a clean design and a real code release, but the evaluation is too thin to support the 'successful agent training' headline. The stress-test note is right—reward curves on training rollouts are not competence.\n\nWhat's actually new: the unified stateful/non-stateful tool abstraction plus the pool-based environment manager is a genuinely useful engineering idea. The decorator API makes it easy to add tools/rewards, which addresses a real blocker for people wanting to do agent RL. Async tool calls and rewards, and the chain-based rollout, look sensible. They integrate four RL algorithms (PPO, REINFORCE++, GRPO, RLOO) and run six tasks including ALFWorld, WebShop, ScienceWorld. The code is on GitHub, which matters.\n\nWhere it's soft: the empirical section is mostly reward curves. No baselines, no final success rates, no held-out eval, no seeds or error bars. The stress-test point lands: with the code interpreter, there's a 0.1 format reward just for calling a tool, so the curve can rise from learning to call tools more often even if answers stay wrong. For ScienceWorld they use the highest subgoal reward, so partial progress inflates the curve. You can't tell from these figures whether the agent actually completes tasks. The REINFORCE++ observation (slightly lower) is a hypothesis about masking, no ablation. The masking itself is a standard PPO adaptation, fine, but they don't test it.\n\nThis is a missing-support problem, not a fundamental flaw. The framework is plausible and the engineering is real. But the central claim needs final metrics and at least some baseline comparison (e.g., untrained model or SFT). The analysis of ALFWorld tool-call statistics is nice and suggests genuine learning behavior, but it's anecdotal.\n\nThe citation pattern is fine—they position against verl, RAGEN, RL-Factory, verl-agent, and AReal, and don't overclaim novelty. One or two of their own prior tools (ToolGen, ScienceWorld) appear, but appropriately.\n\nWho's this for: people building or using agent RL frameworks, especially those hitting multi-turn and tool-parallelism issues. It deserves a serious referee, but only if the authors add evaluation. As is, I'd tell the authors to rerun with success rates, a baseline, and some variance. I'd bring it to reading group—the evaluation gap is a good discussion prompt.\n\nRecommendation: send to peer review, but expect major revision on experiments.","headline":"Useful RL-for-agents engineering with a genuinely clean design and code, but the empirical evidence is training reward curves only—so the 'effectiveness' claim is unproven.","tokens_in":11836,"tokens_out":2299,"would_cite":false,"duration_ms":22436,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgentFly trains language-model agents with multi-turn RL by masking environment tokens, achieving reward gains on six tool-use environments.","keywords":["Agent-RL","reinforcement learning","language model agents","multi-turn training","token-level masking","tool use","scalable framework"],"falsifier":"Re-run AgentFly's WebShop or ScienceWorld training for the 7B model with the same hyperparameters and check whether the reward curve rises above the untrained instruct-model baseline; if it does not, the framework's effectiveness claim fails. To isolate the mask, run a second ablation without the token mask (applying loss over all trajectory tokens): if the unmasked run matches or beats the masked one on a short-horizon task like code-interpreter math, then the mask is not the component doing the work.","tokens_in":10958,"feed_emoji":"🤖","tokens_out":8964,"duration_ms":90687,"temperature":0.7,"pith_summary":"The paper introduces AgentFly, a framework for training language-model agents with reinforcement learning over multi-turn interactions with tools, APIs, and environments. Its central technical move is to apply the policy-gradient loss and advantages only to the language model's own output tokens, masking out all tokens produced by the environment or tools. This converts single-turn RL algorithms like PPO, GRPO, REINFORCE++, and RLOO into multi-turn agent trainers without altering the underlying objective. Across six environments—code interpretation, web search, Wikipedia retrieval, WebShop, ALFWorld, and ScienceWorld—the authors report that both 3B and 7B instruction-tuned models show rising reward curves, and they document that learning manifests as better tool use, such as querying admissible commands rather than guessing actions. The broader point is that agent RL can be supported by a single extensible and scalable pipeline rather than bespoke per-task systems.","feed_headline":"Multi-turn RL training lifts six tool-use tasks for LM agents","feed_subtitle":"AgentFly masks environment tokens so agents learn from their own outputs; rewards rise on code, search, and embodied tasks.","key_machinery":"The load-bearing mechanism is the token mask M_t, which equals 1 for tokens in the LM's own responses and 0 for environment/tool tokens, applied to both the loss and the advantage in multi-turn trajectories. This adapts the single-turn PPO objective to arbitrary turn counts without changing the algorithm's math. The second pillar is the tool abstraction: every external interface (function, API, environment) is a tool, and stateful tools are bound to isolated environment instances drawn from a central resource pool, so rollouts can run asynchronously at scale. The mask supplies the learning-theoretic fix; the asynchronous tool/environment system supplies the scalability.","core_discovery":"The central claim is that with token-level masking, ordinary single-turn RL algorithms can optimize language-model agents in multi-turn settings where the model alternates between generating responses and receiving tool observations. For a trajectory (p, (r1, o1), ..., (rk, ok)), the mask M_t is 1 when the token at_t belongs to one of the model's responses r_i and 0 otherwise, and this mask is applied to both the PPO-style loss and the advantage estimate. The authors argue this prevents the model from being penalized or credited for tokens it did not generate. Empirically, they show that all four algorithms converge to similar rewards on a code-interpreter math task, that reward increases across all six environments, that larger models benefit more, and that RL changes behavior: the ALFWorld agent learns to use 'get admissible commands' and 'get task objective' tools, and hallucination (malformed tool calls) decreases over training. The discovery, on the paper's own terms, is that a masked multi-turn objective plus scalable environment management is sufficient to train capable agent policies from an instruct-tuned base.","pith_inferences":["Masking out tool tokens may weaken long-horizon credit assignment because the advantage reflects only the final outcome; coupling the mask with a learned value function or process rewards on tool feedback could stabilise long trajectories, a testable modification of the paper's scheme.","The same mask-and-tool abstraction could be used for multi-agent training, where each agent's tokens are masked independently and shared environments are handled by the central pool.","The caching of search queries in the Retrieve environment changes the reward landscape over training; one could test whether the agent learns to exploit the cache by rephrasing queries, which would be an emergent, potentially unintended behavior.","The 3B model's failure on ALFWorld under the same framework suggests that a minimum base-model capability is needed for long-horizon tasks; scaling the same setup to larger instruction-tuned models is a natural next experiment."],"forward_implications":["Standard on-policy RL algorithms can be applied to multi-turn agent training without a new objective; only a mask selecting the LM's own tokens is needed.","New environments and rewards plug in with a decorator, so a developer can go from an environment wrapper to a training run with a small amount of code.","Rollout throughput scales with the number of environment instances in the pool, which is the key to making online agent RL feasible at large batch sizes.","The four compared algorithms (PPO, GRPO, REINFORCE++, RLOO) reach similar reward levels on the code task, suggesting the framework is algorithm-agnostic and the gain comes from the masked multi-turn setup.","Increasing the maximum number of turns improves final reward on the tested task but makes both reward and gradient norm fluctuate more, indicating a stability/expressiveness trade-off in rollout length."],"supporting_citations":[{"why":"The open-source RL training engine on which AgentFly's training half is built; supplies distributed PPO/GRPO-style infrastructure.","marker":"[13]"},{"why":"Defines the PPO loss that the token-level mask adapts from single-turn to multi-turn trajectories.","marker":"[11]"},{"why":"GRPO, one of the four algorithms integrated and used for several environment tasks.","marker":"[12]"},{"why":"REINFORCE++, integrated and compared; the paper observes its masked variant is slightly weaker, which motivates the masking discussion.","marker":"[6]"},{"why":"RLOO, integrated and compared on the code-interpreter math task.","marker":"[1]"},{"why":"Search-R1's data and retriever supply the Retrieve tool used in the search experiments.","marker":"[7]"},{"why":"ALFWorld environment, a long-horizon text-based testbed where masking and tool learning are demonstrated.","marker":"[14]"},{"why":"WebShop environment, the e-commerce navigation testbed.","marker":"[22]"},{"why":"ScienceWorld environment, the multi-subgoal science experiment testbed.","marker":"[18]"}],"fun_headline_variants":["Token masking makes RL work for multi-turn LM agents","AgentFly: RL for agents via masked multi-turn training","Scalable RL framework trains LM agents across six tasks","Masked multi-turn RL lifts tool-use performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that computing the policy-gradient loss and advantages only on the language model's generated tokens, and ignoring tokens emitted by tools and environments, yields a correct and stable reinforcement-learning update for multi-turn agent trajectories.","fun_headline_variants_meta":{"raw":{"variants":["Token masking makes RL work for multi-turn LM agents","AgentFly: RL for agents via masked multi-turn training","Scalable RL framework trains LM agents across six tasks","Masked multi-turn RL lifts tool-use performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001014,"raw_usage":{"total_tokens":4289,"prompt_tokens":961,"completion_tokens":3328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":3265}},"tokens_in":577,"tokens_out":3328,"duration_ms":28013,"temperature":1.0,"reasoning_tokens":3265,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:43:53.446989+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run AgentFly's WebShop or ScienceWorld training for the 7B model with the same hyperparameters and check whether the reward curve rises above the untrained instruct-model baseline; if it does not, the framework's effectiveness claim fails. To isolate the mask, run a second ablation without the token mask (applying loss over all trajectory tokens): if the unmasked run matches or beats the masked one on a short-horizon task like code-interpreter math, then the mask is not the component doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"REINFORCE++, integrated and compared; the paper observes its masked variant is slightly weaker, which motivates the masking discussion."},{"cited_title":"Ahmadian, C","cited_arxiv_id":null,"evidence_quote":"RLOO, integrated and compared on the code-interpreter math task."}],"review_version":1}