RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use

Chengqi Dong; Chuhuai Yue; Guojun Yin; Hang He; Jiajun Chai; Jiwen Jiang; Siyu Xia; Wei Lin; Xiaoguang Li; Xiaohan Wang

REVIEW 4 major objections 4 minor 3 cited by

RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes a plug-and-play RL post-training framework for multi-turn tool use and claims it lets a smaller model beat a larger one while training 6.8x faster.

desk verdict Useful engineering integration for tool-use RL post-training, but the headline performance claim is confounded by a base-model swap and an internal number mismatch. read the letter →

arxiv 2509.06980 v1 pith:IK2E4IUY submitted 2025-08-31 cs.LG cs.AI

Jiajun Chai , Guojun Yin , Zekun Xu , Chuhuai Yue , Yi Jia , Siyu Xia , Xiaohan Wang , Jiwen Jiang

show 4 more authors

Xiaoguang Li Chengqi Dong Hang He Wei Lin

This is my paper

classification cs.LGcs.AI

keywords reinforcementlearningpost-trainingmulti-turntooluseLLMagentsobservationtokensasynchronouscallsrewardcomputationMDPstatereconstructionplug-and-playframework

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reinforcement-learning post-training for multi-turn tool use fails in practice because tool interfaces are heterogeneous, tool calls are slow and serial, and reward needs vary by task. It introduces RLFactory, a plug-and-play framework that decouples tool execution from training, runs tool calls asynchronously, and supports rule-based, model-judge, and tool-verification rewards. The central empirical claim is that a 4B-parameter model post-trained with RLFactory scores 0.486 on the Natural Questions benchmark, beating the 0.473 achieved by a 7B-parameter model trained with the baseline method, while raising training throughput by 6.8x. If correct, this would make multi-turn tool-use RL substantially cheaper and more accessible.

What carries the argument

The central mechanism is the observation token: external tool feedback (search results, code logs, verification outputs) is appended to the interaction trajectory to redefine the Markov decision process state, while being masked out of the model's loss so the model is not trained to imitate environment noise. Around this, RLFactory builds a generate-parse-invoke-update closed loop, with an asyncio-based asynchronous caller that lets multiple tool calls run concurrently instead of blocking the rollout.

What would settle it

Hold the base model fixed and run both the baseline recipe and RLFactory on identical hardware with the same batch size and rollout concurrency: if the Natural Questions gap disappears or the throughput multiplier drops far below 6.8x, the framework-specific claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper discovers that reconstructing the RL state with observation tokens—tool outputs appended to the trajectory but excluded from the loss—together with a generate-parse-invoke-update loop, yields a stable and efficient multi-turn tool-use training framework. Using this design with a 4B-parameter base model, the reported Natural Questions test score is 0.486, above the 0.473 of a larger 7B-parameter model trained with the reference baseline approach, and the training throughput is 6.8 times higher. The paper attributes this to asynchronous parallel tool invocation, a decoupled tool/training architecture that lowers environment setup cost, and a flexible reward layer th

Load-bearing premise

The comparison assumes the baseline run is a fair control, but the paper changes both the training framework and the base model (4B vs 7B) while leaving the baseline's hardware, batch size, and rollout settings unspecified.

Editorial extensions

If this is right

Smaller base models can be post-trained for tool use to match or exceed much larger models, cutting GPU and time budgets for agentic RL.
A 6.8x throughput gain makes multi-round rollout RL experiments practical on smaller clusters.
Observation tokens as state-plus-feedback, excluded from loss, could become a standard design for any agent training that consumes environment feedback.
A decoupled tool/training layer means adding a new tool reduces to a configuration-file change rather than framework code modification.
Supporting rule-based, model-judge, and tool-verification rewards lets a single framework cover verifiable tasks and open-ended tasks without swapping training stacks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the headline comparison conflates framework change with base-model change (4B vs 7B) and does not report the baseline's hardware or rollout settings; a controlled same-base replication would be needed to attribute the score gain to the framework itself.
Editorial extension: the 6.8x speedup likely comes mostly from replacing serial tool calls with asyncio-based parallel invocation; that design is testable independently in any RL rollout loop by comparing serial versus concurrent tool execution.
Editorial extension: observation-token state reconstruction should transfer beyond text search to code execution, image returns, and other multimodal tool feedback, which the paper's roadmap partially anticipates.
Editorial extension: a direct test would apply RLFactory to NL2SQL or code generation with tool-verification rewards to see whether the same advantage appears outside search-heavy benchmarks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a genuine engineering contribution, and the empirical evidence is not adequate to support its main claim. The integration is real: asyncio-based parallel tool calling, decoupled tool/training architecture, three reward types (rule, model-judge, tool-verification), MCP-based tool registration, and observation tokens appended to the MDP state. None of these is new alone, but the combination is a practical, plug-and-play package that lowers the barrier for multi-turn tool-use RL. The writing is clear and honest about scope; it does not claim a new algorithm. The code is released, which matters.

The problem is Table 1. It compares RLFactory+Qwen3-4B against Search-R1+Qwen2.5-7B, changing two variables at once. So the 0.486 vs 0.473 gap could easily come from the base model being stronger at tool use, and the 6.8x throughput 'improvement' could come from training a 4B model instead of a 7B model. There is no fixed-base-model ablation toggling the framework alone. No error bars, one dataset only. And there is an internal inconsistency: Section 3 text reports the Qwen2.5-7B baseline as 0.429, while the abstract and Table 1 say 0.473. That undermines confidence in the reported numbers.

The MDP reconstruction with observation tokens is a reasonable formalism and the loss mask for tool output is a sensible detail. The reward layer is pragmatic. The citation pattern looks fine: veRL, Search-R1, Qwen-Agent are the right prior work; no sign of self-citation inflation.

Who it is for: anyone building RL post-training for tool-use agents who wants a head start on the engineering. The paper deserves a serious referee—not because the main empirical claim is established, but because the framework is useful and the evidence is weak enough that referee scrutiny should drive it to a controlled comparison. If I were editor, I would send it out and require the authors to (a) fix the 0.429/0.473 inconsistency, (b) run a fixed-base-model ablation, and (c) specify the throughput comparison setup. With those, the paper could be a solid systems contribution.

Referee Report

4 major / 4 minor

Summary. The paper presents RLFactory, an open-source reinforcement learning post-training framework for multi-turn tool use in LLMs. The claimed contributions are architectural: an asyncio-based asynchronous tool caller, a decoupled tool/training design, a multi-type reward layer (rule-based, model-judge, tool-verification), and an MDP reconstruction that appends tool feedback as observation tokens. The experimental section reports one main result on the Search-R1/NQ setup: a Qwen3-4B model trained with RLFactory achieves a test score of 0.486, exceeding the 0.473 reported for Search-R1-trained Qwen2.5-7B-Instruct-GRPO, with 6.8x higher training throughput. The paper positions this as evidence that RLFactory gives stronger multi-turn tool-use performance at lower computational cost.

Significance. If the empirical claims were properly supported, RLFactory would be a practically useful engineering contribution: asynchronous tool invocation, decoupled environment/training modules, and flexible reward signaling lower the barrier for RL-based tool-use training. The open-source release is a concrete strength. However, the current manuscript does not establish the central quantitative claim. The headline comparison varies both the base model and the training framework, the baseline score is reported inconsistently, the throughput gain lacks a measurement protocol, and there is only one dataset with no reported variance or ablation. These issues are not cosmetic; they directly affect the paper's main conclusion. The framework design itself is reasonable, but the validation is currently too weak to support the abstract's comparative claims.

major comments (4)

[Section 3, Table 1] The central comparison is confounded. Table 1 compares RLFactory+Qwen3-4B-Instruct-GRPO against Search-R1+Qwen2.5-3B/7B-Instruct-GRPO, changing both the training framework and the base model. The 0.486 vs 0.473 advantage and the 6.8x throughput gain could both be caused by the base-model swap: Qwen3-4B may be intrinsically better at tool use, and a 4B model may train faster than a 7B model on identical hardware. A controlled comparison is required, e.g., Search-R1+Qwen3-4B vs RLFactory+Qwen3-4B, and ideally Qwen2.5-7B with both frameworks. Without this, the framework's specific contribution is unidentified.
[Section 3, text vs. Table 1 / Abstract] There is an internal inconsistency in the reported baseline: Section 3 states that Qwen2.5-7B-Instruct-GRPO scored 0.429 trained with the same Search-R1 technique, while the abstract and Table 1 report 0.473. The difference (0.044) is larger than the claimed performance gap between RLFactory and the baseline (0.013). If 0.429 is the correct number, the central claim that RLFactory outperforms the larger baseline is reversed or at least unsupported. The authors must correct the inconsistency and state which number is the official Search-R1 result.
[Section 3, Table 1 and Figure 5] The experimental evidence is limited to a single dataset (NQ) with no error bars, no multiple seeds, and no statistical significance testing. Figure 5 shows 95% confidence intervals over training runs, but the number of runs and the computation of the intervals are not specified. Reporting a single test score without variance is insufficient to support the claimed 0.013-point improvement, especially given the confounded comparison in the same table.
[Section 3, throughput claim] The 6.8x throughput improvement is reported without any measurement protocol. The paper does not specify the hardware, batch size, rollout count, maximum tool-call limit, async concurrency, or whether the baseline used the same settings. Throughput comparisons in RL training are highly sensitive to these choices; without this information the 6.8x figure is not verifiable. The authors should report the exact configurations for both runs and define the throughput metric (e.g., generated tokens per second).

minor comments (4)

[Eq. (1)] The notation in Eq. (1) is imprecise: it says Rrule is the total reward of the i-th rule, but the left-hand side is the total rule reward over n rules. Also, the meaning of ri(s,a,s') is clear from context but should be formally defined.
[Section 2.3.2 / Figure 4] The 'Generate-Parse-Invoke-Update' flow is clear, but the role of the loss mask is only mentioned in a sentence. Since masking tool-return tokens is an important design choice, a more detailed explanation (e.g., which tokens are masked and why) would help reproducibility.
[References] Reference [11] is cited for veRL, but the reference title is 'HybridFlow: A flexible and efficient RLHF framework'. The authors should verify whether the intended citation is the veRL technical report or the HybridFlow paper, and cite appropriately.
[Figure 5] The caption refers to 'mean Critic scores', but GRPO typically does not use a critic. This may be a terminology error from the underlying verl code and should be corrected to 'reward scores' or similar.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are benchmark measurements and framework descriptions, not derivations that reduce to their own inputs.

full rationale

The paper does not derive any result from fitted parameters: the central performance claim (0.486 on NQ) is an external benchmark evaluation, and the throughput claim is a measured comparison against Search-R1/veRL baselines. The MDP state definition and reward equations (Eqs. 1-3) are explicit design definitions, not results derived from the outcome they are meant to explain. Citations to Search-R1 and veRL are external prior work, not self-citations by the present authors, and no uniqueness theorem or ansatz is imported from the authors' own prior publications. The skeptical concern that Table 1 confounds framework change with base-model change is a legitimate experimental-validity issue, and the 0.429 vs 0.473 discrepancy between Section 3 and Table 1 is a consistency concern, but neither is an instance of the target outcome being built into the input by construction. Accordingly, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard RL assumptions and unspecified experimental choices. No new entities are introduced. The main burden is the unstated hyperparameters and the assumption that NQ is representative of multi-turn tool-use performance.

free parameters (2)

Reward weights w_i in Eq. (1) = unspecified
The rule-based reward uses a weighted sum; weights are not reported for the NQ experiments, and the result may depend on them.
Training hyperparameters (learning rate, batch size, rollout count, max tool calls) = unspecified
These settings affect the reported throughput and final score but are not given.

assumptions (3)

standard math MDP state definition with observation tokens as in Section 2.2
Standard RL formulation applied to tool-use; not proved but commonly accepted.
domain assumption Observation tokens are excluded from the loss (Loss Mask)
The paper asserts this decouples model decisions from environmental feedback; it is a design choice, not derived.
domain assumption NQ test score is a valid proxy for multi-turn tool-use capability
Single benchmark used to support the central claim; no evidence of generalizability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use." pith.science (2026). https://pith.science/paper/IK2E4IUY

@misc{pith2026250906980,
  author       = {Pith},
  title        = {Pith review of: RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IK2E4IUY}},
  note         = {Machine review of arXiv:2509.06980}
}

read the original abstract

Large language models excel at basic reasoning but struggle with tasks that require interaction with external tools. We present RLFactory, a plug-and-play reinforcement learning post-training framework for multi-round tool use. RLFactory tackles (i) tool-call stability and adaptability amid tool heterogeneity and interface issues via an asyncio-based asynchronous caller and a decoupled tool/training architecture, and (ii) diverse evaluation needs via a reward layer supporting rule-based, model-judgment, and tool-verification signals. It reconstructs the MDP by introducing observation markers from tool feedback, closing the loop among model, tools, and environment, and implements a generate-parse-invoke-update workflow for dynamic policy optimization. On Search-R1 with Qwen3-4B, RLFactory achieves a 0.486 test score on the Natural Questions (NQ) dataset, surpassing larger models trained with similar techniques (e.g., Qwen2.5-7B-Instruct-GRPO at 0.473), and increases training throughput by 6.8x. RLFactory provides a low-barrier, highly adaptable framework for strengthening multi-round tool use of LLMs in real-world scenarios. Code: https://github.com/Simple-Efficient/RL-Factory.

Figures

Figures reproduced from arXiv: 2509.06980 by the authors.

**Figure 2.** The framework of RLFactory. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** A diagram of the RLFactory layer structure. The basic layer can be used directly without [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: RLFactory multi-round tool call logic diagram using GPRO as an example. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Mean reward score trends across different base model. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

VistaHop: Benchmarking Long-Horizon Visual DeepSearch
cs.CV 2026-06 conditional novelty 7.0 of 10

VistaHop is a new benchmark of 350 multi-hop visual reasoning tasks where the strongest evaluated model achieves 24.31% Pass@1, revealing limitations in visual grounding and long-chain reasoning.
From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation
cs.LG 2026-07 conditional novelty 6.0 of 10

Replaying teacher trajectory prefixes and explicitly optimizing the historical prefix tokens improves small-model agent success rates over distillation and response-only GRPO baselines in TextCraft, BabyAI, and ALFWorld.
LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services
cs.AI 2025-12 conditional novelty 6.0 of 10

LocalSearchBench—1.3M merchant records and 900 multi-hop local-life QA tasks across 9 Chinese cities—shows the best reasoning agent reaches only 35.6% correctness.

Reference graph

Works this paper leans on

11 extracted references · 6 canonical work pages · cited by 3 Pith papers

[1]

Introducing gpt 5

Openai. Introducing gpt 5. https://openai.com/index/introducing-gpt-5/ , August 2025

work page 2025
[2]

Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving, 2025

Xinji Mai, Haotian Xu, Xing W, Weinong Wang, Jian Hu, Yingying Zhang, and Wenqiang Zhang. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving, 2025

work page 2025
[3]

Agentic reinforced policy optimization, 2025

Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. Agentic reinforced policy optimization, 2025

2025
[4]

Agentic reasoning and tool integration for llms via reinforcement learning, 2025

Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for llms via reinforcement learning, 2025

work page 2025
[5]

Search-o1: Agentic search-enhanced large reasoning models, 2025

Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models, 2025

2025
[6]

Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

work page 2025
[7]

Mmsearch-r1: Incentivizing lmms to search, 2025

Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, and Ziwei Liu. Mmsearch-r1: Incentivizing lmms to search, 2025

work page 2025
[8]

Deepresearcher: Scaling deep research via reinforcement learning in real-world environments, 2025

Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments, 2025

work page 2025

Show all 11 references

[9]

Robust planning with llm-modulo framework: Case study in travel planning

Atharva Gundawar, Mudit Verma, Lin Guan, Karthik Valmeekam, Siddhant Bhambri, and Subbarao Kambhampati. Robust planning with llm-modulo framework: Case study in travel planning. arXiv preprint arXiv:2405.20625, 2024

2024 arXiv
[10]

Tripcraft: A benchmark for spatio-temporally fine grained travel planning

Soumyabrata Chaudhuri, Pranav Purkar, Ritwik Raghav, Shubhojit Mallick, Manish Gupta, Abhik Jana, and Shreya Ghosh. Tripcraft: A benchmark for spatio-temporally fine grained travel planning. arXiv preprint arXiv:2502.20508, 2025

2025 arXiv
[11]

Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024. 11

2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Introducing gpt 5

Openai. Introducing gpt 5. https://openai.com/index/introducing-gpt-5/ , August 2025

work page 2025

[2] [2]

Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving, 2025

Xinji Mai, Haotian Xu, Xing W, Weinong Wang, Jian Hu, Yingying Zhang, and Wenqiang Zhang. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving, 2025

work page 2025

[3] [3]

Agentic reinforced policy optimization, 2025

Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. Agentic reinforced policy optimization, 2025

2025

[4] [4]

Agentic reasoning and tool integration for llms via reinforcement learning, 2025

Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for llms via reinforcement learning, 2025

work page 2025

[5] [5]

Search-o1: Agentic search-enhanced large reasoning models, 2025

Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models, 2025

2025

[6] [6]

Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

work page 2025

[7] [7]

Mmsearch-r1: Incentivizing lmms to search, 2025

Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, and Ziwei Liu. Mmsearch-r1: Incentivizing lmms to search, 2025

work page 2025

[8] [8]

Deepresearcher: Scaling deep research via reinforcement learning in real-world environments, 2025

Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments, 2025

work page 2025