{"id":"7aad1894-b0c5-47bf-a139-d860b795074a","arxiv_id":"2508.18669","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"MUA-RL adds an LLM-simulated user to the RL rollout loop for multi-turn tool use, improving small Qwen3 models on TAU2, BFCL-V3 Multi Turn, and ACEBench Agent.","lead":"This paper trains LLM agents to use tools in multi-turn conversations by adding an LLM-simulated user into the reinforcement learning loop. It reports that the resulting 32B model matches or beats several larger open models on four agentic tool-use benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MUA-RL's reported gains may be specific to the GPT-4o/GPT-4.1 user-simulator family: RL trains against GPT-4o and TAU/ACE evals use GPT-4.1, so no evidence the learned dialogue policy transfers to real users.","rationale":"MUA-RL is a plausible engineering contribution: the cold-start data synthesis pipelines include dual verification and real MCP-server tool execution, the GRPO-based training is described in enough detail to reproduce, and the reported improvements over base and cold-start models are consistent across most benchmarks. Those are real strengths. However, the framework's distinctive element—the LLM-simulated user—is also its most vulnerable assumption. The paper does not characterize the simulator's behavior, does not compare training-simulator outputs to human user behavior, and evaluates on simulators from the same GPT-4 family used during training. This makes it impossible to know whether the learned dialogue strategy is genuinely robust or merely adapted to GPT-4-style user responses. The BFCL result, where the 32B model does not beat DeepSeek-V3-0324, further shows that the headline superiority is not universal and depends on the benchmark family. None of this implies the results are fabricated; it means the central generalization claim is currently unverified. A cross-simulator retraining experiment would settle whether the concern lands. The reader's conditional verdict already covers this need for verification, so no verdict adjustment is required beyond making the simulator-robustness check an explicit acceptance condition.","tokens_in":24745,"tokens_out":8356,"duration_ms":95236,"concrete_test":"Retrain MUA-RL-32B from the same cold-start checkpoint and with the same hyperparameters as Section 4.1, but replace the GPT-4o RL user simulator with a non-OpenAI simulator (e.g., Llama-3.3-70B-Instruct or Qwen3-235B) configured with TAU-Bench user-prompt templates. Evaluate the resulting model on TAU2 Retail/Airline/Telecom and ACEBench using the official harness. If the MUA-RL improvement over the cold-start model is largely preserved (>80%), simulator fidelity is not the driver; if the improvement shrinks substantially or reverses, the reported gains are specific to the GPT-4o user simulator and the central claim would need to be weakened to a benchmark-specific result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the GPT-4o-2024-11-20 user simulator used in RL rollouts (Sections 3.3.2, 4.1) adequately captures real user dialogue behavior. The paper provides no distributional evidence for this: no statistics on clarification requests, refusals, mid-task changes of intent, or comparison of simulated vs. human turns. Because reward is binary terminal success under this simulator, the policy can exploit simulator regularities (e.g., an LLM user that reliably confirms, never abandons the task, or tolerates missing information) without learning robust interaction. The evaluation makes this concern concrete: TAU1/TAU2 and ACEBench use GPT-4.1 as the user simulator—the same OpenAI GPT-4 family as the training simulator—so strong TAU/ACE scores are consistent with overfitting to GPT-4-family dialogue patterns. The only clearly out-of-family benchmark, BFCL-V3 Multi Turn, actually shows MUA-RL-32B (28.4) below DeepSeek-V3-0324 (29.8). Thus the headline claim 'outperforming or matching larger models' rests on in-family simulated-user benchmarks, with no evidence of transfer to real users.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MUA-RL, a multi-turn user-interacting agent reinforcement learning framework for tool use. Starting from Qwen3-8B/14B/32B non-thinking models, the authors first perform a cold-start SFT on synthetic tool-use trajectories, then run GRPO with a GPT-4o-simulated user during rollouts and real-time tool execution. Rewards are terminal and binary (task completion only). Evaluations on TAU1-Bench, TAU2-Bench, BFCL-V3 Multi Turn, and ACEBench Agent report improvements over the corresponding base and cold-start models, and the abstract claims the 32B model outperforms or matches much larger open models such as DeepSeek-V3-0324 and Qwen3-235B-A22B. The paper also includes training-dynamics analysis, an ablation study, and plans to release code, data, and model weights.","tokens_in":25123,"tokens_out":8033,"duration_ms":87122,"significance":"If the claims hold, MUA-RL is a practical recipe: injecting an LLM-simulated user into GRPO rollouts with real tool execution improves tool-use agents across three model scales, and the controlled comparison against base/cold-start models is informative. The planned open-source release of code, data, and weights, plus the use of official evaluation harnesses, are clear strengths. However, the headline claim of outperforming larger open models is currently oversold: the abstract's Telecom number is below the stated baseline, the TAU1 evaluation may use training tasks, BFCL gains are not monotone across scales, and all user-interaction evidence comes from the GPT-4 family of simulators. The contribution is therefore plausible but not yet established at the level claimed.","major_comments":[{"comment":"RL training uses \"115 retail and 50 airline datasets from TAU1-Bench,\" and Table 1 reports TAU1-Bench results, but the paper never states that the evaluation tasks are disjoint from the training tasks. If the same tasks are used for GRPO training and evaluation, the TAU1 Retail/Airline comparisons against baselines that never trained on those tasks are not apples-to-apples. Please evaluate on a held-out TAU1 split, or remove/adjust the TAU1 comparisons, and state the split explicitly.","section":"§4.1, Table 1"},{"comment":"The abstract claims MUA-RL-32B achieves 28.3 on TAU2 Telecom, \"outperforming or matching\" DeepSeek-V3-0324, but Table 1 lists DeepSeek-V3-0324 at 32.9. The 32B model is 4.6 points lower. The claim is only true for MUA-RL-14B (33.4), not for the 32B model highlighted in the abstract. This overstatement must be corrected.","section":"Abstract, Table 1"},{"comment":"The text says the MUA-RL series show \"steady performance gains across all model scales\" on BFCL-V3 Multi Turn, but MUA-RL-8B's Overall Acc is 14.6 versus 15.4 for Qwen3-8B Cold-start. On BFCL Base, MUA-RL-8B (21.0) also drops below cold-start (24.0). The claim of monotone gains is contradicted; the scale-dependence should be qualified.","section":"§4.2.2, Table 2"},{"comment":"The trained policy interacts with GPT-4o-2024-11-20 during RL and with GPT-4.1 during TAU/ACEBench evaluation; no non-GPT-4, out-of-family user simulator or human-user evaluation is provided. The paper also gives no distributional analysis of the simulated user (clarification requests, refusals, mid-task changes of intent). Since the reward is terminal success under this simulator, the benchmark gains could reflect adaptation to GPT-4-family dialogue patterns rather than robust multi-turn user interaction. Please narrow the claims or add evidence of transfer.","section":"§3.3.2, §4.1"},{"comment":"Quantitative comparisons lack uncertainty estimates. Section 4.1 says four repeated tests were averaged, but no standard deviations, confidence intervals, or significance tests are reported. Many headline comparisons are small (e.g., TAU2 Retail 67.3 vs 64.7 for DeepSeek-V3-0324; ACEBench 82.5 vs 74.2) and some test sets are small (50 tasks for airline, 50 for ACEBench). Please report per-run variance or otherwise justify that the differences are not noise.","section":"§4.1"}],"minor_comments":[{"comment":"The keywords contain the typo \"Reinforcement Learing.\" Equation (1) has stray LaTeX \"| {z }\" markup and is not readable as typeset.","section":"Keywords, Eq. (1)"},{"comment":"The parenthetical notation in Table 3 should be defined directly in the caption. The row labels \"+ MUA-RLw/oRL\" and \"+ MUA-RLw/ocold-start\" are confusing because only the latter is an actual RL variant in the usual sense; the former is the cold-start model.","section":"Table 3"},{"comment":"The text says evaluations are \"deterministic\" with temperature 0.0 but then averages four repeated tests \"to improve confidence.\" If decoding is deterministic and the user simulator is deterministic, repeated runs add no information; if the user or environment is stochastic, the evaluation is not fully deterministic. Please clarify.","section":"§4.1"},{"comment":"The conclusion says the framework enables models to handle \"dynamic, user-driven interactions,\" but the paper contains no discussion of the simulator-transfer limitation. A limitations paragraph explicitly noting that all user interactions are LLM-simulated would help readers calibrate the claims.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: MUA-RL is a genuinely new recipe — cold-start SFT on synthetic tool-use trajectories, then GRPO with a GPT-4o-simulated user in the rollout loop — and it moves the needle on multi-turn tool-use benchmarks at 8B–32B scale. The gains over cold-start baselines are consistent and sometimes large. But the headline claim of beating DeepSeek-V3-0324 rests on benchmarks where the user simulator is also GPT-4-family, and the paper does not show transfer to real users. Treat the results as promising but not settled.\n\nWhat's new: nobody in the cited literature puts an LLM-simulated user inside the RL rollouts for agentic tool use. The paper does it cleanly: binary task-completion reward, loss masking on user/tool tokens, and a real database/MCP environment for tool execution. That is a real contribution for the agentic RL crowd. The training-dynamics analysis (KL, entropy, rollout turns, tool-invocation trends) is a plus; it shows the improvement comes from more structured interaction, not longer responses.\n\nWhere it's soft:\n- The simulator-transfer issue is the main one. RL trains against GPT-4o-2024-11-20; TAU and ACEBench evaluate with GPT-4.1 as the user. Same family, so those numbers likely overstate what a real user would see. The only out-of-family benchmark is BFCL-V3, and there MUA-RL-32B is below DeepSeek-V3-0324 (28.4 vs 29.8) and only 2.4 points above its own cold-start. That is the honest counterpoint to the abstract.\n- No error bars. Four repeated runs averaged; no variance or significance. Some gaps (e.g., BFCL 28.4 vs 29.8) are within plausible noise.\n- The abstract says “outperforming or matching” DeepSeek-V3-0324, but on TAU2 Telecom it is 28.3 vs 32.9, and on BFCL it is lower. The headline is a bit generous.\n- The reward simplification during training (removing dialogue content requirements) is reasonable but should be stated more prominently; it means the RL policy is optimized for task completion, not for mentioning required info, and the eval still checks the latter. That mismatch could explain part of the TAU gains.\n\nWho this is for: anyone building RL training loops for tool-using agents. The paper deserves referee time — it is a serious engineering attempt, with a clear setup and a reproducible recipe promised. I would send it to review, with the expectation that the authors add error bars, clarify the simulator caveat, and ideally show one out-of-family or human-in-the-loop result.","headline":"MUA-RL is a genuinely new RL-for-agentic-tool-use recipe, but its headline gains are measured against GPT-4-family user simulators, so transfer to real users is unproven.","tokens_in":25623,"tokens_out":2723,"would_cite":true,"duration_ms":27923,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 32B model trained with a simulated user beats far larger rivals","keywords":["reinforcement learning","LLM agents","tool use","multi-turn interaction","simulated user","GRPO","function calling","agentic AI"],"falsifier":"Run MUA-RL with a differently constructed user simulator (e.g., one that never asks follow-up questions, or one that randomly alters requests mid-task) and measure whether the benchmark gains disappear or reverse; if they do, the reported improvements are tied to the specific simulator rather than to dynamic user interaction in general. A stronger version would evaluate the trained agents with real human users on the same retail and airline tasks.","tokens_in":24694,"feed_emoji":"🤖","tokens_out":3448,"duration_ms":37638,"temperature":0.7,"pith_summary":"MUA-RL claims that putting an LLM-simulated user inside the reinforcement-learning loop, rather than relying on fixed pre-scripted queries, lets a model learn to converse and call tools in the same dynamic way real users demand. The paper reports that a 32B-parameter model trained this way matches or outperforms much larger open models such as DeepSeek-V3-0324 and Qwen3-235B-A22B across four multi-turn tool-use benchmarks. The key move is to reward only final task completion, not intermediate tool-call formats, so the agent freely explores how to clarify, confirm, and correct itself while using real database tools. If the claim holds, it means the missing ingredient in prior agentic RL was not a better reward function or more compute, but a living conversation partner during training.","feed_headline":"A 32B model trained with a simulated user beats far larger rivals","feed_subtitle":"Putting an LLM-simulated customer inside RL led a 32B model to match or beat 3x-larger open models on agentic tool use.","key_machinery":"The central mechanism is the multi-turn user-interacting rollout: during each RL step, the policy samples a conversation with an LLM-simulated user, interleaving text clarification, tool invocation, and real database responses, while only the final binary task-completion outcome feeds a group-relative advantage signal (GRPO). The simulated user is what makes the optimization genuinely interactive, allowing the agent to learn when to ask for information, when to confirm changes, and when to call tools, rather than following a static script.","core_discovery":"MUA-RL is a reinforcement-learning framework that, for the first time in agentic tool use, integrates an LLM-simulated user into the RL rollout loop. During training, the policy model converses with a GPT-4o-simulated user, invokes tools against a real operational database, and receives a reward of 1 only if the entire task is ultimately completed, regardless of the exact dialogue path or tool sequence. The paper shows that this dynamic interaction, combined with a lightweight supervised cold-start phase and GRPO, produces models that generalize across benchmarks they were not trained on: MUA-RL-32B achieves 67.3 on TAU2 Retail, 45.4 on TAU2 Airline, 28.3 on TAU2 Telecom, 28.4 on BFCL-V3 Mul","pith_inferences":["The strongest untested implication is that the simulated user transfers to real humans: if a human study shows similar gains, then simulator fidelity is not the bottleneck; if not, the benchmark results may partly reflect simulator-specific dialogue patterns.","A cheap and concrete extension would be to train with a deliberately weaker or narrower user simulator and measure how performance degrades; this would isolate how much of the benefit comes from user diversity versus user quality.","The binary-reward design implies the agent learns clarification behavior implicitly, because asking the right questions is the only way to reach the terminal success state; this suggests a general recipe for teaching agents soft skills without hand-crafting rewards for them.","The same framework should be testable in dual-control environments beyond telecom, where both the user and the agent can call tools, which is where the paper reports its strongest relative gains."],"forward_implications":["A 32B model trained with MUA-RL can match or exceed the non-thinking performance of models several times larger, suggesting that interaction strategy matters more than raw parameter count for agentic tool use.","Both the cold-start phase and the RL phase are necessary; ablations removing either one degrade performance, so the method is a pipeline rather than a single trick.","Training dynamics show gains come from more structured multi-turn interactions (about 21-23 turns), not from longer responses, so the improvement mechanism differs from test-time scaling in reasoning models.","The model learns to reduce reliance on auxiliary tools like Calculate, Think, and Transfer to Human Agent, implying RL can prune unnecessary tool calls and produce more efficient policies.","The framework scales across 8B, 14B, and 32B backbones, with larger models showing smoother training and more consistent gains, suggesting the approach is not tied to one model size."],"supporting_citations":[{"why":"Provides the TAU1-Bench retail and airline tasks used as RL training data, the benchmark that motivated the reward simplification, and the first evaluation set.","marker":"[39]"},{"why":"Supplies the TAU2-Bench evaluation sets, including the dual-control telecom domain where the agent and user can both call tools.","marker":"[41]"},{"why":"Supplies the BFCL-V3 Multi Turn evaluation set with base, miss-param, miss-func, and long-context subsets.","marker":"[42]"},{"why":"Supplies the ACEBench Agent evaluation set covering real-world multi-step and multi-turn tool-use scenarios.","marker":"[43]"},{"why":"Supplies the GRPO algorithm that the framework uses for group-relative advantage estimation and policy optimization.","marker":"[2]"},{"why":"Supplies the GPT-4o model used as the LLM user simulator during RL rollouts, and also serves as a strong baseline in evaluations.","marker":"[40]"},{"why":"Supplies the Qwen3-8B/14B/32B backbone models used for cold-start and RL training.","marker":"[36]"}],"fun_headline_variants":["Simulated user in RL loop lets 32B match 235B models","First RL for agentic tool use with simulated users in loop","32B model trained with simulated user beats 3x-larger rivals","Dynamic user simulation in RL boosts 32B agentic tool use","MUA-RL: Simulated users in RL train 32B to beat 235B"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The whole approach depends on GPT-4o, used as the simulated user during training, behaving enough like a real human user—including asking follow-ups, changing requests, and confirming actions—that a policy trained against it will also succeed with the evaluation simulators and with actual users.","fun_headline_variants_meta":{"raw":{"variants":["Simulated user in RL loop lets 32B match 235B models","First RL for agentic tool use with simulated users in loop","32B model trained with simulated user beats 3x-larger rivals","Dynamic user simulation in RL boosts 32B agentic tool use","MUA-RL: Simulated users in RL train 32B to beat 235B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1598,"prompt_tokens":866,"completion_tokens":732,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":633}},"tokens_in":610,"tokens_out":732,"duration_ms":8534,"temperature":1.0,"reasoning_tokens":633,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:17:47.394012+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MUA-RL with a differently constructed user simulator (e.g., one that never asks follow-up questions, or one that randomly alters requests mid-task) and measure whether the benchmark gains disappear or reverse; if they do, the reported improvements are tied to the specific simulator rather than to dynamic user interaction in general. A stronger version would evaluate the trained agents with real human users on the same retail and airline tasks.","supporting_citations":[{"cited_title":"The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the BFCL-V3 Multi Turn evaluation set with base, miss-param, miss-func, and long-context subsets."}],"review_version":1}