{"id":"9e7af219-4dd3-4ffb-9573-ce4ba7e1b71a","arxiv_id":"2505.16401","paper_version":4,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"A group, fuse, and retrain recipe for multi-game reinforcement learning lets a 32B model reach near-Claude3.5 performance on several TextArena games, though the headline score is internally inconsistent.","lead":"A 32-billion-parameter language model was trained to play 18 text-based games by splitting the games into groups, training specialists, averaging their weights, and retraining on everything. It matches the commercial model Claude3.5 on some games, but the headline win/draw/loss score in the abstract does not match the paper's own result table.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The '7 wins, 4 draws' headline is an artifact of counting single-player games as wins over Claude; the actual head-to-head two-player record is 99W-29D-151L, a decisive loss.","rationale":"I read the paper in good faith and verified the reader's arithmetic. The reader's stated weakest assumption is the fuse step's parameter averaging, but that is not the most load-bearing issue: Table 3 shows that after conquer, performance generally recovers or improves over fuse alone, so the mechanism is at least plausible. The more serious problem is the evaluation protocol behind the central claim. The abstract's 7-4-7 record is not a contradiction of Table 2; it is reproducible only by counting single-player games as victories or draws against Claude, even though those games are played separately against the environment, not head-to-head. Under a fair protocol, the two-player head-to-head record is roughly 99-29-151, a clear loss. Thus the central claim 'comparable to Claude3.5' is unsupported regardless of whether the divide-fuse-conquer recipe itself is effective. The rejection verdict stands, but for the counting/protocol reason rather than the alleged Table 2 contradiction. I also note secondary concerns that reinforce rejection: no error bars, only 20 games per environment, custom game selection (games where the base model does not consistently fail), and no released code or base model. These do not change the verdict but make the empirical support even weaker.","tokens_in":23448,"tokens_out":7352,"duration_ms":57695,"concrete_test":"Recompute the headline tally from Table 2 using two separate protocols: (1) for the 14 two-player games, sum W/D/L across all head-to-head matches and report the aggregate (my count: 99/29/151); (2) for the 4 single-player games, report success rates separately and do not classify them as wins/draws against Claude. If, as expected, the two-player aggregate shows more losses than wins and the single-player games are not head-to-head, the abstract's '7 wins and 4 draws' and 'comparable to Claude3.5' claims are not supported. A secondary check is to compute 95% binomial confidence intervals for each 20-game two-player row; rows such as Poker-v0-long (9/8/3) and SimpleNegotiation-v0 (9/3/8) are within sampling noise of a draw.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims performance comparable to Claude3.5, 'achieving 7 wins and 4 draws' across 18 games. The reader's rationale says Table 2 actually yields 5-2-7 and therefore contradicts the abstract. That specific arithmetic is wrong: the 7-4-7 count is reproducible if the 4 single-player games are included, with TowerOfHanoi-v0 and TowerOfHanoi-v0-medium counted as DFC wins (DFC 20/0/0 vs Claude 17/0/3 and 2/0/18) and GuessTheNumber-v0 and GuessTheNumber-v0-hardcore counted as draws (both 20/0/0). However, this is precisely the load-bearing flaw: single-player games are not head-to-head competitions, so counting them as 'wins' and 'draws' against Claude inflates the headline. Restricting to the 14 genuine two-player games, the aggregate record is approximately 99 wins, 29 draws, and 151 losses out of 280 matches (35.4% win rate), so the trained model loses more than twice as often as it wins against Claude3.5. The claim of being 'comparable to Claude3.5' is therefore unsupported by the paper's own data once the comparison protocol is made explicit. This is a more direct threat to the central claim than the parameter-averaging assumption, because even if fusion works perfectly, the headline result is an artifact of the counting protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Divide-Fuse-Conquer (DFC-MSRL), a multi-scenario reinforcement learning framework for LLMs playing text games. The method divides 18 TextArena games into four heuristic groups, trains a specialized policy per group with GRPO and several stabilizers (format reward shaping, half-negative sampling, mixed prioritized sampling, epsilon-greedy disturbance, randomized seeds, hasty-action penalty), fuses the specialized policies by parameter averaging, and then continues RL training on the merged groups. The authors claim that Qwen2.5-32B-Align trained with this recipe reaches performance comparable to Claude3.5, citing a record of 7 wins and 4 draws across the 18 games, and they report ablations and training curves for the individual techniques.","tokens_in":23866,"tokens_out":7184,"duration_ms":51463,"significance":"If the headline result were sound, the paper would present a useful empirical recipe for scaling RL across heterogeneous game scenarios, with practical engineering techniques and extensive TextArena evaluation. Credit is due for the detailed ablations (Figure 3), hyperparameter sensitivity experiments (Appendix F), and the unusually complete description of the training recipe. However, the central claim is not supported by the paper's own data: the 14 genuine two-player games show an aggregate 99 wins, 29 draws, and 151 losses against Claude3.5, and the advertised 7-4-7 count follows only from treating single-player success rates as head-to-head results. The framework may still yield improvements over the base model, but the claimed comparability to a frontier model is an artifact of the comparison protocol.","major_comments":[{"comment":"The abstract and Section 4.3 claim that Qwen2.5 (DFC-MSRL) is comparable to Claude3.5, achieving 7 wins and 4 draws. This count does not follow from Table 2 for the 14 two-player games, which give 5 wins, 2 draws, and 7 losses under the paper's own majority rule. The 7-4-7 count is obtained only by treating single-player games (GuessTheNumber-v0, GuessTheNumber-v0-hardcore, TowerOfHanoi-v0, TowerOfHanoi-v0-medium) as wins/draws against Claude3.5, but these are not adversarial head-to-head games. Summing the two-player rows of Table 2 gives 99 wins, 29 draws, and 151 losses out of 280 matches, a 35.4% win rate and a decisive aggregate loss. The central claim of comparability to Claude3.5 is therefore unsupported by the paper's own data.","section":"Abstract; Section 4.3; Table 2"},{"comment":"The evaluation suite was selected after preliminary testing with the base model (Section 4.1 states that only games where the base model 'did not consistently fail' were retained and then expanded), and hyperparameters were tuned on the same games (Section 4.2 and Appendix F). This introduces selection bias and a risk of overfitting to the evaluation set. Moreover, all results are single batches of 20 seeds per environment with no reported variance, confidence intervals, or significance tests; many entries are 20/0/0 or 0/0/20, so the paper's headline difference between 7-4-7 and the two-player 5-2-7 may well be within noise. The evaluation should be repeated with multiple independent runs and error bars.","section":"Section 4.1; Section 4.2; Appendix F"},{"comment":"The fuse step assumes that parameter averaging, theta(pi(k)) = 1/2(theta(pi(k-1)) + theta(pi(k))), preserves the competence of each group-specialized policy well enough that continued GRPO training on the merged group improves rather than destroys performance. Table 3 provides only indirect evidence: after fusion, win rates sometimes drop sharply before the conquer phase (for example, SimpleNegotiation-v0-long goes from 8/8/4 at Conquer(g1,g2) to 2/4/14 at Fuse(g1,g2,g3)). No direct test is provided that the fused policy retains the skills of its parents. An ablation comparing DFC-MSRL with a version that skips fusion, or fuses at different stages, is needed to validate this load-bearing mechanism.","section":"Section 3.2.2; Table 3"},{"comment":"The 'Aha Moment' analysis is partly circular: it defines aha moments as improvements in win rate, response length, and step count, which are exactly the metrics optimized by the reward shaping and training objectives. The reported curves therefore show that the policy follows the training signal, not that a distinct reasoning breakthrough occurs. If the paper wishes to claim an aha-moment phenomenon, it should provide a more direct behavioral measure, such as qualitative changes in intermediate reasoning steps, independent of the optimized reward.","section":"Section 4.3 (Aha Moment)"}],"minor_comments":[{"comment":"In the MPS rollout allocation formula, the denominator uses M(t)_{g_j} but should use W(t)_{g_j} to match the definition of the mixed sampling weight; as written, the expression is undefined.","section":"Section 3.2.3"},{"comment":"The word 'Algorighm' is a typo in the algorithm caption; it should be 'Algorithm'.","section":"Algorithm 1"},{"comment":"Table 3 is very difficult to read: game names are concatenated with group labels in the first column, and the column headers such as 'Fuse(g1,g2,g3)' combined with the final 'g1+g2+g3+g4' row are ambiguous. The table should be reformatted with clear separate rows for each game and explicit phase labels.","section":"Table 3"},{"comment":"The base model Qwen2.5-32B-Align is described as an in-house model and is not publicly available; this limits reproducibility. The authors should state how to obtain the model or otherwise make the exact base checkpoint identifiable.","section":"Section 3.1"},{"comment":"Figure 1 displays only 8 of the 18 games, yet the text refers to 'the win rates of Qwen2.5 and Qwen2.5 (DFC-MSRL)' without explaining the selection; the caption should clarify which games are shown and why.","section":"Figure 1"}],"recommendation":"reject","confidential_remarks":"The skeptical note about the headline is correct: the 7W-4D claim in the abstract is an artifact of counting single-player success rates as adversarial wins or draws against Claude3.5. The restricted two-player record is 99-29-151, a decisive loss. Even aside from the headline, the absence of error bars and the post-hoc selection of the game suite weaken the empirical contribution substantially. Because the paper's primary claim is contradicted by its own tables, I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nThe headline is the problem: 'comparable to Claude3.5' is not what Table 2 shows. In the 14 two-player games, the trained model's aggregate record against Claude3.5 is 99 wins, 29 draws, 151 losses—a decisive loss. The '7 wins and 4 draws' from the abstract only makes sense if you count the four single-player games as competitive results. They are not: Claude also solves GuessTheNumber, and its TowerOfHanoi performance is 17/3 vs DFC's 20/0—good, but not a head-to-head match. The reader's note that the abstract contradicts Table 2 is arithmetically off; the abstract is reproducible, but only by treating non-competitions as wins or draws. That is a real flaw in the central claim, not a typo.\n\nWhat is genuinely new is the pipeline: heuristic grouping, per-group GRPO, parameter averaging, and continued training on merged groups. That is a coherent answer to multi-scenario instability, and the ablation figures are informative. The six stabilizers are individually known, but the combination is new for text games. Table 3 is actually the most interesting part: fusion alone often hurts (e.g., LiarsDice), and the conquer phase recovers and exceeds. That suggests the framework is doing something real, not just cherry-picking.\n\nThe main soft spots, in proportion: (1) The evaluation protocol conflates single-player and two-player results. This is load-bearing. (2) No error bars; 20 seeds, and the game suite was selected after initial testing—selection bias is plausible. (3) Hyperparameters were tuned on the same games. (4) No code or base model release, and the base model is an in-house Moonshot model, so independent reproduction is currently impossible. (5) The parameter-averaging fusion is a blunt instrument; the paper gives no comparison to other fusion methods and no analysis of when averaging destroys specialized skills. That is a minor-to-moderate issue, because the conquer phase seems to repair it.\n\nI would send this to peer review, but only with the expectation of major revision. The authors need to report two-player head-to-head results separately, drop or heavily qualify the Claude comparison, add error bars, and ideally release code. If they do that, the divide-fuse-conquer idea is worth the community's time.\n\nFor a reading group, I'd bring it as an example of a promising recipe undermined by an overclaimed headline.","headline":"A promising RL recipe whose headline its own Table 2 contradicts: the two-player record against Claude3.5 is 99W-29D-151L, not 'comparable'.","tokens_in":24360,"tokens_out":4283,"would_cite":false,"duration_ms":32337,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A staged divide-fuse-conquer RL recipe lifts a 32B model to Claude3.5-level play across 18 text games.","keywords":["multi-scenario reinforcement learning","LLM generalization","text games","GRPO","parameter averaging","curriculum learning","training collapse"],"falsifier":"Train the same 18 TextArena games with the identical conquer recipe but replace the fuse step by random parameter interpolation, or by simply concatenating all games without grouping; if the final win/draw/loss record against Claude3.5 stays at 7/4/7 under the same compute budget, then the divide-fuse mechanism is not what drives the result.","tokens_in":23252,"feed_emoji":"🎮","tokens_out":6253,"duration_ms":48232,"temperature":0.7,"pith_summary":"The paper claims that a single 32-billion-parameter language model can be trained with reinforcement learning to play 18 diverse text games at a level comparable to a frontier commercial model, and that the key is not a bigger model or more data but a staged training recipe. The recipe, Divide-Fuse-Conquer, first groups games by their rules and by whether the base model can already win, trains a specialist policy on each group, then averages the parameters of the specialists into one model and continues training on the union of groups, repeating until every group is absorbed. The authors test this on TextArena with Qwen2.5-32B-Align and report that the trained model achieves 7 wins and 4 draws against Claude3.5 across 18 games, where naive multi-scenario RL suffers collapse and single-scenario training stalls. If true, this is evidence that heterogeneous RL tasks can be generalized by deliberately managing similarity and transfer, rather than by scaling compute alone.","feed_headline":"Divide, fuse, conquer: a 32B model rivals Claude3.5 on 18 text games","feed_subtitle":"Grouping games by similarity, averaging specialist policies, then retraining lifts the model to 7 wins and 4 draws.","key_machinery":"The load-bearing operation is the fusion identity $\\theta(\\pi^{(k)}) = \\tfrac{1}{2}\\left(\\theta(\\pi^{(k-1)}) + \\theta(\\pi_k)\\right)$, which averages the parameters of the previously fused policy with the newly trained group specialist. That simple average is what allows knowledge from separate game groups to coexist in one model before the conquer phase retrains it; the conquer phase uses GRPO (Group Relative Policy Optimization) on self-play rollouts, stabilized by format reward shaping, half-negative sampling, mixed priority sampling, epsilon-greedy disturbance, randomized seeds, and a step penalty for hasty actions. The divide step supplies the grouping heuristic: games are split by random versus fixed initial states and by whether the base model has any non-zero win rate, yielding four groups of comparable difficulty and dynamics.","core_discovery":"The central claim is that \"aha moments\" — the sudden jumps in reasoning quality that outcome-only RL can produce in math and coding — can be elicited in multi-scenario games if the scenarios are first separated, conquered, and then recombined. The paper's result is a training trajectory: four groups of games, one specialist per group, parameter averaging as the fusion operation, and a \"conquer\" phase with GRPO, format rewards, half-negative sampling, mixed priority sampling, epsilon-greedy exploration, randomized seeds, and a hasty-action penalty. On 18 TextArena environments, this recipe turns Qwen2.5-32B-Align into a policy that records 7 wins, 4 draws, and 7 losses against Claude3.5, outperforming both single-scenario RL and direct multi-scenario RL on the same benchmark.","pith_inferences":["The reported 7W/4D/7L record is against one fixed opponent; a natural next test is whether the trained policy stays competitive against a panel of opponents with different styles and against the same model in repeated plays with varied seeds.","If the parameter-averaging step is the real mechanism, the recipe could generalize to other multi-task RL settings, such as robotics skill libraries, multi-domain dialogue, or agent tool use, by grouping tasks with shared state-transition structure.","A direct ablation of the divide step itself — random grouping versus rule-based grouping with the same fuse-and-conquer budget — would separate the contribution of grouping quality from the contribution of progressive training."],"forward_implications":["A 32B open-weight model can reach parity with a frontier model on a multi-game benchmark without supervised reasoning data, purely through staged RL on outcome rewards.","Training collapse seen when many heterogeneous games are combined is attributable to scenario diversity, and can be avoided by grouping games before joint optimization.","Fused parameter-averaged policies provide an initialization from which continued GRPO training converges faster and to a higher final win rate than training any single group alone.","Each stabilization technique in the conquer recipe contributes measurably: format rewards prevent instruction-following collapse, half-negative sampling prevents early collapse, and mixed priority sampling accelerates slow-converging games.","The framework should transfer to other RL domains where a single policy must cover many scenarios with distinct mechanics."],"supporting_citations":[{"why":"TextArena supplies the 18 game environments and the Claude3.5 comparison protocol on which the headline result is measured.","marker":"[7]"},{"why":"DeepSeek-R1 establishes the outcome-only RL paradigm (the \"aha moment\") that the paper extends from single tasks to multi-scenario games.","marker":"[9]"},{"why":"DeepSeekMath provides the GRPO objective used in the conquer step and in the base RL recipe.","marker":"[18]"},{"why":"DPO describes the direct preference optimization stage used to build the Qwen2.5-32B-Align base model.","marker":"[16]"},{"why":"Kimi K1.5 supplies the rollout-and-update training recipe that the conquer algorithm follows.","marker":"[24]"},{"why":"The epsilon-greedy disturbance technique is grounded in the cited bandit exploration strategy.","marker":"[5]"}],"fun_headline_variants":["Divide-fuse-conquer: 32B LLM rivals Claude3.5 on 18 text games","Aha moment via RL: divide-fuse-conquer lifts 32B to Claude3.5","Group, conquer, fuse: 32B LLM matches Claude3.5 across 18 games","Specialist fusion: 32B LLM earns 7 wins, 4 draws vs Claude3.5","Divide, fuse, conquer: 32B LLM matches Claude3.5 on 18 games"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework depends on parameter averaging keeping each group's specialized skill intact: if averaging two policies just mixes away the hard-won behaviors, the conquer phase begins from a degraded model and the whole framework loses its mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Divide-fuse-conquer: 32B LLM rivals Claude3.5 on 18 text games","Aha moment via RL: divide-fuse-conquer lifts 32B to Claude3.5","Group, conquer, fuse: 32B LLM matches Claude3.5 across 18 games","Specialist fusion: 32B LLM earns 7 wins, 4 draws vs Claude3.5","Divide, fuse, conquer: 32B LLM matches Claude3.5 on 18 games"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001501,"raw_usage":{"total_tokens":6041,"prompt_tokens":986,"completion_tokens":5055,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":4926}},"tokens_in":602,"tokens_out":5055,"duration_ms":29954,"temperature":1.0,"reasoning_tokens":4926,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:01:53.220662+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 18 TextArena games with the identical conquer recipe but replace the fuse step by random parameter interpolation, or by simply concatenating all games without grouping; if the final win/draw/loss record against Claude3.5 stays at 7/4/7 under the same compute budget, then the divide-fuse mechanism is not what drives the result.","supporting_citations":[{"cited_title":"Finite-time regret bounds for the multiarmed bandit problem","cited_arxiv_id":null,"evidence_quote":"The epsilon-greedy disturbance technique is grounded in the cited bandit exploration strategy."}],"review_version":1}