{"id":"5501a0a4-76e3-49c4-974c-711ebb85e44f","arxiv_id":"2508.21365","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A reinforcement-learning pipeline for predicting macro-actions in Honor of Kings improves action prediction accuracy, but the method is imitation of human replay labels, not the claimed environmental interaction.","lead":"This paper proposes a training pipeline called Think in Games that uses reinforcement learning to teach large language models to choose strategic actions in the mobile game Honor of Kings. The authors claim a 14B model can beat a 671B reasoning model, but the experiments train on human replay labels rather than actual game play, so the headline claim is not supported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward in Eq. (4) is supervised label matching, not environmental feedback; the procedural-knowledge claim is not tested.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: Eq. (4)'s reward is not environmental feedback but a match to human replay labels. This is the crux of the paper because every downstream claim—bridging declarative and procedural knowledge, outperforming Deepseek-R1, learning from interaction—depends on the reward being a genuine signal from the environment. My reading of the full text confirms this: the environment is never actually interacted with; the 'online RL' optimizes agreement with fixed labels. The paper does include some evidence that GRPO improves label-prediction accuracy (Table 2a), but that is consistent with behavioral cloning plus variance-reduced optimization, not with acquiring procedural skill. The Table 3 results further undermine the comparative claim. I found no additional independent evidence—no released code, no live-environment evaluation, no outcome-based reward—that would rescue the central claim. Therefore the appropriate verdict remains REJECT, and this stress-test does not change the reader's conclusion.","tokens_in":15923,"tokens_out":4314,"duration_ms":50787,"concrete_test":"Deploy the final TiG policy in the actual Honor of Kings environment (or a high-fidelity simulator) by executing the predicted macro-action at each decision point, and compare win-rate against the base Qwen-3-14B and Deepseek-R1 under the same interface. If there is no statistically significant win-rate improvement, the central 'procedural understanding via environmental interaction' claim is unsupported regardless of action-prediction accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that TiG acquires procedural knowledge through direct interaction with game environments and online RL from environmental feedback—is not supported by the method as written. In Section 3.2, Eq. (4) defines the reward as rt = 1 iff predicted macro-action equals A*_t obtained from replay data. This is a supervised label-matching objective, not an environmental outcome. The model never executes an action in a live environment, observes a state transition, or receives a win/loss or state-value reward. The data pipeline in Section 3.1.2 further reinforces this: labels are propagated backward and overwritten according to a hand-designed priority hierarchy, so the training signal is entirely derived from human replay annotations. The SFT stage also distills Deepseek-R1 outputs, adding another imitation signal. At best, the results in Table 2a show that GRPO improves the model's ability to fit human replay labels; they do not demonstrate improved decision quality or procedural understanding. This is confirmed by Table 3, where Deepseek-R1 outperforms the trained Qwen-2.5-32B on both strong-related (57.14% vs 40.00%) and weak-related (79.63% vs 37.78%) game-state questions—contradicting the claim that TiG surpasses R1. The paper's own limitation section acknowledges interpretability and generalization gaps but does not acknowledge the more fundamental gap between its framing and its reward signal. The formalization in Section 2.3 defines the task as state-to-action prediction; nothing in the experiments closes the loop to environment interaction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Think in Games (TiG), a framework to train LLMs for macro-level strategic decision-making in the MOBA game Honor of Kings. The method combines SFT on DeepSeek-R1 distilled outputs with GRPO reinforcement learning, where the reward is a binary match between the predicted macro-action and a human replay label. The authors claim that TiG enables LLMs to acquire procedural knowledge through direct interaction with game environments, and that a 14B model can surpass Deepseek-R1 on action prediction while preserving general capabilities. The paper reports action-prediction accuracy, a TiG-QA generalization evaluation, general NLP benchmark results, and qualitative case studies.","tokens_in":16339,"tokens_out":3812,"duration_ms":42753,"significance":"If the central claim were supported, the paper would be significant: it would show that a relatively small, interpretable LLM can match or exceed a much larger reasoning model on game-strategy tasks through environmental interaction, with potential applications to explainable game agents. The paper has several strengths: a clearly specified multi-stage training pipeline, evaluation on general LLM benchmarks showing that domain training does not degrade broad capabilities, a concrete error taxonomy, and qualitative case studies. However, the core claim of learning from environmental feedback is not supported by the method as written, and one of the paper's own tables directly contradicts a headline comparison. These issues are load-bearing for the paper's stated contribution.","major_comments":[{"comment":"The reward is rt = 1 iff the predicted macro-action equals A*_t obtained from replay data. No action is executed in a live environment, no state transition is observed, and no win/loss or outcome-based reward is used. This is supervised label matching, not reinforcement learning from environmental feedback. The abstract, Figure 1 caption, and §7 repeatedly claim 'direct interaction with game environments' and 'environmental feedback', but the method as written does not interact with an environment at all. A concrete test would be to deploy the policy in a live HoK environment or to evaluate it with an outcome-based metric such as win rate; without that, the procedural-knowledge claim is unsupported.","section":"§3.2, Eq. (4)"},{"comment":"The text states that TiG 'achieves results comparable to Deepseek-R1' on TiG-QA, but Table 3 shows Deepseek-R1 ahead on both question types: strong-related correct responses (score 2) are 57.14% for Deepseek-R1 vs 40.00% for Qwen-2.5-32B + GRPO, and weak-related correct responses are 79.63% vs 37.78%. This directly contradicts the 'comparable' claim. The paper's explanation that Deepseek-R1 excels because the questions rely on web knowledge rather than game interaction also undercuts the paper's own argument that TiG's value comes from game-grounded procedural understanding.","section":"§4.4, Table 3"},{"comment":"The main action-prediction evaluation measures accuracy against the same human replay labels A*_t used to compute the training reward in Eq. (4). The reported accuracy is therefore the fitted training objective, not an independent measure of game competence. The TiG-QA evaluation is not circular, but as noted above it does not support the claim that TiG surpasses Deepseek-R1. An independent evaluation—e.g., expert human judgment on held-out states or a live-game outcome metric—is needed before the central claim can be accepted.","section":"§4.3, Table 2a vs. §3.2, Eq. (4)"},{"comment":"The abstract and conclusion claim that TiG achieves 'competitive performance with dramatically lower data and computational demands compared to conventional RL methods.' However, the experiments compare only against LLM baselines (Qwen variants, Deepseek-R1, QwQ-32B); no conventional RL agent (e.g., PPO- or DQN-based MOBA agents) is included. This claim is therefore unsupported by the reported experiments.","section":"§4.1, §6"}],"minor_comments":[{"comment":"Model names are inconsistent: 'Qwen2.5-14B' and 'Qwen-2.5-14B' are used interchangeably. Please standardize.","section":"§4.2"},{"comment":"The table columns are labeled '0', '1', '2' with the caption 'Numbers indicate count (percentage) of correct responses.' It is unclear whether entries are counts, percentages, or both; sample sizes per row should be stated.","section":"Table 3"},{"comment":"QwQ-32B appears in the table but is not listed in the baselines in §4.1 and no citation or access link is provided.","section":"Table 2a"},{"comment":"The caption says the model is updated using a 'rule-based verifier' in the 'game environment.' Since the verifier is Eq. (4), matching a replay label, the caption should not imply live environmental feedback.","section":"Figure 1 caption"},{"comment":"The Limitations section states that TiG 'still requires a substantial amount of environment interaction.' This is inconsistent with the method description in §3, where no environment interaction occurs. Either the limitation should be reworded or the method description should be corrected.","section":"§7"}],"recommendation":"reject","confidential_remarks":"The paper's central claim—that TiG learns procedural knowledge through direct interaction with game environments—is contradicted by the method's reward design in Eq. (4), which is replay-label matching. The TiG-QA results in Table 3 also directly undermine the stated comparison with Deepseek-R1. These are not local issues; they affect the paper's main contribution. The authors may wish to reframe the work as imitation learning with GRPO and add an outcome-based evaluation, but that would be a substantial revision rather than a minor fix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper does one thing solidly—it shows that an SFT+GRPO pipeline can push a 14B LLM to match human macro-action labels in Honor of Kings at 90.91% accuracy—but the accompanying narrative that this constitutes procedural knowledge acquired through game interaction is not supported by the method. The reward in Eq. (4) is a supervised match against replay labels, not an environmental outcome, and the evaluation metric is the same objective.\n\nWhat's actually new: a concrete data pipeline (relabeling algorithm for dense action labels, a 40-action taxonomy, JSON game-state representations) and an application of GRPO to macro-action prediction in a commercial MOBA. The authors also check that standard benchmarks do not degrade, which is worth doing. The case studies are detailed and the system is plausible for a real-time game assistant.\n\nWhere it falls apart: the abstract and introduction repeatedly say \"direct interaction with game environments\" and \"online reinforcement learning based on environmental feedback,\" but the training data is a fixed set of replay frames, and the reward is exactly whether the predicted action equals the human's action. That's behavioral cloning, not RL from the environment. The TiG-QA results in Table 3 directly contradict the \"comparable to Deepseek-R1\" claim—R1 is far ahead on both strong- and weak-related questions. And there is no comparison to an actual gameplay evaluation (win rate, etc.). The limitations section acknowledges generalization gaps but not this fundamental mismatch.\n\nMy take: the pipeline is a legitimate contribution as a high-level action predictor trained on expert replays, and the engineering is clean. But as a framework that \"bridges declarative and procedural knowledge,\" it is not substantiated. The authors should either reframe the contribution or add a real environment evaluation. I'd send it to review—the work is relevant and the data is potentially useful—but would expect a major revision or a rejection if the framing is not corrected.","headline":"A clean behavioral-cloning pipeline with an overblown 'procedural knowledge' framing; worth a referee's time only if the authors reframe or test in a live environment.","tokens_in":16793,"tokens_out":4101,"would_cite":false,"duration_ms":45544,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims RL fine-tuning lets a 14-billion-parameter model predict MOBA macro-actions at 90.91% accuracy, ahead of Deepseek-R1's 86.67%.","keywords":["large language models","reinforcement learning","procedural knowledge","macro-action prediction","MOBA games","GRPO","interpretable decision-making","Honor of Kings"],"falsifier":"Put the trained policy into live Honor of Kings matches (or a high-fidelity simulator) and compare win rate, tower damage, and objective control against the base LLM and against random macro-action baselines; if higher macro-action prediction accuracy does not translate to better in-game outcomes, the claim that TiG instills procedural knowledge is falsified.","tokens_in":15856,"feed_emoji":"🎮","tokens_out":7311,"duration_ms":68294,"temperature":0.7,"pith_summary":"Think-in-Games (TiG) tries to show that an LLM can move from knowing facts about a game to knowing how to steer it, by turning strategic decisions into a language task. Given a JSON-encoded game state, the model must output the best macro-action plus a natural-language reasoning trace; supervised fine-tuning on distilled reasoning is followed by GRPO reinforcement learning with a rule-based reward that checks whether the predicted action matches the recorded human macro-action. On Honor of Kings, a Qwen-3-14B model trained this way reaches 90.91% macro-action prediction accuracy, beating Deepseek-R1's 86.67% and outperforming all untrained baselines. The authors read this as evidence that procedural understanding can be injected into smaller models without sacrificing general reasoning, and that the resulting decisions come with step-by-step explanations. The reason to care is the implied trade-off: much smaller, interpretable models might handle strategic decision-making in constrained action spaces as well as models an order of magnitude larger.","feed_headline":"Small LLM beats 671B reasoning model at MOBA move prediction","feed_subtitle":"A 14-billion-parameter model reaches 90.91% accuracy on strategic game decisions, beating a 671-billion-parameter reasoner.","key_machinery":"The central object is the macro-action space: a fixed set of 40 high-level team objectives (push top lane, secure dragon, defend base, etc.) that abstracts away low-level control. The policy model maps a JSON-encoded game state to a subset of these actions plus a <think> reasoning chain. The training engine is GRPO: for each game-state prompt, a group of completions is generated, each scored by a rule-based verifier comparing the predicted action to the replay-derived ground truth; rewards are normalized within the group into advantages, and the policy is updated with a clipped surrogate loss plus KL regularization to a reference policy. The macro-action abstraction and the binary reward tog","core_discovery":"On its own terms, TiG's central claim is that declarative and procedural knowledge can be bridged by reformulating RL-based decision-making as a language modeling task: the LLM generates a language-guided policy (a set of macro-actions and a reasoning chain), which is then refined iteratively through online reinforcement learning. Training is multi-stage: SFT on reasoning distilled from Deepseek-R1, then GRPO with a rule-based binary reward that assigns 1 if the predicted action equals the replay-derived ground-truth action and 0 otherwise. The reported result is that Qwen-3-14B + SFT + GRPO reaches 90.91% on macro-action prediction, outperforming Deepseek-R1 (86.67%) and Qwen-2.5-32B + GRPO","pith_inferences":["The implemented reward is agreement with recorded human macro-actions, not outcome-based game reward, so TiG as described is closer to RL-assisted behavioral cloning than to learning from win/loss consequences; a live-game test is needed to confirm the procedural-knowledge claim.","Performance likely depends on the expert-designed 40-action taxonomy and priority relabeling; porting TiG to a new game means rebuilding this abstraction, and part of the reported gain may come from this human prior rather than from RL itself.","The reasoning traces may be post-hoc rationalizations; to test faithfulness, one could ablate parts of the game state and check whether the model's stated reasons change only when the relevant feature is removed.","If the accuracy result transfers, a practical route emerges: take a moderately sized instruction-tuned model, distill reasoning from a large reasoner into it, then refine on replay-labeled decisions, obtaining deployable game assistants at a fraction of the inference cost."],"forward_implications":["A 14-billion-parameter model can match or beat a 671-billion-parameter general reasoning model on constrained strategic decision tasks, suggesting parameter count is not the binding constraint once domain-specific RL is applied.","GRPO with rule-based rewards is the main driver of improvement: Qwen-2.5-32B jumps from 66.67% to 86.84% with GRPO alone, and GRPO alone beats SFT alone on Qwen2.5-14B.","Trained models keep their general language abilities: scores on MMLU, CEval, BBH, and IfEval remain stable or improve slightly, so domain-specific RL does not cause catastrophic forgetting.","Because decisions come with natural-language reasoning, the resulting policy is inspectable by human players, unlike black-box RL agents.","The framework is presented as transferable beyond MOBAs to other interactive environments where language can serve as the policy interface."],"supporting_citations":[{"why":"Supplies the rule-based reward precedent, the Deepseek-R1 baseline, and the large reasoning model used to distill SFT training data.","marker":"DeepSeek-AI et al., 2025"},{"why":"Defines GRPO, the online RL algorithm TiG adapts for policy optimization.","marker":"Shao et al., 2024"},{"why":"Provides PPO and the KL divergence estimator used in GRPO's objective.","marker":"Schulman et al., 2017"},{"why":"Represents prior deep RL for MOBA control that TiG contrasts as black-box and data-hungry.","marker":"Ye et al., 2020"},{"why":"RLHF framework that motivates RL-based training of LLMs over pure supervised fine-tuning.","marker":"Ouyang et al., 2022"},{"why":"Survey establishing the challenges of grounding LLMs in game environments and the gap TiG addresses.","marker":"Hu et al., 2024b"}],"fun_headline_variants":["14B model beats 671B reasoner in game macro-actions","TiG: small model wins strategic game moves via RL","Qwen-3-14B tops Deepseek-R1 in MOBA prediction","How RL turns LLM knowledge into game-playing skill","Think in Games: bridging know-that and know-how"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The central claim rests on treating a match between the model's predicted macro-action and a human player's recorded macro-action as the reward signal; if those human labels are imperfect, not the right abstraction, or not predictive of winning outcomes, the model may learn to imitate labeling artifacts rather than true game strategy.","fun_headline_variants_meta":{"raw":{"variants":["14B model beats 671B reasoner in game macro-actions","TiG: small model wins strategic game moves via RL","Qwen-3-14B tops Deepseek-R1 in MOBA prediction","How RL turns LLM knowledge into game-playing skill","Think in Games: bridging know-that and know-how"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1428,"prompt_tokens":791,"completion_tokens":637,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":535,"tokens_out":637,"duration_ms":6599,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:18:47.080333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Put the trained policy into live Honor of Kings matches (or a high-fidelity simulator) and compare win rate, tower damage, and objective control against the base LLM and against random macro-action baselines; if higher macro-action prediction accuracy does not translate to better in-game outcomes, the claim that TiG instills procedural knowledge is falsified.","supporting_citations":[{"cited_title":"Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, et al","cited_arxiv_id":null,"evidence_quote":"RLHF framework that motivates RL-based training of LLMs over pure supervised fine-tuning."}],"review_version":1}