{"id":"d0ed1ded-c73d-475c-8a1e-72f42ee06694","arxiv_id":"2507.03711","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In 50-game matches, an 8-billion-parameter Llama beat a 70-billion-parameter Llama more often than it lost, while larger models generated longer reasoning traces but not reliably better scores.","lead":"This paper tests three sizes of Llama language models as agents in the traditional Vietnamese board game Ô Ăn Quan, recording their win rates, scores, and the lengths of their reasoning traces. It probes whether bigger models plan better in a fully observable, deterministic game, a question relevant to how language models reason and decide under constraints.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that larger models have more advanced planning is built entirely on 'planning depth' = generated step count, a proxy never validated against actual search, move quality, or any external notion of planning; longer traces could simply be verbosity.","rationale":"Reader's assessment is sound. The central claim in V.C is the only strong conclusion about architecture scale, and it depends wholly on 'planning depth' as number of generated steps. The paper does not supply any construct validation of this measure, and the qualitative classification by Gemini is not a gold standard. This is a correctness risk because the alternative explanation—larger models are more verbose—is well-documented and would trivially reproduce Figure 3. The confounded measurement context († row) compounds this: without knowing the opponent and player order for the 70B row, the cross-model comparison is not controlled. I therefore agree with the reader's weakest-assumption identification. That said, the paper has independent value: it presents a new, fully observable rule-based environment, releases source code, and reports basic gameplay outcomes. The descriptive win-rate and persona results, while statistically light, are honest exploratory observations. The limitations section openly acknowledges the simplification and the reliance on reasoning labels. Thus a conditional acceptance—requiring validation of the planning metric and clarification/control of the experimental design—is proportionate. No reason to reject outright given the testbed contribution. The proposed concrete test (correlating step count with search-based move quality and explicit state coverage) would provide the missing construct validity in a single pass. If it yields no correlation, the Section V.C claim should be removed or downgraded to a statement about output length.","tokens_in":7623,"tokens_out":9263,"duration_ms":98753,"concrete_test":"On a sample of 100 logged moves per model, run a fixed-depth minimax search (e.g., depth 5) over the Ô Ăn Quan game tree to compute the game-theoretic value of the model's chosen move relative to the best legal move; also count the number of distinct board states or candidate positions the model's rationale explicitly evaluates. Then compute the within-model correlation between the paper's 'planning depth' (step count) and both (a) move-value rank and (b) explicit state coverage. If either correlation is near zero or negative across all models, the metric in Section V.C is not a measure of planning and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V.C defines 'planning depth' as the number of steps generated per move and uses Figure 3's distribution to conclude that Llama-3.3-70B possesses more advanced planning capabilities. This proxy is never validated: the paper does not compare it against game-tree search depth, the number of legal moves considered, or the quality of the resulting moves, nor does it control for output verbosity. Larger instruction-tuned models are known to produce longer rationales; without an external anchor, high step counts are just as consistent with over-explanation as with deeper planning. The supplementary reasoning-type classification (Section V.D, Table III) is itself produced by Gemini in a zero-shot setting with no human reliability check, so it cannot independently corroborate the metric. Moreover, Figure 3 aggregates games 'played against Llama-3.3-70B-Instruct,' yet the 70B model's own row (marked †) is not described in terms of opponent or player order, raising a confound: the 'depth' of the 70B may be measured under systematically different game states than the 3B/8B. The paper's Limitations section concedes that evaluation is based on reasoning labels rather than full gameplay outcomes. Since the Section V.C claim rests on this single unvalidated measure, it is load-bearing and currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the traditional Vietnamese board game Ô Ăn Quan as a testbed for evaluating the strategic planning and decision-making abilities of three Llama-based instruction-tuned models (3B, 8B, and 70B). The authors implement a closed-loop game environment with additional rules, define several agent personas, and conduct zero-shot experiments. They report persona-based score averages, win/draw rates against the 70B model, distributions of a 'planning depth' measure defined as the number of generated steps per move, and reasoning-type classifications produced by Gemini. They conclude that larger models exhibit deeper planning but not necessarily higher win rates.","tokens_in":7851,"tokens_out":5010,"duration_ms":50491,"significance":"The paper's main contribution is a new, fully observable, deterministic game environment for probing LLM planning, complete with source code. The descriptive results on win rates and reasoning-type distributions, if properly qualified, could be of interest to the community studying LLM strategic behavior. However, the central claim about planning depth is built on an unvalidated proxy (generated step count) that confounds verbosity with planning, and the quantitative comparisons lack statistical support. As a result, the paper currently reads as a promising pilot study rather than a definitive evaluation; with additional validation and statistical rigor, the underlying setup could support stronger claims.","major_comments":[{"comment":"The central claim that \"larger models possess not only better language understanding but also more advanced planning capabilities\" (Section V.C) rests entirely on the measure of \"planning depth\" defined as the number of steps generated per move. This proxy is never validated against any external notion of planning, such as the depth of game-tree search, the number of legal moves considered, or the quality of the resulting moves, nor is output verbosity controlled for. Since larger instruction-tuned models are known to produce longer rationales, the observed step-count differences are equally consistent with over-explanation as with deeper planning; the claim as stated is therefore unsupported.","section":"V.C, Figure 3"},{"comment":"The cross-architecture comparison in Table II lacks statistical support: no error bars, confidence intervals, or significance tests are reported, and the number of games for these matches is not stated (the 50-game figure appears only for the persona experiments in Section V.A). The observed differences, such as 38% vs. 34% win rate, are small and could easily arise from sampling noise, so the conclusion that \"model scale alone is not a sufficient predictor of success\" (Section V.B) is not established. In addition, the table setup is ambiguous: the 70B model appears both as the reference opponent and as a tested agent (with † denoting first player), so it is unclear against whom the 70B row is measured and whether the 3B and 8B models always play second.","section":"V.B, Table II"},{"comment":"The reasoning-type distributions in Table III are produced by Gemini in a zero-shot setting with no human reliability check, inter-annotator agreement, or comparison against a gold standard. These labels are then used to support the claim that larger models show \"reduced impulsiveness\" and \"more coherent and consistent strategic justifications\" (Section V.D). Without classifier validation, the percentages may simply reflect a bias toward labeling longer or more fluent text as \"LONG TERM STRATEGY,\" which would make this analysis circular with the unvalidated planning-depth metric.","section":"V.D, Table III"},{"comment":"The persona comparison in Table I reports average scores without error bars or significance tests, despite the differences between agents being small (e.g., 23.5 vs. 22.4 average points). The claim that \"strategies with better long-term balance tend to perform more consistently across full matches\" (Section V.A) is therefore not statistically supported, and the phase-wise breakdown (EGE/MGE/LGE) is not accompanied by any measure of variance.","section":"V.A, Table I"}],"minor_comments":[{"comment":"The caption states that the data are aggregated across 50 games played against Llama-3.3-70B-Instruct, but panel (c) shows the 70B model itself; clarify who the opponent is for each panel and whether the 70B's own step counts are measured in games against a different opponent.","section":"Figure 3"},{"comment":"The term \"number of steps\" is never precisely defined; specify whether a step is a sentence, a line, a token, or a thought block in the generated reasoning trace.","section":"V.C"},{"comment":"The Limitations section states that \"the models often struggle with multi-step planning and complex capture chains, limiting their strategic depth,\" which appears to contradict the conclusion that larger models exhibit \"superior planning depth\" (Section V.C). Please reconcile these statements.","section":"Limitations"},{"comment":"Generation details such as temperature, max tokens, decoding strategy, and number of random seeds are not reported, which limits reproducibility despite the provided source-code link.","section":"Appendix B"},{"comment":"Reference [18] (\"Machine theory of mind\") is cited as an example of \"structured planning environments and strategic dialogue systems,\" but that paper is about theory-of-mind inference; either replace the reference or adjust the sentence.","section":"Section II"},{"comment":"The model name \"Gemini-2.0-Flask\" appears to be a typo; should it be \"Gemini-2.0-Flash\"?","section":"V.D"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.CL, but its central claim needs rework. The authors should either replace the 'planning depth' terminology with something like 'generated reasoning length' and avoid causal claims, or validate the metric against search depth and move quality. I would also encourage them to report confidence intervals and clarify the experimental design of Table II. The novelty is modest but sufficient for a workshop-style paper; the current manuscript needs a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful thing: this paper introduces a new LLM testbed, and Ô Ăn Quan is a sensible choice. It is fully observable and deterministic, unlike the Werewolf/Avalon/Spyfall line, and it demands rule compliance and multi-step planning. The authors ship code, evaluate three Llama sizes with several personas, and report win rates, planning step counts, and reasoning type distributions. That is a legitimate descriptive contribution, and the headline observation that the 8B model beats the 70B in the reported matchups is a useful antidote to the 'bigger is better' assumption.\n\nThe soft spot is the planning-depth metric. Defining it as the number of generated steps per move and then concluding that the 70B model possesses more advanced planning capabilities is not supported. The metric is just output length, and larger instruction-tuned models tend to produce longer rationales. The paper never validates it against actual search depth, move quality, or any external notion of planning. The stress-test note is right to flag the 70B's own row in Figure 3: the games are aggregated 'against Llama-3.3-70B-Instruct,' but for the 70B row we do not know the opponent or player order, which could confound the comparison. The reasoning-type classification also relies on Gemini zero-shot labels with no human reliability check, so it cannot independently corroborate the metric.\n\nBeyond that, the persona and win-rate differences in Table I are small, and there are no error bars or statistical tests, so the strategy conclusions are fragile. The paper's limitations section does acknowledge some of this—it explicitly says the evaluation is based on reasoning labels rather than full gameplay outcomes—which is honest, but the Section V.C claim is presented without that caution.\n\nWho benefits? Researchers working on LLM planning benchmarks will find the game environment and the code useful. This is a data point, not a result that changes practice. It deserves a serious referee, but the planning-depth analysis needs major revision: either drop the claim or validate the metric. I would recommend conditional acceptance with a request for error bars and a validation experiment.","headline":"New game testbed with a plausible central observation, but the planning-depth metric is unvalidated and the scale-based conclusions go beyond the evidence.","tokens_in":8407,"tokens_out":3751,"would_cite":false,"duration_ms":38845,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that in the game Ô Ăn Quan, larger LLMs generate deeper planning traces and more long-term reasoning, while model size does not predict wins against smaller models.","keywords":["large language models","Ô Ăn Quan","multi-step planning","decision-making","strategic reasoning","persona-based agents","game-theoretic AI"],"falsifier":"Inspect the reasoning traces and count how many distinct future board states the model considers before acting. If long traces mention few or no alternative board configurations, or if instructing a model to be more verbose raises its planning-depth score without improving the quality of its moves, the proxy is invalid and the scale-to-planning claim loses its support.","tokens_in":7424,"feed_emoji":"🎲","tokens_out":6555,"duration_ms":66934,"temperature":0.7,"pith_summary":"This paper asks whether large language models can plan over multiple steps in a fully observable, deterministic game, and uses the traditional Vietnamese board game Ô Ăn Quan as the testbed. The authors build LLM agents with personas such as balanced, defensive, aggressive, and risk-taking, then measure win rates, planning depth (steps generated per move), and the share of short-term vs. long-term reasoning. Their central claim is that larger models plan deeper and show more long-term strategic reasoning, yet this depth does not reliably convert into more wins. They also find that balanced and defensive strategies outperform aggressive ones over a full game and that planning depth declines for all models as the game progresses. The paper itself notes the evaluation is zero-shot, with a simplified environment, and that conclusions rest on reasoning labels rather than full gameplay outcomes.","feed_headline":"Bigger LLMs plan deeper but still lose the Ô Ăn Quan game","feed_subtitle":"Testing 3B, 8B, and 70B models shows scale deepens reasoning but not winning—a caution for LLM benchmarks.","key_machinery":"The central object is a closed-loop agent framework built on the Ô Ăn Quan game: each turn, the model receives the current board state, the history of previous moves, the rule set (including extra rules like the Immature Mandarin and forced capture chains), and a natural-language persona prompt, and returns a rationale plus an action of position and direction. The load-bearing mechanism is the planning-depth metric, defined as the number of steps the model generates per move, combined with a zero-shot classifier that assigns each rationale to short-term gain, long-term strategy, or ambiguous. Together these measurements support the paper's claims about how scale changes planning and strategy.","core_discovery":"The paper reports that Llama-3.3-70B-Instruct obtains the highest planning depth, with some moves exceeding 250 generated steps, followed by Llama-3.1-8B-Instruct at around 150 steps, while Llama-3.2-3B-Instruct mostly stays below 100 steps. A Gemini-2.0-Flash classifier labels the models' rationales and shows the 70B model favoring long-term strategy (64.86%) and the 3B model favoring short-term gain (70.18%), with the 8B model in between. In head-to-head games, however, the 8B model beats the 70B model 38% to 34%, and the 70B does not dominate smaller opponents. The paper concludes that model scale tracks planning depth and long-horizon reasoning but not competitive success, and that the game's early rounds are where the models plan most deeply.","pith_inferences":["The paper's planning-depth proxy is text-based; a natural extension would count the distinct board states a model actually evaluates, or test whether forcing longer reasoning through prompting alone raises the metric without improving moves.","Pitting the LLM agents against a known-state-space search baseline, such as minimax or Monte Carlo tree search, would reveal whether the depth-versus-outcome dissociation is specific to LLMs or inherent to the game's complexity.","If personas steer planning style, the same model should produce measurably different move distributions under aggressive versus defensive prompts; that is a direct test of the persona mechanism the paper assumes."],"forward_implications":["If larger models genuinely plan deeper, then scale should matter most for long-horizon tasks even when short-run outcome metrics are flat.","The dissociation between planning depth and win rate implies that game outcomes alone are an incomplete benchmark for LLM planning ability.","The universal decline in planning depth as games progress suggests planning effort is state-dependent and may weaken as uncertainty or constraints grow.","The stronger full-game results of balanced and defensive personas argue for testing multiple persona strategies rather than a single style."],"supporting_citations":[{"why":"Supplies the Llama 3 herd of models evaluated in the experiments.","marker":"[4]"},{"why":"Supplies the Gemini family of models used zero-shot to classify reasoning traces.","marker":"[5]"},{"why":"Provides the chain-of-thought prompting basis for eliciting step-by-step reasoning.","marker":"[19]"},{"why":"Provides the ReAct formulation that shapes the agent's rationale-and-action loop.","marker":"[20]"},{"why":"Provides the persona-based agent modeling template used for the different player strategies.","marker":"[21]"},{"why":"Establishes the prior benchmark of LLM gaming ability in multi-agent environments that this work extends to a deterministic game.","marker":"[10]"}],"fun_headline_variants":["LLM scale boosts planning depth but not game wins","In Ô Ăn Quan, 8B model beats 70B despite less planning","Deep planning doesn't make LLMs win Ô Ăn Quan","Scale-up LLM plans deeper, still loses Ô Ăn Quan","In Ô Ăn Quan, bigger LLMs plan deeper but don't win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the number of steps a model writes before choosing a move measures how deeply it plans, with no check that longer output corresponds to wider or deeper search rather than verbosity or repeated deliberation.","fun_headline_variants_meta":{"raw":{"variants":["LLM scale boosts planning depth but not game wins","In Ô Ăn Quan, 8B model beats 70B despite less planning","Deep planning doesn't make LLMs win Ô Ăn Quan","Scale-up LLM plans deeper, still loses Ô Ăn Quan","In Ô Ăn Quan, bigger LLMs plan deeper but don't win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2929,"prompt_tokens":935,"completion_tokens":1994,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1889}},"tokens_in":551,"tokens_out":1994,"duration_ms":15327,"temperature":1.0,"reasoning_tokens":1889,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:02:54.719453+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the reasoning traces and count how many distinct future board states the model considers before acting. If long traces mention few or no alternative board configurations, or if instructing a model to be more verbose raises its planning-depth score without improving the quality of its moves, the proxy is invalid and the scale-to-planning claim loses its support.","supporting_citations":[{"cited_title":"Generative agents: Interactive simulacra of human behavior,","cited_arxiv_id":null,"evidence_quote":"Provides the persona-based agent modeling template used for the different player strategies."}],"review_version":1}