{"id":"b87c9653-9bbc-4fb0-831e-9dabe202f8e7","arxiv_id":"2506.18928","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"In a zero-sum horse-race game with provided random choices, stronger LLMs randomize more when explicitly hinted and weaker models stay deterministic, but the equilibrium claim and metrics are not reliably established.","lead":"This paper tests whether large language models can decide to randomize when randomness is the optimal move, using a horse-race game where the system supplies random choices. It reports that weaker models stay deterministic while stronger models randomize when told to, but the game's theoretical basis and measurement design have serious flaws.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The uniform-random equilibrium premise is actually correct, so the reader's main attack misses; the load-bearing flaw is that the Bayes-factor metric measures compliance with the system's suggestion, not strategic randomization.","rationale":"The reader's stated weakest assumption is that the paper's Section 3.1 claim about uniform randomization being the Nash equilibrium is false. This is not correct. For N horses, a uniformly random permutation guarantees expected reward N/2 against any fixed opponent sequence because each round's marginal distribution is uniform over the remaining horses; since total reward per round is 1, the game is constant-sum and no player can force more than N/2 against this strategy. So the hinted prompt teaches an actually optimal strategy, and the Bayes factor reference point is not wrong on those grounds.\n\nHowever, the paper still fails to support its central claim as stated. The Bayes factor in Section 3.3 is constructed so that H1 is favored whenever the model matches the system-provided random suggestion. This conflates the cognitive decision to randomize with mere acceptance of a system suggestion, which is especially problematic under the hinted prompt where the model is directly told that uniform randomization is the equilibrium. A model that follows instructions, or that simply trusts the system's suggestion, will score high without any strategic reasoning. The paper also reports only aggregate win/loss matrices and averaged Bayes factors over K=10 tournaments, with no error bars, no seed variation, and no control for base-rate suggestion-following. The claim that strong models 'converge toward equilibrium play when facing peers' implies learning over repeated tournaments, but the Limitations section explicitly says LLMs have no reinforcement learning mechanism and rely solely on prompt context; no learning curve or per-tournament analysis is provided.\n\nBecause these are empirical and methodological gaps rather than a false theoretical premise, I would keep the reader's REJECT verdict, but for different reasons. The central finding as written is not reliably demonstrated. A control condition and per-tournament analysis could potentially rehabilitate a weaker version of the claim, which is why the concrete test focuses on that control.","tokens_in":5172,"tokens_out":8149,"duration_ms":98045,"concrete_test":"Add an instruction-following control condition: replace the hinted prompt's game-theoretic rationale with a matched-length instruction to use the system-provided random horse about 65% of the time and otherwise choose uniformly, with no mention of Nash equilibrium or randomization as a strategy. Run the same K=10 protocol and compare the resulting log Bayes factors to the hinted-prompt condition. If strong models show equally high Bayes factors under this control, the hinted-prompt results reflect general compliance rather than strategic randomization; if they do not, the randomization interpretation survives this particular confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest_assumption targets Section 3.1, claiming uniform random selection is not the Nash equilibrium. That attack does not land: against any fixed opponent permutation, a uniformly random permutation gives expected reward N/2 because each round's marginal is uniform, and the constant-sum structure means no unilateral deviation improves on this. Uniform randomization is a genuine Nash equilibrium. The real soft spot is the empirical metric in Section 3.3. H1 in the Bayes factor is defined as 'tends to use the system-provided random option,' so the measure rewards any tendency to follow the system suggestion, regardless of whether the model recognizes that uniform play is strategically optimal. Under the hinted prompt, the system explicitly states that uniform randomization is the Nash equilibrium, so strong models producing high Bayes factors is exactly what generic instruction-following predicts. The paper provides no control for base-rate compliance, no error bars on the K=10 tournament results, and the Limitations section (Section 5.4) concedes that LLMs 'lack mechanisms for reinforcement learning during gameplay' and 'rely solely on prompt context,' yet the title and conclusions claim learning 'through experience' and convergence to equilibrium. No per-tournament learning curve or evidence that prior tournaments are even in context is reported. Thus the central empirical claim of meaningful strategic randomization and adaptation is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a zero-sum game inspired by the Tian Ji horse race in which the authors claim the Nash equilibrium is uniform random play over remaining horses. Five LLMs are evaluated in round-robin tournaments under three prompt variants (framed, neutral, hinted), with system-generated random choices offered to the models each round. The authors measure win/loss outcomes and a log Bayes factor designed to distinguish randomized from deterministic play. They conclude that stronger models randomize under explicit hints, exploit weaker opponents with deterministic strategies, and converge toward equilibrium play when facing peers. The paper includes a public code repository and fixed random seeds.","tokens_in":5465,"tokens_out":6569,"duration_ms":71888,"significance":"If the empirical claims were established, the paper would make a useful contribution by separating the decision to randomize from the mechanical generation of randomness, a distinction that prior work has often blurred. The game is a clean and novel testbed, and the public implementation and seeded reproducibility are strengths. However, the central measure does not actually isolate strategic randomization, the theoretical anchor is asserted rather than demonstrated, and the claims about learning from experience are not supported by the reported experiments. The paper is therefore not yet a reliable account of LLMs' strategic randomization abilities.","major_comments":[{"comment":"The Bayes factor metric measures compliance with the system-provided suggestion, not randomization. Under H1, the likelihood is high exactly when the model matches the suggested random horse; under H2, the likelihood is uniform over all legal choices. A model that randomizes uniformly on its own, independently of the system suggestion, has exactly the same expected likelihood under H1 and H2 and therefore yields a log Bayes factor of 0. A model that blindly echoes the suggested horse receives a large positive log Bayes factor. The paper repeatedly interprets high Bayes factors as deliberate strategic randomization, but the metric is essentially an instruction-following score. To support the central claim, the authors should either redefine the hypotheses in terms of marginal uniformity of the chosen actions, or add control conditions for non-strategic suggestion-following, and report the results separately.","section":"Section 3.3, Eq. (1)"},{"comment":"The Nash equilibrium claim is load-bearing because the hinted prompt explicitly tells models that uniform random play is optimal, and the Bayes factor interpretation depends on that premise, yet the manuscript provides no proof. The text says the equilibrium was found by numerically solving a Bellman equation under the assumption that the opponent plays the maximal entropy strategy, with details deferred to a supplementary document that is not included. The claim is in fact true, but the reader cannot verify it from the manuscript. A short analytic argument, such as noting that a uniformly random permutation gives expected reward N/2 against any fixed opponent permutation, would settle the matter and should be included.","section":"Section 3.1"},{"comment":"The conclusions about convergence and learning through experience are not supported by the experimental design. Each model pair plays only K=10 tournaments, yet the win/loss matrices and Bayes factor matrices are reported without error bars, confidence intervals, or significance tests. Moreover, the interaction loop in Algorithm 1 gives no indication that the models receive any information about previous tournaments, and Section 5.4 concedes that LLMs rely solely on prompt context and lack reinforcement-learning mechanisms. The abstract and conclusion nonetheless claim learning 'through experience' and convergence toward equilibrium against peers. The authors must either provide per-tournament learning curves and specify how prior games enter the context, or remove the adaptation and learning claims.","section":"Section 5.1 and Algorithm 1"}],"minor_comments":[{"comment":"The first sentence of the Introduction begins with lowercase 'large language models'; the capitalization should be fixed.","section":"Introduction, first sentence"},{"comment":"The human experiment is described anecdotally with no sample size, protocol, or quantitative results. It should either be moved to a clearly labeled pilot with methodological details or removed.","section":"Section 3.1, paragraph 2"},{"comment":"The word 'williness' should be 'willingness'.","section":"Section 5.2"},{"comment":"The game is described as zero-sum with 'expected utility of zero,' but the total reward per tournament is N, so the zero-sum framing requires subtracting N/2 or speaking of payoff differences. The wording should be made precise.","section":"Section 3.1"},{"comment":"The algorithm says to prompt models with game background information, but the prompt variant (framed, neutral, hinted) is not an explicit input to the algorithm. This should be clarified.","section":"Algorithm 1, line 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is attractive, and the equilibrium premise is actually correct, but the empirical measure is not doing the work the paper claims. The missing supplementary proof and the absence of any learning curve are the main substantive gaps. I would encourage the editor to treat the revision as an opportunity to add a suggestion-following control and to reframe the title and conclusions if the learning claim cannot be supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper has a real idea—separating the decision to randomize from the mechanics of random generation—and the empirical comparison across five LLMs and three prompt styles is new. But the central claim that stronger models 'randomize strategically' is not supported by the metric. The Bayes factor in Section 3.3 defines H1 as 'tends to use the system-provided random option.' That is not randomization, that's compliance. Under the hinted prompt the system explicitly tells the model that uniform play is the equilibrium, so high Bayes factors for strong models are exactly what generic instruction-following predicts.\n\nThe reader's main attack—that uniform random is not the Nash equilibrium—does not land. For this game, playing a uniformly random permutation guarantees expected payoff N/2 against any opponent, and the game is zero-sum symmetric, so that is the value. The stress-test note is right. The paper's theoretical anchor is fine, at least for the claim that uniform play is optimal.\n\nWhat's actually good: the decoupling design is a genuine improvement over Vidler and Walsh, and the win/loss matrices across prompt conditions are a useful descriptive dataset. The paper is clearly written and the limitations section is honest.\n\nSoft spots in proportion: (1) The metric conflates following a suggestion with strategic reasoning; you need a control where the system provides a non-equilibrium random suggestion, or compare against a base rate of accepting system suggestions on neutral tasks. (2) K=10 tournaments per pair, no error bars or significance tests. (3) The title and abstract claim adaptation 'through experience,' but no learning curve or evidence that prior tournaments are in context; the limitations section concedes LLMs lack RL mechanisms. So the empirical conclusions are weaker than claimed.\n\nWho's this for: people building LLM evaluation benchmarks for strategic reasoning, especially anyone interested in randomization. It deserves a serious referee—the design seed is worth preserving—but it needs substantial revision before the claims can stand.","headline":"The decoupling idea is worth a look, but the metric measures instruction-following, not strategic randomization; the reader's equilibrium attack misses.","tokens_in":5914,"tokens_out":2513,"would_cite":false,"duration_ms":26642,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Whether an LLM deliberately randomizes depends on model strength and framing: weak models stay deterministic under every prompt, while strong models randomize on explicit hints and exploit weak opponents instead.","keywords":["strategic randomization","Nash equilibrium","mixed strategies","large language models","game theory","Bayes factor","prompt engineering","horse-race game"],"falsifier":"Re-derive the game's equilibrium directly: solve the symmetric zero-sum dynamic program for $N=7$ (or even $N=3$) and check whether any policy achieves strictly positive expected payoff against uniform randomization; if such a policy exists, or if uniform play itself is beatable, then the paper's equilibrium premise is false and its measurements of when a model should randomize are miscalibrated.","tokens_in":4996,"feed_emoji":"🎲","tokens_out":12939,"duration_ms":114049,"temperature":0.7,"pith_summary":"This paper asks whether large language models know when to randomize in a strategic game, treating that as a separate cognitive skill from being able to generate random-looking outputs. It introduces a zero-sum game with seven horses per player, inspired by the ancient Chinese horse-race story, and argues that its Nash equilibrium is uniform random selection among the remaining horses. Each round the system privately suggests a uniformly random move; because the randomness is supplied externally, a model's willingness to use that suggestion measures its deliberate choice to randomize, quantified with a log Bayes factor. Across five LLMs and three prompt styles, the paper reports that weak models stay deterministic everywhere, strong models randomize when the equilibrium is spelled out, and strong models switch back to deterministic exploitation against weaker opponents.","feed_headline":"Stronger LLMs randomize on hint; weaker models stay rigid","feed_subtitle":"A horse-race game with built-in random suggestions shows the gap is knowing when to randomize, not generating it.","key_machinery":"The load-bearing mechanism is the decoupling harness: each round the system privately samples a horse uniformly from the model's remaining set and offers it as a suggestion, leaving the model free to adopt it or ignore it. Because the randomness is provided externally, the model's acceptance rate is a direct measure of the deliberate decision to randomize, and the log Bayes factor $B_{12}=\\sum_i \\log P[\\text{choice}_i\\mid H_1]-\\sum_i \\log P[\\text{choice}_i\\mid H_2]$ turns that rate into a quantitative spectrum between full randomization and full determinism, with $H_1$ penalized by a conservative $\\eta=0.4$ deviation rate. The game itself is the second piece: a deliberately opaque symmetric contest whose asserted equilibrium is maximal entropy, giving the evaluation a correct target that is not obvious to human intuition or to weaker models.","core_discovery":"The paper's discovery is that strategic randomization in LLMs is an isolable cognitive faculty that varies with model strength and prompt content. The game it uses is a symmetric, zero-sum tournament: both players have $N=7$ distinct-speed horses and, over seven rounds, simultaneously reveal one unused horse each; the fastest revealed horse scores a point, split equally on ties, and the highest total wins. The authors claim that the unique Nash equilibrium is the maximal-entropy policy, uniform random choice among remaining horses, supported by a numerical Bellman solve. They pit five LLMs against each other in round-robin tournaments under framed, neutral, and hinted prompts, and score each action sequence with a log Bayes factor comparing the hypothesis that the model follows the system's random suggestion against the hypothesis that it follows its own deterministic plan. The headline pattern is consistent: weak models never randomize, strong models randomize only when told to, and strong models abandon randomization when facing a weak, exploitable opponent.","pith_inferences":["We infer that the protocol transfers to any game with a known mixed equilibrium; applying it to matching pennies or rock-paper-scissors would let researchers isolate framing effects against a fully proven equilibrium, sidestepping the numerical-equilibrium assumption in this paper.","We infer that a model's Bayes factor under the hinted prompt is a proxy for how well its internal reasoning approximates mixed-strategy play; comparing that proxy across different values of $N$ would separate computational overload from conceptual misunderstanding.","We infer that the win-rate asymmetry against weak opponents predicts a testable pattern: if strong models knowingly exploit weak deterministic play, eliciting their verbal confidence or rationale would show explicit reference to the opponent's predictability rather than to equilibrium reasoning.","We infer that a natural extension is a version with imperfect information or asymmetric horse sets; if randomization acceptance drops as state complexity rises, that would suggest capacity limits in maintaining mixed strategies rather than a categorical inability."],"forward_implications":["The decision to randomize can be measured apart from random-number generation, so future LLM evaluations can reuse the system-suggestion plus Bayes-factor protocol without relying on models to generate their own chaos.","Prompt design is a real lever on strategic play: a single sentence stating the Nash equilibrium changes strong models' behavior, so deployment contexts should treat prompt controls as strategic variables.","Matchup asymmetry matters: strong models may appear non-strategic when measured against weaker opponents, so evaluation protocols should control opponent strength when inferring randomized-play competence.","Because weak models did not respond to hints, the gap is unlikely to be a surface instruction-following failure; it points to limits in the model's underlying reasoning or training."],"supporting_citations":[{"why":"Documents that LLMs generate poor uniform randomness even with temperature tuning, motivating the paper's decoupling of randomness generation from the decision to randomize.","marker":"Vidler and Walsh [2025]"},{"why":"Supplies the game-theoretic benchmark the paper positions against, arguing that prior evaluations overlook randomization as a distinct faculty.","marker":"Duan et al. [2024]"},{"why":"A diverse strategic game benchmark for LLM agents whose heuristic metrics the paper distinguishes from equilibrium-grounded evaluation.","marker":"Tang et al. [2025]"}],"fun_headline_variants":["LLMs randomize only when told by hints; weak models never","Horse-race game reveals LLMs' randomization gap","Strong LLMs randomize on hint, exploit weak opponents","Strategic randomization in LLMs depends on model strength","Hinted LLMs randomize, but strong ones exploit weak foes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire benchmark rests on the claim that uniformly random play over remaining horses is the Nash equilibrium of this game; the paper asserts this on the strength of a numerical Bellman solve and supplies no closed-form proof, and if the claim fails the hinted prompt teaches a suboptimal strategy and the Bayes-factor reference point loses its meaning.","fun_headline_variants_meta":{"raw":{"variants":["LLMs randomize only when told by hints; weak models never","Horse-race game reveals LLMs' randomization gap","Strong LLMs randomize on hint, exploit weak opponents","Strategic randomization in LLMs depends on model strength","Hinted LLMs randomize, but strong ones exploit weak foes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2748,"prompt_tokens":960,"completion_tokens":1788,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":1705}},"tokens_in":576,"tokens_out":1788,"duration_ms":12548,"temperature":1.0,"reasoning_tokens":1705,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:07:45.281043+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-derive the game's equilibrium directly: solve the symmetric zero-sum dynamic program for $N=7$ (or even $N=3$) and check whether any policy achieves strictly positive expected payoff against uniform randomization; if such a policy exists, or if uniform play itself is beatable, then the paper's equilibrium premise is false and its measurements of when a model should randomize are miscalibrated.","supporting_citations":[],"review_version":2}