REVIEW 3 major objections 5 minor 3 references
Do LLMs Know When to Flip a Coin? Strategic Randomization through Reasoning and Experience
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Whether an LLM deliberately randomizes depends on model strength and framing: weak models stay deterministic under every prompt, while strong models randomize on explicit hints and exploit weak opponents instead.
desk verdict The decoupling idea is worth a look, but the metric measures instruction-following, not strategic randomization; the reader's equilibrium attack misses. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the decoupling harness: each round the system privately samples a horse uniformly from the model's remaining set and offers it as a suggestion, leaving the model free to adopt it or ignore it. Because the randomness is provided externally, the model's acceptance rate is a direct measure of the deliberate decision to randomize, and the log Bayes factor $B_{12}=\sum_i \log P[\text{choice}_i\mid H_1]-\sum_i \log P[\text{choice}_i\mid H_2]$ turns that rate into a quantitative spectrum between full randomization and full determinism, with $H_1$ penalized by a conservative $\eta=0.4$ deviation rate. The game itself is the second piece: a deliberately opaque symmetric contest whose asserted equilibrium is maximal entropy, giving the evaluation a correct target that is not obvious to human intuition or to weaker models.
What would settle it
Re-derive the game's equilibrium directly: solve the symmetric zero-sum dynamic program for $N=7$ (or even $N=3$) and check whether any policy achieves strictly positive expected payoff against uniform randomization; if such a policy exists, or if uniform play itself is beatable, then the paper's equilibrium premise is false and its measurements of when a model should randomize are miscalibrated.
Extended reading notes
Core claim
The paper's discovery is that strategic randomization in LLMs is an isolable cognitive faculty that varies with model strength and prompt content. The game it uses is a symmetric, zero-sum tournament: both players have $N=7$ distinct-speed horses and, over seven rounds, simultaneously reveal one unused horse each; the fastest revealed horse scores a point, split equally on ties, and the highest total wins. The authors claim that the unique Nash equilibrium is the maximal-entropy policy, uniform random choice among remaining horses, supported by a numerical Bellman solve. They pit five LLMs against each other in round-robin tournaments under framed, neutral, and hinted prompts, and score each action sequence with a log Bayes factor comparing the hypothesis that the model follows the system's random suggestion against the hypothesis that it follows its own deterministic plan. The headline pattern is consistent: weak models never randomize, strong models randomize only when told to, and strong models abandon randomization when facing a weak, exploitable opponent.
Load-bearing premise
The entire benchmark rests on the claim that uniformly random play over remaining horses is the Nash equilibrium of this game; the paper asserts this on the strength of a numerical Bellman solve and supplies no closed-form proof, and if the claim fails the hinted prompt teaches a suboptimal strategy and the Bayes-factor reference point loses its meaning.
Editorial extensions
If this is right
- The decision to randomize can be measured apart from random-number generation, so future LLM evaluations can reuse the system-suggestion plus Bayes-factor protocol without relying on models to generate their own chaos.
- Prompt design is a real lever on strategic play: a single sentence stating the Nash equilibrium changes strong models' behavior, so deployment contexts should treat prompt controls as strategic variables.
- Matchup asymmetry matters: strong models may appear non-strategic when measured against weaker opponents, so evaluation protocols should control opponent strength when inferring randomized-play competence.
- Because weak models did not respond to hints, the gap is unlikely to be a surface instruction-following failure; it points to limits in the model's underlying reasoning or training.
Reading between the lines
- We infer that the protocol transfers to any game with a known mixed equilibrium; applying it to matching pennies or rock-paper-scissors would let researchers isolate framing effects against a fully proven equilibrium, sidestepping the numerical-equilibrium assumption in this paper.
- We infer that a model's Bayes factor under the hinted prompt is a proxy for how well its internal reasoning approximates mixed-strategy play; comparing that proxy across different values of $N$ would separate computational overload from conceptual misunderstanding.
- We infer that the win-rate asymmetry against weak opponents predicts a testable pattern: if strong models knowingly exploit weak deterministic play, eliciting their verbal confidence or rationale would show explicit reference to the opponent's predictability rather than to equilibrium reasoning.
- We infer that a natural extension is a version with imperfect information or asymmetric horse sets; if randomization acceptance drops as state complexity rises, that would suggest capacity limits in maintaining mixed strategies rather than a categorical inability.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a zero-sum game inspired by the Tian Ji horse race in which the authors claim the Nash equilibrium is uniform random play over remaining horses. Five LLMs are evaluated in round-robin tournaments under three prompt variants (framed, neutral, hinted), with system-generated random choices offered to the models each round. The authors measure win/loss outcomes and a log Bayes factor designed to distinguish randomized from deterministic play. They conclude that stronger models randomize under explicit hints, exploit weaker opponents with deterministic strategies, and converge toward equilibrium play when facing peers. The paper includes a public code repository and fixed random seeds.
Significance. If the empirical claims were established, the paper would make a useful contribution by separating the decision to randomize from the mechanical generation of randomness, a distinction that prior work has often blurred. The game is a clean and novel testbed, and the public implementation and seeded reproducibility are strengths. However, the central measure does not actually isolate strategic randomization, the theoretical anchor is asserted rather than demonstrated, and the claims about learning from experience are not supported by the reported experiments. The paper is therefore not yet a reliable account of LLMs' strategic randomization abilities.
major comments (3)
- [Section 3.3, Eq. (1)] The Bayes factor metric measures compliance with the system-provided suggestion, not randomization. Under H1, the likelihood is high exactly when the model matches the suggested random horse; under H2, the likelihood is uniform over all legal choices. A model that randomizes uniformly on its own, independently of the system suggestion, has exactly the same expected likelihood under H1 and H2 and therefore yields a log Bayes factor of 0. A model that blindly echoes the suggested horse receives a large positive log Bayes factor. The paper repeatedly interprets high Bayes factors as deliberate strategic randomization, but the metric is essentially an instruction-following score. To support the central claim, the authors should either redefine the hypotheses in terms of marginal uniformity of the chosen actions, or add control conditions for non-strategic suggestion-following, and report the results separately.
- [Section 3.1] The Nash equilibrium claim is load-bearing because the hinted prompt explicitly tells models that uniform random play is optimal, and the Bayes factor interpretation depends on that premise, yet the manuscript provides no proof. The text says the equilibrium was found by numerically solving a Bellman equation under the assumption that the opponent plays the maximal entropy strategy, with details deferred to a supplementary document that is not included. The claim is in fact true, but the reader cannot verify it from the manuscript. A short analytic argument, such as noting that a uniformly random permutation gives expected reward N/2 against any fixed opponent permutation, would settle the matter and should be included.
- [Section 5.1 and Algorithm 1] The conclusions about convergence and learning through experience are not supported by the experimental design. Each model pair plays only K=10 tournaments, yet the win/loss matrices and Bayes factor matrices are reported without error bars, confidence intervals, or significance tests. Moreover, the interaction loop in Algorithm 1 gives no indication that the models receive any information about previous tournaments, and Section 5.4 concedes that LLMs rely solely on prompt context and lack reinforcement-learning mechanisms. The abstract and conclusion nonetheless claim learning 'through experience' and convergence toward equilibrium against peers. The authors must either provide per-tournament learning curves and specify how prior games enter the context, or remove the adaptation and learning claims.
minor comments (5)
- [Introduction, first sentence] The first sentence of the Introduction begins with lowercase 'large language models'; the capitalization should be fixed.
- [Section 3.1, paragraph 2] The human experiment is described anecdotally with no sample size, protocol, or quantitative results. It should either be moved to a clearly labeled pilot with methodological details or removed.
- [Section 5.2] The word 'williness' should be 'willingness'.
- [Section 3.1] The game is described as zero-sum with 'expected utility of zero,' but the total reward per tournament is N, so the zero-sum framing requires subtracting N/2 or speaking of payoff differences. The wording should be made precise.
- [Algorithm 1, line 2] The algorithm says to prompt models with game background information, but the prompt variant (framed, neutral, hinted) is not an explicit input to the algorithm. This should be clarified.
Circularity Check
The randomization metric is defined as following the system's random suggestion, and the hinted prompt explicitly instructs that same behavior, so the headline result that stronger models randomize under hints is largely a compliance effect by construction.
-
self definitional
[Section 3.2 (Hinted Prompt) and Section 3.3 (Bayes Factor Computation)]
"H 1: Randomized Strategy—The agent chooses to randomize and tends to use the system-provided random option. ... Hinted Prompt: Explicitly states that uniform random selection over remaining horses is the Nash equilibrium."
The dependent variable 'randomized strategy' is operationally defined as 'tends to use the system-provided random option.' The hinted prompt's only strategic content is the assertion that uniform random selection is the Nash equilibrium—i.e., the instruction to do exactly what H1 rewards. Consequently, a model that follows the explicit hint will produce a high log Bayes factor by construction. The paper's headline finding that stronger models 'exhibit increased randomization under explicit hints' therefore reduces to the claim that stronger models follow an explicit instruction that is encoded verbatim in the metric.
-
renaming known result
[Section 3.3 (Bayes Factor Computation) and Section 5.2 (Bayes Factor Analysis)]
"H 2: Strategic Deviation—The agent adheres to its own internal strategy, disregarding the system-provided random option entirely; in this sense, its choices exhibit a form of relative randomness with respect to the reference provided by the system’s suggestion. ... Less capable models show low Bayes factors regardless of prompt, indicating deterministic behavior."
Under the paper's own equation, P[choice_i|H2] = 1/|legal choices|, a uniform random baseline. A low Bayes factor therefore means only that choices did not preferentially match the system-provided suggestion; it does not distinguish a deterministic strategy from an independent uniform randomizer. Yet the results section reads low Bayes factors as 'indicating deterministic behavior.' This relabels the complement of the H1 definition—not following the system's random option—as the paper's target conclusion of determinism. The claim that weaker models 'remain deterministic' is thus an interpretive renaming of the metric's input rather than an independently measured property.
full rationale
The game-theoretic equilibrium derivation in Section 3.1 is a numerical self-consistency check rather than a circular step, and the win/loss matrices provide an independent behavioral measure that is not definitionally tied to the randomization metric. However, the paper's central randomization claim is structurally aligned with its measurement instrument: H1 is defined as using the system-provided random option, and the hinted prompt explicitly tells the model that uniform random selection is the Nash equilibrium. High Bayes factors under the hinted condition are therefore expected from any instruction-following model, making the headline 'increased randomization under explicit hints' largely a compliance effect. The secondary move of interpreting low Bayes factors as determinism is also a relabeling, since H2 is a uniform random likelihood. No load-bearing self-citations appear. The paper contains independent elements, so it is not wholly circular, but the main construct-level conclusion reduces by construction to instruction-following and metric relabeling.
Assumptions & free parameters
free parameters (1)
- eta (Bayes factor deviation tendency) =
0.4
assumptions (3)
- ad hoc to paper Uniform random selection over remaining horses is the Nash equilibrium of the game.
- domain assumption LLMs cannot generate uniform randomness, so using the system-provided random choice is a valid proxy for deliberate randomization.
- ad hoc to paper The H1/H2 likelihood model with eta=0.4 adequately represents randomized versus non-randomized behavior.
Cite this review
Pith. "Pith review of Do LLMs Know When to Flip a Coin? Strategic Randomization through Reasoning and Experience." pith.science (2026). https://pith.science/paper/3ESC47GC
@misc{pith2026250618928,
author = {Pith},
title = {Pith review of: Do LLMs Know When to Flip a Coin? Strategic Randomization through Reasoning and Experience},
year = {2026},
howpublished = {\url{https://pith.science/paper/3ESC47GC}},
note = {Machine review of arXiv:2506.18928}
}
read the original abstract
Strategic randomization is a key principle in game theory, yet it remains underexplored in large language models (LLMs). Prior work often conflates the cognitive decision to randomize with the mechanical generation of randomness, leading to incomplete evaluations. To address this, we propose a novel zero-sum game inspired by the Tian Ji Horse Race, where the Nash equilibrium corresponds to a maximal entropy strategy. The game's complexity masks this property from untrained humans and underdeveloped LLMs. We evaluate five LLMs across prompt styles -- framed, neutral, and hinted -- using competitive multi-tournament gameplay with system-provided random choices, isolating the decision to randomize. Results show that weaker models remain deterministic regardless of prompts, while stronger models exhibit increased randomization under explicit hints. When facing weaker models, strong LLMs adopt deterministic strategies to exploit biases, but converge toward equilibrium play when facing peers. Through win/loss outcomes and Bayes factor analysis, we demonstrate meaningful variation in LLMs' strategic reasoning capabilities, highlighting opportunities for improvement in abstract reasoning and adaptive learning. We make our implementation publicly available at https://github.com/ocelopus/llm-when-to-throw-coin to ensure full reproducibility.
Figures
Reference graph
Works this paper leans on
-
[3]
URLhttps://arxiv.org/abs/2503.02582. 7 Appendix ContributionsThis work was entirely designed, implemented, analyzed, and written by the sole author. Computational Resources and CostThe experiments were conducted using API access to multiple large language models provided by Meituan, under an academic sponsorship arranged by instructor ying.wen@sjtu.edu.cn...
-
[2024]
Wenjie Tang, Yuan Zhou, Erqiang Xu, Keyan Cheng, Minne Li, and Liquan Xiao
URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ 3191170938b6102e5c203b036b7c16dd-Paper-Conference.pdf. Wenjie Tang, Yuan Zhou, Erqiang Xu, Keyan Cheng, Minne Li, and Liquan Xiao. Dsgbench: A diverse strategic game benchmark for evaluating llm-based agents in complex decision-making environments,
work page 2024
-
[2025]
URLhttps://arxiv.org/abs/2503.06047. Alicia Vidler and Toby Walsh. Playing games with large language models: Randomness and strategy,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.