Pith. sign in

REVIEW 3 major objections 6 minor 19 references

LLMs as Agentic Cooperative Players in Multiplayer UNO

T0 review · 3 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A pretrained 70-billion-parameter language model, given only a text prompt, can significantly improve a teammate's chance of winning a three-player game of UNO.

desk verdict Small, honest, reproducible study of LLMs as cooperative UNO players; the cooperative-gain claim is real but fragile and needs a competence control before it can carry weight. read the letter →

arxiv 2509.09867 v1 pith:TKJ63OYT submitted 2025-09-11 cs.AI cs.CL

Yago Romano Matinez , Jesse Roberts This is my paper
classification cs.AIcs.CL
keywords LLMagentscooperativeAIUNOpromptengineeringclozepromptingcounterfactualmodelscalingcardgame
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a large language model can do more than answer questions: can it actively help another player win a game? The authors wire decoder-only LLMs (1B to 70B parameters) into the card game UNO, prompt them with the full game state in plain English, and test two prompting styles in two settings: solo play against a random agent, and three-player play where the LLM sits in seat 2 and tries to help the rule-based agent in seat 1. They find that all tested models beat a random baseline when playing solo, but only one configuration—the 70B model with cloze prompting—raises the partner's win rate above the unassisted baseline with statistical significance (35.00% to 35.96%, p<0.05). If that result holds, it suggests a genuinely useful kind of AI assistance: an off-the-shelf LLM can act as a cooperative teammate in a structured, goal-driven game, with practical implications for game NPCs and human-AI collaboration.

What carries the argument

The central mechanism is a prompt-shift aggregation scheme. Because LLMs exhibit a base-rate bias toward early letter options, the paper repeats the decision prompt once per legal action, rotating the letter labels (A, B, C, ...) and the order of the actions, then sums token probabilities across shifts to select the move. In cloze prompting the model assigns probabilities to letter labels; in counterfactual prompting the model scores each action as 'good' versus 'bad' and takes the difference. This machinery turns a raw LLM into a decision-making agent whose choices can be compared statistically against baselines using one-sided z-tests.

What would settle it

Run the same three-player cooperative setup with the 70B LLM in seat 2 but prompt it to win for itself (no mention of helping seat 1). If seat 1's win rate rises to about 35.96% anyway, the paper's 'cooperative support' conclusion collapses; if it stays near 35.00%, the original result is readily attributed to cooperative prompting.

Watch

Extended reading notes

Core claim

On the paper's own terms, pretrained decoder-only LLMs, given no fine-tuning and only a text prompt describing the current game state, can exceed a random-agent baseline in head-to-head UNO play; the 24B model with cloze prompting reached a 52.10% win rate. More importantly, when tasked explicitly with helping a rule-based teammate win a three-player game, the 70B model with cloze prompting raised that teammate's win rate from 35.00% to 35.96%—the only model–prompt pairing to reach p<0.05. The authors interpret this as evidence that a sufficiently large LLM can provide statistically significant strategic support to another player, and that the effect depends nonlinearly on model scale and pr

Load-bearing premise

The statistical conclusion assumes the unassisted baseline win rate of a rule-based player in seat 1, measured when seat 2 plays randomly, is a valid comparison for the same seat when the LLM replaces the random agent—even though the LLM's actions change hand counts, turn order, and card flow, so any win-rate difference could come from generic play, not from an intent to help.

Editorial extensions

If this is right

  • Pretrained LLMs can play UNO above random chance without any training, performing comparably to a learned DQN agent in solo play.
  • A 70B LLM with the right prompting can improve a teammate's win probability by a small but statistically significant margin; smaller models do not reliably clear that bar.
  • Prompting method matters as much as scale: cloze prompting produced the only significant cooperative gain and the most consistent solo gains.
  • The open-source tool lets other researchers test additional LLMs and games without retraining agents from scratch.
  • Cooperative benefit is currently small (about one percentage point) and depends on seat order and card flow, so practical applications would need larger effects or more targeted environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is real, similar cooperative gains might appear in other turn-based games where legal actions can be enumerated and the goal is well defined—for example cooperative card games like Hanabi—making this a testable extension.
  • The result may be an underestimate: larger or closed proprietary models, or teammates that adapt to the LLM's play, could produce larger cooperative gains than the 70B open model does.
  • The paper's baseline transfer is the fragile link: the 35.00% baseline comes from games where seat 2 plays randomly, but the LLM changes the game dynamics, so the observed gain could partly reflect generic play rather than cooperative intent. A control using an LLM told to win for itself would separate these explanations.
  • The nonlinear scale effect (the 1B model beating the 8B model) suggests that model family and instruction-following quality may matter more than raw parameter count for cooperative behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces an RLCard extension that lets decoder-only LLMs participate in UNO via text prompts, and reports two evaluations: autonomous 1v1 play against a random agent (RQ1) and cooperative three-player play where an LLM in seat 2 is instructed to assist a rule-based teammate in seat 1 (RQ2). Four model sizes (1B–70B) and two prompting strategies (cloze, counterfactual) are compared. All models beat the random baseline in at least one autonomous configuration; in cooperative play, three configurations approached significance and LLaMA3.3-70B with cloze yielded 35.96% versus a 35.00% baseline (p<0.05). The authors conclude that pretrained LLMs can deliver statistically significant cooperative gains. The central concern is that the cooperative baseline is not an adequate counterfactual for seat-2 competence, so the headline cooperation claim is not established by the reported experiments.

Significance. The autonomous-play contribution is credible and useful: the paper ships a reproducible tool, uses large game counts (10,000 per configuration for most models), and compares against fixed, seat-adjusted baselines. If the cooperative result were established, it would be a meaningful step toward LLM agents acting as active teammates in structured games. However, the cooperative claim rests on a single p=0.046 result out of eight configurations, with no control for the possibility that the LLM simply plays better than random in seat 2. The paper is therefore best read as a solid infrastructure and autonomous-play study, with the cooperative conclusion requiring additional experiments before it can be accepted.

major comments (3)
  1. [Section VI] The cooperative test has the wrong null. The unassisted baseline of 35.00% is seat 1's win rate when seat 2 is a random agent. The treatment replaces seat 2 with an LLM that RQ1 shows is a stronger player than random. A more competent seat-2 player changes game dynamics—hand counts, turn order, draw penalties, and which player faces which states—independently of any intent to assist. The one-proportion z-test in Section IV-D therefore tests whether the LLM seat differs from a random seat, not whether it is cooperative. A control with the same LLM instructed to play selfishly, or with a competent non-cooperative agent in seat 2, is needed to isolate the cooperative effect. Without it, the 0.96 percentage point gain (z≈1.68 for n=7,000) is not interpretable as evidence of cooperation.
  2. [Section VI] Multiple comparisons and baseline uncertainty are not handled. Eight model–prompt configurations are tested against the same baseline; only one gives p=0.046. With a Bonferroni-corrected threshold of 0.05/8=0.00625, none of the cooperative results are significant. The text also refers to the baseline as '100,000 games' in the RQ2 answer while Section VI and Figure 6 say 10,000 baseline games; the baseline's own sampling error should be incorporated (e.g., a two-sample test) rather than treating p0=0.35 as an exact constant. The manuscript should report adjusted p-values or a pre-specified familywise procedure, and reconcile the baseline n.
  3. [Section V] The seat assignment for the autonomous 1v1 experiments is not stated. The dashed line is the second-player random baseline (48.96%), and Figure 2 shows a first-player advantage. If the LLM is not always in the second seat, the wrong baseline is used. Please specify the LLM's seat in the 1v1 trials and, if it varies, report per-seat results or compare against the appropriate seat-specific baseline. This is needed to make RQ1's 'beats random' claim fully unambiguous.
minor comments (6)
  1. [Throughout] LLaMA naming is inconsistent (LLaMa3.2-1B vs LLaMA3.1-8B vs LLaMA3.3-70B); please standardize to the official capitalization.
  2. [Section IV-D] The z-test formula is garbled in the text ('p p0(1-p0)/n'); please typeset it properly as z=(p_hat-p0)/sqrt(p0(1-p0)/n).
  3. [Section VI] The baseline size is inconsistent: the RQ2 answer says 100,000 games, Section VI says 10,000 games per model (7,000 for 70B), and Figure 6 reports 10,000 games per model. Please state the exact number of baseline games.
  4. [Section VIII] The limitations section acknowledges the reduced sample size for the 70B model and the use of rule-based teammates, but it does not mention the seat-2 competence confound. Adding this caveat would strengthen the paper's honesty.
  5. [Section IV-A] The prompt description is written primarily for the cooperative setting ('assisting a specific rule-based agent to victory'); please clarify how autonomous-play prompts differ, if at all.
  6. [Section IX] Typo: 'assitants' should be 'assistants'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: RQ2's cooperative gain is an empirical measurement against a fixed baseline, and the self-citations are not load-bearing.

full rationale

The paper's central derivations are self-contained empirical comparisons. RQ1 tests observed 1v1 win rates against a fixed second-player baseline p0=0.4896 measured from random-vs-random games (Section IV-D); RQ2 compares the rule-based seat-1 win rate with an LLM in seat 2 against a fixed 35.00% baseline measured from rule-rule-random games (Section VI). Neither comparison fits a parameter from the data being predicted, and neither defines the target quantity in terms of the input: the LLM's actions could plausibly have lowered, left unchanged, or raised seat 1's win rate, so the observed 35.96% is not forced by construction. The only self-citations ([5], [11], [17], [18]) support background claims about decoder-only models, prior game-agent work, and prompt-design techniques (cloze/counterfactual prompting and the BRP effect); they are published external results and are not invoked as uniqueness theorems or as substitutes for the experiments. The main threat to RQ2 is a validity/confound concern—the 'unassisted' baseline uses a random agent in seat 2 while the assisted condition uses a competent LLM, so the 0.96-point gain may be due to seat-2 competence rather than cooperative intent. That is a correctness/experimental-design issue, not circularity. The Limitations section itself acknowledges related concerns (fixed rule-based partners, reduced power for 70B), but no step in the paper's derivation reduces to its own inputs. Accordingly the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on measured win rates and fixed baselines, not on fitted constants. The main liabilities are the transferability of the cooperative baseline, the first-token probability proxy, and the statistical independence assumption. No new physical or mathematical entities are introduced.

assumptions (4)
  • domain assumption RLCard's patched UNO game class correctly instantiates multi-player games and assigns payoffs.
    The entire evaluation depends on the patched game class in Section IV-B being a faithful, multi-player UNO simulator.
  • domain assumption A single high-probability first token from a decoder-only LLM, aggregated over rotated prompt orders, is a valid proxy for the model's action choice.
    Both cloze and counterfactual prompting in Section IV-A select actions from first-token probabilities; if this proxy is invalid, the reported strategies are not what the models would do in free-form play.
  • domain assumption The 35.00% unassisted baseline measured in rule-rule-random games is a valid counterfactual for the rule-rule-LLM assistance condition.
    Section VI adopts this baseline directly; any game-dynamics shift caused by replacing random seat 2 with an LLM would confound the comparison.
  • domain assumption Games are independent trials, so one-sided one-proportion z-tests apply.
    Section IV-D relies on the normal approximation to the binomial; within-tournament dependencies are not discussed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs as Agentic Cooperative Players in Multiplayer UNO." pith.science (2026). https://pith.science/paper/TKJ63OYT

@misc{pith2026250909867,
  author       = {Pith},
  title        = {Pith review of: LLMs as Agentic Cooperative Players in Multiplayer UNO},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKJ63OYT}},
  note         = {Machine review of arXiv:2509.09867}
}
read the original abstract

LLMs promise to assist humans -- not just by answering questions, but by offering useful guidance across a wide range of tasks. But how far does that assistance go? Can a large language model based agent actually help someone accomplish their goal as an active participant? We test this question by engaging an LLM in UNO, a turn-based card game, asking it not to win but instead help another player to do so. We built a tool that allows decoder-only LLMs to participate as agents within the RLCard game environment. These models receive full game-state information and respond using simple text prompts under two distinct prompting strategies. We evaluate models ranging from small (1B parameters) to large (70B parameters) and explore how model scale impacts performance. We find that while all models were able to successfully outperform a random baseline when playing UNO, few were able to significantly aid another player.

Figures

Figures reproduced from arXiv: 2509.09867 by the authors.

Figure 1
Figure 1. Win rates by seat for three-player random agent and rule-based agent matches (10,000 games each), highlighting the first-player advantage in UNO. Rule-vs-Rule Rand-vs-Rand 48 49 50 51 52 50.58 51.04 49.42 48.96 Win Rate (%) First-Player Advantage in 1v1 UNO Player 1 Player 2 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Win rates by seat in 1v1 UNO matches (10,000 games each), showing a consistent first-player advantage for both Rule-Based and Random agents. This standardized setup makes RLCard particularly useful for evaluating how well LLMs perform both individually and cooperatively in structured multiplayer games like UNO. E. UNO and Opening-Player Advantage UNO is a popular multiplayer turn-based card game known for its simple… view at source ↗
Figure 3
Figure 3. Sample decision trace from Mistral-24B in a UNO match: the model cycles through legal actions (A: green 2, B: green 9, C: red skip) over three prompt shifts, reports per-shift probabilities, aggregates cumulative scores, and selects the highest-scoring move. consistent bias toward the first listed option—a known issue referred to as Base Rate Probability (BRP) [18]. BRP is the intrinsic likelihood of a token being s… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: System architecture for LLM-assisted UNO play in single-node and multi-node modes, showing the controller, agent classes, RLCard integration, and data flow during gameplay. C. System Architecture and Execution A single Controller configures the UNO environment, swaps i…
Figure 6
Figure 6. Figure 6: In a three-player “1v1v1” setting with two rule-based agents in seats 0 and 1 and an LLM in seat 2, we compare the Rule 1 seat win rates across four LLMs under cloze (red) and counterfactual (blue) prompting (10,000 games per model and prompt; LLaMA3.3-70B on 7,000 gam…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 6 linked inside Pith

  1. [1]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskeveret al., “Improving language understanding by generative pre-training,” 2018

    2018

  2. [2]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  3. [3]

    Language models are few-shot learners,

    B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwalet al., “Language models are few-shot learners,”arXiv preprint arXiv:2005.14165, vol. 1, no. 3, p. 3, 2020

    arXiv 2005

  4. [4]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  5. [5]

    How powerful are decoder-only transformer neural models?

    J. Roberts, “How powerful are decoder-only transformer neural models?” in2024 International Joint Conference on Neural Networks (IJCNN). IEEE, 2024, pp. 1–8

    2024

  6. [6]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language mod- els are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  7. [7]

    Palm: Scal- ing language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmannet al., “Palm: Scal- ing language modeling with pathways,”Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023

    2023

  8. [8]

    Rlcard: A toolkit for reinforcement learning in card games,

    D. Zha, K.-H. Lai, Y . Cao, S. Huang, R. Wei, J. Guo, and X. Hu, “Rlcard: A toolkit for reinforcement learning in card games,”arXiv preprint arXiv:1910.04376, 2019

    arXiv 1910

Show all 19 references
  1. [9]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” inPro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolog...

    2019

  2. [10]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  3. [11]

    Do large language models learn human-like strategic preferences?

    J. Roberts, K. Moore, and D. Fisher, “Do large language models learn human-like strategic preferences?” inProceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025), E. Kamalloo, N. Gontier, X. H. Lu, N. Dziri, S. Murty, and A. Lacoste, Eds. Vienna, Aus...

    2025

  4. [12]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  5. [13]

    Neural machine translation of rare words with subword units,

    R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,”arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  6. [14]

    Agents play thousands of 3d video games,

    Z. Xu, X. Wang, S. Li, T. Yu, L. Wang, Q. Fu, and W. Yang, “Agents play thousands of 3d video games,”arXiv preprint arXiv:2503.13356, 2025

    2025 arXiv

  7. [15]

    Collaborative quest completion with llm-driven non- player characters in minecraft,

    S. Rao, W. Xu, M. Xu, J. Leandro, K. Lobb, G. DesGarennes, C. Brock- ett, and B. Dolan, “Collaborative quest completion with llm-driven non- player characters in minecraft,”arXiv preprint arXiv:2407.03460, 2024

    2024 arXiv

  8. [16]

    Pangea: Procedural artificial narrative using generative ai for turn-based video games,

    S. Buongiorno, L. J. Klinkert, T. Chawla, Z. Zhuang, and C. Clark, “Pangea: Procedural artificial narrative using generative ai for turn-based video games,”arXiv preprint arXiv:2404.19721, 2024

    2024 arXiv

  9. [17]

    Chain of thought still thinks fast: Apricot helps with thinking slow,

    K. Moore, J. Roberts, T. T. M. Pham, and D. Fisher, “Chain of thought still thinks fast: Apricot helps with thinking slow,” inProceedings of the Annual Meeting of the Cognitive Science Society, vol. 47, 2025

    2025

  10. [18]

    The base-rate effect on LLM benchmark performance: Disambiguating test-taking strategies from benchmark performance,

    K. Moore, J. Roberts, T. Pham, O. Ewaleifoh, and D. Fisher, “The base-rate effect on LLM benchmark performance: Disambiguating test-taking strategies from benchmark performance,” inFindings of the Association for Computational Linguistics: EMNLP 2024, Y . Al- Onaizan, M. Bansa...

    2024

  11. [19]

    D. S. Moore, G. P. McCabe, and B. A. Craig,Introduction to the Practice of Statistics. WH Freeman New York, 2016, vol. 9

    2016

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.