Pith. sign in

REVIEW 3 major objections 4 minor 38 references

Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes that LLM agents do not share one strategic policy in an AI development race, and that a validity audit must precede any behavioral interpretation.

desk verdict Useful audit-first protocol and a cautionary tale about sample-size-matched diversity claims; the title overreaches. read the letter →

arxiv 2608.01193 v1 pith:A4LXRRMD submitted 2026-08-02 cs.AI cs.CYcs.GTcs.LGcs.MA

Phu Hoa Pham , Duy Minh Dao Sy , Trung Kiet Huynh , Phu Quy Nguyen Lam , Chi Nguyen Tran , Minh Trung Le , Phong Hao Le , Dinh Nam Nguyen
show 4 more authors
Thien Ky Nguyen Dong Elias Fernandez Domingos Le Hong Trang The Anh Han
This is my paper
classification cs.AIcs.CYcs.GTcs.LGcs.MA
keywords largelanguagemodelagentsAIdevelopmentracesmulti-agentsystemsrepeatedgamesstrategicdecision-makingpromptsensitivitytaskvaliditybehavioraldiversity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language model agents can be said to behave strategically in an idealised AI development race, and it argues that the answer is only visible after a validity audit, not from aggregate action rates. It reproduces a repeated two-player game of Safe versus Unsafe development, extends it to three to five players, and tests seven model endpoints against an evolutionary game-theory benchmark and published human data. The central finding is that the models do not share one policy: similar aggregate Unsafe rates hide large differences in responses to risk, opponent history, and race position, while human participants spread across nearly the full behavioral range. The authors conclude that multi-agent AI-race simulations should report comprehension accuracy, parser health, and trajectory-level diversity before their outputs are described as strategic, human-like, or safety-aware, and they scope all findings to the tested models, prompts, and decoding settings.

What carries the argument

The load-bearing object is the idealised AI development race itself: a repeated simultaneous-move game in which Safe advances progress by 1.0 and Unsafe by 1.5, Unsafe strictly dominates the stage payoff, and accumulated Unsafe choices create private setback risk that is drawn only against the winner. The argument is carried by a four-level audit gate placed before behavioral interpretation: mechanical validity of the game engine, task validity measured through rule recall, state reconstruction, state transition, terminal scoring, and expected-payoff calculation, representation robustness under equivalent wordings and response mappings, and comparison with human trajectories. Trajectories ar

What would settle it

Run the full 41-probe comprehension battery on every tested endpoint and exclude every race in which any model fails a pre-registered state-tracking or payoff threshold; if the between-model spread in Unsafe rates and archetype membership shrinks or collapses, the claim of distinct strategic policies is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that strategic behavior in an idealised AI race is not a fixed or common property of LLMs: the tested models differ from each other as much as they differ from humans, but in the opposite direction. Humans spread across nearly the whole spectrum of unsafe play and behavioral archetypes, while each model concentrates in a narrow, model-specific band. The discovery is framed as a methodological one: a correctly formatted action is not evidence that the agent understands the game. Rule recall can be near-perfect while state tracking, state transition, and expected-payoff calculation are weak, and equivalent presentations such as a decision card or opaque response co

Load-bearing premise

The load-bearing premise is that the comprehension audit run on a single checkpoint is representative of all seven models; if other models misunderstand the game in different ways, the observed cross-model policy diversity could be an artifact of differential task misunderstanding rather than distinct strategies.

Editorial extensions

If this is right

  • An LLM's aggregate Unsafe rate should not be read as its policy: models at similar rates can differ sharply in reciprocity, position sensitivity, and risk responsiveness.
  • Matching the human mean is not evidence of human-like behavior; a model near the human average can be concentrated in a single cautious archetype while humans occupy all four archetypes.
  • Assigned risk personas in prompts act as strong instructions, moving play by tens of percentage points, rather than as measured human risk preferences.
  • Adding players to the race does not produce one monotone group-size effect; the pattern is model- and persona-specific.
  • Reports of LLM gameplay should include comprehension-admission scores and parser health, because plausible trajectories can coexist with poor state tracking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the comprehension gate was run on one checkpoint only, the cross-model behavioral differences could partly reflect differences in task understanding; a natural next step is to run the full probe battery on every endpoint and split trajectories by admission status.
  • The authors' position-effect reversal under risk personas is entangled with selection, since unsafe players mechanically become leaders; conditioning on or randomising prior own-action history could test whether rank itself has a causal effect.
  • If the audit-gate approach generalises, human-LLM behavioral comparisons should be treated as distributional and archetype-based rather than mean-matching, which would make 'human-like' claims falsifiable.
  • With comprehension verified per model, equivalent narrative skins could serve as a stress test: if policy differences persist after all models pass, that would support genuine strategic diversity rather than prompt artifacts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes an audit-first evaluation of LLM agents in an idealised AI development race game. After validating the game engine, the authors run a 685-probe task audit on Qwen2.5-7B-Instruct, test robustness to disclosed arithmetic and opaque symbolic mappings, compare two-player LLM trajectories with an evolutionary game-theoretic benchmark and a published human dataset, and pilot three-to-five-player races. The headline finding is that aggregate unsafe rates hide trajectory-level diversity: humans occupy multiple behavioural archetypes while the tested frontier LLMs concentrate in narrow, model-specific regions; models also differ in which state variables predict unsafe choices. The authors are careful to scope conclusions to tested checkpoints, prompts, and decoding settings, and to label multi-player pilots exploratory.

Significance. The paper’s main value is methodological and cautionary. It reproduces the human benchmark coefficients almost exactly (Appendix A.2), pre-registers evidence strata, retains prompts/seeds, includes a source-level citation audit, and presents multiple converging analyses (probe battery, paired diagnostics, HDBSCAN, nested logistic interactions, SHAP). If the sample-size and audit-coverage concerns are addressed, this would be a useful contribution to multi-agent LLM evaluation and AI-safety simulation methodology. As it stands, the central ‘humans are more diverse’ claim is not yet supported at the required strength because the key HDBSCAN comparison is confounded by sample size, and the validity gate is only run on one of the eight populations whose strategic behaviour is interpreted.

major comments (3)
  1. [§4.4, Figure 4, Table 4] The archetype-coverage comparison is confounded by sample size. HDBSCAN is run with min_cluster_size=15 on 340 human trajectories but only 60 per model. With N=60, at most four disjoint clusters of size ≥15 can exist; with N=340, many more clusters are possible. Thus the statement that all eleven archetypes occur among humans while GPT-5-nano falls into three cannot distinguish a genuine diversity difference from a cluster-resolution artefact. The same issue appears in Figure 8 (human n=98 vs model n=40). Please re-run with sample-size-matched human subsamples (e.g., 60 trajectories per draw, repeated over many draws) or use a size-adjusted diversity index. Without this, the title claim rests on a biased comparison. Table 4’s fixed four-archetype projection is less vulnerable, but the HDBSCAN analysis is still presented as primary support.
  2. [§4.4, decision-tree accuracy] The decision-tree evidence is presented against the wrong baseline. The eight populations are not balanced: n=340 humans vs n=60 per model, so a no-skill majority classifier achieves 340/760 = 44.7% accuracy. The reported 38.9% ± 3.3 is therefore below the majority baseline, not above it; ‘random guessing among eight balanced classes’ (12.5%) is not the relevant null. The claim that population identity is visible in the trajectories should be supported by balanced accuracy, macro-F1, or a comparison against the majority baseline after class balancing. As written, this statistic weakens rather than supports the diversity argument.
  3. [§4.1 vs §4.4–§4.6] The audit gate is only run on Qwen2.5-7B-Instruct, yet the subsequent trajectory comparisons and multi-player analyses include seven other endpoints (GPT-5-nano, GPT-5.4-nano, Gemini-3-Flash, Gemini-3.1-Flash-Lite, Gemini-3.5-Flash-Lite, Claude Opus 5, Claude Sonnet 5). The paper states in §4.2 that the comprehension check is a prerequisite for interpreting behavioural results, and §5 says reports should show comprehension accuracy together with action rates. Those requirements are not met for six of the seven models. If the other checkpoints fail comprehension differently, the observed cross-model diversity could partly reflect differential task misunderstanding rather than distinct strategic policies. Since RQ1 and RQ2 are framed as strategic behaviour, please either run the same gate on each reported endpoint or explicitly downgrade the cross-model comparisons to un-audited descriptiv
minor comments (4)
  1. [Keywords] Typo: ‘Lsarge language model agents’ should be ‘Large language model agents’.
  2. [Appendix A] The HDBSCAN sensitivity check is referenced as ‘(§ ??)’; the correct section number should be supplied.
  3. [§4.2] The paper flags the repetition-parity confounding and the failed comprehension gate only in §5’s limitation list. The same caveat should appear at the point where the opaque-symbol result is first reported, since the result is explicitly ‘confounded’ and cannot support claims about task understanding.
  4. [Figure 8] The panels compare human n=98 with model n=40 or n=20 using log-scale smoothed densities. Please state the smoothing/bandwidth rule and justify comparability across unequal sample sizes, or use a shared bandwidth followed by a sensitivity check.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: external human/evolutionary benchmarks are independently verified, not fitted to LLM output.

full rationale

This is an empirical behavioural study rather than a derivation. The game model is taken from the authors' prior work [14,17], but it is an input mechanism under test, not a result derived from LLM data; using one's own game definition as the experimental apparatus is not circular. The human benchmark [14] is also from the same research group, but the paper does not simply cite it: §A.2 refits the published dynamic specification on the public de-identified dataset and reproduces the original coefficients almost exactly (largest discrepancy 0.002), so the self-citation is independently verified evidence under the review rules. The evolutionary benchmark is explicitly kept separate: §4.3 states 'The evolutionary model describes strategy frequencies under selection and mutation; the LLM experiment samples prompted self-play decisions. Their units are different,' and the paper reports a qualitative mismatch rather than claiming the LLMs confirm the benchmark. The diversity/archetype analyses (HDBSCAN, t-SNE, decision tree, k-means projection) are descriptive fits to observed trajectories; none of these quantities is fitted to a subset and then reported as a prediction of a closely related quantity. The decision-tree baseline (12.5% vs 38.9%) and the human-vs-model sample-size imbalance are statistical validity concerns, not definitional reductions: the title claim could be confounded by n=340 vs n=60 clustering capacity, but the paper's evidence does not reduce to its inputs by construction. The paper also flags its own limits (comprehension gate on one model, confounded mapping, small multiplayer pilots) rather than hiding them. Score 1 reflects the presence of several self-citations to the same game/human line of work, but none is load-bearing for the central claim.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on the game mechanics inherited from prior literature, the external human and evolutionary benchmarks, and the assumption that a single-model audit informs the interpretation of all models. No numbers are fitted to derive the main conclusions, so free parameters are limited to analytical choices that do not affect the core claim.

free parameters (2)
  • HDBSCAN minimum cluster size = 15
    Chosen by hand for trajectory archetype analysis in Section 4.4; does not affect the central claim about aggregate rates hiding diversity.
  • HDBSCAN minimum samples = 6
    Chosen by hand for the same clustering step; does not affect the main conclusions.
assumptions (4)
  • domain assumption Simultaneous sealed actions and the risk formula q_i(T) = p_max * n_U_i(T)/T from Han et al. [17]
    The game mechanics are taken from prior literature; the paper does not derive them.
  • domain assumption The evolutionary benchmark from Domingos and Han [14] is taken as a qualitative reference; its predicted Unsafe rates (99.2%, 98.0%, 1.9%) are not fitted to LLM output.
    The benchmark is presented as a qualitative comparison, not a fitted model of LLM behavior.
  • domain assumption The human dataset from Domingos and Han [14] is treated as a valid external reference for behavior.
    The paper validates its reconstruction by refitting the original model, but the underlying data quality is assumed.
  • ad hoc to paper The Qwen2.5-7B audit results are assumed to inform the interpretation of behavior in other models that were not audited.
    The comprehension gate is only run on Qwen; the paper extends this framing to all models without individual audits.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races." pith.science (2026). https://pith.science/paper/A4LXRRMD

@misc{pith2026260801193,
  author       = {Pith},
  title        = {Pith review of: Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A4LXRRMD}},
  note         = {Machine review of arXiv:2608.01193}
}
read the original abstract

An AI development race creates a multi-agent safety dilemma. Each company can develop slowly and safely, or move faster while taking a risk that may remove its final reward. We use this repeated game to study strategic safety behaviour among large language model (LLM) agents in races with two to five players. However, a valid action does not show that an agent understands the game. We therefore place an audit gate before behavioural interpretation. We first verify the game engine, then test rule recall, state tracking, payoff calculation, and stability under different but equivalent task descriptions. We then compare LLM action sequences with an evolutionary game-theory benchmark and published human data, and explore differences across models, risk conditions, personas, and two- to five-player races. The audit shows that strong rule recall can coexist with weak state tracking and expected-payoff calculation. Providing verified arithmetic and changing the response representation can also change later actions, even when the game rules stay fixed. Across seven tested model endpoints, aggregate rates hide large differences in action sequences, responses to opponents, and responses to race position. Patterns across the tested three- to five-player races are also model-specific rather than a single effect of adding competitors. These results show why multi-agent AI-race simulations need validity checks and trajectory-level analysis before their outputs are described as strategic, human-like, or safety-aware. Our findings are exploratory and apply only to the tested models, prompts, and decoding settings.

Figures

Figures reproduced from arXiv: 2608.01193 by the authors.

Figure 1
Figure 1. Overview of the AI-race mechanism shared by the two-player and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of the three persona-related framing families used to describe each company’s executive. The canonical [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 5
Figure 5. t-SNE visualisation of the pooled 15-dimensional [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figures from the paper (6 more)
Figure 6
Figure 6. Figure 6: The same t-SNE coordinates as Figure 5, recoloured [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Mean round-by-round profile for each population, [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Smoothed distribution of player-level unsafe rates by risk cap (columns) and model family (rows), with the human benchmark repeated in every panel as the shaded reference. Density is log-scaled: model peaks span close to three orders of magnitude, and a linear axis wou…
Figure 9
Figure 9. Figure 9: unsafe rate by relative position (Leader / Middle / Trailer) within each assigned persona band, for GPT-5-nano (left) and GPT-5.4-nano (right). Bars marked † in Tables 5–6 have 𝑛 < 20 and should be read as noisy pilot estimates rather than stable rates [PITH_FULL_IMAG…
Figure 10
Figure 10. Figure 10: A measured human disposition against an assigned model label. Left: mean [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Population-specific predictive structure for round- [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 16 canonical work pages

  1. [1]

    Arriaga, and Adam Tauman Kalai

    Gati V Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using Large Lan- guage Models to Simulate Multiple Humans and Replicate Human Subject Studies. InProceedings of the 40th International Conference on Machine Learning (Proceed- ings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sab...

    work page 2023

  2. [2]

    Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. 2025. Playing repeated games with large language models.Nature Human Behaviour9, 7 (01 Jul 2025), 1380–1390. https://doi.org/10.1038/s41562- 025-02172-y

    doi:10.1038/s41562- 2025

  3. [3]

    Richardson, Austin C

    Jacy Reese Anthis, Ryan Liu, Sean M. Richardson, Austin C. Kozlowski, Bernard Koch, Erik Brynjolfsson, James Evans, and Michael S. Bernstein. 2025. Position: LLM Social Simulations Are a Promising Research Method. InProceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, Vancouver, C...

    work page 2025

  4. [4]

    Argyle, Ethan C

    Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of One, Many: Using Language Models to Simulate Human Samples.Political Analysis31, 3 (2023), 337–351. https: //doi.org/10.1017/pan.2023.2

    doi:10.1017/pan.2023.2 2023

  5. [5]

    Stuart Armstrong, Nick Bostrom, and Carl Shulman. 2016. Racing to the precipice: a model of artificial intelligence development.AI & SOCIETY31, 2 (01 May 2016), 201–206. https://doi.org/10.1007/s00146-015-0590-y

    doi:10.1007/s00146-015-0590-y 2016

  6. [6]

    Hadfield

    Amanda Askell, Miles Brundage, and Gillian K. Hadfield. 2019. The Role of Cooperation in Responsible AI Development. arXiv:1907.04534 [cs.CY] https: //arxiv.org/abs/1907.04534

    arXiv 2019

  7. [7]

    Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M

    James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. 2024. Synthetic Replacements for Human Survey Data? The Perils of Large Language Models.Political Analysis32, 4 (2024), 401–416. https: //doi.org/10.1017/pan.2024.5

    doi:10.1017/pan.2024.5 2024

  8. [8]

    Buscemi, D

    A. Buscemi, D. Proverbio, A. Di Stefano, T.A. Han, G. Castignani, and P. Liò. 2025. FAIRGAME: a Framework for AI Agents Bias Recognition using Game Theory. InFront. Art. Int. Appl., Vol. 413: ECAI2025. IOS Press

    work page 2025

Show all 38 references
  1. [9]

    Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich. 2025. Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations. InProceedings of NAACL

    2025

  2. [10]

    ÓhÉigeartaigh

    Stephen Cave and Seán S. ÓhÉigeartaigh. 2018. An AI Race for Strategic Advan- tage: Rhetoric and Risks. InProceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society(New Orleans, LA, USA)(AIES ’18). Association for Computing Machinery, New York, NY, USA, 36–40. http...

    2018

  3. [11]

    Santos, Luís Moniz Pereira, Tom Lenaerts, and The Anh Han

    Theodor Cimpeanu, Francisco C. Santos, Luís Moniz Pereira, Tom Lenaerts, and The Anh Han. 2022. Artificial intelligence development races in heterogeneous settings.Scientific Reports12, 1 (02 Feb 2022), 1723. https://doi.org/10.1038/ s41598-022-05729-3

    2022

  4. [12]

    Wang, and Eric Schulz

    Julian Coda-Forno, Marcel Binz, Jane X. Wang, and Eric Schulz. 2024. CogBench: A Large Language Model Walks into a Psychology Lab. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, Vienna, Austria...

    2024

  5. [13]

    Maria del Rio-Chanona, Marco Pangallo, and Cars Hommes

    R. Maria del Rio-Chanona, Marco Pangallo, and Cars Hommes. 2025. Can Gen- erative AI Agents Behave Like Humans? Evidence from Laboratory Market Experiments. arXiv:2505.07457 [econ.GN] https://arxiv.org/abs/2505.07457 Preprint

    2025 arXiv

  6. [14]

    Elias Fernández Domingos and The Anh Han. 2026. Falling Behind Drives Unsafe Development in an Idealised AI Race Experiment. arXiv:2607.26034 [cs.AI] https://arxiv.org/abs/2607.26034

    2026 arXiv

  7. [15]

    Salvatore Giorgi, Tingting Liu, Ankit Aich, Kelsey Jane Isman, Garrick Sherman, Zachary Fried, João Sedoc, Lyle Ungar, and Brenda Curtis. 2024. Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas. InFindings of the Association for Computat...

    2024

  8. [16]

    Oliver Gürtler, Lennart Struth, and Max Thon. 2023. Competition and risk- taking.European Economic Review160 (2023), 104592. https://doi.org/10.1016/j. euroecorev.2023.104592

    2023

  9. [17]

    Santos, and Tom Lenaerts

    The Anh Han, Luís Moniz Pereira, Francisco C. Santos, and Tom Lenaerts. 2020. To regulate or not: A social dynamics analysis of an idealised ai race.Journal of Artificial Intelligence Research69 (2020), 881–921. https://doi.org/10.1613/JAIR.1. 12225

    2020 doi

  10. [18]

    Nathan Herr, Fernando Acero, Roberta Raileanu, María Pérez-Ortiz, and Zhibin Li

  11. [19]

    Trung-Kiet Huynh, Duy-Minh Dao-Sy, Thanh-Bang Cao, Phong-Hao Le, Hong- Dan Nguyen, Phu-Quy Nguyen-Lam, Minh-Luan Nguyen-Vo, Hong-Phat Pham, Phu-Hoa Pham, Thien-Kim Than, Chi-Nguyen Tran, Huy Tran, Gia-Thoai Tran-Le, Alessio Buscemi, Le Hong Trang, and The Anh Han. 2025. Under-...

    2025 doi

  12. [20]

    Trung-Kiet Huynh, Duy-Minh Dao-Sy, Thanh-Bang Cao, Phong-Hao Le, Hong- Dan Nguyen, Phu-Quy Nguyen-Lam, Minh-Luan Nguyen-Vo, Hong-Phat Pham, Phu-Hoa Pham, Thien-Kim Than, Chi-Nguyen Tran, Huy Tran, Gia-Thoai Tran- Le, Alessio Buscemi, Le Hong Trang, and The Anh Han. 2026. Payof...

    2026 doi arXiv

  13. [21]

    arXiv:2407.04467 [cs.AI] https://arxiv.org/abs/2407.04467

    Are Large Language Models Strategic Decision Makers? A Study of Perfor- mance and Bias in Two-Player Non-Zero-Sum Games. arXiv:2407.04467 [cs.AI] https://arxiv.org/abs/2407.04467

    arXiv

  14. [22]

    Petra Nieken and Dirk Sliwka. 2010. Risk-taking tournaments – Theory and experimental evidence.Journal of Economic Psychology31, 3 (2010), 254–268. https://doi.org/10.1016/j.joep.2009.03.009

    2010 doi

  15. [23]

    Serkan Ozbeklik and Janet Kiholm Smith. 2017. Risk taking in competition: Evidence from match play golf tournaments.Journal of Corporate Finance44 (2017), 506–523. https://doi.org/10.1016/j.jcorpfin.2014.05.003

    2017 doi

  16. [24]

    Yuxuan Lu, Jing Huang, Yan Han, Bingsheng Yao, Sisong Bei, Yaochen Xie, Yisi Sang, Qi He, and Dakuo Wang. 2026. Can LLM Agents Simulate Multi- Turn Human Behavior? Evidence from Real Online Customer Behavior Data. InProceedings of the 64th Annual Meeting of the Association for...

    2026 doi

  17. [25]

    Pouya Pezeshkpour and Estevam Hruschka. 2024. Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions. InFind- ings of the Association for Computational Linguistics: NAACL 2024. Associa- tion for Computational Linguistics, Mexico City, Mexico, 20...

    2024 doi

  18. [26]

    Isaac Robinson and John Burden. 2025. Framing the Game: How Context Shapes LLM Decision-Making. arXiv:2503.04840 [cs.CL] https://arxiv.org/abs/2503.04840

    2025 arXiv

  19. [27]

    Saptarshi Pal, Abhishek Mallela, Lenz Pracher, Chiyu Wei, Feng Fu, Santiago Schnell, and Martin A Nowak. 2026. Large language models instantiate evolu- tionarily robust strategies of cooperation.PNAS nexus5, 6 (2026), pgag210

    2026

  20. [28]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quan- tifying Language Models’ Sensitivity to Spurious Features in Prompt De- sign or: How I Learned to Start Worrying About Prompt Formatting. InInternational Conference on Learning Representations. OpenReview.n...

    2024

  21. [29]

    Jia Wang, Ziyu Zhao, Tingjuntao Ni, and Zhongyu Wei. 2025. SocioBench: Modeling Human Behavior in Sociological Surveys with Large Language Models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistic...

    2025 doi

  22. [30]

    Abel Salinas and Fred Morstatter. 2024. The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance. arXiv:2401.03729 [cs.CL] https://arxiv.org/abs/2401.03729

    2024 arXiv

  23. [31]

    Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2024. Reasoning or Reciting? Explor- ing the Capabilities and Limitations of Language Models Through Counterfactual Tasks. InProceedings of the 2024 Conferenc...

    2024 doi

  24. [32]

    Jiayi Yao, Cong Chen, and Baosen Zhang. 2026. Competition and Cooperation of LLM Agents in Games. arXiv:2604.00487 https://arxiv.org/abs/2604.00487

    2026 arXiv

  25. [33]

    Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models. InFindings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics,...

    2024

  26. [34]

    A Helpful Assistant

    Mingqian Zheng, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. 2024. When “A Helpful Assistant” Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models. In Findings of the Association for Computational Linguisti...

    2024 doi

  27. [37]

    Kehan Zheng, Jinfeng Zhou, and Hongning Wang. 2025. Beyond Nash Equilib- rium: Bounded Rationality of LLMs and Humans in Strategic Decision-Making. arXiv:2506.09390 https://arxiv.org/abs/2506.09390

    2025 arXiv

  28. [2024]

    https://doi.org/10.18653/v1/2024.findings-emnlp.420

    Association for Computational Linguistics, Miami, Florida, USA, 7174–7188. https://doi.org/10.18653/v1/2024.findings-emnlp.420

    2024 doi

  29. [2025]

    https://doi.org/10.18653/v1/2025.naacl-long.162

    Association for Computational Linguistics, Albuquerque, New Mexico, 3141–3154. https://doi.org/10.18653/v1/2025.naacl-long.162

    2025 doi

  30. [5621]

    https://doi.org/10.18653/v1/2024.findings-acl.333

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.