Emergent LLM behaviors are observationally equivalent to data leakage

Christopher Barrie; Petter T\"ornberg

REVIEW 4 major objections 5 minor 4 cited by

Emergent LLM behaviors are observationally equivalent to data leakage

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that LLM agents playing the naming game reproduce pretraining knowledge of coordination games, making the reported 'emergent social conventions' observationally equivalent to data leakage.

desk verdict The paper makes a fair point about a real contamination risk, but its 'observational equivalence' claim is a plausible hypothesis, not a demonstrated result, because it never tests whether verbal game recognition drives in-game choices. read the letter →

arxiv 2505.23796 v1 pith:UYJRAEGZ submitted 2025-05-26 cs.CL cs.GT

Christopher Barrie , Petter Törnberg This is my paper · ORCID

classification cs.CLcs.GT

keywords LLMagentsdatacontaminationleakagenaminggameemergentsocialconventionsgenerativeagent-basedmodelscoordinationpretrainingmemorization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This commentary argues that the 'emergent social conventions' reported in a recent LLM naming-game study are observationally equivalent to data leakage. The authors show that when the original game prompt is given to a range of language models as a meta-prompt, most models correctly identify the setup as a coordination game, state that the optimal move after a match is to repeat the same action, and many predict that the game will converge globally on one convention. Because the same models are then asked to play the game without this recognition being elicited, the observed convergence can be fully explained by recitation of pretraining knowledge rather than spontaneous creation of norms. If the argument is correct, the published result is an artifact of memorization, and any LLM-based social simulation that reproduces a known finding faces the same data-leakage threat.

What carries the argument

The mechanism carrying the argument is a meta-prompt probe built from the original game's own prompt. It asks a language model three questions: whether the setup corresponds to an existing social-science model or theory, what the optimal move is after a successful match, and how the game will converge globally. The replies are classified along three dimensions, which lets recognition be measured without requiring the model to name a specific paper. This operationalizes data leakage by showing that the knowledge needed to produce the observed naming-game convergence is already present in the models, while the game mechanics—random pairing, reward for matching, and pruning the lexicon toward the last matched word—provide the arena in which that memorized knowledge is expressed.

What would settle it

Run the original 100-round naming game with models that fail the meta-prompt test—models that do not identify the setup as a coordination game, cannot state the optimal post-success move, or do not predict global convergence—and compare their convergence with models that pass; if the failing models converge just as readily, the leakage explanation loses its causal force.

Watch

Extended reading notes

Core claim

The paper's central claim is that the convergence observed in the original LLM naming-game simulation is observationally indistinguishable from models retrieving their pretraining knowledge. Using the original study's own game description, the authors ask a suite of LLMs whether the setup evokes an existing model or theory, what the optimal move is after a successful match, and how the game will converge globally. Across most models tested, including those from the original study, the models identify the scenario as a coordination game, state that after a success the right move is to keep choosing the same action, and often predict rapid global convergence to a single convention. The paper maintains that these three recognized elements are exactly what produces the agent-level convergence in the original simulation, so the behavior can be accounted for by the models' prior exposure to the relevant research literature. The conclusion is that without strict control over training data, claims of emergent behavior from off-the-shelf LLMs cannot be distinguished from memorization.

Load-bearing premise

The paper assumes that a model's correct verbal answers in a meta-prompt reflect the mechanism driving its choices during the original game, where no such recognition is prompted.

Editorial extensions

If this is right

If the central claim is correct, the published finding of emergent social conventions in LLM populations would be a pretraining-memorization artifact rather than a discovery about how LLM agents coordinate.
Generative agent-based simulations that reproduce well-known results cannot take convergence as evidence of emergence unless training data are strictly controlled or the target mechanism is absent from pretraining.
Data leakage becomes a standard validity check for LLM-based social simulations, on par with prompt-template checks and token-bias controls.
Testing LLMs on a genuinely novel game would make 'emergence' a measurable property rather than an assumption, because the pretraining answer would no longer be available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A cheap diagnostic for any claimed emergent LLM behavior is to ask the model directly what theory or optimal strategy the setup evokes; strong verbal recognition makes the behavior observationally equivalent to retrieval, even if the action labels are abstract symbols.
Because human experimental subjects also respond differently once they know the relevant theory, the leakage problem for LLMs mirrors a known reflexivity in social science; models trained on that literature may systematically converge to textbook equilibria.
A testable extension of the paper's logic is to vary only the framing, from a classic coordination game to a structurally identical game described in unfamiliar terms; if convergence drops sharply under unfamiliar framing, the leakage account is strengthened.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a short, honest critique that demonstrates a real risk in the Ashery et al. naming-game result—many LLMs can articulate that the setup is a coordination game, what to do after a success, and how it converges globally. That is a useful diagnostic for the generative ABM community, and the paper is transparent about its methods and code. But the headline claim that the original emergent behavior is observationally equivalent to data leakage is stronger than the evidence supports, because the probe asks the model to recognize the game and the original game never does.

The probes are simple and reproducible. The authors use gpt-4.1 to annotate the outputs and manually verify every annotation, which grounds the evaluation outside the model. They also correctly point out that the original paper's mitigation steps—symmetry of payoffs, token bias checks, prompt variations—don't eliminate the contamination concern, since a model could recognize the game structure even with nonsensical tokens. The point that LLM-based ABM findings can be confounded by pretraining knowledge is well taken and deserves attention.

The central assumption is that verbal recognition licenses a mechanistic inference about in-game choices. That bridge is untested. The authors do not run the original game with their own controls, do not compare probe predictions to actual game trajectories, and do not include a novel-game baseline. They even state that a completely novel game would be needed, but stop short of running it. The 'observational equivalence' framing is therefore an argument about plausibility, not a demonstrated equivalence. The observed convergence could still arise from the hard-coded lexicon-pruning rule plus simple in-context payoff following—which the authors themselves call trivial—so the data-leakage explanation is one of several that fit the same observations.

This paper will be most useful to readers working on LLM-based social simulation and on data contamination. It is a solid cautionary comment, not a definitive refutation. With a softened conclusion or a direct test it would be a stronger contribution. I would send it to peer review because it raises a legitimate methodological challenge to a high-profile Science Advances claim and the literature needs this debate aired. I'd bring it to a reading group for the discussion it would provoke. Recommendation: engage with it, but push the authors to either run a novel-game control or temper the 'observational equivalence' conclusion.

Referee Report

4 major / 5 minor

Summary. This comment-style paper argues that the emergent naming-game conventions reported for LLM populations by Ashery et al. are better explained by data leakage. The authors probe 14 LLMs with a meta-prompt based on the original game description, asking whether the setup corresponds to a known social-science model, what the optimal move after a success is, and whether the game converges globally. A majority of models answer these questions correctly. The paper concludes that LLM agents in the original game recognize the structure and recall its outcomes, and that the emergent behaviors are observationally equivalent to memorization of the training corpus.

Significance. The paper addresses an important and timely concern for generative agent-based modeling. It makes a valuable contribution by empirically demonstrating, with reproducible code and manually verified annotations, that off-the-shelf LLMs can verbally identify a coordination-game structure and its optimal conditional strategy. If the full claim were supported, it would cast doubt on a high-profile published result and sharpen the discussion of contamination in LLM-based simulations. However, the paper does not establish the causal link from that verbal capability to actual in-game decision behavior, and it does not compare the original game trajectories with predictions from a leakage model. The strengths are the multi-model evidence and the explicit GitHub repository; the weakness is the inferential gap between recognition and usage.

major comments (4)

[Probe experiment, section beginning 'Fortunately, we can easily determine whether or not the LLM is aware...'] The probe asks models to name a related theory and to state the optimal move after a success, but the original game presents no such meta-prompt and asks only for an action. The paper does not show that a model that verbalizes 'repeat the same action after a success' actually follows that rule during zero-shot gameplay in the original prompt. Because the original simulation code hard-codes the lexicon-pruning rule (as the paper notes when citing the GitHub link), the in-game convergence could arise from that mechanic alone in combination with simple in-context pattern matching. To support observational equivalence, the authors need to compare the actual game trajectories from Ashery et al.'s setup with predictions derived from the probe responses (e.g., win-stay behavior after a match) or to run the original game with their own controls. This comparison is absent, so the central claim is not tested.
[Abstract and conclusions] The abstract states that the observed behaviors 'are indistinguishable from memorization' and that the results are 'better explained by data leakage,' but the paper reports no original game-play experiment. It relies on the convergence reported in Ashery et al. as a given. Observational equivalence requires a comparison of the behavioral time series under two generative models—one with leakage and one without—or at least a demonstration that the game's outcome is sensitive to the model's knowledge of coordination games. Without that comparison, the strongest supported claim is that LLMs can identify the game when explicitly prompted, which is a risk of contamination rather than proof that the published convergence is an artifact.
[Discussion of Ashery et al.'s supplementary mitigation, quoted in the full text] Ashery et al. argued that the symmetry of payoffs across action labels means no predetermined optimal strategy exists. The probe shows that models know a conditional strategy (repeat the same action after a success), which is compatible with that symmetry argument: it does not imply a preference for token F over token J. The paper therefore does not actually refute the quoted mitigation; it only shows that models can describe a general coordination-game principle. To make the leakage argument stick, the authors must show that this conditional knowledge changes the model's in-game choices relative to a model without such knowledge, or that the original convergence pattern cannot be reproduced with a novel, contamination-free game.
[Penultimate section ('So what would we need to see...')] The paper concedes that inventing a completely novel game is required to test the leakage hypothesis and that whether existing detection techniques would help is 'an open question.' This concession directly undercuts the conclusion that the original result is 'better explained by data leakage.' At minimum, the authors should perform the novel-game control they propose, which is within the scope of their existing code and infrastructure, or else explicitly reframe the paper's contribution as a cautionary note rather than a demonstrated explanation.

minor comments (5)

[Figure 1] Figure 1 reports the percentage of runs per model without error bars or confidence intervals; with only ten runs per model, the precision of these estimates is limited, and error bars should be added.
[Section defining data leakage] The text 'the it may simply be reproducing' in the paragraph defining data leakage contains a typo ('the it').
[Abstract] The abstract's phrase 'multiple analyses' overstates what is in fact a single probe experiment with several models; recommend rewording.
[Methods details] The paper does not report the decoding parameters (e.g., temperature) used for the LLM probes, which are relevant for reproducibility; these should be stated in a methods note or appendix.
[Reproducibility] The GitHub repository for the original game is cited by URL, but the paper does not specify which commit or version of Ashery et al.'s code was used when interpreting their results; a pinned version would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the critique rests on independent probe evidence, not on fitted parameters, self-citations, or definitional reductions.

full rationale

This paper does not present a derivation in which a predicted quantity is equivalent to an input by construction. The central claim is an empirical/observational critique: that the LLM naming-game convergence reported by Ashery et al. is better explained by data leakage than by spontaneous emergence. The evidence is a separate meta-prompt probe in which the same game description is shown to 14 LLMs and they are asked to identify the game, state the optimal move after a success, and predict global convergence. The responses are annotated by gpt-4.1, but every annotation is manually verified against extracted text snippets, so the evaluation is grounded outside the annotating model. There are no fitted parameters that are then renamed as predictions, and no load-bearing self-citations: the reference list contains no prior work by the present authors that is invoked to justify the argument. The inferential step from 'the model can verbalize a coordination-game strategy when prompted' to 'the original zero-shot game behavior is memorization' is a validity gap, not a circular reduction: the probe is a distinct behavioral measurement, and the conclusion is explicitly tentative ('may not be', 'could just be'). The absence of a novel-game control weakens causal inference, but under the stipulated rules that is a correctness/robustness concern, not circularity. The paper is self-contained against external evidence and does not reduce to its own inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's argument is interpretative rather than derivational. It introduces no fitted parameters or new entities. Its burden lies in three domain assumptions about how LLM self-reports, model selection, and simulation mechanics relate to the claimed data-leakage mechanism.

assumptions (3)

domain assumption Verbal self-report of game recognition reflects the knowledge used during in-game decisions.
The probing prompt asks models to identify the game and state optimal moves; the paper infers from these answers that the original in-game convergence was caused by pretraining knowledge. This transfer from meta-cognitive report to gameplay mechanism is assumed, not tested.
domain assumption The models probed are representative of the models used in the original study.
The probe uses 14 models, but the original paper used Llama 2, Llama 3.0, Llama 3.1, and Claude 3.5. The overlap is partial, and the transfer of probe results to the exact original models is assumed.
domain assumption The hardcoded inventory-pruning rule makes subsequent LLM convention choices trivial.
The paper argues that the pruning rule in the simulation code mechanically increases future success probability, but this addresses simulation mechanics, not whether the LLM's specific outputs are generated from pretraining knowledge; the connection is asserted.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Emergent LLM behaviors are observationally equivalent to data leakage." pith.science (2026). https://pith.science/paper/UYJRAEGZ

@misc{pith2026250523796,
  author       = {Pith},
  title        = {Pith review of: Emergent LLM behaviors are observationally equivalent to data leakage},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UYJRAEGZ}},
  note         = {Machine review of arXiv:2505.23796}
}

read the original abstract

Ashery et al. recently argue that large language models (LLMs), when paired to play a classic "naming game," spontaneously develop linguistic conventions reminiscent of human social norms. Here, we show that their results are better explained by data leakage: the models simply reproduce conventions they already encountered during pre-training. Despite the authors' mitigation measures, we provide multiple analyses demonstrating that the LLMs recognize the structure of the coordination game and recall its outcomes, rather than exhibit "emergent" conventions. Consequently, the observed behaviors are indistinguishable from memorization of the training corpus. We conclude by pointing to potential alternative strategies and reflecting more generally on the place of LLMs for social science models.

Figures

Figures reproduced from arXiv: 2505.23796 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Reason-Mediated Behavioral Models for Auditing LLM Social Simulators
cs.AI 2026-07 conditional novelty 6.0 of 10

Signed rationale-derived reason states improve held-out purchase-intent prediction in a 94-person concept test, while LLM-simulated reasons fail the same human readout.
Large language models replicate and predict human cooperation across experiments in game theory
cs.AI 2025-11 conditional novelty 6.0 of 10

Llama-3.1-8B with a multi-step reasoning-and-filter prompt reproduces human cooperation rates across 121 dyadic games (MSD=0.031, r=0.89), outperforming Nash-equilibrium predictions (MSD=0.096, r=0.78).
Addressing Longstanding Challenges in Cognitive Science with Language Models
cs.AI 2025-10 conditional novelty 4.0 of 10

A review proposes that LLMs can serve as tools for a more integrative and cumulative cognitive science when used under human oversight.
Reply to "Emergent LLM behaviors are observationally equivalent to data leakage"
cs.CL 2025-06 conditional novelty 4.0 of 10

The authors argue that data leakage cannot explain the convention formation, collective bias, and critical mass dynamics seen in their LLM population experiments, and they add meta-prompting evidence that LLMs perceiv...

Reference graph

Works this paper leans on

7 extracted references · 4 canonical work pages · cited by 4 Pith papers

[1]

Emergent social conventions and collective bias in llm populations

Ariel Flint Ashery, Luca Maria Aiello, and Andrea Baronchelli. Emergent social conventions and collective bias in llm populations. Science Advances, 11(20):eadu9368, May 2025

work page 2025
[2]

Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms

Simone Balloccu, Patr ´ ıcia Schmidtov´ a, Mateusz Lango, and Ondˇ rej Duˇ sek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms. arXiv preprint arXiv:2402.03927 , 2024

arXiv 2024
[3]

Are economists different, and if so, why? Journal of Economic Perspectives , 5(2):171–177, 1991

John R Carter and Michael D Irons. Are economists different, and if so, why? Journal of Economic Perspectives , 5(2):171–177, 1991

work page 1991
[4]

Sparse autoencoders find highly interpretable features in language models

Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600 , 2023

arXiv 2023
[5]

Generalization or memorization: Data contamination and trustwor- thy evaluation for large language models

Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustwor- thy evaluation for large language models. arXiv preprint arXiv:2402.15938 , 2024

arXiv 2024
[6]

New rules of sociological method

Anthony Giddens. New rules of sociological method . Stanford University Press, 1993

work page 1993
[7]

Latesteval: Addressing data contamination in language model evaluation through dynamic and time- sensitive test construction

Yucheng Li, Frank Guerin, and Chenghua Lin. Latesteval: Addressing data contamination in language model evaluation through dynamic and time- sensitive test construction. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):18600–18607, Mar. 2024. 7 1 Appendix System Prompt: You are an expert annotation assistant. When given a model’s resp...

work page 2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Emergent social conventions and collective bias in llm populations

Ariel Flint Ashery, Luca Maria Aiello, and Andrea Baronchelli. Emergent social conventions and collective bias in llm populations. Science Advances, 11(20):eadu9368, May 2025

work page 2025

[2] [2]

Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms

Simone Balloccu, Patr ´ ıcia Schmidtov´ a, Mateusz Lango, and Ondˇ rej Duˇ sek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms. arXiv preprint arXiv:2402.03927 , 2024

arXiv 2024

[3] [3]

Are economists different, and if so, why? Journal of Economic Perspectives , 5(2):171–177, 1991

John R Carter and Michael D Irons. Are economists different, and if so, why? Journal of Economic Perspectives , 5(2):171–177, 1991

work page 1991

[4] [4]

Sparse autoencoders find highly interpretable features in language models

Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600 , 2023

arXiv 2023

[5] [5]

Generalization or memorization: Data contamination and trustwor- thy evaluation for large language models

Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustwor- thy evaluation for large language models. arXiv preprint arXiv:2402.15938 , 2024

arXiv 2024

[6] [6]

New rules of sociological method

Anthony Giddens. New rules of sociological method . Stanford University Press, 1993

work page 1993

[7] [7]

Latesteval: Addressing data contamination in language model evaluation through dynamic and time- sensitive test construction

Yucheng Li, Frank Guerin, and Chenghua Lin. Latesteval: Addressing data contamination in language model evaluation through dynamic and time- sensitive test construction. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):18600–18607, Mar. 2024. 7 1 Appendix System Prompt: You are an expert annotation assistant. When given a model’s resp...

work page 2024