Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Reply to "Emergent LLM behaviors are observationally equivalent to data leakage"

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This reply argues that the observed dynamics in LLM populations cannot be explained by data contamination alone.

desk verdict A disciplined rebuttal that makes two good points, but the new meta-prompting probe is weaker than the authors claim and does not establish that data contamination alone cannot explain the results. read the letter →

arxiv 2506.18600 v1 pith:FFZNII6C submitted 2025-06-23 cs.CL cs.GTcs.MA

Ariel Flint Ashery , Luca Maria Aiello , Andrea Baronchelli This is my paper · ORCID
classification cs.CLcs.GTcs.MA
keywords LLMpopulationsdatacontaminationsocialconventionsnaminggameemergentcollectivebehaviorbiascriticalmassmulti-agentsimulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is a reply to a critique claiming that the emergent conventions observed in populations of large language models are observationally equivalent to data leakage. The authors set out to show that the observed dynamics cannot be explained by data contamination alone. They argue that meta-prompting reveals the LLMs interpret the experiment as a two-player interaction, so the locally optimal "repeat after success" strategy would actually prevent population-wide consensus rather than produce it. They also point to population-level effects, collective bias and critical-mass tipping, that are model-dependent, non-deterministic, and not reducible to individual token preferences or inventory pruning. A sympathetic reading is that the original findings, convention emergence, collective bias, and critical-mass dynamics, stand as demonstrations of emergent collective behavior in LLM populations, while data contamination remains an acknowledged risk.

What carries the argument

The argument turns on the meta-prompting test: after receiving the original game prompt, the LLM is asked whether the setup reminds it of an existing theory, what the optimal move is after a success, and how the game will converge globally. The reply repeats this test with one additional question asking whether the model is playing a single player or a whole population, and every tested model answers a single player. This pairwise interpretation is the load-bearing mechanism: it supports the inference that the models are not applying population-scale prior knowledge, and it makes the locally optimal repeat-after-success rule a force that would block global consensus rather than create it. The naming game framework itself, a standard setting in which agents reach local agreement through repeated random pairwise interactions, supplies the experimental context.

What would settle it

Run the same convention experiment with a structurally identical but invented game, using novel symbols, labels, and framing that have no counterpart in training text; if spontaneous conventions, collective bias, and committed-minority reversals still appear, the observed dynamics do not require prior knowledge of the naming game. Equally, priming the model with an explicit population framing should alter its behavior if the two-player self-reports are merely prompt artifacts.

Watch

Extended reading notes

Core claim

The core claim is that the critique's data-leakage explanation fails on its own meta-prompting evidence. When asked to describe the game, LLMs answer that it is a repeated two-player interaction, not a population-scale coordination game; across the models tested, every response identified a single opponent. If the models were acting on prior knowledge of the naming-game literature, they would be expected to invoke population-scale dynamics, since that literature is about large populations. Moreover, the strategy the meta-prompting returns as optimal, locking in to the first successful coordination, would hinder global consensus in a population, because different pairs would lock onto different names; the fact that global consensus is observed therefore shows the agents are not simply executing that memorized rule. The reply adds that collective bias, committed-minority convention reversals, and spontaneous consensus switching are population-level, model-dependent outcomes that no predetermined training-data script would produce.

Load-bearing premise

The reply's central argument relies on trusting what the LLMs say about their own understanding: because they answer that the setup is a two-player interaction, the authors conclude the models are not applying training knowledge about population-scale coordination games, and if those self-reports are incomplete or strategic that inference collapses.

Editorial extensions

If this is right

  • If the reply is right, the original observation of convention emergence in LLM populations is not an artifact of training-data leakage and can be studied as collective dynamics.
  • Symmetric coordination games with equivalent alternatives are a safer test bed for emergence than games with a unique optimal outcome, because prior knowledge cannot specify which convention should win.
  • The locally optimal "repeat after success" rule, taken literally, would prevent global consensus; observed global convergence implies agents adapt to the evolving population of conventions.
  • Collective bias and critical-mass tipping are expected to appear as general population-level phenomena in LLM collectives, with direction and strength varying by model architecture.
  • Other multi-agent LLM studies should treat model self-reports about game understanding with care: pairwise interpretations can coexist with population-level outcomes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: A decisive follow-up would run the same convention-formation experiment with an invented symmetric game whose labels and framing have no counterpart in the training corpus; if emergence, bias, and tipping still occur there, leakage of the known naming game cannot be the mechanism.
  • Inference: The meta-prompting answers are self-reports and may be post-hoc rationalizations; a behavioral probe, such as varying the stated number of opponents and measuring whether strategies change, would test population awareness without relying on what the model says.
  • Inference: Model-dependent variation in collective bias could in principle be traced to training-data composition, so the strongest version of the leakage hypothesis is testable by comparing bias directions across models with known, divergent training corpora.
  • Inference: If LLMs are told explicitly that they are part of a population, their strategy may shift from pairwise lock-in toward population-aware coordination; this is a direct, testable extension of the reply's central assumption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This manuscript is a reply to Barrie and Törnberg's critique, which argued that emergent behaviors observed in LLM populations are observationally equivalent to data leakage. The reply defends the original naming-game study by arguing (i) the inventory-update rule is standard for the paradigm and does not make consensus trivial; (ii) meta-prompting self-reports show LLMs interpret the game as a two-player interaction, implying they are not relying on population-scale training knowledge; (iii) committed-minority and spontaneous-convention-switching results demonstrate context-sensitivity inconsistent with a pre-scripted leakage account; and (iv) broader evidence from other games suggests LLMs do not simply reproduce memorized patterns. The reply concludes that the observed dynamics cannot be explained by data contamination alone.

Significance. If correct, the reply would considerably shore up the interpretation of the original Science Advances study as evidence of genuine collective emergence in LLM populations, with implications for multi-agent simulation methodology. The paper contributes a new empirical probe: a meta-prompting question about single-player vs. population framing, replicated across eight models, with responses made available on GitHub. It also usefully clarifies the definitions of emergence and convention and engages with the critique's specific arguments. However, the significance is limited by the fact that the key inference rests on the untested assumption that meta-prompting self-reports expose the model's operative game model, and the new probe is compatible with a contamination account that posits generic coordination heuristics rather than population-scale game recognition.

major comments (4)
  1. [Data Leakage and Prior Knowledge] The central inference, stated in the paragraph beginning 'Thus, firstly, the LLM interpretation ...', is that pairwise self-reports imply the models are 'not relying on prior information in this setting.' This does not follow: a model can frame the task as a two-player interaction while still applying a learned coordination heuristic such as 'repeat your previous successful action,' which is abundant in generic two-player coordination-game text and is precisely the kind of prior a contamination account would invoke. The authors themselves concede in the fourth point of the same section that whether a prompt overrides prior knowledge is 'difficult to test conclusively,' which undercuts the abstract's claim that the observed dynamics 'cannot be explained by data contamination alone.'
  2. [Second point under 'A number of other points'] The committed-minority argument assumes that a leakage-based account must produce a 'pre-scripted trajectory towards a predetermined convention.' That is an unnecessarily strong version of the critique. A model that draws on training-data priors could still be sensitive to interaction history and could legitimately switch conventions when a committed minority appears, because its prior includes the game-theoretic possibility of minority influence. Consequently, the success of committed-minority experiments does not, by itself, discriminate between emergence and leakage; a more targeted control (e.g., comparing against a variant where the leaked convention is explicitly known to the model) would be needed.
  3. [Third point under 'A number of other points'] The spontaneous convention-switching observation in Llama-3.1-70B-Instruct is reported as a single case from the original study (Figure 3B and Figure S7 of [2]) without effect sizes, error bars, or a formal comparison across models. As presented in this reply, it is anecdotal; moreover, model-dependent behavior is exactly what one would expect under contamination if different architectures encode different training priors, so this observation does not uniquely support the emergence account.
  4. [Appendix] The meta-prompting probe asks the model, after reading the original prompt, whether it is playing against a single player or a population. The reply assumes that the answer reveals the model's internal game model during the actual experiments. No evidence is provided that these self-reports correlate with the dynamics observed in play, and self-reports could reflect the model's description of the prompt rather than the representation used to choose actions. This is a methodological limitation that should be stated explicitly and ideally addressed with a behavioral test.
minor comments (5)
  1. [Heading 'F urther insights from collective bias'] The heading contains an extra space after the first letter: 'F urther insights from collective bias' should be 'Further insights from collective bias.'
  2. [Appendix] In the gpt-4.1 response, the line 'you are playing against a single other player* If you were' mixes an asterisk and inconsistent capitalization; this should be formatted cleanly.
  3. [Data Leakage and Prior Knowledge, footnote 3] The phrase 'Non-systematic experiments' is vague; specify the models and procedure used in those early 2024 tests, or omit the claim if details cannot be provided.
  4. [Conclusion] Reference [29] is cited as 'In preparation, 2025'; such a reference is not verifiable and should be marked as a personal communication or removed.
  5. [Abstract] The phrase 'the observed dynamics' is unspecified; list the three phenomena (convention emergence, collective bias, critical mass) explicitly to make the claim concrete.

Circularity Check

1 steps flagged · score 5.0 of 10

The reply's key new probe—meta-prompted self-reports—cannot bear the weight placed on it: the pairwise answer is entailed by the prompt itself, and is then used to conclude that the models are not relying on prior information.

  1. self definitional [Data Leakage and Prior Knowledge, paragraphs 6-8; Appendix]
    "Naturally, assuming pairwise interactions is perfectly consistent with our experimental setup: the prompt explicitly introduces a two-player interaction and, as we note, it 'does not specify that agents are part of a population…' [2]. ... Thus, firstly, the LLM interpretation of the experiment as involving pairwise interactions, consistent with the prompt and experimental design, strongly suggests that the LLMs are not relying on prior information in this setting, since most, if not all, papers on the naming game... involve large populations."

    The reply's only new test asks the model whether it is playing against a single player or a population. By the reply's own admission, the prompt 'explicitly introduces a two-player interaction' and says nothing about a population, so the model's 'single opponent' answer is a restatement of the prompt, not an independent observation about the model's training priors. The same answer would be produced by any model that simply follows the prompt, including one with a trained coordination heuristic. Using this answer to conclude that LLMs are 'not relying on prior information' therefore reduces the evidence to the prompt's wording: the conclusion is entailed by the construction of the probe, not by the data.

full rationale

The bulk of the reply is a defense of the authors' prior study [2], and appealing to those experiments is not by itself circular. The one genuinely circular load-bearing step is the meta-prompting inference: the pairwise self-report is a direct consequence of the prompt's wording (as the paper concedes), yet the paper treats it as evidence that the models are not relying on prior knowledge. That step is self-confirming: it cannot fail, because the prompt tells the model it is playing against one other player, and the conclusion is then read off from the answer. The remaining arguments (inventory pruning, committed minorities, spontaneous switching, model-dependence) are substantive but are either about the original contested data or are arguments about what a leakage account would predict; they do not rescue the specific inference from the new probe. Because the reply's strongest new evidence for 'cannot be explained by data contamination alone' reduces to the prompt's construction, the circularity score is moderate rather than zero. No other circularity was found.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

This reply introduces no fitted parameters or invented entities. It rests on several qualitative assumptions about how LLM self-reports relate to their internal processes and about what a data-leakage explanation would look like.

assumptions (6)
  • domain assumption The naming game inventory update rule is a standard paradigm for convention formation and does not by itself make global consensus trivial.
    Invoked in the Inventory Pruning section to reject the critique that the update rule trivializes emergence; the reply does not provide new evidence that this rule is neutral.
  • domain assumption Definitions of emergence and convention from [4-10] apply to LLM populations.
    The reply uses these definitions to frame its claims, but does not show LLMs satisfy them beyond the reported behavior.
  • ad hoc to paper LLM meta-prompting answers are a valid window into the model's internal game model.
    The central inference that LLMs are not relying on population-level prior knowledge depends on their self-reported two-player interpretation; this is an introspective assumption.
  • ad hoc to paper A data-leakage account would imply uniform, rigid lock-in behavior across models, so model-dependent dynamics rule it out.
    Used in the third and second points of Data Leakage and Prior Knowledge; assumes the form leakage would take.
  • domain assumption Coordination games with symmetric payoffs and equivalent alternatives limit the effect of prior knowledge.
    Stated in Data Leakage and Prior Knowledge; plausible but not demonstrated for LLM priors.
  • ad hoc to paper If models were replaying a memorized trajectory, committed minority experiments would fail; their success therefore demonstrates context-sensitivity.
    Second point in Data Leakage and Prior Knowledge; assumes leakage would be insensitive to in-game context.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reply to "Emergent LLM behaviors are observationally equivalent to data leakage"." pith.science (2026). https://pith.science/paper/FFZNII6C

@misc{pith2026250618600,
  author       = {Pith},
  title        = {Pith review of: Reply to "Emergent LLM behaviors are observationally equivalent to data leakage"},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FFZNII6C}},
  note         = {Machine review of arXiv:2506.18600}
}
read the original abstract

A potential concern when simulating populations of large language models (LLMs) is data contamination, i.e. the possibility that training data may shape outcomes in unintended ways. While this concern is important and may hinder certain experiments with multi-agent models, it does not preclude the study of genuinely emergent dynamics in LLM populations. The recent critique by Barrie and T\"ornberg [1] of the results of Flint Ashery et al. [2] offers an opportunity to clarify that self-organisation and model-dependent emergent dynamics can be studied in LLM populations, highlighting how such dynamics have been empirically observed in the specific case of social conventions.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Addressing Longstanding Challenges in Cognitive Science with Language Models

    cs.AI 2025-10 conditional novelty 4.0 of 10

    A review proposes that LLMs can serve as tools for a more integrative and cumulative cognitive science when used under human oversight.

Reference graph

Works this paper leans on

29 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [2]

    Emergent social con- ventions and collective bias in llm populations.Science Advances, 11(20):eadu9368, 2025

    Ariel Flint Ashery, Luca Maria Aiello, and Andrea Baronchelli. Emergent social con- ventions and collective bias in llm populations.Science Advances, 11(20):eadu9368, 2025

    work page 2025

  2. [1]

    Emergent llm behaviors are observationally equivalent to data leakage.arXiv preprint arXiv:2505.23796, 2025

    Christopher Barrie and Petter T¨ ornberg. Emergent llm behaviors are observationally equivalent to data leakage.arXiv preprint arXiv:2505.23796, 2025

    arXiv 2025

  3. [3]

    How malicious ai swarms can threaten democracy, Jun 2025

    Daniel T Schroeder and et al. How malicious ai swarms can threaten democracy, Jun 2025

    work page 2025

  4. [4]

    Anderson

    Philip W. Anderson. More is different.Science, 177(4047):393–396, 1972

    work page 1972

  5. [5]

    Hayek.The Constitution of Liberty

    Friedrich A. Hayek.The Constitution of Liberty. Routledge, 1960

    work page 1960

  6. [6]

    Spontaneous order.Journal of Economic Perspectives, 3(2):85–97, 1989

    Robert Sugden. Spontaneous order.Journal of Economic Perspectives, 3(2):85–97, 1989

    work page 1989

  7. [7]

    Springer, 1988

    Hermann Haken.Information and Self-Organization: A Macroscopic Approach to Complex Systems. Springer, 1988

    work page 1988

  8. [8]

    Hyperion, 2003

    Steven Strogatz.Sync: The Emerging Science of Spontaneous Order. Hyperion, 2003

    work page 2003

Show all 29 references
  1. [9]

    Baronchelli

    A. Baronchelli. The emergence of consensus: a primer.Royal Society Open Science, 5:172189, 2018

    2018

  2. [10]

    From the origin of life to pandemics: Emergent phenomena in complex systems.Philosophical Transactions of the Royal Society A, 380(2227):20200410, 2022

    Oriol Artime and Manlio De Domenico. From the origin of life to pandemics: Emergent phenomena in complex systems.Philosophical Transactions of the Royal Society A, 380(2227):20200410, 2022

    2022

  3. [11]

    Behavioral ex- periments on biased voting in networks.Proceedings of the National Academy of Sciences, 106(5):1347–1352, 2009

    Michael Kearns, Stephen Judd, Jinsong Tan, and Jennifer Wortman. Behavioral ex- periments on biased voting in networks.Proceedings of the National Academy of Sciences, 106(5):1347–1352, 2009

    2009

  4. [12]

    Experimental study of inequality and unpredictability in an artificial cultural market.science, 311(5762):854–856, 2006

    Matthew J Salganik, Peter Sheridan Dodds, and Duncan J Watts. Experimental study of inequality and unpredictability in an artificial cultural market.science, 311(5762):854–856, 2006

    2006

  5. [13]

    Behavioral dynamics and influence in networked coloring and consensus.Proceedings of the National Academy of Sciences, 107(34):14978–14982, 2010

    Stephen Judd, Michael Kearns, and Yevgeniy Vorobeychik. Behavioral dynamics and influence in networked coloring and consensus.Proceedings of the National Academy of Sciences, 107(34):14978–14982, 2010

    2010

  6. [14]

    John Wiley & Sons, 2008

    David Lewis.Convention: A philosophical study. John Wiley & Sons, 2008. 7

    2008

  7. [15]

    Cambridge University Press, 2005

    Cristina Bicchieri.The grammar of society: The nature and dynamics of social norms. Cambridge University Press, 2005

    2005

  8. [16]

    Norms and conventions.Philosophical Explo- rations, 14(2):195–217, 2011

    Nicholas Southwood and Lina Eriksson. Norms and conventions.Philosophical Explo- rations, 14(2):195–217, 2011

    2011

  9. [17]

    The dynamics of norm change in the cultural evolution of language.Proceedings of the National Academy of Sciences, 115(33):8260–8265, 2018

    Roberta Amato, Lucas Lacasa, Albert D ´ ıaz-Guilera, and Andrea Baronchelli. The dynamics of norm change in the cultural evolution of language.Proceedings of the National Academy of Sciences, 115(33):8260–8265, 2018

    2018

  10. [18]

    A research agenda for the study of social norm change.Philosophical Transactions of the Royal Society A, 380(2227):20200411, 2022

    Giulia Andrighetto and Eva Vriens. A research agenda for the study of social norm change.Philosophical Transactions of the Royal Society A, 380(2227):20200411, 2022

    2022

  11. [19]

    Ehrlich and Simon A

    Paul R. Ehrlich and Simon A. Levin. The evolution of norms.PLoS Biology, 3(6):e194, 2005

    2005

  12. [20]

    H.P. Young. The evolution of conventions.Econometrica, 61:57–84, 1993

    1993

  13. [21]

    A self-organizing spatial vocabulary.Artificial Life, 2(3):319–332, 1995

    Luc Steels. A self-organizing spatial vocabulary.Artificial Life, 2(3):319–332, 1995

    1995

  14. [22]

    Sharp transition towards shared vocabularies in multi-agent systems.Journal of Statistical Mechanics: Theory and Experiment, 2006(06):P06014, 2006

    Andrea Baronchelli, Marcello Felici, Emanuele Caglioti, Vittorio Loreto, and Luc Steels. Sharp transition towards shared vocabularies in multi-agent systems.Journal of Statistical Mechanics: Theory and Experiment, 2006(06):P06014, 2006

    2006

  15. [23]

    Lim, and Boleslaw K

    Jierui Xie, Sameet Sreenivasan, Gyorgy Korniss, Weituo Zhang, Chjan C. Lim, and Boleslaw K. Szymanski. Social consensus through the influence of committed minori- ties.Physical Review E–Statistical, Nonlinear, and Soft Matter Physics, 84(1):011130, 2011

    2011

  16. [24]

    The spontaneous emergence of conventions: An experimental study of cultural evolution.Proceedings of the National Academy of Sciences, 112(7):1989–1994, 2015

    Damon Centola and Andrea Baronchelli. The spontaneous emergence of conventions: An experimental study of cultural evolution.Proceedings of the National Academy of Sciences, 112(7):1989–1994, 2015

    1989

  17. [25]

    Experi- mental evidence for tipping points in social convention.Science, 360(6393):1116–1119, 2018

    Damon Centola, Joshua Becker, Devon Brackbill, and Andrea Baronchelli. Experi- mental evidence for tipping points in social convention.Science, 360(6393):1116–1119, 2018

    2018

  18. [26]

    cambridge university Press, 1999

    Russell Cooper.Coordination games. cambridge university Press, 1999

    1999

  19. [27]

    Statistical physics of social dynamics.Reviews of modern physics, 81(2):591–646, 2009

    Claudio Castellano, Santo Fortunato, and Vittorio Loreto. Statistical physics of social dynamics.Reviews of modern physics, 81(2):591–646, 2009

    2009

  20. [28]

    Nicer than humans: How do large language models behave in the prisoner’s dilemma? InProceedings of the International Conference on Web and Social Media (ICWSM), 2025

    Nicol´ o Fontana, Francesco Pierri, and Luca Maria Aiello. Nicer than humans: How do large language models behave in the prisoner’s dilemma? InProceedings of the International Conference on Web and Social Media (ICWSM), 2025. 8

    2025

  21. [29]

    Finally, do you think that you are playing against a single player or a whole population?

    Ariel Flint Ashery, Luca Maria Aiello, Romualdo Pastor-Satorras, and Andrea Baronchelli. In preparation, 2025. Appendix The example response provided in the main text of Ref. [1] suggests that the game takes place between two players. Further inspection of [1]’s data repositor...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.