REVIEW 4 major objections 7 minor 19 references
Reviewed by Pith at T0; open to challenge.
T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →
T0 review · glm-5.2
LLM agents reproduce polarization without being told to
2026-07-09 12:16 UTC pith:YUTEI2W7
load-bearing objection LLM-driven ABM for vaccine opinion dynamics: functional framework with a real circularity problem in the 'emergence' claim the 4 major comments →
A Large Language Model-Driven Agent-Based Modeling Framework with Multi-Round Communication for Simulating Vaccine Opinion Dynamics
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
When LLM-driven agents converse and reflect, they spontaneously produce both assimilative influence (31-41% of interactions depending on scenario) and repulsive influence (29-39%), along with a threshold effect where moderate opinion differences lead to convergence but extreme differences trigger resistance. This matches the predictions of social judgment theory (Sherif and Hovland, 1961) and bounded confidence models (Jager and Amblard, 2005), but the paper claims these patterns arise from the language model's conversational generation rather than from exogenously imposed rules. Additionally, the memory module increases repulsive responses (from 31.0% to 38.6%) and lowers vaccination rates,
What carries the argument
Qwen3-8B
Load-bearing premise
The paper claims that non-linear social influence patterns (assimilation, repulsion, thresholds) 'emerge naturally from LLM-based interactions,' but this assumes the LLM is not simply reproducing patterns it learned from training data that already contains descriptions of these exact social psychology theories.
What would settle it
If the LLM were prompted with a system instruction that explicitly forbids any opinion movement away from a neighbor's position (no repulsion), and the scatter plots still showed substantial second/fourth quadrant activity, the claim of emergent repulsive influence would be undermined. Conversely, if a different LLM architecture not trained on social science literature produced the same patterns, the emergence claim would be strengthened.
Where Pith is reading between the lines
- The paper's claim of emergence rests on the LLM not having been explicitly prompted to produce assimilative or repulsive behavior. However, the LLM's training data likely contains descriptions of social judgment theory, bounded confidence models, and polarization dynamics. If the model has internalized these patterns as textual regularities, the 'emergence' may be circular: the LLM reproduces the
- The fixed openness parameter α=0.5 and the mathematical aggregation layer (Equation 1) constrain how LLM-generated opinion shifts propagate through the network. The non-linear patterns observed in the scatter plots could be partly shaped by this hybrid architecture rather than purely by the LLM's dialogue generation. Disentangling the LLM's contribution from the aggregation mechanism would require
- The finding that memory amplifies repulsive influence suggests a testable hypothesis: agents with longer memory windows should show progressively higher repulsion rates and stronger polarization, as accumulated negative experiences compound resistance. This could be tested by varying memory length as a continuous parameter.
- The 95-agent scale limits the ability to distinguish between genuine emergent polarization and finite-size effects. Repulsive influence at 31% in a small population could produce apparent polarization that would wash out or intensify at larger scales, making the scaling experiment the authors propose a critical validation step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces an LLM-driven agent-based modeling framework using Qwen3-8B to simulate vaccination opinion dynamics. Agents engage in multi-round dialogues, reflect on conversations to update their opinion scores, and update opinions via a mathematical aggregation layer (Eq. 1). The authors evaluate four scenarios varying cognitive modules (memory, prompt diversity) across 95 agents over 10 time steps. They report that memory reduces vaccination rates while prompt diversity increases them, and claim that non-linear social influence patterns (assimilative and repulsive influence, threshold effects) emerge naturally from LLM-based interactions rather than from imposed rules, suggesting potential for level 3 validation of agent-based models.
Significance. The paper addresses a timely question in computational social science: how LLM-based dialogue can replace static numerical rules in agent-based opinion dynamics models. The modular experimental design (Table 2) and the micro-level scatter analysis (Figure 5, Table 3) are commendable attempts to move beyond aggregate outcome reporting to mechanism-level inspection. The framework produces falsifiable, quadrant-based predictions about assimilative versus repulsive influence. The use of synthetic population data with heterogeneous social networks provides demographic grounding.
major comments (4)
- §3.3, final paragraph: The claim that non-linear influence patterns 'emerge naturally from LLM-based interactions' and are 'not imposed exogenously' is not adequately supported. The reflection module (§2.3) feeds the LLM explicit numeric opinion scores for both agents and asks it to output a new numeric opinion score. If Qwen3-8B has learned from training data that large opinion differences lead to resistance (as described in the social judgment theory and bounded confidence literature the paper itself cites — Sherif & Hovland 1961, Jager & Amblard 2005), then the threshold and repulsion patterns in Figure 5 could be the LLM reproducing a learned textual regularity rather than an emergent property of multi-round dialogue. The paper does not discuss this alternative explanation. This is load-bearing because the word 'emergent' is central to the paper's strongest claim. The authors should:
- §2.2.1, Table 1: The openness parameter α=0.5 is fixed for all agents based on Yin et al. (2024), but that work uses a different model architecture (rule-based ABM, not LLM-driven). Since α directly controls the magnitude of opinion change in Eq. (1), the scatter patterns in Figure 5 and the quadrant statistics in Table 3 are partly determined by this fixed parameter. No sensitivity analysis is provided. The paper should either vary α and show that the qualitative patterns persist, or explicitly acknowledge that the reported non-linearities are conditional on this parameter choice.
- §2.3 and §3.3: The opinion space is bounded at [-1,1] and vaccinated agents are pinned at o=1.0. This boundary could create artificial non-linearity in the scatter plots (Figure 5) independent of LLM behavior — agents near the upper boundary have asymmetric room for opinion movement, which could inflate the apparent repulsion share in QII+QIV. The paper does not control for this boundary effect when interpreting the scatter patterns. The authors should either exclude interactions involving agents near the boundary from the scatter analysis, or demonstrate that the non-linear patterns persist away from the boundaries.
- §3, verification paragraph: The filtering of LLM refusals ('As an AI' responses) is mentioned, but the paper does not report what fraction of responses were filtered, whether the filter rate varies across scenarios, or whether filtering introduces systematic bias (e.g., filtering may remove interactions where the LLM was uncomfortable with extreme opinion distances, biasing the remaining data toward assimilation). Since the filtered interactions are part of the dialogue-reflection pipeline that generates the opinion changes analyzed in §3.3, this is relevant to the validity of the micro-level results. The authors should report filter rates per scenario and discuss potential bias.
minor comments (7)
- Table 3 caption references 'Fig.8A–D' but the corresponding figures are labeled Figure 5A–D in the text. This inconsistency should be corrected.
- §2.2.1: The initial opinion distribution is described as N(0, 0.3²) in one place and N(0, 0.3) in another (§2.2.2). These are different distributions. The notation should be consistent.
- §2.3, Eq. (1): The term dO_{i,j,t} is described as 'the direct opinion shift derived from the reflection score following the dialogue,' but the precise computation from the reflection output to dO is not specified. This makes the equation difficult to reproduce.
- §3.1: The claim that 'opinion polarization emerges under all four scenarios' (Figure 3C) is partly an artifact of the design: vaccinated agents are pinned at o=1.0, which mechanically increases the standard deviation. The authors should note this confound when interpreting the polarization trend.
- Figure 2: The example dialogue is useful but the figure quality should be improved for readability in print.
- §2.2.1: The number of dialogue rounds is fixed at two. The paper's title emphasizes 'multi-round communication,' but two rounds is minimal. The authors should justify this choice or discuss how the number of rounds might affect the results.
- The paper would benefit from citing and comparing to Chuang et al. (2024), which is referenced in the introduction but not discussed in relation to the paper's findings on LLM-based opinion dynamics.
Simulated Author's Rebuttal
We thank the referee for a careful and constructive report. The comments identify four substantive issues: (1) the 'emergence' claim may be confounded by the LLM reproducing learned textual regularities from social judgment theory; (2) the fixed openness parameter α=0.5 lacks sensitivity analysis; (3) boundary effects from the [-1,1] opinion space may inflate apparent non-linearity in scatter plots; and (4) LLM refusal filtering rates and potential bias are not reported. We agree that all four points require revision. We can fully address points 1, 3, and 4 with textual additions and supplementary analysis. Point 2 requires a sensitivity analysis that we can partially conduct but may be constrained by computational resources for a full sweep.
read point-by-point responses
-
Referee: §3.3, final paragraph: The claim that non-linear influence patterns 'emerge naturally from LLM-based interactions' and are 'not imposed exogenously' is not adequately supported. The reflection module feeds the LLM explicit numeric opinion scores and asks it to output a new numeric opinion score. If Qwen3-8B has learned from training data that large opinion differences lead to resistance (as in social judgment theory), the threshold and repulsion patterns could be the LLM reproducing a learned textual regularity rather than an emergent property of multi-round dialogue.
Authors: The referee raises a valid and important concern. We agree that the distinction between genuine emergence and reproduction of learned textual regularities is critical, and our current manuscript does not adequately address this alternative explanation. In the revision, we will revise the language in §3.3 to qualify the emergence claim more carefully. Specifically, we will replace 'emerges naturally from LLM-based interactions' with language acknowledging that the observed non-linear patterns arise from the LLM's generative process without being explicitly coded as numerical rules, while noting that the LLM may have internalized regularities from its training corpus that align with social judgment theory. We will add a dedicated paragraph discussing this confound explicitly, noting that our framework does not impose a bounded-confidence threshold or a repulsion rule as a mathematical function (as in Jager & Amblard 2005), but that the LLM's behavior may reflect learned associations from training data that encode similar patterns. We believe this is still a meaningful result—the non-linearities arise from natural language dialogue rather than from an imposed numerical rule—but we agree the word 'emergent' overstates the claim without further evidence. We will also note that disentangling these explanations would require controlled experiments (e.g., ablating the numeric opinion scores from the reflection prompt) that we leave to future work. revision: yes
-
Referee: §2.2.1, Table 1: The openness parameter α=0.5 is fixed for all agents based on Yin et al. (2024), but that work uses a different model architecture (rule-based ABM, not LLM-driven). Since α directly controls the magnitude of opinion change in Eq. (1), the scatter patterns in Figure 5 and the quadrant statistics in Table 3 are partly determined by this fixed parameter. No sensitivity analysis is provided.
Authors: We agree that the lack of sensitivity analysis for α is a gap, and that citing Yin et al. (2024) for this parameter value is insufficient given the architectural difference between their rule-based ABM and our LLM-driven framework. We will conduct a sensitivity analysis by running the baseline and combined scenarios with α values of 0.3, 0.5, and 0.7, and will report whether the qualitative patterns (presence of repulsive interactions, threshold effects) persist across these values. We expect the quadrant proportions to be robust to α because α scales the magnitude of opinion changes uniformly in Eq. (1) without changing their sign, meaning the quadrant classification (which depends on the sign of Δo) should be largely unaffected. However, the boundary effects and the distribution of points within quadrants may shift. We will add this analysis as a supplementary section and explicitly acknowledge in §2.2.1 that the reported patterns are conditional on this parameter choice. If computational constraints prevent a full sweep across all four scenarios, we will at minimum report results for the baseline scenario and discuss the expected generalization. revision: partial
-
Referee: §2.3 and §3.3: The opinion space is bounded at [-1,1] and vaccinated agents are pinned at o=1.0. This boundary could create artificial non-linearity in the scatter plots (Figure 5) independent of LLM behavior — agents near the upper boundary have asymmetric room for opinion movement, which could inflate the apparent repulsion share in QII+QIV.
Authors: This is a well-taken point. The boundary at [-1,1] and the pinning of vaccinated agents at o=1.0 do create asymmetric movement potential that could confound the scatter analysis. In the revision, we will address this in two ways. First, we will re-run the quadrant analysis excluding interactions where either agent's opinion is within 0.15 of either boundary (i.e., |o| > 0.85), and report the revised quadrant proportions. If the repulsion share remains substantially above zero in this restricted sample, this demonstrates that the non-linear patterns are not purely boundary artifacts. Second, we will add a discussion paragraph in §3.3 explicitly acknowledging the boundary effect and its potential to inflate the QII+QIV share, and we will report both the full-sample and boundary-excluded statistics in Table 3 so readers can assess the magnitude of this confound. We note that the vaccinated-agent pinning is a design choice reflecting the irreversibility of vaccination decisions, but we agree it should not contaminate the micro-level influence analysis without explicit acknowledgment. revision: yes
-
Referee: §3, verification paragraph: The filtering of LLM refusals is mentioned, but the paper does not report what fraction of responses were filtered, whether the filter rate varies across scenarios, or whether filtering introduces systematic bias.
Authors: We agree this information should be reported. In the revision, we will add a table reporting the filter rate (percentage of responses containing refusal phrases such as 'As an AI' or 'I cannot provide') per scenario per time step, along with the total number of filtered interactions. We will also discuss whether the filter rate varies systematically across scenarios (e.g., whether the memory scenario, which involves more context, triggers more refusals). Regarding potential bias, we will add a discussion noting that if the LLM tends to refuse more often in interactions involving extreme opinion distances or sensitive vaccination arguments, the filtered sample could be biased toward assimilative interactions, which would mean our reported repulsion shares are conservative estimates. We will also report the filter rate broken down by opinion distance bins to check for this pattern. If the data show no systematic relationship between opinion distance and filter rate, we will state this explicitly; if a relationship exists, we will acknowledge the direction of the bias. revision: yes
Circularity Check
No significant circularity; the 'emergence' concern is a validity/confound issue, not a construction-level circularity.
full rationale
The paper's central claim is that non-linear influence patterns (assimilation, repulsion, thresholds) 'emerge naturally from LLM-based interactions' and are 'not imposed exogenously' (Section 3.3). Tracing the derivation chain: (1) agents are initialized with opinion scores from N(0, 0.3²) — not defined in terms of the output patterns; (2) the LLM (Qwen3-8B) generates dialogue and then a reflection output containing a new numeric opinion score; (3) Eq. 1 is a linear aggregation: O_{i,t+1} = O_{i,t} + α · Σ u_{i,j} · dO_{i,j,t}, where dO comes from the LLM reflection module; (4) the scatter plots (Figure 5) map opinion disparity against opinion change, where the non-linearity must originate in the LLM's input→output mapping since Eq. 1 is linear. The paper does not fit parameters to the target patterns and then 'predict' them. The openness α=0.5 is set from an external citation (Yin et al. 2024), not self-cited. The theoretical frameworks cited (Sherif & Hovland 1961, Jager & Amblard 2005, DeGroot 1974, Flache et al. 2017) are all external authors. The one self-citation (Jiang et al. 2024) is for the synthetic population dataset — data infrastructure, not load-bearing for the emergence claim. The reader's concern that the LLM may reproduce theory-consistent patterns from training data is a legitimate validity risk (training-data contamination confound), but it is not circularity in the technical sense: the paper's equations do not reduce to their inputs by construction, no parameter is fitted to the target quantity and renamed as a prediction, and no self-citation chain forces the conclusion. The 'emergence' claim may be overstated or under-tested, but that is a correctness concern, not a circularity one. Score 2 reflects the minor self-citation for data infrastructure and the fact that the emergence claim, while not circular, is not independently validated against the training-data-contamination alternative the reader raises.
Axiom & Free-Parameter Ledger
free parameters (6)
- Openness parameter α =
0.5 (uniform for all agents)
- Initial opinion distribution =
N(0, 0.3²) truncated to [-1,1]
- Vaccination threshold =
O_i > 0
- Number of dialogue rounds =
2
- Number of time steps =
10
- Agent count =
95
axioms (4)
- domain assumption LLM-generated dialogues and reflections are valid proxies for human opinion formation processes.
- domain assumption The Qwen3-8B model's outputs are not biased by training-data exposure to the social psychology theories the paper claims to reproduce.
- domain assumption The mathematical aggregation in Eq. 1 (weighted linear sum of pairwise opinion shifts) is an appropriate model for combining LLM-generated opinion changes.
- ad hoc to paper Filtering LLM refusals ('As an AI' responses) does not introduce systematic bias into the opinion dynamics.
read the original abstract
Recently, Large Language Models (LLMs) have been utilized in various applications of computational social science and provide the possibility to integrate such models into agent-based modeling to explore the cognitive processes. However, how specific cognitive modules drive individual decisions and macro-level opinion dynamics remains unclear. Therefore, this study introduces a framework that integrates an LLM (Qwen3-8B) into agent-based modeling to investigate this problem, using vaccination opinion dynamics as a case study. We utilize this framework to simulate opinion dynamics among agents with heterogeneous profiles and social networks, evaluating scenarios by enabling different cognitive modules: a memory module and a prompt diversity module. The simulation results reveal that different cognitive modules have opposite impacts on our emergent opinion. Furthermore, the framework reproduces the non-linear behavior patterns of social influence observed in existing research, demonstrating our framework's validity and potential to reach the level 3 validation of agent-based models.
Figures
Reference graph
Works this paper leans on
-
[1]
Emergent Abilities in Large Language Models: A Survey
“Emergent Abilities in Large Language Models: A Survey”.arXiv preprint arXiv: 2503.05788. Chu, Z., Y . Wang, F. Zhu, L. Yu, L. Li, and J. Gu
work page internal anchor Pith review Pith/arXiv arXiv
-
[2]
“Professional Agents – Evolving Large Language Models into Autonomous Experts with Human-Level Competencies”.arXiv preprint arXiv: 2402.03628. Chuang, Y .-S., A. Goyal, N. Harlalka, S. Suresh, R. Hawkins, S. Yang,et al
work page internal anchor Pith review Pith/arXiv arXiv
-
[3]
Simulating Opinion Dynamics with Networks of LLM-based Agents
“Simulating Opinion Dynamics with Networks of LLM-based Agents”.arXiv preprint arXiv: 2311.09618. DeGroot, M. H
work page internal anchor Pith review Pith/arXiv arXiv
-
[4]
Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?
“Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?”. NBER Working Paper w31122, National Bureau of Economic Research. Available at: https://ssrn.com/abstract=4413859 (Accessed: 13 February 2026). Jager, W., and F. Amblard
work page 2026
-
[5]
“Uniformity, bipolarization and pluriformity captured as generic stylized behavior with an agent-based simulation model of attitude change”.Computational & Mathematical Organization Theory10(4):295– 303 https://doi.org/10.1007/s10588-005-6282-2. Zhang and Jiang Jiang, N., F. Yin, B. Wang, X. Luo, H. Purohit, H. Kavak,et al. 2024, nov. “A Large-Scale Geogr...
-
[6]
Reimagining Agent-based Modeling with Large Language Model Agents via Shachi
“Reimagining Agent-based Modeling with Large Language Model Agents via Shachi”.arXiv preprint arXiv: 2509.21862. Larooij, M., and P. Törnberg
-
[7]
“Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations”.arXiv preprint arXiv: 2504.03274. Li, C. J., J. Wu, Z. Mo, A. Qu, Y . Tang, K. I. Zhao,et al
work page internal anchor Pith review Pith/arXiv arXiv
-
[8]
Simulating Society Requires Simulating Thought
“Simulating Society Requires Simulating Thought”.arXiv preprint arXiv: 2506.06958. Liu, J., C. Chu, Y . Zhao, G. Aoki, and Z. Xiao
-
[9]
Generative Agents: Interactive Simulacra of Human Behavior
“Generative Agents: Interactive Simulacra of Human Behavior”.arXiv preprint arXiv: 2304.03442. Sherif, M., and C. I. Hovland. 1961.Social Judgment: Assimilation and Contrast Effects in Communication and Attitude Change. Oxford, England: Yale University Press. Shinn, N., F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao
work page internal anchor Pith review Pith/arXiv arXiv 1961
-
[10]
Reflexion: Language Agents with Verbal Reinforcement Learning
“Reflexion: Language Agents with Verbal Reinforcement Learning”.arXiv preprint arXiv: 2303.11366. Taillandier, P., J. D. Zucker, A. Grignard, B. Gaudou, N. Q. Huynh, and A. Drogoul
work page internal anchor Pith review Pith/arXiv arXiv
-
[11]
Integrating LLM in Agent-Based Social Simulation: Opportunities and Challenges
“Integrating LLM in Agent-Based Social Simulation: Opportunities and Challenges”.arXiv preprint arXiv: 2507.19364. U.S. Census Bureau
-
[12]
2016-2020 American Community Survey 5-Year Public Use Microdata Sample
“2016-2020 American Community Survey 5-Year Public Use Microdata Sample”. Data set. Accessed: 2025-05-19. Vanhée, L., M. Borit, P.-O. Siebers, R. Cremades, C. Frantz, Önder Gürcan,et al
work page 2016
-
[13]
“Large Language Models for Agent-Based Modelling: Current and possible uses across the modelling cycle”.arXiv preprint arXiv: 2507.05723. Yan, Y ., J. Piao, X. Lan, C. Shao, P. Hui, and Y . Li
work page internal anchor Pith review Pith/arXiv arXiv
-
[14]
Simulating Generative Social Agents via Theory-Informed Workflow Design
“Simulating Generative Social Agents via Theory-Informed Workflow Design”.arXiv preprint arXiv: 2508.08726. Yang, A., A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng,et al
work page internal anchor Pith review Pith/arXiv arXiv
-
[15]
“Qwen3 Technical Report”.arXiv preprint arXiv: 2505.09388. Yao, S., J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhanet al
work page internal anchor Pith review Pith/arXiv arXiv
-
[16]
ReAct: Synergizing Reasoning and Acting in Language Models
“ReAct: Synergizing Reasoning and Acting in Language Models”.arXiv preprint arXiv: 2210.03629. Yin, F., A. Crooks, and L. Yin
work page internal anchor Pith review Pith/arXiv arXiv
-
[17]
“Igniting Language Intelligence: The Hitchhiker’s Guide From Chain-of-Thought Reasoning to Language Agents”.arXiv preprint arXiv: 2311.11797. Zhou, J., J. tse Huang, X. Zhou, M. H. Lam, X. Wang, H. Zhu,et al
work page internal anchor Pith review Pith/arXiv arXiv
-
[18]
The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
“The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies”.arXiv preprint arXiv: 2509.18052. Zhou, Y ., J. Li, Y . Xiang, H. Yan, L. Gui, and Y . He
work page internal anchor Pith review Pith/arXiv arXiv
-
[19]
The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis
“The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis”.arXiv preprint arXiv: 2311.00237. AUTHOR BIOGRAPHIES BO ZHANGis a Research Assistant in the Thrust of Urban Governance and Design at The Hong Kong University of Science and Technology (Guangzhou) and Mphil student of Xi’an Jiaotong University. His research focuses ...
work page internal anchor Pith review Pith/arXiv arXiv
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.