{"id":"c433cfdb-2c3b-4405-9fa6-32bb3a7cf498","arxiv_id":"2411.10294","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM agents playing repeated prisoner's dilemma did not show the network-stabilized cooperation seen in humans, and GPT-3.5 barely responded to network structure.","lead":"This study asked whether chatbots that play cooperation games act like humans when placed on a fixed social network. They did not: unlike people, the language model agents did not cooperate more on networks, and one model cooperated more in random encounters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central claim that LLMs do not alter cooperative behavior in response to network structure or evolving social contexts is directly contradicted by its own Fig. 7 (large cooperation drops when neighbors defect) and by GPT-4's higher cooperation in well-mixed than structured networks.","rationale":"I read the paper as an empirical caution about using 2024 LLMs as human surrogates in networked social dilemmas. The authors deserve credit for transferring a classic human experiment to multiple models and comparing directly to human data. However, the central claim as stated in the title/abstract is broader than the evidence and, more importantly, is inconsistent with the paper's own figures. The reader's identified weak point—that the prompt may not convey the graph topology, so the null result could reflect missing information rather than missing capacity—is real and would be a good follow-up experiment. But the internal contradiction is more load-bearing because it does not depend on any external assumption: Fig. 7 shows strong behavioral adjustment to evolving neighbor composition, and Fig. 4/3 show GPT-4 responding differently to well-mixed versus ring structures. Therefore the blanket statement 'LLMs do not alter their cooperative behavior in response to network structures or evolving social contexts' is false as written. The appropriate fix is to restate the conclusion to say that static network structure does not stabilize LLM cooperation relative to well-mixed settings, and that LLM adjustments are weaker or opposite to human patterns. This supports the reader's CONDITIONAL verdict: the core negative finding may hold for the tested conditions, but the paper's own results require re-qualification. No new experiments are strictly needed to identify the contradiction; a direct quantitative re-analysis of Figures 7 and 4 would settle the magnitude of the discrepancy.","tokens_in":17559,"tokens_out":7806,"duration_ms":73738,"concrete_test":"Re-analyze the underlying data for Fig. 7 and Fig. 4: compute per-model mean cooperation during the first phase (all four neighbors cooperate) versus the final phase (three defectors, one cooperator), and the well-mixed versus ring difference for GPT-4 at k=2, b/c=6. If the phase difference and the well-mixed/ring difference are statistically significant (e.g., via confidence intervals from the five runs), then the paper's own results contradict the claim that LLMs do not alter behavior in response to evolving social contexts or network structure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—'LLMs do not alter their cooperative behavior in response to network structures or evolving social contexts' (Abstract; also Discussion: 'LLMs do not distinguish between well-mixed and structured settings')—is falsified by the paper's own data. In the controlled-stimuli experiment (Fig. 7), Claude and GPT-4 drop from near-total cooperation to near-zero when cooperative neighbors are replaced by defectors; the paper itself states these models 'are sensitive to shifts in their social environment.' In the main experiments (Fig. 4D vs 4A, Fig. 3), GPT-4 cooperates more in the well-mixed condition than in the fixed ring, so it does distinguish the two structures, albeit in the opposite direction from humans. The abstract's later phrase 'with limited adjustment to networked contexts' concedes some adjustment, which is inconsistent with 'do not alter.' This is not an overgeneralization or a missing reference; it is an internal inconsistency between the stated conclusion and the presented results. At minimum, the conclusion must be narrowed to 'static network structure does not increase LLM cooperation relative to well-mixed settings in these models,' and the blanket 'do not alter' phrasing must be removed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports experiments in which LLM agents (GPT-3.5, GPT-4, Claude, Mixtral) play repeated Prisoner's Dilemma games in both well-mixed and static ring-network conditions, closely following the design of Rand et al. (2014). The authors find that, unlike human participants, LLMs do not show higher cooperation in static networks relative to well-mixed populations: GPT-3.5 is largely insensitive to network parameters, GPT-4 cooperates more in well-mixed settings, and a controlled stimulus experiment shows that some models respond to the composition of neighbors. The paper concludes that static network structure cannot stabilize cooperation among LLM agents and that LLMs do not adapt to network structures or evolving social contexts.","tokens_in":17729,"tokens_out":5242,"duration_ms":48533,"significance":"If the findings are robust, they provide a cautionary data point for the growing use of LLMs as human surrogates in behavioral experiments. The paper includes a replication of a well-known human experiment, reports error bands and some inferential statistics, and makes its prompt content available, which are strengths. However, the significance is currently limited by overgeneralized claims, an internal inconsistency between the stated conclusion and the presented data, and several methodological ambiguities. The most defensible contribution is a negative result for the specific models and conditions tested; the universal 'cannot' claim is not supported.","major_comments":[{"comment":"The abstract and Discussion state that 'LLMs do not alter their cooperative behavior in response to network structures or evolving social contexts,' but this is contradicted by the paper's own data. In Fig. 7, Claude and GPT-4 drop from near-total cooperation to near-zero when cooperative neighbors are replaced by defectors, and the Results explicitly say these models 'are sensitive to shifts in their social environment.' Moreover, GPT-4 shows higher cooperation in the well-mixed condition than in the fixed-ring condition (Fig. 3C vs. Fig. 3B; Fig. 4D vs. Fig. 4A), so it does distinguish the two structures, albeit in the opposite direction from humans. This internal inconsistency undermines the central claim as stated. The conclusion should be narrowed to 'LLMs do not show the human pattern of increased cooperation in static networks,' and the blanket 'do not alter' phrasing should be removed.","section":"Abstract and Discussion"},{"comment":"The title and abstract assert that 'static network structure cannot stabilize cooperation among Large Language Model agents,' a universal negative, but the evidence comes from only four model families (GPT-3.5, GPT-4, Claude, Mixtral), with no model versions or sampling temperatures reported, and from only one network topology (circulant rings with k = 2, 4, 6). This is insufficient to support a claim about all LLMs and all static network structures. The claim should be restricted to the tested models and settings, and the title should be revised accordingly.","section":"Title and Experimental Setup"},{"comment":"The prompt tells the participant 'You will be connected to the same people every round' and shows neighbor choices, but it never provides the network structure, such as the ring topology, the node's degree, or the fact that neighbors are connected to each other. The Discussion attributes the absence of network-adaptive behavior to a lack of capacity to infer network structure, but an alternative explanation is that the models simply lack the information needed to infer the structure. The paper needs a control condition in which the graph structure is explicitly described to distinguish between incapacity and missing information; without this, the conclusion that LLMs 'lack the capacity to infer network structure' does not follow.","section":"Methods, 'Prompting the LLMs'"},{"comment":"The well-mixed condition is severely underspecified. The paper contrasts 'well-mixed' with 'structured networks' but does not state how the LLM is paired with other players in each round (e.g., random rematching, a complete graph, or something else). In the human benchmark (Rand et al. 2014), well-mixed means random, one-shot interactions without repeated partners. For LLM agents, the implementation matters because the dialogue format may carry over information between rounds, making the comparison with the human condition ambiguous. The manuscript must specify the matching algorithm and verify its comparability to the human protocol.","section":"Methods, 'Experimental setup'"},{"comment":"No model version (e.g., gpt-3.5-turbo vs. text-davinci-003), temperature, or other sampling parameters are reported for any of the four models. LLM behavior is known to vary substantially with these choices, and the paper's conclusion depends on a small number of model calls. This omission prevents independent replication and weakens the generality of the findings. At a minimum, the authors should report these details; ideally, they should probe sensitivity to them.","section":"General reproducibility"}],"minor_comments":[{"comment":"In the paragraph beginning 'In contrast, GPT-4 shows a more dynamic approach,' the text says 'when the network structure is sparse, with fewer connections (such as k = 2k)'; this appears to be a typo for 'k = 2'.","section":"Results"},{"comment":"The name of the Mixtral model is written as 'Mistral' in the Discussion section; the authors should use one consistent spelling.","section":"Throughout"},{"comment":"There are duplicate references: Ref. 8 and Ref. 15 are both Lorè, N. & Heydari, B. (2023), and Ref. 13 and Ref. 31 are both Rand et al. (2014). The reference list should be consolidated.","section":"References"},{"comment":"The caption says the neighborhood changes from four to three cooperators in panel A at 'timestep 5,' but the text describes 'a progressive replacement of cooperators with defectors in three stages' over the course of 25 rounds; the exact timing and mapping of panels to stages should be clarified for readability.","section":"Figure 7"},{"comment":"The figure captions contain apparent text-extraction artifacts of the form '/gid00002/gid00001/...' that make parts of the captions unreadable; these need to be cleaned up in the final version.","section":"Figure captions"},{"comment":"The paper says experiments are repeated five times for averages, but Fig. 7 states values are averaged over 10 runs; this discrepancy should be clarified.","section":"Experimental setup"}],"recommendation":"major_revision","confidential_remarks":"The paper's central 'cannot' claim is overstated relative to the evidence, and the internal inconsistency between the abstract and the data (notably Fig. 7 and GPT-4's well-mixed vs. structured behavior) will need to be resolved in revision. The underlying data on these specific models is a useful addition to the literature on LLM behavior in social dilemmas, and the replication of Rand et al. is a reasonable benchmark. I see no integrity issues; the main problems are framing, experimental detail, and overgeneralization, which are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: a useful empirical caution, but the title and abstract claim more than the data support. The paper's own Figure 7 shows Claude and GPT-4 dropping from near-total cooperation to near-zero when cooperative neighbors are replaced by defectors, and GPT-4 cooperates more in the well-mixed condition than in the fixed ring. So the blanket line that LLMs 'do not alter their cooperative behavior in response to network structures or evolving social contexts' is directly contradicted by their results. What they actually show is that LLMs do not reproduce the human pattern of network-enhanced cooperation; some models even do the opposite.\n\nWhat is genuinely new: they are the first to transfer the Rand et al. network protocol to LLM agents, and the negative result—static network structure does not stabilize cooperation the way it does for humans—is worth knowing. The paper compares four models, includes a controlled stimulus experiment, and keeps the human comparison explicit. Error bands and t-tests are reported.\n\nSoft spots: the overclaim is the big one, but the experimental details are also thin. No model versions or temperature settings are given, the well-mixed implementation is underspecified, and the full prompts, code, and data are not released. The semi-automated repair of dialogues ('rectify dialogues gone awry') leaves room for experimenter judgment. The conclusion rests on two OpenAI models plus Claude and Mixtral, one network topology, and a few parameter combinations. That warrants caution, not a general law.\n\nAlso, the prompt never gives the LLM the graph structure explicitly; it says 'you will be connected to the same people every round' and shows neighbor choices. If the model cannot extract the network from the dialogue, the conclusion that they lack the capacity to infer network structure does not follow.\n\nRecommendation: this deserves a serious referee—the question is live and the data are useful—but it needs major revision: narrow the claim to what the data show, add the missing methods, and release artifacts. As it stands, I wouldn't cite it for the strong conclusion, but the experiments are worth engaging with.","headline":"Useful cautionary data on LLM agents in networked prisoner's dilemma, but the title overclaims: their own figures show LLMs do respond to social context, just not like humans.","tokens_in":18324,"tokens_out":3956,"would_cite":false,"duration_ms":33737,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that current large language model agents do not cooperate more in static social networks than in well-mixed populations, so static network structure cannot stabilize cooperation among them as it does in humans.","keywords":["large language models","prisoner's dilemma","network reciprocity","cooperation","social dilemmas","LLM agents","static networks","human-AI comparison"],"falsifier":"If, when the prompt explicitly supplies the complete list or diagram of neighbors and their past actions, LLM cooperation rises above 50 percent and tracks the $b/c > k$ threshold across values of $k$, the claim that static network structure cannot stabilize LLM cooperation would be disproven.","tokens_in":17310,"feed_emoji":"🤖","tokens_out":6589,"duration_ms":57415,"temperature":0.7,"pith_summary":"This paper asks whether large language models, cast as agents in a repeated prisoner's dilemma—a game in which cooperating costs the player but benefits a partner—reproduce the human tendency to cooperate more in fixed social networks than in fluid, well-mixed populations. It reports that they do not: GPT-3.5 hovers near 50 percent cooperation regardless of network degree or benefit-to-cost ratio, while GPT-4 cooperates more in well-mixed settings and less on static networks, the opposite of the human pattern. The authors conclude that current LLMs lack the reciprocity norms and network inference that let humans stabilize cooperation when the benefit-to-cost ratio exceeds degree, and that claims of LLM-human equivalence in behavioral experiments do not generalize to networked social dilemmas.","feed_headline":"LLMs ignore static networks when deciding to cooperate","feed_subtitle":"In repeated prisoner's dilemma games, GPT-3.5 and GPT-4 show little of the network reciprocity humans display.","key_machinery":"The load-bearing setup is the repeated prisoner's dilemma on circulant ring networks, where each of the $k$ nearest neighbors receives benefit $b$ if the agent cooperates, and cooperation costs $c$; the game is played for about 15 rounds with the same neighbors while the ratio $b/c$ is varied from 2 to 6. The human benchmark is the threshold $b/c > k$, under which cooperation becomes stable in static networks through assortment. The LLM agents are prompted with the human instructions from that experiment and asked to respond with C or D, and their cooperation trajectories are compared across well-mixed and networked conditions, with controlled perturbations of neighbor behavior serving as an additional probe of reciprocity.","core_discovery":"The central claim is that static network structure cannot stabilize cooperation among LLM agents as of 2024. Replicating the design of a classical human experiment—a repeated prisoner's dilemma on ring networks with degree $k=2,4,6$ and benefit-to-cost ratios $b/c=2,4,6$—the authors find that humans cooperate more in the network when $b/c > k$, while GPT-3.5's cooperation stays nearly flat and GPT-4's cooperation is higher in the well-mixed condition than on the network. They interpret this as evidence that LLMs do not strategically adjust to network structure or evolving social context, do not infer the neighborhood they are embedded in, and instead follow rigid, model-specific behavioral patterns. The paper's conclusion is that claims that LLMs can replace human participants in behavioral experiments are overgeneralized.","pith_inferences":["The authors do not vary the information given to the model; a natural extension would present the full neighbor list or a network diagram and test whether cooperation then tracks $b/c > k$, which would separate missing information from missing capability.","Because the result is a snapshot of specific 2024 models, it should be treated as a benchmark rather than a law; future LLMs trained on more social-interaction data may behave differently.","The same controlled-stimulus protocol could be used to measure whether prompting LLMs to reason explicitly about reciprocity, or giving them a norm-rich backstory, restores network-stabilized cooperation.","If the pattern holds across architectures, the experimental template becomes a useful test for social intelligence in AI agents, complementing game-theoretic measures."],"forward_implications":["LLMs cannot currently be used as drop-in human substitutes for studying how network structure shapes cooperation.","Cooperation in LLM social dilemmas is driven more by the model's intrinsic behavior than by the structural parameters $b/c$ and $k$.","The human pattern of network reciprocity—cooperation rising when $b/c > k$—is absent, so future LLM design must explicitly encode social norms or network awareness to close the gap.","The GPT-4 result, higher cooperation in well-mixed than in networked settings, runs opposite to human behavior, so simply making LLMs more cooperative is not the fix."],"supporting_citations":[{"why":"Supplies the human experimental baseline and the $b/c > k$ threshold that the LLM agents fail to reproduce.","marker":"[13]"},{"why":"Provides prior evidence that LLM chatbots show model-specific 'personality' in strategic games, which the authors invoke to explain rigid behavior.","marker":"[5]"},{"why":"Articulates the claim that LLMs can simulate human samples, which this paper argues does not generalize to networked dilemmas.","marker":"[4]"},{"why":"Reports that LLMs behave 'nicer than humans' in the prisoner's dilemma, a contrasting baseline for the present network results.","marker":"[17]"},{"why":"Shows LLM human-likeness varies with cultural context, supporting the paper's caution about generalization.","marker":"[9]"},{"why":"Warns against using LLMs as human surrogates, a position the results reinforce.","marker":"[34]"}],"fun_headline_variants":["LLMs ignore network structure when deciding to cooperate","Static networks fail to stabilize LLM cooperation","No network reciprocity: LLMs don't adapt cooperation","LLMs miss the network effect on cooperation","Human-like network cooperation eludes LLM agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiments assume that telling an LLM it will face the same neighbors each round and showing it their choices gives the model enough information to perceive the network, so its failure to cooperate more on the network is a behavioral limitation rather than a missing-information artifact.","fun_headline_variants_meta":{"raw":{"variants":["LLMs ignore network structure when deciding to cooperate","Static networks fail to stabilize LLM cooperation","No network reciprocity: LLMs don't adapt cooperation","LLMs miss the network effect on cooperation","Human-like network cooperation eludes LLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1374,"prompt_tokens":987,"completion_tokens":387,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":317}},"tokens_in":603,"tokens_out":387,"duration_ms":3987,"temperature":1.0,"reasoning_tokens":317,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:45:35.728044+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If, when the prompt explicitly supplies the complete list or diagram of neighbors and their past actions, LLM cooperation rises above 50 percent and tracks the $b/c > k$ threshold across values of $k$, the claim that static network structure cannot stabilize LLM cooperation would be disproven.","supporting_citations":[{"cited_title":"& Jackson, M","cited_arxiv_id":null,"evidence_quote":"Provides prior evidence that LLM chatbots show model-specific 'personality' in strategic games, which the authors invoke to explain rigid behavior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Articulates the claim that LLMs can simulate human samples, which this paper argues does not generalize to networked dilemmas."},{"cited_title":"J., Park, P","cited_arxiv_id":null,"evidence_quote":"Shows LLM human-likeness varies with cultural context, supporting the paper's caution about generalization."}],"review_version":1}