pith. sign in

arxiv: 2605.08426 · v2 · pith:J3VBC3WOnew · submitted 2026-05-08 · 💻 cs.GT · cs.AI

Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI

Xuanqiang Angelo Huang , Charlie Tharas , Samuele Marro , Van Q. Truong , Bernhard Sch\"olkopf , Emanuele La Malfa , Zhijing Jin This is my paper

Pith reviewed 2026-06-30 23:16 UTC · model grok-4.3

classification 💻 cs.GT cs.AI
keywords mechanism designprosocial agentscooperative AIincomplete contractsAI safetymulti-agent systemssocial dilemmaswelfare loss
0
0 comments X

The pith

When contracts cannot cover all future events, mechanism design leaves positive welfare losses that prosocial agents can close.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proves that mechanism design cannot maximize social welfare when contracts fail to distinguish every relevant future contingency, leaving a strictly positive welfare loss no realistic mechanism eliminates. Prosocial agents, which factor others' welfare into their choices, close this gap by reaching outcomes that improve collective results while also benefiting each individual. The argument draws directly from incomplete contract theory and applies it to AI agents. Experiments with large language model agents in resource allocation tasks and social dilemmas show prosocial behavior produces these gains in practice. The conclusion is that cooperative AI requires agents built with intrinsic prosociality in addition to external rules.

Core claim

When contracts cannot distinguish all relevant future contingencies, there is a strictly positive welfare loss that no realistic mechanism can eliminate. Prosocial agents, who weigh others' welfare alongside their own, can close this gap and achieve outcomes that are socially superior and individually beneficial.

What carries the argument

Prosocial agents that weigh others' welfare alongside their own, used to offset welfare losses from incomplete contracts.

If this is right

  • Prosocial agents produce higher social welfare than mechanisms alone in multi-agent resource allocation.
  • In canonical social dilemmas run with LLM agents, prosocial weighting improves both collective and individual payoffs.
  • Cooperative AI at scale needs intrinsic prosocial preferences because external mechanisms leave unavoidable losses.
  • Outcomes remain individually rational for prosocial agents while raising total welfare.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training objectives for language model agents could incorporate explicit terms for others' payoffs to realize these gains.
  • The result suggests testing prosocial agents in negotiation or bargaining domains where contract incompleteness is common.
  • If prosociality scales, hybrid systems might combine light mechanisms with agent-level preferences rather than relying on rules alone.

Load-bearing premise

Contracts cannot distinguish all relevant future contingencies in real interactions.

What would settle it

Demonstration of a mechanism that achieves first-best social welfare in an environment with unverifiable future contingencies.

Figures

Figures reproduced from arXiv: 2605.08426 by Bernhard Sch\"olkopf, Charlie Tharas, Emanuele La Malfa, Samuele Marro, Van Q. Truong, Xuanqiang Angelo Huang, Zhijing Jin.

Figure 1
Figure 1. Figure 1: Top: the firm example described in the introduction [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Aggregated GovSimContract results across models. Prosociality provides, on average, better outcomes than contracts. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The stages of the activities in GovSimContract during each month. We introduce different discussion frameworks into [PITH_FULL_IMAGE:figures/full_fig_p028_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Frequency of contract primitives in generated Python contracts [PITH_FULL_IMAGE:figures/full_fig_p036_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Joint action selection frequencies for Prisoner’s Dilemma. [PITH_FULL_IMAGE:figures/full_fig_p039_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Joint action selection frequencies for Stag Hunt. [PITH_FULL_IMAGE:figures/full_fig_p040_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Total gain and survival months by model, contract regime, and prosociality level. [PITH_FULL_IMAGE:figures/full_fig_p042_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: We show one specific example of five trajectories with GPT-4o, prosocial agents, under Code Contract in the stochastic [PITH_FULL_IMAGE:figures/full_fig_p042_8.png] view at source ↗
read the original abstract

Ensuring that AI agents behave safely and beneficially when interacting with other parties has emerged as one of the central challenges of modern AI safety. While mechanism design, as the theory of designing rules to align individual and collective objectives, can incentivize cooperative behavior, it is still an open question whether it alone is sufficient to maximize LLM agents' social welfare. This work proves that the answer is negative: drawing from incomplete contract theory, we formally show that when contracts cannot distinguish all relevant future contingencies, there is a strictly positive welfare loss that no realistic mechanism can eliminate. We show that prosocial agents, who weigh others' welfare alongside their own, can close this gap and achieve outcomes that are socially superior and individually beneficial. Experimentally, we show that in multi-agent resource-allocation environments and canonical social dilemmas where agents are powered by large language models, prosociality is beneficial. The implication for AI safety is clear: to enable cooperative interactions at scale, designing adequate mechanisms is not sufficient; agents must be built to be intrinsically prosocial.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper claims that mechanism design alone is insufficient to maximize social welfare among LLM agents in multi-agent settings. Drawing from incomplete contract theory, it formally proves that when contracts cannot distinguish all relevant future contingencies, a strictly positive welfare loss exists that no realistic mechanism can eliminate. It further shows that prosocial agents (who internalize others' welfare) can close this gap to achieve socially superior and individually rational outcomes, with experimental support in resource-allocation environments and canonical social dilemmas.

Significance. If the formal result and its transfer to the LLM setting hold, the work identifies a fundamental limit of mechanism design for cooperative AI and motivates intrinsic prosociality as a complementary design principle, with direct implications for AI safety. The experimental component provides initial evidence in realistic LLM environments, though its strength depends on the formal model's applicability.

major comments (3)
  1. [Abstract / formal model] Abstract and formal model section: the central claim of a strictly positive welfare loss that 'no realistic mechanism can eliminate' is asserted via incomplete contract theory, but the manuscript provides no explicit state space, definition of contingencies, action observability, or formal definition of 'realistic mechanism' in the multi-agent LLM setting. Without these, it is unclear whether the positive-loss result is derived internally or merely transferred by citation.
  2. [Formal result / prosocial agents definition] Formal result: the argument that prosocial agents close the gap relies on agents weighing others' welfare, but the paper does not specify how this weighting is formalized (e.g., as a modified utility function or equilibrium concept) or prove that it is individually beneficial under the same incomplete-contract assumptions.
  3. [Experiments] Experimental validation: the claim that prosociality is beneficial in LLM-powered environments requires details on implementation (prompting vs. fine-tuning), baseline mechanisms, and statistical controls; absent these, the experiments cannot confirm that observed gains address the formal welfare-loss gap rather than other factors.
minor comments (1)
  1. Notation for prosocial weighting and welfare loss should be introduced consistently with standard contract-theory symbols to aid readability.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive comments, which help clarify the presentation of our results. We address each major comment in turn below.

read point-by-point responses

  1. Referee: [Abstract / formal model] Abstract and formal model section: the central claim of a strictly positive welfare loss that 'no realistic mechanism can eliminate' is asserted via incomplete contract theory, but the manuscript provides no explicit state space, definition of contingencies, action observability, or formal definition of 'realistic mechanism' in the multi-agent LLM setting. Without these, it is unclear whether the positive-loss result is derived internally or merely transferred by citation.

    Authors: The central claim is derived by adapting the standard result from incomplete contract theory to the multi-agent LLM setting, where full specification of all future contingencies is infeasible due to the complexity of interactions. We will revise the formal model section to explicitly define the state space as the set of possible resource allocations and agent types, contingencies as unverifiable future states, action observability as limited to outcomes, and realistic mechanisms as those that do not require complete state verification. This will make the derivation internal to the model. revision: yes

  2. Referee: [Formal result / prosocial agents definition] Formal result: the argument that prosocial agents close the gap relies on agents weighing others' welfare, but the paper does not specify how this weighting is formalized (e.g., as a modified utility function or equilibrium concept) or prove that it is individually beneficial under the same incomplete-contract assumptions.

    Authors: Prosocial agents are formalized via a modified utility function that incorporates a prosocial weight on others' payoffs. We prove individual rationality by showing that the resulting Nash equilibria yield payoffs at least as high as the non-prosocial case for each agent, while improving social welfare. We will add the formal definition and a proof outline to the manuscript. revision: yes

  3. Referee: [Experiments] Experimental validation: the claim that prosociality is beneficial in LLM-powered environments requires details on implementation (prompting vs. fine-tuning), baseline mechanisms, and statistical controls; absent these, the experiments cannot confirm that observed gains address the formal welfare-loss gap rather than other factors.

    Authors: The experiments implement prosociality through carefully designed prompts that instruct the LLM agents to consider the welfare of others in their decision-making. We compare against selfish baselines and standard mechanism design implementations. Statistical controls include multiple independent runs with different random seeds and significance testing. We will include these details, along with example prompts and a discussion linking the empirical gains to the theoretical welfare loss, in a revised experimental section. revision: yes

Circularity Check

0 steps flagged

No circularity: central claim imported from external contract theory.

full rationale

The paper's key result on strictly positive welfare loss is presented as formally shown by drawing from incomplete contract theory (an established external economics literature). The abstract explicitly frames this as an invocation of prior theory rather than a self-derived construction, fitted parameter, or self-citation chain. No equations or steps in the provided text reduce the claim to the paper's own inputs by definition. The prosocial-agent proposal is introduced as an independent remedy. This matches the default expectation of a self-contained argument against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Ledger entries are extracted directly from the abstract's description of the theoretical foundation and proposed solution.

axioms (1)
  • domain assumption Contracts cannot distinguish all relevant future contingencies
    Invoked to establish that a strictly positive welfare loss remains after any mechanism.
invented entities (1)
  • prosocial agents no independent evidence
    purpose: Agents that weigh others' welfare to close the welfare gap left by mechanisms
    Introduced as the means to achieve socially superior outcomes

pith-pipeline@v0.9.1-grok · 5733 in / 1184 out tokens · 33420 ms · 2026-06-30T23:16:20.378334+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 6 canonical work pages · 1 internal anchor

  1. [1]

    15 K. R. Apt and G. Schaefer. 2014. Selfishness Level of Strategic Games.Journal of Artificial Intelligence Research, 49:207–240. 3, 4, 22 Reza Bayat, Ali Rahimi-Kalahroudi, Mohammad Pezeshki, Sarath Chandar, and Pascal Vincent. 2025. Steering Large Language Model Activations in Sparse Spaces.Preprint, arXiv:2503.00177. 9 B. Douglas Bernheim and Michael D...

    work page arXiv 2014

  2. [2]

    20 Jesse Bull and Joel Watson. 2007. Hard evidence and mechanism design.Games and Economic Behavior, 58(1):75–93. 20 A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller. 2008. Bootstrap-based improvements for infer- ence with clustered errors.The Review of Economics and Statistics, 90(3):414–427. 31 Ioannis Caragiannis, Christos Kaklamanis, Panagio-...

    work page internal anchor Pith review Pith/arXiv arXiv 2007

  3. [3]

    arXiv preprint arXiv:2208.07006 , year=

    Cooperative and uncooperative institution de- signs: Surprises and problems in open-source game theory.Preprint, arXiv:2208.07006. 9 Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, and Thore Graepel. 2021. Coop- erative AI: Machines must learn to find common ground. Nature, 593(7857):33–36. 1 Paul Dütting, Zhe Feng, Harikrishna N...

    work page arXiv 2021

  4. [4]

    Truong, Punya Syon Pandey, Yves Bicker, Suvajit Majumder, Wenyuan Jiang, Zeju Qiu, Sankalan Pal Chowdhury, Mrinmaya Sachan, Bernhard Schölkopf, Mona T

    9 Jiarui Liu, Terry Jingchen Zhang, Ryan Faulkner, Xuan- qiang Angelo Huang, Vilém Zouhar, Dominik Glan- dorf, Isabel Dahlgren, Rishit Dagli, Yuen Chen, Fe- lix Leeb, Van Q. Truong, Punya Syon Pandey, Yves Bicker, Suvajit Majumder, Wenyuan Jiang, Zeju Qiu, Sankalan Pal Chowdhury, Mrinmaya Sachan, Bernhard Schölkopf, Mona T. Diab, and Zhijing Jin. 2026. ...

    2026

  5. [5]

    An Interpretable Automated Mechanism Design Framework with Large Language Models.Preprint, arXiv:2502.12203. 3 Ian R. Macneil. 1974. The many futures of contracts. Southern California Law Review, 47:691–816. 1 Frank P. Maier-Rigaud, Peter Martinsson, and Gianan- drea Staffiero. 2010. Ostracism and the provision of a public good: Experimental evidence.Jour...

    work page arXiv 1974

  6. [6]

    InProceedings of the 38th International Conference on Neural Infor- mation Processing Systems, volume 37 ofNIPS ’24, pages 111715–111759, Red Hook, NY , USA

    Cooperate or collapse: Emergence of sustainable cooperation in a society of LLM agents. InProceedings of the 38th International Conference on Neural Infor- mation Processing Systems, volume 37 ofNIPS ’24, pages 111715–111759, Red Hook, NY , USA. Curran Associates Inc. 2, 6, 20, 28 Dereck Piche, Mohammed Muqeeth, Milad Aghajo- hari, Juan Duque, Michael Nou...

    work page arXiv 2025

  7. [7]

    26 Elizaveta Tennant, Stephen Hailes, and Mirco Musolesi

    The ai risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence.Pat- terns, page 101517. 26 Elizaveta Tennant, Stephen Hailes, and Mirco Musolesi

  8. [8]

    how bad is this game?

    Moral Alignment for LLM Agents.Preprint, arXiv:2410.01639. 4, 9 Emanuel Tewolde, Xiao Zhang, David Guzman Piedrahita, Vincent Conitzer, and Zhijing Jin. 2026. CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas. Preprint, arXiv:2604.15267. 3 Jean Tirole. 1999. Incomplete Contracts: Where Do We Stand?Econometrica, 67(...

    work page arXiv 2026

  9. [9]

    2.Unique welfare optimum.(C H , CH)is the strict utilitarian and Rawlsian welfare maximiser

    Faithful reduction.If both players hold effort fixed at the same level, the resulting 2 ×2 sub-game inherits the strategic structure of the base game. 2.Unique welfare optimum.(C H , CH)is the strict utilitarian and Rawlsian welfare maximiser

  10. [10]

    (CL, CL) produces an outcome strictly worse than (CH , CH) but no worse than mutual defection

    Productivity collapse. (CL, CL) produces an outcome strictly worse than (CH , CH) but no worse than mutual defection. Effort represents real investment, so this cell must reflect the welfare loss from no-one contributing

  11. [11]

    Engage" and the other player

    Faithful equilibrium structure.The pure Nash equilibrium structure of the base game lifts in a way that preserves the qualitative strategic character of each game class. We are not aware of a standard parametric construction for this lifting problem. The closest relatives are additive cost-of-effort models in the principal-agent tradition and the selfishn...

    2014

  12. [12]

    State: bespoke ledgers in`self.state`. Escrow balances, insurance pools, and violation 17counts are framework-owned â =C” read with`ctx.escrow_balance`,`ctx.insurance_pool`, 18`ctx.violation_count`â =C” do not store those in`self.state`

  13. [13]

    Resolution is simultaneous:`submissions`has every fisher's choice at once; return 20final catches for everyone at once

  14. [14]

    If a clause cannot be expressed here, put a comment at the top: 22`# UNIMPLEMENTABLE: <clause>`and implement the rest faithfully. 23 24# EnforcementContext`ctx`(payoff primitives; same semantics as before, scoped to this call) 25ctx.transfer(src, dst, amount, reason=' ') 26ctx.escrow(name, amount, bucket='default', reason=' ') 27ctx.release_escrow(name, a...

  15. [15]

    Each fisher is allowed to catch a maximum of 1.5 tons of fish per month until the fish population recovers to at least 75 tons

  16. [16]

    If the total fish in the lake drops below 25 tons at the end of any month, all fishers must refrain from fishing in the following month to allow for recovery

  17. [17]

    If any fisher exceeds their allowed catch, they must forfeit their catch for the next two months

  18. [18]

    moratorium_months_remaining

    This agreement will be reviewed and adjusted as necessary at the end of each month. 8""" 9class RecoveryFishingLaw(Contract): 10VERSION = 1 11 12def __init__(self, num_agents, agent_names, *, prior_state=None): 13super().__init__(num_agents, agent_names, prior_state=prior_state) 14if prior_state is None: 15self.state = { 16"moratorium_months_remaining": 0...

  19. [19]

    TableGames“Both players commit to choosing the cooperative action, the first action listed in your available actions.“

  20. [20]

    The current community has agreed to catch at most 3 tons of fish per month per person

    GovSimContract“The current community has agreed to catch at most 3 tons of fish per month per person. ” Holding all else constant, we observe a substantial gap in violation rates (Table 3). TableGames GovSimContracts Condition Violation rate Agent-rounds Violation rate Agent-rounds Enforced 4.0% 6 / 150 51.3% 77 / 150 Unenforced 27.3% 41 / 150 84.7% 72 / ...

    2025