REVIEW 3 major objections 4 minor 1 cited by
AI agents that trade delegated cognitive work can outperform self-sufficient agents, but the market rules enabling those gains are the decisive, fragile part.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 16:36 UTC pith:LB36JDY4
load-bearing objection A genuinely useful agent-market testbed with non-obvious findings, but the headline Market-vs-Autarky comparison bundles exchange with the settlement regime, so the size of the trade gains is provisional. the 3 major comments →
Diagon: A Programmable Testbed for AI-Agent Cognitive Labor Markets
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that AI agents trading delegated cognitive work through a programmable market outperform self-sufficient agents on profit and quality, but this advantage is highly dependent on institutional structure. The authors instantiate one concrete market form—first-price sealed-bid auction, incomplete contracts with discretionary payment, bilateral reputation, and periodic evolutionary selection—and vary each mechanism individually. They find that market exchange generates roughly 1.6× the per-task profit of autarky and higher task quality, but that identity transparency reduces cross-family trade by a large effect (d = −1.66), 'honest' instructions increase disputes, and
What carries the argument
Diagon is a programmable market system whose core mechanism is a seven-step round protocol—post, bid, select, execute, evaluate, pay, update—mapped to three design desiderata: allocation (first-price sealed-bid auction), contracting (incomplete contract with a discretionary payment ratio between 0.5 and 1.0), and enforcement (bilateral reputation records plus replicator-style elimination and reproduction every six rounds). The testbed's work is to make each institutional rule individually manipulable while keeping every strategic decision (posting, bidding, selecting, evaluating, paying) a live agent call, so the causal effect of each rule on trade, quality, and wealth becomes observable.
Load-bearing premise
The market's productivity advantage over autarky rests on the two regimes differing only in whether agents trade, but autarky settles payments with deterministic ground-truth scores while the market leaves payment to the poster's subjective discretion—so the gains could come from the evaluation regime rather than from exchange itself.
What would settle it
Run the market baseline with payment settled by the oracle quality score instead of the poster's discretionary ratio (and run autarky with poster-style discretion). If the market's per-task profit advantage over autarky disappears or reverses, the headline gains are an artifact of the settlement regime. The paper's own oracle-calibrated poster model (Table 4) could be substituted for the live poster on the same 1,957 transactions; a material change in the profit gap or dispute rate would show that settlement rules, not trade, drive the result.
If this is right
- Trade among heterogeneous AI agents can create measurable gains over self-sufficiency, with per-task profit about 1.58× higher and quality improved, suggesting specialization is a real source of value in agent economies.
- Institutional rules are not neutral: identity transparency fragments trade along model lines (d = −1.66, p < 0.001), and 'honest' instructions increase disputes, so mechanisms inherited from human markets may backfire for agents.
- The evaluation bottleneck is structural: posters distinguish excellent from failing work but not intermediate quality (within-bin quality–payment correlation r = 0.16 for q < 0.5), so institutional design must accommodate rather than eliminate dispute friction.
- Stronger competitive selection degrades most metrics at once, implying selection pressure alone sustains neither cooperation nor diversity in agent markets.
- Agent-generated natural language exposes decision reasoning—poster text recovers R² = 0.86 of payment-ratio variance—making agent beliefs inspectable for market governance.
Where Pith is reading between the lines
- Inference: The market-vs-autarky comparison is confounded by settlement regime: autarky settles payments with deterministic ground-truth scores while the market uses subjective poster discretion, so a portion of the reported productivity advantage may stem from the evaluation regime rather than from exchange itself.
- Inference: A direct test would swap settlement mechanisms—run the market with oracle-based deterministic payment and autarky with poster-style discretionary payment; if the profit gap reverses or shrinks, the headline gains would be attribution to evaluation infrastructure, not trade.
- Inference: The paper's framing implies that verification infrastructure—who can cheaply check quality—is the binding constraint in agent markets, so investment in evaluation tools may matter more than tweaking auction or reputation rules.
- Inference: The monoculture ablation degrading premium-tier hires suggests population diversity is itself an institutional asset; a testable extension would add diversity-preserving mechanisms (such as quotas or diversity-aware selection) and test whether the long-run dispute regime stabilizes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Diagon, a programmable experimental testbed for agent-to-agent markets in delegated cognitive work. Twenty-five LLM-backed trader agents post tasks, bid, select, execute, evaluate, and settle payments under configurable institutional rules. The paper instantiates one market form (first-price sealed-bid auction, incomplete contracts with poster-controlled payment ratios, bilateral reputation, and replicator-style selection) and reports experiments comparing this market against an autarky baseline, along with six single-mechanism ablations (honest/adversarial/collaborative prompts, identity transparency, monoculture, fierce selection). The headline claims are that market exchange yields higher per-task profit and quality than self-sufficiency, that the market exhibits persistent disputes and partial-lemons dynamics, and that institutional interventions such as identity transparency and stronger selection degrade performance. Code and data are released.
Significance. If the empirical claims hold, Diagon would be a useful and timely contribution: it makes institutional rules experimentally manipulable in a full agent economy, ships with open code and data, and includes several careful validation pieces — execution-replay fidelity checks (Appendix F.4), dispute-cutoff sensitivity (Appendix F.1), prompt-paraphrase robustness (Appendix F.3), and long-horizon stability checks (Appendix F.6). The paper is also commendably explicit about its operating parameters and their rationale (Appendix A.1). However, the central causal claim that 'market exchange generates more productivity gains over self-sufficient agents' is not yet established, because the Market/Autarky comparison varies the settlement regime together with the market structure, and the statistical inference pools transactions across only three seeds in a way that likely overstates significance. These are fixable concerns, so the contribution is promising but needs a major revision.
major comments (3)
- [§4.4, §D.3, Table 5] The Market-vs-Autarky comparison is confounded. Autarky 'settlement uses deterministic ground-truth scores' (§4.4), while Market settlement is by subjective poster discretion over ρ∈[0.5,1] (§4.2 step 6 and §D.4). The headline 1.58× per-task profit advantage and the dispute-rate gap (39.1% vs 50.3%, Table 7) therefore bundle the effect of exchange with the effect of the settlement regime. Appendix D.3 asserts 'only the internal redistribution differs,' but the settlement rule is itself part of the redistribution and feeds back into effort, selection, and entry/exit decisions. The qR-based metrics in Table 5 are invariant to the realized ρ, not to the settlement regime. To support the abstract's causal language, run at least one control: e.g., a Market variant with oracle-based or formulaic payment (like Autarky's ground-truth settlement) while keeping bidding and selection, or an Autarky
- [§5 opening; Appendix F.2] The statistical inference pools 1,957 transactions across three baseline seeds and then computes bootstrap percentiles and p-values over the pooled transaction set. Transactions within a seed share agent decisions, reputation dynamics, and evolutionary selection, so they are not exchangeable. With only three independent seeds, the reported p-values (e.g., d=−1.66, p<0.001 for transparency; d=−0.74, p<0.001 for monoculture) are likely anti-conservative. Please report seed-level effect sizes and use a cluster-robust or seed-permutation test, or state clearly that the pooled test is descriptive. This is load-bearing for every ablation claim in §5.3.
- [§5.3 vs Appendix F.2] The main text states 'Telling agents to be “honest” increases disputes' and later 'the effect appears in four of five model families.' Appendix F.2 reports that the honest disposition reaches only borderline significance on the pooled dispute rate (Δ=+0.020, paired t(4)=3.37, p≈0.05) and is omitted from Figure 4, which plots only p<0.05 contrasts after multi-seed bootstrapping. The main-text claim is stronger than the appendix supports. Please either add the additional evidence needed to make the claim robust or temper the wording to match the reported significance.
minor comments (4)
- [Appendix A.1, Table 1] The surge-pricing amplification α=0.15 and the cooldown rate (−5%) are defined in Appendix B.5 but are absent from the operating-point parameter table in Appendix A.1. Since α is an operating parameter, it should be listed with its rationale.
- [§5.1, Table 5] The notation qR in Table 5 is not defined in the main text or appendix. Define it explicitly (e.g., quality-adjusted reward) at first use to avoid ambiguity with the payment ratio ρ.
- [§4.5] The claim that 'every Trader decision (bidding, evaluating, paying) is still a fresh agent call' should note that the cached draw is sampled uniformly at random and that the contractor's plan is preserved; this is clear in §D.2 but the main text could be more precise about what 'replay' does not replay.
- [Appendix F.2] The paired t-test is reported as t(4)=3.37 with three seeds. Please clarify the effective degrees of freedom; if the test is across five model-family means rather than seeds, this should be stated to avoid confusion about the level of replication.
Circularity Check
No meaningful circularity: the paper is an empirical simulation testbed with disclosed operating parameters; the only related concern is a settlement-regime confound, not a circular construction.
full rationale
Diagon is not a derivation-to-prediction paper; its results are observed outcomes of a configured simulation. The profit accounting (Eqs. 1, 4-5) and the Table 5 metrics are definitions, but the Market-vs-Autarky ranking is an empirical contrast, not a consequence of those definitions. Nothing is fitted to make Market win: μ and f are disclosed operating points, and Appendix F.5 reports the market collapses at μ=1, a sensitivity result rather than a hidden tuning. The closest thing to a circularity concern is that Autarky "settlement uses deterministic ground-truth scores" (§4.4) while Market uses poster discretion (ρ∈[0.5,1]), so the exchange treatment is bundled with a settlement treatment; but this is a confound/validity issue, not an equivalence-by-construction. The qR-based metrics are invariant to realized ρ for fixed q but not to the settlement regime. Self-citations appear only in Related Work (Liu et al. 2025, 2026a, 2026b) and are not load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The acknowledged limitation in Appendix H (long-run dispute regime left to future work) is a scope statement, not circularity.
Axiom & Free-Parameter Ledger
free parameters (7)
- mu (task amplifier) =
10
- f (reward-to-cost ratio) =
5.0
- rho_min (payment floor) =
0.50
- K (elimination period) =
6 rounds
- w0 (initial balance) =
$1
- rho_c (dispute cutoff) =
0.95
- surge alpha (dynamic pricing) =
0.15
axioms (3)
- domain assumption LLM agents' strategic decisions in the simulated market are a meaningful proxy for future AI-agent market behavior
- domain assumption Execution cache replay does not alter strategic dynamics
- domain assumption Pooled-transaction bootstrap is valid for significance testing
read the original abstract
AI agents are emerging as market participants that trade delegated cognitive work with one another on behalf of their users. Each agent can act as both a task poster and a contractor: posting tasks, bidding for work, completing tasks, evaluating outputs, and settling payments. As these markets take shape, their rules become costly to change once embedded in infrastructure and transaction histories. Existing market institutions evolved around human constraints. AI agents operate under different conditions: they interact rapidly, vary widely in capability and cost across configurations. This raises the question: how should markets for delegated cognitive work be designed for AI agents? We present diagon, a programmable system for controlled experiments on how market rules shape agent-to-agent trade in delegated cognitive work. Rules for allocation, contracting, and enforcement are configurable, while posting, bidding, selecting, executing, evaluating, and paying remain agent decisions. We use diagon to study trade under different agent configurations and market rules. Our results show that changes to agent configuration and market rules can reshape trade, quality, and wealth. By enabling these consequences to be identified and evaluated before deployment, diagon takes a step toward agent marketplaces that provide reliable work and accountable transactions for users and organizations. Code and data are available at https://github.com/assassin808/diagon.
Figures
Forward citations
Cited by 1 Pith paper
-
When Shippers Become Algorithms: Candidate Exposure, Information Design, and the Concentration of LLM-Mediated Freight Markets
LLM shipper agents in a capacity-constrained freight market concentrate demand on one carrier on day one, and disclosing remaining carrier capacity is the one platform intervention that reliably cuts concentration and...
Reference graph
Works this paper leans on
-
[1]
the agent market is not viable. Each contract carries fixed backbone overhead (Trader decisions on bidding, selection, evaluation, payment), so when contract size is comparable to that overhead the trade margin is exhausted by decision cost itself; agent markets become economical only when a single negotiated contract represents enough delivered work to a...
1987
-
[3]
Robert Axelrod.The Evolution of Cooperation
Accessed 2025-03-31. Robert Axelrod.The Evolution of Cooperation. Basic Books, New York,
2025
-
[5]
Mechanism design for large language models
Paul Duetting, Vahab Mirrokni, Renato Paes Leme, Haifeng Xu, and Song Zuo. Mechanism design for large language models. InProceedings of the ACM Web Conference 2024, pp. 144–155,
2024
-
[7]
Agam Goyal, Olivia Pal, Hari Sundaram, Eshwar Chandrasekharan, and Koustuv Saha
Accessed 2025-03-30. Agam Goyal, Olivia Pal, Hari Sundaram, Eshwar Chandrasekharan, and Koustuv Saha. Social simulacra in the wild: AI agent communities on Moltbook.arXiv preprint arXiv:2603.16128,
arXiv 2025
-
[10]
The effective reward becomes µ·R(τ j) and the effective execution cost becomes µ·c ex i , while backbone (thinking) cost remains unscaled. This ensures that execution cost dominates the agent’s budget— matching the economics of real outsourcing where the cost ofdoingthe work far exceeds the cost ofdecidingto do it (Williamson, 1985). Skill clusters.The se...
1985
-
[11]
doi: 10.1038/s41586-025-09505-x. Blake LeBaron. Agent-based computational finance. InHandbook of Computational Economics, volume 2, pp. 1187–1233. Elsevier,
-
[12]
Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670,
-
[13]
Strategic collusion of LLM agents: Market division in multi-commodity competitions
Ryan Y Lin, Siddhartha Ojha, Kevin Cai, and Maxwell F Chen. Strategic collusion of LLM agents: Market division in multi-commodity competitions. InNeurIPS 2024 Workshop on Language Gamification,
2024
-
[14]
Xianyang Liu, Shangding Gu, and Dawn Song. AgenticPay: A multi-agent LLM negotiation system for buyer–seller transactions.arXiv preprint arXiv:2602.06008, 2026a. Xuan Liu, Jie Zhang, HaoYang Shang, Song Guo, Chengxu Yang, and Quanyan Zhu. Ex- ploring prosocial irrationality for LLM agents: A social cognition view. InThe Thirteenth International Conference...
-
[15]
Xuan Liu, HaoYang Shang, and Haojian Jin
URL https://openreview.net/ forum?id=u8VOQVzduP. Xuan Liu, HaoYang Shang, and Haojian Jin. Cobra: Programming cognitive bias in social agents using classic social science experiments. InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI ’26, New York, NY, USA, 2026b. Association for Computing Machinery. ISBN 9798400722783. d...
arXiv 2026
-
[16]
Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T Hancock
Accessed 2025-03-30. Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T Hancock. Finding deceptive opinion spam by any stretch of the imagination. InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pp. 309–319,
2025
-
[17]
Kenneth Payne and Baptiste Alloui-Cros. Strategic intelligence in large language models: Evidence from evolutionary game theory.arXiv preprint arXiv:2507.02618,
-
[18]
doi: 10.1038/s42256-023-00646-0. Alvin E Roth. The economist as engineer: Game theory, experimentation, and computation as tools for design economics.Econometrica, 70(4):1341–1378,
-
[20]
Michelle Vaccaro, Michael Caosun, Harang Ju, Sinan Aral, and Jared R Curhan. Advancing AI negotiations: New theory and evidence from a large-scale autonomous negotiation competition.arXiv preprint arXiv:2503.06416,
-
[23]
Beyond numeric rewards: In-context dueling bandits with LLM agents
Fanzeng Xia, Hao Liu, Yisong Yue, and Tongxin Li. Beyond numeric rewards: In-context dueling bandits with LLM agents. InFindings of the Association for Computational Linguistics: ACL 2025,
2025
-
[24]
The automated but risky game: Modeling agent-to-agent negotiations and transactions in consumer markets
Shenzhe Zhu, Jiao Sun, Yi Nian, Tobin South, Alex Pentland, and Jiaxin Pei. The automated but risky game: Modeling agent-to-agent negotiations and transactions in consumer markets. InICML 2025 Workshop on Reliable and Responsible Foundation Models,
2025
-
[26]
and comparative advan- tage (Ricardo, 2005): an agent whose cluster si matches a task’s domain dj receives skill packages (documentation and helper scripts) that are injected into the Worker’s execution environment via prompt, improving execution quality. Skills therefore reside in thetask environment, not in the model itself; the Trader’s role is to sele...
2005
-
[27]
cex i can be smaller or larger than cj ref depending on tier choice; it is scaled by µ in the posted contract
cex i realised execution costactually incurred by agent ai as contractor: the token spend of the Worker tier the contractor chose (Haiku 4.5 / Sonnet 4.6 / Opus 4.6 at the selected reasoning effort), under the prompt augmented with any matching skill packages. cex i can be smaller or larger than cj ref depending on tier choice; it is scaled by µ in the po...
1985
-
[1978]
Language models as agent models
Jacob Andreas. Language models as agent models. InFindings of the Association for Computa- tional Linguistics: EMNLP 2022,
2022
-
[1979]
John J Horton. Large language models as simulated economic agents: What can we learn from homo silicus?arXiv preprint arXiv:2301.07543,
-
[1984]
Concretely, every K= 6 rounds, all agents are ranked by wealth
—repeated play, payoff accumulation, periodic update—to a multi-agent market with het- erogeneous costs, asymmetric poster/contractor roles, and bilateral reputation (Nowak & Sigmund, 1998). Concretely, every K= 6 rounds, all agents are ranked by wealth. The bottom E= 1 agent isdeactivated: it ceases to participate in trading but its historical record—wea...
1998
-
[1985]
Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Aky¨urek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 1...
2024
-
[1992]
An economy of AI agents.arXiv preprint arXiv:2509.01063,
Gillian K Hadfield and Andrew Koh. An economy of AI agents.arXiv preprint arXiv:2509.01063,
-
[1997]
Isadora White, Kolby Nottingham, Ayush Maniar, Max Robinson, Hansen Lillemark, Mehul Maheshwari, Lianhui Qin, and Prithviraj Ammanabrolu. Collaborating action by action: A multi-agent LLM framework for embodied reasoning.arXiv preprint arXiv:2504.17950,
-
[1998]
where trust must be earned, not inherited. This mechanism yields 4% active-population turnover per cycle (1 /25), applied every 6 rounds—a high-frequency, low-amplitude approximation to the continuous replicator dynamic (Weibull, 1997), chosen to minimise per-event market disruption while maintaining meaningful selection pressure. Over a 100-round experim...
1997
-
[2002]
Virtual agent economies.arXiv preprint arXiv:2509.10147,
12 Nenad Tomasev, Matija Franklin, Joel Z Leibo, Julian Jacobs, William A Cunningham, Iason Gabriel, and Simon Osindero. Virtual agent economies.arXiv preprint arXiv:2509.10147,
-
[2006]
A cooldown of −5% per successful match prevents persistent inflation
that clears the market: unattrac- tive tasks become progressively more lucrative until some agent accepts. A cooldown of −5% per successful match prevents persistent inflation. Surge tasks are offered before fresh tasks each round (drain-first policy), ensuring that no contract is permanently stranded. C Implementation This section describes howDiagonruns...
1979
-
[2020]
doi: 10.1257/aer.20190623. Alan Chan, Kevin Wei, Sihao Huang, Nitarshan Rajkumar, Elija Perrier, Seth Lazar, Gillian K Hadfield, and Markus Anderljung. Infrastructure for AI agents.Transactions on Machine Learning Research,
-
[2023]
Sayash Kapoor, Noam Kolt, and Seth Lazar
doi: 10.3982/ECTA19978. Sayash Kapoor, Noam Kolt, and Seth Lazar. Position: Build agent advocates, not platform agents. InInternational Conference on Machine Learning,
-
[2024]
Modular pluralism: Pluralistic alignment via multi-LLM collaboration
Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. Modular pluralism: Pluralistic alignment via multi-LLM collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4151–4171,
2024
-
[2025]
doi: 10.1038/s41562-025-02172-y. George A Akerlof. The market for “lemons”: Quality uncertainty and the market mechanism. InUncertainty in economics, pp. 235–251. Elsevier,
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.