REVIEW 3 major objections 4 minor 3 cited by
Learning from Synthetic Labs: Language Models as Auction Participants
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read LLM bidders reproduce classic human auction behaviors — risk aversion, clock-auction clarity, and the winner's curse — in a synthetic lab.
desk verdict Useful proof-of-concept with honest reporting, but the proxy-validity claim is overstated and the memorization concern is untested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the synthetic-lab protocol: a plan-bid-reflect loop in which each GPT-4 bidder independently writes a bidding plan, places a bid given its private value, observes the outcome and all bids, then reflects and revises its plan before the next round. This loop, powered by chain-of-thought prompting (asking the model to reason step-by-step before answering), is what generates the behavior under study. Its low per-round cost lets the authors run more than a thousand auctions and compare aggregate bid patterns against theoretical equilibria and published human data.
What would settle it
Run the same protocol on a novel auction format with no published human benchmark, and record whether the model's chain-of-thought references specific known studies when it produces human-typical bids; absence of such references on novel formats, together with preserved behavioral patterns, would support emergent reasoning, while patterns that only appear for formats covered in the training literature would point to recitation.
Extended reading notes
Core claim
The central claim is that LLM agents, when endowed with chain-of-thought planning and reflection, reproduce the main behavioral regularities of human auction experiments across multiple classic formats. In independent private-value auctions, their first-price bids sit above the risk-neutral Bayes-Nash equilibrium, consistent with the risk-averse overbidding documented in human subjects, while their second-price bids skew below value, an asymmetry not found in the human data. In affiliated private-value settings, LLM bidders drop out near their value in ascending clock auctions far more often than they bid their value in the strategically equivalent second-price sealed-bid auction, mirroring the obvious strategy-proofness effect. In common-value auctions, they bid as if their private signal were an unbiased estimate of the common value and suffer a winner's curse that deepens as the number of bidders grows. The paper also shows that naive prompt changes, such as language and currency, matter little, while prompting with the language of Nash deviations substantially improves play, and that an extended closing rule eliminates last-second sniping in an eBay-style environment.
Load-bearing premise
The paper assumes that GPT-4's bids come from reasoning about the described auction context rather than from memorizing the results of the auction-experiment literature during training.
Editorial extensions
If this is right
- If LLM proxies are valid, auction researchers can screen designs and interventions for about $10 per experiment instead of $15,000, making large parameter sweeps feasible.
- The similarity with human risk aversion and the winner's curse means LLM agents can generate synthetic data on behavioral deviations, not just rational benchmarks.
- The finding that clock formats improve truth-telling suggests that obvious strategy-proofness design principles carry over to LLM agents, and that prompting interventions can be tested in silico before human trials.
- The eBay closing-rule result offers a low-cost path to revisit classic marketplace design debates, such as the competing ending rules that distinguished Amazon and eBay auctions.
- Naive prompt robustness, across languages and currencies, increases confidence that the observed patterns are not artifacts of a specific wording choice.
Reading between the lines
- A direct test of the emergent-reasoning interpretation would be to run the same protocol on a novel auction format with no published human benchmark; if the model still displays human-typical biases there, the proxy claim is strengthened.
- The framework's flexibility suggests the same plan-bid-reflect loop could be applied to other mechanism-design settings, such as matching markets or contests, to generate behavioral data before expensive human validation.
- The contrast between LLM underbidding and human overbidding in second-price sealed-bid auctions hints at a model-specific bias that prompt engineering might adjust, potentially letting LLM proxies match not just the magnitude but the direction of human error.
- Because the paper runs multi-round auctions, the observed overbidding may be partly an artifact of the agent's explicit instruction to explore and gather data, rather than a stable preference; single-round ablation results in the appendix suggest this is the case.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes using large language model (LLM) agents as low-cost proxies for human bidders in auction experiments. The authors run a large number of simulated auctions with GPT-4 agents under chain-of-thought reasoning, covering first-, second-, third-price, all-pay, ascending clock, common-value, and eBay-style auctions. They report that LLM bidders overbid relative to the risk-neutral benchmark in first-price auctions, bid closer to truth in clock formats than in sealed-bid formats, suffer the winner's curse in common-value settings, and exhibit strategic behaviors such as sniping in eBay-style auctions. They also study how different prompt interventions affect bidding accuracy. The central claim is that LLM agents reproduce key empirical regularities from the human experimental auctions literature, thereby serving as a valid and cheap synthetic lab.
Significance. If the proxy-validity claim were established, this would be a useful methodological contribution: it would let researchers explore auction designs at a fraction of the cost of human experiments, and the open-source framework would enable broader use. The paper's strengths are its breadth of auction formats, the explicit planning-and-reflection protocol, the cost transparency, and the inclusion of robustness checks on prompts, languages, and currencies. However, the abstract and introduction overstate the agreement with the human literature, and the paper does not address the most important threat to its central claim: that GPT-4's training data includes the very experimental results used as benchmarks, so the observed agreement could reflect memorization rather than emergent strategic reasoning. This unresolved issue, together with an internal contradiction in the reported language-robustness results, means the paper currently supports a proof-of-concept but not yet the stronger proxy-validity claim.
major comments (3)
- [Abstract; Section 3.1.5, Table 2] The abstract and introduction state that LLM bidders 'agree with the experimental literature in auctions' and specifically 'produce results consistent with risk-averse human bidders.' However, Section 3.1.5 and Table 2 show that in the SPSB auction, LLMs underbid their value 65.78% of the time, while the paper itself notes that Kagel and Levin (1993) report humans overbid 67% of the time. The direction of the dominant deviation is therefore opposite to the human data, not merely different in magnitude. This is a direct contradiction of the unqualified 'agree' claim, and it is load-bearing because the paper uses this agreement as evidence that LLMs can serve as proxies. The authors should either substantially qualify the central claim in the abstract and introduction or reframe the SPSB result as a systematic difference between LLM and human behavior that itself requires explanation before proxy validity can be assumed.
- [Section 2.1; Section 3.1.5; Appendix D] The central proxy-validity claim rests on the premise that GPT-4's bidding behavior reflects emergent, context-driven strategic reasoning rather than memorized reproduction of the auction-theory and experimental-literature content in its training data. The paper never tests this premise. The protocol (Section 2.1) intentionally avoids persona anchoring, but it still uses the exact benchmark settings and vocabulary (the same value ranges, the same APV and common-value structures, and descriptions that echo the experimental papers). Appendix D show that removing the plan-bid-reflect loop produces qualitatively different behavior (e.g., no overbidding in SPSB), which is consistent with the alternative explanation that the observed regularities are sensitive to priming and surface cues rather than reflecting a stable reasoning disposition. Because the intended use of the proxy is to learn about new auction designs, reproducing memorized results would not validate that use. The paper should include a control that distinguishes these hypotheses, for example: (i) auctions with novel formats or payment rules that are unlikely to appear verbatim in training data, (ii) paraphrase or distractor tests that remove auction-theory vocabulary while preserving the incentive structure, or (iii) an analysis of whether the LLM's stated reasoning in chain-of-thought actually derives the equilibrium logic rather than citing known results. Without such a control, the agreement with human experiments remains ambiguous evidence for proxy validity.
- [Section C.3, Figure 11; Abstract] The abstract claims that LLMs are 'not very sensitive to naive changes in prompts (e.g., language, currency),' but the robustness check in Appendix C.3 reports that the Russian-language FPSB prompt produces a stark outlier: agents bid close to their true value rather than shading bids as in the English version. This is a qualitative change in the key treatment, not a small quantitative difference. The paper should either remove the overgeneralized claim, or report this outlier prominently in the main text and explain why it does not undermine the conclusion that behavior is robust to prompt variation. The current presentation gives the reader an inaccurate impression of the evidence.
minor comments (4)
- [Section 3.1.4] The sentence 'In particular, 90bids exceeded their value in this auction' appears to be missing a percentage sign or a word (likely '90% of bids'); please correct this typo.
- [Section 4.1; Section 4.3] The claim that LLMs replicate sniping 'with no prompting' is somewhat overstated, because the prompt explicitly tells bidders that they may place bids on the final day and that no bidder knows if they are the last; this is a design cue that likely encourages late bidding. Please phrase this as 'with a standard rules description that mentions final-day bidding' rather than 'with no prompting.'
- [Section 5.1, Table 7] The text says the risk-neutrality intervention 'has about the same performance as the proxy intervention,' but Table 7 reports R^2 = 0.4004 for the risk intervention, which is lower than the no-intervention R^2 = 0.4845 and much lower than the proxy intervention's 0.2966 (or the Nash intervention's 0.8365). The qualitative description should be tied more carefully to the reported fit statistics, or the table should be annotated to clarify that the comparison is limited to the overbidding component.
- [Section 3.1.5 and Section 5 use multiple statistical tests] Several tables report pairwise t-tests or chi-square tests without any correction for multiple comparisons or clustering of bids within the 5 experiments per condition. Because the LLM agents share a common history across rounds, the effective sample size is smaller than the number of bids. Please either report cluster-robust standard errors or explicitly acknowledge this limitation in the statistical analysis.
Circularity Check
No definitional or fitted-input circularity; core benchmarks are external. Minor self-citations are not load-bearing.
full rationale
The paper's central claims are empirical comparisons of LLM bidding behavior against external experimental and theoretical benchmarks (Kagel and Levin 1986/1993, Li 2017, Roth and Ockenfels 2002, Cox et al. 1988). The derivation chain does not fit parameters to those benchmarks and then call the fit a prediction: the LLM agents are given auction rules and values, and their bids are generated without reference to the target experimental outcomes. There is no self-definitional step in which a quantity is defined in terms of the quantity it is claimed to predict. The protocol does instruct agents to maximize long-run profit and to use chain-of-thought reasoning, but this is a standard experimental instruction, not a fitted parameter tied to the benchmark results. Sections 3.1.5, 3.2.4, and 3.3.4 report outcomes that agree with, but are not derived from, the cited human experiments. The paper also explicitly acknowledges remaining open questions about how much prompting is needed to match empirical evidence, which is consistent with a non-circular, exploratory design. There are self-citations (e.g., Manning et al. 2024, Horton 2023, and the EDSL package), but these concern the simulation framework and social-science methodology, not the load-bearing auction benchmarks; they do not substitute for the external evidence. The concern that GPT-4 may have memorized the auction literature in its training data is a real external-validity or soundness risk, but it is not an internal circularity of the derivation chain: the paper does not define its conclusions in terms of its inputs or fit its inputs to its outputs. Accordingly, no specific circular step can be exhibited, and the appropriate circularity score is low rather than elevated.
Assumptions & free parameters
free parameters (1)
- GPT-4 sampling temperature =
0.5
assumptions (4)
- standard math Standard Bayes-Nash equilibrium formulas for IPV auctions with uniform values and risk neutrality (Table 1).
- domain assumption Discrete bids on a $1 grid approximate continuous-theory equilibria.
- domain assumption The plan-bid-reflect protocol and universal profit-maximization instruction elicit genuine strategic behavior from the LLM rather than text reproduction of known auction results.
- domain assumption Human experimental results (Kagel-Levin, Cox et al., Li 2017, Roth-Ockenfels) are accurate and transferable benchmarks.
Cite this review
Pith. "Pith review of Learning from Synthetic Labs: Language Models as Auction Participants." pith.science (2026). https://pith.science/paper/CRHB6QQH
@misc{pith2026250709083,
author = {Pith},
title = {Pith review of: Learning from Synthetic Labs: Language Models as Auction Participants},
year = {2026},
howpublished = {\url{https://pith.science/paper/CRHB6QQH}},
note = {Machine review of arXiv:2507.09083}
}
abstract
This paper investigates the behavior of simulated AI agents (large language models, or LLMs) in auctions, introducing a novel synthetic data-generating process to help facilitate the study and design of auctions. We find that LLMs -- when endowed with chain of thought reasoning capacity -- agree with the experimental literature in auctions across a variety of classic auction formats. In particular, we find that LLM bidders produce results consistent with risk-averse human bidders; that they perform closer to theoretical predictions in obviously strategy-proof auctions; and, that they succumb to the winner's curse in common value settings. On prompting, we find that LLMs are not very sensitive to naive changes in prompts (e.g., language, currency) but can improve dramatically towards theoretical predictions with the right mental model (i.e., the language of Nash deviations). We run 1,000$+$ auctions for less than $\$$400 with GPT-4 models (three orders of magnitude cheaper than modern auction experiments) and develop a framework flexible enough to run auction experiments with any LLM model and a wide range of auction design specifications, facilitating further experimental study by decreasing costs and serving as a proof-of-concept for the use of LLM proxies.
Figures
Figures from the paper (10 more)
Forward citations
Cited by 3 Pith papers
-
Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce
LLM merchant agents in a new dynamic auction benchmark capture at most 32% of hindsight-optimal profit; profit tracks margin per win more than win rate, and fast pre-shock learners adapt poorly to preference shocks.
-
Choose Your Agent: Tradeoffs in Adopting AI Advisors, Coaches, and Delegates in Multi-Party Negotiation
Users prefer an AI Advisor but gain most with a Delegate, because human editing filters out the AI's best proposals.
-
When Is Delegated Play Truthful? Within-Range Regret and the Trilemma of Aligned Delegation
The gain from misreporting to your own proxy equals the proxy's within-range regret, so honest reporting is optimal exactly when the proxy already plays the best reachable action; guardrails then face a binding–truthf...
Reference graph
Works this paper leans on
-
[2]
Your value for the prize is equal to the common value plus your private taste adjustment
Then, for each bidder, a private taste adjustment will be drawn between 0 and {{private}}, with all values equally likely. Your value for the prize is equal to the common value plus your private taste adjustment. You will not learn the common value or your private taste adjustment separately. This means that each person in your group may have a different ...
-
[5]
Your value for the prize is equal to the common value plus your private taste adjustment
Then, for each bidder, a private taste adjustment will be drawn between 0 and {{private}}, with all values equally likely. Your value for the prize is equal to the common value plus your private taste adjustment. You will not learn the common value or your private taste adjustment separately. This means that each person in your group may have a different ...
-
[6]
First we will randomly draw a common value between {{ common_low}} and {{common_high}}, with all values equally likely
-
[7]
Your value for the prize is equal to the common value plus your private taste adjustment
Then, for each bidder, a private taste adjustment will be drawn between 0 and {{private}}, with all values equally likely. Your value for the prize is equal to the common value plus your private taste adjustment. You will not learn the common value or your private taste adjustment separately. This means that each person in your group may have a different ...
-
[8]
For each round, a common value will be drawn between {{ common_low}} and {{common_high}}, with all values equally likely to be drawn
-
[9]
For each person, a private noisy adjustment will be drawn between -{{private}} and {{private}}, with all values equally likely to be drawn. We will tell you your perceived value, the sum of the common value and the private noise adjustment. However, everyone’s 49 true value for the prize is equal to the shared common value . After learning your perceived ...
-
[2025]
kelly is a warm person, joseph is a role model
Automated mechanism design: A survey. ACM SIGecom Exchanges 22, 2 (2025), 102–120. Constantinos Daskalakis, Ian Gemp, Yanchen Jiang, Renato Paes Leme, Christos Papadimitriou, and Geor- gios Piliouras. 2024. Charting the Shapes of Stories with Game Theory.arXiv preprint arXiv:2412.05747 (2024). Emmanuel Dechenaux, Dan Kovenock, and Roman M Sheremeta. 2015....
arXiv 2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.