Pith. sign in

REVIEW 3 major objections 6 minor 105 references

When Experimental Economics Meets Large Language Models: Evidence-based Tactics

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Experimental design choices—persona, dialogue format, and answer format—shift what large language models reveal about preferences and rationality, and the effects depend on model scale.

desk verdict Useful and transparent empirical mapping of LLM experiment design choices, but the headline 'personas affect preferences not rationality' overreaches because it compares different tasks. read the letter →

arxiv 2505.21371 v2 pith:ROOPSLZW submitted 2025-05-27 econ.GN q-fin.EC

Shu Wang , Zijun Yao , Shuhuai Zhang , Jianuo Gai , Tracy Xiao Liu , Songfa Zhong This is my paper · ORCID
classification econ.GNq-fin.EC
keywords largelanguagemodelsexperimentaleconomicseconomicrationalityCCEIpersonaeffectsdialogueformatanswerreplicability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the results of economic experiments run on large language models are not fixed properties of the models; they depend on design decisions such as whether a persona is assigned, whether questions come one at a time or all at once, and whether answers are open-ended or multiple-choice. Through two case studies—budgetary tasks that measure economic rationality and behavioral games that measure preferences—it finds that assigned personas change preferences but not rationality, while single-turn dialogue and forced-choice answers lower the rationality of smaller open-source models (Llama and Qwen) but not of larger models (GPT and DeepSeek). The authors distill these findings into seven practical tactics for designing LLM experiments. The point is that LLM experiments need standardized protocols, just as human experiments do.

What carries the argument

The measurement engine is Afriat's critical cost efficiency index (CCEI), a 0-to-1 score from revealed-preference analysis that measures consistency with utility maximization, applied to 25-round budgetary allocation tasks in risk and social preference domains. Preferences are measured through five decision scenarios drawn from four canonical games: dictator, ultimatum proposer, ultimatum responder, public goods, and bomb risk. To aggregate the influence of design variations, the paper constructs a sensitivity score $\lambda(S)=\frac{1}{|I||M||S|}\sum_{i,m,s}\mathbf{1}\{p_{ims}<0.05\}$, the fraction of model-by-domain-by-condition comparisons that reach significance at $p<0.05$. The baseline condition mirrors human-experiment protocols—default temperature, no assigned persona, a hypothetical payment rule, an illustrative example, comprehension questions, multi-turn dialogue, and open-ended JSON-formatted answers—and each experimental condition changes exactly one element at a time.

What would settle it

Run the same single-turn versus multi-turn and open-ended versus multiple-choice comparisons on a small model trained for sequential reasoning, or on matched models from one family with different parameter counts; if the rationality decline disappears or appears in a large model, the parameter-size attribution is wrong.

Watch

Extended reading notes

Core claim

The central empirical claim is that protocol choices in LLM experiments produce systematic, model-dependent shifts in measured economic behavior. In budgetary tasks scored by the critical cost efficiency index (CCEI), assigned personas—gender, age, education, ethnicity, or occupation—do not significantly change rationality for any of the four models, yielding a sensitivity score of 0%. In behavioral games, the same personas change decisions in 35.7% of comparisons, with occupational personas more influential than demographic ones. Single-turn dialogue, which packs all 25 decisions into one prompt, significantly reduces the CCEI of Llama and Qwen in both risk and social preference domains, while GPT and DeepSeek remain essentially unchanged. Switching from open-ended to multiple-choice answers similarly reduces rationality for Llama and Qwen and changes outputs in 55% of the behavioral-game comparisons. The paper interprets these patterns as evidence that experimental design is a first-order determinant of what LLM experiments reveal, and it proposes seven tactics—such as using default temperature, avoiding personas, using multi-turn dialogue, and preferring open-ended answers—to make LLM experiments more comparable and replicable.

Load-bearing premise

The load-bearing premise is that the two model pairs differ only in scale for these tasks—that parameter count, rather than training data, alignment, or API behavior, explains why Llama and Qwen lose rationality under single-turn and multiple-choice formats.

Editorial extensions

If this is right

  • If design choices shift LLM outcomes, published LLM-experiment results should be read as conditional on prompt protocol rather than as stable model traits.
  • Multi-turn, open-ended, persona-free prompts with examples and comprehension checks become the recommended default for comparability with human experiments.
  • Small and large models should not be pooled in LLM experiments, because the same protocol can measure rationality for one class and break it for another.
  • Reporting invalid-answer rates as a standard metric would improve the replicability and interpretability of LLM experiments.
  • A Turing test that an LLM 'passes' under one protocol may fail under another, so behavioral-similarity claims need to specify the protocol under which they hold.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the parameter-size attribution is right, the newest reasoning-oriented small models may behave more like the large models in this study, which would make the 'small model' part of the tactics time-sensitive.
  • The persona results suggest that LLM 'demographic' responses may reflect corpus stereotypes rather than genuine identity effects; a direct extension would compare persona-induced preference shifts against a human sample matched on the same identities.
  • The finding that Qwen's rationality declines as stake sizes increase is a testable lead: if larger hypothetical stakes degrade consistency for small open-source models, similar incentive-magnitude effects should appear in other small models.
  • The sensitivity score $\lambda$ could become a standard protocol-reporting metric, analogous to robustness checks, letting readers see at a glance how much of an LLM experiment's results depend on design choices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper develops evidence-based recommendations for running economic experiments with large language models. The authors review 12 published LLM experiments, extract seven design and implementation variables, and test several of them in two case studies: Case Study 1 uses 25-round budget tasks (risk and social preference) to measure economic rationality via CCEI, and Case Study 2 uses one-shot decisions in dictator, ultimatum, public goods, and bomb risk games to measure economic preferences. Four LLMs (GPT-4o, DeepSeek-V3, Llama-3.1-8B, Qwen2.5-7B) are run with 100 simulations per condition, with variations in persona, dialogue type, answer type, temperature, incentives, and examples. The paper reports that assigned personas affect preferences but not rationality, that single-turn dialogue and multiple-choice answer formats reduce rationality for Llama and Qwen but not for GPT and DeepSeek, and that the answer type shifts decisions in roughly half of the game scenarios; these findings motivate seven practical tactics.

Significance. The paper's main value is as a methodological check for a fast-growing literature: it provides transparent prompt templates, FDR-corrected comparisons, a power analysis, invalid-response rates, and tests four models rather than a single API. If the results survive reanalysis, the seven tactics would be useful and actionable for researchers designing LLM experiments. However, the headline dissociation between preferences and rationality is currently a cross-experiment comparison rather than a within-task result, and the model-size interpretation is confounded with provider and training differences; these claims need to be tightened. The absence of a data/code repository is also notable given the paper's explicit replicability motivation. The experimental work itself is careful in execution, with 100 simulations per condition and transparent reporting of prompts.

major comments (3)
  1. [Sections 3.2.2, 4.2.2, and 6] The central claim that assigned personas 'significantly affect preferences but not rationality' is not established by the reported analyses. In Case Study 1, persona variations are evaluated only through the CCEI; no preference parameters (e.g., risk aversion or social-preference weights) are estimated from the same 25-round allocation data. In Case Study 2, persona variations are evaluated only through one-shot game decisions, with no rationality measure. The direct comparison of λ(S)=0 (Case Study 1) with λ(S)=35.7% (Case Study 2) therefore confounds the rationality/preference distinction with differences in task type, repetition, and incentive structure. Please either estimate preference parameters from the Case Study 1 allocation data and test persona effects on them, or restrict the abstract and Section 6 to the task-specific findings, e.g., 'personas did not affect CCEI in the budget tasks and did affect decisions in the behavioral games.'
  2. [Section 3.2.2 and Section 6] The interpretation that the dialogue-type and answer-type effects 'may arise from the limitation of single-turn dialogue in supporting sequential reasoning processes' and that the problem 'persists for smaller models such as Llama and Qwen' is not identified. The design compares two small open-source models (Llama-3.1-8B, Qwen2.5-7B) with two larger models (GPT-4o, DeepSeek-V3) that also differ in provider, training data, alignment, and API behavior; parameter size is therefore confounded with these factors. To support the size attribution, the authors would need a within-family or within-scale comparison (e.g., Llama-3.1-8B versus Llama-3.1-70B) or should soften the claim to 'the two smaller models we studied.' This matters because Tactic 5 generalizes the finding to 'smaller parameter sizes.'
  3. [Sections 3.1.3 and 4.1.2, and Section 5.2] The multiple-choice manipulation changes two things at once: the answer format and the feasible set, which is discretized into 21 options. The observed reductions in CCEI for Llama and Qwen, and the shifts in game decisions, could be driven by the coarsened choice set rather than by the open-ended-versus-multiple-choice format per se. The paper itself acknowledges in Section 5.2 that 'discretizing continuous feasible sets has been shown to significantly decrease the performance of models,' which supports this concern. Please add a condition that varies the response format while keeping the feasible set constant (e.g., open-ended answers with the same grid listed), or explicitly rephrase the findings and Tactic 6 as effects of discretized choice sets rather than of answer type.
minor comments (6)
  1. [Section 3.1.2 and Appendix D.1.1] The text states that the baseline condition 'evaluates the degree of comprehension using understanding questions,' but the reproduced prompts contain only an illustrative example and an instruction to answer, not a separate comprehension question; please clarify or provide the actual questions.
  2. [Supplementary Table 2] Several cells are blank or contain '−' without a consistent legend explaining whether a blank means 'not applicable' or 'not reported' (e.g., the Temperature row for Suri et al. and the Persona row for Webb et al.); please make the table self-contained.
  3. [Figures 2 and 4] The figures would be more informative if statistically significant differences were marked (e.g., with asterisks) or if the corresponding p-values were available in a table, since the text repeatedly references these comparisons.
  4. [Section 4.2.2] The proportion tests comparing λ(S) values (e.g., 55.0% vs. 35.7%, p=0.084) treat the model-condition-scenario tests within each family as independent observations; given the FDR corrections and the shared data across tests, these proportion-test p-values should be interpreted as descriptive rather than confirmatory.
  5. [General] The manuscript does not provide a data or code repository. Given that the paper's main contribution is replicability of LLM experiments, making the simulation code and raw outputs available would substantially strengthen the contribution.
  6. [Section 5.1, Tactic 1] The recommendation to 'fix the random seed' is not tested in the experiments; the paper only reports variations in temperature. Please mark this as an auxiliary suggestion rather than an evidence-based tactic, or report whether seeds were fixed in the present data.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: the paper's claims are empirical comparisons, and its self-citations are not load-bearing.

full rationale

The paper's central claims are empirical comparisons of LLM outputs across experimental conditions (persona, dialogue type, answer type) in two case studies using standard tasks (budgetary decisions and behavioral games). No parameter is fitted to data and then renamed as a prediction; the λ(S) sensitivity scores are simply the fraction of statistically significant t-tests and are not constructed so as to force any particular value. The main self-citations are to Chen et al. (2023), which supplies the budgetary task, the human reference CCEI distribution, and a sample-size standard deviation; these are published, externally falsifiable data and are not used to derive the paper's findings. The 'personas affect preferences but not rationality' claim rests on a comparison of the Case Study 1 CCEI results with the Case Study 2 game-decision results; although this is a between-task inference that could be criticized on confound grounds, it is not circular because 'rationality' and 'preferences' are not defined in terms of the outcomes that are later asserted to be (un)affected. The tactics in Section 5 are recommendations grounded in the experiments and in standard experimental-economics principles, not renamings of a predetermined result. Accordingly, no load-bearing step reduces to its own inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper does not fit any parameters; its claims rest on the validity of established economic measurements (CCEI/GARP), the transferability of human experiment principles to LLMs, the representativeness of the four tested models, and the comparability of the human benchmark datasets.

assumptions (5)
  • domain assumption GARP and CCEI index validly measure economic rationality.
    Used throughout Case Study 1 (Section 3.2) to score rationality; assumes GARP violations reflect irrationality.
  • domain assumption The budget tasks from Chen et al. (2023) and the games from Mei et al. (2024) are externally valid measures of preferences and rationality.
    The paper adopts these tasks wholesale (Sections 3.1.1 and 4.1.1) and interprets results as evidence about LLM rationality and preferences.
  • domain assumption The human data from the prior papers are comparable benchmarks.
    Comparisons in Figures 1 and 3 treat the human samples from [22] and [67] as appropriate reference distributions.
  • domain assumption The four selected LLMs are representative of larger and smaller model classes.
    Section 3.1 selects GPT-4o, DeepSeek-V3, Llama-3.1-8B, and Qwen2.5-7B and interprets size-based differences (Section 3.2.2).
  • domain assumption Simulations are independent draws from the model's conditional distribution.
    The t-tests across 100 simulations assume independent observations, though no seed control is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Experimental Economics Meets Large Language Models: Evidence-based Tactics." pith.science (2026). https://pith.science/paper/ROOPSLZW

@misc{pith2026250521371,
  author       = {Pith},
  title        = {Pith review of: When Experimental Economics Meets Large Language Models: Evidence-based Tactics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ROOPSLZW}},
  note         = {Machine review of arXiv:2505.21371}
}
read the original abstract

Advancements in large language models (LLMs) have sparked a growing interest in measuring and understanding their behavior through experimental economics. However, there is still a lack of established guidelines for designing economic experiments for LLMs. Inspired by principles from experimental economics with insights from LLM research in artificial intelligence, we outline key considerations in the experimental design and implementation stage, and perform two sets of experiments to assess the impact of these considerations on LLMs' responses. Based on our findings, we discuss seven practical tactics for conducting experiments with LLMs. Our study enhances the design, replicability, and generalizability of LLM experiments, and broadens the scope of experimental economics in the digital age.

Figures

Figures reproduced from arXiv: 2505.21371 by the authors.

Figure 1
Figure 1. Baseline results of CCEI scores. Human data are from Chen et al. [ [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Comparison in rationality of LLMs. The markers represent differences in average CCEI [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Baseline results of decisions. Human data are from Mei et al. [ [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison in decisions of LLMs. The markers represent differences in the mean value [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

105 extracted references · 80 canonical work pages

  1. [1]

    Efficiency estimation of production functions

    Afriat, Sydney N. 1972. “Efficiency estimation of production functions.”International Eco- nomic Review, 13(3): 568–598

    1972

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Alayrac, Jean-Baptiste, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, et al. 2022. “Flamingo: a visual language model for few-shot learning.” InProceedings of the 36th Inter- national Conference on Neural Information Processing Systems

    2022

  3. [3]

    Giving according to GARP: an experimental test of the consistency of preferences for altruism

    Andreoni, James, and John Miller.2002. “Giving according to GARP: an experimental test of the consistency of preferences for altruism.” Econometrica, 70(2): 737–753

    2002

  4. [4]

    Controlling the false discovery rate: a prac- tical and powerful approach to multiple testing

    Benjamini, Yoav, and Yosef Hochberg.1995. “Controlling the false discovery rate: a prac- tical and powerful approach to multiple testing.” Journal of the Royal Statistical Society: Series B (Methodological), 57(1): 289–300

    1995

  5. [5]

    The economics of large language models: token allocation, fine-tuning, and optimal pricing

    Bergemann, Dirk, Alessandro Bonatti, and Alex Smolin. 2025. “The economics of large language models: token allocation, fine-tuning, and optimal pricing.” Working Paper

    2025

  6. [6]

    Using cognitive psychology to understand GPT-3

    Binz, Marcel, and Eric Schulz. 2023. “Using cognitive psychology to understand GPT-3.” Proceedings of the National Academy of Sciences, 120(6): e2218523120

    2023

  7. [7]

    Interpreting neural networks through the polytope lens

    Black, Sid, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ram´on Guevara, Beren Millidge, Gabriel Alfour, et al.2022. “Interpreting neural networks through the polytope lens.” Working Paper

    2022

  8. [8]

    Bommasani, Rishi, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al

Show all 105 references
  1. [9]

    Promoting helping behavior with framing in dictator games

    Bra˜nas-Garza, Pablo. 2007. “Promoting helping behavior with framing in dictator games.” Journal of Economic Psychology, 28(4): 477–486

    2007

  2. [10]

    Paid and hypothetical time preferences are the same: lab, field and online evidence

    Bra˜nas-Garza, Pablo, Diego Jorrat, Antonio M Esp ´ın, and Angel S ´anchez. 2023. “Paid and hypothetical time preferences are the same: lab, field and online evidence.”Experimental Economics, 26(2): 412–434

    2023

  3. [11]

    The power of nonparametric tests of preference maximization

    Bronars, Stephen G. 1987. “The power of nonparametric tests of preference maximization.” Econometrica, 55(3): 693–698

    1987

  4. [12]

    Playing games with GPT: what can we learn about a large language model from canonical strategic games?

    Brookins, Philip, and Jason DeBacker. 2024. “Playing games with GPT: what can we learn about a large language model from canonical strategic games?” Economics Bulletin, 44(1): 25–37

    2024

  5. [13]

    Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

  6. [14]

    Evaluating replicability of laboratory experiments in economics

    Camerer, Colin F, Anna Dreber, Eskil Forsell, Teck-Hua Ho, J ¨urgen Huber, Mag- nus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan, et al. 2016. “Evaluating replicability of laboratory experiments in economics.” Science, 351(6280): 1433–1436

    2016

  7. [15]

    Camerer, Colin F, Anna Dreber, Felix Holzmeister, Teck-Hua Ho, J¨urgen Huber, Mag- nus Johannesson, Michael Kirchler, Gideon Nave, Brian A Nosek, Thomas Pfeiffer, et al

  8. [16]

    12 best practices for leveraging generative AI in experimental research

    Chang, Samuel, Andrew Kennedy, Aaron Leonard, and John A List. 2024. “12 best practices for leveraging generative AI in experimental research.”Working Paper

    2024

  9. [17]

    The next generation of experi- mental research with LLMs

    Charness, Gary, Brian Jabarian, and John A List. 2025. “The next generation of experi- mental research with LLMs.” Nature Human Behaviour, 9(5): 833–835

    2025

  10. [18]

    RLHF deciphered: a critical analysis of reinforcement learning from human feedback for LLMs

    Chaudhari, Shreyas, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ash- win Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva.2024. “RLHF deciphered: a critical analysis of reinforcement learning from human feedback for LLMs.” Working Paper

    2024

  11. [19]

    Evaluating large language models trained on code

    Chen, Mark, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. “Evaluating large language models trained on code.” Working Paper

    2021

  12. [20]

    How general are measures of choice consistency? Evidence from experimental and scanner data

    Chen, Mingshi, Tracy Xiao Liu, You Shan, Shu Wang, Songfa Zhong, and Yanju Zhou. 2025a. “How general are measures of choice consistency? Evidence from experimental and scanner data.” Working Paper

  13. [21]

    A manager and an AI walk into a bar: does chatGPT make biased decisions like we do?

    Chen, Yang, Samuel N Kirshner, Anton Ovchinnikov, Meena Andiappan, and Tracy Jenkin. 2025b. “A manager and an AI walk into a bar: does chatGPT make biased decisions like we do?” Manufacturing & Service Operations Management, 27(2): 354–368

  14. [22]

    The emer- gence of economic rationality of GPT

    Chen, Yiting, Tracy Xiao Liu, You Shan, and Songfa Zhong. 2023. “The emer- gence of economic rationality of GPT.” Proceedings of the National Academy of Sciences , 120(51): e2316205120

    2023

  15. [23]

    Consistency and heterogeneity of individual behavior under uncertainty

    Choi, Syngjoo, Raymond Fisman, Douglas Gale, and Shachar Kariv.2007. “Consistency and heterogeneity of individual behavior under uncertainty.” American Economic Review , 97(5): 1921–1938

    2007

  16. [24]

    Who is (more) rational?

    Choi, Syngjoo, Shachar Kariv, Wieland M ¨uller, and Dan Silverman. 2014. “Who is (more) rational?” American Economic Review, 104(6): 1518–1550

    2014

  17. [25]

    The Economic Potential of Generative AI: The Next Productivity Frontier

    Chui, Michael, Eric Hazan, Roger Roberts, Alex Singla, Kate Smaje, Alex Sukharevsky, Lareina Yee, and Rodney Zemmel. 2023. “The Economic Potential of Generative AI: The Next Productivity Frontier.” https://www.mckinsey.com/capabilities/mckinsey-digital/our- insights/the-econom...

    2023

  18. [26]

    Training verifiers to solve math word problems

    Cobbe, Karl, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. “Training verifiers to solve math word problems.”Working Paper

    2021

  19. [27]

    The method of experimental economics

    Croson, Rachel. 2005. “The method of experimental economics.”International Negotiation, 10(1): 131–148

    2005

  20. [28]

    Gender differences in preferences

    Croson, Rachel, and Uri Gneezy. 2009. “Gender differences in preferences.” Journal of Economic Literature, 47(2): 448–474

    2009

  21. [29]

    Davis, Douglas D, and Charles A Holt. 1993. Experimental economics. Princeton Univer- sity Press

    1993

  22. [30]

    Can LLM agents recognize demographic heterogeneity in economic games?

    Deng, Zebang, Rosemarie Nagel, Jubo Yan, and Tong Zhang. 2025. “Can LLM agents recognize demographic heterogeneity in economic games?” Working Paper

    2025

  23. [31]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. “BERT: pre-training of deep bidirectional transformers for language understanding.” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu...

    2019

  24. [32]

    Individual risk attitudes: measurement, determinants, and behav- ioral consequences

    Dohmen, Thomas, Armin Falk, David Huffman, Uwe Sunde, J ¨urgen Schupp, and Gert G Wagner. 2011. “Individual risk attitudes: measurement, determinants, and behav- ioral consequences.” Journal of the European Economic Association, 9(3): 522–550

    2011

  25. [33]

    Driess, Danny, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al

  26. [34]

    The money pump as a measure of revealed preference violations

    Echenique, Federico, Sangmok Lee, and Matthew Shum. 2011. “The money pump as a measure of revealed preference violations.” Journal of Political Economy, 119(6): 1201– 1223

    2011

  27. [35]

    Integrating machine behavior into human subject experiments: a user-friendly toolkit and illustrations

    Engel, Christoph, Max RP Grossmann, and Axel Ockenfels. 2023. “Integrating machine behavior into human subject experiments: a user-friendly toolkit and illustrations.” Working Paper

    2023

  28. [36]

    Switch transformers: scaling to trillion parameter models with simple and efficient sparsity

    Fedus, William, Barret Zoph, and Noam Shazeer. 2022. “Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.” Journal of Machine Learning Research, 23(120): 1–39

    2022

  29. [37]

    AI and per- ception biases in investments: an experimental study

    Fedyk, Anastassia, Ali Kakhbod, Peiyao Li, and Ulrike Malmendier. 2025. “AI and per- ception biases in investments: an experimental study.”Working Paper

    2025

  30. [38]

    The distributional preferences of an elite

    Fisman, Raymond, Pamela Jakiela, Shachar Kariv, and Daniel Markovits. 2015. “The distributional preferences of an elite.” Science, 349(6254): aab0096. 54

    2015

  31. [39]

    The distributional preferences of Americans, 2013–2016

    Fisman, Raymond, Pamela Jakiela, Shachar Kariv, and Silvia Vannutelli. 2023. “The distributional preferences of Americans, 2013–2016.” Experimental Economics, 26(4): 727– 748

    2023

  32. [40]

    Experimental economics: Past and future

    Fr´echette, Guillaume R, Kim Sarnoff, and Leeat Yariv. 2022. “Experimental economics: Past and future.” Annual Review of Economics, 14(1): 777–794

    2022

  33. [41]

    Friedman, Daniel, and Shyam Sunder. 1994. Experimental methods: a primer for economists. Cambridge University Press

    1994

  34. [42]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars

    Gandhi, Kanishk, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Good- man. 2025. “Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.”Working Paper

    2025

  35. [43]

    How real is hypothetical? a high-stakes test of the allais paradox

    Gneezy, Uri, Yoram Halevy, Brian Hall, Theo Offerman, and Jeroen van de Ven. 2024. “How real is hypothetical? a high-stakes test of the allais paradox.” Working Paper

    2024

  36. [44]

    Frontiers: can large language models capture hu- man preferences?

    Goli, Ali, and Amandeep Singh. 2024. “Frontiers: can large language models capture hu- man preferences?” Marketing Science, 43(4): 709–722

    2024

  37. [45]

    Reinforcement learning from human feedback in LLMs: whose culture, whose values, whose perspectives?

    Gonz´alez Barman, Kristian, Simon Lohse, and Henk W de Regt. 2025. “Reinforcement learning from human feedback in LLMs: whose culture, whose values, whose perspectives?” Philosophy & Technology, 38(2): 1–26

    2025

  38. [46]

    Human-like intuitive behav- ior and reasoning biases emerged in large language models but disappeared in chatGPT

    Hagendorff, Thilo, Sarah Fabi, and Michal Kosinski. 2023. “Human-like intuitive behav- ior and reasoning biases emerged in large language models but disappeared in chatGPT.” Nature Computational Science, 3(10): 833–838

    2023

  39. [47]

    Parametric recoverability of pref- erences

    Halevy, Yoram, Dotan Persitz, and Lanny Zrill. 2018. “Parametric recoverability of pref- erences.” Journal of Political Economy, 126(4): 1558–1593

    2018

  40. [48]

    Risk aversion and incentive effects

    Holt, Charles A, and Susan K Laury.2002. “Risk aversion and incentive effects.”American Economic Review, 92(5): 1644–1655

    2002

  41. [49]

    Holzmeister, Felix, Magnus Johannesson, Colin F Camerer, Yiling Chen, Teck-Hua Ho, Suzanne Hoogeveen, Juergen Huber, Noriko Imai, Taisuke Imai, Lawrence Jin, et al

  42. [50]

    Large language models as simulated economic agents: What can we learn from homo silicus?

    Horton, John J. 2023. “Large language models as simulated economic agents: What can we learn from homo silicus?” Working Paper

    2023

  43. [51]

    A survey on hal- lucination in large language models: principles, taxonomy, challenges, and open questions

    Huang, Lei, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al.2025. “A survey on hal- lucination in large language models: principles, taxonomy, challenges, and open questions.” ACM Transactions on ...

    2025

  44. [52]

    On the psychology of prediction

    Kahneman, Daniel, and Amos Tversky. 1973. “On the psychology of prediction.” Psycho- logical Review, 80(4): 237. 55

    1973

  45. [53]

    Scaling laws for neural language models

    Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Re- won Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. “Scaling laws for neural language models.” Working Paper

    2020

  46. [54]

    Kim, Hyuncheol Bryant, Syngjoo Choi, Booyuel Kim, and Cristian Pop-Eleches

  47. [55]

    Learning to be homo economicus: Can an LLM learn preferences from choice

    Kim, Jeongbin, Matthew Kovach, Kyu-Min Lee, Euncheol Shin, and Hector Tzavel- las. 2024. “Learning to be homo economicus: Can an LLM learn preferences from choice.” Working Paper

    2024

  48. [56]

    Evaluating large language models in theory of mind tasks

    Kosinski, Michal. 2024. “Evaluating large language models in theory of mind tasks.” Pro- ceedings of the National Academy of Sciences, 121(45): e2405460121

    2024

  49. [57]

    Insurance decisions for low-probability losses

    Laury, Susan K, Melayne Morgan McInnes, and J Todd Swarthout. 2009. “Insurance decisions for low-probability losses.” Journal of Risk and Uncertainty, 39: 17–44

    2009

  50. [58]

    Can LLMs mimic human-like mental accounting and behavioral biases?

    Leng, Yan. 2024. “Can LLMs mimic human-like mental accounting and behavioral biases?” Working Paper

    2024

  51. [59]

    Do LLM agents exhibit social behavior?

    Leng, Yan, and Yuan Yuan. 2023. “Do LLM agents exhibit social behavior?” Working Paper

    2023

  52. [60]

    The role of education interventions in improving economic rationality

    “The role of education interventions in improving economic rationality.” Science, 362(6410): 83–86

  53. [61]

    Experimental evidence of physician social preferences

    Li, Jing, Lawrence P Casalino, Raymond Fisman, Shachar Kariv, and Daniel Markovits. 2022. “Experimental evidence of physician social preferences.” Proceedings of the National Academy of Sciences, 119(28): e2112726119

    2022

  54. [62]

    Social preferences of future physi- cians

    Li, Jing, William H Dow, and Shachar Kariv. 2017. “Social preferences of future physi- cians.” Proceedings of the National Academy of Sciences, 114(48): E10291–E10300

    2017

  55. [63]

    Lost in the middle: how language models use long con- texts

    Liu, Nelson F, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. “Lost in the middle: how language models use long con- texts.” Transactions of the Association for Computational Linguistics, 12: 157–173

  56. [64]

    Large language models assume people are more rational than we really are

    Liu, Ryan, Jiayi Geng, Joshua C Peterson, Ilia Sucholutsky, and Thomas L Griffiths. 2024b. “Large language models assume people are more rational than we really are.”Working Paper

  57. [65]

    (Ir)rationality and cognitive biases in large language models

    Macmillan-Scott, Olivia, and Mirco Musolesi. 2024. “(Ir)rationality and cognitive biases in large language models.” Royal Society Open Science, 11(6): 240255

    2024

  58. [66]

    Reduce preference disparity be- tween LLMs and humans: calibration to compose human-like ensembles

    Leng, Yan, Yunxin Sang, and Ashish Agarwal. 2024. “Reduce preference disparity be- tween LLMs and humans: calibration to compose human-like ensembles.” Working Paper

    2024

  59. [67]

    A turing test of whether ai chatbots are behaviorally similar to humans

    Mei, Qiaozhu, Yutong Xie, Walter Yuan, and Matthew O. Jackson. 2024. “A turing test of whether ai chatbots are behaviorally similar to humans.” Proceedings of the National Academy of Sciences, 121(9): e2313925121

    2024

  60. [68]

    AI emerges as the frontier in behavioral science

    Meng, Juanjuan. 2024. “AI emerges as the frontier in behavioral science.” Proceedings of the National Academy of Sciences, 121(10): e2401336121

    2024

  61. [69]

    Efficient es- timation of word representations in vector space

    Mikolov, Tomas, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. “Efficient es- timation of word representations in vector space.” In The First International Conference on Learning Representations

    2013

  62. [70]

    Muennighoff, Niklas, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Ha- jishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand`es, and Tatsunori Hashimoto

  63. [71]

    Experiments: Why, How, and A Users Guide for Producers as well as Consumers

    Niederle, Muriel. 2025. “Experiments: Why, How, and A Users Guide for Producers as well as Consumers.” Working Paper

    2025

  64. [72]

    The AI Index 2025 Annual Report

    Maslej, Nestor, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Capstick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, et al. 2025. “The AI Index 2025 Annual Report.” AI Index Steering Committee, Institute for Human-Centered AI, Stanf...

    2025

  65. [73]

    How ethical should AI be? How AI alignment shapes the risk preferences of LLMs

    Ouyang, Shumiao, Hayong Yun, and Xingjian Zheng. 2024. “How ethical should AI be? How AI alignment shapes the risk preferences of LLMs.” Working Paper

    2024

  66. [74]

    BLEU: a method for automatic evaluation of machine translation

    Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. “BLEU: a method for automatic evaluation of machine translation.” In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311–318

    2002

  67. [75]

    Is tem- perature the creativity parameter of large language models?

    Peeperkorn, Max, Tom Kouwenhoven, Dan Brown, and Anna Jordanous.2024. “Is tem- perature the creativity parameter of large language models?” In International Conference on Computational Creativity 2024

    2024

  68. [76]

    Deep contextualized word representations

    Peters, Matthew E, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. “Deep contextualized word representations.” In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisti...

    2018

  69. [77]

    s1: simple test-time scaling

    “s1: simple test-time scaling.” Working Paper

  70. [78]

    Improv- ing language understanding by generative pre-training

    Radford, Alec, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. “Improv- ing language understanding by generative pre-training.” OpenAI. Technical report

    2018

  71. [79]

    Zoom In: An Introduction to Circuits

    Olah, Chris, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. “Zoom In: An Introduction to Circuits.” Distill. https://distill.pub/2020/circuits/zoom-in

    2020

  72. [80]

    The effect of sampling temperature on prob- lem solving in large language models

    Renze, Matthew, and Erhan Guven. 2024. “The effect of sampling temperature on prob- lem solving in large language models.” In Findings of the Association for Computational Linguistics: EMNLP 2024. 7346–7356

    2024

  73. [81]

    ChatGPT sets record for fastest-growing user base - analyst note

    Reuters. 2023. “ChatGPT sets record for fastest-growing user base - analyst note.” https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst- note-2023-02-01/ (last accessed May 25, 2025)

    2023

  74. [82]

    A systematic survey of prompt engineering in large language models: techniques and applications

    Sahoo, Pranab, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. “A systematic survey of prompt engineering in large language models: techniques and applications.” Working Paper

    2024

  75. [83]

    Outrageously large neural networks: the sparsely-gated Mixture-of-Experts layer

    Shazeer, Noam, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geof- frey Hinton, and Jeff Dean. 2017. “Outrageously large neural networks: the sparsely-gated Mixture-of-Experts layer.” In The Fifth International Conference on Learning Representa- tions

    2017

  76. [84]

    An experimental examination of two ex- change institutions

    Plott, Charles R, and Vernon L Smith. 1978. “An experimental examination of two ex- change institutions.” Review of Economic Studies, 45(1): 133–153

    1978

  77. [85]

    Microeconomic systems as an experimental science

    Smith, Vernon L. 1982. “Microeconomic systems as an experimental science.” American Economic Review, 72(5): 923–955

    1982

  78. [86]

    Benchmarking prompt sensitivity in large language models

    Razavi, Amirhossein, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zi- hayat, and Ebrahim Bagheri. 2025. “Benchmarking prompt sensitivity in large language models.” Working Paper. 57

    2025

  79. [87]

    Do large language models show decision heuristics similar to humans? A case study using GPT-3.5

    Suri, Gaurav, Lily R Slater, Ali Ziaee, and Morgan Nguyen. 2024. “Do large language models show decision heuristics similar to humans? A case study using GPT-3.5.”Journal of Experimental Psychology: General, 153(4): 1066

    2024

  80. [88]

    Attention is all you need

    Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. “Attention is all you need.” In Pro- ceedings of the 31st International Conference on Neural Information Processing Systems . 6000–6010

    2017

  81. [89]

    Large language mod- els that replace human participants can harmfully misportray and flatten identity groups

    Wang, Angelina, Jamie Morgenstern, and John P Dickerson.2025. “Large language mod- els that replace human participants can harmfully misportray and flatten identity groups.” Nature Machine Intelligence, 7: 400–411

    2025

  82. [90]

    Will the real Linda please stand up... to large language models? Examining the representativeness heuristic in LLMs

    Wang, Pengda, Zilin Xiao, Hanjie Chen, and Frederick L Oswald. 2024. “Will the real Linda please stand up... to large language models? Examining the representativeness heuristic in LLMs.” In First Conference on Language Modeling

    2024

  83. [91]

    Experimental economics: induced value theory

    Smith, Vernon L. 1976. “Experimental economics: induced value theory.” American Eco- nomic Review, 66(2): 274–279

    1976

  84. [92]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. “Chain-of-thought prompting elicits reasoning in large lan- guage models.” Advances in Neural Information Processing Systems, 35: 24824–24837. 58

    2022

  85. [93]

    Large Language Mod- els are overconfident and amplify human bias

    Sun, Fengfei, Ningke Li, Kailong Wang, and Lorenz Goette.2025. “Large Language Mod- els are overconfident and amplify human bias.” Working Paper

    2025

  86. [94]

    Effi- cient streaming language models with attention sinks

    Xiao, Guangxuan, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.2023. “Effi- cient streaming language models with attention sinks.” Working Paper

    2023

  87. [95]

    Logic-RL: unleashing llm reasoning with rule-based reinforcement learning

    Xie, Tian, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. 2025. “Logic-RL: unleashing llm reasoning with rule-based reinforcement learning.” Working Paper

    2025

  88. [96]

    How different AI chatbots behave? Benchmarking large language models in behavioral economics games

    Xie, Yutong, Yiyao Liu, Zhuang Ma, Lin Shi, Xiyuan Wang, Walter Yuan, Matthew O Jackson, and Qiaozhu Mei. 2024. “How different AI chatbots behave? Benchmarking large language models in behavioral economics games.” Working Paper

    2024

  89. [97]

    Language models trained to do arithmetic predict human risky and intertemporal choice

    Zhu, Jian-Qiao, Haijiang Yan, and Thomas L Griffiths. 2025. “Language models trained to do arithmetic predict human risky and intertemporal choice.” In The Thirteenth Interna- tional Conference on Learning Representations

    2025

  90. [98]

    Emergent analogical reasoning in large language models

    Webb, Taylor, Keith J Holyoak, and Hongjing Lu.2023. “Emergent analogical reasoning in large language models.” Nature Human Behaviour, 7(9): 1526–1541

    2023

  91. [100]

    Retrieval head mechanistically explains long-context factuality

    Wu, Wenhao, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu.2024. “Retrieval head mechanistically explains long-context factuality.”Working Paper

    2024

  92. [105]

    Experimenter demand effects in economic experiments

    Zizzo, Daniel John. 2010. “Experimenter demand effects in economic experiments.”Exper- imental Economics, 13: 75–98. 59

    2010

  93. [2018]

    Evaluating the replicability of social science experiments in Nature and Science be- tween 2010 and 2015

    “Evaluating the replicability of social science experiments in Nature and Science be- tween 2010 and 2015.” Nature Human Behaviour, 2(9): 637–644

    2010

  94. [2020]

    Language models are few-shot learners

    “Language models are few-shot learners.” In Proceedings of the 34th International Conference on Neural Information Processing Systems. 52

  95. [2021]

    On the opportunities and risks of foundation models

    “On the opportunities and risks of foundation models.” Working Paper

  96. [2023]

    PaLM-E: an embodied multimodal language model

    “PaLM-E: an embodied multimodal language model.” In Proceedings of the 40th In- ternational Conference on Machine Learning

  97. [2025]

    Examining the replicability of online experiments selected by a decision market

    “Examining the replicability of online experiments selected by a decision market.” Nature Human Behaviour, 9(2): 316–330

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.