REVIEW 5 major objections 5 minor 12 references
Evaluating Binary Decision Biases in Large Language Models: Implications for Fair Agent-Based Financial Simulations
T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read LLMs cannot reliably produce fair, memory-less binary decisions under independent sampling.
desk verdict Useful bias data on GPT subversions, but Table 2's chi-squares don't match the counts and undercut the central non-Markovian claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the binary response sequence produced by an LLM under two query regimes: one-shot (one decision per independent API call) and few-shot (100 comma-separated decisions in a single call). The paper evaluates each sequence with three statistical instruments: a chi-square goodness-of-fit test against a uniform 50/50 distribution, a chi-square test comparing $P(\text{Yes}|\text{yes on the previous decision})$ with $P(\text{Yes})$ to test the Markov (memory-less) property, and a sliding-window recency measure, switching rate after runs of length $w$ minus baseline switching rate, to detect negative recency. A temperature sweep modifies the softmax scale $\beta = 1/T$ to test whether output randomness can be controlled. These instruments carry the argument because they turn raw API responses into verdicts about uniformity, independence, and human-like bias.
What would settle it
Recompute the transition chi-square statistics from the raw response sequences. For GPT-4o-Mini Q1, the reported $P(\text{Yes})=0.32$ and $P(\text{Yes}|\text{Yes})=0.28$ imply a chi-square value near $0.2$, not the reported $21.9$; if the corrected statistic is below the critical value of $3.841$, the one-shot non-Markovian claim for that model and prompt is unsupported.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that state-of-the-art GPT models cannot reliably reproduce a uniform binary distribution under independent sampling, and that their response sequences violate the Markov property that financial simulations typically assume. The evidence is a two-prompt benchmark ('yes or no' and 'Answer randomly, yes or no') run on three specific model versions. GPT-4o-mini-2024-07-18 produced the least biased output, 32-43% yes across prompts, with one prompt statistically indistinguishable from uniformity, while GPT-4-0125-preview and GPT-3.5-turbo-0125 produced 87-99% yes. In one-shot testing, all response sequences except one were judged sequentially dependent, and even the near-uniform GPT-4o-mini sequence was non-Markovian. Few-shot sampling averaged over batches approximated 50/50 in most cases but produced pervasive within-batch dependence, and temperature settings from 0.5 to 2.0 failed to yield both uniform and Markovian output in any model. The paper concludes that sampling method, model sub-version, and prompt all materially change the bias, and that none of the tested configurations provides a fair, memory-less binary decision source.
Load-bearing premise
The load-bearing premise is that the statistical tests for sequential dependence were computed correctly from the raw response counts; if those numbers are wrong, the claim that one-shot responses are not memory-less collapses.
Editorial extensions
If this is right
- One-shot LLM queries cannot be treated as fair random coin flips inside an agent-based financial simulation, because a model producing 87-99% yes responses would systematically skew every binary trading decision.
- Few-shot batch responses are not a safe substitute for independent agent decisions, because averaging hides strong sequential dependence within the batch.
- Violations of the Markov property mean that simulated agents' decisions are influenced by their own previous decisions, which can look like spurious herding or information leakage in a market model.
- Choosing which model sub-version is deployed is a first-order design decision: GPT-4-0125-preview and GPT-4o-mini-2024-07-18 differ by roughly 60 percentage points in yes-rate on the same prompt.
- Temperature adjustments do not provide a reliable randomness dial for binary choices, since no tested temperature produced both a uniform distribution and Markovian independence.
Reading between the lines
- A natural extension the authors do not develop is calibration: using the measured yes-bias to weight or reject-sample LLM responses could let financial ABMs keep the richness of LLM decisions without inheriting the bias.
- The Common Crawl frequency analysis points to a testable training-data explanation: if 'yes' is systematically over-represented in pretraining text relative to 'no', then the bias should appear across LLM families and languages roughly in proportion to corpus imbalance.
- The one-shot versus few-shot distinction maps onto two different ABM deployment patterns, individually queried agents versus centrally batched populations, so batch-generated agents are not exchangeable with independently sampled ones even when marginal distributions match.
- The binary benchmark is cheap enough to serve as a pre-deployment screening test: run a few hundred one-shot 'yes or no' queries, check the yes-rate and the transition dependence, and reject configurations that fail before embedding them in a financial simulator.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports an empirical study of binary yes/no generation by three GPT models (GPT-4o-mini-2024-07-18, GPT-4-0125-preview, GPT-3.5-turbo-0125) under one-shot independent API calls and few-shot batched calls. It tests uniformity with chi-square goodness-of-fit tests, first-order Markovianity with chi-square tests on transition probabilities, the effect of temperature on both properties, the frequency of yes/no in Common Crawl data, and negative-recency behavior relative to Random.org binary sequences. The headline claim is that no model simultaneously achieves a uniform distribution and Markovian properties in one-shot testing, with GPT-4o-mini closest (uniform for Q2 but claimed non-Markovian), while few-shot averaging improves uniformity but introduces strong sequential alternation.
Significance. If the central claims survive correction, the paper is a timely and useful benchmark for LLM-as-agent design in financial agent-based models. The raw vote-share findings are informative and falsifiable: GPT-4-0125-preview and GPT-3.5-turbo-0125 show extreme yes bias, GPT-4o-mini is relatively balanced, and one-shot versus few-shot sampling produce strikingly different distributions. The temperature study and the comparison against Random.org and Common Crawl provide useful external reference points, and the paper is candid about the practical limitations of few-shot sampling for ABMs. However, the Markovianity analysis, which carries the paper's strongest negative claim, is not statistically sound as presented; the contribution will be credible only after the transition statistics are recomputed and the conclusions re-evaluated.
major comments (5)
- [Results, Table 2] The chi-square statistics in Table 2 cannot be reproduced from the reported conditional probabilities and YY/n counts. For GPT-4o-Mini Q1, P(Y)=0.32 and YY/n=9/32 imply roughly 9 Yes→Yes, 23 Yes→No, 23 No→Yes, and 45 No→No transitions, giving a 2x2 chi-square statistic of about 0.3 on 1 degree of freedom, not 21.9. For Q2, P(Y)=0.43 and YY/n=20/42 give a statistic of about 0.6, not 36.2. Similar discrepancies appear in Table 4; for example, GPT-3.5 Q1 with P(Y)=0.50, P(Y|Y)=0.06, and YY/N=32/498 should produce a 2x2 chi-square near 700, not 187.4. The paper must report the full 2x2 transition tables and the exact test used, and the one-shot non-Markovian conclusion for GPT-4o-Mini is unsupported until this is corrected.
- [Results, Table 2 (GPT-3.5 Q1)] The all-yes sequence for GPT-3.5 Q1 is labeled 'Reject* due to near-perfect dependence,' but a constant sequence trivially satisfies H0: P(Yes_n|Yes_{n-1}) = P(Yes_n) because both probabilities are 1. The chi-square test is degenerate in this case and provides no evidence against Markovianity. Rejecting H0 for deterministic or near-deterministic sequences on the grounds of 'near-perfect dependence' is not a valid statistical procedure; if such sequences are excluded, the exclusion should be stated and justified a priori.
- [Temperature section, Table 5] Table 5 states that 'All tests: Failed to reject H0 at α = 0.05' for GPT-4o-Mini across both questions and all temperature settings, and the text says this model 'produced Markovian response ... across both questions and all temperature settings.' This directly contradicts Table 2, which reports rejection of H0 for GPT-4o-Mini Q1 and Q2 at T=1. If Table 5 is correct, then GPT-4o-Mini Q2 in the one-shot regime may satisfy both uniformity (Table 1, p=0.162) and Markovianity, contradicting the abstract's central claim. The paper must resolve this internal inconsistency and clarify whether Tables 1/2 and Table 5 use the same or different API runs.
- [Testing methodology, Hypothesis 2] The statement of H0 for the Markov test is incomplete: a first-order Markov test should compare P(Yes_n|Yes_{n-1}) with P(Yes_n|No_{n-1}), not with the marginal P(Yes_n). The paper never reports P(Yes|No) or the full transition count matrix, making the reported chi-square results uninterpretable. Please provide the full transition tables for every model/question/sampling condition and use a standard chi-square test of independence on those tables.
- [Results, Table 1] The methodology states that one-shot testing collects 100 responses per prompt, but Table 1 reports only 87 responses for GPT-3.5 Q1 (87 yes, 0 no). No explanation is given for the missing 13 responses, and other rows do not show their totals explicitly. This affects every subsequent statistic and must be clarified, including whether invalid or malformed responses were discarded.
minor comments (5)
- [Throughout] There are several typographical errors in model names: 'GPT-4o-mini-204-07-18' in the Few-Shot Results section, 'GPT-4o-Mini-2014-07-18' in the Temperature section, and 'GPT-4o-Mini-2034,07,18' in the Conclusion.
- [Tables 2, 4, 5] The notation 'E[P(Y—Y)]' is nonstandard and should be written as an estimated conditional probability, e.g., \hat{P}(Y|Y). The expression 'YY/n' is also ambiguous: in Tables 2 and 5 the denominator appears to be the number of Yes responses, not the total sequence length, so it should be defined explicitly.
- [Table 7] The legend of Table 7 is unclear: R, A, R=NR, P=4, and P=5 are not defined in the text, and the table mixes three symbols per cell without a full caption explaining which test each symbol refers to.
- [Table 6] The header 'Prob. No Cond. Word % Prob.' is confusing and should be replaced with explicit column names such as P(Yes), P(Yes|Yes or No), and the fraction of pages containing neither term.
- [Temperature section] Table 5 does not report uniformity test statistics, only P(Y), P(Y|Y), and YY/n, so the sweeping claim that temperature adjustments do not fix distributional bias is not fully supported by the displayed evidence; chi-square or p-values for uniformity should be added.
Circularity Check
No significant circularity: the paper's claims are empirical comparisons against uniform and true-random benchmarks, and its self-citations do not supply any load-bearing derivation.
full rationale
This paper is an empirical evaluation, not a derivation. The test statistics (chi-square for uniformity, Markov property, and recency switching rates) are computed from raw LLM outputs and compared against external benchmarks (a p = 0.5 uniform null and Random.org sequences). The one-shot/few-shot distinction is an operational definition of API call structure, not a fitted quantity, and no parameter is fitted to data and then renamed as a prediction. The only self-citations (Vidler and Walsh 2024a, 2024b) motivate why LLM bias matters for agent-based financial simulations and are not used to define, fit, or predict any measured result. No equation in the paper serves as both input and derived outcome. The reader-flagged arithmetic inconsistencies in Table 2's chi-square values would, if confirmed, be a correctness or statistical-error issue rather than a circularity issue, because the claimed conclusion is not true by construction of the test. For these reasons, no circular step can be exhibited, and the appropriate score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Independent API calls with 1-second delays constitute independent samples from the model's internal distribution.
- standard math Chi-square goodness-of-fit and chi-square independence tests are applied correctly.
- domain assumption The prompts 'yes or no' and 'Answer randomly, yes or no' are treated as equivalent to a fair coin toss.
- domain assumption Common Crawl sample (CC-MAIN-2024-30) is representative of GPT-4o-Mini training data.
Cite this review
Pith. "Pith review of Evaluating Binary Decision Biases in Large Language Models: Implications for Fair Agent-Based Financial Simulations." pith.science (2026). https://pith.science/paper/SCPAJSUO
@misc{pith2026250116356,
author = {Pith},
title = {Pith review of: Evaluating Binary Decision Biases in Large Language Models: Implications for Fair Agent-Based Financial Simulations},
year = {2026},
howpublished = {\url{https://pith.science/paper/SCPAJSUO}},
note = {Machine review of arXiv:2501.16356}
}
read the original abstract
Large Language Models (LLMs) are increasingly being used to simulate human-like decision making in agent-based financial market models (ABMs). As models become more powerful and accessible, researchers can now incorporate individual LLM decisions into ABM environments. However, integration may introduce inherent biases that need careful evaluation. In this paper we test three state-of-the-art GPT models for bias using two model sampling approaches: one-shot and few-shot API queries. We observe significant variations in distributions of outputs between specific models, and model sub versions, with GPT-4o-Mini-2024-07-18 showing notably better performance (32-43% yes responses) compared to GPT-4-0125-preview's extreme bias (98-99% yes responses). We show that sampling methods and model sub-versions significantly impact results: repeated independent API calls produce different distributions compared to batch sampling within a single call. While no current GPT model can simultaneously achieve a uniform distribution and Markovian properties in one-shot testing, few-shot sampling can approach uniform distributions under certain conditions. We explore the Temperature parameter, providing a definition and comparative results. We further compare our results to true random binary series and test specifically for the common human bias of Negative Recency - finding LLMs have a mixed ability to 'beat' humans in this one regard. These findings emphasise the critical importance of careful LLM integration into ABMs for financial markets and more broadly.
Figures
Reference graph
Works this paper leans on
-
[2]
A Survey on Evaluation of Large Language Models. arXiv:2307.03109. Chopra, A.; Kumar, S.; Giray-Kuru, N.; Raskar, R.; and Quera-Bofarull, A
-
[4]
DOI:10.13140/RG.2.2.27881.47207
Does GPT-4 Play Dice ? Experiment : Gen- erating only one number each time Experiment : Generating random sequence. DOI:10.13140/RG.2.2.27881.47207. Luo, H.; Sun, Q.; Xu, C.; Zhao, P.; Lou, J.; Tao, C.; Geng, X.; Lin, Q.; Chen, S.; and Zhang, D
-
[5]
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct. arXiv:2308.09583. Ng, A
-
[6]
Generative Agents: Interactive Simu- lacra of Human Behavior. arXiv:2304.03442:2304.03442. Peeperkorn, M.; Kouwenhoven, T.; Brown, D.; and Jor- danous, A
-
[7]
Is Temperature the Creativity Parameter of Large Language Models? arXiv:2405.00492:2405.00492. Renda, A.; Hopkins, A. K.; and Carbin, M
-
[8]
ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 4(1): 1–22
Can LLMs Generate Random Numbers? Evaluating LLM Sampling in Controlled Domains. ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 4(1): 1–22. Renze, M.; and Guven, E
work page 2023
-
[9]
The Effect of Sampling Temperature on Problem Solving in Large Language Mod- els. arXiv:2402.05201. Tessema, B. M.; Kedia, A.; and Chung, T.-S
-
[10]
UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages
UnifiedCrawl: Aggregated Common Crawl for Afford- able Adaptation of LLMs on Low-Resource Languages. arXiv:2411.14343. Tjuatja, L.; Chen, V .; Wu, T.; Talwalkwar, A.; and Neubig, G
Show all 12 references
-
[12]
arXiv:1606.04080
Matching Networks for One Shot Learning. arXiv:1606.04080. Wang, Y .; Zhang, Z.; Chen, H.; and Shen, H
-
[2017]
In Proceedings of the 31st Inter- national Conference on Neural Information Processing Sys- tems, NIPS’17, 6000–6010
At- tention is all you need. In Proceedings of the 31st Inter- national Conference on Neural Information Processing Sys- tems, NIPS’17, 6000–6010. Red Hook, NY , USA: Curran Associates Inc. ISBN 9781510860964. Vidler, A.; and Walsh, T. 2024a. TraderTalk: An LLM Behavioural ABM...
-
[2023]
arXiv preprint arXiv:2303.12712
Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. Chang, Y .; Wang, X.; Wang, J.; Wu, Y .; Yang, L.; Zhu, K.; Chen, H.; Yi, X.; Wang, C.; Wang, Y .; Ye, W.; Zhang, Y .; Chang, Y .; Yu, P. S.; Yang, Q.; and Xie, X
-
[2024]
arXiv:2406.00092
How Random is Random? Evaluating the Randomness and Humaness of LLMs’ Coin Flips. arXiv:2406.00092. Liu, Q
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.