REVIEW 4 major objections 5 minor 21 references
StockSim: A Dual-Mode Order-Level Simulator for Evaluating Multi-Agent LLMs in Financial Markets
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read StockSim promises identical reruns and realistic microstructure for LLM trading agents.
desk verdict Useful open-source testbed for LLM trading agents, but fidelity claims outrun the evidence: the demo uses the simplified candlestick fill rule and determinism is not correctness. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Exchange Simulation Engine, the asynchronous message-routing and order-matching core of StockSim. It maintains order state and market-impact state, computes technical indicators on the fly, and routes market data, news, and fills between data providers and agents. The order-level matcher uses price-time priority in a limit-order book, so executions depend on queue position, available volume, and the timing of other participants. The candlestick matcher uses the simpler rule that an order fills if its price lies inside the current bar's high-low range. The same agent messages flow through both modes, which is why the engine's design, rather than any individual strategy, carries the paper's reproducibility and generalizability claims.
What would settle it
Feed one identical agent order stream into StockSim's order-level engine and its candlestick engine over the same historical OHLCV data, using a recorded tick sequence for the same period, and compare the executed trades, cash, and final portfolio. If the candlestick engine fills orders that the recorded queue depth or tick path would reject, or misses fills the order book would have provided, then the claim that the two modes consistently model the same market breaks for that data.
Extended reading notes
Core claim
The central discovery is architectural: a single asynchronous Exchange Simulation Engine can resolve agent orders in two modes without changing the agent-facing API. In order-level mode, orders enter a limit-order book and are matched by price-time priority, with full and partial fills, queue position, latency, and slippage. In candlestick mode, an order is deemed executable when its price falls within the high-low range of an OHLCV bar. The paper reports that running identical deterministic agents through this engine yields identical simulation outputs across repeated runs, and that resource use grows roughly linearly up to about 150 agents. That reproducibility, paired with the dual-mode design, is what the paper offers as a foundation for fair, controlled evaluation of LLM trading behavior.
Load-bearing premise
That a bar's high-low range is enough to decide whether an order fills: in candlestick mode, an order at any price inside the high-low range is treated as executable, which assumes away intra-bar price paths, queue priority, partial fills, and latency.
Editorial extensions
If this is right
- An agent written once for StockSim can be stress-tested in both microstructure-aware order-level mode and long-history candlestick mode by changing only configuration.
- Deterministic replay means two multi-agent strategies can be compared on the same simulated market sequence, isolating strategy differences from market noise.
- Researchers can report a single configuration and expect other labs to reproduce the same orders and performance metrics.
- Capturing latency, slippage, and queue placement makes reported LLM trading results closer to what would happen in a real electronic market.
- The no-code configuration layer lets behavioral studies of LLM traders be run without building custom execution infrastructure.
Reading between the lines
- Not a paper claim: candlestick mode's fill rule could be made path-aware by generating a plausible intra-bar price path and checking every pending order against it; the paper already mentions simulated intra-bar paths for stop orders, so this is a small extension that would let the two execution modes be compared head-to-head.
- Not a paper claim: reproducibility was demonstrated only with deterministic agents, so a fair evaluation protocol for stochastic LLMs will need explicit seeds, temperature settings, or multiple trajectories before cross-paper comparisons are meaningful.
- Not a paper claim: the reported gap between the two tested LLMs comes from a single two-month, single-stock setup with one prompt, so it should be read as a demonstration of the platform's diagnostics, not as a general model ranking.
- Not a paper claim: if StockSim becomes a community standard, the bottleneck shifts from simulator realism to data licensing and prompt and configuration versioning, because identical simulation output across labs still depends on sharing the same market data and agent prompts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents StockSim, an open-source dual-mode market simulator for evaluating LLM-based trading agents. It offers an order-level execution mode operating on a limit-order book with latency, queueing, and partial fills, and a candlestick-level execution mode based on OHLCV bars. The system includes a role-based multi-agent framework, external data integration, technical indicators, and an evaluator with standard performance metrics. The authors claim that StockSim fully models market dynamics, is deterministic and correct, scales almost linearly up to ~150 agents, and they demonstrate its use by comparing GPT-o4-mini and GPT-o3 on a two-month NVIDIA trading scenario, concluding that GPT-o3 is more selective and profitable.
Significance. If substantiated, StockSim would fill a real gap in open-source infrastructure for LLM trading-agent research by combining a high-fidelity order-level engine with a scalable bar-level mode, an extensible agent framework, and a built-in evaluator. The open-source release, modular design, and feature comparison with existing simulators are genuine contributions. However, the paper's central claims are currently under-supported: the candlestick-mode fill rule is a simplistic high-low touch heuristic whose bias is unquantified, determinism is equated with correctness, the LLM demonstration is anecdotal and lacks baselines or error bars, and the scalability test uses deterministic agents rather than LLM agents. Because the headline evaluation and the 'faithful assessment' claim rest on these points, the manuscript requires substantial revision before the conclusions can be accepted.
major comments (4)
- [§3.1, Table 2] The candlestick-level fill rule—'if the agent submits an order at a price that falls within the range of a given candle, the order can be executed; otherwise, it cannot'—is the standard high-low touch heuristic. This rule ignores intra-bar price paths, queue priority, available volume, and partial fills, and it is known to systematically overfill limit and stop orders, especially when the high or low is touched only briefly or during illiquid periods. The GPT-o4-mini and GPT-o3 results in Table 2 are produced entirely in this mode, so the reported ROI, Sharpe, Sortino, Win Rate, and Profit Factor differences inherit this optimistic bias. The paper itself concedes in §3.1 that this mode 'fails to capture critical dynamics' and in the Limitations section that 'market simulation, while realistic, cannot fully capture all complexities of actual trading environments,' which directly contradicts the Abstract's claim that StockSim 'fully models market dynamics' and 'incorporates critical real-world factors, such as latency, slippage, and order-book microstructure.' Please either restrict the 'faithful assessment' claim to the order-level mode, or quantify the fill-rule bias by comparing candlestick-mode fills against order-level fills on the same underlying data and report the resulting differences in the performance metrics.
- [§4, Consistency paragraph] The sentence 'This repeatability empirically verifies the platform's deterministic behavior and validates its correctness, since any deviation would indicate flaws in the design or execution logic' is logically invalid. Determinism only establishes reproducibility; correctness requires fidelity to real market processes. A deterministically buggy engine would pass this test. The paper provides no external validation: no comparison of simulated fills against real LOB data, no calibration of the candlestick surrogate against the order-level engine, and no statistical test of fill rates, spread dynamics, or price impact. Please rephrase the claim as one of reproducibility only, and add at least one validation experiment—for example, replaying a day from LOBSTER and comparing aggregate execution statistics, or showing that candlestick-mode fills approximate order-level fills within a stated tolerance.
- [§4, Table 2, 'LLM Trading Behavior'] The two-model comparison is a single scenario (NVIDIA, April 28–June 28, 2025) with no repeated runs, no error bars, no baselines (such as buy-and-hold, a random agent, or a simple rule-based trader), and no significance tests. The conclusion that GPT-o3 exhibits 'superior ability' is unsupported: a Profit Factor of 999.0 signals a near-zero gross-loss denominator, which is typical of a tiny sample (9 trades), and a perfect Win Rate is not informative without a confidence interval. The statement that 'the evaluator's results effectively capture and distinguish the underlying strategic differences between the two LLMs' is therefore anecdotal. Please either add a rigorous evaluation with repeated runs, baselines, and appropriate statistics, or clearly label this section as a platform demonstration rather than a model comparison.
- [§4, Scalability paragraph] The scalability test uses deterministic agents (the paper explicitly says 'we exclude LLMs'), yet the text concludes that 'STOCK SIM can handle such scale with ease' in a discussion about running 500 concurrent LLM agents. LLM agents introduce stochastic latency, token generation, and API call overhead, so the resource measurements from deterministic agents do not transfer. The super-linear growth beyond 150 agents (mean CPU rising to 123% at 300 agents and 418% at 500 agents) also suggests that LLM workloads may degrade much earlier. Please either run the scalability test with representative LLM agents or rephrase the claim to state specifically that the measurement applies to deterministic agents only, and temper the 'with ease' conclusion accordingly.
minor comments (5)
- [Throughout] The name 'STOCK SIM' is used inconsistently with 'StockSim' and 'StockSim' in figures and code; please standardize the capitalization.
- [§1, Introduction] The phrase 'Comprehensive ordel-level simulation' contains a typo: 'ordel' should be 'order.'
- [Figure 4 caption] The caption refers to the stock 'EXON' for what appears to be ExxonMobil; the correct ticker symbol is XOM. If the figure uses a different symbol, please correct the text.
- [Appendix B] The definitions of Sharpe Ratio and Sortino Ratio omit practical details required for reproducibility: the risk-free rate, the return periodicity (daily, per-trade), and the annualization factor. Please specify these parameters.
- [§3.1, Candlestick Level Execution] The text says 'STOCK SIM simulates realistic price paths within each bar' to support conditional orders like stop losses, but no details of this path model are given. Please specify the path-generation method (e.g., Brownian bridge, random walk), its parameters, and how it interacts with the high-low touch rule.
Circularity Check
No significant circularity: StockSim's claims are empirical platform demonstrations, not derivations from fitted parameters or self-citations.
full rationale
The paper contains no derivation chain in which a predicted quantity is constructed from the same data it is meant to explain. StockSim is presented as a simulator, and its evaluation claims are empirical demonstrations: deterministic agents are run to check reproducibility, and two LLMs are run to illustrate behavioral differences. No parameter is fitted to the LLM trading results, and no reported metric is used to define or calibrate the simulator; the candlestick execution rule (fill if the order price falls within the candle's range) is an explicit simplifying assumption stated in Section 3.1, not a result derived from agent outputs. The paper also explicitly concedes in the Limitations section that the market simulation 'cannot fully capture all complexities of actual trading environments,' which undermines the abstract's overstatement but does not constitute circular reasoning. There are no self-citations serving as load-bearing evidence, no imported uniqueness theorems, and no renaming of a known result as a new derivation. The consistency test in Section 4 is an empirical reproducibility check of a deterministic system, and while it does not by itself validate market realism, it is not circular. Any concern about the fidelity of the candlestick fill heuristic is a correctness or validity risk, not a circularity defect. The appropriate finding is no significant circularity, score 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Candlestick-level execution rule: an order is filled if its price is within the high-low range of a candle.
- domain assumption Order book event data from LOBSTER or generated logs accurately represent real market microstructure.
- domain assumption Latency and slippage parameters can be set to realistic values without calibration to a specific market.
Cite this review
Pith. "Pith review of StockSim: A Dual-Mode Order-Level Simulator for Evaluating Multi-Agent LLMs in Financial Markets." pith.science (2026). https://pith.science/paper/4VK3DVML
@misc{pith2026250709255,
author = {Pith},
title = {Pith review of: StockSim: A Dual-Mode Order-Level Simulator for Evaluating Multi-Agent LLMs in Financial Markets},
year = {2026},
howpublished = {\url{https://pith.science/paper/4VK3DVML}},
note = {Machine review of arXiv:2507.09255}
}
read the original abstract
We present StockSim, an open-source simulation platform for systematic evaluation of large language models (LLMs) in realistic financial decision-making scenarios. Unlike previous toolkits that offer limited scope, StockSim delivers a comprehensive system that fully models market dynamics and supports diverse simulation modes of varying granularity. It incorporates critical real-world factors, such as latency, slippage, and order-book microstructure, that were previously neglected, enabling more faithful and insightful assessment of LLM-based trading agents. An extensible, role-based agent framework supports heterogeneous trading strategies and multi-agent coordination, making StockSim a uniquely capable testbed for NLP research on reasoning under uncertainty and sequential decision-making. We open-source all our code at https: //github.com/harrypapa2002/StockSim.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
David Byrd, Maria Hybinette, and Tucker Hybinette Balch. 2020. Abides: Towards high-fidelity multi-agent market simulation. In Proceedings of the 2020 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation, pages 11--22
work page 2020
-
[4]
Á. Cartea and S. Jaimungal. 2015. https://doi.org/10.1111/mafi.12023 Risk metrics and fine tuning of high‐frequency trading strategies . Mathematical Finance, 25
-
[5]
Dingyang Chen, Qi Zhang, and Yinglun Zhu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.517 Efficient sequential decision making with large language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9157--9170, Miami, Florida, USA. Association for Computational Linguistics
-
[6]
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. https://doi.org/10.18653/v1/2024.findings-acl.716 Generalization or memorization: Data contamination and trustworthy evaluation for large language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 12039--12050, Bangkok, Thailand. Associ...
-
[7]
Sascha Yves Frey, Kang Li, Peer Nagy, Silvia Sapora, Christopher Lu, Stefan Zohren, Jakob Foerster, and Anisoara Calinescu. 2023. Jax-lob: A gpu-accelerated limit order book simulator to unlock large scale reinforcement learning for trading. In Proceedings of the Fourth ACM International Conference on AI in Finance, pages 583--591
work page 2023
-
[8]
Haohang Li, Yupeng Cao, Yangyang Yu, Shashidhar Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, Koduvayur Subbalakshmi, Guojun Xiong, et al. 2024 a . Investorbench: A benchmark for financial decision-making tasks with llm-based agent. arXiv preprint arXiv:2412.18174
arXiv 2024
Show all 21 references
-
[9]
Yuan Li, Bingqiao Luo, Qian Wang, Nuo Chen, Xu Liu, and Bingsheng He. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.63 C rypto T rade: A reflective LLM -based agent to guide zero-shot cryptocurrency trading . In Proceedings of the 2024 Conference on Empirical Methods in...
2024 doi
-
[10]
Xiao-Yang Liu, Ziyi Xia, Jingyang Rui, Jiechao Gao, Hongyang Yang, Ming Zhu, Christina Wang, Zhaoran Wang, and Jian Guo. 2022. Finrl-meta: Market environments and benchmarks for data-driven financial reinforcement learning. Advances in Neural Information Processing Systems, 35...
2022
-
[11]
Xiao-Yang Liu, Hongyang Yang, Qian Chen, Runjia Zhang, Liuqing Yang, Bowen Xiao, and Christina Dan Wang. 2020. Finrl: A deep reinforcement learning library for automated stock trading in quantitative finance. CoRR
2020
-
[12]
Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng, Jinyi Niu, Zixuan Wang, Jiajie Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, et al. 2025. Fin-r1: A large language model for financial reasoning through reinforcement learning. arXiv preprint arXiv:2503.16252
2025
-
[13]
Guilong Lu, Xuntao Guo, Rongjunchen Zhang, Wenqiao Zhu, and Ji Liu. 2025. Bizfinbench: A business-driven real-world financial benchmark for evaluating llms. arXiv preprint arXiv:2505.19457
2025 arXiv
-
[14]
Chris Mascioli, Anri Gu, Yongzhao Wang, Mithun Chakraborty, and Michael Wellman. 2024. A financial market simulation environment for trading agents using deep reinforcement learning. In Proceedings of the 5th ACM International Conference on AI in Finance, pages 117--125
2024
-
[15]
Ashimiyu Nafiu, Salaam Olawale Balogun, Courage Oko-Odion, and Olanrewaju Olukoya Odumuwagun. 2025. Risk management strategies: Navigating volatility in complex financial market environments
2025
-
[16]
Simon Rudkin, Wanling Qiu, and Pawe D otko. 2023. Uncertainty, volatility and the persistence norms of financial time series. Expert Systems with Applications, 223:119894
2023
-
[17]
Aaditya K Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes. 2024. Evaluation data contamination in llms: how do we measure it and (when) does it matter? arXiv preprint arXiv:2411.03923
2024 arXiv
-
[18]
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. 2024. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314, 4
2024 arXiv
-
[19]
New concepts in technical trading systems, trend research
J Welles Wilder Jr. New concepts in technical trading systems, trend research. PO BOX, 450:130
-
[20]
Yijia Xiao, Edward Sun, Di Luo, and Wei Wang. 2024. Tradingagents: Multi-agents llm financial trading framework. arXiv preprint arXiv:2412.20138
2024 arXiv
-
[21]
Gaurang Singh Yadav, Apratim Guha, and Anindya S Chakrabarti. 2020. Measuring complexity in financial data. Frontiers in Physics, 8:339
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.