Pith. sign in

REVIEW 4 major objections 5 minor 34 references

This paper argues that LLM trading benchmarks that score direct buy/hold/sell actions are unreliable because repeated runs diverge wildly, and it replaces them with a benchmark that scores executable strategy code, which is reproducible, te

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:39 UTC pith:UE3TT3NE

load-bearing objection The instability diagnosis for action-based LLM trading is real and worth taking seriously; the proposed code-generation benchmark is a reasonable direction, but its validity claims currently outrun the evidence. the 4 major comments →

arxiv 2602.18481 v2 pith:UE3TT3NE submitted 2026-02-10 q-fin.TR cs.AI

AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models

Wentao Zhang , Mingxuan Zhao , Jincheng Gao , Jieshun You , Huaiyu Jia , Yilei Zhao , Bo An , Shuo Sun This is my paper
classification q-fin.TR cs.AI
keywords LLM evaluationtrading benchmarkdecision instabilityaction flippingstrategy code generationalpha factor generationbacktestingreproducibility
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

AlphaForgeBench argues that existing trading benchmarks are broken in a specific way: when an LLM is asked to emit buy/hold/sell actions, repeated runs under identical settings produce wildly different action sequences and outcome metrics, even with greedy decoding. The paper attributes this to the stateless, autoregressive nature of LLMs and to the unstable mapping of continuous market signals to discrete actions. Its fix is to reposition the LLM as a quantitative researcher: the model must write executable alpha-factor and strategy code, which a deterministic backtest engine then runs on historical data. Across 903 queries, six frontier LLMs, and seven assets, the resulting scores are temperature-invariant, reproducible, and more discriminative than action-based evaluation. A reader should care because it offers a way to evaluate financial reasoning in LLMs without trusting their noisy moment-to-moment decisions.

Core claim

The paper's central discovery is a failure mode and a workaround. Direct-trading LLMs exhibit extreme decision instability: pairwise agreement between runs can fall below 0.4, action sequences differ even at temperature zero, and small divergences compound into dramatically different returns and drawdowns. The paper argues this is structural, not incidental: a stateless autoregressive model re-decides at every step, and the continuous-to-discrete action mapping amplifies small input changes into flips. The workaround is to force the LLM to formalize its decision boundaries as explicit algorithmic rules in code; stochasticity is then confined to generation, and execution is strictly determini

What carries the argument

The load-bearing object is a generate-and-backtest pipeline: each natural-language query is turned into a standardized prompt, the LLM returns a self-contained `generate_signal` function implementing an alpha factor or trading rule, and a unified deterministic backtest engine executes that code on daily OHLCV data across seven assets, computing Sharpe, return, drawdown, and related metrics. What makes it work is the alpha-factor/strategy-code artifact itself: it is executable, auditable, and lets the engine manage position state, so LLM stochasticity cannot leak into the score. A secondary mechanism is the 3x3 level-grade taxonomy (logic translation, logic completion, goal-oriented generatio

Load-bearing premise

The benchmark's rankings stand on the assumption that LLM-generated queries faithfully capture the intent of the original real-world strategies and that backtest scores measure financial reasoning rather than code-generation skill or query-recall bias.

What would settle it

Blind human experts compare a random sample of Stage 1 queries against their source documents to judge whether each query preserves the core strategy intent; if a large share of queries are judged distorted or infeasible as written, the benchmark's rankings would reflect query-generation artifacts instead of financial reasoning. Alternatively, an ablation replacing the LLM-extracted queries with human-written versions of the same strategies would reveal whether rankings survive.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Existing online and offline trading benchmarks that score single-run action sequences should be treated as unreliable; reported model rankings may largely reflect decoding noise, not financial skill.
  • LLM stochasticity can be confined to a generation step: once the model outputs code, evaluation is deterministic, so backtests are reproducible across runs and temperatures.
  • Code-translation ability is nearly saturated across frontier models, while open-ended strategy design exposes large capability gaps; benchmarks should emphasize goal-oriented tasks to rank models.
  • The fixed risk-return profiles (aggressive vs conservative) that emerge suggest choosing a model for automated strategy generation is a risk-preference decision, not just a quality decision.
  • The paradigm extends naturally to portfolio-level and multi-asset strategies, which the paper identifies as future work, so the same approach can cover broader quantitative-research workflows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the queries themselves are produced by LLMs (extraction and generation stages), the benchmark inherits whatever blind spots those models have; a human-validated subset would be the natural next check that rankings reflect real strategy quality rather than query-generation bias.
  • The instability result likely generalizes beyond finance: any sequential decision benchmark that asks an LLM to emit per-step discrete actions (tool use, web navigation, control) may be measuring sampling noise; 'generate a plan or code, then execute deterministically' is a general remedy worth testing.
  • The frictionless, long-only, daily setup may reward strategies that are turnover-heavy or cost-blind; adding transaction costs and short-selling constraints could change the ranking and would test whether the reported risk personalities are robust.
  • A direct testable extension: feed the same generated code to a live-market forward-testing engine; if backtest rankings do not persist out-of-sample, then the benchmark measures in-sample code quality rather than tradable alpha.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that LLM-based direct trading agents suffer from severe run-to-run and temperature-driven decision instability, making existing action-based trading benchmarks unreliable. It proposes AlphaForgeBench, which repurposes LLMs as code generators: models receive a natural-language strategy query and must return executable Python code implementing a trading signal; the code is then executed in a deterministic backtest engine over seven assets and five years. The benchmark comprises 633 'real-world' queries extracted from financial sources by gemini-3-flash-preview and rewritten by gpt-5.2, plus 270 synthetic queries generated by gpt-5.2 under a 3x3 level-grade taxonomy. Six frontier LLMs are evaluated with five runs per query at T=0.7 (and additionally T=0 for Stage 2), for 35,190 generated implementations. The main reported findings are that model rankings are stable across runs and temperatures, that a three-tier hierarchy emerges, and that the difficulty taxonomy progressively separates models. The paper concludes that the code-generation paradigm eliminates execution-induced instability and yields a rigorous, discriminative benchmark for financial reasoning.

Significance. If the instability measurements and the benchmark-validity assumptions hold, this is a useful contribution: it documents a concrete failure mode in action-emitting LLM trading agents (Appendix C) and proposes a plausible alternative evaluation protocol with a deterministic backtest engine, explicit interface contracts, and a large-scale multi-run protocol. The paper's reproducible setup (k=5 runs per query, both temperatures, unified engine) and its eventual public release of code/data are strengths. However, the central claim that AlphaForgeBench measures financial reasoning rather than query-recall or code-generation fluency depends on unvalidated LLM-generated queries and has no fidelity check; without addressing these gaps, the benchmark's validity and the 'more discriminative' claim remain unsupported.

major comments (4)
  1. [§3.1, §D.1.2, §D.2.3] The benchmark queries are themselves produced by the models under evaluation: gemini-3-flash-preview extracts all Stage 1 entries (§D.1.2) and gpt-5.2 rewrites them and generates all 270 Stage 2 queries (§D.2.3). gpt-5.2 is one of the six benchmarked models. No human validation, inter-annotator agreement, or query-fidelity audit is reported. This is load-bearing because the benchmark's validity requires that the 903 queries faithfully represent real-world strategies and are not biased toward particular models' preferences. I would expect (i) a human-validated subset of queries with agreement rates, (ii) a ranking-stability analysis excluding gpt-5.2-generated queries or controlling for query source, and (iii) at least a qualitative audit of extraction/rewriting errors. Without this, 'financial reasoning' rankings may reflect query-recall or code fluency.
  2. [§3.2 Step 3; §4.3.1, Table 2 Level 1] The evaluation pipeline never checks whether the generated code actually implements the stated query. Step 3 only backtests the code. Consequently, a model can achieve high backtest scores by producing a generic profitable strategy (e.g., momentum trend-following) that is unrelated to the specified rules. This is particularly problematic for Level 1 'Logic Translation', where Table 2 shows near-identical SR values across all models (range ≈ 0.03); without a fidelity check, this cannot be interpreted as uniformly successful translation; it may simply indicate all models default to similar generic logic. The Level 1 claims in §4.3.1 and the cross-level 'dissociable capabilities' argument in §4.3.6 therefore rest on an unmeasured quantity. Please add an automated or human fidelity metric (e.g., comparing the generated code's decision boundaries against the query's rules) and report translat
  3. [§4.2, §4.3, Conclusion (and Appendix C)] The abstract and conclusion claim the benchmark is 'more discriminative' than direct-trading baselines, but no direct-trading comparison is run on the same queries, models, assets, or metrics. Appendix C measures instability in a different environment (BTC only, 2025, buy/hold/sell actions) and reports variance, not discrimination. The internal evidence for 'discriminative' is the widening spread from Level 1 to Level 3 in Table 2, which is a property of the Stage 2 taxonomy, not a comparison against action emission. To support the headline claim, please include a same-query direct-trading variant (or a statistical variance-component analysis on the benchmark itself) showing that the code-generation protocol yields smaller run-to-run variance and larger between-model separation than action emission.
  4. [§4.2.1 and §3.3] The key stability claim—'intra-query standard deviation across 5 runs is typically an order of magnitude smaller than inter-query standard deviation'—is not reported in any table or figure. Tables 1 and 2 report mean ± pooled standard deviation, which mixes run-to-run and query-to-query variation; for example, a Level 3 entry in Table 2 with SR=0.734±0.167 at T=0 cannot be decomposed into generation noise vs query difficulty. Because the central advantage of the benchmark is reproducibility, please report a variance decomposition (within-query σ across 5 runs, between-query σ, and between-model σ) for at least the overall results, with the actual numbers.
minor comments (5)
  1. [§2.2 (reference [14])] Text refers to 'AlphaFin [14]' but the reference list entry [14] is 'Stockbench: Can llm agents trade stocks profitably...'. Please correct the citation or reference.
  2. [Table 2 caption] In the caption, 'asser' should be 'asset'.
  3. [Appendix C.5, Figure 12 panel B] The caption reports 'Disagreement Steps: 68.4%' for T=0, but the accompanying text in §C.5 says '88.4%'. Please make these consistent.
  4. [Tables 2 and 10] The headers use 'SOR' in Table 2 but 'SoR' elsewhere; standardize to 'SoR'.
  5. [§1] The claim that existing online trading benchmarks are 'rendering their evaluations unreliable' is broad; Appendix C tests only six models on a single BTC task. Consider softening or adding evidence across assets.

Circularity Check

0 steps flagged

No significant circularity: the instability measurements and code-generation evaluation are empirically self-contained; the LLM-generated-query validity concerns are construct-validity risks, not circular reductions.

full rationale

I walked the claimed derivation chain and found no step in which a predicted result reduces by construction to a fitted input, a self-citation chain, or a definitional equivalence. The direct-trading instability results (Appendix C) are independent empirical measurements of action-sequence agreement and metric variance; they do not reuse AlphaForgeBench as evidence. The code-generation evaluation (Sec. 3.2, Appx. E.1) is self-contained: queries are assembled into standardized prompts, code is generated, and a deterministic backtest engine computes financial metrics. The temperature-invariance claim (Sec. 4.3) is reported empirically (max SR difference < 0.008) rather than assumed; the deterministic backtest is a stated design property, not a fitted parameter renamed as a prediction. There are no author self-citations used as load-bearing support, no imported uniqueness theorem, and no ansatz smuggled in via citation. The genuine validity concerns—Stage 1 queries rewritten by gpt-5.2 with explicit instruction to 'approximate the intent using technical indicators' (Appx. D.1.4), all 270 Stage 2 queries generated by gpt-5.2 (Appx. D.2.3), which is itself one of the evaluated models, and the absence of a code-query fidelity check in Step 3—are benchmark-validity and reliability risks. They do not exhibit the specific kind of reduction required for circularity: no equation is equal to its input by construction, and no fitted value is relabeled as a prediction. Accordingly, the honest finding is no significant circularity, score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters are fitted to data: the design choices (5 runs, 300-day lookback, temperature 0.7) are experimental settings, not fitted values. The axioms listed are the assumptions the paper relies on to turn its measurements into a general benchmark. No new physical or conceptual entity is postulated.

axioms (4)
  • domain assumption LLM-generated queries (Stage 1 extraction via gemini-3-flash-preview, Stage 2 generation via gpt-5.2) faithfully represent real-world financial strategies and are free of model-specific bias.
    Section 3.1: the benchmark queries are produced by LLMs without human verification, yet the benchmark's validity depends on their fidelity.
  • domain assumption Backtest performance on seven assets over 2021-2026 with zero transaction costs is a valid proxy for financial reasoning ability.
    Section 4.1 and E.3.3: the evaluation uses frictionless, long-only, single-asset backtests; this assumes the resulting SR/ARR/MDD rankings reflect reasoning rather than risk-preference or overfitting artifacts.
  • domain assumption The decision instability observed on BTC daily trading in 2025 generalizes to all direct-trading benchmarks.
    Appendix C tests only one asset and one time window, but the paper generalizes to 'existing online and offline trading benchmarks' in the abstract and Section 1.
  • ad hoc to paper LLM action instability is caused by statelessness, continuous-to-discrete sensitivity, and classification rather than policy optimization.
    Section 1 presents these as attribution ('Fundamentally, this phenomenon can be primarily attributed to...') without controlled experiments that isolate these mechanisms.

pith-pipeline@v1.3.0-alltime-deepseek · 58319 in / 11528 out tokens · 111358 ms · 2026-08-03T02:39:12.187020+00:00 · methodology

0 comments
read the original abstract

The rapid advancement of Large Language Models (LLMs) has led to a surge of financial benchmarks, evolving from static knowledge evaluation toward interactive trading simulations. However, existing frameworks for evaluating real-time trading largely overlook a critical failure mode: the severe behavioral instability of LLMs in sequential decision-making under financial uncertainty. Through extensive experiments, we show that when deployed as trading agents, LLMs exhibit extreme run-to-run variance, generate inconsistent action sequences even under deterministic decoding, and frequently produce irrational action flipping across adjacent time steps. We attribute these behaviors to the stateless autoregressive nature of LLMs, which lack persistent memory of prior actions, together with their sensitivity to continuous-to-discrete action mappings in portfolio allocation tasks. These deficiencies fundamentally undermine the reliability and reproducibility of many existing online and offline trading benchmarks. To address these limitations, we propose AlphaForgeBench, a principled evaluation framework that redefines LLMs as quantitative researchers rather than stochastic trading agents. Instead of producing discrete trading actions, AlphaForgeBench requires models to generate executable alpha factors and compose factor-based trading strategies grounded in financial knowledge. This paradigm decouples reasoning from execution mechanics, enabling deterministic and reproducible evaluation while remaining aligned with real-world quantitative research workflows. Extensive experiments across multiple state-of-the-art LLMs demonstrate that AlphaForgeBench eliminates execution-induced instability and provides a rigorous benchmark for evaluating financial reasoning, strategy formulation, and alpha discovery. Webpage at https://finbrain-lab-hkustgz.github.io/AlphaForgeBench

Figures

Figures reproduced from arXiv: 2602.18481 by Bo An, Huaiyu Jia, Jieshun You, Jincheng Gao, Mingxuan Zhao, Shuo Sun, Wentao Zhang, Yilei Zhao.

Figure 1
Figure 1. Figure 1: The framework of AlphaForgeBench. across runs even under deterministic decoding (Section C), ren￾dering single-run live evaluations statistically unreliable. While existing benchmarks have advanced the evaluation of autonomous trading agents, they overlook the necessity of a holistic alpha min￾ing pipeline that integrates strategy formulation, executable code generation, and rigorous backtesting, and large… view at source ↗
Figure 3
Figure 3. Figure 3: Sharpe Ratio by difficulty level on LLM-augmented [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Sharpe Ratio by asset on LLM-augmented queries [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Aligned cumulative return curves on LLM [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Cross-run instability of gemini-3-pro-preview under different decoding temperatures. Run-to-run Variability [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Decision instability across decoding temperatures [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cross-run instability of gemini-3-flash-preview under different decoding temperatures. Run-to-run Variability [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Decision instability across decoding temperatures [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Cross-run instability of deepseek-v3.2 under different decoding temperatures. Run-to-run Variability [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Decision instability across decoding temperatures [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Cross-run instability of grok-4.1-fast under different decoding temperatures. Run-to-run Variability [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Decision instability across decoding temperatures ( [PITH_FULL_IMAGE:figures/full_fig_p015_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Cross-run instability of claude-sonnet-4.5 under different decoding temperatures. Run-to-run Variability [PITH_FULL_IMAGE:figures/full_fig_p016_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Decision instability across decoding temperatures [PITH_FULL_IMAGE:figures/full_fig_p016_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Cross-run instability of gpt-5.2 under different decoding temperatures. Run-to-run Variability [PITH_FULL_IMAGE:figures/full_fig_p017_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Decision instability across decoding temperatures ( [PITH_FULL_IMAGE:figures/full_fig_p017_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Normalized radar chart of model performance on the Stage 1 real-world benchmark across five metrics. Each [PITH_FULL_IMAGE:figures/full_fig_p039_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Grouped bar chart comparing Sharpe Ratio (SR), Annualized Return Rate (ARR), and Sortino Ratio (SoR) across six [PITH_FULL_IMAGE:figures/full_fig_p040_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Per-asset grouped bar chart comparing six LLMs across key metrics on the Stage 1 real-world benchmark. Each [PITH_FULL_IMAGE:figures/full_fig_p040_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Per-asset box plot of strategy performance distributions on the Stage 1 real-world benchmark. Each box summarizes [PITH_FULL_IMAGE:figures/full_fig_p041_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Distribution of core financial metrics across six LLMs on the Stage 1 real-world benchmark. Each box summarizes 633 [PITH_FULL_IMAGE:figures/full_fig_p044_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Aligned return curves across all assets (smoothed with 20-query moving average, 25–75% quantile band). Queries [PITH_FULL_IMAGE:figures/full_fig_p046_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Per-asset aligned return curves (Part 1: BTCUSDT, ETHUSDT, AAPL, GOOGL). [PITH_FULL_IMAGE:figures/full_fig_p047_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Per-asset aligned return curves (Part 2: MSFT, NVDA, TSLA). [PITH_FULL_IMAGE:figures/full_fig_p048_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Radar chart of normalized model performance on the Stage 2 benchmark across five metrics at both temperature [PITH_FULL_IMAGE:figures/full_fig_p051_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Heatmap of Sharpe Ratio across models and difficulty levels on the Stage 2 benchmark. Darker colors indicate [PITH_FULL_IMAGE:figures/full_fig_p052_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Grouped bar chart of Sharpe Ratio across difficulty levels (L1, L2, L3) for six LLMs on the Stage 2 benchmark. [PITH_FULL_IMAGE:figures/full_fig_p054_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Performance trends across difficulty levels (model-averaged). Core metrics generally decline from Level 1 to Level [PITH_FULL_IMAGE:figures/full_fig_p054_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Detailed performance breakdown across 9 fine-grained difficulty levels. Each level (L1, L2, L3) is subdivided into easy, [PITH_FULL_IMAGE:figures/full_fig_p055_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Boxplot distributions across 9 fine-grained difficulty levels. Box boundaries represent the interquartile range (IQR), [PITH_FULL_IMAGE:figures/full_fig_p056_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Boxplot distributions aggregated by three main difficulty levels (L1, L2, L3). This coarse-grained view highlights the [PITH_FULL_IMAGE:figures/full_fig_p057_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: Comparison of standard deviation and mean performance across temperature settings (T=0.0 vs T=0.7). The minimal [PITH_FULL_IMAGE:figures/full_fig_p058_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: Grouped bar chart of model performance across all seven assets (BTCUSDT, ETHUSDT, AAPL, GOOGL, MSFT, NVDA, [PITH_FULL_IMAGE:figures/full_fig_p060_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: Model performance ranking visualization. Bar heights represent aggregate performance scores computed across all [PITH_FULL_IMAGE:figures/full_fig_p061_35.png] view at source ↗
Figure 36
Figure 36. Figure 36: Grouped bar chart comparing all models across core metrics. Each metric is normalized to facilitate cross-metric [PITH_FULL_IMAGE:figures/full_fig_p061_36.png] view at source ↗
Figure 37
Figure 37. Figure 37: Boxplot distributions by model across all evaluation instances. Box width (IQR) indicates consistency, with narrow [PITH_FULL_IMAGE:figures/full_fig_p062_37.png] view at source ↗
Figure 38
Figure 38. Figure 38: Robustness analysis showing performance stability across multiple runs. [PITH_FULL_IMAGE:figures/full_fig_p065_38.png] view at source ↗
Figure 39
Figure 39. Figure 39: Cross-asset robustness analysis. Box plots show performance distribution across 7 different assets. [PITH_FULL_IMAGE:figures/full_fig_p065_39.png] view at source ↗
Figure 40
Figure 40. Figure 40: gemini-3-pro-preview performance across 9 difficulty levels. G.6.1 gemini-3-pro-preview. Overview. gemini-3-pro-preview is Google’s flagship model with advanced reasoning and multimodal capabili￾ties. Within AlphaForgeBench it serves as the representative of the aggressive-creative archetype, consistently prioritizing high-conviction signal logic over capital preservation. Analysis. As shown in [PITH_FUL… view at source ↗
Figure 41
Figure 41. Figure 41: Best-performing strategies generated by gemini-3-pro-preview [PITH_FULL_IMAGE:figures/full_fig_p067_41.png] view at source ↗
Figure 42
Figure 42. Figure 42: gpt-5.2 performance across 9 difficulty levels. gpt-5.2 achieves the lowest Maximum Drawdown (0.119 at 𝜏=0) and Volatility (0.163) among all evaluated models, reflecting a distinctly conservative strategy generation profile. While its overall Sharpe Ratio (0.415) places it in the lower tier, the competitive Calmar Ratio (1.599) reveals efficient return-to-drawdown management: the model sacrifices upside p… view at source ↗
Figure 43
Figure 43. Figure 43: Best-performing strategies generated by gpt-5.2 [PITH_FULL_IMAGE:figures/full_fig_p069_43.png] view at source ↗
Figure 44
Figure 44. Figure 44: claude-sonnet-4.5 performance across 9 difficulty levels. G.6.3 claude-sonnet-4.5. Overview. claude-sonnet-4.5 is Anthropic’s flagship model, known for strong reasoning capabilities. Within AlphaForgeBench it exemplifies the balanced-stable archetype, achieving the most favorable trade-off between return generation, risk control, and cross-run consistency. Analysis. As shown in [PITH_FULL_IMAGE:figures/f… view at source ↗
Figure 45
Figure 45. Figure 45: Best-performing strategies generated by claude-sonnet-4.5 [PITH_FULL_IMAGE:figures/full_fig_p071_45.png] view at source ↗
Figure 46
Figure 46. Figure 46: gemini-3-flash-preview performance across 9 difficulty levels. G.6.4 gemini-3-flash-preview. Overview. gemini-3-flash-preview is Google’s efficient model optimized for speed and cost-effectiveness. Within AlphaForgeBench it occupies the balanced-stable archetype alongside claude-sonnet-4.5, serving as a cost-effective alternative to its Pro-tier sibling with a remarkably flat difficulty profile. Analysis.… view at source ↗
Figure 47
Figure 47. Figure 47: Best-performing strategies generated by gemini-3-flash-preview [PITH_FULL_IMAGE:figures/full_fig_p073_47.png] view at source ↗
Figure 48
Figure 48. Figure 48: deepseek-v3.2 performance across 9 difficulty levels. G.6.5 deepseek-v3.2. Overview. deepseek-v3.2 is an open-weights model with strong performance on coding and reasoning tasks. Within AlphaForgeBench it falls under the conservative-rigid archetype, exhibiting the most pronounced dissociation between code-translation competence and open-ended strategic reasoning among all evaluated models. Analysis. deep… view at source ↗
Figure 49
Figure 49. Figure 49: Best-performing strategies generated by deepseek-v3.2 [PITH_FULL_IMAGE:figures/full_fig_p075_49.png] view at source ↗
Figure 50
Figure 50. Figure 50: grok-4.1-fast performance across 9 difficulty levels. G.6.6 grok-4.1-fast. Overview. grok-4.1-fast is xAI’s model evaluated in this benchmark. It falls under the conservative-rigid archetype but is further distinguished by the highest run-to-run variance among all models, making it the least predictable generator in AlphaForgeBench. Analysis. As shown in [PITH_FULL_IMAGE:figures/full_fig_p076_50.png] view at source ↗
Figure 51
Figure 51. Figure 51: Best-performing strategies generated by grok-4.1-fast [PITH_FULL_IMAGE:figures/full_fig_p077_51.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 7 linked inside Pith

  1. [1]

    Finqa: A dataset of numerical reasoning over financial data

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, 2021

    2021

  2. [2]

    Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance.arXiv preprint arXiv:2105.07624, 2021

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance.arXiv preprint arXiv:2105.07624, 2021

    Pith/arXiv arXiv 2021

  3. [3]

    Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering.arXiv preprint arXiv:2210.03849, 2022

    Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering.arXiv preprint arXiv:2210.03849, 2022

    Pith/arXiv arXiv 2022

  4. [4]

    Bloomberggpt: A large language model for finance.arXiv preprint arXiv:2303.17564, 2023

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Se- bastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance.arXiv preprint arXiv:2303.17564, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Fingpt: Democratizing internet-scale data for financial large language models.arXiv preprint arXiv:2307.10485, 2023

    Xiao-Yang Liu, Guoxuan Wang, Hongyang Yang, and Daochen Zha. Fingpt: Democratizing internet-scale data for financial large language models.arXiv preprint arXiv:2307.10485, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Pixiu: A comprehensive benchmark, instruction dataset and large language model for finance.Advances in Neural Information Processing Systems, 36:33469–33484, 2023

    Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. Pixiu: A comprehensive benchmark, instruction dataset and large language model for finance.Advances in Neural Information Processing Systems, 36:33469–33484, 2023

    2023

  7. [7]

    Finben: A holistic financial benchmark for large language models.Advances in Neural Information Processing Systems, 37:95716–95743, 2024

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. Finben: A holistic financial benchmark for large language models.Advances in Neural Information Processing Systems, 37:95716–95743, 2024

    2024

  8. [8]

    Ai trading in real markets, 2026

    Alpaha Arena. Ai trading in real markets, 2026. URL https://nof1.ai/. Accessed: 2026-01-23

    2026

  9. [9]

    Fintextqa: A dataset for long-form financial question answering

    Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. Fintextqa: A dataset for long-form financial question answering. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6025–6047, 2024

    2024

  10. [10]

    Cfinbench: A comprehensive chinese financial benchmark for large language models

    Ying Nie, Binwei Yan, Tianyu Guo, Hao Liu, Haoyu Wang, Wei He, Binfan Zheng, Weihao Wang, Qiang Li, Weijian Sun, et al. Cfinbench: A comprehensive chinese financial benchmark for large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (...

    2025

  11. [11]

    Fin-eva version 1.0: A chinese financial evaluation benchmark for large language models, 2025

    Ant Group and Shanghai University of Finance and Economics. Fin-eva version 1.0: A chinese financial evaluation benchmark for large language models, 2025. URL https://github.com/alipay/financial_evaluation_dataset. Accessed: 2026-01- 23

    2025

  12. [12]

    Ucfe: A user-centric financial expertise benchmark for large language models

    Yuzhe Yang, Yifei Zhang, Yan Hu, Yilin Guo, Ruoli Gan, Yueru He, Mingcong Lei, Xiao Zhang, Haining Wang, Qianqian Xie, et al. Ucfe: A user-centric financial expertise benchmark for large language models. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 5429–5448, 2025

    2025

  13. [13]

    Yangyang Yu, Zhiyuan Yao, Haohang Li, Zhiyang Deng, Yuechen Jiang, Yupeng Cao, Zhi Chen, Jordan Suchow, Zhenyu Cui, Rong Liu, et al. Fincon: A synthe- sized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making.Advances in Neural Information Processing Systems, 37:137010–137045, 2024

    2024

  14. [14]

    Stockbench: Can llm agents trade stocks profitably in real-world markets?arXiv preprint arXiv:2510.02209, 2025

    Yanxu Chen, Zijun Yao, Yantao Liu, Jin Ye, Jianing Yu, Lei Hou, and Juanzi Li. Stockbench: Can llm agents trade stocks profitably in real-world markets?arXiv preprint arXiv:2510.02209, 2025

    arXiv 2025

  15. [15]

    Can chatgpt forecast stock price movements? return predictability and large language models.arXiv preprint arXiv:2304.07619, 2023

    Alejandro Lopez-Lira and Yuehua Tang. Can chatgpt forecast stock price movements? return predictability and large language models.arXiv preprint arXiv:2304.07619, 2023

    arXiv 2023

  16. [16]

    The wall street neophyte: A zero-shot analysis of chatgpt over multimodal stock movement prediction challenges.arXiv preprint arXiv:2304.05351, 2023

    Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, and Jimin Huang. The wall street neophyte: A zero-shot analysis of chatgpt over multimodal stock movement prediction challenges.arXiv preprint arXiv:2304.05351, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Investorbench: A benchmark for financial decision-making tasks with llm-based agent

    Haohang Li, Yupeng Cao, Yangyang Yu, Shashidhar Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, Kp Subbalakshmi, Jimin Huang, et al. Investorbench: A benchmark for financial decision-making tasks with llm-based agent. InProceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages...

    2025

  18. [18]

    Time travel is cheating: Going live with deepfund for real-time fund investment benchmarking.arXiv preprint arXiv:2505.11065, 2025

    Changlun Li, Yao Shi, Chen Wang, Qiqi Duan, Runke Ruan, Weijie Huang, Haonan Long, Lijun Huang, Nan Tang, and Yuyu Luo. Time travel is cheating: Going live with deepfund for real-time fund investment benchmarking.arXiv preprint arXiv:2505.11065, 2025

    arXiv 2025

  19. [19]

    Futurex: An advanced live benchmark for llm agents in future prediction.arXiv preprint arXiv:2508.11987, 2025

    Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianci He, Yali Liao, Yixiao Tian, Jinpeng Wang, Zaiyuan Wang, Yang Yang, Lingyue Yin, et al. Futurex: An advanced live benchmark for llm agents in future prediction.arXiv preprint arXiv:2508.11987, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Prophet arena: Live llm trading competition platform, 2025

    Prophet Arena. Prophet arena: Live llm trading competition platform, 2025. URL https://www.prophetarena.co/. Accessed: 2026-01-23

    2025

  21. [21]

    Rockalpha: Llm-powered quantitative trading platform, 2025

    RockFlow AI. Rockalpha: Llm-powered quantitative trading platform, 2025. URL https://rockalpha.rockflow.ai/. Accessed: 2026-01-23

    2025

  22. [22]

    Livetradebench: Seeking real-world alpha with large language models.arXiv preprint arXiv:2511.03628, 2025

    Haofei Yu, Fenghai Li, and Jiaxuan You. Livetradebench: Seeking real-world alpha with large language models.arXiv preprint arXiv:2511.03628, 2025

    arXiv 2025

  23. [23]

    Worldquant: Quantitative research platform

    WorldQuant. Worldquant: Quantitative research platform. https://www. worldquant.com/, 2026. Accessed: 2026-02-09

    2026

  24. [24]

    Joinquant: Quantitative research platform

    JoinQuant. Joinquant: Quantitative research platform. https://www.joinquant. com/, 2026. Accessed: 2026-02-09

    2026

  25. [25]

    Qlib: An ai-oriented quantitative investment platform.arXiv preprint arXiv:2009.11189, 2020

    Xiao Yang, Weiqing Liu, Dong Zhou, Jiang Bian, and Tie-Yan Liu. Qlib: An ai-oriented quantitative investment platform.arXiv preprint arXiv:2009.11189, 2020

    Pith/arXiv arXiv 2009

  26. [26]

    Openfe: Automated feature generation with expert-level performance

    Tianping Zhang, Zheyu Aqa Zhang, Zhiyuan Fan, Haoyan Luo, Fengyuan Liu, Qian Liu, Wei Cao, and Li Jian. Openfe: Automated feature generation with expert-level performance. InInternational Conference on Machine Learning, pages 41880–41901. PMLR, 2023

    2023

  27. [27]

    Www’18 open challenge: financial opinion mining and question answering

    Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www’18 open challenge: financial opinion mining and question answering. InCompanion proceedings of the the web conference 2018, pages 1941–1942, 2018

    2018

  28. [28]

    most reasonable

    Ai trading in real markets. https://nof1.ai/, 2025. Accessed: 2026-01-14. KDD’26, August 9 to 13, 2026, Jeju, Korea Wentao Zhang et al. A Code and Data Availability All code and data will be publicly available upon acceptance of this paper, including the benchmark query set, evaluation pipeline, backtest engine, and supplementary scripts for reproducing t...

    2025

  29. [29]

    ema_{period}

    Technical Indicators • ema: Exponential Moving Average gives more weight to recent prices. When price crosses above EMA, it signals upward momentum; crossing below suggests downward trend. Formula:ema_w = EMA(close, w)| Scale: price-scale | Usage:df["ema_{period}"] • sma: Simple Moving Average calculates the arithmetic mean of prices over a period. Used t...

    2026

  30. [30]

    std_{period}

    Statistical Factors • std: Standard Deviation measures price volatility relative to current price. Higher values indicate greater price dispersion; useful for volatility-based strategies. Formula:std_w = ts_std_dev(close, w) / close| Scale:≥0 | Usage:df["std_{period}"] • vstd: Volume Standard Deviation measures volume volatility. High vstd indicates errat...

  31. [31]

    max_{period}

    Time Series Factors • max: Period High Ratio compares period maximum to current price. Values close to 1 indicate price near recent highs; higher values suggest price has fallen from highs. Formula:max_w = ts_max(close, w) / close| Scale:≥1.0 | Usage:df["max_{period}"] • min: Period Low Ratio compares period minimum to current price. Values close to 1 ind...

    2026

  32. [32]

    klow2"] •kmid2: Body Ratio (normalized by candle range). Measures body direction relative to candle range. Formula:kmid2 = (close - open) / (high - low)| Scale: -1 to 1 | Usage:df[

    Candlestick Pattern Factors •klen: Candle Body Length measures the total range of the candle. Higher values indicate larger price swings; useful for volatility assessment. Formula:klen = (high - low) / open| Scale:≥0 | Usage:df["klen"] •kup: Upper Shadow Length measures rejection from highs. Long upper shadows indicate selling pressure; often seen at resi...

  33. [33]

    vma_{period}

    Volume Factors • vma: Volume Moving Average Ratio compares average volume to current volume. Values > 1 indicate current volume below average; values < 1 indicate above average (high activity). Formula:vma_w = ts_mean(volume, w) / volume| Scale:≥0 | Usage:df["vma_{period}"] •logvol: Log Volume normalizes volume data for easier comparison. Useful for cross...

  34. [34]

    cntp_{period}

    Counting Factors •cntp: Positive Return Ratio counts the proportion of up days. High values indicate bullish momentum; low values suggest bearish sentiment. Formula:cntp_w = count(ret > 0, w) / w| Scale: 0–1 | Usage:df["cntp_{period}"] • cntn: Negative Return Ratio counts the proportion of down days. High values indicate bearish momentum; low values sugge...

    arXiv 2026