Pith. sign in

REVIEW 4 major objections 5 minor 5 references

Predictive Power of LLMs in Financial Markets

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompted GPT-3.5 can extract stock-bond correlation signals from Federal Reserve Beige Book text, but its apparent edge is look-ahead bias and a fine-tuned BERT model performs better out of sample.

desk verdict Fine-tuned vs zero-shot confound undercuts the headline 'traditional models triumph' claim, but the Beige Book correlation task and look-ahead bias test are worth engaging. read the letter →

arxiv 2411.16569 v1 pith:P22ANSHW submitted 2024-11-25 q-fin.PM q-fin.CP

Jerick Shi , Burton Hollifield This is my paper · ORCID
classification q-fin.PMq-fin.CP
keywords LargeLanguageModelsGPT-3.5BERTBeigeBookstock-bondcorrelationlook-aheadbiasportfoliooptimizationSharperatio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a large language model can turn Federal Reserve Beige Book text into useful predictions of how stock and bond returns will co-move. It compares prompted GPT-3.5 with a BERT model fine-tuned to classify the same articles into correlation categories, then feeds the predicted correlations into minimum-variance portfolios. The paper concludes that the Beige Book does contain information about asset correlations, but GPT-3.5 carries too much look-ahead bias for its predictions to be trusted out of sample, and the traditional BERT model performs better. The practical upshot for a reader is that generative prompting alone is not a reliable way to harvest economic text for portfolio decisions, and a simpler supervised classifier can beat it.

What carries the argument

The load-bearing objects are two prediction pipelines over the same Federal Reserve Beige Book articles: a prompted GPT-3.5 with temperature zero, asked to return a coarse correlation category, and a BERT model fine-tuned as a classifier on the actual historical correlation labels. The argument uses RMSE computed over non-overlapping n-month blocks, with the pretraining cutoff defining the train/test boundary, and then feeds the predicted correlations into minimum-variance portfolio optimization, using a closed-form two-asset weight and a Lagrangian multi-asset version. The train/test split is what isolates look-ahead bias, and the portfolio simulation is what turns correlation forecasts into an economic comparison.

What would settle it

Run the same prompts on a GPT model with a pretraining cutoff no later than 2019 and compare RMSE on months after that cutoff; if the test-period error no longer exceeds the training-period error, the look-ahead-bias story would be weakened. Alternatively, rewrite the Beige Book articles with dates shifted by one or two years and ask GPT-3.5 for correlations; if the outputs barely change, the model is responding to textual content rather than to memorized calendar outcomes.

Watch

Extended reading notes

Core claim

The paper's central claim is that GPT-3.5's apparent skill at predicting stock-bond correlations from Beige Book text is substantially an artifact of look-ahead bias, not genuine forecasting ability. In the paper's design, the model is prompted as a financial agent with no knowledge after the article date, but because GPT-3.5 was pretrained through September 2021, the entire 1980-2021 training period is visible to it. When the same prompt is evaluated on 2021-10 through 2024-06, errors rise significantly, especially in the binned-response version, while a BERT model trained as a classifier on actual correlation labels holds up better out of sample. In portfolio simulations, BERT-based weights yield higher Sharpe ratios than GPT-based weights in both two-asset and multi-asset settings. The authors summarize it as: the Beige Book contains correlation-relevant information, yet the GPT model has too much look-ahead bias and traditional models still triumph.

Load-bearing premise

The central claim depends on GPT-3.5's September 2021 pretraining cutoff being a clean information boundary, so that a lower in-sample error is evidence of look-ahead bias rather than of a genuinely better model.

Editorial extensions

If this is right

  • If the paper is right, a GPT-style model that has seen the test period during pretraining cannot serve as a clean out-of-sample forecaster for financial text, even when the prompt instructs it to ignore future knowledge.
  • Prompting GPT with recent historical correlations does not help: in the original three-choice version the errors got worse, which suggests noisy numerical context can degrade the text signal.
  • A supervised BERT classifier trained on actual outcomes is the better way to extract correlation information from the Beige Book, and the resulting portfolios have higher Sharpe ratios than GPT-driven or rolling-average portfolios.
  • The Beige Book text itself carries information about stock-bond comovement; the failure is in the extraction method, not the data source.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to retest with a model whose pretraining cutoff lies well before the evaluation window; if the out-of-sample gap vanishes, the paper's main evidence for look-ahead bias would be sharpened.
  • The comparison is tilted by design: BERT is fine-tuned on labels from the same period, while GPT-3.5 is used zero-shot, so the result says more about prompting versus supervised fine-tuning than about the architectures themselves.
  • The same template could be applied to other scheduled economic releases, such as FOMC statements or jobs reports, where text timing is clean, to see whether classification beats generative prompting for other macro-financial quantities.
  • The paper implicitly shows that the sign and coarse bin of the correlation matter more than exact magnitude for minimum-variance allocation; a test using continuous correlation targets could reveal whether GPT's extra granularity is wasted.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript asks whether GPT-3.5 Turbo can predict monthly stock-bond correlations from Federal Reserve Beige Book text, compares it with a fine-tuned BERT model and with a rolling-covariance baseline, and evaluates the resulting minimum-variance portfolios via Sharpe ratios. The authors report evidence of 'look-ahead bias' in GPT (lower in-sample RMSE than out-of-sample, especially in the binned version), no benefit from feeding historical correlations into the prompt, and superior out-of-sample RMSE and portfolio Sharpe ratios for BERT over GPT. They conclude that the Beige Book contains correlation-relevant information but that GPT has too much look-ahead bias and that 'traditional models still triumph.'

Significance. If established, the result would be a useful cautionary data point for using pretrained LLMs in financial forecasting, and the Beige Book correlation-prediction task is a reasonable, relatively clean test bed. The authors deserve credit for attempting an explicit hypothesis-testing framework, for using out-of-sample evaluation for the BERT model, and for recognizing the look-ahead-bias issue. However, the central comparisons are confounded by the training-protocol difference between GPT (zero-shot) and BERT (fine-tuned on labels), the look-ahead-bias test conflates memorization with time-period effects, and the Sharpe-ratio comparisons lack any inferential support. The paper is therefore a suggestive preliminary study rather than a definitive test of the claims in the abstract.

major comments (4)
  1. [Sections II.B, II.C, and Tables V-VIII] The comparison between GPT and BERT is confounded by training protocol. GPT-3.5 is used zero-shot with a prompt, while BERT is fine-tuned on 1980-2021 labels derived from the actual stock-bond correlations. The out-of-sample tables showing lower RMSE for BERT (Tables VII-VIII and the simulation results in Section V) therefore do not identify whether the BERT architecture or the supervised fine-tuning drives the improvement. The abstract's claim that 'traditional models still triumph' over GPT is not supported unless GPT is also fine-tuned on the same labels or the claim is explicitly restricted to zero-shot GPT prompting.
  2. [Section IV.A (Look-Ahead Bias)] The look-ahead-bias test compares RMSE on 1980-01 to 2021-09 ('training set') with RMSE on 2021-10 to 2024-06 ('testing set'). Because GPT-3.5's pretraining data include financial and web text up to September 2021, the historical period is in-distribution for the model, and its lower in-sample RMSE is better characterized as memorization of past outcomes than as look-ahead bias in the forecasting sense. The test also cannot separate a cutoff effect from a market-regime change between the two periods, and the t-test on monthly RMSE values ignores autocorrelation in correlation forecast errors; for n=12 the test set yields only two non-overlapping blocks, so the test is severely underpowered. The conclusion of 'unavoidable look-ahead bias' is therefore not established by the reported evidence.
  3. [Section V, Tables IX-X] The Sharpe ratios are point estimates with no standard errors, confidence intervals, or hypothesis tests. The differences reported (e.g., 2.605 vs 2.768 vs 2.334 in Table IX; 0.423 vs 1.9208 vs 0.445 in Table X) could easily be within sampling variation given the small post-Covid sample and the absence of any significance testing. The statements that 'BERT tends to perform the best' and that GPT 'does even worse after Covid' are not statistically substantiated. The authors should report bootstrap/t-test results or at least the number of rebalancing periods used.
  4. [Section V.A (Choice of V3)] The simulation section selects 'BERT: Original Correlations V3' and 'GPT: Original Correlations V3' because 'V3 seems to produce a more accurate correlation than both V1 and V2.' If this selection was made using the same test period used in the simulations, the portfolio results inherit a selection bias. The paper must state on which sample (training set only) V3 was chosen, or present results for all three versions to demonstrate robustness.
minor comments (5)
  1. [Tables V-VIII] Table VI is captioned 'Training with Bins' but the text states that out-of-sample results are shown in Tables VI and VIII; the entries in Table VI (all 1.000) appear to refer to the test set, not the training set. This labeling error makes the results difficult to interpret.
  2. [Section II.B] The prompt includes both 'You don't know anything that happens during this year or afterwards' and the year of the article. Since the Beige Book article is about conditions during that month, it would be clearer to state explicitly whether the requested correlation is for the same month (a nowcast) or for a subsequent month, because the look-ahead-bias interpretation depends on this timing.
  3. [Section IV.A (Equation for RMSE)] The RMSE formula is missing parentheses and the index range; the equation should read RMSE = sqrt((1/n) * sum_{i=1}^n (y_i - yhat_i)^2).
  4. [Section III.B (Simulation, Algorithm 2)] The algorithm refers to 'w_I' (presumably a risk-free or third asset weight) without defining how it is chosen or whether it is part of the two-asset minimization; the notation should be clarified.
  5. [General] The paper does not provide code, data availability, or the exact GPT/BERT model identifiers (e.g., gpt-3.5-turbo version date and BERT checkpoint), which limits reproducibility; at least the exact model versions and prompt templates should be included in an appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: predictions are evaluated against realized market data and holdout periods, with no fitted parameter renamed as a prediction.

full rationale

The paper's derivation chain is not circular. GPT-3.5 is prompted zero-shot with Beige Book articles and past correlations, and its outputs are compared to realized stock-bond correlations; the prompt is not calibrated to the labels, and the temperature-zero sampling does not fit any parameter to the target variable. The look-ahead bias test compares GPT's in-sample RMSE (1980-01 to 2021-09) with out-of-sample RMSE (2021-10 to 2024-06), relying on the model's stated September 2021 pretraining cutoff; even if that cutoff is an imperfect information boundary, the test is an empirical comparison of errors, not a definitional equivalence. BERT is fine-tuned on labels derived from actual correlations up to September 2021 and then evaluated on a strictly later period, which is standard supervised training with a temporal holdout, not a fitted input called a prediction. The portfolio simulations use genuinely realized price data and compare strategies whose inputs include Beige-Book-derived correlations against rolling-average baselines. The GPT-versus-BERT comparison is confounded by training protocol (fine-tuned BERT versus zero-shot GPT), and the 'traditional models still triumph' headline is therefore not fully established, but that is a methodological fairness concern, not circularity. There are no load-bearing self-citations, no uniqueness theorems imported from the authors' prior work, and no renaming of a known result. Accordingly, the circularity score is 0.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The empirical claims rest on several modeling choices: the definition of true monthly correlation, the use of GPT-3.5's knowledge cutoff as a clean training/test boundary, the BERT training protocol, and the prompt/scaling design. No new physical or theoretical entities are introduced; the main load-bearing choices are the 'true correlation' definition and the clean-cutoff assumption for look-ahead testing. The free parameters listed are the ones that were fitted or chosen by hand and that the conclusions depend on.

free parameters (6)
  • Bond regression variable subset = corporate, 1y, 2y, 5y, 7y, 10y; R2=0.9901
    Algorithm 1 searches all combinations of treasury and corporate return series to predict AGG, selecting the best out-of-sample R2; this reconstructed bond price series is used to compute historical monthly correlations.
  • BERT fine-tuning configuration = 10 epochs, 3 or 11 classes
    The BERT model is trained on actual rounded correlations from 1980 to 2021-09, with the exact optimizer, learning rate, and validation split unspecified; its outputs define the BERT condition.
  • Prompt version V3 for simulations = Original correlations, scaled strength p-(1-p)
    Section V.A selects V3 because it 'seems to produce a more accurate correlation', a post hoc choice based on the same period evaluated; this can inflate the reported portfolio results.
  • Number of past correlations in the prompt = 3 months
    The prompt 'previous {numcorrelations} month's correlations' is set to 3 (see Algorithm 2, rho_month <- askGPT(beige data[month], rho_-3::-1)), an arbitrary choice not varied in the main tests.
  • Exponential moving average window = not specified
    Algorithm 2 uses a rolling average for sigma_S and sigma_B and a baseline of exponential moving covariances, but no window length is stated; the baseline comparison depends on this choice.
  • RMSE block length n = 1, 3, 6, 12 months
    Hypothesis tests are repeated for four block sizes; the look-ahead significance appears only at some n values, so the conclusions depend on which n is emphasized.
assumptions (6)
  • domain assumption The true monthly stock-bond correlation is the Pearson correlation of daily percentage changes within the calendar month.
    Section II.B defines this as the 'true' correlation with no justification for the monthly window or return measure.
  • ad hoc to paper GPT-3.5's pretraining cutoff at September 2021 cleanly separates information it can and cannot access.
    Section IV.A splits training and test at this cutoff; the paper does not test for other leakage channels.
  • domain assumption Temperature=0 makes GPT-3.5 outputs deterministic enough for replication.
    Section II.B relies on this for replicability, though closed-source APIs can still introduce nondeterminism.
  • ad hoc to paper The scaled strength transformation p-(1-p) is a valid way to weight correlation predictions.
    Section II.B introduces this transformation without empirical or theoretical justification; results vary across the three strength versions.
  • domain assumption The selected Beige Book articles for a month are contemporaneous with the month whose correlation is predicted.
    Section II.A describes scraping articles by month and state but never aligns publication dates with return months, which is essential for causal prediction.
  • standard math The one-sided t-test on non-overlapping RMSE blocks satisfies independence and distributional assumptions.
    Section IV uses t-tests on RMSE blocks; the normality and variance assumptions are not checked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Predictive Power of LLMs in Financial Markets." pith.science (2026). https://pith.science/paper/P22ANSHW

@misc{pith2026241116569,
  author       = {Pith},
  title        = {Pith review of: Predictive Power of LLMs in Financial Markets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P22ANSHW}},
  note         = {Machine review of arXiv:2411.16569}
}
read the original abstract

Predicting the movement of the stock market and other assets has been valuable over the past few decades. Knowing how the value of a certain sector market may move in the future provides much information for investors, as they use that information to develop strategies to maximize profit or minimize risk. However, market data are quite noisy, and it is challenging to choose the right data or the right model to create such predictions. With the rise of large language models, there are ways to analyze certain data much more efficiently than before. Our goal is to determine whether the GPT model provides more useful information compared to other traditional transformer models, such as the BERT model. We shall use data from the Federal Reserve Beige Book, which provides summaries of economic conditions in different districts in the US. Using such data, we then employ the LLM's to make predictions on the correlations. Using these correlations, we then compare the results with well-known strategies and determine whether knowing the economic conditions improves investment decisions. We conclude that the Beige Book does contain information regarding correlations amongst different assets, yet the GPT model has too much look-ahead bias and that traditional models still triumph.

Figures

Figures reproduced from arXiv: 2411.16569 by the authors.

Figure 1
Figure 1. Movement of Stocks over Time to capture insignificant information, making it difficult to generalize for future times. For example, if you were to predict how the market moves in the early twentyfirst century, it is highly unlikely that the model will be able to predict that Covid would happen, in which stocks plummeted. Hence, even though there are several models that performed well in previous years, few models pe… view at source ↗
Figure 2
Figure 2. Error Distributions of Predicted Prices We observe that since the errors are all quite small, so our model should produce relatively accurate prices for AGG. We also show in figure 3 predictions for earlier AGG prices (blue being our prediction, orange being the actual prices) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Predicted Prices (Blue) of Bonds [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Example Prompt for GPT Another way we shall ask for it is in bins, where, rather than returning numbers between 0 and 2, we can ask for numbers between 0 and 10 to be more specific. With each of those answers, the model also outputs a probability p, with respect to the…
Figure 6
Figure 6. Figure 6: 2-Variable Portfolio Values post-Covid We notice that in both periods, the BERT model tends to perform the best, whereas the GPT model sometimes even performs worse than simple rolling covariances. In this case, it seems that using the BERT model to analyze the Beige B…
Figure 7
Figure 7. Figure 7: Multi-Variable Portfolio Values pre-Covid [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Multi-Variable Portfolio Values post-Covid [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Errors for out-of-sample (Original) [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Errors for Adding History (Original) [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 13
Figure 13. Figure 13: Training Errors for BERT vs GPT (Original) [PITH_FULL_IMAGE:figures/full_fig_p008_13.png]
Figure 15
Figure 15. Figure 15: Testing Errors for BERT vs GPT (Original) [PITH_FULL_IMAGE:figures/full_fig_p009_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 4 canonical work pages

  1. [1]

    Kalyani, J., Bharathi, Prof. H. N., Jyothi, Prof. R. (2016, July 7). Stock trend prediction using news sentiment analysis. arXiv.org. https://arxiv.org/abs/1607.01958

    arXiv 2016

  2. [2]

    (2020, July 27)

    Author links open overlay panelYinghao Ren a, a, Abstract With the in-depth study of the stock market. (2020, July 27). Impact of news on the trend of stock price change: An analysis based on the deep bidirectiona LSTM model. Procedia Computer Science. https://www.sciencedirect.com/science/article/pii/S1877050920315829

    work page 2020

  3. [3]

    (2023, November)

    Bybee, L. (2023, November). The Ghost in the Machine: Generating Beliefs with Large Language Models

    work page 2023

  4. [4]

    Federal Reserve Board - Beige Book

    Beige Book. Federal Reserve Board - Beige Book. (n.d.). https://www.federalreserve.gov/monetarypolicy/publications/beige-book-default.htm

    work page

  5. [5]

    Ԅ; ˏ 9"j8 6 tzyy l6@ kp' 7l6 d2X Q q Gr P m6o&EQo>qD0 dY ٗ_ ڵk (Y ` ^ݻw ޽ xSSS:j|Ǐ|YUnKh0 `` 歷2 ap8 4M˲o 9r xUUj H / Q eCP4 T < _ \ Ȋ p8l0 X t:L&F oQw 7 h a&6c(ʲl27 D

    11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

    work page

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.