Pith. sign in

REVIEW 3 major objections 4 minor 25 references

Sentiment trading with large language models

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sentiment scores from the GPT-3-style OPT model predict next-day stock returns well enough that a long-short portfolio built on them earns a Sharpe ratio of 3.05.

desk verdict The paper's headline Sharpe ratio and regression coefficients are artifacts of a lookahead design: sentiment labels include the next-day return, and the backtest runs on the full sample. read the letter →

arxiv 2412.19245 v1 pith:PS7K46XD submitted 2024-12-26 q-fin.CP cs.LGecon.EMq-fin.PMq-fin.TR

Kemal Kirtac , Guido Germano This is my paper · ORCID
classification q-fin.CPcs.LGecon.EMq-fin.PMq-fin.TR
keywords largelanguagemodelssentimentanalysisstockreturnpredictionOPTBERTFinLoughran-McDonalddictionarylong-shortportfolio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that large language models trained on general text extract more tradeable information from U.S. financial news than the dictionary-based sentiment scores that dominate empirical finance. It fine-tunes OPT and BERT on news articles labelled by the sign of each stock's three-day excess return, and compares them with FinBERT and the Loughran-McDonald dictionary. The central result is that OPT scores carry a statistically strong positive association with next-day stock returns, while dictionary scores do not, and that a value-weighted long-short portfolio built on OPT scores earns a Sharpe ratio of 3.05 with a 355% cumulative gain from August 2021 to July 2023. If true, this would shift text-based return prediction away from lexicons toward large pre-trained models.

What carries the argument

The load-bearing object is the vector of per-article sentiment scores $\mathbf{x}_{i,n}$ produced by each model and entered into the panel regression $r_{i,n+1} = a_i + b_n + \boldsymbol{\gamma}\cdot\mathbf{x}_{i,n} + \epsilon_{i,n}$, where $r_{i,n+1}$ is the next-day return, and $a_i$, $b_n$ are firm and date fixed effects. Labels for fine-tuning are the sign of the three-day cumulative excess return from publication day onward, following event-study practice. For the portfolio exercise, long (short) portfolios hold the top (bottom) 20% of stocks by daily sentiment score, are value-weighted, rebalanced daily with a 10 basis point transaction cost, and trades are timed to news release. These connected pieces — model score, regression equation, labelling rule, and portfolio rule — carry the argument from text to profit.

What would settle it

Re-run the analysis with a strict chronological split — train on articles before a cutoff date, test only on later articles, and define labels using returns that end before the article's publication — and recompute the regression coefficient and the long-short Sharpe ratio. If the OPT coefficient falls to near zero or the Sharpe ratio drops to the market level, the reported predictive edge would be shown to come from the overlap between the label window and the forecast target rather than from genuine news sentiment.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the sentiment scores produced by the GPT-3-style OPT model are a robust predictor of the next trading day's stock return. In panel regressions with firm and date fixed effects, the coefficient on the OPT score is 0.274 (t = 5.37) alone and 0.254 with other model scores included, whereas the Loughran-McDonald coefficient is 0.083 and statistically insignificant. The same ordering appears in classification accuracy on the 20% test set: OPT 74.4%, BERT 72.5%, FinBERT 72.2%, dictionary 50.1%. The paper concludes that general-purpose LLMs, and OPT in particular, outperform the dictionary approach for both directional forecasting and portfolio construction, with the long-short OPT strategy delivering the highest risk-adjusted return among the models tested.

Load-bearing premise

The claim rests on the assumption that a randomly chosen 80/20 train-test split, with news labels defined from the three-day excess return including the next day, gives an unbiased out-of-sample measure of return prediction, and that the portfolio backtest does not use articles that also appeared in training.

Editorial extensions

If this is right

  • If OPT scores genuinely predict next-day returns, dictionary-based sentiment indices in finance should be re-examined; the paper's comparison suggests they capture almost none of the signal in contemporary news.
  • A daily rebalanced, value-weighted long-short portfolio built on OPT scores would have earned 355% over August 2021 to July 2023 net of 10 bps per trade, versus roughly 1% for value- and equal-weighted market benchmarks.
  • The predictive content is strongest in the larger, general-purpose model (OPT) rather than the finance-specific FinBERT, implying that broad pre-training matters for financial text.
  • Regulators and asset managers, the paper argues, need to account for the possibility that LLM-based signals concentrate informational advantage in those with access to such models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's evaluation design is the part I would test first: because sentiment labels are the sign of a three-day return that starts on publication day, and the regression target is the next-day return, the label window overlaps the target window; a strict temporal split is needed to confirm the coefficients are not lookahead artifacts.
  • The 355% figure covers a specific two-year window; a natural extension is to ask how the strategy behaves in a longer out-of-sample period and under larger, more realistic transaction costs and short-sale constraints.
  • If the result survives a temporal split, it suggests a broader research program: probing whether other causal signals in news text, not just sentiment polarity, can be extracted by LLMs for return prediction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper evaluates the ability of large language models (OPT, BERT, FinBERT) and the Loughran-McDonald dictionary to predict U.S. stock returns from financial news. Sentiment labels are derived from the sign of the 3-day excess return following each article, and the models are fine-tuned on these labels. The authors report high classification accuracy for OPT (74.4%), significant positive regression coefficients of OPT scores on next-day returns (0.274 and 0.254 in Table 4), and a long-short portfolio with a Sharpe ratio of 3.05 and a 355% gain in the August 2021–July 2023 period. The paper claims that LLM-based sentiment scores substantially outperform the dictionary approach.

Significance. The question of whether LLMs extract tradeable signals from financial news is timely and important. The dataset is large (965,375 articles) and the comparison across several models, including a dictionary benchmark, is a useful design. If the predictive claims were valid, the paper would provide relevant evidence on the value of modern language models for return prediction. However, the core evidence is compromised by a serious lookahead in the label construction and by an evaluation design that does not ensure out-of-sample inference. The strengths of the paper do not outweigh the validity problems in its central claims.

major comments (3)
  1. [Section 2.2 and Eq. (1)] The sentiment labels are defined as the sign of the 3-day excess return from the article date n through day n+2, and the regression in Eq. (1) uses the next-day return r_{i,n+1} as the dependent variable. Because the fine-tuning label includes r_{i,n+1}, the OPT/BERT/FinBERT scores x_{i,n} contain information about the very return they are later used to predict. The coefficients in Table 4 are therefore partly mechanical and do not demonstrate predictive ability; they reflect that the model was trained to forecast a label that embeds the target variable.
  2. [Section 2.2 and Table 4] The regression reported in Table 4 uses all 965,375 observations, i.e., the full sample, not the held-out 20% test set. Combined with the random (non-chronological) 80/20 split described in Section 2.2, this means the regression is an in-sample fit that includes articles on which the models were fine-tuned. The reported t-statistics and R² values are thus not out-of-sample evidence, and the comparison against the Loughran-McDonald dictionary is not a test of predictive accuracy.
  3. [Section 3.3 and Table 5] The portfolio backtest period (August 2021 to July 2023) overlaps with the randomly assigned training and validation data. The paper does not state that the backtest is restricted to the 20% test set, so the long-short OPT strategy's Sharpe ratio of 3.05 and the 355% gain in Fig. 1 likely include positions based on articles used in training. This is a second source of lookahead bias that invalidates the portfolio performance claims as out-of-sample results.
minor comments (4)
  1. [Table 5] The table contains typographical irregularities such as '1 .42', '0 .25', and '3 .05'; these appear to be missing spaces or alignment issues and should be corrected.
  2. [Table 1 note] The note states that articles are excluded if too similar to content published within the 'prior five business days', while Section 2.1 states the similarity filter uses a 20-day window; these two descriptions are inconsistent.
  3. [Section 3.1] The term 'laggs' in Section 3.3 ('The Loughran-McDonald dictionary model-based strategy (L-S LM dictionary) laggs behind') is a typo for 'lags'.
  4. [Table 4] The AIC and BIC values for Regression 6 (AIC = 135,783, BIC = 123,382) violate the standard inequality BIC > AIC for a model with more than one parameter, suggesting a reporting error or an unusual calculation that should be checked.

Circularity Check

2 steps flagged · score 8.0 of 10

Training labels embed the predicted return, and a random split makes the backtest in-sample, so the headline Sharpe ratio and return coefficients are partly mechanical.

  1. self definitional [Section 2.2 (Methods), Eq. (1); Section 3.2, Table 4]
    "Sentiment labels are assigned to each news article based on the sign of this aggregated three-day excess return. ... The regression is modelled as 𝑟𝑖,𝑛+1 =𝑎𝑖 +𝑏𝑛 + 𝜸 ⋅ 𝐱𝑖,𝑛 +𝜖𝑖,𝑛, (1) where𝑟𝑖,𝑛+1 is the return of stock 𝑖 on the subsequent trading day 𝑛 + 1, 𝐱𝑖,𝑛 is a vector of scores from language models."

    By construction, the fine-tuning label is the sign of the three-day excess return starting on the article day n, which includes the next-day return r_{i,n+1}. The LLM is trained to make x_{i,n} predict that label, so x_{i,n} encodes information about r_{i,n+1}. Eq. (1) then regresses r_{i,n+1} on x_{i,n}, recovering a relationship that the training procedure inserted rather than an independent news-to-return association. This mechanically inflates the OPT coefficients 0.274 and 0.254 reported in Table 4.

  2. fitted input called prediction [Section 2.2 (random split); Section 3.3, Fig. 1]
    "We allocated 20% of the data randomly for testing and, from the remaining data pool, allocated another 20% randomly for validation purposes, resulting in a training set of 193,070 articles. ... Finally, we examine the outcomes of trading strategies based on news sentiment including a 10 bps trading cost from August 2021 to July 2023. Fig. 1 illustrates the performance of various strategies, notably highlighting the long-short OPT strategy with an impressive 355% gain."

    The 80/20 split is random, not chronological, so with 965,375 articles the training set almost certainly contains articles from August 2021 to July 2023, the very window used for the backtest. The OPT sentiment scores driving the long-short portfolios therefore come from a model fine-tuned on labels from the evaluation period, making the 355% gain and the Sharpe ratio of 3.05 largely in-sample rather than out-of-sample predictions.

full rationale

The paper's headline quantities — the OPT coefficients 0.274/0.254 in Table 4 and the long-short OPT Sharpe ratio of 3.05 / 355% gain — are not derived from an independent out-of-sample exercise. Section 2.2 defines the sentiment label as the sign of the aggregated three-day excess return from the article day n through n+2, and Eq. (1) uses r_{i,n+1} as the dependent variable. Fine-tuning OPT to predict that label makes x_{i,n} carry information about r_{i,n+1}; the regression of r_{i,n+1} on x_{i,n} is therefore partly a mechanical restatement of the training target. In parallel, the 80/20 random split, rather than a chronological split, means the training set includes articles from the August 2021-July 2023 backtest window, so the portfolio results are largely in-sample. Both leaks are visible in the paper's own text and equations, not in external assumptions; no self-citation or imported uniqueness theorem is needed to establish the reduction. The accuracy comparison in Table 3 is a more conventional label-prediction benchmark, but it does not rescue the return-prediction and trading claims. Score 8 reflects that the central results reduce, by the paper's own design, to quantities that contain their own targets.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The work rests on assumptions about sentiment labels, split validity, return alignment, and implementability. The most important are the label window and the random split, both of which are violated in ways that inflate the reported predictive performance.

free parameters (4)
  • Portfolio quantile threshold = Top and bottom 20%
    Stocks in the top and bottom 20% of sentiment scores define long and short portfolios; the Sharpe ratios in Table 5 depend directly on this cutoff.
  • Transaction cost = 10 bps per trade
    A fixed cost assumption used in the backtest; net returns and the 355% gain change with this value.
  • Sentiment labeling window = 3 trading days
    Labels are the sign of the aggregated 3-day excess return after an article; this window overlaps with the next-day return in Eq. (1).
  • Deduplication thresholds = Cosine similarity 0.8, 20-day horizon
    These filters shrink the sample from 1,865,372 to 965,375 articles and affect all downstream metrics.
assumptions (5)
  • domain assumption The sign of the 3-day excess return is a valid ground-truth label for news sentiment.
    Section 2.2 assigns sentiment labels from future returns; the accuracy metrics in Table 3 inherit this assumption.
  • domain assumption A random 80/20 split prevents lookahead in the return prediction and backtest.
    The predictive claim and the Sharpe ratios require this; the split is random rather than chronological and the label window overlaps the predicted return.
  • domain assumption Each news article maps cleanly to the next trading day return for stock i.
    Eq. (1) pairs score x_i,n with r_i,n+1; timing details are only partially specified.
  • standard math Linear panel regression with firm and date fixed effects and double-clustered errors yields unbiased coefficients on sentiment scores.
    Standard econometric assumptions are invoked without diagnostics.
  • domain assumption The value-weighted portfolios can be traded daily at 10 bps cost.
    Section 2.2 assumes immediate execution relative to news release times; capacity and price impact are not modeled.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sentiment trading with large language models." pith.science (2026). https://pith.science/paper/PS7K46XD

@misc{pith2026241219245,
  author       = {Pith},
  title        = {Pith review of: Sentiment trading with large language models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PS7K46XD}},
  note         = {Machine review of arXiv:2412.19245}
}
read the original abstract

We investigate the efficacy of large language models (LLMs) in sentiment analysis of U.S. financial news and their potential in predicting stock market returns. We analyze a dataset comprising 965,375 news articles that span from January 1, 2010, to June 30, 2023; we focus on the performance of various LLMs, including BERT, OPT, FINBERT, and the traditional Loughran-McDonald dictionary model, which has been a dominant methodology in the finance literature. The study documents a significant association between LLM scores and subsequent daily stock returns. Specifically, OPT, which is a GPT-3 based LLM, shows the highest accuracy in sentiment prediction with an accuracy of 74.4%, slightly ahead of BERT (72.5%) and FINBERT (72.2%). In contrast, the Loughran-McDonald dictionary model demonstrates considerably lower effectiveness with only 50.1% accuracy. Regression analyses highlight a robust positive impact of OPT model scores on next-day stock returns, with coefficients of 0.274 and 0.254 in different model specifications. BERT and FINBERT also exhibit predictive relevance, though to a lesser extent. Notably, we do not observe a significant relationship between the Loughran-McDonald dictionary model scores and stock returns, challenging the efficacy of this traditional method in the current financial context. In portfolio performance, the long-short OPT strategy excels with a Sharpe ratio of 3.05, compared to 2.11 for BERT and 2.07 for FINBERT long-short strategies. Strategies based on the Loughran-McDonald dictionary yield the lowest Sharpe ratio of 1.23. Our findings emphasize the superior performance of advanced LLMs, especially OPT, in financial market prediction and portfolio management, marking a significant shift in the landscape of financial analysis tools with implications to financial regulation and policy analysis.

Figures

Figures reproduced from arXiv: 2412.19245 by the authors.

Figure 1
Figure 1. Cumulative returns from investing $1 with value-weighted, zero-cost long-short portfolios based on OPT (red), BERT (yellow), FinBERT (dark blue) and the Loughran-McDonald dictionary (green), rebalanced daily with a 10 bps transaction cost. For comparison, we also show a value-weighted market portfolio (light blue) and an equal-weighted market portfolio (orange), both without transaction costs. Finally, we examine th… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 5 canonical work pages

  1. [3]

    Measuring economic policy uncertainty. Q. J. Econ. 131 (4), 1593–1636. http://dx.doi.org/10.1093/qje/qjw024. Baker, M., Wurgler, J.,

    doi:10.1093/qje/qjw024

  2. [6]

    How news and its context drive risk and returns around the world. J. Financ. Econ. 133 (2), 299–336. http: //dx.doi.org/10.1016/j.jfineco.2018.11.009. Campbell, J.L., Chen, H., Dhaliwal, D.S., Lu, H.-m., Steele, L.B.,

    doi:10.1016/j.jfineco.2018.11.009 2018

  3. [9]

    In: Burstein, J., Doran, C., Solorio, T

    BERT: Pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minne...

    doi:10.18653/v1/n19-1423 2019

  4. [11]

    dictionary methods

    Disclosure sentiment: Machine learning vs. dictionary methods. Manage. Sci. 68 (7), 5514–5532. http://dx.doi.org/10. 1287/mnsc.2021.4156. Garcia, D.,

    arXiv 2021

  5. [15]

    Word power: A new approach for content analysis. J. Financ. Econ. 110 (3), 712–729. http://dx.doi.org/10.1016/j.jfineco.2013. 08.018. Ke, Z., Kelly, B.T., Xiu, D.,

    work page doi:10.1016/j.jfineco.2013 2013

  6. [17]

    The impact of investor sentiment on the market’s reaction to stock splits. Rev. Financ. Stud. 27 (5), 1367–1401. http: //dx.doi.org/10.1093/rfs/hhu006. Loughran, T., Mcdonald, B.,

    work page doi:10.1093/rfs/hhu006

  7. [19]

    Event studies in economics and finance. J. Econ. Lit. 35 (1), 13–39, URL: http://www.jstor.org/stable/2729691. Malo, P., Sinha, A., Korhonen, P., Wallenius, J., Takala, P.,

    arXiv

  8. [20]

    Good debt or bad debt: Detecting semantic orientations in economic texts. J. Assoc. Inf. Sci. Technol. 65 (4), 782–796. http://dx.doi.org/10.1002/asi.23062. Finance Research Letters 62 (2024) 105227 9 K. Kirtac and G. Germano Manela, A., Moreira, A.,

    doi:10.1002/asi.23062 2024

Show all 25 references
  1. [22]

    Measuring news sentiment. J. Econometrics 228 (2), 221–243. http://dx.doi.org/10.1016/j.jeconom.2020.07.053. Tetlock, P.C.,

    2020 doi

  2. [24]

    In: Liu, Q., Schlangen, D

    Transformers: State-of-the-art natural language processing. In: Liu, Q., Schlangen, D. (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Minneapolis, Minnesota, pp. ...

    2020 doi

  3. [25]

    arXiv:2205.01068

    OPT: Open pre-trained transformer language models. arXiv:2205.01068. http: //dx.doi.org/10.48550/arXiv.2205.01068

    doi arXiv

  4. [1993]

    Common risk factors in the returns on stocks and bonds. J. Financ. Econ. 33 (1), 3–56. http://dx.doi.org/10.1016/0304- 405X(93)90023-5. Frankel, R., Jennings, J., Lee, J.,

    doi

  5. [1997]

    On persistence in mutual fund performance. J. Finance 52 (1), 57–82. http://dx.doi.org/10.1111/j.1540-6261.1997.tb03808.x. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.,

    1997

  6. [2006]

    Investor sentiment and the cross-section of stock returns. J. Finance 61 (4), 1645–1680. http://dx.doi.org/10.1111/j.1540- 6261.2006.00885.x. Bybee, L., Kelly, B.T., Manela, A., Xiu, D.,

    2006

  7. [2007]

    Giving content to investor sentiment: The role of media in the stock market. J. Finance 62 (3), 1139–1168. http://dx.doi.org/10.1111/J.1540- 6261.2007.01232.X. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., D...

    2007

  8. [2011]

    When is a liability not a liability? Textual analysis, dictionaries, and 10-ks. J. Finance 66 (1), 35–65. http://dx.doi.org/10. 1111/j.1540-6261.2010.01625.x. Loughran, T., McDonald, B.,

    2010

  9. [2013]

    Sentiment during recessions. J. Finance 68 (3), 1267–1300. http://dx.doi.org/10.1111/jofi.12027. Hansen, S., McMahon, M., Prat, A.,

    doi

  10. [2014]

    The information content of mandatory risk factor disclosures in corporate filings. Rev. Account. Stud. 19 (1), 396–455. http://dx.doi.org/10.1007/s11142-013-9258-3. Carhart, M.M.,

    doi

  11. [2016]

    arXiv:1610.01644

    Understanding intermediate layers using linear classifier probes. arXiv:1610.01644. http://dx.doi.org/10.48550/arXiv.1610.01644. Baker, S.R., Bloom, N., Davis, S.J.,

    doi arXiv

  12. [2017]

    News implied volatility and disaster concerns. J. Financ. Econ. 123 (1), 137–162. http://dx.doi.org/10.1016/j.jfineco.2016.01.032. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.,

    2016 doi

  13. [2018]

    Transparency and deliberation within the FOMC: A computational linguistics approach. Q. J. Econ. 133 (2), 801–870. http://dx.doi.org/10.1093/qje/qjx045. Hoberg, G., Phillips, G.,

    doi

  14. [2019]

    SSRN 3446225

    The structure of economic news. SSRN 3446225. http://dx.doi.org/10.2139/ssrn.3446225. Calomiris, C.W., Mamaysky, H.,

    doi

  15. [2020]

    SSRN 3389884

    Predicting returns with text data. SSRN 3389884. http://dx.doi.org/10.2139/ssrn.3389884. Lemmon, M., Ni, S.X.,

    doi

  16. [2022]

    Artificial intelligence and jobs: Evidence from online vacancies. J. Labor Econ. 40 (S1), S293–S340. http://dx.doi.org/10.1086/718327. Alain, G., Bengio, Y.,

    doi

  17. [2023]

    FinBERT: A large language model for extracting information from financial text. Contemp. Account. Res. 40 (2), 806–841. http://dx.doi.org/10.1111/1911-3846.12832. Hugging Face,

    1911

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.