{"id":"d3c8855d-2aff-4a74-9ab7-2d55faffb0c5","arxiv_id":"2412.19245","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A comparison of sentiment models claims OPT, a GPT-3-style language model, best predicts stock returns, but the backtest is contaminated by future information.","lead":"Researchers compared four ways to read financial news sentiment, from a word-count dictionary to large language models like BERT and OPT, and found OPT most accurate at matching news to short-term stock moves. The paper claims a trading strategy based on OPT would have earned a 355% return, but the test design appears to let the models peek at the returns they are asked to predict.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Forward-looking sentiment labels and a random train/test split leak the predicted return into the model, invalidating the regression and backtest claims.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the random split and forward-looking labels compromise the out-of-sample evaluation. My stress-test agrees and sharpens the mechanism. The label window overlaps the regression's dependent variable by construction (day n+1), so even a perfectly trained model would show a positive coefficient because the score is a function of the target it is asked to predict. The random split compounds this by placing test-period articles in the training set, making the backtest in-sample. Both issues are visible from the manuscript text alone and do not require external data to suspect. The accuracy numbers in Table 3 are also affected because they measure prediction of a forward-looking label on a randomly split set; the model has seen similar articles from the same period during training. A chronological split with a non-overlapping label window is the minimal change needed to produce credible evidence. As the paper stands, the central claim that LLM sentiment has robust predictive power for next-day returns is not supported. The reader's REJECT verdict is appropriate, and no adjustment to the verdict is needed.","tokens_in":11926,"tokens_out":5584,"duration_ms":52692,"concrete_test":"Re-train the OPT sentiment model with the same architecture but with labels defined as the sign of the 3-day excess return over days n-3 to n-1 (fully in the past), and evaluate on a chronological holdout (e.g., train on articles before 2021-08-01, test on articles from 2021-08-01 onward). Re-run Eq. (1) and the long-short backtest on the held-out period. If the OPT coefficient becomes statistically insignificant and the Sharpe ratio falls toward the dictionary baseline (≈1.2), the headline results are artifacts of lookahead in label construction and random splitting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that OPT sentiment scores predict next-day returns (Table 4: coefficients 0.274 and 0.254) and deliver a Sharpe ratio of 3.05 (Table 5, Fig. 1) depends on a training/evaluation design with two forms of lookahead. First, sentiment labels are assigned as the sign of the 3-day excess return from the article date n through n+2 (Section 2.2). The regression in Eq. (1) uses r_{i,n+1} as the dependent variable, so the label window includes exactly the return the score is supposed to predict. The LLM is fine-tuned to predict this label, thereby building information about r_{i,n+1} into the score x_{i,n}; the subsequent regression of r_{i,n+1} on x_{i,n} is partly mechanical, and the coefficient is inflated regardless of any true news-sentiment association. Second, the 80/20 train-test split is random rather than chronological, so the model is trained on articles drawn from the entire 2010-2023 sample, including the August 2021-July 2023 backtest window. The backtest in Section 3.3 therefore runs largely in-sample. These flaws jointly contaminate the accuracy comparison, the regression coefficients, and the portfolio performance, undermining the out-of-sample interpretation and the comparison against the Loughran-McDonald dictionary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates the ability of large language models (OPT, BERT, FinBERT) and the Loughran-McDonald dictionary to predict U.S. stock returns from financial news. Sentiment labels are derived from the sign of the 3-day excess return following each article, and the models are fine-tuned on these labels. The authors report high classification accuracy for OPT (74.4%), significant positive regression coefficients of OPT scores on next-day returns (0.274 and 0.254 in Table 4), and a long-short portfolio with a Sharpe ratio of 3.05 and a 355% gain in the August 2021–July 2023 period. The paper claims that LLM-based sentiment scores substantially outperform the dictionary approach.","tokens_in":12171,"tokens_out":3385,"duration_ms":33388,"significance":"The question of whether LLMs extract tradeable signals from financial news is timely and important. The dataset is large (965,375 articles) and the comparison across several models, including a dictionary benchmark, is a useful design. If the predictive claims were valid, the paper would provide relevant evidence on the value of modern language models for return prediction. However, the core evidence is compromised by a serious lookahead in the label construction and by an evaluation design that does not ensure out-of-sample inference. The strengths of the paper do not outweigh the validity problems in its central claims.","major_comments":[{"comment":"The sentiment labels are defined as the sign of the 3-day excess return from the article date n through day n+2, and the regression in Eq. (1) uses the next-day return r_{i,n+1} as the dependent variable. Because the fine-tuning label includes r_{i,n+1}, the OPT/BERT/FinBERT scores x_{i,n} contain information about the very return they are later used to predict. The coefficients in Table 4 are therefore partly mechanical and do not demonstrate predictive ability; they reflect that the model was trained to forecast a label that embeds the target variable.","section":"Section 2.2 and Eq. (1)"},{"comment":"The regression reported in Table 4 uses all 965,375 observations, i.e., the full sample, not the held-out 20% test set. Combined with the random (non-chronological) 80/20 split described in Section 2.2, this means the regression is an in-sample fit that includes articles on which the models were fine-tuned. The reported t-statistics and R² values are thus not out-of-sample evidence, and the comparison against the Loughran-McDonald dictionary is not a test of predictive accuracy.","section":"Section 2.2 and Table 4"},{"comment":"The portfolio backtest period (August 2021 to July 2023) overlaps with the randomly assigned training and validation data. The paper does not state that the backtest is restricted to the 20% test set, so the long-short OPT strategy's Sharpe ratio of 3.05 and the 355% gain in Fig. 1 likely include positions based on articles used in training. This is a second source of lookahead bias that invalidates the portfolio performance claims as out-of-sample results.","section":"Section 3.3 and Table 5"}],"minor_comments":[{"comment":"The table contains typographical irregularities such as '1 .42', '0 .25', and '3 .05'; these appear to be missing spaces or alignment issues and should be corrected.","section":"Table 5"},{"comment":"The note states that articles are excluded if too similar to content published within the 'prior five business days', while Section 2.1 states the similarity filter uses a 20-day window; these two descriptions are inconsistent.","section":"Table 1 note"},{"comment":"The term 'laggs' in Section 3.3 ('The Loughran-McDonald dictionary model-based strategy (L-S LM dictionary) laggs behind') is a typo for 'lags'.","section":"Section 3.1"},{"comment":"The AIC and BIC values for Regression 6 (AIC = 135,783, BIC = 123,382) violate the standard inequality BIC > AIC for a model with more than one parameter, suggesting a reporting error or an unusual calculation that should be checked.","section":"Table 4"}],"recommendation":"reject","confidential_remarks":"The fundamental flaw is not a matter of presentation: the label construction in Section 2.2 makes the regression in Eq. (1) partly tautological, and the random split plus full-sample regression and backtest provide no out-of-sample validation. Correcting these issues would require redefining the prediction target, re-fine-tuning the models, and rebuilding the backtests, i.e., effectively a new empirical study. I do not see how a revision within the current scope could repair the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this paper claims a 3.05 Sharpe ratio and 355% two-year return from OPT-based news sentiment, but those numbers are not credible because the evaluation design leaks the target return into the model. The regression in Eq. (1) uses r_{i,n+1} as the dependent variable while the sentiment labels are the sign of the 3-day excess return covering r_{i,n} through r_{i,n+2} — so r_{i,n+1} is inside the label. Fine-tuning the model to predict that label mechanically inflates the predictive coefficients in Table 4. On top of that, the 80/20 split is random, not chronological, and the backtest in Section 3.3 appears to use the full 2010–2023 sample, including training articles. The backtest is therefore largely in-sample, and the Sharpe ratio is not out-of-sample evidence.\n\nWhat the paper does well: the dataset is substantial (965,375 articles), and the authors compare several LLMs against the Loughran-McDonald dictionary under a common pipeline. The method section is transparent enough that the lookahead problems are visible — that transparency is itself a virtue. The accuracy comparison in Table 3 may be internally valid as a test of whether the models can predict the sign of a 3-day excess return, but that is a much weaker claim than \"predicts next-day returns\" and does not rescue the portfolio results.\n\nThe soft spots are load-bearing, not cosmetic. Random splitting in time-series data is a classic error, and combining it with a label window that overlaps the predicted return gives the LLM access to the answer. The dictionary comparison is also unfair: LM scores are not fine-tuned on the label, so the gap in Table 3 partly reflects that the LLMs were trained to predict the very thing they are then evaluated on. The reported R² and AIC/BIC values do not address this.\n\nWho should read this: as a cautionary example of how easy it is to build lookahead into a sentiment strategy, it is useful for a methods reading group. As a source of trading signals or evidence about LLM superiority, it should not be used.\n\nRecommendation: if this came to my desk, I would reject it, but I would send it to a knowledgeable referee rather than desk-reject, because the flaws are instructive and the paper has real empirical scaffolding. The referee's job would be to articulate the lookahead clearly and, if the authors can salvage anything with a chronological split and a clean label window, invite a rewrite.","headline":"The paper's headline Sharpe ratio and regression coefficients are artifacts of a lookahead design: sentiment labels include the next-day return, and the backtest runs on the full sample.","tokens_in":12734,"tokens_out":1825,"would_cite":false,"duration_ms":18779,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sentiment scores from the GPT-3-style OPT model predict next-day stock returns well enough that a long-short portfolio built on them earns a Sharpe ratio of 3.05.","keywords":["large language models","sentiment analysis","stock return prediction","OPT","BERT","FinBERT","Loughran-McDonald dictionary","long-short portfolio"],"falsifier":"Re-run the analysis with a strict chronological split — train on articles before a cutoff date, test only on later articles, and define labels using returns that end before the article's publication — and recompute the regression coefficient and the long-short Sharpe ratio. If the OPT coefficient falls to near zero or the Sharpe ratio drops to the market level, the reported predictive edge would be shown to come from the overlap between the label window and the forecast target rather than from genuine news sentiment.","tokens_in":11690,"feed_emoji":"📈","tokens_out":5307,"duration_ms":44888,"temperature":0.7,"pith_summary":"The paper sets out to show that large language models trained on general text extract more tradeable information from U.S. financial news than the dictionary-based sentiment scores that dominate empirical finance. It fine-tunes OPT and BERT on news articles labelled by the sign of each stock's three-day excess return, and compares them with FinBERT and the Loughran-McDonald dictionary. The central result is that OPT scores carry a statistically strong positive association with next-day stock returns, while dictionary scores do not, and that a value-weighted long-short portfolio built on OPT scores earns a Sharpe ratio of 3.05 with a 355% cumulative gain from August 2021 to July 2023. If true, this would shift text-based return prediction away from lexicons toward large pre-trained models.","feed_headline":"GPT-style news sentiment hits 3.05 Sharpe in long-short test","feed_subtitle":"On 965,375 news articles, OPT predicts next-day returns and a long-short portfolio gains 355%.","key_machinery":"The load-bearing object is the vector of per-article sentiment scores $\\mathbf{x}_{i,n}$ produced by each model and entered into the panel regression $r_{i,n+1} = a_i + b_n + \\boldsymbol{\\gamma}\\cdot\\mathbf{x}_{i,n} + \\epsilon_{i,n}$, where $r_{i,n+1}$ is the next-day return, and $a_i$, $b_n$ are firm and date fixed effects. Labels for fine-tuning are the sign of the three-day cumulative excess return from publication day onward, following event-study practice. For the portfolio exercise, long (short) portfolios hold the top (bottom) 20% of stocks by daily sentiment score, are value-weighted, rebalanced daily with a 10 basis point transaction cost, and trades are timed to news release. These connected pieces — model score, regression equation, labelling rule, and portfolio rule — carry the argument from text to profit.","core_discovery":"On its own terms, the paper's discovery is that the sentiment scores produced by the GPT-3-style OPT model are a robust predictor of the next trading day's stock return. In panel regressions with firm and date fixed effects, the coefficient on the OPT score is 0.274 (t = 5.37) alone and 0.254 with other model scores included, whereas the Loughran-McDonald coefficient is 0.083 and statistically insignificant. The same ordering appears in classification accuracy on the 20% test set: OPT 74.4%, BERT 72.5%, FinBERT 72.2%, dictionary 50.1%. The paper concludes that general-purpose LLMs, and OPT in particular, outperform the dictionary approach for both directional forecasting and portfolio construction, with the long-short OPT strategy delivering the highest risk-adjusted return among the models tested.","pith_inferences":["The paper's evaluation design is the part I would test first: because sentiment labels are the sign of a three-day return that starts on publication day, and the regression target is the next-day return, the label window overlaps the target window; a strict temporal split is needed to confirm the coefficients are not lookahead artifacts.","The 355% figure covers a specific two-year window; a natural extension is to ask how the strategy behaves in a longer out-of-sample period and under larger, more realistic transaction costs and short-sale constraints.","If the result survives a temporal split, it suggests a broader research program: probing whether other causal signals in news text, not just sentiment polarity, can be extracted by LLMs for return prediction."],"forward_implications":["If OPT scores genuinely predict next-day returns, dictionary-based sentiment indices in finance should be re-examined; the paper's comparison suggests they capture almost none of the signal in contemporary news.","A daily rebalanced, value-weighted long-short portfolio built on OPT scores would have earned 355% over August 2021 to July 2023 net of 10 bps per trade, versus roughly 1% for value- and equal-weighted market benchmarks.","The predictive content is strongest in the larger, general-purpose model (OPT) rather than the finance-specific FinBERT, implying that broad pre-training matters for financial text.","Regulators and asset managers, the paper argues, need to account for the possibility that LLM-based signals concentrate informational advantage in those with access to such models."],"supporting_citations":[{"why":"Supplies the dictionary baseline whose sentiment scores are the comparator throughout the paper.","marker":"Loughran and Mcdonald, 2011"},{"why":"Provides the master Loughran-McDonald word dictionary used to compute the traditional sentiment score.","marker":"Loughran and McDonald, 2022"},{"why":"Provides the fine-tuning methodology and text-based return prediction setup that the paper adapts for LLM sentiment scores.","marker":"Ke et al., 2020"},{"why":"Introduces BERT, the architecture on which both BERT and FinBERT sentiment scores are based.","marker":"Devlin et al., 2019"},{"why":"Introduces OPT, the GPT-3-style model that produces the paper's best-performing sentiment scores.","marker":"Zhang et al., 2022"},{"why":"Supplies FinBERT, the finance-specific BERT variant used as a benchmark model.","marker":"Huang et al., 2023"},{"why":"Grounds the three-day excess-return event window used to assign sentiment labels to news articles.","marker":"MacKinlay, 1997"}],"fun_headline_variants":["OPT sentiment beats dictionary: 3.05 Sharpe ratio","LLM sentiment scores predict next-day stock returns","OPT tops LLM sentiment models with 74.4% accuracy","Long-short OPT strategy hits 3.05 Sharpe ratio","Dictionary model lags LLMs in stock sentiment prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that a randomly chosen 80/20 train-test split, with news labels defined from the three-day excess return including the next day, gives an unbiased out-of-sample measure of return prediction, and that the portfolio backtest does not use articles that also appeared in training.","fun_headline_variants_meta":{"raw":{"variants":["OPT sentiment beats dictionary: 3.05 Sharpe ratio","LLM sentiment scores predict next-day stock returns","OPT tops LLM sentiment models with 74.4% accuracy","Long-short OPT strategy hits 3.05 Sharpe ratio","Dictionary model lags LLMs in stock sentiment prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1980,"prompt_tokens":1076,"completion_tokens":904,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":823}},"tokens_in":692,"tokens_out":904,"duration_ms":8303,"temperature":1.0,"reasoning_tokens":823,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:47:19.280066+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the analysis with a strict chronological split — train on articles before a cutoff date, test only on later articles, and define labels using returns that end before the article's publication — and recompute the regression coefficient and the long-short Sharpe ratio. If the OPT coefficient falls to near zero or the Sharpe ratio drops to the market level, the reported predictive edge would be shown to come from the overlap between the label window and the forecast target rather than from genuine news sentiment.","supporting_citations":[],"review_version":1}