Backtesting Sentiment Signals for Trading: Evaluating the Viability of Alpha Generation from Sentiment Analysis

Antoine Doucet; Carlos-Emiliano Gonz\'alez-Gallardo; Elvys Linhares Pontes; Georgeta Bordea; Jos\'e G. Moreno; Mohamed Ben Jannet; Yuxuan Zhao

REVIEW 5 major objections 6 minor 14 references

Backtesting Sentiment Signals for Trading: Evaluating the Viability of Alpha Generation from Sentiment Analysis

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A sentiment-only trading strategy generated 50.63% over 28 months on Dow 30 stocks, the paper reports.

desk verdict A well-motivated comparison of sentiment models for trading that is undermined by missing timestamp provenance, in-sample threshold tuning, and internally inconsistent performance metrics. read the letter →

arxiv 2507.03350 v1 pith:K54MMTYY submitted 2025-07-04 cs.CL cs.AI

Elvys Linhares Pontes , Carlos-Emiliano González-Gallardo , Georgeta Bordea , José G. Moreno , Mohamed Ben Jannet , Yuxuan Zhao , Antoine Doucet This is my paper · ORCID

classification cs.CLcs.AI

keywords sentimentanalysistradingstrategyfinancialbacktestingalphagenerationDowJones30FinBERTRoBERTa+Transfsregression

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether news sentiment alone, without price or technical data, can drive profitable trading decisions. It backtests daily sentiment scores for Dow Jones 30 stocks from January 2020 to April 2022, using three BERT-based models to turn news articles into buy, neutral, or sell orders. The regression model RoBERTa+Transfs reached a 50.63% cumulative return over 28 months, beating the 26.96% Buy&Hold benchmark and producing a 23.67% alpha with a much smaller maximum drawdown. The paper's conclusion is that continuous sentiment scores are a viable standalone input for alpha generation and that three-class sentiment labels are too coarse to beat the benchmark. If right, this makes sentiment analysis a practical component of investment strategies rather than just an accuracy benchmark task.

What carries the argument

The load-bearing mechanism is the daily per-asset sentiment aggregation and the threshold-triggered order rule. Each morning, sentiment scores from articles published between the previous and current market open are averaged for each Dow 30 company and rescaled to a 0-100 range; aggregated scores above 60 generate buy orders, scores below 40 generate sell orders, and scores in between generate neutral or position-closing orders. Orders are executed at the opening price with equal dollar amounts and a 0.05% commission. The regression model RoBERTa+Transfs supplies continuous values in [-1, +1], and the paper argues that this continuous resolution is what lets the thresholds separate tradable signals, unlike the three-class outputs of FinBERT and DualGCN.

What would settle it

Recover true publication timestamps for the 1,332,446 articles and rerun the daily aggregation and orders; if the 50.63% return and 23.67% alpha cannot be reproduced with correctly timestamped articles, the headline result was an artifact of the missing meta-information. A companion test is to shuffle the aggregated sentiment scores across stocks and days and check whether the strategy's returns still exceed the shuffled distribution.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a sentiment-based trading system can generate positive alpha when sentiment is predicted as a continuous value. Over the backtest period, the RoBERTa+Transfs regression model produced a 50.63% cumulative return versus 26.96% for Buy&Hold, an alpha of 23.67%, and a maximum drawdown of -5.27% versus -32.38% for the index. The paper attributes the gap to the floating-point predictions, which capture degrees of positivity and negativity, while the two classification models (FinBERT and DualGCN) collapse sentiment into three labels and end up with negative alpha (-1.73% and -11.47%). The intended message is that news sentiment, aggregated daily and used as the sole criterion for market entry, can create a profitable strategy with relatively low risk on large-cap stocks.

Load-bearing premise

The whole strategy rests on assigning each article to the correct daily window between the previous market open and the current market open, but the corpus is described as raw and lacking publication timestamps, so that assignment has no documented basis.

Editorial extensions

If this is right

Under the paper's claims, an NLP model's sentiment score is a tradable signal on its own, without price, volume, or macroeconomic inputs.
Regression-style sentiment scoring should be preferred over three-class classification in trading pipelines because the extra granularity changes whether alpha appears.
The reported risk metrics suggest that sentiment-driven daily rebalancing can cut maximum drawdown relative to passive holding, even during the Covid crash and the 2022 selloff.
The same aggregation-plus-threshold evaluation protocol can be applied to any new sentiment model to decide whether its scores are actionable before deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural control the paper does not run is to shuffle sentiment signals across days and stocks; if shuffled strategies produce similar returns, the headline alpha is a property of the backtest rule rather than of sentiment content.
Because the corpus is described as lacking publication timestamps, a follow-up study should date each article and re-run the backtest; until then, the 50.63% return should be read as conditional on the daily-window assignment actually used.
The 40/60 thresholds were selected by inspecting the same backtest period, so the reported alpha likely includes selection bias; an out-of-sample threshold choice is a direct robustness test.
If continuous sentiment works this well on Dow 30 megacaps, the natural next test is high-momentum or small-cap stocks, where news coverage is sparser and sentiment signals may be noisier.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does something genuinely worth doing: it compares two classification sentiment models and one regression model in a solo-sentiment trading backtest on a large corpus of Dow Jones 30 news, and it reports that the continuous-valued model beats the discrete ones. That comparison is a legitimate extension of prior work, and the finding that continuous scores outperform hard classes is plausible and worth testing. The corpus is large, the pipeline is described step by step, and the authors cite Glasserman and Lin on look-ahead bias, so they are aware of the methodological hazards.

But the central result is not currently supported. The most serious problem is timing. Section 3.1.1 says the articles are raw and contain no meta-information, yet Section 3.1.3 aggregates sentiment over articles published between the previous and current market open. Without publication timestamps, the daily sentiment score is undefined. If timestamps were inferred from crawl dates or file order, the backtest could be mixing look-ahead and stale signals, which can manufacture spurious profits. The paper never explains this, and Figure 3 only shows a date axis for volume, not the source of the dates.

Second, the BUY_SIGNAL and SELL_SIGNAL thresholds were chosen by testing on the same backtest period (Section 3.1.4). That is in-sample tuning, and the reported alpha is conditional on that choice. Third, the comparison is skewed: the sentiment strategy uses an equal-value approach that leaves cash idle, so its lower drawdown and volatility are partly a cash-drag artifact, and the 'alpha' is just the difference in cumulative returns, not a risk-adjusted measure.

Fourth, the metrics in Table 1 are internally inconsistent. A 50.63% cumulative return over 28 months implies an annual compound return of roughly 19%, not the reported 65.26%. The annual return of 23.15% is also incompatible with a 65.26% compound figure. These numbers cannot all be correct, which undermines confidence in the other reported metrics.

This is a paper with a useful question and a plausible hypothesis, but the evidence as presented does not establish the claim. The timestamp provenance issue is load-bearing and may be unfixable if the data are truly raw. Still, I think this deserves peer review rather than desk rejection, because a serious referee can force the authors to clarify the timing, move threshold selection out-of-sample, and fix the metrics. The topic matters to financial NLP, and the classification-versus-regression comparison is worth getting right. My own verdict would be reject with major revision, but it is a paper worth engaging with.

Referee Report

5 major / 6 minor

Summary. The paper describes a backtesting framework in which sentiment scores from three BERT-based models (FinBERT, DualGCN, and RoBERTa+Transfs) are computed for news articles on Dow Jones 30 stocks between January 2020 and April 2022, aggregated into a daily score per stock, and converted into buy, neutral, or sell orders using thresholds. The authors compare the resulting equal-value portfolio with a Buy&Hold benchmark and report that the regression model RoBERTa+Transfs achieves a 50.63% cumulative return and a 23.67% alpha with lower drawdown and volatility than Buy&Hold. The paper interprets this as evidence that news sentiment can generate profitable, low-risk trading strategies.

Significance. If the results were valid, the paper would provide useful evidence on the practical value of continuous sentiment scores relative to discrete classification, and its comparison of three models on a large news corpus is a sensible design. The paper also cites the look-ahead bias literature, showing awareness of backtest pitfalls. However, the current manuscript does not establish the central claim because of missing timestamp provenance, in-sample threshold selection, and a benchmark comparison that is distorted by idle cash; the reported metrics also contain internal inconsistencies. These issues are load-bearing rather than cosmetic.

major comments (5)

[Section 3.1.1 and 3.1.3] The paper states that articles 'are raw and contain no meta-information' (Section 3.1.1) but then defines the daily sentiment score using 'articles published from the previous market open day to the current market open day' (Section 3.1.3). The manuscript never explains how publication timestamps were obtained for 1,332,446 articles or how each article was assigned to a 24-hour trading window. Without this provenance, the backtest is not reproducible and may contain look-ahead or stale signals; every return in Table 1 depends on this step. Figure 3 plots article volume over time, so some date information exists, but its source is not stated.
[Section 3.1.4] The thresholds SELL_SIGNAL=40 and BUY_SIGNAL=60 are selected by testing 45/55, 40/60, and 35/65 on the same backtest period. The reported 50.63% cumulative return and 23.67% alpha are therefore conditional on parameters fitted in-sample. There is no held-out period, walk-forward analysis, or sensitivity check to show that the strategy would have performed similarly out of sample. The central claim of alpha generation is not supported without such validation.
[Table 1 and Appendix C, Eq. (2)] The 'Annual Compound Return' column in Table 1 is inconsistent with the paper's own definitions and with the cumulative returns. For Buy&Hold, a 26.96% cumulative return over 28 months implies a CAGR of about 10.7%, not 23.44%; the 23.44% figure equals the product of the three calendar-year returns reported in Section 4.2 (9.98%, 18.24%, -5.07%). Similarly, RoBERTa+Transfs' 65.26% is the compounded three-year return, not the annual compound return defined by Eq. (2). The table's metric names need to be corrected or the computations need to be revised.
[Section 3.1.4 and Table 1] The equal-value strategy leaves most capital idle ('most of the time, our money will be idle'), so the sentiment portfolios are not fully invested. Their lower volatility, lower drawdown, and high Sharpe and Sortino ratios partly reflect holding cash, not superior signal quality. Comparing the cumulative return of a partially invested strategy with a fully invested Buy&Hold benchmark and calling the difference 'alpha' is not an apples-to-apples comparison. The authors should report a fully invested variant, such as scaling orders to deploy all capital, and report the average gross exposure of the sentiment strategy.
[Section 4.2] Alpha is defined as the simple difference between portfolio return and benchmark return. This is not the usual risk-adjusted alpha; it ignores beta, timing, and statistical significance. Given the idle-cash issue, the reported alpha of 23.67% is not evidence of trading skill. A regression-based alpha with a significance test, or at least an information ratio, is needed to support the claim that the sentiment strategy generates positive risk-adjusted returns.

minor comments (6)

[Table 1] The header contains a typo: 'Annual V olatility' should be 'Annual Volatility'.
[Section 5] The conclusion writes 'alpha of 23,67%' with a comma as the decimal separator, which is inconsistent with the rest of the paper.
[General] The paper does not state whether code and data are available. Given the backtest's dependence on a proprietary news corpus, a detailed data description or reproducibility appendix is essential.
[Section 4.2] The claim that floating-point predictions are superior is based on a single regression model compared with two classification models; confounding factors such as training data and model architecture are not controlled, so the comparison is only suggestive.
[References] Some references are incomplete or informal, such as Bird et al. (2023), described only as 'Rapport interne', and Glasserman & Lin (2023), which lacks publication venue details.
[Section 4.2] The discussion of monthly returns is qualitative; a table of yearly returns or monthly return statistics would make it easier to verify the claims about 2020, 2021, and 2022 performance.

Circularity Check

1 steps flagged · score 6.0 of 10

The reported 50.63% return / 23.67% alpha is conditional on BUY/SELL thresholds chosen by inspecting the same backtest window; no out-of-sample validation is provided.

fitted input called prediction [Section 3.1.4 (Trading Decisions); results in Table 1 / Section 4.2]
"To determine the optimal parameters for SELL_SIGNAL and BUY_SIGNAL, we performed comprehensive analyses to achieve stable transactions and consistent results. Beginning with the neutral value of 50, we systematically tested signal thresholds at 45 and 55, 40 and 60, and 35 and 65. ... Consequently, we established SELL_SIGNAL and BUY_SIGNAL at 40 and 60, respectively, as they demonstrated the most desirable balance between transaction stability and signal activation frequency."

The headline results (50.63% cumulative return, 23.67% alpha, Table 1) are produced by a strategy whose only free decision parameters are SELL_SIGNAL and BUY_SIGNAL. Those thresholds were selected by testing 45/55, 40/60, and 35/65 on the very same 2020-01-01 to 2022-04-30 period that is later used as the evaluation window. The paper provides no hold-out, walk-forward, or out-of-sample validation. The reported return is therefore conditional on parameters chosen after observing the backtest, so the 'alpha' is in part a fitted outcome rather than an independent forecast of sentiment-driven profitability. This matches the pattern of fitting a parameter to the evaluation set and then presenting the resulting performance as evidence for the strategy.

full rationale

The central empirical claim is that news sentiment alone generates positive alpha. The sentiment models themselves are external or from prior work (FinBERT, DualGCN, RoBERTa+Transfs) and are benchmarked in Appendix B, so the sentiment-score pipeline is not definitionally tied to the price returns. No uniqueness theorem or load-bearing self-citation chain is used: the citation to Linhares Pontes & Benjannet (2021) describes a model whose reproduced cosine similarity (0.848) is reported, and the strategy performance is computed from the described backtest rather than derived from that citation. The principal circular step is in Section 3.1.4, where SELL_SIGNAL/BUY_SIGNAL are chosen by testing candidate thresholds on the same January 2020-April 2022 window in which Table 1 performance is then reported. Since no validation split or threshold-sensitivity analysis is given, the headline alpha is partly a fitted backtest artifact. A related data-integrity concern is the missing article-timestamp provenance: Section 3.1.1 states the articles 'are raw and contain no meta-information,' while Section 3.1.3 requires assigning articles to windows between market opens; this is a serious reproducibility and look-ahead risk, but it is a correctness issue rather than a circularity reduction. Overall, the sentiment signal has independent content, but the central performance claim is partially inflated by in-sample parameter selection, so the paper receives a moderate circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The headline alpha depends on in-sample threshold selection and on unverified assumptions about article timestamps, entity recognition accuracy, and model transfer. No new entities are introduced.

free parameters (4)

BUY_SIGNAL = 60
Selected by testing 45/55, 40/60, and 35/65 thresholds on the same backtest period (Section 3.1.4). Reported returns and alpha are conditional on this in-sample choice.
SELL_SIGNAL = 40
Selected jointly with BUY_SIGNAL on the same backtest period (Section 3.1.4). Lower or higher thresholds change signal frequency and returns.
ORDER_VALUE = $10,000
Fixed dollar amount per order, set so that 30 stocks exhaust the $300,000 initial capital. This creates idle cash and makes the strategy not fully invested, affecting the comparison with Buy&Hold.
Commission fee = 0.05%
Assumed per-trade cost (Section 3.1.4). No sensitivity analysis is reported, though results depend on it.

assumptions (4)

domain assumption News articles used in the backtest can be assigned to the correct daily aggregation window (previous market open to current market open) even though the corpus is described as raw with no meta-information.
Section 3.1.1 says articles 'are raw and contain no meta-information,' yet Section 3.1.3 requires publication timestamps to form daily signals; if timestamps are inferred or unreliable, the backtest timing is unverified.
domain assumption The BERT-CRF entity recognizer correctly identifies Dow Jones 30 company mentions in the news corpus.
Section 3.1.1 uses the recognizer to isolate sentences per asset, but no accuracy evaluation is reported on this corpus; misattributed sentiment would corrupt signals.
domain assumption Sentiment scores from models trained on Financial Phrase Bank and SemEval transfer to the news corpus without calibration drift.
Appendix B describes training on small public datasets, then applies models to 1.33M news articles; no distribution-shift check is provided.
ad hoc to paper The in-sample selected thresholds (40/60) would generalize out-of-sample.
Section 3.1.4 selects thresholds based on backtest stability, so the reported alpha assumes this selection is valid beyond the test window.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Backtesting Sentiment Signals for Trading: Evaluating the Viability of Alpha Generation from Sentiment Analysis." pith.science (2026). https://pith.science/paper/K54MMTYY

@misc{pith2026250703350,
  author       = {Pith},
  title        = {Pith review of: Backtesting Sentiment Signals for Trading: Evaluating the Viability of Alpha Generation from Sentiment Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K54MMTYY}},
  note         = {Machine review of arXiv:2507.03350}
}

read the original abstract

Sentiment analysis, widely used in product reviews, also impacts financial markets by influencing asset prices through microblogs and news articles. Despite research in sentiment-driven finance, many studies focus on sentence-level classification, overlooking its practical application in trading. This study bridges that gap by evaluating sentiment-based trading strategies for generating positive alpha. We conduct a backtesting analysis using sentiment predictions from three models (two classification and one regression) applied to news articles on Dow Jones 30 stocks, comparing them to the benchmark Buy&Hold strategy. Results show all models produced positive returns, with the regression model achieving the highest return of 50.63% over 28 months, outperforming the benchmark Buy&Hold strategy. This highlights the potential of sentiment in enhancing investment strategies and financial decision-making.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 13 canonical work pages

[1]

the most significant decline in the history of the Dow Jones 30 index and the subsequent recovery period

Further details about this corpus can be consulted at Appendix A. the most significant decline in the history of the Dow Jones 30 index and the subsequent recovery period. Specifically, the article selection process involved curating content from a variety of respected financial sources, including Accesswire 2, Benzinga 3, Infocast 4, Informa 5, MT Newswi...

work page 2020
[2]

https://www.accesswire.com

work page
[3]

https://www.benzinga.com/

work page
[4]

https://infocastinc.com

work page
[5]

https://www.informa.com

work page
[6]

https://www.mtnewswires.com

work page
[7]

https://www.reuters.com

work page
[8]

https://www.seekingalpha.com

work page

Show all 14 references

[9]

https://www.americanbankingnews.com
[10]

https://finance.yahoo.com/
[11]

https://huggingface.co/ProsusAI/finbert
[12]

Further details and training procedures can be consulted at Appendix B. 3.1.3 Sentiment Aggregation To establish distinct sentiment measures for individual assets and facilitate the formulation of effective trading strategies, we aggregate sentiment scores derived from article...
[13]

Additional details about the metrics can be found at Appendix C. Metrics Buy&Hold DualGCN FinBERT RoBERTa+Transfs Annual Return 9.12% 6.58% 10.91% 23.15% Annual Compound Return 23.44% 16.62% 28.39% 65.26% Annual Cumulative Return 26.96% 15.49% 25.23% 50.63% Calmar Ratio 0.28 1...

2019 arXiv
[1998]

is calculated by subtracting the risk-free rate of return (Rf ) from the expected or average return of the investment (Rp), and then dividing the result by the standard deviation of the investment returns (σp) as follows : Sharpe Ratio = Rp − Rf σp (6) Sortino Ratio It is a ri...

1994

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

the most significant decline in the history of the Dow Jones 30 index and the subsequent recovery period

Further details about this corpus can be consulted at Appendix A. the most significant decline in the history of the Dow Jones 30 index and the subsequent recovery period. Specifically, the article selection process involved curating content from a variety of respected financial sources, including Accesswire 2, Benzinga 3, Infocast 4, Informa 5, MT Newswi...

work page 2020

[2] [2]

https://www.accesswire.com

work page

[3] [3]

https://www.benzinga.com/

work page

[4] [4]

https://infocastinc.com

work page

[5] [5]

https://www.informa.com

work page

[6] [6]

https://www.mtnewswires.com

work page

[7] [7]

https://www.reuters.com

work page

[8] [8]

https://www.seekingalpha.com

work page