REVIEW 3 major objections 5 minor 26 references
Innovative Sentiment Analysis and Prediction of Stock Price Using FinBERT, GPT-4 and Logistic Regression: A Data-Driven Approach
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that on 14 years of Nigerian stock-market news, a tuned logistic regression on TF-IDF features predicts index direction more accurately than fine-tuned FinBERT or a predefined GPT-4 approach, and at far lower compute cost.
desk verdict A new Nigerian-news benchmark where logistic regression beats FinBERT and GPT-4, but a label-as-input sentence and missing artifacts leave the headline numbers unverified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The comparison rests on three model-specific text representations. FinBERT (a 12-layer BERT architecture with 768 hidden size) receives BERT embeddings with a maximum sequence length of 128 and is fine-tuned with automatic mixed precision and early stopping. GPT-4 is used in a predefined mode: headline text is sent to the API, which classifies sentiment by built-in heuristics with no task-specific training. Logistic regression operates on TF-IDF vectors and outputs a sentiment probability through the sigmoid function $S(z)=1/(1+e^{-z})$, with a 0.5 decision threshold separating 'Class 1' (index gain) from 'Class 0' (no gain or fall). What carries the argument is the evaluation protocol: chronological 70/15/15 splits with time-series cross-validation over five folds, Optuna hyperparameter search with the F1 score as objective, and the same five metrics applied to all three models, so the accuracy gap between the linear model and the transformers is measured under identical temporal conditions.
What would settle it
Run the identical pipeline with the label column removed from the input features and with each day's news timestamped strictly before that day's market close; if logistic regression's test accuracy drops from 81.83% toward the roughly 50% base rate, the central claim fails as a leakage artifact, whereas if accuracy stays near 80%, the result is genuine.
Extended reading notes
Core claim
The discovery is that the study's second hypothesis fails: the authors expected domain-specific FinBERT and general-purpose GPT-4 to capture market sentiment better than a classic linear model, but on the NGX All-Share Index dataset the opposite held. A logistic regression with L2 regularization and a liblinear solver, trained on TF-IDF vectors of cleaned headlines, reached 81.83% test accuracy, 82.57% precision, 81.15% recall, 81.85% F1 score, and 89.76% ROC AUC, with training accuracy of 80.93% indicating little overfitting. FinBERT, fine-tuned on the same chronological split with Optuna-selected hyperparameters (learning rate around $3.56\times 10^{-5}$, batch size 16), plateaued at 63.33% accuracy and 65.59% AUC after roughly 110 minutes on an A100 GPU, and its predicted probabilities clustered near 0.5, a sign of persistent uncertainty. The GPT-4 predefined approach, which uses the model's built-in heuristic sentiment classification rather than fine-tuning, scored 54.19% accuracy with high precision but low recall, which the paper attributes to a lack of adaptation to financial jargon. The authors conclude that simple models generalize better when the underlying sentiment signal is close to linearly separable, and that FinBERT and GPT-4 remain useful as components of future hybrid systems rather than as standalone predictors on this dataset.
Load-bearing premise
The entire accuracy ranking assumes that the outcome labels (whether the index rose or fell that day) were used only as supervision targets and never as model features; Section 2.2 states that 'data labels based on the stock index categorization were added to the news dataset as an input feature,' and that single sentence, if taken literally, collapses the central claim.
Editorial extensions
If this is right
- A practitioner facing a similar news-to-price-movement task should benchmark a tuned logistic regression on TF-IDF features before spending GPU budget on transformer fine-tuning, since the simple model set the highest bar on this dataset.
- FinBERT's modest 63% accuracy and probability mass near 0.5 mean that fine-tuning a financial BERT on roughly 3,500 daily observations does not by itself unlock reliable stock-direction signals, at least for this market.
- The GPT-4 predefined sentiment approach, with precision near 72.66% but recall of only 32.69%, should not be used as a standalone predictor: its positive calls are often right but it misses most of them.
- The paper's recommended path is hybrid: use FinBERT or GPT-4 to produce sentiment scores and feed those as features into a logistic regression or ensemble, rather than relying on either model's raw output.
- Because the logistic regression generalizes well (training accuracy 80.93% versus test accuracy 81.83%), the result indicates that the news-text signal for NGX direction is nearly linearly separable after TF-IDF feature engineering.
Reading between the lines
- If the paper's Section 2.2 statement that outcome labels were added to the news dataset as an input feature is literal, the reported ranking is compromised; a replication that removes the label column from the features would test whether logistic regression's 81.83% survives at all.
- The ranking is likely dataset-dependent: Nigerian market terminology is less represented in FinBERT's pretraining data, so a similar comparison on deeper English financial-news coverage might narrow or reverse the gap between the linear model and the transformers.
- A testable extension is to lag the news by one trading day, predicting tomorrow's index move from today's headlines: if logistic regression's edge shrinks substantially, much of its accuracy comes from same-day news published after the market open, which real deployment cannot exploit.
- The daily aggregation into 3,573 observations averages away intraday structure; feeding the same headlines through a document-level sentiment score used as a regression feature, rather than a binary classifier, is the natural next experiment the authors' hybrid recommendation points to.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares three approaches for sentiment-based prediction of daily NGX All-Share Index movements using Nigerian financial news: FinBERT, a GPT-4 predefined (prompt-based) sentiment approach, and a logistic regression classifier on TF-IDF features. The data consist of 24,923 news headlines (2010–2024) aggregated into 3,573 daily observations, split chronologically 70/15/15. Hyperparameters are tuned with Optuna under five-fold time-series cross-validation. The authors report that logistic regression outperforms the other two, with 81.83% test accuracy and 89.76% ROC AUC, versus 63.33% for FinBERT and 54.19% for the GPT-4 predefined approach. They conclude that a simple, well-regularized linear model is the most practical choice for this task and recommend future hybrid approaches.
Significance. If the reported results are valid, the paper provides a useful empirical data point for emerging-market financial sentiment: a tuned logistic regression on TF-IDF can beat state-of-the-art large language models on this specific news-index prediction task at a tiny fraction of the computational cost. The study uses a real-world dataset (Nigerian market news), respects temporal order in the data split, and reports a full battery of classification metrics. The main obstacle is that a key sentence in the methodology suggests the target labels were used as input features, which would invalidate the central claim. The paper also lacks uncertainty quantification and a precise description of the news-to-label temporal alignment. These issues must be resolved before the comparative conclusion can be accepted.
major comments (3)
- [Section 2.2, Data Preparation] The sentence 'data labels based on the stock index categorization were added to the news dataset as an input feature of the model' is a direct statement that the target label was an input feature. If taken literally, the model had access to the outcome during both training and testing, making every reported accuracy and ROC AUC circular and meaningless. This is inconsistent with Sections 2.5 and 2.6, where the labels are described as the predicted classes. The authors must clarify definitively whether labels were used only as the prediction target, and if so, revise the sentence. This is the most load-bearing issue in the manuscript because all three model comparisons inherit it.
- [Section 2.2, Data aggregation and temporal alignment] The paper states that 24,923 headlines were aggregated into 3,573 'distinct temporal observations' but does not specify the aggregation rule (e.g., concatenation, count, average sentiment) or the exact mapping from headlines to daily NGX labels. In particular, it is not clear whether news published during trading day t is used to predict day t's return (which would introduce look-ahead) or day t+1's return. The authors should define the time cutoff and confirm that no information from day t's price move was available to the model when predicting that same day's label, as this is essential for the no-leakage claim.
- [Section 3.3, Table 3 and Section 3.1, Table 1] The headline results (LR 81.83% accuracy, 89.76% ROC AUC) are point estimates from a single chronological test set. Time-series cross-validation is used only for hyperparameter selection, not for reporting final performance. Without confidence intervals, per-fold results, or a significance test (e.g., McNemar or a bootstrap comparison), the reader cannot assess whether the gap between LR and FinBERT (81.83% vs. 63.33%) is meaningful or within noise. Please report variability across folds or confidence bounds for the test metrics.
minor comments (5)
- [Section 3.2.1, Table 2 vs. Section 4, Table 4] The recall and F1 values for the GPT predefined approach are swapped between Table 2 (Recall 32.69%, F1 45.09%) and Table 4 (Recall 45.09%, F1 32.69%). The text in Section 3.2.1 supports Table 2, so Table 4 should be corrected.
- [Section 3.3, Model Evaluation] The paper notes that training accuracy (80.93%) and test accuracy (81.83%) are 'very close' and concludes there is no overfitting. While the similarity is reassuring, test accuracy slightly exceeding training accuracy is unusual and may warrant a brief comment (e.g., regularization effects or label noise), to avoid overstating the generalization claim.
- [Throughout] Several typos appear in the abstract and body: 'Finaance', 'Generatice', 'Transsformers', and awkward phrasing such as 'predefined approach of versatile GPT-4'. The manuscript would benefit from careful proofreading.
- [Data Availability Statement] The statement 'Data are available upon request' is vague and does not follow current reproducibility best practices. Please provide a repository with the preprocessed (or raw) data, the exact train/validation/test split timestamps, and the code for all three models, or explain why the data cannot be shared.
- [Section 3.2, GPT-4 setup] The comparison with GPT-4 uses a predefined (zero-shot) prompt approach, not a fine-tuned model. This is acknowledged in the text, but the abstract and conclusions should also make clear that GPT-4 was not trained on the NGX-labeled data, so the comparison is primarily against a zero-shot baseline rather than a fully trained model.
Circularity Check
Target label added as input feature makes the reported stock-movement prediction circular as written.
-
self definitional
[Section 2.2 (Data Preparation)]
"Additionally, data labels based on the stock index categorization were added to the news dataset as an input feature of the model. The labels help the model to differentiate news categories, reduce noise, and enable efficient model training through clear mappings between input features and the desired outcomes [11]."
The task is to predict daily NGX All-Share Index movement, with labels defined in the same section as 'Class 1' for daily share price gain and 'Class 0' for unchanged or fall. If those exact stock-index-categorization labels are added as an input feature, then the model is given the answer it is supposed to predict. The reported test accuracy of 81.83% and ROC AUC of 89.76% for Logistic Regression would be trivially forced by reading the label column, not by learning from news text. The same leakage is echoed in Section 1.3: 'adding NGX labels for easy topic classification as part of input features.' The observed accuracy below 100% suggests the sentence may be loose wording, but as written the central prediction claim reduces by construction to the input label.
full rationale
The only load-bearing circular step is the explicit statement in Section 2.2 that stock-index-categorization labels were included as input features. Since those labels are the target variable for the prediction task, the reported comparison among FinBERT, GPT-4, and Logistic Regression is undermined as written. If the sentence is literal, every reported accuracy is invalid because the model had direct access to the class labels. If it is an imprecise description, the manuscript does not provide enough detail (no code or data, only 'available upon request') to resolve the ambiguity. No other circularity was found: the models are conventional, the hyperparameter tuning is standard, and the external citations are not used to justify the paper's own results.
Assumptions & free parameters
free parameters (7)
- Logistic Regression regularization strength C =
3.037
- Logistic Regression solver and penalty =
liblinear, l2
- FinBERT learning rate =
3.5649e-5
- FinBERT batch size =
16
- Decision threshold =
0.5
- Data split ratio =
70/15/15
- Time series cross-validation folds =
5
assumptions (6)
- domain assumption NGX All-Share Index daily direction is an appropriate ground-truth label for financial news sentiment.
- domain assumption Headlines from Nairametric and Proshare are representative of the market-moving news for the NGX.
- ad hoc to paper Aggregating 24,923 headlines into 3,573 temporal observations preserves the news-price alignment without look-ahead.
- domain assumption No target information leaks into model inputs.
- domain assumption GPT-4's predefined sentiment outputs are comparable to the supervised labels without fine-tuning.
- standard math TF-IDF linear separability and logistic regression assumptions hold sufficiently for the text classification task.
Cite this review
Pith. "Pith review of Innovative Sentiment Analysis and Prediction of Stock Price Using FinBERT, GPT-4 and Logistic Regression: A Data-Driven Approach." pith.science (2026). https://pith.science/paper/CSEFTVJE
@misc{pith2026241206837,
author = {Pith},
title = {Pith review of: Innovative Sentiment Analysis and Prediction of Stock Price Using FinBERT, GPT-4 and Logistic Regression: A Data-Driven Approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/CSEFTVJE}},
note = {Machine review of arXiv:2412.06837}
}
read the original abstract
This study explores the comparative performance of cutting-edge AI models, i.e., Finaance Bidirectional Encoder representations from Transsformers (FinBERT), Generatice Pre-trained Transformer GPT-4, and Logistic Regression, for sentiment analysis and stock index prediction using financial news and the NGX All-Share Index data label. By leveraging advanced natural language processing models like GPT-4 and FinBERT, alongside a traditional machine learning model, Logistic Regression, we aim to classify market sentiment, generate sentiment scores, and predict market price movements. This research highlights global AI advancements in stock markets, showcasing how state-of-the-art language models can contribute to understanding complex financial data. The models were assessed using metrics such as accuracy, precision, recall, F1 score, and ROC AUC. Results indicate that Logistic Regression outperformed the more computationally intensive FinBERT and predefined approach of versatile GPT-4, with an accuracy of 81.83% and a ROC AUC of 89.76%. The GPT-4 predefined approach exhibited a lower accuracy of 54.19% but demonstrated strong potential in handling complex data. FinBERT, while offering more sophisticated analysis, was resource-demanding and yielded a moderate performance. Hyperparameter optimization using Optuna and cross-validation techniques ensured the robustness of the models. This study highlights the strengths and limitations of the practical applications of AI approaches in stock market prediction and presents Logistic Regression as the most efficient model for this task, with FinBERT and GPT-4 representing emerging tools with potential for future exploration and innovation in AI-driven financial analytics
Reference graph
Works this paper leans on
-
[1]
Transforming sentiment analysis in the financial domai n with ChatGPT
Fatouros, G.; Soldatos, J.; Kouroumali, K.; Makridis, G.; Kyriazis, D. Transforming sentiment analysis in the financial domai n with ChatGPT. Mach. Learn. Appl. 2023, 14, 100508. [CrossRef]
work page 2023
-
[2]
Shapiro, A.H.; Sudhof, M.; Wilson, D.J. Measuring news sentiment. J. Econom. 2022, 228, 221–243. [CrossRef]
work page 2022
-
[3]
FinBERT: A pre -trained financial language representation model for financial text mining
Liu, Z.; Huang, D.; Huang, K.; Li, Z.; Zhao, J. FinBERT: A pre -trained financial language representation model for financial text mining. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, Yokohama, Japan, 7 –15 January 2021. Available online: https://www.ijcai.org/proceedings/2020/0622.pdf (accessed on 25 July 2024)
work page 2021
-
[4]
Sentiment spin: Attacking financial sentiment with GPT-3
Leippold, M. Sentiment spin: Attacking financial sentiment with GPT-3. Finance Res. Lett. 2023, 55, 103957. [CrossRef]
work page 2023
-
[5]
Yang, J.; Wang, Y.; Li, X. Prediction of stock price direction using the LASSO -LSTM model combining technical indicators and financial sentiment analysis. PeerJ Comput. Sci. 2022, 8, e1148. [CrossRef] [PubMed]
work page 2022
-
[6]
Stock price prediction using FinBERT and LSTM
Sidogi, T.; Mbuvha, R.; Marwala, T. Stock price prediction using FinBERT and LSTM. In Proceedings of the 2021 IEEE Internatio nal Conference Systems Man and Cybernetics, Melbourne, Australia, 17 –20 October 2021. Available online: https://ieeexplore.ieee. org/abstract/document/9659283 (accessed on 25 July 2024)
-
[7]
Simple heuristics to run a research group
Gigerenzer, G. Simple heuristics to run a research group. PsyCh J. 2022, 11, 133–135. [CrossRef] [PubMed]
work page 2022
-
[8]
A Context-Aware Lemmatization Model for Setswana Language Using Machine Learning
Bafitlhile, K.D. A Context-Aware Lemmatization Model for Setswana Language Using Machine Learning. Master’s Thesis, Botswana International University of Science and Technology, Palapye, Botswana, 2022. Available online: http://repository.biust. ac.bw/handle/123456789/536 (accessed on 25 July 2024)
arXiv 2022
Show all 26 references
-
[9]
What are different research approaches? Comprehensive Review of Qualitative, quantitative, and mixed method research, their applications, types, and limitations
Taherdoost, H. What are different research approaches? Comprehensive Review of Qualitative, quantitative, and mixed method research, their applications, types, and limitations. J. Manag. Sci. Eng. Res. 2022, 5, 53–63. [CrossRef]
2022
-
[10]
Harnessing machine learning for stock price prediction with random forest and simple moving average techniques
Priyatno, A.M.; Ningsih, L.; Noor, M. Harnessing machine learning for stock price prediction with random forest and simple moving average techniques. J. Eng. Sci. Appl. 2024, 1, 1–8. [CrossRef]
2024
-
[11]
Semi -Supervised Classification of Network Data Using Very Few Labels
Lin, F.; Cohen, W.W. Semi -Supervised Classification of Network Data Using Very Few Labels. In Proceedings of the 2010 International Conference on Advances in Social Networks Analysis and Mining, Odense, Denmark, 9 –11 August 2010. Available online: https://ieeexplore.ieee.org...
2010
-
[12]
FinBERT: A Large Language Model for Extracting Information from Financial Text
Huang, A.H.; Wang, H.; Yang, Y. FinBERT: A Large Language Model for Extracting Information from Financial Text. Contemp. Account. Res. 2023, 40, 806–841. [CrossRef]
2023
-
[13]
Financial Text Classification with Deep Learning Using FinBERT
Keita, Z. Financial Text Classification with Deep Learning Using FinBERT. 2022. Available online: https://zoumanakeita.medium. com/financial-text-classification-with-deep-learning-using-finbert-51a73b393a08 (accessed on 9 October 2024)
2022
-
[14]
On the Validity of Pre-trained Transformers for Natural Language Processing in the Software Engineering Domain
Von der Mosel, J.; Trautsch, A.; Herbold, S. On the Validity of Pre-trained Transformers for Natural Language Processing in the Software Engineering Domain. IEEE Trans. Softw. Eng. 2022, 49, 1487–1507. [CrossRef]
2022
-
[15]
EFSA: Towards Event -Level Financial Sentiment Analysis
Chen, T.; Zhang, Y.; Yu, G.; Zhang, D.; Zeng, L.; He, Q. EFSA: Towards Event -Level Financial Sentiment Analysis. Computation and Language. arXiv 2024, arXiv:2404.08681
2024 arXiv
-
[16]
Sentiment trading with large language models
Kirtac, K.; Germano, G. Sentiment trading with large language models. Finance Res. Lett. 2024, 62, 105227. [CrossRef]
2024
-
[17]
Dynamics of Nonlinear Causality: Exploring the Influence of Positive and Negative Financial News on the Indian Equity Market
Varghese, R.R.; Mohan, B.R. Dynamics of Nonlinear Causality: Exploring the Influence of Positive and Negative Financial News on the Indian Equity Market. In Proceedings of the 2023 Annual International Conference on Emerging Research Areas: International Conference on Intellig...
2023
-
[18]
A Decision Support System for Crop Recommendation Using Machine Learning Classification Algorithms
Senapaty, M.K.; Ray, A.; Padhy, N. A Decision Support System for Crop Recommendation Using Machine Learning Classification Algorithms. Agriculture 2024, 14, 1256. [CrossRef]
2024
-
[19]
Survey on algorithmic trading using sentiment analysis
Bagate, R.; Joshi, A.; Trivedi, A.; Pandey, A.; Tripathi, D. Survey on algorithmic trading using sentiment analysis. In Proceedings of the 6th International Conference on Advance Computing and Intelligent Engineering: ICACIE 2021; Springer Nature: Singapore, 2022; pp. 241–252
2021
-
[20]
Ethical Considerations in AI -Driven Predictive Analytics: Addressing Bias and Fairness Issues
Paripati, L.; Hajari, V.R.; Narukulla, N.; Prasad, N.; Shah, J.; Agarwal, A. Ethical Considerations in AI -Driven Predictive Analytics: Addressing Bias and Fairness Issues. Darpan Int. Res. Anal. 2024, 12, 34–50
2024
-
[21]
Smart Expert System: Large Language Models as Text Classifiers
Wang, Z.; Pang, Y.; Lin, Y. Smart Expert System: Large Language Models as Text Classifiers. arXiv 2024, arXiv:2405.10523
2024 arXiv
-
[22]
Stock Market Prediction Based on BERT Embedding and News Sentiment Analysis
Yang, H.; Ye, C.; Lin, X.; Zhou, H. Stock Market Prediction Based on BERT Embedding and News Sentiment Analysis. In Service Science, Proceedings of the CCF 16th International Conference, ICSS 2023, Harbin, China, 13–14 May 2023, Revised Selected Papers; Wang, Z., Wang, S., Xu,...
2023
-
[23]
Exploring the sentiment in Borsa Istanbul with deep learning
Atak, A. Exploring the sentiment in Borsa Istanbul with deep learning. Borsa Istanb. Rev. 2023, 23, S84–S95. [CrossRef]
2023
-
[24]
Performance analysis of Machine Learnin g Algorithms to classify text data
Idate, S.; Srinivasa Rao, T.; Gayakwad, M.; Paygude, P.; Chavan, P.; Pawar RKadam, K. Performance analysis of Machine Learnin g Algorithms to classify text data. J. Electr. Syst. 2024, 20, 1588–1599. [CrossRef] Big Data Cogn. Comput. 2024, 8, 143 21 of 21
2024
-
[25]
Model complexity and financial forecasting: A comparative study
Zhang, L.; Lee, J. Model complexity and financial forecasting: A comparative study. J. Financ. Anal. 2022, 15, 45–60
2022
-
[26]
Transformer-based Models for Language Identification: A Comparative Study
Kumar, R.P.; Elakkiya, R. Transformer-based Models for Language Identification: A Comparative Study. In Proceedings of the 2023 International Conference on System, Computation, Automation and Networking (ICSCAN), Puducherry, India, 17 –18 November 2023. Available online: https...
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.