{"id":"e9cfe342-d9cd-4964-b0cc-bdc674f4a937","arxiv_id":"2412.06837","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"On Nigerian financial news from 2010 to 2024, logistic regression with TF-IDF outperformed FinBERT and a predefined GPT-4 approach for predicting NGX index direction, with 81.83% accuracy.","lead":"This paper compared three AI approaches, FinBERT, GPT-4, and logistic regression, for predicting Nigerian stock index movements from financial news headlines. It found the simple logistic regression model was the most accurate, but the result is hard to verify because data and code were not released and one methods sentence hints at a possible data leak.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Target-label leakage threat: Section 2.2 states stock index labels were added as input features, directly undermining the reported LR accuracy.","rationale":"The reader identified the same load-bearing concern: the apparent inclusion of target labels as input features in Section 2.2. My reading of the full text confirms that this is the most serious threat to the central claim, because it affects the validity of all reported metrics, not just the logistic regression numbers. I considered whether other issues—the opaque GPT-4 setup, the missing confidence intervals, the chronological alignment of 24,923 headlines into 3,573 daily observations—might be more fundamental, but each of those is secondary to the leakage question. If labels really entered as features, no amount of hyperparameter tuning or time-series cross-validation can rescue the result. I also note that the manuscript is self-contradictory: the introduction says labeled data 'allows the models to associate input news features with specific outcomes' (supervised learning), while Section 2.2 says labels were 'added to the news dataset as an input feature.' This contradiction is precisely why the sentence cannot be dismissed as a mere slip of phrasing without direct confirmation from the authors. I therefore maintain the reader's CONDITIONAL verdict: the paper should not be accepted as-is; it requires either a clear retraction of that sentence or access to code/data demonstrating that labels were not used as features. My verdict is unchanged from the reader's, so I set it to UNCHANGED.","tokens_in":132,"tokens_out":3382,"duration_ms":43078,"concrete_test":"Request the authors' preprocessing and training scripts (or the exact feature matrix for the logistic regression model) and check whether the target label column ('Class 1'/'Class 0') appears among the input features. Then re-run the logistic regression pipeline with the label column removed from the feature set, keeping the train/test split unchanged. If test accuracy drops to near chance (≈50–55%), the reported 81.83% was driven by target leakage. If accuracy remains high, the claim is supported. Additionally, ask the authors to state explicitly whether the target label was ever used as an input feature for any model, and to reconcile the Section 2.2 sentence with the supervised-labeling description in Section 2.6.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that logistic regression reaches 81.83% test accuracy and 89.76% ROC AUC—rests on the assumption that the target variable (daily NGX price move) was used only for supervision, never as a model feature. Section 2.2 explicitly says: 'data labels based on the stock index categorization were added to the news dataset as an input feature of the model.' This sentence is internally inconsistent with the supervised training described elsewhere (e.g., Sections 2.5 and 2.6, where labels are the predicted classes). If interpreted literally, the model would have direct access to the target during both training and testing, and the reported accuracies would be meaningless. The 81.83% figure being below 100% does not disprove leakage; it may reflect a formatting error, a partially applied label column, or a separate 'category' feature—but without the data or code, the reader cannot distinguish these possibilities. The paper ships no artifacts, and the Data Availability statement says only 'available upon request,' so the ambiguity cannot be resolved from the manuscript. This is the single most load-bearing weakness: every comparison of the three models inherits it. If the label-as-feature sentence is an error, the claim may be salvageable; if it is accurate, the headline result is invalid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three approaches for sentiment-based prediction of daily NGX All-Share Index movements using Nigerian financial news: FinBERT, a GPT-4 predefined (prompt-based) sentiment approach, and a logistic regression classifier on TF-IDF features. The data consist of 24,923 news headlines (2010–2024) aggregated into 3,573 daily observations, split chronologically 70/15/15. Hyperparameters are tuned with Optuna under five-fold time-series cross-validation. The authors report that logistic regression outperforms the other two, with 81.83% test accuracy and 89.76% ROC AUC, versus 63.33% for FinBERT and 54.19% for the GPT-4 predefined approach. They conclude that a simple, well-regularized linear model is the most practical choice for this task and recommend future hybrid approaches.","tokens_in":17064,"tokens_out":3827,"duration_ms":36247,"significance":"If the reported results are valid, the paper provides a useful empirical data point for emerging-market financial sentiment: a tuned logistic regression on TF-IDF can beat state-of-the-art large language models on this specific news-index prediction task at a tiny fraction of the computational cost. The study uses a real-world dataset (Nigerian market news), respects temporal order in the data split, and reports a full battery of classification metrics. The main obstacle is that a key sentence in the methodology suggests the target labels were used as input features, which would invalidate the central claim. The paper also lacks uncertainty quantification and a precise description of the news-to-label temporal alignment. These issues must be resolved before the comparative conclusion can be accepted.","major_comments":[{"comment":"The sentence 'data labels based on the stock index categorization were added to the news dataset as an input feature of the model' is a direct statement that the target label was an input feature. If taken literally, the model had access to the outcome during both training and testing, making every reported accuracy and ROC AUC circular and meaningless. This is inconsistent with Sections 2.5 and 2.6, where the labels are described as the predicted classes. The authors must clarify definitively whether labels were used only as the prediction target, and if so, revise the sentence. This is the most load-bearing issue in the manuscript because all three model comparisons inherit it.","section":"Section 2.2, Data Preparation"},{"comment":"The paper states that 24,923 headlines were aggregated into 3,573 'distinct temporal observations' but does not specify the aggregation rule (e.g., concatenation, count, average sentiment) or the exact mapping from headlines to daily NGX labels. In particular, it is not clear whether news published during trading day t is used to predict day t's return (which would introduce look-ahead) or day t+1's return. The authors should define the time cutoff and confirm that no information from day t's price move was available to the model when predicting that same day's label, as this is essential for the no-leakage claim.","section":"Section 2.2, Data aggregation and temporal alignment"},{"comment":"The headline results (LR 81.83% accuracy, 89.76% ROC AUC) are point estimates from a single chronological test set. Time-series cross-validation is used only for hyperparameter selection, not for reporting final performance. Without confidence intervals, per-fold results, or a significance test (e.g., McNemar or a bootstrap comparison), the reader cannot assess whether the gap between LR and FinBERT (81.83% vs. 63.33%) is meaningful or within noise. Please report variability across folds or confidence bounds for the test metrics.","section":"Section 3.3, Table 3 and Section 3.1, Table 1"}],"minor_comments":[{"comment":"The recall and F1 values for the GPT predefined approach are swapped between Table 2 (Recall 32.69%, F1 45.09%) and Table 4 (Recall 45.09%, F1 32.69%). The text in Section 3.2.1 supports Table 2, so Table 4 should be corrected.","section":"Section 3.2.1, Table 2 vs. Section 4, Table 4"},{"comment":"The paper notes that training accuracy (80.93%) and test accuracy (81.83%) are 'very close' and concludes there is no overfitting. While the similarity is reassuring, test accuracy slightly exceeding training accuracy is unusual and may warrant a brief comment (e.g., regularization effects or label noise), to avoid overstating the generalization claim.","section":"Section 3.3, Model Evaluation"},{"comment":"Several typos appear in the abstract and body: 'Finaance', 'Generatice', 'Transsformers', and awkward phrasing such as 'predefined approach of versatile GPT-4'. The manuscript would benefit from careful proofreading.","section":"Throughout"},{"comment":"The statement 'Data are available upon request' is vague and does not follow current reproducibility best practices. Please provide a repository with the preprocessed (or raw) data, the exact train/validation/test split timestamps, and the code for all three models, or explain why the data cannot be shared.","section":"Data Availability Statement"},{"comment":"The comparison with GPT-4 uses a predefined (zero-shot) prompt approach, not a fine-tuned model. This is acknowledged in the text, but the abstract and conclusions should also make clear that GPT-4 was not trained on the NGX-labeled data, so the comparison is primarily against a zero-shot baseline rather than a fully trained model.","section":"Section 3.2, GPT-4 setup"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be the version published in Big Data Cogn. Comput. 2024. The label-as-feature sentence in Section 2.2 is serious enough that, if confirmed as accurate, the reported results are invalid and the paper would need a correction or retraction rather than a revision. Given that the sentence is contradicted elsewhere in the paper, I assume it is a wording error; however, the authors must be asked to clarify and to re-run or re-state the methodology precisely. I would also recommend that the editor ask for the data/code to verify the temporal alignment, since the news-to-label mapping is not described."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a workmanlike empirical comparison on a new dataset, with one unresolved sentence that could sink it, and no artifacts to check.\n\nThe real contribution is the corpus: 14 years of Nigerian financial news headlines from Nairametric and Proshare, mapped to daily NGX All-Share Index direction. That's not in the prior literature, and it's a sensible resource for a market that gets little NLP attention. The three-way comparison (TF-IDF + logistic regression, fine-tuned FinBERT, GPT-4 with a 'predefined' prompt) on that corpus is new, even if each component is standard. The headline result, that LR gets 81.83% accuracy and 89.76% AUC versus FinBERT's 63.33% and GPT-4's 54.19%, is plausible and fits the well-known pattern that linear models on bag-of-words features often hold their own against large pretrained models on small-to-mid-size text classification tasks. The hyperparameters are reported, and the use of TSCV on the validation side is appropriate.\n\nThe soft spots are real though. The big one is Section 2.2, which says 'data labels based on the stock index categorization were added to the news dataset as an input feature of the model.' If that is literal, the test labels are inside the feature matrix and every number is meaningless. I think it is more likely a misuse of the word 'input feature' to mean 'the label column was attached for supervision' — Section 2.6 describes standard supervised LR with labels as targets, and the accuracies aren't near ceiling — but without code or data this is an ambiguity that a referee cannot resolve. That sentence has to be fixed and the pipeline clarified. Second, there is a single chronological test split with no confidence intervals or repeated runs, so the LR-vs-FinBERT gap may be less dramatic than the point estimates suggest. Third, the GPT-4 evaluation is opaque: no prompt, no temperature, no examples of the 'predefined approach', so those results are essentially unverifiable. Fourth, no data or code is shipped; 'available upon request' isn't enough for the central empirical claim to be checked. There are also smaller issues, like Table 4 swapping F1 and recall for GPT, and a typo-riddled abstract.\n\nWho is this for? Someone working on sentiment-based price prediction in emerging African markets might find the dataset and the pragmatic takeaway (start with a simple linear model) useful. A reader looking for a rigorous model comparison will be frustrated by the missing artifacts and uncertainty.\n\nI would send it out for peer review if it were submitted today — the corpus and the empirical question are worth refereeing — but my own verdict would be conditional on the authors fixing the leakage sentence and releasing the aggregation script and prompts. As it stands, I wouldn't cite the accuracy numbers without first getting clarification.","headline":"A new Nigerian-news benchmark where logistic regression beats FinBERT and GPT-4, but a label-as-input sentence and missing artifacts leave the headline numbers unverified.","tokens_in":17646,"tokens_out":3435,"would_cite":false,"duration_ms":30501,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that on 14 years of Nigerian stock-market news, a tuned logistic regression on TF-IDF features predicts index direction more accurately than fine-tuned FinBERT or a predefined GPT-4 approach, and at far lower compute cost.","keywords":["sentiment analysis","stock price prediction","FinBERT","GPT-4","logistic regression","TF-IDF","time series cross-validation","NGX All-Share Index"],"falsifier":"Run the identical pipeline with the label column removed from the input features and with each day's news timestamped strictly before that day's market close; if logistic regression's test accuracy drops from 81.83% toward the roughly 50% base rate, the central claim fails as a leakage artifact, whereas if accuracy stays near 80%, the result is genuine.","tokens_in":16587,"feed_emoji":"📈","tokens_out":10118,"duration_ms":83736,"temperature":0.7,"pith_summary":"The paper tries to settle a practical question: for predicting whether the Nigerian stock index rises or falls from the day's financial news, is an advanced language model worth the cost? It compares three approaches on 24,923 news headlines aggregated into 3,573 daily observations spanning 2010 to 2024: a fine-tuned FinBERT, a predefined GPT-4 sentiment pipeline, and a logistic regression on TF-IDF text features, all tuned with Optuna and evaluated with time-series cross-validation. The central claim is that logistic regression wins clearly, with 81.83% test accuracy and 89.76% ROC AUC, against 63.33% accuracy and 65.59% AUC for FinBERT and 54.19% accuracy and 65.37% AUC for the GPT-4 predefined approach. If that claim is right, it matters because a cheap, interpretable linear model is the better default for this kind of sentiment-to-movement task, and the transformer machinery adds cost without adding accuracy on this dataset. The paper reads the result as an instance of Occam's razor and recommends hybrid designs that feed advanced-model sentiment scores into simple classifiers.","feed_headline":"Simple logistic regression beats GPT-4 and FinBERT on stock news","feed_subtitle":"On 14 years of Nigerian market headlines, the cheap linear model hit 81.83% accuracy and beat both large AI models.","key_machinery":"The comparison rests on three model-specific text representations. FinBERT (a 12-layer BERT architecture with 768 hidden size) receives BERT embeddings with a maximum sequence length of 128 and is fine-tuned with automatic mixed precision and early stopping. GPT-4 is used in a predefined mode: headline text is sent to the API, which classifies sentiment by built-in heuristics with no task-specific training. Logistic regression operates on TF-IDF vectors and outputs a sentiment probability through the sigmoid function $S(z)=1/(1+e^{-z})$, with a 0.5 decision threshold separating 'Class 1' (index gain) from 'Class 0' (no gain or fall). What carries the argument is the evaluation protocol: chronological 70/15/15 splits with time-series cross-validation over five folds, Optuna hyperparameter search with the F1 score as objective, and the same five metrics applied to all three models, so the accuracy gap between the linear model and the transformers is measured under identical temporal conditions.","core_discovery":"The discovery is that the study's second hypothesis fails: the authors expected domain-specific FinBERT and general-purpose GPT-4 to capture market sentiment better than a classic linear model, but on the NGX All-Share Index dataset the opposite held. A logistic regression with L2 regularization and a liblinear solver, trained on TF-IDF vectors of cleaned headlines, reached 81.83% test accuracy, 82.57% precision, 81.15% recall, 81.85% F1 score, and 89.76% ROC AUC, with training accuracy of 80.93% indicating little overfitting. FinBERT, fine-tuned on the same chronological split with Optuna-selected hyperparameters (learning rate around $3.56\\times 10^{-5}$, batch size 16), plateaued at 63.33% accuracy and 65.59% AUC after roughly 110 minutes on an A100 GPU, and its predicted probabilities clustered near 0.5, a sign of persistent uncertainty. The GPT-4 predefined approach, which uses the model's built-in heuristic sentiment classification rather than fine-tuning, scored 54.19% accuracy with high precision but low recall, which the paper attributes to a lack of adaptation to financial jargon. The authors conclude that simple models generalize better when the underlying sentiment signal is close to linearly separable, and that FinBERT and GPT-4 remain useful as components of future hybrid systems rather than as standalone predictors on this dataset.","pith_inferences":["If the paper's Section 2.2 statement that outcome labels were added to the news dataset as an input feature is literal, the reported ranking is compromised; a replication that removes the label column from the features would test whether logistic regression's 81.83% survives at all.","The ranking is likely dataset-dependent: Nigerian market terminology is less represented in FinBERT's pretraining data, so a similar comparison on deeper English financial-news coverage might narrow or reverse the gap between the linear model and the transformers.","A testable extension is to lag the news by one trading day, predicting tomorrow's index move from today's headlines: if logistic regression's edge shrinks substantially, much of its accuracy comes from same-day news published after the market open, which real deployment cannot exploit.","The daily aggregation into 3,573 observations averages away intraday structure; feeding the same headlines through a document-level sentiment score used as a regression feature, rather than a binary classifier, is the natural next experiment the authors' hybrid recommendation points to."],"forward_implications":["A practitioner facing a similar news-to-price-movement task should benchmark a tuned logistic regression on TF-IDF features before spending GPU budget on transformer fine-tuning, since the simple model set the highest bar on this dataset.","FinBERT's modest 63% accuracy and probability mass near 0.5 mean that fine-tuning a financial BERT on roughly 3,500 daily observations does not by itself unlock reliable stock-direction signals, at least for this market.","The GPT-4 predefined sentiment approach, with precision near 72.66% but recall of only 32.69%, should not be used as a standalone predictor: its positive calls are often right but it misses most of them.","The paper's recommended path is hybrid: use FinBERT or GPT-4 to produce sentiment scores and feed those as features into a logistic regression or ensemble, rather than relying on either model's raw output.","Because the logistic regression generalizes well (training accuracy 80.93% versus test accuracy 81.83%), the result indicates that the news-text signal for NGX direction is nearly linearly separable after TF-IDF feature engineering."],"supporting_citations":[{"why":"Supplies the FinBERT pre-trained financial language model that the paper fine-tunes and compares against.","marker":"[3]"},{"why":"The prior LASSO-LSTM-FinBERT hybrid approach that the study positions itself against and extends.","marker":"[5]"},{"why":"Cited as the basis for using time-series cross-validation to maintain temporal order and avoid data leakage.","marker":"[10]"},{"why":"Justifies logistic regression as a text-classification baseline and later supports the claim that it gives high-granularity sentiment signals.","marker":"[12]"},{"why":"The contrary evidence that FinBERT outperforms conventional models in financial sentiment, which this study's result overturns on NGX data.","marker":"[22]"},{"why":"Cited to support the Occam's razor reading that simpler models give more reliable forecasts in volatile market conditions.","marker":"[25]"}],"fun_headline_variants":["Logistic regression tops GPT-4 and FinBERT on stock news","Cheap linear model beats big AI on Nigerian stock sentiment","81.83% accuracy: logistic regression wins over deep learning","Study: simple model outperforms GPT-4 and FinBERT on market data","Financial AI showdown: logistic regression outperforms transformers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire accuracy ranking assumes that the outcome labels (whether the index rose or fell that day) were used only as supervision targets and never as model features; Section 2.2 states that 'data labels based on the stock index categorization were added to the news dataset as an input feature,' and that single sentence, if taken literally, collapses the central claim.","fun_headline_variants_meta":{"raw":{"variants":["Logistic regression tops GPT-4 and FinBERT on stock news","Cheap linear model beats big AI on Nigerian stock sentiment","81.83% accuracy: logistic regression wins over deep learning","Study: simple model outperforms GPT-4 and FinBERT on market data","Financial AI showdown: logistic regression outperforms transformers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1697,"prompt_tokens":1143,"completion_tokens":554,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":759,"completion_tokens_details":{"reasoning_tokens":468}},"tokens_in":759,"tokens_out":554,"duration_ms":4950,"temperature":1.0,"reasoning_tokens":468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:36:57.654030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical pipeline with the label column removed from the input features and with each day's news timestamped strictly before that day's market close; if logistic regression's test accuracy drops from 81.83% toward the roughly 50% base rate, the central claim fails as a leakage artifact, whereas if accuracy stays near 80%, the result is genuine.","supporting_citations":[{"cited_title":"FinBERT: A pre -trained financial language representation model for financial text mining","cited_arxiv_id":null,"evidence_quote":"Supplies the FinBERT pre-trained financial language model that the paper fine-tunes and compares against."},{"cited_title":"Prediction of stock price direction using the LASSO -LSTM model combining technical indicators and financial sentiment analysis","cited_arxiv_id":null,"evidence_quote":"The prior LASSO-LSTM-FinBERT hybrid approach that the study positions itself against and extends."},{"cited_title":"Harnessing machine learning for stock price prediction with random forest and simple moving average techniques","cited_arxiv_id":null,"evidence_quote":"Cited as the basis for using time-series cross-validation to maintain temporal order and avoid data leakage."},{"cited_title":"FinBERT: A Large Language Model for Extracting Information from Financial Text","cited_arxiv_id":null,"evidence_quote":"Justifies logistic regression as a text-classification baseline and later supports the claim that it gives high-granularity sentiment signals."},{"cited_title":"Stock Market Prediction Based on BERT Embedding and News Sentiment Analysis","cited_arxiv_id":null,"evidence_quote":"The contrary evidence that FinBERT outperforms conventional models in financial sentiment, which this study's result overturns on NGX data."},{"cited_title":"Model complexity and financial forecasting: A comparative study","cited_arxiv_id":null,"evidence_quote":"Cited to support the Occam's razor reading that simpler models give more reliable forecasts in volatile market conditions."}],"review_version":1}