REVIEW 4 major objections 6 minor 24 references
Dual-Model Sentiment Analysis of Consumer Reviews in the Retail Coffee Sector Using Machine Learning and Deep Learning Approaches
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims that on a real-world imbalanced set of Starbucks reviews, SVM leads the machine-learning models and BiLSTM leads the deep-learning models, with class weighting behind both.
desk verdict Routine but readable benchmark whose ranking claim is undercut by an internal inconsistency in class weighting and an anecdotal 'generalization' test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the dual-model pipeline: TF-IDF unigram and bigram features feeding classical classifiers, and padded word sequences feeding embedding-based recurrent and convolutional networks. The identity that carries the argument is the star-rating binarization rule (4-5 becomes positive, 1-3 becomes negative) combined with class weighting for SVM and BiLSTM. That rule defines the supervised target, and the class weighting is what keeps the two winning models from simply predicting the majority negative class, so the comparison is really about which model best uses the text signal under that label rule.
What would settle it
A reader could settle the claim by drawing a random sample of reviews, having annotators label sentiment without seeing star ratings, and comparing model predictions to those labels; if agreement on negative reviews is near chance, or if SVM and BiLSTM accuracy on that gold set falls well below 91% and 92%, the comparative ranking is an artifact of the label rule. A simpler check is to retrain with 3-star reviews removed or treated as neutral and see whether the two winners still lead.
Extended reading notes
Core claim
On a dataset of over 700 Starbucks reviews, with star ratings collapsed to binary sentiment (4-5 positive, 1-3 negative), the paper claims that SVM with an RBF kernel reaches 91% accuracy and a 0.90 weighted F1-score, the best of the five machine-learning models, and that BiLSTM reaches 92% accuracy and 0.91 weighted F1-score, the best of the five deep-learning models. The paper further claims that BiLSTM demonstrated good generalization on unseen data based on correct classification of two manually written reviews. The reported pattern is that bidirectional recurrent models capture context better than CNNs and one-directional RNNs on this data, and that class imbalance mainly suppresses recall of the minority positive class across models.
Load-bearing premise
The load-bearing premise is that binarized star ratings (4-5 positive, 1-3 negative) from an unverified third-party Starbucks review collection faithfully represent the true sentiment of each review; if those labels are noisy, skewed, or unrepresentative, every reported accuracy and F1 number loses its meaning.
Editorial extensions
If this is right
- On review sets of similar size and imbalance, SVM and BiLSTM are the safest of the ten tested choices when the goal is accuracy and F1.
- Accuracy alone is misleading here; because positive reviews are the minority, monitoring recall on positive sentiment is necessary to avoid silently predicting the majority class.
- Because the paper deliberately avoided resampling, the reported numbers describe natural-deployment conditions under class weighting, not what would happen with SMOTE or undersampling.
- The two manual test sentences provide only a minimal check of generalization; the paper itself states that broader validation with more diverse reviews is needed before deployment.
Reading between the lines
- A natural next experiment is to rerun the same ten models with SMOTE or random oversampling and with 3-star reviews treated as neutral; the current ranking could change if the winning models' edge comes from class weighting rather than from the text signal.
- The 91% and 92% figures should be read as upper bounds relative to the unvalidated star-rating labels; a human-annotated gold set would likely lower them and would provide a stronger comparison for transformer-based models.
- Because the dataset includes location and timestamp fields, the same pipeline could be extended to test whether models trained on one region or season transfer to another, which the paper's exploratory analysis hints at but does not evaluate.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents a comparative sentiment analysis study of Starbucks customer reviews sourced from a Kaggle/ConsumerAffairs dataset containing over 700 reviews. Star ratings are binarized (4–5 positive, 1–3 negative), yielding an imbalanced negative-dominated corpus. Five classical machine learning classifiers (Logistic Regression, SVM, Decision Tree, Random Forest, Naive Bayes) and five deep learning models (LSTM, RNN, BiLSTM, GRU, CNN) are evaluated under an 80/20 stratified split using accuracy, precision, recall, and F1-score. The authors report that SVM achieves the highest machine learning accuracy (91%) and BiLSTM the highest deep learning accuracy (92%), and claim that BiLSTM generalizes to unseen data. The paper also includes exploratory data analysis of temporal and geographic patterns and a qualitative demonstration on two manually written example reviews.
Significance. If the central claims were fully supported, the paper would provide a useful, if incremental, practitioner-oriented comparison of standard ML and DL pipelines on a small, imbalanced, real-world review dataset. The work has several strengths: it uses a clearly described preprocessing pipeline, it evaluates multiple metrics rather than accuracy alone, it preserves the natural class imbalance, and the writing is generally accessible. The authors also make an effort to connect EDA insights to modeling choices. However, the significance is substantially limited by the lack of controlled experimental comparison, the absence of statistical support for the reported rankings, the inconsistent description of class-weighting, and the anecdotal nature of the generalization claim. As presented, the top-model claim is not established beyond configuration-choice and sampling noise.
major comments (4)
- [Sections 3.5 and 3.6.1] The manuscript is internally inconsistent about which models received class weighting. Section 3.5 states that class weighting was implemented for SVM and BiLSTM, whereas Section 3.6.1 states that class weighting was applied to LR and SVM, and Section 3.6.2 says class weights were used 'where appropriate' without naming the models. Because the two winning models (SVM and BiLSTM) may have received a different imbalance-handling treatment than the other models, the reported differences in accuracy and F1-score could reflect configuration choices rather than model quality. The authors must specify exactly which models received class weights, and ideally use a uniform protocol (all models with class weights, or none) to make the comparison fair.
- [Tables 2 and 3, Section 3.4] The performance rankings rest on single-point estimates from one 80/20 stratified split. With roughly 700 reviews, the test set contains about 140 instances, so the differences between 91% and 92% accuracy, or between 90% and 91%, are within sampling noise. The paper reports no confidence intervals, no repeated runs, no random seeds, and no significance tests. The introduction and contribution list also state that confusion matrices are part of the evaluation, but no confusion matrices appear anywhere in the manuscript. To support the central ranking claim, the authors should provide repeated stratified cross-validation or bootstrap confidence intervals, report statistical significance (e.g., McNemar's test), and include the promised confusion matrices.
- [Abstract and Sections 4.3–4.4] The claim that BiLSTM 'demonstrated good generalization on unseen data' is supported only by two manually written sentences ('Taste was very good' and 'Smells bad'). These examples contain obvious unigram polarity cues that any lexicon-based or bag-of-words baseline would classify correctly, so they do not provide evidence of generalization beyond the training distribution. The authors themselves acknowledge the sample is minimal and call for extended validation. The generalization claim in the abstract and conclusions should be removed or replaced by a proper external held-out evaluation with quantitative metrics, or the wording should be limited to a qualitative illustration.
- [Section 3.1] The ground-truth labels are derived by binarizing star ratings from an unverified third-party Kaggle upload. The paper does not report basic dataset statistics after cleaning, such as the exact number of reviews, the class counts, or the number of unique reviews versus duplicate or near-duplicate texts. It also does not discuss whether the Kaggle upload faithfully represents the original ConsumerAffairs data or whether any manual validation of labels was performed. If the labels are noisy or the data is not representative, every reported accuracy and F1 value loses meaning. The authors should describe dataset provenance, licensing, class distribution after binarization, and any verification steps.
minor comments (6)
- [Data Availability] The first sentence contains a typo: 'he dataset used in this study' should read 'The dataset used in this study.'
- [References] References [13] and [15] are the same arXiv paper by Zhang, Wang, and Liu; this duplicate should be removed or replaced. Citation style is also inconsistent, mixing numbered citations with author-year forms such as '(Socher et al., 2013)'.
- [Tables 2 and 3] The manuscript does not specify whether precision and recall are macro-averaged, micro-averaged, or weighted averages. Since several models report identical values for accuracy, precision, and recall, this is likely a rounding artifact or a particular averaging choice, and it should be clarified.
- [Section 3.1 and elsewhere] The term 'dual-model approach' is used repeatedly, but it is never formally defined. The paper actually compares two families of models, so 'dual-paradigm comparison' or 'dual-pipeline comparison' would be more accurate and less ambiguous.
- [Sections 4.3 and 4.4] The phrase 'Own Data' in the section titles is misleading, as the examples are manually crafted single sentences rather than a new dataset. Rename these sections to something like 'Qualitative Prediction Examples' to reflect their scope.
- [Figures 1–6] Figure captions are sometimes placed after the figure or separated from the visual, making it hard to match the caption to the plot. Please reformat so each caption immediately precedes or follows its figure.
Circularity Check
No circularity: empirical benchmark with standard train/test split; top-model claim is a measurement, not a construction.
full rationale
The paper makes no theoretical derivation whose output could reduce to its input. The central claim (SVM best among machine learning models, BiLSTM best among deep learning models on this dataset) is an empirical comparison: features (TF-IDF, padded token sequences) are constructed from review text independently of the labels, labels come from binarized star ratings, models are trained on an 80% stratified split and evaluated on the held-out 20% split (Section 3.4). The accuracy/F1 numbers in Tables 2 and 3 are therefore measurements, not quantities fixed by construction, and no fitted parameter is renamed as a prediction. The 'own data' tests (Sections 4.3-4.4) are illustrative predictions on two hand-written sentences; they were not used to fit any constant or to select a model, so they do not make the claim circular, though they are too small to support a generalization claim. No load-bearing self-citation appears: the references are external prior work, and the authors' own lab affiliation is not used to justify any result. Inconsistencies about which models received class weighting (Section 3.5 vs. Section 3.6.1) and the absence of repeated trials or statistical significance tests are methodological weaknesses affecting the strength of the comparison, but they are not circularity under the stated hard rules because no specific reduction of the conclusion to its inputs can be exhibited. Accordingly, the paper is self-contained as an empirical study and receives a circularity score of 0.
Assumptions & free parameters
free parameters (8)
- TF-IDF max_features =
10000
- TF-IDF min_df =
5
- Sequence padding length =
100
- Embedding dimension =
100
- Training epochs =
10
- Batch size =
32
- Learning rate =
0.001
- Class weights for SVM and BiLSTM =
unspecified
assumptions (6)
- domain assumption Star-rating binarization is a valid sentiment ground truth
- domain assumption The Kaggle upload is authentic and representative
- domain assumption TF-IDF and randomly initialized embeddings capture enough signal
- ad hoc to paper A single stratified 80/20 split is sufficient to rank models
- ad hoc to paper Selective class weighting yields a fair comparison
- ad hoc to paper Two hand-written examples demonstrate generalization
Cite this review
Pith. "Pith review of Dual-Model Sentiment Analysis of Consumer Reviews in the Retail Coffee Sector Using Machine Learning and Deep Learning Approaches." pith.science (2026). https://pith.science/paper/7SMHI5FJ
@misc{pith2026260812007,
author = {Pith},
title = {Pith review of: Dual-Model Sentiment Analysis of Consumer Reviews in the Retail Coffee Sector Using Machine Learning and Deep Learning Approaches},
year = {2026},
howpublished = {\url{https://pith.science/paper/7SMHI5FJ}},
note = {Machine review of arXiv:2608.12007}
}
read the original abstract
Consumer reviews play an important role in shaping brand perception and business strategies, particularly in service-driven industries such as retail coffee. This study presents a comparative sentiment analysis framework for Starbucks customer reviews using classical machine learning and deep learning approaches. The dataset, collected from ConsumerAffairs, contains more than 700 reviews and was analyzed through preprocessing and exploratory data analysis to identify temporal and geographic patterns. Sentiment labels were generated by binarizing star ratings, with ratings of 4 and 5 classified as positive and ratings of 1 to 3 as negative. The resulting dataset was substantially imbalanced toward negative sentiment. Five machine learning classifiers, including Logistic Regression, Support Vector Machine (SVM), Decision Tree, Random Forest, and Naive Bayes, were evaluated alongside five deep learning models: LSTM, RNN, Bidirectional LSTM, GRU, and CNN. Model performance was assessed using accuracy, precision, recall, and F1-score. SVM achieved the highest accuracy among the machine learning models at 91.0 percent, while Bidirectional LSTM showed the strongest performance among the deep learning models and demonstrated good generalization on unseen data. The findings also show that class imbalance negatively affected positive sentiment recall across several models. Overall, this study provides a comparative evaluation of machine learning and deep learning approaches for real-world consumer sentiment analysis and highlights the importance of appropriate model selection and preprocessing for customer experience analytics in the retail coffee sector.
Reference graph
Works this paper leans on
-
[1]
K. Meng, M. S. Mahapatra, and J. J. Xiao, “Artificial Intelligence and Consumer Financial Behavior: A Systematic Literature Review and Agenda for Future Research,” J. Consum. Behav. , p. cb.2497, Apr. 2025, doi: 10.1002/cb.2497
-
[2]
D. Ivanov, A. Tsipoulanidis, and J. Schönberger, Global Supply Chain and Operations Management: A Decision- Oriented Introdu ction to the Creation of Value . in Springer Texts in Business and Economics. Cham: Springer International Publishing, 2021. doi: 10.1007/978-3-030-72331-6
-
[3]
Sentiment Analysis of Pr oduct Reviews Using Machine Learning and Pre -Trained LLM,
P. S. Ghatora, S. E. Hosseini, S. Pervez, M. J. Iqbal, and N. Shaukat, “Sentiment Analysis of Pr oduct Reviews Using Machine Learning and Pre -Trained LLM,” Big Data Cogn. Comput. , vol. 8, no. 12, p. 199, Dec. 2024, doi: 10.3390/bdcc8120199
-
[4]
P. P. Putra, “Optimizing Sentiment Analysis on Imbalanced Hotel Review Data Using SMOTE and Ensemble Machine Learning Techniques,” J. Appl. Data Sci., vol. 6, no. 2, pp. 921 –935, May 2025, doi: 10.47738/jads.v6i2.618
-
[5]
Using machine learning to develop customer insights from user-generated content,
M. Mustak, H. Hallikainen, T. Laukkanen, L. Plé, L. D. Hollebeek, and M. Aleem, “Using machine learning to develop customer insights from user-generated content,” J. Retail. Consum. Serv. , vol. 81, p. 104034, Nov. 2024, doi: 10.1016/j.jretconser.2024.104034
arXiv 2024
-
[6]
SenticNet 6: Ensemble Application of Symbolic and Subsymbolic AI for Sentiment Analysis,
E. Cambria, Y. Li, F. Z. Xing, S. Poria, and K. Kwok, “SenticNet 6: Ensemble Application of Symbolic and Subsymbolic AI for Sentiment Analysis,” in Proceedings of the 29th ACM International Conference on Information & Knowledge Management , Virtual E vent Ireland: ACM, Oct. 2020, pp. 105 –114. doi: 10.1145/3340531.3412003
arXiv 2020
-
[7]
B. I. Adekunle, E. C. Chukwuma -Eke, E. D. Balogun, and K. O. Ogunsola, “Sentiment Analysis for Customer Behavior Insights: A Natural Language Processing Approach to Business Decision-Making,” Int. J. Soc. Sci. Except. Res. , vol. 3, no. 1, pp. 272 –282, 2024, doi: 10.54660/IJSSER.2024.3.1.272-282
-
[8]
A systematic study of the class imbalance problem in convolutional neural networks,
M. Buda, A. Maki, and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional neural networks,” Neural Netw., vol. 106, pp. 249 –259, Oct. 2018, doi: 10.1016/j.neunet.2018.07.011
Show all 24 references
-
[9]
Magnets for Sarcasm: Making Sarcasm Detection Timely, Contextual and Very Personal,
A. Ghosh and T. Veale, “Magnets for Sarcasm: Making Sarcasm Detection Timely, Contextual and Very Personal,” in Proceedings of the 2017 Conference on Empirical Methods in Natura l Language Processing, Copenhagen, Denmark: Association for Computational Linguistics, 2017, pp. 48...
2017 doi
-
[10]
Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,
C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” Nat. Mach. Intell., vol. 1, no. 5, pp. 206– 215, May 2019, doi: 10.1038/s42256-019-0048-x
2019 doi
-
[11]
Sentiment Analysis using various Machine Learning and Deep Learning Techniques,
V. Umarani, A. Julian, and J. Deepa, “Sentiment Analysis using various Machine Learning and Deep Learning Techniques,” J. Niger. Soc. Phys. Sci., pp. 385– 394, Nov. 2021, doi: 10.46481/jnsps.2021.308
2021 doi
-
[12]
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank,
R. Socher et al., “Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank,” in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, D. Yarowsky, T. Baldwin, A. Korhonen, K. Livescu, and S. Bethard, Eds., Seattle, Washin...
2013
-
[14]
Thumbs up? Sentiment Classification using Machine Learning Techniques,
B. Pang, L. Lee, and S. Vaithyanathan, “Thumbs up? Sentiment Classification using Machine Learning Techniques,” 2002, arXiv. doi: 10.48550/ARXIV.CS/0205070
2002 doi
- [15]
- [16]
-
[17]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,
J. Devlin, M. -W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North, Minneapolis, Minnesota: Association for Computational Linguistics, 2019, pp. 4171–4186. do...
2019 doi
-
[18]
Sentiment Analysis of IMDB Movie Reviews
“Sentiment Analysis of IMDB Movie Reviews.” Accessed: Jun. 18, 2025. [Online]. Available: https://kaggle.com/code/lakshmi25npathi/sentiment- analysis-of-imdb-movie-reviews
2025
-
[19]
Sentiment Analysis on the Yelp Reviews Dataset
“Sentiment Analysis on the Yelp Reviews Dataset.” Accessed: Jun. 18, 2025. [Online]. Available: https://kaggle.com/code/omkarsabnis/sentiment- analysis-on-the-yelp-reviews-dataset
2025
-
[20]
A Deep Text Mining-Based Cosmetics Consumer Sentiment Analysis Model,
X. Zou, “A Deep Text Mining-Based Cosmetics Consumer Sentiment Analysis Model,” in 2025 4th International Symposium on Computer Applications and Information Technology (ISCAIT) , Xi’an, China: IEEE, Mar. 2025, pp. 246 –249. doi: 10.1109/ISCAIT64916.2025.11010384
2025
-
[22]
Social Media Sentiment Analysis for Airline Customer Satisfaction,
S. Chaudhary, K. Garg, P. Gupta, and A. K. Dwivedi, “Social Media Sentiment Analysis for Airline Customer Satisfaction,” in 2025 3rd International Conference on Disruptive Technologies (ICDT), Greater Noida, India: IEEE, Mar. 2025, pp. 1519–1522. doi: 10.1109/ICDT63985.2025.10986313
2025
-
[23]
Improvising E-Commerce Sentiment Analysis with Hybrid VADER-BERT Ensemble Model,
S. Sharma and N. Desai, “Improvising E-Commerce Sentiment Analysis with Hybrid VADER-BERT Ensemble Model,” in 2024 International Conference on Computer Engineering, Network, and Intelligent Multimedia (CENIM), Surabaya, Indonesia: IEEE, Nov. 2024, pp. 1–8. doi: 10.1109/CENIM64...
2024
-
[24]
Sentiment Analysis of P2P Lending Fintech Service User Comments Using CNN-ROS-NCL on Imbalanced Data,
A. D. Widiantoro, M. Mustafid, and R. Sanjaya, “Sentiment Analysis of P2P Lending Fintech Service User Comments Using CNN-ROS-NCL on Imbalanced Data,” in 2024 8th International Conference on Information Technology (InCIT), Chonburi, Thailand: IEEE, Nov. 2024, pp. 787–792. doi:...
2024
-
[25]
Starbucks Reviews Dataset
H. H, “Starbucks Reviews Dataset.” Accessed: Jun. 18,
-
[2024]
Available: https://www.kaggle.com/datasets/harshalhonde/starbuck s-reviews-dataset
[Online]. Available: https://www.kaggle.com/datasets/harshalhonde/starbuck s-reviews-dataset
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.