REVIEW 3 major objections 6 minor 12 references
Enhancing Granular Sentiment Classification with Chain-of-Thought Prompting in Large Language Models
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Chain-of-thought prompting raises GPT-4's granular sentiment accuracy on Amazon app reviews from 84% to 93%.
desk verdict The 9-point CoT gain is arithmetically inconsistent with the paper's own error analysis, and the prompt was tuned on the evaluation set, so the headline result should not be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the chain-of-thought prompt: a five-step instruction sequence that forces the model to parse sentiment-bearing expressions and weigh them before rating. Its distinctive component is a set of keyword hints—words like 'easy,' 'love,' 'amazing,' and 'convenient' for positive sentiment and 'slow,' 'buggy,' 'frustrating,' and 'unreliable' for negative sentiment—which the authors say were refined through manual annotation of the dataset. These hints anchor the model's reasoning to concrete textual cues and are what the paper credits for aligning model ratings with human judgment.
What would settle it
Run the exact CoT prompt on a held-out set of reviews the authors never inspected while building the keyword hints; if accuracy falls to roughly the simple-prompt level (around 84%), the reported gain was an artifact of test-set familiarity.
Extended reading notes
Core claim
The central discovery is that a five-step chain-of-thought prompt, enriched with keyword hints for sentiment-bearing language, improves GPT-4's granular sentiment classification accuracy on Amazon app reviews from 84% to 93% when measured against majority-vote human labels. The model is asked to read the review, identify specific positive and negative expressions, evaluate the dominant tone, assign a rating from 1 (Very Negative) to 5 (Very Positive), and justify the rating. The authors show that this structured format corrects about 80% of the errors made by the simple prompt, particularly for reviews with conflicting or layered sentiments. The remaining errors involve vague expressions, sarcasm, and very short reviews.
Load-bearing premise
The keyword hints were refined by manually examining the same 2,000 reviews used for evaluation, so the 93% figure assumes this refinement did not effectively tune the prompt to the test set.
Editorial extensions
If this is right
- Organizations analyzing app-store feedback can adopt the CoT prompt without retraining or labeled data beyond the prompt text, and expect about nine percentage points better agreement with human sentiment ratings.
- The method's benefit concentrates on mixed-sentiment reviews, where simple prompting defaults to the sentiment with more words; CoT's reasoning step corrects the majority of those errors.
- The remaining error classes—vague language, sarcasm, and very short reviews—mark the boundary of prompt-only CoT for this task.
- Adding keyword hints to CoT prompts is a transferable design choice, since the same pattern could apply to other domains with known sentiment-bearing vocabulary.
Reading between the lines
- The paper's stated arithmetic is internally inconsistent: 84% accuracy on 2,000 reviews implies roughly 320 errors, not the 'about 80' the error-correction section describes; a direct review-by-review trace would settle which numbers are right.
- Because the keyword hints were refined on the same 2,000 reviews used for evaluation, a natural extension is to test CoT with generic hints versus dataset-specific hints on a fresh corpus to measure how much of the gain comes from the prompt structure versus tuned vocabulary.
- The 9-point gain is demonstrated on GPT-4 only; repeating the same protocol with smaller or open-weight models would show whether the benefit transfers or is tied to the model's reasoning strength.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper investigates whether Chain-of-Thought (CoT) prompting improves granular five-class sentiment classification of 2,000 Amazon App store reviews using GPT-4. The authors report that CoT prompting achieves 93% accuracy versus 84% for a simple prompt and present a qualitative error analysis. They attribute the improvement to explicit reasoning steps and keyword hints in the CoT prompt.
Significance. If the reported effect were valid, the result would be a useful demonstration that structured prompting can substantially improve fine-grained sentiment classification in app reviews, a task with practical applications. The paper includes human annotation of the test set by three reviewers and a qualitative error analysis. However, the evaluation has serious methodological flaws that preclude drawing this conclusion.
major comments (3)
- [§5.4 and Table 1] The claim that CoT corrected 'about 64 out of 80' misclassified reviews is arithmetically inconsistent with the reported accuracies. With 2,000 reviews, simple prompting at 84% accuracy makes 320 errors, not 80. If CoT corrected 80% of those 320 errors (256 reviews), it would leave 64 errors, implying 96.8% accuracy, not the reported 93% (140 errors). Conversely, if CoT made 140 errors and they represent 20% of simple prompting's errors, simple prompting would have 700 errors (65% accuracy), contradicting 84%. No assignment of the stated counts satisfies all three reported numbers. Since the headline result is an accuracy comparison, this contradiction invalidates the reported effect size.
- [§4.2] The CoT prompt's keyword hints (e.g., 'easy', 'love', 'amazing', 'slow', 'buggy') were 'refined through manual annotation of the dataset'. This means the prompt was tuned using the same 2,000 reviews on which the model is then evaluated. The paper reports no held-out set, cross-validation, or separate development/test split. Consequently, the measured 93% accuracy includes the effect of test-set-specific prompt engineering and does not provide an unbiased estimate of out-of-sample performance.
- [§3, §4.1, §5.2] The example reviews used in Sections 4.1, 4.2, and 5.2 are not from the Amazon App store. Section 4.1 uses a review of a book, Section 5.2 includes 'The scenery was breathtaking, but the guide was rude' (a travel review) and 'The movie had stunning visuals' (a film review). If these examples are meant to illustrate the model's behavior on the actual dataset, this discrepancy suggests the dataset is not as described, or the examples are fabricated. This undermines the validity of the qualitative error analysis and the claim that the evaluation is on app store reviews.
minor comments (6)
- [§5] No confidence intervals or significance tests are provided for the accuracies in Table 1, so readers cannot assess whether the 9-point difference is within sampling noise.
- [§4.2] The sentence 'These patterns refined through manual annotation of the dataset set' contains a duplicated word ('dataset set').
- [§5.3] The 'Manual Intervention Needed' item describes an error where the model's initial rating was revised after human review; this should be clarified as a limitation of the autonomous system rather than a final classification error.
- [§5 and Conclusion] The abstract and Table 1 report 93% accuracy, but the conclusion says 'up to 93%'; please make the reported number consistent.
- [References] Reference [12] is a ResearchGate preprint without a DOI or peer-review information; the citation should be updated to a permanent venue or the lack of archival status should be noted.
- [§3] The paper does not report the label distribution, star-rating breakdown, or inter-annotator agreement (e.g., Cohen's kappa) for the 2,000-review dataset, making it impossible to assess gold-label reliability.
Circularity Check
The CoT prompt's keyword hints were refined on the same 2,000-review evaluation set, so part of the 84% to 93% accuracy gain is a test-set fit rather than an unbiased prediction; Section 5.4's error counts are also arithmetically inconsistent with Table 1.
-
fitted input called prediction
[Section 4.2 (CoT prompt design) vs. Section 3 and Section 5/Table 1 (evaluation on the same 2,000 reviews)]
"We incorporated keyword hints into the CoT prompt design to guide the model in identifying sentiment-bearing expressions. These patterns refined through manual annotation of the dataset set—played a key role in aligning the model's reasoning with human judgment. ... CoT prompting reached a notable 93% accuracy, validating the impact of detailed reasoning steps."
The CoT prompt is not a fixed, independently specified method: its keyword hints and sentiment patterns were refined through manual annotation of the dataset on which the model is later scored. Section 3 defines the dataset as 2,000 annotated Amazon app reviews, and Section 5 reports accuracies on the same 2,000 reviews. Thus the CoT condition enters the evaluation with information extracted from the test set (which words and tones signal positive, negative, or neutral sentiment in these particular reviews), while the simple-prompt condition receives no such tuning.
full rationale
The only load-bearing circular step is in the design of the CoT prompt. The paper explicitly says the keyword hints were 'refined through manual annotation of the dataset set' (Section 4.2), and that same dataset — 2,000 Amazon app reviews annotated by human reviewers — is the dataset on which Table 1 reports 93% accuracy. The CoT condition therefore enters the evaluation with information extracted from the evaluation set, while the simple-prompt baseline does not. This makes the 84%-to-93% comparison a partially fitted-input-called-prediction result. I assign a score of 6 rather than higher because the CoT prompt also contains a generic step-by-step reasoning instruction that could transfer to other reviews, and there is no evidence that the entire result is definitionally forced or reduced entirely to a self-citation chain. No other circularity patterns are present: the paper contains no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. Separately, but not as a circularity issue, Section 5.4's claim that CoT corrected 'about 64 out of 80' simple-prompt errors is arithmetically inconsistent with the reported 84% and 93% accuracies on 2,000 reviews; that inconsistency is a correctness and internal-coherence problem rather than a derivation-that-reduces-to-its-inputs problem, but it compounds the concern that the reported effect size is not reliable.
Assumptions & free parameters
free parameters (1)
- CoT prompt keyword hints =
positive: easy, love, amazing, convenient; negative: slow, frustrating, unreliable, buggy; neutral: okay, average
assumptions (3)
- domain assumption Human majority-vote labels on 2,000 reviews are the ground truth for sentiment.
- ad hoc to paper The 2,000 reviews are from the Amazon App store.
- ad hoc to paper The CoT prompt's keyword hints generalize to unseen reviews.
Cite this review
Pith. "Pith review of Enhancing Granular Sentiment Classification with Chain-of-Thought Prompting in Large Language Models." pith.science (2026). https://pith.science/paper/RQUENKDN
@misc{pith2026250504135,
author = {Pith},
title = {Pith review of: Enhancing Granular Sentiment Classification with Chain-of-Thought Prompting in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/RQUENKDN}},
note = {Machine review of arXiv:2505.04135}
}
read the original abstract
We explore the use of Chain-of-Thought (CoT) prompting with large language models (LLMs) to improve the accuracy of granular sentiment categorization in app store reviews. Traditional numeric and polarity-based ratings often fail to capture the nuanced sentiment embedded in user feedback. We evaluated the effectiveness of CoT prompting versus simple prompting on 2000 Amazon app reviews by comparing each method's predictions to human judgements. CoT prompting improved classification accuracy from 84% to 93% highlighting the benefit of explicit reasoning in enhancing sentiment analysis performance.
Reference graph
Works this paper leans on
-
[1]
B. Pang and L. Lee, “Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales,” in Proceedings of the 43rd annual meeting of the association for computational linguistics (ACL’05), pp. 115– 124, 2005
work page 2005
-
[2]
Chain of thought prompting elicits reasoning in large language models,
J. Wei, X. Wang, D. Schuurmans, M. Bosma, V. Zhao, K. Guu, et al. , “Chain of thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems , vol. 35, pp. 2483–2494, 2022
work page 2022
-
[3]
How do users like this feature? a fine-grained sentiment analy- sis of app reviews,
E. Guzman and W. Maalej, “How do users like this feature? a fine-grained sentiment analy- sis of app reviews,” in 2015 IEEE 23rd Inter- national Requirements Engineering Conference (RE), pp. 153–162, IEEE, 2015
work page 2015
-
[4]
Scare: The sentiment corpus of app reviews with fine-grained annotations,
A. Gupta, M. Gupta, V. Singh, and P. Ku- maraguru, “Scare: The sentiment corpus of app reviews with fine-grained annotations,” in Pro- ceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), 2016
work page 2016
-
[5]
Senticnet 5: Discovering conceptual primi- tives for advancing sentiment analysis,
E. Cambria, S. Poria, D. Hazarika, and K. Kwok, “Senticnet 5: Discovering conceptual primi- tives for advancing sentiment analysis,” AAAI, vol. 34, no. 05, pp. 5076–5083, 2020
work page 2020
-
[6]
Language models are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, et al. , “Language models are few-shot learners,” in Advances in Neural In- formation Processing Systems, vol. 33, pp. 1877– 1901, 2020
work page 1901
-
[7]
How Effectively Do LLMs Extract Feature-Sentiment Pairs from App Reviews?
F. A. Shah, A. Sabir, R. Sharma, and D. Pfahl, “How effectively do llms extract feature-sentiment pairs from app reviews?,” arXiv preprint arXiv:2409.07162 , 2024
work page Pith review arXiv 2024
-
[8]
Large language models are zero- shot reasoners,
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero- shot reasoners,” in NeurIPS, 2022
work page 2022
Show all 12 references
-
[9]
Controllable data augmentation for few-shot text mining with chain-of-thought attribute manipulation,
L. Peng, Y. Zhang, and J. Shang, “Controllable data augmentation for few-shot text mining with chain-of-thought attribute manipulation,” arXiv preprint arXiv:2307.07099, 2023
2023 arXiv
-
[10]
Least- to-most prompting enables complex reason- ing in large language models,
D. Zhou, N. Sch¨ arli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bous- quet, Q. V. Le, and E. H. Chi, “Least- to-most prompting enables complex reason- ing in large language models,” arXiv preprint arXiv:2205.10625, 2022
2022 arXiv
-
[11]
Multimodal pear chain-of-thought reasoning for multimodal senti- ment analysis,
Y. Zhang, R. Chen, and Q. Li, “Multimodal pear chain-of-thought reasoning for multimodal senti- ment analysis,” Proceedings of the ACM on Mul- timedia, vol. 2024, no. 10, pp. 1–12, 2024
2024
-
[12]
Evaluating chatgpt-4 and machine learning models for sentiment analysis on a multi-script moroccan arabic corpus,
M. Hannani, A. Soudi, and K. Van Laer- hoven, “Evaluating chatgpt-4 and machine learning models for sentiment analysis on a multi-script moroccan arabic corpus,” Re- searchGate Preprint , 2025. Available at: https://www.researchgate.net/publication/390432057. 5
2025
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.