{"id":"cfc78397-d885-4be4-b10c-be3a5d3aa862","arxiv_id":"2505.04135","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Chain-of-thought prompting raised GPT-4's granular sentiment classification accuracy on 2,000 Amazon app reviews from 84% to 93%, though the paper's own numbers and example data are inconsistent.","lead":"This paper tests whether asking a large language model to reason step-by-step before rating the sentiment of app store reviews improves accuracy. It reports an improvement from 84% to 93% on 2,000 Amazon app reviews, but the evidence contains internal contradictions and data provenance problems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 accuracies (84%, 93%) are arithmetically inconsistent with Section 5.4's claim that CoT corrected 64 of 80 simple-prompt errors; the central 9-point gain is not internally coherent.","rationale":"The reader's REJECT verdict is appropriate, but the single most load-bearing concern is not the weakest assumption the reader named (test-set contamination). The contamination concern is real and important, but it depends on an inference about how the keyword hints were refined and whether that refinement used the evaluation labels; that pathway is somewhat inferential. The arithmetic inconsistency in Section 5.4 is self-contained: it requires only the paper's own numbers. Three quantities cannot all be true: 84% accuracy on 2,000 reviews (320 errors), 93% CoT accuracy (140 errors), and CoT correcting 80% of simple-prompt errors (if simple-prompt errors are 320, the corrected subset should be 256, leaving 64 CoT errors, not 140). This does not merely weaken the 9-point effect size; it makes the reported accuracies and the error analysis mutually contradictory, so the central empirical claim is not reliably established. The provenance issues and non-app-review examples in the paper reinforce the rejection but are secondary to this internal inconsistency. The reader did flag the arithmetic in their rationale, so there is partial agreement, even though their formal weakest_assumption field points to contamination. The recommendation remains REJECT, hence UNCHANGED relative to the reader's verdict.","tokens_in":5316,"tokens_out":2539,"duration_ms":24827,"concrete_test":"Request from the authors the raw per-review predictions for both prompting methods, the human labels, and the exact prompt templates, then recompute the full confusion matrices on all 2,000 reviews. Specifically verify whether simple prompting produced 320 errors, as 84% accuracy requires, and whether CoT corrected 256 of those errors, as the '80% of the misclassified reviews (about 64 out of 80)' statement would require at the reported error scale. If the error counts are actually 80 and 64, then Table 1's accuracies are wrong; if Table 1 is exact, then Section 5.4's error denominator is wrong. Either way, the claimed 9-point improvement must be re-reported with mutually consistent counts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CoT prompting raises granular sentiment classification accuracy from 84% to 93% on 2,000 Amazon app reviews. The paper's own error analysis makes this claim internally incoherent. At 84% accuracy, simple prompting makes 320 errors (16% of 2,000). Section 5.4 states that CoT 'correctly predicted about 64 out of 80' errors made by simple prompting. That sentence implies simple prompting made 80 errors (4%, i.e. 96% accuracy) and, if 20% of errors persisted, that CoT made 16 errors (0.8%, i.e. 99.2% accuracy). Neither number matches Table 1. Conversely, if the reported accuracies are exact, simple prompting has 320 errors and CoT has 140 errors. For CoT to have corrected 80% of simple-prompt errors, it would need to fix 256 of 320 errors, leaving only 64 errors, not 140; for 140 CoT errors to represent 20% of the original errors, simple prompting would need 700 errors, i.e. 65% accuracy, not 84%. No assignment of the stated counts satisfies all three reported numbers simultaneously. Because the headline result is an accuracy comparison, this arithmetic contradiction directly undermines the claim that CoT improved accuracy from 84% to 93%. The additional risk of evaluation-set contamination from 'manual annotation of the dataset' in Section 4.2 compounds the problem, but the internal inconsistency alone is sufficient to invalidate the reported effect size.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether Chain-of-Thought (CoT) prompting improves granular five-class sentiment classification of 2,000 Amazon App store reviews using GPT-4. The authors report that CoT prompting achieves 93% accuracy versus 84% for a simple prompt and present a qualitative error analysis. They attribute the improvement to explicit reasoning steps and keyword hints in the CoT prompt.","tokens_in":5629,"tokens_out":4230,"duration_ms":37842,"significance":"If the reported effect were valid, the result would be a useful demonstration that structured prompting can substantially improve fine-grained sentiment classification in app reviews, a task with practical applications. The paper includes human annotation of the test set by three reviewers and a qualitative error analysis. However, the evaluation has serious methodological flaws that preclude drawing this conclusion.","major_comments":[{"comment":"The claim that CoT corrected 'about 64 out of 80' misclassified reviews is arithmetically inconsistent with the reported accuracies. With 2,000 reviews, simple prompting at 84% accuracy makes 320 errors, not 80. If CoT corrected 80% of those 320 errors (256 reviews), it would leave 64 errors, implying 96.8% accuracy, not the reported 93% (140 errors). Conversely, if CoT made 140 errors and they represent 20% of simple prompting's errors, simple prompting would have 700 errors (65% accuracy), contradicting 84%. No assignment of the stated counts satisfies all three reported numbers. Since the headline result is an accuracy comparison, this contradiction invalidates the reported effect size.","section":"§5.4 and Table 1"},{"comment":"The CoT prompt's keyword hints (e.g., 'easy', 'love', 'amazing', 'slow', 'buggy') were 'refined through manual annotation of the dataset'. This means the prompt was tuned using the same 2,000 reviews on which the model is then evaluated. The paper reports no held-out set, cross-validation, or separate development/test split. Consequently, the measured 93% accuracy includes the effect of test-set-specific prompt engineering and does not provide an unbiased estimate of out-of-sample performance.","section":"§4.2"},{"comment":"The example reviews used in Sections 4.1, 4.2, and 5.2 are not from the Amazon App store. Section 4.1 uses a review of a book, Section 5.2 includes 'The scenery was breathtaking, but the guide was rude' (a travel review) and 'The movie had stunning visuals' (a film review). If these examples are meant to illustrate the model's behavior on the actual dataset, this discrepancy suggests the dataset is not as described, or the examples are fabricated. This undermines the validity of the qualitative error analysis and the claim that the evaluation is on app store reviews.","section":"§3, §4.1, §5.2"}],"minor_comments":[{"comment":"No confidence intervals or significance tests are provided for the accuracies in Table 1, so readers cannot assess whether the 9-point difference is within sampling noise.","section":"§5"},{"comment":"The sentence 'These patterns refined through manual annotation of the dataset set' contains a duplicated word ('dataset set').","section":"§4.2"},{"comment":"The 'Manual Intervention Needed' item describes an error where the model's initial rating was revised after human review; this should be clarified as a limitation of the autonomous system rather than a final classification error.","section":"§5.3"},{"comment":"The abstract and Table 1 report 93% accuracy, but the conclusion says 'up to 93%'; please make the reported number consistent.","section":"§5 and Conclusion"},{"comment":"Reference [12] is a ResearchGate preprint without a DOI or peer-review information; the citation should be updated to a permanent venue or the lack of archival status should be noted.","section":"References"},{"comment":"The paper does not report the label distribution, star-rating breakdown, or inter-annotator agreement (e.g., Cohen's kappa) for the 2,000-review dataset, making it impossible to assess gold-label reliability.","section":"§3"}],"recommendation":"reject","confidential_remarks":"The manuscript resembles a preliminary workshop-style empirical study, but the internal arithmetic inconsistency and the apparent test-set contamination in the prompt design are load-bearing issues that cannot be resolved with minor edits. A complete re-evaluation with proper held-out data and consistent reporting would be needed before the claims could be considered credible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the central claim doesn't survive the paper's own arithmetic, and there's enough methodological leakage that the 84-to-93 number shouldn't be trusted. But it's not a nonsense paper—the setup is clear and the prompts are fully specified.\n\nThe good: the paper defines a five-class sentiment taxonomy, uses 2,000 Amazon app reviews, compares two prompt templates on GPT-4, and reports human-majority labels. The CoT prompt is printed in full, and the qualitative error analysis is easy to follow. The writing is plain and the related work is adequate; they cite CoTAM and PEAR, so they know CoT applied to sentiment is not new.\n\nThe soft spots are load-bearing. First, the numbers. 84% of 2,000 means 320 simple-prompt errors; 93% means 140 CoT errors. Section 5.4 says CoT correctly predicted about 64 out of 80 misclassified reviews (i.e., simple errors ≈ 80) and the remaining 20% persisted (CoT errors ≈ 16). You can't make those three statements true at the same time. Pick any two and the third fails. This kills the credibility of the headline gain.\n\nSecond, the prompt was tuned on the same 2,000 reviews. Section 4.2 says the keyword hints were 'refined through manual annotation of the dataset set'—that is direct evaluation-set information leaking into the prompt. Third, the examples in Sections 4 and 5 are about a book, a movie, scenery, and tutorials—not Amazon app reviews. That makes me wonder whether the qualitative analysis actually matches the dataset, or whether the examples were written to illustrate the taxonomy.\n\nFourth, there are no error bars or inter-annotator agreement numbers, and temperature is set to 0.3 but there's no seed or repeated sampling. Section 5.3 even mentions 'Manual Intervention Needed' to correct CoT outputs, which is not a clean comparison unless those interventions are clearly excluded from the accuracy numbers. The paper doesn't say.\n\nWhat's genuinely new? Not much. CoT for sentiment is established; the dataset is new but also not released. A careful reader might still find the prompt design useful as a starting point, but the evidence for the 9-point gain is not there.\n\nWho is this for? Someone looking for a quick prompt recipe for fine-grained app-review tagging. They'd need to redo the evaluation properly first. This draft should not go to peer review; it needs its arithmetic corrected, the evaluation protocol cleaned up, and the data released before it's a serious empirical contribution. I'd desk-reject the current version.","headline":"The 9-point CoT gain is arithmetically inconsistent with the paper's own error analysis, and the prompt was tuned on the evaluation set, so the headline result should not be trusted.","tokens_in":6100,"tokens_out":3259,"would_cite":false,"duration_ms":29799,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chain-of-thought prompting raises GPT-4's granular sentiment accuracy on Amazon app reviews from 84% to 93%.","keywords":["chain-of-thought prompting","granular sentiment classification","app store reviews","GPT-4","prompt engineering","sentiment analysis","Amazon app reviews","accuracy comparison"],"falsifier":"Run the exact CoT prompt on a held-out set of reviews the authors never inspected while building the keyword hints; if accuracy falls to roughly the simple-prompt level (around 84%), the reported gain was an artifact of test-set familiarity.","tokens_in":5115,"feed_emoji":"🧠","tokens_out":3878,"duration_ms":32712,"temperature":0.7,"pith_summary":"This paper claims that asking GPT-4 to reason step by step through an app review before assigning a 1-to-5 sentiment score produces markedly better granular sentiment labels than asking for a rating directly. On 2,000 Amazon app reviews, the chain-of-thought prompt reached 93% agreement with human judges, versus 84% for the simple prompt. The argument is that explicit reasoning steps—identifying positive and negative expressions, weighing the dominant tone, and justifying the rating—help the model handle mixed or ambiguous reviews that defeat direct prompting. If accurate, the result gives an inexpensive, prompt-only recipe for improving fine-grained sentiment analysis without retraining.","feed_headline":"Step-by-step prompts lift GPT-4 sentiment ratings from 84% to 93%","feed_subtitle":"Adding explicit reasoning steps and keyword hints helps GPT-4 classify nuanced app reviews more like human judges.","key_machinery":"The central object is the chain-of-thought prompt: a five-step instruction sequence that forces the model to parse sentiment-bearing expressions and weigh them before rating. Its distinctive component is a set of keyword hints—words like 'easy,' 'love,' 'amazing,' and 'convenient' for positive sentiment and 'slow,' 'buggy,' 'frustrating,' and 'unreliable' for negative sentiment—which the authors say were refined through manual annotation of the dataset. These hints anchor the model's reasoning to concrete textual cues and are what the paper credits for aligning model ratings with human judgment.","core_discovery":"The central discovery is that a five-step chain-of-thought prompt, enriched with keyword hints for sentiment-bearing language, improves GPT-4's granular sentiment classification accuracy on Amazon app reviews from 84% to 93% when measured against majority-vote human labels. The model is asked to read the review, identify specific positive and negative expressions, evaluate the dominant tone, assign a rating from 1 (Very Negative) to 5 (Very Positive), and justify the rating. The authors show that this structured format corrects about 80% of the errors made by the simple prompt, particularly for reviews with conflicting or layered sentiments. The remaining errors involve vague expressions, sarcasm, and very short reviews.","pith_inferences":["The paper's stated arithmetic is internally inconsistent: 84% accuracy on 2,000 reviews implies roughly 320 errors, not the 'about 80' the error-correction section describes; a direct review-by-review trace would settle which numbers are right.","Because the keyword hints were refined on the same 2,000 reviews used for evaluation, a natural extension is to test CoT with generic hints versus dataset-specific hints on a fresh corpus to measure how much of the gain comes from the prompt structure versus tuned vocabulary.","The 9-point gain is demonstrated on GPT-4 only; repeating the same protocol with smaller or open-weight models would show whether the benefit transfers or is tied to the model's reasoning strength."],"forward_implications":["Organizations analyzing app-store feedback can adopt the CoT prompt without retraining or labeled data beyond the prompt text, and expect about nine percentage points better agreement with human sentiment ratings.","The method's benefit concentrates on mixed-sentiment reviews, where simple prompting defaults to the sentiment with more words; CoT's reasoning step corrects the majority of those errors.","The remaining error classes—vague language, sarcasm, and very short reviews—mark the boundary of prompt-only CoT for this task.","Adding keyword hints to CoT prompts is a transferable design choice, since the same pattern could apply to other domains with known sentiment-bearing vocabulary."],"supporting_citations":[{"why":"Introduces chain-of-thought prompting, the method under evaluation.","marker":"[2]"},{"why":"Evaluates GPT-4's ability to extract feature-sentiment pairs from app reviews, motivating the choice of model and domain.","marker":"[7]"},{"why":"Shows that zero-shot prompts can elicit reasoning chains, supporting the prompt-only, no-examples design.","marker":"[8]"},{"why":"Defines rating-scale sentiment categorization, the framework the granular labels extend.","marker":"[1]"},{"why":"Provides earlier fine-grained sentiment analysis of app reviews, establishing the task context.","marker":"[3]"}],"fun_headline_variants":["CoT prompting lifts GPT-4 sentiment accuracy to 93% on app reviews","Step-by-step reasoning lets GPT-4 nail 93% of granular app sentiments","CoT prompts give GPT-4 a 9-point edge in sentiment accuracy","Explicit reasoning steps turn GPT-4 into a sharper sentiment rater","Chain-of-thought prompts boost GPT-4's fine-grained app ratings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The keyword hints were refined by manually examining the same 2,000 reviews used for evaluation, so the 93% figure assumes this refinement did not effectively tune the prompt to the test set.","fun_headline_variants_meta":{"raw":{"variants":["CoT prompting lifts GPT-4 sentiment accuracy to 93% on app reviews","Step-by-step reasoning lets GPT-4 nail 93% of granular app sentiments","CoT prompts give GPT-4 a 9-point edge in sentiment accuracy","Explicit reasoning steps turn GPT-4 into a sharper sentiment rater","Chain-of-thought prompts boost GPT-4's fine-grained app ratings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001284,"raw_usage":{"total_tokens":5158,"prompt_tokens":768,"completion_tokens":4390,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":384,"completion_tokens_details":{"reasoning_tokens":4289}},"tokens_in":384,"tokens_out":4390,"duration_ms":29103,"temperature":1.0,"reasoning_tokens":4289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:35:54.154027+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact CoT prompt on a held-out set of reviews the authors never inspected while building the keyword hints; if accuracy falls to roughly the simple-prompt level (around 84%), the reported gain was an artifact of test-set familiarity.","supporting_citations":[{"cited_title":"Chain of thought prompting elicits reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Introduces chain-of-thought prompting, the method under evaluation."},{"cited_title":"How Effectively Do LLMs Extract Feature-Sentiment Pairs from App Reviews?","cited_arxiv_id":"2409.07162","evidence_quote":"Evaluates GPT-4's ability to extract feature-sentiment pairs from app reviews, motivating the choice of model and domain."},{"cited_title":"Large language models are zero- shot reasoners,","cited_arxiv_id":null,"evidence_quote":"Shows that zero-shot prompts can elicit reasoning chains, supporting the prompt-only, no-examples design."},{"cited_title":"Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales,","cited_arxiv_id":null,"evidence_quote":"Defines rating-scale sentiment categorization, the framework the granular labels extend."},{"cited_title":"How do users like this feature? a fine-grained sentiment analy- sis of app reviews,","cited_arxiv_id":null,"evidence_quote":"Provides earlier fine-grained sentiment analysis of app reviews, establishing the task context."}],"review_version":1}