REVIEW 4 major objections 6 minor 24 references
Exploring Zero-Shot App Review Classification with ChatGPT: Challenges and Potential
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A zero-shot GPT-4o mini, prompted with an expert requirements-analyst role, an emotional cue, and step-by-step reasoning at temperature 0.2, classifies app reviews as functional, non-functional, both, or neither with micro-F1 0.842.
desk verdict Useful dataset and prompt comparison for app-review classification, but the headline F1 is a selection-on-test estimate that needs a proper held-out evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is Prompt 3, a single-turn prompt that wraps the system instruction "Imagine you are an expert requirements analyst... Provide only the category—no explanations" together with brief definitions of the four classes, the chain-of-thought phrase "Let's think this out in step by step way to be sure we have the right answer," and the emotion cue "This is important to my career." The paper's argument is that combining role prompting, emotion prompting, and chain-of-thought lets a zero-shot model classify without labeled examples, while a low temperature of 0.2 stabilizes the stochastic output. The benchmark's majority-consensus annotation procedure is the other load-bearing element: it converts raw reviews into the ground-truth labels against which every reported F1 number is measured.
What would settle it
Run the exact Prompt 3/temperature-0.2 pipeline on the 120 non-consensus reviews, scored against an independent panel's labels: if the F1 on that set falls substantially below 0.842 (for instance below 0.7), the headline result is being carried by the curated, easy-to-agree reviews rather than by a general zero-shot capability.
Extended reading notes
Core claim
The central discovery is that prompt design and decoding settings, not model fine-tuning, unlock accurate zero-shot app-review classification. On the curated 1,880-review benchmark, built from ten app domains with an inter-annotator agreement of 0.76 and with 120 non-consensus reviews removed, Prompt 3—brief definitions of the four classes, the step-by-step reasoning instruction, and the emotional cue—at temperature 0.2 yields recall 0.848, precision 0.858, and micro-F1 0.842. Per-class results are strong for functional reviews (0.91), non-functional (0.87), and neither (0.94), but weak for the "both" category (0.54). Review length has almost no effect on accuracy, while readability does: correctly classified reviews average an FKGL score of 6.34 versus 9.24 for misclassified ones. Manual error analysis attributes most misclassifications to overlapping FR/NFR characteristics, ambiguous language, negative-sentiment bias, and emotionally charged wording.
Load-bearing premise
The benchmark's ground truth is the majority vote of five annotators, and the 120 reviews they could not agree on were removed before evaluation; if that ground truth is systematically wrong or if dropping the hard cases inflates agreement, every reported F1 is measured against a biased yardstick.
Editorial extensions
If this is right
- A zero-shot prompt can replace trained classifiers for initial app-review triage: on the paper's benchmark, Prompt 3 at temperature 0.2 outperforms every classical baseline by a wide margin with no app-specific training data.
- Because the "both" category is the weak point at F1 0.54, a practical deployment should treat "both" predictions as low-confidence and route them to human review or a second classifier.
- Because readability predicts accuracy, adding a text-simplification stage before classification is a concrete next step with a directly measurable expected payoff.
- Because higher temperatures degrade performance across all tested prompts, production use should fix the temperature low rather than raise it to encourage variability.
Reading between the lines
- A direct test the paper does not run is whether the 120 excluded non-consensus reviews are disproportionately "both"-class; if they are, the four-way taxonomy itself, not the model, would explain much of the reported weakness.
- The negative-sentiment bias could be probed generically by taking paired reviews with the same content but opposite sentiment and measuring how much F1 shifts; this perturbation design would separate emotion effects from wording effects.
- If zero-shot FR/NFR classification transfers across app domains, the same prompt recipe may extend to other software text triage tasks such as issue-tracker bug-versus-feature routing, but that transfer is a hypothesis this paper does not test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper evaluates whether GPT-4o mini can classify mobile app reviews into functional requirements (FR), non-functional requirements (NFR), both, or neither in a zero-shot setting, using a manually annotated benchmark of 1,880 reviews from ten app domains. The authors engineer eight prompts, select three on a 100-review sample, and report micro-F1 on the full dataset across three temperature settings; Prompt 3 (combining role, emotion, and chain-of-thought prompting) at temperature 0.2 yields F1 0.842, outperforming classical ML classifiers on a 470-review test split (F1 0.82). The paper also investigates how review length and readability (FKGL) affect classification, and manually categorizes error patterns. The authors release source code and annotated data via Zenodo.
Significance. If the headline F1 survived a proper held-out protocol, the paper would make a useful empirical contribution: zero-shot LLM classification could reduce the need for domain-specific training data in requirements engineering. The strengths include a publicly released dataset and code, a multi-class benchmark with four categories, and a thoughtful manual error taxonomy that goes beyond numeric reporting. However, the central quantitative claim is currently supported only by a selection-on-test estimate, and the discrepancy between Tables 2 and 3 blurs which number is the held-out result. I therefore see the contribution as potentially valuable but not yet established.
major comments (4)
- [Section 2.2 and Table 2] The experimental protocol selects prompts using a 100-review sample drawn from the same 1,880-review benchmark, then evaluates the selected prompts on the full benchmark. Consequently, the reported F1 0.842 for Prompt 3 at temperature 0.2 is a selection-on-test estimate: the configuration was chosen using labels from the same distribution as the evaluation set, with no disjoint validation split or nested procedure described. This is load-bearing because the abstract's 'robust F1 score of 0.842' is the paper's central claim. Please report a held-out evaluation (e.g., a validation fold used for prompt selection and a test fold never used for selection) or a nested cross-validation scheme, and state the F1 obtained under that protocol.
- [Table 2 vs. Table 3] The comparison with classical ML models reports GPT-4o mini F1 0.82 on a 1410/470 split, while Table 2 reports F1 0.842 for the same configuration (Prompt 3, temperature 0.2). The paper does not explain whether Table 3's 0.82 is the held-out result for the optimal configuration; if it is, the abstract's 0.842 is not the test-set estimate, and if it is not, the paper never reports a held-out ChatGPT result. Please reconcile these numbers and clearly label which evaluation split underlies each table.
- [Section 2.3] The paper acknowledges that LLM outputs are non-deterministic (citing [6]) and varies temperature to explore randomness, but every review is scored with a single API call per configuration, and no repeated runs or confidence intervals are reported. A single point estimate cannot support the word 'robust' in the abstract. Please add multiple runs (e.g., 3-5 repetitions per configuration), report mean and variance, and consider significance tests against the ML baselines in Table 3.
- [Section 2.1] The evaluation's ground truth is the five annotators' majority label on the 1,880 reviews that survived exclusion of 120 non-consensus reviews. Because the excluded reviews are likely the most ambiguous, the reported F1 may overstate performance on live app-store data where no such curation occurs. Please provide class-level kappa, describe the distribution of excluded reviews across the four classes, and, if possible, report model performance on the 120 non-consensus reviews as a sensitivity check.
minor comments (6)
- [Section 2.2, Table 1] The fourth class is named 'None' in Prompts 1 and 2 but 'neither' in the dataset description and elsewhere; this inconsistency should be harmonized for clarity and prompt consistency.
- [Equation (1)] The FKGL formula is typeset with Unicode symbols and ambiguous division; use standard math notation and define all variables explicitly.
- [Tables 2 and 3] Add a table note stating that each reported value is a single API pass (or the mean over runs, after revision), and specify whether Table 3's ChatGPT row was computed on the same 470-review split as the ML models.
- [Abstract] The phrase 'robust F1 score of 0.842' is stronger than the evidence supports; once repeated runs or confidence intervals are added, qualify the claim accordingly, or replace 'robust' with a neutral descriptor.
- [Title page] The title contains 'ChatGP T' with a space; fix this typographical artifact.
- [Section 6] Reference [19] is cited as 'our prior work' in the future-work discussion; if this paper is a direct extension, the relationship should be stated explicitly in the introduction.
Circularity Check
No circularity: the reported F1 scores are empirical measurements against human-annotated labels, not derivations from the model's own outputs.
full rationale
The paper makes no formal derivation; it reports an empirical benchmark evaluation of zero-shot GPT-4o mini on 1,880 manually annotated reviews. The classification labels are human-generated and are not defined in terms of the model's outputs, so no self-definitional circularity is present. The only self-citation ([19]) appears in future work as a suggested review-simplification step and is not load-bearing for the reported F1 scores. The prompt-selection procedure in Section 2.2 uses a 100-review sample from the same curated benchmark to choose the top prompts and later reports Prompt 3's F1 on the complete dataset; this is a legitimate validity concern about selection-on-test evaluation and the absence of a disjoint held-out ChatGPT split, but it is not a circular reduction: the prompt is not fitted to the labels in the sense of a parameter, and the reported score is a measurement rather than a quantity derived from its own inputs. Under the required standard of exhibiting an equation-level or definition-level reduction, no circular step is found.
Assumptions & free parameters
free parameters (1)
- Temperature and prompt configuration =
Temperature 0.2 with Prompt 3
assumptions (4)
- domain assumption Manual labels produced by five software engineers with Fleiss kappa 0.76 are a valid gold standard.
- domain assumption Excluding 120 reviews without majority agreement does not bias the benchmark.
- domain assumption A single API call per review at temperature 0.2 is representative of model performance.
- domain assumption Reviews from ten selected apps represent the diversity of app store reviews.
Cite this review
Pith. "Pith review of Exploring Zero-Shot App Review Classification with ChatGPT: Challenges and Potential." pith.science (2026). https://pith.science/paper/4RPIPSR5
@misc{pith2026250504759,
author = {Pith},
title = {Pith review of: Exploring Zero-Shot App Review Classification with ChatGPT: Challenges and Potential},
year = {2026},
howpublished = {\url{https://pith.science/paper/4RPIPSR5}},
note = {Machine review of arXiv:2505.04759}
}
read the original abstract
App reviews are a critical source of user feedback, offering valuable insights into an app's performance, features, usability, and overall user experience. Effectively analyzing these reviews is essential for guiding app development, prioritizing feature updates, and enhancing user satisfaction. Classifying reviews into functional and non-functional requirements play a pivotal role in distinguishing feedback related to specific app features (functional requirements) from feedback concerning broader quality attributes, such as performance, usability, and reliability (non-functional requirements). Both categories are integral to informed development decisions. Traditional approaches to classifying app reviews are hindered by the need for large, domain-specific datasets, which are often costly and time-consuming to curate. This study explores the potential of zero-shot learning with ChatGPT for classifying app reviews into four categories: functional requirement, non-functional requirement, both, or neither. We evaluate ChatGPT's performance on a benchmark dataset of 1,880 manually annotated reviews from ten diverse apps spanning multiple domains. Our findings demonstrate that ChatGPT achieves a robust F1 score of 0.842 in review classification, despite certain challenges and limitations. Additionally, we examine how factors such as review readability and length impact classification accuracy and conduct a manual analysis to identify review categories more prone to misclassification.
Reference graph
Works this paper leans on
-
[6]
Prompt engineering with chatgpt: a guide fo r academic writers
Louie Giray. Prompt engineering with chatgpt: a guide fo r academic writers. Annals of biomedical engineering , 51(12):2629–2633, 2023
work page 2023
-
[1]
Ulrike Abelein, Helen Sharp, and Barbara Paech. Does inv olving users in soft- ware development really influence system success? IEEE software, 30(6):17–23, 2013
work page 2013
-
[2]
Identification and Classification of Non-Functional Requirements from User Re views
Md Ashif Aziz, Nafisa Mehjabin, and Khandaker Rifah Tasni a. Identification and Classification of Non-Functional Requirements from User Re views. PhD thesis, Department of Computer Science and Engineering (CSE), Isla mic University of Technology (IUT), 2022
work page 2022
-
[3]
User involvement in softw are development and system success: A systematic literature review
Muneera Bano and Didar Zowghi. User involvement in softw are development and system success: A systematic literature review. 2013
work page 2013
-
[4]
Mining and searching app reviews for requirements engineering: Evaluation and replication studies
Jacek Dąbrowski, Emmanuel Letier, Anna Perini, and Ange lo Susi. Mining and searching app reviews for requirements engineering: Evaluation and replication studies. Information Systems, 114:102181, 2023
work page 2023
-
[5]
Automatically clas- sifying requirements from app stores: A preliminary study
Roger Deocadez, Rachel Harrison, and Daniel Rodriguez. Automatically clas- sifying requirements from app stores: A preliminary study. In 2017 IEEE 25th international requirements engineering conference works hops (REW), pages 367–
work page 2017
-
[7]
Mohammad A Hadi and Fatemeh H Fard. Evaluating pre-train ed models for user feedback analysis in software engineering: A study on c lassification of app- reviews. Empirical Software Engineering, 28(4):88, 2023
work page 2023
-
[8]
A critical review of la rge language models: Sensitivity, bias, and the path toward specialized ai
Arash Hajikhani and Carolyn Cole. A critical review of la rge language models: Sensitivity, bias, and the path toward specialized ai. Quantitative Science Studies, 5(3):736–756, 2024
work page 2024
Show all 24 references
-
[9]
Eval- uation of chatgpt on biomedical tasks: A zero-shot comparis on with fine-tuned generative transformers
Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimm y Huang. Eval- uation of chatgpt on biomedical tasks: A zero-shot comparis on with fine-tuned generative transformers. arXiv preprint arXiv:2306.04504 , 2023
2023 arXiv
-
[10]
Performance compariso n of word embedding methods in text classification for various number of feature s
Elif Kabullar and Ilker Türker. Performance compariso n of word embedding methods in text classification for various number of feature s. In International Istanbul Modern Scientific Research Congress-IV , 2022
2022
-
[11]
Derivation of new readability formulas (automated readabi lity index, fog count and flesch reading ease formula) for navy enlisted personnel
J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers , and Brad S Chissom. Derivation of new readability formulas (automated readabi lity index, fog count and flesch reading ease formula) for navy enlisted personnel . 1975
1975
-
[12]
Better zero-shot reasoning w ith role-play prompting
Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. Better zero-shot reasoning w ith role-play prompting. arXiv preprint arXiv:2308.07702 , 2023
2023 arXiv
-
[13]
As sessment of quality of experience (qoe) of image compression in social cloud com puting
Asif Laghari, Hui He, Muhammad Shafiq, and Asiya Khan. As sessment of quality of experience (qoe) of image compression in social cloud com puting. 2018
2018
-
[14]
Large language models und erstand and can be enhanced by emotional stimuli
Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxi n Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. Large language models und erstand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2307.11760 , 2023
2023 arXiv
-
[15]
Automatic classification of non-functional re- quirements from augmented app user reviews
Mengmeng Lu and Peng Liang. Automatic classification of non-functional re- quirements from augmented app user reviews. 2017
2017
-
[16]
App sto re mining for feature extraction: analyzing user reviews
Zulfiqar Ali Memon, Nida Munawar, and Maha Kamal. App sto re mining for feature extraction: analyzing user reviews. Acta Scientiarum. Technology, 46(1), 2024
2024
-
[17]
Harnessing large language models’ zero-shot and few-shot learning capabili ties for regulatory re- search
Hamed Meshkin, Joel Zirkle, Ghazal Arabidarrehdor, An ik Chaturbedi, Shilpa Chakravartula, John Mann, Bradlee Thrasher, and Zhihua Li. Harnessing large language models’ zero-shot and few-shot learning capabili ties for regulatory re- search. Briefings in Bioinformatics , 25(5):...
2024
-
[18]
Too many user-revi ews! what should app developers look at first? IEEE Transactions on Software Engineering , 47(2):367–378, 2019
Ehsan Noei, Feng Zhang, and Ying Zou. Too many user-revi ews! what should app developers look at first? IEEE Transactions on Software Engineering , 47(2):367–378, 2019
2019
-
[19]
Refining app reviews: Dataset, methodology, and evaluation
Amrita Singh, Chirag Jain, Mohit Chaudhary, and Preeth u Anish. Refining app reviews: Dataset, methodology, and evaluation. InProceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing: In dustry Track, pages 595–608, 2024
2024
-
[20]
Dcnr: A transfe r learning ap- proach for detecting and classifying non-functionalrequi rements from user re- views
Khandaker Rifah Tasnia, Nafisa Mehjabin, Md Ashif Aziz, Abu Raihan Mostofa Kamal, Afrina Khatun, and Lutfun Nahar Lota. Dcnr: A transfe r learning ap- proach for detecting and classifying non-functionalrequi rements from user re- views. 2022
2022
-
[21]
Can app changelogs improve requirements classification fro m app reviews? an exploratory study
Chong Wang, Fan Zhang, Peng Liang, Maya Daneva, and Mart en Van Sinderen. Can app changelogs improve requirements classification fro m app reviews? an exploratory study. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measuremen...
2018
-
[22]
Chain-of-thought prompting eli cits reasoning in large language models, 2022
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma , Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting eli cits reasoning in large language models, 2022
2022
-
[23]
Identification and classificati on of requirements from app user reviews
Hui Yang and Peng Liang. Identification and classificati on of requirements from app user reviews. In SEKE, pages 7–12. Citeseer, 2015
2015
-
[24]
Y. Yao, W. Jiang, Y. Wang, P. Song, and B. Wang. Non-funct ional requirements analysis based on application reviews in the android app mar ket. Information Resources Management Journal, 35:1–17, 2022
2022
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.