Pith. sign in

REVIEW 4 major objections 6 minor 24 references

Exploring Zero-Shot App Review Classification with ChatGPT: Challenges and Potential

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read A zero-shot GPT-4o mini, prompted with an expert requirements-analyst role, an emotional cue, and step-by-step reasoning at temperature 0.2, classifies app reviews as functional, non-functional, both, or neither with micro-F1 0.842.

desk verdict Useful dataset and prompt comparison for app-review classification, but the headline F1 is a selection-on-test estimate that needs a proper held-out evaluation. read the letter →

arxiv 2505.04759 v1 pith:4RPIPSR5 submitted 2025-05-07 cs.SE cs.AI

classification cs.SEcs.AI
keywords appreviewclassificationzero-shotlearningChatGPTpromptengineeringrequirementfunctionalrequirementsnon-functionallargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a large language model can sort app-store reviews into requirement categories—functional (what the app should do), non-functional (how well it performs), both, or neither—without being given any domain-specific training examples. It reports that a zero-shot GPT-4o mini, using a prompt that combines role assignment, an emotional cue, and chain-of-thought instructions at temperature 0.2, reaches a micro-F1 of 0.842 on 1,880 manually annotated reviews from ten apps. The same model outperforms classical machine-learning classifiers such as random forest, SVM, XGBoost, and logistic regression even though those models were trained on labeled data. If the result holds, app teams could automate the first pass of requirements triage on user feedback without curating expensive, domain-specific labeled datasets.

What carries the argument

The load-bearing object is Prompt 3, a single-turn prompt that wraps the system instruction "Imagine you are an expert requirements analyst... Provide only the category—no explanations" together with brief definitions of the four classes, the chain-of-thought phrase "Let's think this out in step by step way to be sure we have the right answer," and the emotion cue "This is important to my career." The paper's argument is that combining role prompting, emotion prompting, and chain-of-thought lets a zero-shot model classify without labeled examples, while a low temperature of 0.2 stabilizes the stochastic output. The benchmark's majority-consensus annotation procedure is the other load-bearing element: it converts raw reviews into the ground-truth labels against which every reported F1 number is measured.

What would settle it

Run the exact Prompt 3/temperature-0.2 pipeline on the 120 non-consensus reviews, scored against an independent panel's labels: if the F1 on that set falls substantially below 0.842 (for instance below 0.7), the headline result is being carried by the curated, easy-to-agree reviews rather than by a general zero-shot capability.

Watch

Extended reading notes

Core claim

The central discovery is that prompt design and decoding settings, not model fine-tuning, unlock accurate zero-shot app-review classification. On the curated 1,880-review benchmark, built from ten app domains with an inter-annotator agreement of 0.76 and with 120 non-consensus reviews removed, Prompt 3—brief definitions of the four classes, the step-by-step reasoning instruction, and the emotional cue—at temperature 0.2 yields recall 0.848, precision 0.858, and micro-F1 0.842. Per-class results are strong for functional reviews (0.91), non-functional (0.87), and neither (0.94), but weak for the "both" category (0.54). Review length has almost no effect on accuracy, while readability does: correctly classified reviews average an FKGL score of 6.34 versus 9.24 for misclassified ones. Manual error analysis attributes most misclassifications to overlapping FR/NFR characteristics, ambiguous language, negative-sentiment bias, and emotionally charged wording.

Load-bearing premise

The benchmark's ground truth is the majority vote of five annotators, and the 120 reviews they could not agree on were removed before evaluation; if that ground truth is systematically wrong or if dropping the hard cases inflates agreement, every reported F1 is measured against a biased yardstick.

Editorial extensions

If this is right

  • A zero-shot prompt can replace trained classifiers for initial app-review triage: on the paper's benchmark, Prompt 3 at temperature 0.2 outperforms every classical baseline by a wide margin with no app-specific training data.
  • Because the "both" category is the weak point at F1 0.54, a practical deployment should treat "both" predictions as low-confidence and route them to human review or a second classifier.
  • Because readability predicts accuracy, adding a text-simplification stage before classification is a concrete next step with a directly measurable expected payoff.
  • Because higher temperatures degrade performance across all tested prompts, production use should fix the temperature low rather than raise it to encourage variability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper does not run is whether the 120 excluded non-consensus reviews are disproportionately "both"-class; if they are, the four-way taxonomy itself, not the model, would explain much of the reported weakness.
  • The negative-sentiment bias could be probed generically by taking paired reviews with the same content but opposite sentiment and measuring how much F1 shifts; this perturbation design would separate emotion effects from wording effects.
  • If zero-shot FR/NFR classification transfers across app domains, the same prompt recipe may extend to other software text triage tasks such as issue-tracker bug-versus-feature routing, but that transfer is a hypothesis this paper does not test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper evaluates whether GPT-4o mini can classify mobile app reviews into functional requirements (FR), non-functional requirements (NFR), both, or neither in a zero-shot setting, using a manually annotated benchmark of 1,880 reviews from ten app domains. The authors engineer eight prompts, select three on a 100-review sample, and report micro-F1 on the full dataset across three temperature settings; Prompt 3 (combining role, emotion, and chain-of-thought prompting) at temperature 0.2 yields F1 0.842, outperforming classical ML classifiers on a 470-review test split (F1 0.82). The paper also investigates how review length and readability (FKGL) affect classification, and manually categorizes error patterns. The authors release source code and annotated data via Zenodo.

Significance. If the headline F1 survived a proper held-out protocol, the paper would make a useful empirical contribution: zero-shot LLM classification could reduce the need for domain-specific training data in requirements engineering. The strengths include a publicly released dataset and code, a multi-class benchmark with four categories, and a thoughtful manual error taxonomy that goes beyond numeric reporting. However, the central quantitative claim is currently supported only by a selection-on-test estimate, and the discrepancy between Tables 2 and 3 blurs which number is the held-out result. I therefore see the contribution as potentially valuable but not yet established.

major comments (4)
  1. [Section 2.2 and Table 2] The experimental protocol selects prompts using a 100-review sample drawn from the same 1,880-review benchmark, then evaluates the selected prompts on the full benchmark. Consequently, the reported F1 0.842 for Prompt 3 at temperature 0.2 is a selection-on-test estimate: the configuration was chosen using labels from the same distribution as the evaluation set, with no disjoint validation split or nested procedure described. This is load-bearing because the abstract's 'robust F1 score of 0.842' is the paper's central claim. Please report a held-out evaluation (e.g., a validation fold used for prompt selection and a test fold never used for selection) or a nested cross-validation scheme, and state the F1 obtained under that protocol.
  2. [Table 2 vs. Table 3] The comparison with classical ML models reports GPT-4o mini F1 0.82 on a 1410/470 split, while Table 2 reports F1 0.842 for the same configuration (Prompt 3, temperature 0.2). The paper does not explain whether Table 3's 0.82 is the held-out result for the optimal configuration; if it is, the abstract's 0.842 is not the test-set estimate, and if it is not, the paper never reports a held-out ChatGPT result. Please reconcile these numbers and clearly label which evaluation split underlies each table.
  3. [Section 2.3] The paper acknowledges that LLM outputs are non-deterministic (citing [6]) and varies temperature to explore randomness, but every review is scored with a single API call per configuration, and no repeated runs or confidence intervals are reported. A single point estimate cannot support the word 'robust' in the abstract. Please add multiple runs (e.g., 3-5 repetitions per configuration), report mean and variance, and consider significance tests against the ML baselines in Table 3.
  4. [Section 2.1] The evaluation's ground truth is the five annotators' majority label on the 1,880 reviews that survived exclusion of 120 non-consensus reviews. Because the excluded reviews are likely the most ambiguous, the reported F1 may overstate performance on live app-store data where no such curation occurs. Please provide class-level kappa, describe the distribution of excluded reviews across the four classes, and, if possible, report model performance on the 120 non-consensus reviews as a sensitivity check.
minor comments (6)
  1. [Section 2.2, Table 1] The fourth class is named 'None' in Prompts 1 and 2 but 'neither' in the dataset description and elsewhere; this inconsistency should be harmonized for clarity and prompt consistency.
  2. [Equation (1)] The FKGL formula is typeset with Unicode symbols and ambiguous division; use standard math notation and define all variables explicitly.
  3. [Tables 2 and 3] Add a table note stating that each reported value is a single API pass (or the mean over runs, after revision), and specify whether Table 3's ChatGPT row was computed on the same 470-review split as the ML models.
  4. [Abstract] The phrase 'robust F1 score of 0.842' is stronger than the evidence supports; once repeated runs or confidence intervals are added, qualify the claim accordingly, or replace 'robust' with a neutral descriptor.
  5. [Title page] The title contains 'ChatGP T' with a space; fix this typographical artifact.
  6. [Section 6] Reference [19] is cited as 'our prior work' in the future-work discussion; if this paper is a direct extension, the relationship should be stated explicitly in the introduction.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported F1 scores are empirical measurements against human-annotated labels, not derivations from the model's own outputs.

full rationale

The paper makes no formal derivation; it reports an empirical benchmark evaluation of zero-shot GPT-4o mini on 1,880 manually annotated reviews. The classification labels are human-generated and are not defined in terms of the model's outputs, so no self-definitional circularity is present. The only self-citation ([19]) appears in future work as a suggested review-simplification step and is not load-bearing for the reported F1 scores. The prompt-selection procedure in Section 2.2 uses a 100-review sample from the same curated benchmark to choose the top prompts and later reports Prompt 3's F1 on the complete dataset; this is a legitimate validity concern about selection-on-test evaluation and the absence of a disjoint held-out ChatGPT split, but it is not a circular reduction: the prompt is not fitted to the labels in the sense of a parameter, and the reported score is a measurement rather than a quantity derived from its own inputs. Under the required standard of exhibiting an equation-level or definition-level reduction, no circular step is found.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No mathematical derivation is made, so the ledger records evaluation assumptions. The main assumption is that the self-built annotated dataset is a trustworthy gold standard, and the main hand-picked configuration is the temperature and prompt selection.

free parameters (1)
  • Temperature and prompt configuration = Temperature 0.2 with Prompt 3
    The reported F1 of 0.842 depends on choosing temperature 0.2 and Prompt 3 after comparing three prompts and three temperatures on a 100-review sample of the same benchmark.
assumptions (4)
  • domain assumption Manual labels produced by five software engineers with Fleiss kappa 0.76 are a valid gold standard.
    Section 2.1: every F1 score measures agreement with these labels, and no external validated benchmark is used.
  • domain assumption Excluding 120 reviews without majority agreement does not bias the benchmark.
    Section 2.1: the excluded reviews are likely the ambiguous cases, which later show lower F1, so removal may inflate measured performance.
  • domain assumption A single API call per review at temperature 0.2 is representative of model performance.
    Section 2.3: non-determinism is acknowledged but no repeated runs or confidence intervals are reported.
  • domain assumption Reviews from ten selected apps represent the diversity of app store reviews.
    Section 2.1 and Section 4: the authors acknowledge limited size and rely on domain diversity for generalizability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Zero-Shot App Review Classification with ChatGPT: Challenges and Potential." pith.science (2026). https://pith.science/paper/4RPIPSR5

@misc{pith2026250504759,
  author       = {Pith},
  title        = {Pith review of: Exploring Zero-Shot App Review Classification with ChatGPT: Challenges and Potential},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4RPIPSR5}},
  note         = {Machine review of arXiv:2505.04759}
}
read the original abstract

App reviews are a critical source of user feedback, offering valuable insights into an app's performance, features, usability, and overall user experience. Effectively analyzing these reviews is essential for guiding app development, prioritizing feature updates, and enhancing user satisfaction. Classifying reviews into functional and non-functional requirements play a pivotal role in distinguishing feedback related to specific app features (functional requirements) from feedback concerning broader quality attributes, such as performance, usability, and reliability (non-functional requirements). Both categories are integral to informed development decisions. Traditional approaches to classifying app reviews are hindered by the need for large, domain-specific datasets, which are often costly and time-consuming to curate. This study explores the potential of zero-shot learning with ChatGPT for classifying app reviews into four categories: functional requirement, non-functional requirement, both, or neither. We evaluate ChatGPT's performance on a benchmark dataset of 1,880 manually annotated reviews from ten diverse apps spanning multiple domains. Our findings demonstrate that ChatGPT achieves a robust F1 score of 0.842 in review classification, despite certain challenges and limitations. Additionally, we examine how factors such as review readability and length impact classification accuracy and conduct a manual analysis to identify review categories more prone to misclassification.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 21 canonical work pages

  1. [6]

    Prompt engineering with chatgpt: a guide fo r academic writers

    Louie Giray. Prompt engineering with chatgpt: a guide fo r academic writers. Annals of biomedical engineering , 51(12):2629–2633, 2023

  2. [1]

    Does inv olving users in soft- ware development really influence system success? IEEE software, 30(6):17–23, 2013

    Ulrike Abelein, Helen Sharp, and Barbara Paech. Does inv olving users in soft- ware development really influence system success? IEEE software, 30(6):17–23, 2013

  3. [2]

    Identification and Classification of Non-Functional Requirements from User Re views

    Md Ashif Aziz, Nafisa Mehjabin, and Khandaker Rifah Tasni a. Identification and Classification of Non-Functional Requirements from User Re views. PhD thesis, Department of Computer Science and Engineering (CSE), Isla mic University of Technology (IUT), 2022

  4. [3]

    User involvement in softw are development and system success: A systematic literature review

    Muneera Bano and Didar Zowghi. User involvement in softw are development and system success: A systematic literature review. 2013

  5. [4]

    Mining and searching app reviews for requirements engineering: Evaluation and replication studies

    Jacek Dąbrowski, Emmanuel Letier, Anna Perini, and Ange lo Susi. Mining and searching app reviews for requirements engineering: Evaluation and replication studies. Information Systems, 114:102181, 2023

  6. [5]

    Automatically clas- sifying requirements from app stores: A preliminary study

    Roger Deocadez, Rachel Harrison, and Daniel Rodriguez. Automatically clas- sifying requirements from app stores: A preliminary study. In 2017 IEEE 25th international requirements engineering conference works hops (REW), pages 367–

  7. [7]

    Evaluating pre-train ed models for user feedback analysis in software engineering: A study on c lassification of app- reviews

    Mohammad A Hadi and Fatemeh H Fard. Evaluating pre-train ed models for user feedback analysis in software engineering: A study on c lassification of app- reviews. Empirical Software Engineering, 28(4):88, 2023

  8. [8]

    A critical review of la rge language models: Sensitivity, bias, and the path toward specialized ai

    Arash Hajikhani and Carolyn Cole. A critical review of la rge language models: Sensitivity, bias, and the path toward specialized ai. Quantitative Science Studies, 5(3):736–756, 2024

Show all 24 references
  1. [9]

    Eval- uation of chatgpt on biomedical tasks: A zero-shot comparis on with fine-tuned generative transformers

    Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimm y Huang. Eval- uation of chatgpt on biomedical tasks: A zero-shot comparis on with fine-tuned generative transformers. arXiv preprint arXiv:2306.04504 , 2023

  2. [10]

    Performance compariso n of word embedding methods in text classification for various number of feature s

    Elif Kabullar and Ilker Türker. Performance compariso n of word embedding methods in text classification for various number of feature s. In International Istanbul Modern Scientific Research Congress-IV , 2022

  3. [11]

    Derivation of new readability formulas (automated readabi lity index, fog count and flesch reading ease formula) for navy enlisted personnel

    J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers , and Brad S Chissom. Derivation of new readability formulas (automated readabi lity index, fog count and flesch reading ease formula) for navy enlisted personnel . 1975

  4. [12]

    Better zero-shot reasoning w ith role-play prompting

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. Better zero-shot reasoning w ith role-play prompting. arXiv preprint arXiv:2308.07702 , 2023

  5. [13]

    As sessment of quality of experience (qoe) of image compression in social cloud com puting

    Asif Laghari, Hui He, Muhammad Shafiq, and Asiya Khan. As sessment of quality of experience (qoe) of image compression in social cloud com puting. 2018

  6. [14]

    Large language models und erstand and can be enhanced by emotional stimuli

    Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxi n Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. Large language models und erstand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2307.11760 , 2023

  7. [15]

    Automatic classification of non-functional re- quirements from augmented app user reviews

    Mengmeng Lu and Peng Liang. Automatic classification of non-functional re- quirements from augmented app user reviews. 2017

  8. [16]

    App sto re mining for feature extraction: analyzing user reviews

    Zulfiqar Ali Memon, Nida Munawar, and Maha Kamal. App sto re mining for feature extraction: analyzing user reviews. Acta Scientiarum. Technology, 46(1), 2024

  9. [17]

    Harnessing large language models’ zero-shot and few-shot learning capabili ties for regulatory re- search

    Hamed Meshkin, Joel Zirkle, Ghazal Arabidarrehdor, An ik Chaturbedi, Shilpa Chakravartula, John Mann, Bradlee Thrasher, and Zhihua Li. Harnessing large language models’ zero-shot and few-shot learning capabili ties for regulatory re- search. Briefings in Bioinformatics , 25(5):...

  10. [18]

    Too many user-revi ews! what should app developers look at first? IEEE Transactions on Software Engineering , 47(2):367–378, 2019

    Ehsan Noei, Feng Zhang, and Ying Zou. Too many user-revi ews! what should app developers look at first? IEEE Transactions on Software Engineering , 47(2):367–378, 2019

  11. [19]

    Refining app reviews: Dataset, methodology, and evaluation

    Amrita Singh, Chirag Jain, Mohit Chaudhary, and Preeth u Anish. Refining app reviews: Dataset, methodology, and evaluation. InProceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing: In dustry Track, pages 595–608, 2024

  12. [20]

    Dcnr: A transfe r learning ap- proach for detecting and classifying non-functionalrequi rements from user re- views

    Khandaker Rifah Tasnia, Nafisa Mehjabin, Md Ashif Aziz, Abu Raihan Mostofa Kamal, Afrina Khatun, and Lutfun Nahar Lota. Dcnr: A transfe r learning ap- proach for detecting and classifying non-functionalrequi rements from user re- views. 2022

  13. [21]

    Can app changelogs improve requirements classification fro m app reviews? an exploratory study

    Chong Wang, Fan Zhang, Peng Liang, Maya Daneva, and Mart en Van Sinderen. Can app changelogs improve requirements classification fro m app reviews? an exploratory study. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measuremen...

  14. [22]

    Chain-of-thought prompting eli cits reasoning in large language models, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma , Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting eli cits reasoning in large language models, 2022

  15. [23]

    Identification and classificati on of requirements from app user reviews

    Hui Yang and Peng Liang. Identification and classificati on of requirements from app user reviews. In SEKE, pages 7–12. Citeseer, 2015

  16. [24]

    Y. Yao, W. Jiang, Y. Wang, P. Song, and B. Wang. Non-funct ional requirements analysis based on application reviews in the android app mar ket. Information Resources Management Journal, 35:1–17, 2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.