{"id":"1d0f835d-50fb-4a90-b6be-7ce1d49b5af4","arxiv_id":"2505.04759","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Zero-shot GPT-4o mini classifies app reviews into functional, non-functional, both, or neither with F1 0.84, outperforming classical ML models on a 1,880-review benchmark.","lead":"App reviews can be automatically sorted into functional and non-functional requirements by ChatGPT without any training data, reaching an F1 score of 0.84 on a hand-labeled set of 1,880 reviews. The result matters because it suggests app teams could analyze user feedback cheaply, though the evaluation has gaps in statistical rigor.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 0.842 is a selection-on-test estimate: the winning prompt and temperature were selected on the same 1,880-review benchmark and then scored on it, with no disjoint held-out split or repeated runs.","rationale":"I agree that the paper is conditionally acceptable, but my load-bearing concern differs from the reader's. The reader's weakest assumption concerns the ground-truth labels; those are documented with kappa 0.76 and majority consensus, and while not independently audited, the larger immediate threat to the central numerical claim is that prompt and temperature were selected using the same benchmark on which the headline F1 is reported. This is an internal-validity issue that can be fixed by a proper held-out evaluation, so it does not warrant rejection. I would keep the CONDITIONAL verdict but require the authors to report results on a disjoint test set with multiple runs and confidence intervals, and to reconcile Table 3's 0.82 with the abstract's 0.842.","tokens_in":10108,"tokens_out":4595,"duration_ms":48046,"concrete_test":"Using the Zenodo artifact, designate the 100-review sample from Section 2.2 as the prompt-development set and a disjoint held-out test set, e.g., the 470-review split used for Table 3. Run Prompt 3 at temperature 0.2 at least 10 independent times on the held-out set and report the mean and 95% confidence interval of micro-F1. If the held-out mean is materially below 0.842 (or equals the 0.82 already shown in Table 3), the headline 0.842 is an overfit selection estimate; if it reproduces 0.842 within noise, the concern is retired.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is evaluation protocol, not annotation quality. Section 2.2 says the authors applied eight prompts to a 100-review sample drawn from the curated 1,880-review dataset and then selected the top three prompts for further analysis on the complete dataset. Section 3 then reports that Prompt 3 at temperature 0.2 achieved a micro-F1 of 0.842 on the complete dataset. Because the same benchmark labels were used to choose among 8 prompts × 3 temperatures and to compute the headline score, 0.842 is a selection-on-test estimate, not an unbiased estimate of zero-shot performance on new app reviews. The paper itself notes that LLM outputs are non-deterministic, but no repeated runs or error bars are reported, so a single 0.842 point is not enough to establish robustness. Table 3 reinforces the concern: on the 470-review split used for the ML comparison, GPT-4o mini is listed with F1 0.82, not 0.842. If Table 3 is the held-out comparison, the abstract's headline number is not the held-out number; if it is not, the paper never reports a held-out ChatGPT evaluation. This matters directly for the central claim that zero-shot ChatGPT classification is practically usable without curated training data. The promised Zenodo artifacts are useful, but they do not by themselves resolve the absence of a clear train/selection/test separation in the reported headline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates whether GPT-4o mini can classify mobile app reviews into functional requirements (FR), non-functional requirements (NFR), both, or neither in a zero-shot setting, using a manually annotated benchmark of 1,880 reviews from ten app domains. The authors engineer eight prompts, select three on a 100-review sample, and report micro-F1 on the full dataset across three temperature settings; Prompt 3 (combining role, emotion, and chain-of-thought prompting) at temperature 0.2 yields F1 0.842, outperforming classical ML classifiers on a 470-review test split (F1 0.82). The paper also investigates how review length and readability (FKGL) affect classification, and manually categorizes error patterns. The authors release source code and annotated data via Zenodo.","tokens_in":10362,"tokens_out":5376,"duration_ms":51456,"significance":"If the headline F1 survived a proper held-out protocol, the paper would make a useful empirical contribution: zero-shot LLM classification could reduce the need for domain-specific training data in requirements engineering. The strengths include a publicly released dataset and code, a multi-class benchmark with four categories, and a thoughtful manual error taxonomy that goes beyond numeric reporting. However, the central quantitative claim is currently supported only by a selection-on-test estimate, and the discrepancy between Tables 2 and 3 blurs which number is the held-out result. I therefore see the contribution as potentially valuable but not yet established.","major_comments":[{"comment":"The experimental protocol selects prompts using a 100-review sample drawn from the same 1,880-review benchmark, then evaluates the selected prompts on the full benchmark. Consequently, the reported F1 0.842 for Prompt 3 at temperature 0.2 is a selection-on-test estimate: the configuration was chosen using labels from the same distribution as the evaluation set, with no disjoint validation split or nested procedure described. This is load-bearing because the abstract's 'robust F1 score of 0.842' is the paper's central claim. Please report a held-out evaluation (e.g., a validation fold used for prompt selection and a test fold never used for selection) or a nested cross-validation scheme, and state the F1 obtained under that protocol.","section":"Section 2.2 and Table 2"},{"comment":"The comparison with classical ML models reports GPT-4o mini F1 0.82 on a 1410/470 split, while Table 2 reports F1 0.842 for the same configuration (Prompt 3, temperature 0.2). The paper does not explain whether Table 3's 0.82 is the held-out result for the optimal configuration; if it is, the abstract's 0.842 is not the test-set estimate, and if it is not, the paper never reports a held-out ChatGPT result. Please reconcile these numbers and clearly label which evaluation split underlies each table.","section":"Table 2 vs. Table 3"},{"comment":"The paper acknowledges that LLM outputs are non-deterministic (citing [6]) and varies temperature to explore randomness, but every review is scored with a single API call per configuration, and no repeated runs or confidence intervals are reported. A single point estimate cannot support the word 'robust' in the abstract. Please add multiple runs (e.g., 3-5 repetitions per configuration), report mean and variance, and consider significance tests against the ML baselines in Table 3.","section":"Section 2.3"},{"comment":"The evaluation's ground truth is the five annotators' majority label on the 1,880 reviews that survived exclusion of 120 non-consensus reviews. Because the excluded reviews are likely the most ambiguous, the reported F1 may overstate performance on live app-store data where no such curation occurs. Please provide class-level kappa, describe the distribution of excluded reviews across the four classes, and, if possible, report model performance on the 120 non-consensus reviews as a sensitivity check.","section":"Section 2.1"}],"minor_comments":[{"comment":"The fourth class is named 'None' in Prompts 1 and 2 but 'neither' in the dataset description and elsewhere; this inconsistency should be harmonized for clarity and prompt consistency.","section":"Section 2.2, Table 1"},{"comment":"The FKGL formula is typeset with Unicode symbols and ambiguous division; use standard math notation and define all variables explicitly.","section":"Equation (1)"},{"comment":"Add a table note stating that each reported value is a single API pass (or the mean over runs, after revision), and specify whether Table 3's ChatGPT row was computed on the same 470-review split as the ML models.","section":"Tables 2 and 3"},{"comment":"The phrase 'robust F1 score of 0.842' is stronger than the evidence supports; once repeated runs or confidence intervals are added, qualify the claim accordingly, or replace 'robust' with a neutral descriptor.","section":"Abstract"},{"comment":"The title contains 'ChatGP T' with a space; fix this typographical artifact.","section":"Title page"},{"comment":"Reference [19] is cited as 'our prior work' in the future-work discussion; if this paper is a direct extension, the relationship should be stated explicitly in the introduction.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an empirical software engineering venue, and the public release of dataset and code is commendable. The main concern is methodological: the headline F1 is a selection-on-test estimate with no disjoint holdout, and the reported numbers are internally inconsistent between Tables 2 and 3. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I would be willing to review a revised version that adds a proper held-out protocol, repeated runs, and a reconciled presentation of results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a compact empirical study of zero-shot GPT-4o mini for classifying app reviews into functional/non-functional/both/neither. The dataset of 1,880 manually labeled reviews and the prompt engineering comparison are useful for the requirements engineering niche. But the headline F1 of 0.842 is not a reliable estimate of zero-shot performance because the prompt and temperature were selected on the same benchmark they were scored on, and there are no repeated runs.\n\nWhat's new: previous work on app review classification used classical ML, semi-supervised learning, and fine-tuned transformers. Applying zero-shot LLM classification to this four-class problem with a prompt comparison is a legitimate new application, and the paper does it transparently. The authors release the data and code on Zenodo, report inter-annotator agreement (Fleiss kappa 0.76), and build a reasonable misclassification taxonomy from 100 errors. That part is solid.\n\nWhere it falls short: the evaluation protocol. The authors picked the top 3 out of 8 prompts on a 100-review sample drawn from the same 1,880-review dataset, then evaluated those on the full dataset. They also picked the temperature (0.2) by looking at full-dataset results across three temperatures. So 0.842 is the best of nine configurations scored on the same data. There is no disjoint validation set and no error bars, so the number is likely optimistically biased. Table 3 makes this worse: on the 470-review test split used for the ML comparison, the same model gets F1 0.82. The paper never explains why the two numbers differ, and that inconsistency undermines the abstract's claim. The single-run nature is acknowledged in principle but not handled.\n\nOther concerns are minor: removing 120 non-consensus reviews is reasonable, and the 'Both' class has low F1 (0.54), which the authors discuss. The ML baselines are, as usual, easy to outperform, but that is not a fatal issue.\n\nOverall, the core idea is plausible and the artifacts are useful, but the paper overstates what is demonstrated. With a proper held-out split, repeated runs, and a reconciliation of the two F1 numbers, this would be a solid contribution to EASE. As is, treat the headline as preliminary.\n\nWho it's for: researchers in requirements engineering or app-store mining who want a zero-shot baseline and a new annotated dataset. I would send it to peer review rather than desk reject, because the flaws are fixable and the dataset has value.\n\nCheers.","headline":"Useful dataset and prompt comparison for app-review classification, but the headline F1 is a selection-on-test estimate that needs a proper held-out evaluation.","tokens_in":10894,"tokens_out":3112,"would_cite":true,"duration_ms":28940,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A zero-shot GPT-4o mini, prompted with an expert requirements-analyst role, an emotional cue, and step-by-step reasoning at temperature 0.2, classifies app reviews as functional, non-functional, both, or neither with micro-F1 0.842.","keywords":["app review classification","zero-shot learning","ChatGPT","prompt engineering","requirement engineering","functional requirements","non-functional requirements","large language models"],"falsifier":"Run the exact Prompt 3/temperature-0.2 pipeline on the 120 non-consensus reviews, scored against an independent panel's labels: if the F1 on that set falls substantially below 0.842 (for instance below 0.7), the headline result is being carried by the curated, easy-to-agree reviews rather than by a general zero-shot capability.","tokens_in":9896,"feed_emoji":"📱","tokens_out":9266,"duration_ms":86472,"temperature":0.7,"pith_summary":"This paper asks whether a large language model can sort app-store reviews into requirement categories—functional (what the app should do), non-functional (how well it performs), both, or neither—without being given any domain-specific training examples. It reports that a zero-shot GPT-4o mini, using a prompt that combines role assignment, an emotional cue, and chain-of-thought instructions at temperature 0.2, reaches a micro-F1 of 0.842 on 1,880 manually annotated reviews from ten apps. The same model outperforms classical machine-learning classifiers such as random forest, SVM, XGBoost, and logistic regression even though those models were trained on labeled data. If the result holds, app teams could automate the first pass of requirements triage on user feedback without curating expensive, domain-specific labeled datasets.","feed_headline":"0.842 F1: zero-shot GPT-4o mini sorts app reviews","feed_subtitle":"Role, emotion, and step-by-step prompting beat classical ML models with no domain-specific labels.","key_machinery":"The load-bearing object is Prompt 3, a single-turn prompt that wraps the system instruction \"Imagine you are an expert requirements analyst... Provide only the category—no explanations\" together with brief definitions of the four classes, the chain-of-thought phrase \"Let's think this out in step by step way to be sure we have the right answer,\" and the emotion cue \"This is important to my career.\" The paper's argument is that combining role prompting, emotion prompting, and chain-of-thought lets a zero-shot model classify without labeled examples, while a low temperature of 0.2 stabilizes the stochastic output. The benchmark's majority-consensus annotation procedure is the other load-bearing element: it converts raw reviews into the ground-truth labels against which every reported F1 number is measured.","core_discovery":"The central discovery is that prompt design and decoding settings, not model fine-tuning, unlock accurate zero-shot app-review classification. On the curated 1,880-review benchmark, built from ten app domains with an inter-annotator agreement of 0.76 and with 120 non-consensus reviews removed, Prompt 3—brief definitions of the four classes, the step-by-step reasoning instruction, and the emotional cue—at temperature 0.2 yields recall 0.848, precision 0.858, and micro-F1 0.842. Per-class results are strong for functional reviews (0.91), non-functional (0.87), and neither (0.94), but weak for the \"both\" category (0.54). Review length has almost no effect on accuracy, while readability does: correctly classified reviews average an FKGL score of 6.34 versus 9.24 for misclassified ones. Manual error analysis attributes most misclassifications to overlapping FR/NFR characteristics, ambiguous language, negative-sentiment bias, and emotionally charged wording.","pith_inferences":["A direct test the paper does not run is whether the 120 excluded non-consensus reviews are disproportionately \"both\"-class; if they are, the four-way taxonomy itself, not the model, would explain much of the reported weakness.","The negative-sentiment bias could be probed generically by taking paired reviews with the same content but opposite sentiment and measuring how much F1 shifts; this perturbation design would separate emotion effects from wording effects.","If zero-shot FR/NFR classification transfers across app domains, the same prompt recipe may extend to other software text triage tasks such as issue-tracker bug-versus-feature routing, but that transfer is a hypothesis this paper does not test."],"forward_implications":["A zero-shot prompt can replace trained classifiers for initial app-review triage: on the paper's benchmark, Prompt 3 at temperature 0.2 outperforms every classical baseline by a wide margin with no app-specific training data.","Because the \"both\" category is the weak point at F1 0.54, a practical deployment should treat \"both\" predictions as low-confidence and route them to human review or a second classifier.","Because readability predicts accuracy, adding a text-simplification stage before classification is a concrete next step with a directly measurable expected payoff.","Because higher temperatures degrade performance across all tested prompts, production use should fix the temperature low rather than raise it to encourage variability."],"supporting_citations":[{"why":"Supplies the emotion-prompting technique whose cue, \"This is important to my career,\" is part of Prompt 3.","marker":"[14]"},{"why":"Supplies the role-play prompting technique behind the expert requirements analyst system prompt.","marker":"[12]"},{"why":"Supplies the chain-of-thought instruction used in Prompt 3 to elicit step-by-step reasoning.","marker":"[22]"},{"why":"Provides the FKGL readability formula used to measure review complexity and explain misclassification patterns.","marker":"[11]"},{"why":"Provides the sentence-transformer embedding method used to build the classical ML baselines that ChatGPT is compared against.","marker":"[10]"},{"why":"Documents LLM output non-determinism, motivating the temperature comparison that identifies 0.2 as the best setting.","marker":"[6]"}],"fun_headline_variants":["Zero-shot ChatGPT achieves 0.842 F1 on app reviews","Prompt design unlocks zero-shot app review classification","F1 0.842 in zero-shot app review split without training data","Zero-shot ChatGPT app review accuracy depends on readability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is the majority vote of five annotators, and the 120 reviews they could not agree on were removed before evaluation; if that ground truth is systematically wrong or if dropping the hard cases inflates agreement, every reported F1 is measured against a biased yardstick.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot ChatGPT achieves 0.842 F1 on app reviews","Prompt design unlocks zero-shot app review classification","F1 0.842 in zero-shot app review split without training data","Zero-shot ChatGPT app review accuracy depends on readability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000551,"raw_usage":{"total_tokens":2643,"prompt_tokens":974,"completion_tokens":1669,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":1601}},"tokens_in":590,"tokens_out":1669,"duration_ms":11027,"temperature":1.0,"reasoning_tokens":1601,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:21:43.707588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact Prompt 3/temperature-0.2 pipeline on the 120 non-consensus reviews, scored against an independent panel's labels: if the F1 on that set falls substantially below 0.842 (for instance below 0.7), the headline result is being carried by the curated, easy-to-agree reviews rather than by a general zero-shot capability.","supporting_citations":[{"cited_title":"Chain-of-thought prompting eli cits reasoning in large language models, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought instruction used in Prompt 3 to elicit step-by-step reasoning."},{"cited_title":"Derivation of new readability formulas (automated readabi lity index, fog count and flesch reading ease formula) for navy enlisted personnel","cited_arxiv_id":null,"evidence_quote":"Provides the FKGL readability formula used to measure review complexity and explain misclassification patterns."},{"cited_title":"Performance compariso n of word embedding methods in text classification for various number of feature s","cited_arxiv_id":null,"evidence_quote":"Provides the sentence-transformer embedding method used to build the classical ML baselines that ChatGPT is compared against."},{"cited_title":"Prompt engineering with chatgpt: a guide fo r academic writers","cited_arxiv_id":null,"evidence_quote":"Documents LLM output non-determinism, motivating the temperature comparison that identifies 0.2 as the best setting."}],"review_version":1}