{"id":"3f555087-0e57-46da-a353-9e61c970e696","arxiv_id":"2605.12657","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs can detect usability content in user reviews with F-scores comparable to humans, though performance depends strongly on prompt design.","lead":"This paper tests if large language models can spot usability issues in app user reviews as well as human experts by labeling 300 reviews with prompts based on Nielsen heuristics. A smart generalist might read it to see how AI could make gathering user feedback faster and cheaper for software teams without needing huge labeled training sets.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Ground truth shaky: no inter-rater agreement reported and prompt tuned on same 300 reviews with no held-out test","rationale":"Reader's weakest assumption matches the load-bearing gap exactly. The abstract-only view already flags missing metrics; the full-text description of the 300-review dataset and two-iteration prompt development confirms the absence of agreement statistics or generalization checks, so the CONDITIONAL verdict holds.","tokens_in":1775,"tokens_out":291,"duration_ms":10997,"concrete_test":"Re-label the 300 reviews independently with a third rater and compute Cohen's kappa between all pairs; if any kappa < 0.65, recompute LLM F-scores against the majority-vote labels. Separately, apply the final prompt to a fresh 100-review sample from a fourth app category and compare to new human labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on F-score comparison between LLM and 'human raters' as ground truth. With only two raters and no reported Cohen's kappa or equivalent, label consistency is unknown. The prompt was iteratively refined on this exact dataset using Nielsen-derived guidelines, so any reported F-score may reflect overfitting rather than general recognition ability. No external validation set or cross-LLM test is described, directly undermining the claim that performance 'is strongly dependent on the prompt' in a reliable way.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper explores using large language models (LLMs) to identify usability-related content in user reviews as a precursor to requirements engineering. It contributes a labeled dataset of 300 reviews from three app types (labeled by two human raters and an LLM), develops an initial prompt via two iterations based on Nielsen's 10 usability heuristics, and claims that LLMs can generally recognize usability as a non-functional requirement with F-scores comparable to human raters, though performance and reliability depend strongly on the prompt.","tokens_in":1864,"tokens_out":460,"duration_ms":52494,"significance":"If the central claims hold after proper validation, the work could support low-cost, scalable extraction of usability requirements from abundant user reviews without needing large manually labeled training sets typical of ML/DL approaches. The provision of a fully coded dataset is a clear strength for reproducibility and community use.","major_comments":[{"comment":"Abstract: The abstract reports F-score comparisons on 300 reviews but provides no numerical F-score values, no inter-rater agreement statistics, and no details on the prompt iterations or exact LLM used, which prevents verification of the claim that LLMs perform comparably to humans.","section":null},{"comment":"Methodology / Labeling Process: The prompt was developed and refined iteratively on the exact same 300 reviews used for evaluation, with no held-out test set, cross-validation, or external validation described; this raises a direct risk of overfitting and weakens the conclusion that performance 'is strongly dependent on the prompt' in a generalizable way.","section":null},{"comment":"Labeling Process: Only two human raters are used with no reported inter-rater reliability metric (e.g., Cohen's kappa or percentage agreement), so the reliability of the ground-truth labels is unknown and the F-score comparison to the LLM rests on an unverified foundation.","section":null}],"minor_comments":[{"comment":"Abstract: Consider adding the actual F-score numbers and a brief note on the LLM model/version to make the central result immediately assessable.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like a workshop or short conference paper than a full journal article; the limited scope and missing validation details may affect fit for a top-tier venue."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our precursor study. We address each major comment below and will revise the manuscript to improve clarity, transparency, and acknowledgment of limitations where appropriate.","responses":[{"response":"We agree that the abstract should be more informative. In the revised manuscript, we will include the specific F-score values for the LLM (which were comparable to those of the human raters), the inter-rater agreement statistic, the exact LLM model employed, and a concise description of the two prompt engineering iterations. This will enable readers to directly assess the comparability claim.","revision_made":"yes","referee_comment":"Abstract: The abstract reports F-score comparisons on 300 reviews but provides no numerical F-score values, no inter-rater agreement statistics, and no details on the prompt iterations or exact LLM used, which prevents verification of the claim that LLMs perform comparably to humans."},{"response":"This observation is correct and highlights a limitation inherent to our small-scale precursor study. With only 300 reviews available, iterative prompt refinement was conducted on the full set, which is a common practice in early-stage prompt engineering but does carry overfitting risk. We will revise the methodology and discussion sections to explicitly state this limitation, note the absence of a held-out set, and qualify the generalizability of the prompt-dependence conclusion. We will also stress that the publicly released labeled dataset allows other researchers to perform independent validation on new data.","revision_made":"partial","referee_comment":"Methodology / Labeling Process: The prompt was developed and refined iteratively on the exact same 300 reviews used for evaluation, with no held-out test set, cross-validation, or external validation described; this raises a direct risk of overfitting and weakens the conclusion that performance 'is strongly dependent on the prompt' in a generalizable way."},{"response":"We accept this criticism. The revised manuscript will include the inter-rater reliability metric (Cohen's kappa and percentage agreement) computed between the two human raters. This addition will provide a clearer basis for interpreting the LLM's F-score performance relative to human labeling.","revision_made":"yes","referee_comment":"Labeling Process: Only two human raters are used with no reported inter-rater reliability metric (e.g., Cohen's kappa or percentage agreement), so the reliability of the ground-truth labels is unknown and the F-score comparison to the LLM rests on an unverified foundation."}],"tokens_in":1460,"tokens_out":537,"duration_ms":45743,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"This paper's main output is a coded dataset of 300 user reviews from three app types, labeled by two humans and an LLM, plus an initial prompt built from Nielsen heuristics after two iterations. That gives a concrete resource for pulling usability-related requirements from reviews without large training sets for ML models. The shift to prompt engineering on pre-trained LLMs is the practical angle they highlight against earlier classification work. They also release the data, which lowers the barrier for others to test or extend the approach. The prompt guidelines tied to established heuristics are a clear step that makes the method more reproducible than ad-hoc prompting. The evaluation side is thinner. The abstract refers to F-score comparisons with human raters but reports none of the actual numbers, no inter-rater agreement measure, and no details on how the prompt iterations were assessed. Since the prompt was refined on this exact set of reviews, any performance figures risk reflecting that tuning rather than broader capability. No separate test set or checks across different LLMs appear, so the statement that results depend strongly on the prompt stays hard to verify. This is aimed at requirements engineering researchers who want lightweight LLM tools for user-centered elicitation. A reader could pull the dataset and guidelines as a starting point for their own experiments, even if they would need to add stronger validation themselves. I would send it for peer review. The precursor framing fits, and referees could push for the missing metrics and a held-out evaluation so the work becomes more usable as a foundation.","headline":"A small labeled dataset and Nielsen-based prompt for LLM usability review filtering, but the F-score claims rest on missing numbers and no held-out validation.","tokens_in":2314,"tokens_out":372,"would_cite":false,"duration_ms":33108,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"LLM prompt engineering for usability review classification unrelated to recognition-cost forcing","alignment":"orthogonal","rationale":"Paper operates entirely in cs.SE / requirements engineering: iterative prompt refinement on Nielsen heuristics for binary classification of 300 reviews, F-score vs. human raters, Cohen's kappa. No reference to J-cost, φ-ladders, 8-tick periodicity, distinction-forcing, or any RS theorem. RS framework (e.g., reality_from_one_distinction, Jcost uniqueness in Cost.FunctionalEquation) has no opinion on this domain.","tokens_in":47861,"confidence":"high","tokens_out":136,"duration_ms":7530,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Large language models can identify usability requirements in user reviews with F-scores comparable to human raters when the prompt is well designed.","keywords":["user reviews","usability requirements","large language models","requirements engineering","prompt engineering","Nielsen usability heuristics","non-functional requirements","app feedback"],"falsifier":"Apply the final prompt to a fresh set of reviews independently labeled by new human raters and check whether the resulting F-score stays within the same range as the original human-to-human agreement.","tokens_in":2676,"feed_emoji":"📝","tokens_out":652,"duration_ms":25210,"temperature":0.7,"pith_summary":"User reviews hold practical feedback on usability but require effort to sift through at scale. The paper tests whether pre-trained LLMs can classify reviews that discuss usability issues without needing task-specific training data. The authors assembled a dataset of 300 reviews from three app categories, labeled them according to Nielsen heuristics, and iterated on a prompt to guide the LLM. They report that the LLM reaches human-level F-scores on the classification task, yet the outcome varies sharply with small changes to the prompt wording. This points to a low-cost workflow that lets development teams turn existing review streams into usability requirements without building new labeled corpora.","feed_headline":"LLMs match human accuracy in spotting usability requirements in reviews","feed_subtitle":"Prompt engineering turns existing app feedback into design insights without building new labeled training sets.","key_machinery":"An iteratively refined prompt, built from Nielsen's 10 Usability Heuristics, that directs the LLM to filter user reviews for usability-relevant content.","core_discovery":"LLMs are generally able to recognize usability as a non-functional requirement in user reviews in terms of their F-score, but the performance and reliability is strongly dependent on the prompt. The study supplies a fully coded dataset of 300 reviews labeled by two human raters and an LLM, together with an initial prompt derived from two engineering iterations and coding guidelines based on the 10 Nielsen Usability Heuristics.","pith_inferences":["The same prompt strategy could be adapted to extract other non-functional requirements such as security or performance concerns from reviews.","Testing the prompt on a stream of live app-store reviews would reveal how well it handles new phrasing and emerging issues.","Embedding the classification step inside requirements-management tools could reduce the manual triage burden on product teams."],"forward_implications":["Development teams can process large volumes of user reviews quickly and at low cost to surface usability requirements.","LLMs provide an alternative to training dedicated machine-learning classifiers for requirements classification tasks.","The approach supports user-centered requirements elicitation by leveraging existing review data rather than new manual labeling.","Prompt refinement becomes a central engineering activity whose outcome directly affects the quality of extracted requirements."],"fun_headline_variants":["LLMs match human raters on usability in user reviews","New dataset tests LLMs extracting usability from app feedback","Initial prompts enable LLMs to filter usability relevant reviews","LLM performance on review usability varies with prompt design"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Human raters supply consistent ground-truth labels for usability aspects and the prompt developed on this dataset will produce reliable results on new reviews or different LLMs.","fun_headline_variants_meta":{"raw":{"variants":["LLMs match human raters on usability in user reviews","New dataset tests LLMs extracting usability from app feedback","Initial prompts enable LLMs to filter usability relevant reviews","LLM performance on review usability varies with prompt design"]},"model":"grok-4.3","cost_usd":0.00581,"raw_usage":{"total_tokens":2719,"prompt_tokens":736,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":58103000,"prompt_tokens_details":{"text_tokens":736,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1921,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":736,"tokens_out":62,"duration_ms":22181,"temperature":1.0,"reasoning_tokens":1921,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-14T20:20:58.447696+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Apply the final prompt to a fresh set of reviews independently labeled by new human raters and check whether the resulting F-score stays within the same range as the original human-to-human agreement.","supporting_citations":[],"review_version":1}