{"id":"9e59a5e0-cc79-4f6f-9147-c539e9ebf3f4","arxiv_id":"2507.07344","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A ChatGPT-based pipeline can draft explainability requirements and explanations from app reviews, but engineers still prefer manually written requirements and users find AI explanations stylish yet less correct.","lead":"This paper tests whether a chatbot can turn app-store reviews into formal explainability requirements and user-facing explanations, comparing the AI output with texts written by human requirements engineers. It finds that engineers mostly prefer the human-written requirements, while users are split on explanations, with AI occasionally winning on style but not on correctness.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported chi-square significance for RQ1 and RQ2 treats repeated votes by the same participants as independent, inflating p-values and leaving the paper's quantitative preference claims unsupported as analyzed.","rationale":"I read the paper as an empirical feasibility study whose central contribution is a tool, a released dataset, and a comparative evaluation. The tool and dataset are real contributions, and the 'human validation remains essential' conclusion is robust to reanalysis. The single most load-bearing weakness is the statistical inference: the chi-square tests in Sections V-A and V-B assume independence across all votes, but the design is a fully crossed repeated-measures setup (8/14 participants x 58 reviews). This is a concrete, verifiable flaw that directly affects the headline 'statistically significant' language. I agree with the reader's weakest assumption in part: the independence violation is the decisive concern, while the gold-standard provenance issue is secondary because even an imperfect manual baseline demonstrates feasibility and the comparative direction is not likely to reverse. The reported effect sizes are large enough that a mixed-effects reanalysis may still favor manual requirements, but the current p-values and the abstract's characterization are not justified. This supports, rather than changes, the reader's conditional verdict: the paper should be accepted subject to a clustered reanalysis and softened statistical claims. I do not see grounds for rejection or for promoting the current statistical claims as definitive.","tokens_in":13213,"tokens_out":4665,"duration_ms":56563,"concrete_test":"Download the released Zenodo dataset and reanalyze both preference studies with mixed-effects logistic regression: preference (manual=1, AI=0) ~ source + (1|participant) + (1|review), reporting the source coefficient with a profile-likelihood confidence interval. Also compute each participant's majority preference and run a participant-level sign test. If the RQ1 source effect remains significant at p<0.05 after clustering and RQ2 does not, the paper's directional claims stand but every reported chi-square p-value must be replaced with clustered estimates. If RQ1 loses significance or RQ2 gains significance, the abstract's quantitative conclusions require substantive revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing quantitative support is the claimed 'statistically significant' preference for manual artifacts (Sections V-A and V-B: chi2=35.31, p=2.81e-09 for requirements; chi2=8.28, p=0.004 for explanations). Those chi-square tests treat all 464 requirement votes and 812 explanation votes as independent, but each of the 8 engineers and 14 users voted on all 58 reviews, and each review was rated by every participant. Votes are therefore clustered by participant and by review, so the effective sample size is far smaller than the vote count and the reported p-values are inflated. The authors' own Fleiss' kappa values (0.49 and 0.16) show substantial rater dependence, yet the significance tests ignore it. For RQ1, the direction (296 vs 168) would plausibly survive a clustered reanalysis, but the specific p-value and the 'strong evidence' wording in Section V-A do not follow from the data as analyzed. For RQ2, the 447 vs 365 split is weak enough that correcting for clustering could erase statistical significance entirely, which matters because the abstract leans on the explanation preference result. The gold-standard and single-company concerns are real but secondary; the immediate issue is the invalid independence assumption behind the headline statistics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a tool-supported pipeline that uses ChatGPT to automatically derive explainability requirements from user reviews and generate corresponding UI explanations. The authors evaluate the pipeline on 58 Spotify user reviews by comparing its outputs against manually crafted artifacts produced by four requirements engineers in a workshop. Two online surveys are reported: eight requirements engineers voted on which requirement they preferred per review (RQ1), and fourteen end users voted on which explanation they preferred per review (RQ2). The paper reports a statistically significant preference for manual requirements (296 vs. 168 votes; chi-square = 35.31, p = 2.81e-09) and a statistically significant but weaker preference for manual explanations (447 vs. 365 votes; chi-square = 8.28, p = 0.004), along with per-criterion reason analyses and an evaluation of the tool's efficiency and acceptance.","tokens_in":13418,"tokens_out":6466,"duration_ms":66285,"significance":"If the results hold, the paper makes a useful contribution to requirements engineering: it provides an open dataset of 58 reviews with paired manual and AI-generated artifacts, a reproducible prompt-based tool, and empirical evidence that human validation remains essential in AI-assisted requirements derivation. The transparent reporting of raw vote counts and the public release of data and code are notable strengths. However, the central quantitative claims rest on chi-square tests that ignore the repeated-measures structure of the data, and some interpretive claims in the abstract overstate what the data show. The underlying direction of the RQ1 finding is plausible, but the strength of the evidence for both the requirements and explanation preferences is not established by the reported analysis. The paper's value would be substantially improved by a correct statistical analysis.","major_comments":[{"comment":"The chi-square tests reported in Sections V-A and V-B treat all 464 requirement votes and 812 explanation votes as independent observations. This assumption is violated because each of the 8 engineers and 14 users voted on all 58 reviews, so votes are clustered by both participant and review. The authors' own Fleiss' kappa values (0.49 for requirements, 0.16 for explanations) confirm substantial rater dependence. The reported p-values (2.81e-09 and 0.004) are therefore inflated and cannot support the 'statistically significant' preference claims. For RQ2, the 447 vs. 365 split is weak enough that a clustered reanalysis could eliminate significance entirely, which matters because the abstract and Section VI-A rely on the explanation-preference result. Please reanalyze using a method that accounts for clustering, such as a mixed-effects logistic regression with random intercepts for participant and review, or at minimum a paired within-participant analysis (e.g., a sign test on per-review outcomes or a Wilcoxon signed-rank test on per-participant preference counts), and report effect sizes with confidence intervals.","section":"V-A and V-B"},{"comment":"The abstract and Section VI-A state that 'AI-generated explanations are frequently preferred for their clarity and style.' This claim is not directly supported by the reported data. Table III shows Clarity and Style were cited as decisive reasons in a minority of votes (56 and 53 of the main-reason counts, respectively, versus 58 each for Correctness and Relevance), and the paper does not provide a per-source breakdown of votes by criterion. Given the overall preference for manual explanations and the non-significant per-criterion tests in Table IV, the evidence supports only a claim that AI explanations were sometimes preferred for stylistic reasons. Please either provide the missing breakdown or temper the wording.","section":"Abstract and VI-A"},{"comment":"The tool evaluation claims 'significant efficiency gains' in the conclusion (Section VII), but Section V-C reports only descriptive percentage reductions in formulation time (44.2% in Group A, 52.5% in Group B) with no inferential test, confidence intervals, or accounting for the crossover design. With n=8 participants, these differences should be tested with a paired analysis (e.g., Wilcoxon signed-rank test) or the wording should be softened to 'descriptive reductions in time' rather than 'significant efficiency gains.'","section":"V-C and VII"}],"minor_comments":[{"comment":"The language of the artifacts presented to participants is not stated. The pre-processing prompt in Section III-A instructs 'Use German,' while the requirement prompt uses the English sentence structure 'The system must explain...' and the reviews are described in Section VI-C as English-language Spotify reviews. Please clarify whether the manual and AI-generated requirements and explanations were in English or German when shown to the German participants, since a language mismatch could confound the preference results.","section":"III-A and IV"},{"comment":"The six quality criteria (tone, style, clarity, correctness, relevance, level of detail) are described as 'based on established standards in RE and explainability research,' but no specific reference is provided for this selection. Please cite the source(s) for these criteria.","section":"IV-C"},{"comment":"The per-criterion chi-square tests in Tables II and IV also ignore that participants could select multiple reasons for a single vote, so the counts are not independent. Since these results are not used to support a positive claim, a brief caveat would suffice.","section":"V-A, Table II"},{"comment":"The name 'Vogelsang' is rendered as 'V ogelsang' in multiple places in the text and reference [29]; please correct the typographical spacing.","section":"II"},{"comment":"The phrase 'randomly selected 58 reviews, ensuring that all types and categories of explanation needs were represented' is ambiguous; if the selection was stratified random, please state that explicitly.","section":"IV-B"},{"comment":"The statement 'The overall agreement rate was low (5%)' is unclear; please specify that this means 3 out of 58 reviews received unanimous agreement (approximately 5%).","section":"V-B"}],"recommendation":"major_revision","confidential_remarks":"The main statistical flaw (ignoring the repeated-measures structure in the chi-square tests) is serious because it directly affects the paper's headline quantitative claims. However, the flaw is fixable with a reanalysis, and the paper's dataset, tool, and empirical setup have sufficient value to justify a major revision rather than rejection. If the reanalysis shows the RQ2 preference is not significant, the authors need to substantially revise the abstract and discussion. The reviewer also notes that the paper's timing claim ('ChatGPT 3.5—the latest available model at the time of the study') seems inconsistent with a 2025 submission; the authors should specify the data collection period."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful, modest RE paper with a genuinely new pipeline and a released benchmark, but the headline significance tests are not valid as run. The core finding that engineers prefer manual requirements is directionally credible; the explanation-preference result is weaker than the abstract implies.\n\nWhat's new: first automatic pipeline from explanation needs in user reviews to structured explainability requirements plus matching UI explanations, with a public 58-review dataset and tool. The empirical setup is more careful than most: blinded, randomized artifact order, separate participant pools for requirements and explanations, Fleiss' kappa reported, and limitations acknowledged. Releasing the data and code is real value.\n\nSoft spots, in rough order. First, the chi-square tests in V-A and V-B treat every vote as independent. Each of the 8 engineers voted 58 times and each of 14 users voted 58 times, so votes cluster by participant and by review. The reported p-values are inflated; the stress-test note has this right. For RQ1, the 296 vs 168 split is large enough that a clustered analysis would very likely still favor manual requirements, but the specific p=2.81e-09 and the 'strong evidence' wording don't follow. For RQ2, the 447 vs 365 split is modest, kappa is only 0.16, and 31 of 58 reviews had no majority; correcting for clustering could easily erase significance. The abstract's 'frequently preferred for clarity and style' is not supported by the overall preference counts. Second, both the manual gold standard and the evaluators come from one industrial partner; even though the eight evaluators did not create the artifacts, they share company context and training, so the manual-versus-AI comparison is somewhat company-specific. That is a real but minor threat, and the paper acknowledges related external-validity limits. Third, the quality criteria and the one-to-one pairing of artifacts are author-defined; fine for a benchmark, but worth remembering when comparing numbers across future work.\n\nOverall: this deserves a serious referee. The statistical analysis needs re-doing with a mixed model or per-participant summaries, and the abstract should be reworded to match the actual evidence. If the authors fix that, the dataset alone is worth publishing, and the hybrid human-in-the-loop conclusion is sensible.\n\nRecommendation: send to peer review, with a request for a clustered reanalysis.","headline":"Useful new pipeline and benchmark for explainability requirements, but the headline significance tests ignore repeated-measures clustering and the abstract overstates the explanation-preference result.","tokens_in":13970,"tokens_out":2277,"would_cite":true,"duration_ms":24343,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A prompt-based ChatGPT pipeline derives explainability requirements and explanations from app reviews, but requirements engineers still significantly prefer human-written requirements.","keywords":["explainability","requirements engineering","user reviews","app reviews","large language models","ChatGPT","non-functional requirements","human-in-the-loop"],"falsifier":"Repeat the two preference studies with an analysis that accounts for repeated voting by the same participants, for example a mixed-effects model with participant random intercepts; if the explanation preference no longer differs from chance and the requirement preference shrinks toward the margin, the paper's claim of statistically significant preferences is not supported.","tokens_in":12979,"feed_emoji":"💬","tokens_out":8844,"duration_ms":89654,"temperature":0.7,"pith_summary":"User reviews express explanation needs in informal language, and turning those needs into structured explainability requirements is largely unautomated. This paper tries to close that gap with a tool-supported ChatGPT pipeline that summarizes a review, derives a requirement in a fixed “The system must explain...” form, and generates a matching user-facing explanation. In a comparative study, requirements engineers significantly preferred manually written requirements (296 vs 168 votes; $\\chi^2 = 35.31$, $p = 2.81 \\times 10^{-9}$), while end users only slightly preferred manual explanations (447 vs 365; $\\chi^2 = 8.28$, $p = 0.004$). The paper concludes that the pipeline is a useful drafting aid but that human validation is essential, especially for the correctness and relevance of requirements.","feed_headline":"AI drafts explainability requirements; engineers favor manual ones","feed_subtitle":"A test on app reviews: manual requirements win on correctness, AI explanations sometimes read better.","key_machinery":"The mechanism is a three-stage prompt-based pipeline built on a general-purpose pretrained language model: a pre-processing prompt summarizes the review and removes emotional content; a derivation prompt forces the output into the objective sentence pattern “The system must explain...”; and a generation prompt writes a short UI explanation from the derived requirement rather than from the raw review. The fixed requirement pattern is load-bearing because it makes outputs comparable, checkable, and traceable from need to requirement to explanation. The evaluation machinery is a blind paired-preference survey using six criteria (tone, style, clarity, correctness, relevance, and level of detail), with chi-square tests against a 50:50 baseline and an inter-rater agreement measure.","core_discovery":"The central claim is that a prompt-based generative language model can perform the whole arc from a raw user review to a structured explainability requirement and then to a user-facing explanation, and that the result is useful but not yet reliable enough to replace a human reviewer. The evidence is a paired, blinded comparison on 58 app reviews: eight requirements engineers voted on requirements and 14 end users voted on explanations. The paper reports a significant overall preference for manual requirements, a weaker but still significant preference for manual explanations, moderate inter-rater agreement for requirements ($\\kappa = 0.49$) and only slight agreement for explanations ($\\kappa = 0.16$). The authors read this as evidence that automation is feasible, that AI explanations can compete on clarity and style, and that correctness and relevance remain the decisive weaknesses for AI-generated artifacts.","pith_inferences":["Because every participant voted on all 58 items, the reported chi-square tests treat repeated measurements as independent; re-analyzing with per-participant random effects would likely widen the intervals, leaving the requirement preference intact but making the explanation preference fragile.","A direct next experiment would swap the application domain and the participant pool: if the manual-requirement preference persists among reviewers unfamiliar with the product, it reflects requirement quality; if it shrinks, part of the gap is product familiarity.","The correctness weakness suggests a concrete remedy the paper leaves untested: grounding the explanation step in system-specific documentation or feature specifications, which should eliminate plausible-but-wrong statements before users see them."],"forward_implications":["A development team can feed a backlog of app reviews into the pipeline and receive first-draft explainability requirements plus UI explanation texts, cutting per-task formulation time by roughly 44–52% in the tool trial.","Because engineers favored manual requirements mainly on relevance and correctness, AI-generated requirements should be routed through human review before entering a formal requirements document.","AI-generated explanations are competitive with human-written ones on clarity and style, so their most defensible near-term use is user-facing draft text that is verified before release.","The 58-review paired dataset of manual and AI artifacts gives later work a benchmark for measuring progress on automatic explainability requirement generation."],"supporting_citations":[{"why":"supplies the taxonomy of explainability needs that guided the selection and categorization of the 58 reviews.","marker":"[12]"},{"why":"provides the underlying review dataset from which explanation-need reviews were drawn.","marker":"[39]"},{"why":"frames LLM use in requirements engineering into understanding and generation tasks, positioning the prompt-based approach.","marker":"[29]"},{"why":"defines quality factors for textual requirements that inform both the derivation prompt and the evaluation criteria.","marker":"[7]"},{"why":"establishes the prior capability the paper extends: detecting and categorizing explanation needs in app reviews.","marker":"[9]"},{"why":"makes the study artifacts and code publicly available as a benchmark.","marker":"[14]"},{"why":"provides the validity-category framework used to structure the threats to validity.","marker":"[41]"}],"fun_headline_variants":["AI drafts explainability specs; humans still win on correctness","Explainability automation from reviews: promising but not ready","AI explanations read well but lack correctness, study finds","Automating explainability from user reviews: human edge remains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative conclusion rests on the manual artifacts created in a workshop with four requirements engineers from one industrial partner being a valid gold standard, and on the chi-square tests' assumption that all votes are independent even though each participant voted on all 58 items.","fun_headline_variants_meta":{"raw":{"variants":["AI drafts explainability specs; humans still win on correctness","Explainability automation from reviews: promising but not ready","AI explanations read well but lack correctness, study finds","Automating explainability from user reviews: human edge remains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000483,"raw_usage":{"total_tokens":2367,"prompt_tokens":909,"completion_tokens":1458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1393}},"tokens_in":525,"tokens_out":1458,"duration_ms":10910,"temperature":1.0,"reasoning_tokens":1393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:44:12.277635+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the two preference studies with an analysis that accounts for repeated voting by the same participants, for example a mixed-effects model with participant random intercepts; if the explanation preference no longer differs from chance and the requirement preference shrinks toward the margin, the paper's claim of statistically significant preferences is not supported.","supporting_citations":[{"cited_title":"Dataset: Gold standard dataset for explainability need detection in app reviews","cited_arxiv_id":null,"evidence_quote":"provides the underlying review dataset from which explanation-need reviews were drawn."},{"cited_title":"Explanation needs in app reviews: Taxonomy and automated detection,","cited_arxiv_id":null,"evidence_quote":"establishes the prior capability the paper extends: detecting and categorizing explanation needs in app reviews."},{"cited_title":"Dataset: Automatic generation of explainability requirements and software explanations from user reviews,","cited_arxiv_id":null,"evidence_quote":"makes the study artifacts and code publicly available as a benchmark."},{"cited_title":"Wohlin, P","cited_arxiv_id":null,"evidence_quote":"provides the validity-category framework used to structure the threats to validity."}],"review_version":1}