{"id":"11893b99-21ce-4ad2-8908-9bdebfe66fc4","arxiv_id":"2603.28677","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Grouping candidate requirements by user-feedback topics before matching them to reviews improves automated requirements prioritization and, with ChatGPT-extracted 'requires' pairs, strengthens NSGA-II release planning on a word-processor benchmark.","lead":"A software-engineering study modifies an existing feedback-driven requirements prioritizer so that candidate features are grouped by user-review topics before priorities are inferred, and tests the modified pipeline on release-planning data from four apps. The authors report better recall and precision than the baseline and show that LLM-discovered feature dependencies can be plugged into a search-based release planner.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparison baseline is a re-implemented ReFeed with altered similarity threshold and intention classifier, not the original ReFeed; claim of outperforming ReFeed is unsupported until compared with the original.","rationale":"The reader's weakest assumption was the validity of release timestamps as ground truth, which is a reasonable concern but is standard practice in NRP evaluations. A more concrete and directly testable issue is that the baseline ReFeed is not the original method, so the comparative claim is not yet established against the state of the art. The reader did mention this in the rationale ('compare against the original ReFeed settings') but did not make it the weakest assumption. My concern supports the same CONDITIONAL verdict, so no change to the reader's verdict is needed. The concrete test of rerunning the original ReFeed is specific and would resolve the ambiguity, whereas the ground truth concern is harder to settle empirically without a subjective judgment of priority.","tokens_in":18026,"tokens_out":9585,"duration_ms":107880,"concrete_test":"Run the original ReFeed method with its exact settings (Jaccard similarity threshold 0, ontology-based intention score, StanfordCoreNLP sentiment) on the same 94 prioritization instances and compute recall, precision, F1, and F2 at the same cutoffs. If iReFeed still significantly outperforms this original ReFeed, the claim holds; if not, the comparison is unfair and the conclusion needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that iReFeed consistently outperforms ReFeed rests on a comparison against a re-implemented baseline that differs from the original ReFeed in two key ways: (1) the similarity metric/threshold was changed from Jaccard with threshold 0 (as in [13]) to cosine similarity with threshold 0.1, and (2) the intention score was computed via a random forest classifier trained on [31] instead of the ontology used in [13]. These changes are applied to both ReFeed and iReFeed, so the experiments isolate the effect of clustering, but they do not establish superiority over the original ReFeed. If the original ReFeed's settings yield different (possibly better) prioritization performance, the claim 'iReFeed outperforms ReFeed' may be false. The paper acknowledges these modifications (Section 3, Steps 2–3) but does not compare against the original implementation or justify that the modified baseline is equivalent or better. This is a direct threat to the headline comparative claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes iReFeed, an extension of ReFeed (Kifetew et al., 2021) for user-feedback-driven requirements prioritization. Instead of associating each user message with a single requirement, iReFeed first clusters requirements topically using LDA or BERTopic over user reviews, then associates feedback at the cluster level, and finally computes per-requirement priorities using a modified ReFeed formula with cluster-coherence weighting. The paper evaluates four iReFeed variants against a reimplemented ReFeed on 94 prioritization instances from Discord, Microsoft 365 Word, Webex, and Zoom, using release-note timestamps as ground truth. A second experiment (RQ2) uses ChatGPT to extract asymmetric 'requires' pairs from the Word Processor benchmark, comparing whole-set prompting against cluster-wise prompting. A third experiment (RQ3) adds a dependency value derived from these pairs as a third NSGA-II objective and reports that iReFeed solutions occupy a larger share of the reference Pareto front than the baseline. The paper concludes that interconnectedness improves prioritization, enables automated dependency discovery, and strengthens search-based release planning.","tokens_in":18330,"tokens_out":6338,"duration_ms":62075,"significance":"The work is potentially valuable: it operationalizes requirements interconnectedness in a feedback-driven prioritization pipeline, and it makes a credible attempt to connect CrowdRE with SBSE. The authors provide publicly available datasets and implementations, which is exemplary for reproducibility. The RQ2 use of modern LLMs for dependency extraction is timely. However, the headline claims are weakened by evaluation choices: the ReFeed comparison is against a reimplemented baseline, the release-timestamp ground truth is a proxy whose validity is not established, the RQ3 comparison uses different objective spaces, and the leakage discussion in RQ2 is logically incorrect. If these concerns are addressed with additional experiments and re-analysis, the core idea could become a solid contribution.","major_comments":[{"comment":"The comparison against ReFeed is not against the original method. The paper reimplements ReFeed with cosine similarity threshold 0.1 instead of the Jaccard threshold 0 used in [13], and replaces the ontology-based intention score with a random forest classifier trained on [31]. These changes are applied to both ReFeed and iReFeed, so the experiment isolates cluster-level association, but the claim 'iReFeed consistently outperforms ReFeed' (Abstract, §4.2) is unsupported for the original ReFeed. Please compare against the original configuration or provide evidence that the modified baseline is equivalent or stronger.","section":"§3, Steps 2–3 and §4.2"},{"comment":"The evaluation labels requirements released in the earlier of two consecutive periods as the true top-k set. The statement 'The release timestamps give rise to the prioritization’s ground truth in an authoritative way' is not justified: release timing is often determined by marketing windows, dependencies, contracts, or other factors unrelated to feedback-driven priority. Because both methods are evaluated on this proxy, the measured recall/precision may reflect release-order prediction rather than prioritization quality. This threatens the principal RQ1 conclusion. Please validate the ground truth against explicit priority indicators or discuss and mitigate this confound.","section":"§4.1, ground truth definition"},{"comment":"The RQ3 comparison is not methodologically fair. The baseline NSGA-II optimizes two objectives (value, cost), while iReFeed NSGA-II optimizes three (value, cost, D-value). The reference Pareto front is then constructed using only value and cost, and performance is measured by the fraction of each algorithm's solutions on that 2-D front. The third objective is ignored in the evaluation, so a 3-objective search can produce different 2-D projections for reasons unrelated to solution quality. Please either solve the same bi-objective problem with D-value as a constraint, evaluate in the 3-objective space (e.g., hypervolume), or otherwise make the comparison comparable.","section":"§6, Table 6 and Fig. 5"},{"comment":"The data-leakage reasoning is backwards. Because the Word Processor benchmark was published in 2016 and ChatGPT 4.5/4o were trained on web data up to their release dates, the dataset is more likely, not less, to have been memorized. The low precision (≤0.30) of the extracted pairs also means the D-value fed to NSGA-II in RQ3 is largely noise. The paper should test for leakage (e.g., pre-registered prompts, random splits) and validate the 'requires' pairs before using them as an objective.","section":"§5, discussion of data leakage"}],"minor_comments":[{"comment":"The figure reports averages without error bars or confidence intervals, and the text mentions combined Wilcoxon p-values without giving the actual p-values or effect sizes. Please include per-instance distributions or at least standard deviations.","section":"Fig. 3"},{"comment":"For ChatGPT 4o, iReFeed finds 32 pairs, the same count as baseline; the claim that 'iReFeed does help uncover additional pairs' needs an overlap analysis to show the pairs are different and not simply a reordering.","section":"Table 4"},{"comment":"α(FC) is defined as min(1, average pairwise similarity); the notation makes it look like a per-summation weight. Clarify that α(FC) is constant for a cluster and should be moved outside the sum.","section":"Eq. (3)"},{"comment":"The change from Jaccard threshold 0 to cosine threshold 0.1 is described as 'slightly increased'; these thresholds operate on different similarity scales and are not directly comparable. Please justify the equivalence.","section":"§3, Step 2"},{"comment":"Reference [50] is a Substack blog post; replace with a peer-reviewed model comparison.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core idea and open artifacts, but the central comparison is against a modified baseline and the ground truth's validity is questionable. The RQ3 comparison is unfair because the objective spaces differ, and the RQ2 leakage discussion is logically flawed. These issues require substantial additional experiments and re-analysis rather than minor edits. I do not see evidence of misconduct; the concerns are about experimental design and interpretation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my quick take. The paper proposes iReFeed, an enhancement of ReFeed that clusters user-feedback topics, groups requirements by cluster, and then uses LLMs to extract \"requires\" pairs, finally feeding a dependency objective into NSGA-II. The main evidence is a comparison on 94 instances from four apps claiming iReFeed consistently outperforms ReFeed. That claim is not yet supported, because the baseline is a reimplementation of ReFeed with two material changes: cosine similarity threshold 0.1 instead of the original Jaccard threshold 0, and a random-forest intention classifier instead of the ontology in Kifetew et al. The changes are symmetric — both ReFeed and iReFeed use them — so the experiment does isolate the clustering effect, but the paper never shows that the reimplementation behaves like the original ReFeed. If the original settings give different or better results, the headline conclusion may not hold.\n\nWhat's genuinely good: the cluster-level feedback association is a sensible extension, and the coherence weighting (LDA-C, BERTopic-C) is a reasonable addition. The LLM prompting experiment is a nice idea — feeding clusters rather than all 50 requirements — and the NSGA-II integration is a useful bridge between CrowdRE and SBSE. The paper shares data and code on Zenodo, uses external release notes as ground truth, runs Wilcoxon tests, and is unusually transparent about its limitations, including the low LLM recall. That counts for something.\n\nThe soft spots, in rough order: the baseline mismatch is the big one; it needs a direct comparison against the original ReFeed or a justification that the changed components are equivalent. Second, the ground truth from release timing is a proxy for priority — marketing windows, contractual obligations, resource constraints all affect release decisions — so calling it \"authoritative\" overstates the case. Third, the RQ2 result is real but weak: the best combined F1 is 0.22. The paper acknowledges this, but the conclusion is about additional pairs found, not about accurate dependency identification. Fourth, the RQ3 evaluation uses the share of the reference Pareto front, a metric that can reward generating more diverse solutions; the paper mentions hypervolume as future work, which is honest, but the positive conclusion rests on that metric. Finally, several free parameters (topic count, association threshold, lookback window) are tuned on the same data; a sensitivity analysis would strengthen the claims.\n\nBottom line: this is a paper for the CrowdRE and SBSE community. It deserves a serious referee, but the comparative claim needs to be reworked — either by running against the original ReFeed or by arguing why the changed baseline is a fair comparison. I would send it to review, conditional on major revision.","headline":"The clustering idea and the shared artifacts are solid, but the claim that iReFeed consistently outperforms ReFeed rests on a re-implemented baseline with different settings, so the head-to-head comparison needs to be redone or justified before the headline holds.","tokens_in":18755,"tokens_out":3592,"would_cite":true,"duration_ms":38838,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Requirements rank better when user feedback is pooled by topic clusters, outperforming a state-of-the-art independent-ranking method.","keywords":["requirements prioritization","user feedback","CrowdRE","requirements clustering","topic modeling","requires relations","NSGA-II","next release problem"],"falsifier":"Compare iReFeed against ReFeed on prioritization instances where ground truth is set by explicit expert judgment of priority rather than release timestamps. Concretely, take the Discord, Word, Webex, and Zoom release notes, have independent practitioners rank each instance's top-k requirements, and recompute F2 at top-k; if iReFeed's advantage disappears or reverses, the claim that interconnectedness improves prioritization is not supported.","tokens_in":17935,"feed_emoji":"🧩","tokens_out":3221,"duration_ms":31810,"temperature":0.7,"pith_summary":"The paper tries to show that requirements prioritization improves when requirements are treated as interconnected rather than independent. It proposes iReFeed, which uses topic modeling over user app reviews to cluster candidate requirements, associates feedback at the cluster level, and ranks requirements within that context. On 94 prioritization instances from four real-world apps, iReFeed outperforms ReFeed across recall, precision, F1, and F2. The paper also shows that feeding topically clustered requirements to ChatGPT surfaces additional 'requires' pairs, and adding a dependency value as a third NSGA-II objective increases the share of solutions on the reference Pareto front, improving search-based release planning.","feed_headline":"Clustering requirements by topic beats ranking them alone","feed_subtitle":"Clustering user reviews by topic lets iReFeed beat independent ranking and sharpens release planning.","key_machinery":"The central mechanism is the topic cluster: user feedback is topic-modeled, candidate requirements are grouped by the resulting topics, and all feedback associated with a cluster's requirements is pooled before priority scoring. Equations (2) and (3) replace ReFeed's per-requirement feedback mapping with a cluster-level mapping; a coherence factor α(FC) weights clusters by internal pairwise similarity. The second mechanism is the D-value, a count of how often a requirement appears as the right-hand side of a ChatGPT-discovered 'requires' pair, added as a third NSGA-II objective to bias release planning toward requirements that are prerequisites for others.","core_discovery":"The central claim is that interconnectedness itself, as derived from user feedback topics, is a source of prioritization signal. iReFeed first runs topic modeling (LDA or BERTopic) over large sets of Google Play reviews, groups the candidate requirements of a release into topically coherent clusters, then associates each cluster with the union of feedback messages from its member requirements. Priorities are computed from sentiment, intention, and similarity scores at the cluster level rather than per requirement. On 94 instances from Discord, Microsoft 365 Word, Webex, and Zoom, iReFeed consistently outperforms ReFeed, with the coherence-weighted LDA-C variant recommended. The paper further","pith_inferences":["If release timing itself encodes customer priorities imperfectly, the reported margin over ReFeed may shrink under ground truth derived from explicit stakeholder ranking; the release-timestamp assumption deserves direct testing.","The same topic-cluster step could be reused as a feature in other LLM-based requirements engineering tasks, such as traceability or inconsistency detection, not just prioritization.","The inverse D-value variant's poor performance suggests dependency-aware search is not a random perturbation; the direction of the dependency objective matters and could be tested on other search-based planning algorithms.","The 'requires' pairs from ChatGPT are noisy, so treating them as a soft objective rather than hard constraints may be why NSGA-II still helps; this soft-versus-hard design choice is testable."],"forward_implications":["Prioritization systems that treat requirements as independent leave usable signal on the table; cluster-level feedback pooling captures this signal and improves ranking quality.","Topic modeling over user reviews can double as a dependency-discovery aid: cluster-wise LLM prompting finds 'requires' pairs that whole-set prompting misses.","Adding dependency counts as an objective in search-based release planning shifts the Pareto front toward solutions that respect prerequisite structure.","The coherence-weighting variant suggests not all clusters are equal; more internally coherent clusters deserve stronger priority influence.","The method is fully automated from reviews to ranked requirements, so it scales to settings where manual AHP-style prioritization breaks down."],"fun_headline_variants":["iReFeed links related requirements to outrank independent ones","Cluster user feedback to prioritize requirements together","Interconnected requirements win: iReFeed beats ReFeed on 94 tasks","Grouping requirements by topic yields better priorities"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation treats vendor release timestamps as authoritative ground truth for which requirements were correctly prioritized; if release timing is driven by marketing windows, contracts, dependencies, or resource constraints rather than user-feedback-aligned priority, the comparison may not measure prioritization quality at all.","fun_headline_variants_meta":{"raw":{"variants":["iReFeed links related requirements to outrank independent ones","Cluster user feedback to prioritize requirements together","Interconnected requirements win: iReFeed beats ReFeed on 94 tasks","Grouping requirements by topic yields better priorities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000574,"raw_usage":{"total_tokens":2546,"prompt_tokens":742,"completion_tokens":1804,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":1749}},"tokens_in":486,"tokens_out":1804,"duration_ms":12818,"temperature":1.0,"reasoning_tokens":1749,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T05:36:22.604003+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare iReFeed against ReFeed on prioritization instances where ground truth is set by explicit expert judgment of priority rather than release timestamps. Concretely, take the Discord, Word, Webex, and Zoom release notes, have independent practitioners rank each instance's top-k requirements, and recompute F2 at top-k; if iReFeed's advantage disappears or reverses, the claim that interconnectedness improves prioritization is not supported.","supporting_citations":[],"review_version":1}