{"id":"6136c560-95f7-4c8c-a922-67e95a5bc89e","arxiv_id":"2507.09039","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A BiLSTM-LSTM seq2seq model with self-attention and CRF extracts software requirements from app reviews, scoring 0.96 F1 on one crowdsourced dataset but only 0.47 on a manually annotated dataset.","lead":"Researchers turned app review text into a sequence-labeling task and tested a neural sequence-to-sequence model with attention and a CRF layer. On a large crowdsourced review set the model reached 0.96 F1, but on a smaller manually labeled set it was only comparable to the best baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.96 F1 rests on an undefined 'requirement-level' matching rule; without exact-span F1, the comparison to RE-BERT and T-FREX is not meaningful.","rationale":"The reader's weakest assumption targets the comparability of the baseline numbers in Tables III and IV, specifically that they were produced under the same splits, preprocessing, and evaluation metric. I agree that this is a serious defect and that the paper gives no evidence of how the baseline values were obtained. My stress-test reading points to an even more fundamental issue directly upstream of that comparison: the paper does not define the matching rule for the proposed model's own 'requirement-level' F1. If the authors counted partial overlaps or any token overlap, then even the Seq2seq 0.96 figure is not a standard NER span-F1 and cannot be compared to typical RE-BERT/T-FREX scores, which presumably use exact span matching. This is not an external-consensus disagreement; it is an internal ambiguity in the evaluation design. The concern is concrete and checkable because the code is released. If the exact-span F1 reproduces the reported numbers and equals or exceeds the baselines under identical conditions, the central claim would be substantially repaired. Absent that check, the evidence does not support the claimed outperformance, so the reader's REJECT verdict remains appropriate. I mark agreement as 'partial' because the reader identified baseline provenance, whereas my concern centers on the undefined matching criterion that makes any numeric comparison ambiguous, including the proposed model's own score.","tokens_in":12193,"tokens_out":5354,"duration_ms":71194,"concrete_test":"Using the released Zenodo code and Dataset 2, compute Seq2seq F1 under three matching rules: (1) exact span equality, where a predicted BIO span must exactly equal the gold span; (2) partial overlap, where any token overlap counts; and (3) token-level BIO accuracy. Then re-run or re-implement RE-BERT and T-FREX under the same cross-domain folds and the same exact-span evaluator. If Seq2seq's exact-span F1 drops below RE-BERT's reported 0.88, or if the 0.96 value holds only under partial-overlap matching, the headline comparison collapses. As a secondary diagnostic, report a stratified error analysis showing whether failures concentrate on multi-word requirements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the Seq2seq framework outperforms state-of-the-art methods on Dataset 2 with F1 0.96 vs 0.88 for RE-BERT and 0.44 for T-FREX. For this claim to hold, the scores being compared must be computed under the same matching criterion. Section V-D says only that predicted B/I tokens are unified into spans and then 'compared' with ground truth; it never states whether a match requires exact span equality, partial overlap, or token overlap. This ambiguity is load-bearing because Dataset 2 is dominated by single-word requirements (30,501 of 34,059 gold requirements are single tokens), so a lenient overlap rule can yield high F1 even when predicted span boundaries are wrong. Moreover, Tables III and IV report RE-BERT and T-FREX numbers without stating the protocol used to obtain them, whether they were re-run under identical cross-domain folds, or whether those baselines used the same matching rule. Without this information, the headline 'outperformed existing state-of-the-art methods' may reflect evaluation-metric mismatch rather than genuine model superiority. The released code makes this concern testable, which is a mitigating factor, but the paper as written does not establish metric equivalence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a sequence-to-sequence (Seq2seq) framework for extracting software requirements from mobile app reviews, reformulating the task as BIO-tagging/NER. The architecture is a BiLSTM encoder, an LSTM decoder, self-attention, GloVe embeddings, and a CRF layer. The model is evaluated on two datasets: a manually annotated set of 1,000 reviews (Dataset 1) and a crowdsourced set of 23,816 reviews (Dataset 2), using a leave-one-domain-out cross-validation strategy. The reported mean F1 is 0.47 on Dataset 1 and 0.96 on Dataset 2, compared with RE-BERT and T-FREX. The paper characterizes the results as preliminary and includes a research plan for further validation.","tokens_in":12393,"tokens_out":5002,"duration_ms":57600,"significance":"If the reported F1 of 0.96 on Dataset 2 is credible and comparable to the baseline numbers, the framework would be a substantially faster and simpler alternative to transformer-based methods for requirements extraction from app reviews, with the additional strength of publicly released code and evaluation on two independently curated datasets. The use of a cross-domain protocol and averaging over 15 runs are also positive aspects. However, the evaluation-matching rule is underspecified and the provenance of the baseline scores is unclear, so the central comparative claim is not yet established.","major_comments":[{"comment":"The requirement-level evaluation is not precisely defined. Section V-D states that predicted B/I tokens are unified into spans and then compared with ground truth, but it never states whether a match requires exact span equality, partial overlap, or token-level overlap. This is load-bearing because Table II shows that 30,501 of 34,059 gold requirements in Dataset 2 are single-word; a lenient overlap rule can yield high F1 even when multi-word boundaries are wrong. The authors should specify the matching rule and report exact-span F1, ideally alongside token-level F1, for both datasets.","section":"V-D, Tables III and IV"},{"comment":"The baseline scores for RE-BERT and T-FREX are presented without any statement of how they were produced. The paper does not say whether these numbers were re-run under the same cross-domain folds, the same preprocessing, the same tag vocabulary, or the same matching rule used for the proposed model. T-FREX, for example, uses B-feature/I-feature tags, and its original evaluation protocol may differ in other ways. If the numbers are copied from the original papers, the comparison is not valid; if they were re-run, the configuration and code should be described. Without this information, the claim that Seq2seq outperformed state-of-the-art methods is unsupported.","section":"V-D, Tables III and IV"},{"comment":"The proposed architecture is under-specified. There are no equations for the BiLSTM encoder states, the self-attention scores, the LSTM decoder update, or the CRF scoring function. The description says the decoder generates tag representations but also states that the decoder is not directly used to generate BIO tags; it is unclear whether the CRF is applied to decoder outputs, to encoder outputs, or as a global re-ranker, and what loss function is optimized. This makes the framework non-reproducible from the paper alone despite the availability of code.","section":"IV"},{"comment":"The paper states that 15 runs were performed for each app/category, but Tables III and IV report only mean F1 values with no standard deviations, ranges, or significance tests. The reported difference between Seq2seq (0.96) and RE-BERT (0.88) on Dataset 2 could fall within run-to-run variation, so the authors should report variance across the 15 runs and test the statistical significance of the differences.","section":"V-C and VI, Tables III and IV"}],"minor_comments":[{"comment":"The phrase 'condition random fields' should be 'conditional random fields'.","section":"I"},{"comment":"The app name is given as 'Photo editor' in Table I but as 'Phone editor' in Table III; the naming should be consistent.","section":"Tables I and III"},{"comment":"References [12] and [20] both cite the same GloVe paper; they should be consolidated into a single reference.","section":"References"},{"comment":"The runtime comparison of 50 minutes per iteration for RE-BERT versus 15 minutes for Seq2seq is anecdotal; the authors should specify what 'iteration' includes and whether the comparison was made on the same hardware and folds.","section":"VI"},{"comment":"The research plan itself acknowledges the lack of an ablation study and sensitivity analysis; these should be performed before attributing the results to specific components such as the CRF layer or self-attention.","section":"VIII"}],"recommendation":"major_revision","confidential_remarks":"The central contribution could be defensible after a substantial revision, but the evaluation protocol needs to be made rigorous: exact-span F1, re-run baselines under an identical protocol, and variance reporting. If the Dataset 2 result drops substantially under exact-span matching, the contribution should be reframed accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first time I've seen a seq2seq NER framework applied to extracting requirements from app reviews, and the authors release their code. That's a real thing. The empirical claim—0.96 F1 on the T-FREX dataset—is striking, but as written it doesn't hold up because the evaluation protocol is under-defined.\n\nThe paper does some things right. The task framing is clear, they use two separately curated datasets (manual vs crowdsourced), and they adopt the cross-domain train/test splits from the originals. They report 15 runs per fold, which shows they thought about stability, even though they don't give variance. They also explicitly call the work preliminary and list what would strengthen it.\n\nThe soft spots are where the reader's report and the stress-test note land. Section V-D says predicted B/I tokens are unified into spans and then 'compared' with ground truth, but never says whether a match requires exact span equality, partial overlap, or token overlap. That matters because 30,501 of 34,059 gold requirements in Dataset 2 are single tokens, so a lenient overlap rule inflates F1. The same section gives no source or protocol for the RE-BERT and T-FREX numbers in Tables III–IV. If those came from the original papers with different splits or matching rules, the comparison is meaningless. The architecture description is also vague: the decoder and CRF interaction is never spelled out, and there are no ablations for the self-attention or CRF components.\n\nNone of this makes the idea wrong. The code is public, so a referee could re-run with a defined metric and the baselines under the same folds. But the paper as written does not establish the superiority claim, and the 0.96 in particular is not credible without error analysis and a defined evaluation.\n\nWho's this for? Someone working on app-review mining or RE-NER who wants to see a seq2seq baseline and has time to verify it. I'd send it to peer review—the question is testable and the artifact is there—but I'd expect the referee to require a precise evaluation protocol, baseline re-runs, and variance reporting before accepting it. I wouldn't cite it as evidence until that happens.","headline":"A plausible seq2seq NER application to app-review requirements, but the headline 0.96 F1 is not interpretable until the matching rule and baseline protocols are pinned down.","tokens_in":12946,"tokens_out":3112,"would_cite":false,"duration_ms":33602,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that extracting software requirements from app reviews can be reformulated as a sequence-to-sequence NER task, and that a lightweight BiLSTM-LSTM model with self-attention and CRF outperforms transformer-based baselines…","keywords":["requirements extraction","app reviews","sequence-to-sequence modeling","named entity recognition","BiLSTM","self-attention","conditional random fields","software engineering"],"falsifier":"Run RE-BERT and T-FREX on the same train/test app and category splits described in Section V-C, applying the same tokenization, lemmatization, and requirement-level F1 scoring; if the reproduced baseline scores differ materially from 0.48/0.12 on Dataset 1 and 0.88/0.44 on Dataset 2, the claim that Seq2seq outperforms existing state-of-the-art no longer has a valid basis.","tokens_in":11967,"feed_emoji":"📱","tokens_out":6920,"duration_ms":72860,"temperature":0.7,"pith_summary":"Mobile app reviews contain a running list of feature requests, but they are noisy and informal, so automated extraction of requirements has had only modest success (F1 around 0.4-0.5 for prior state-of-the-art). This paper tries to improve that by treating requirements extraction as a named-entity-recognition task where the model generates a BIO tag sequence (B for beginning of a requirement, I for inside, O for outside) for each review. Its Seq2seq framework, built from a BiLSTM encoder, an LSTM decoder, self-attention, GloVe embeddings, and a CRF layer, is reported to reach F1 of 0.96 on a 23,816-review crowdsourced dataset, versus 0.88 for RE-BERT and 0.44 for T-FREX, and a comparable 0.47 on a 1,000-review manually annotated dataset. The reason to care is that this would make requirements extraction faster and much cheaper: the authors report 15-minute training iterations where the BERT-based baseline needed 50 minutes on the large dataset.","feed_headline":"Seq2seq tagger beats BERT on app-review requirements","feed_subtitle":"Lightweight BiLSTM-LSTM with self-attention and CRF hits 0.96 F1 on 23,816 reviews and trains in 15 minutes.","key_machinery":"The load-bearing object is the Seq2seq NER framework itself: an encoder-decoder architecture that takes a sequence of GloVe word vectors, encodes the review with a BiLSTM, applies self-attention to emphasize requirement-bearing spans, decodes with a single-layer LSTM, and finishes with a CRF that constrains tag transitions so the final sequence of BIO tags is legal. The CRF is the element that prevents nonsensical outputs such as two consecutive B tags for one requirement, and the self-attention component is what allows the model to focus on the requirement-like parts of a noisy sentence. The whole pipeline is what carries the claim that requirements extraction can be treated as tag-sequence generation rather than per-token classification.","core_discovery":"The central claim is that the sequence-to-sequence generation formulation is a better fit for app-review requirements extraction than the token-classification formulation used by prior work. Concretely, the authors show that a BiLSTM encoder reading each review in both directions, followed by self-attention, an LSTM decoder, and a CRF tagger, can predict BIO tags over the review tokens. On the crowdsourced T-FREX dataset the model attains a mean F1 of 0.96, exceeding the RE-BERT baseline at 0.88 and far exceeding T-FREX at 0.44, with consistent per-category scores between 0.94 and 0.98. On the manually labeled RE-BERT dataset of eight apps, the same framework reaches 0.47, essentially tied with RE-BERT's 0.48 and well above T-FREX's 0.12. The authors attribute the large dataset gap to annotation style and dataset size, and they present the runtime advantage (15 vs 50 minutes per iteration) as a practical benefit.","pith_inferences":["What the paper leaves untested is the individual contribution of the CRF layer; an ablation removing CRF (which the authors list as future work) would show whether the gain comes from the tag-transition constraints or from the attention-plus-seq2seq backend.","The near-perfect scores on the crowdsourced dataset may partly reflect the way Dataset 2 was created: requirements were transferred into reviews from an external crowdsourced list, so the tag patterns may be simpler and more repetitive than in naturally written feature requests; testing on a fresh manually annotated sample from the same 10 categories would separate this effect.","The same encoder-decoder tagging scheme should transfer to other noisy software texts, such as issue comments or forum posts, where requirement-like spans appear in informal language; if it fails there, the method is tuned to review-specific phrasing."],"forward_implications":["If the reported numbers hold, a lightweight recurrent model can replace transformer-based encoders for this task, cutting training time from about 50 minutes to 15 minutes per iteration on the large dataset.","The cross-domain training strategy means the model can be applied to an entirely new app category without requiring labeled reviews from that category.","Requirement extraction becomes part of the standard NER toolchain for app-review mining, so downstream tasks like requirement clustering and sentiment analysis can consume the extracted spans directly.","The reformulation as a generation task, rather than classification, gives other review-based RE tasks a template for using encoder-decoder architectures."],"supporting_citations":[{"why":"supplies Dataset 1, the RE-BERT baseline scores, and the cross-domain training strategy the paper reproduces.","marker":"[8]"},{"why":"supplies Dataset 2, the T-FREX baseline scores, and the crowdsourced annotation methodology behind the large dataset.","marker":"[11]"},{"why":"provides the GloVe word embeddings the framework uses for input representation.","marker":"[12]"},{"why":"introduces the self-attention mechanism that focuses the model on requirement-bearing spans.","marker":"[14]"},{"why":"provides the CRF model used to enforce legal BIO tag transitions.","marker":"[16]"},{"why":"demonstrates the seq2seq framework for fine-grained NER that this paper adapts to app reviews.","marker":"[21]"},{"why":"supplies the evidence that GPT-style decoder-only models perform poorly and are costly for domain-specific NER, motivating the seq2seq choice.","marker":"[22]"}],"fun_headline_variants":["Seq2seq with attention and CRF: 0.96 F1 on app-review requirements","Seq2seq generation beats token tagging for app-review requirements","Lightweight BiLSTM-LSTM+attention+CRF hits 0.96 F1 on 23k reviews","Sequence-to-sequence framework improves app review requirement extraction","From app reviews to requirements: a seq2seq NER approach"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison's reliability rests entirely on the baseline F1 values in Tables III and IV having been measured under the exact same cross-domain splits, preprocessing steps, and requirement-level evaluation metric as the Seq2seq system; the paper reports those baseline numbers without showing how they were produced.","fun_headline_variants_meta":{"raw":{"variants":["Seq2seq with attention and CRF: 0.96 F1 on app-review requirements","Seq2seq generation beats token tagging for app-review requirements","Lightweight BiLSTM-LSTM+attention+CRF hits 0.96 F1 on 23k reviews","Sequence-to-sequence framework improves app review requirement extraction","From app reviews to requirements: a seq2seq NER approach"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000538,"raw_usage":{"total_tokens":2605,"prompt_tokens":992,"completion_tokens":1613,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1511}},"tokens_in":608,"tokens_out":1613,"duration_ms":14951,"temperature":1.0,"reasoning_tokens":1511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:05:59.614195+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RE-BERT and T-FREX on the same train/test app and category splits described in Section V-C, applying the same tokenization, lemmatization, and requirement-level F1 scoring; if the reproduced baseline scores differ materially from 0.48/0.12 on Dataset 1 and 0.88/0.44 on Dataset 2, the claim that Seq2seq outperforms existing state-of-the-art no longer has a valid basis.","supporting_citations":[{"cited_title":"Re-bert: automatic extraction of software requirements from app reviews using bert language model,","cited_arxiv_id":null,"evidence_quote":"supplies Dataset 1, the RE-BERT baseline scores, and the cross-domain training strategy the paper reproduces."},{"cited_title":"T- frex: A transformer-based feature extraction method from mobile app reviews,","cited_arxiv_id":null,"evidence_quote":"supplies Dataset 2, the T-FREX baseline scores, and the crowdsourced annotation methodology behind the large dataset."},{"cited_title":"GloVe: Global vectors for word representation,","cited_arxiv_id":null,"evidence_quote":"provides the GloVe word embeddings the framework uses for input representation."},{"cited_title":"An introduction to conditional random fields,","cited_arxiv_id":null,"evidence_quote":"provides the CRF model used to enforce legal BIO tag transitions."},{"cited_title":"Fine grained named entity recognition via seq2seq framework,","cited_arxiv_id":null,"evidence_quote":"demonstrates the seq2seq framework for fine-grained NER that this paper adapts to app reviews."},{"cited_title":"Decoding domain-specific ner: A performance evaluation of chatgpt, bi-lstm, and bert,","cited_arxiv_id":null,"evidence_quote":"supplies the evidence that GPT-style decoder-only models perform poorly and are costly for domain-specific NER, motivating the seq2seq choice."}],"review_version":1}