{"id":"14e38cff-3528-40c5-bd4b-bddef3fd76b8","arxiv_id":"1908.00614","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SecureReqNet, a CNN trained on word2vec embeddings from CVE entries and issue trackers, classifies security-related issues with 96% accuracy on open-source data and 71.6% on industrial requirements.","lead":"This paper presents SecureReqNet, a two-stage neural network that reads issue descriptions and decides whether they are security-related. The authors report 96% accuracy on open-source issues and 71.6% on a small set of industrial requirements.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Word embeddings are trained on the test set text, so the 96% open-source accuracy does not come from truly unseen issues.","rationale":"The reader's weakest assumption (ground-truth label noise) is plausible, but the paper partially addresses it by manually verifying 10% of non-SR samples, and the 'security' tag on major projects is a reasonable proxy. The more fundamental problem is that the evaluation does not satisfy the 'unseen' condition the paper itself advertises: the word2vec embeddings are trained on the full Embedding Dataset, which includes the test issues. This is a concrete, verifiable violation of train/test separation, whereas label noise is an external threat that may or may not be severe. If the embeddings are retrained without test data and accuracy remains high, the core claim survives; if accuracy drops, the headline result is an artifact of transductive learning. Either way, the current paper's 96% open-source result is not a valid estimate of performance on truly unseen issues. The industrial result (71.6% on 69 stories) is unaffected but too small to carry the central claim alone. Thus the paper remains CONDITIONAL: the authors should re-run the open-source evaluation with the corrected embedding split before the accuracy claim can be accepted. This is consistent with the reader's verdict, so I recommend no change to the verdict.","tokens_in":8059,"tokens_out":14637,"duration_ms":144397,"concrete_test":"Retrain the Word2Vec embeddings using only the training and validation splits (plus the CVE and Wikipedia text, which are not in the test split), completely excluding the 1,032 test issues. Then re-vectorize the test issues, retrain SecureReqNet (shallow) with the same hyperparameters and early-stopping protocol, and record test accuracy and AUC. Compare to the reported 0.96 accuracy and 0.984 AUC. If the retrained accuracy drops materially (e.g., below 0.90) or out-of-vocabulary test words are frequent, the reported performance is inflated by test-set leakage; if the accuracy remains near 96%, the embedding leakage is not material. The provided GitHub repository should make this test runnable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is a violation of the paper's claimed 'unseen' test set through unsupervised embedding pretraining. In Section III-A the authors state that they split the Embedding Dataset according to a 70%/20%/10% train/validation/test split, and Table I lists for each open-source source a single total count in the Embedding Dataset column (e.g., 4,575 GitHub SR) with training/validation/test counts that sum to it. The Word2Vec model (Section II-A) is trained on the entire Embedding Dataset, which therefore includes the 1,032 test issues. The resulting word vectors are then used to vectorize the test issues before they are fed to the CNN, so the model's input representation for each test issue was learned from the text of that very issue. This contradicts the claim that the test issues are 'unseen by the model' (Section III-A). In deployment, embeddings would be trained only on data available at training time, so this transductive setup can inflate accuracy by adapting the feature space to the test corpus, including its rare or project-specific vocabulary. The industrial test is not affected (Table I shows Industry documents only in the Testing column), but the headline open-source result is compromised.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SecureReqNet, a two-stage neural approach for automatically classifying software issue-tracker descriptions as security-related (SR) or non-SR. The first stage trains 100-dimensional skip-gram Word2Vec embeddings on a large corpus of CVE descriptions, GitHub/GitLab issues, and Wikipedia articles. The second stage uses these embeddings to vectorize issue texts and trains four CNN variants (shallow, deep, Alex-style, and an α-SecureReqNet hybrid) to perform binary classification. The authors evaluate the models on 1,032 temporally held-out open-source issues, reporting 96% accuracy for the best configuration, and on 69 industrial user stories from Cisco, reporting 71.6% accuracy. The paper claims a novel contribution in adapting CNNs for security-issue identification and provides an online appendix with code, data, and pre-trained models.","tokens_in":8279,"tokens_out":7855,"duration_ms":72555,"significance":"If validated, SecureReqNet would be a practical automation tool for agile security processes, helping teams identify security-critical issues, triage bug reports, and prioritize security requirements. The paper's strengths are its use of a balanced open-source test set, an explicit temporal-split design for the supervised stage, an industrial evaluation with a commercial partner, and the public release of code and data. However, the central performance claim is currently compromised by the fact that the unsupervised embedding model trains on the same test documents whose representations are later fed to the CNN, and the absence of baseline classifiers prevents the reader from attributing the observed accuracy to the proposed architecture rather than to the data or the label distribution.","major_comments":[{"comment":"The claim that test issues are 'unseen by the model' is not valid for the embedding stage. The Word2Vec model is trained on the entire Embedding Dataset, which includes the test documents (e.g., the GitHub SR row lists 4,575 documents in the Embedding Dataset and 458 in the Testing set). The resulting word vectors are used to vectorize the test issues before classification, so the feature representation for each test issue is learned from the text of that very issue. This transductive setup can inflate the reported 96% open-source accuracy by adapting the embedding space to the test corpus's vocabulary and wording. The authors should retrain the embeddings using only data available before the temporal cutoff of the test issues (or at least excluding test issues from the embedding corpus) and report the resulting accuracy.","section":"Section III-A and Table I"},{"comment":"The ground-truth non-SR labels are defined by the absence of a 'security' tag, with manual verification of only a 10% random sample. The paper does not report the outcome of this verification, nor any inter-rater agreement statistics. If a non-negligible fraction of the tag-based non-SR issues actually contain security-related content, then the accuracy numbers, which treat the tag proxy as ground truth, will overstate the classifier's ability to identify true security-relatedness. The authors should provide verification statistics (e.g., precision/recall of the tag proxy) or use a more rigorously validated labeling procedure.","section":"Section III-A"},{"comment":"The evaluation reports only the raw performance of the four architectures and contains no comparison against standard baselines, such as majority-class prediction, a keyword/regular-expression classifier, TF-IDF with logistic regression, or the k-NN approach of Riaz et al. [22]. Without such comparisons, the paper cannot support the claim that the two-phase CNN architecture itself is responsible for the observed accuracy. At a minimum, the authors should add a conventional baseline and a paired significance test (e.g., McNemar's test) on the same test set.","section":"Section IV"},{"comment":"The data-split description is internally inconsistent: for GitHub (Non-SR), Table I lists an Embedding Dataset total of 47,483, but the Training, Validation, and Testing counts (33,238 + 9,497 + 458) sum to 43,193, not 47,483; similar discrepancies appear for the CVE row. This makes it impossible to verify the claimed 70/20/10 temporal split and undermines the reproducibility of the evaluation. The table and accompanying text should be corrected and clarified, including exactly how the test set was sampled from the full corpus.","section":"Section III-A and Table I"}],"minor_comments":[{"comment":"The sentence 'We observed a test validation loss of 0.1313 and accuracy of 0.9736 compared to 0.0272 and 0.9936 for the α-SecureReqNet respectively' is confusing and does not directly match the values reported in Table II; please clarify which dataset (validation or test) and which configuration these numbers refer to.","section":"Section IV-A"},{"comment":"The caption states that the Word2Vec output is a softmax and that the network is trained with mean squared error, which is atypical for word2vec training; please specify the actual training objective (e.g., negative sampling or hierarchical softmax).","section":"Figure 1"},{"comment":"There is a typo: 'generalizablility' should be 'generalizability'. Also, the paper uses 'requirements' and 'issues' interchangeably for the industrial user stories; the terminology should be unified.","section":"Section III-A"},{"comment":"For reproducibility, the paper should report the Word2Vec hyperparameters (window size, negative samples, number of training epochs, and subsampling rate) in addition to the embedding dimension and dropout rate already given.","section":"Section II-A"}],"recommendation":"major_revision","confidential_remarks":"The embedding leakage is the key threat to the headline result; I recommend that the revision require the authors to re-run the evaluation with the test documents excluded from embedding training. The Table I inconsistency should also be checked during revision, and the online appendix code should be inspected to confirm whether the current implementation actually trains on the test text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Palacio et al. is an early application of word2vec + CNN to a real problem: flagging security-related issues in issue trackers. That's a useful triage aid, and the paper does the right things around it—released code and data, a temporal split that tries to mimic deployment, and a separate small industrial study. The combination of CVE text, issue text, and Wikipedia for embeddings is sensible.\n\nThe problem is the main result. The authors split the Embedding Dataset into train/validation/test, but they train the word2vec embeddings on the full Embedding Dataset. That includes the 1,032 test issues. So those test issues aren't 'unseen' in the way the paper claims—the embedding phase has already read their text. That's a transductive leak, and it can inflate the open-source accuracy (96%). The industrial evaluation isn't affected, since those 69 user stories are only in the test column, but 69 with no confidence intervals is a lot thinner than the abstract suggests.\n\nThe other gaps are the usual ones. No baselines—no bag-of-words, no logistic regression, no comparison to Han et al. or Riaz et al. That makes it hard to know how much the CNN adds. The labels are only as good as GitHub's 'security' tag, with a 10% manual check of non-SR text and no equivalent check on the SR side. And there are no error bars or statistical tests anywhere.\n\nNone of this is fatal to the idea. The leak is straightforward to fix by retraining embeddings only on the training+validation text, or on a corpus that excludes the test issues. With that fix, the accuracy may drop, but the approach would still be credible. The missing baselines are a matter of effort, not method.\n\nSo this is a paper for people who want to know whether neural text classification can be dropped into a security-requirements workflow. It's not a breakthrough, but it's a legitimate empirical artifact with real weaknesses that a referee can ask to have cleaned up. I'd send it to peer review rather than desk-reject it, and I'd expect a revision that fixes the embedding leak, adds baselines, and reports confidence intervals on the industrial set.","headline":"The paper is a reasonable engineering contribution, but the 96% open-source figure is compromised by training word embeddings on the test set, and the missing baselines and lack of error bars further limit the claims.","tokens_in":8815,"tokens_out":3813,"would_cite":true,"duration_ms":36817,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SecureReqNet claims a CNN can identify security-related issues from issue text alone, reaching 96% accuracy on open-source issues.","keywords":["security-related issues","issue tracking systems","convolutional neural networks","word embeddings","natural language processing","software security","requirements identification","deep learning"],"falsifier":"Ask several security experts to independently label a random sample of issues that the model was not trained on, including both tagged and untagged issues; if the model's agreement with expert labels is substantially below 96%, the high accuracy reflects noise in the tag-based labels rather than true security-relatedness.","tokens_in":7868,"feed_emoji":"🛡️","tokens_out":4788,"duration_ms":42063,"temperature":0.7,"pith_summary":"SecureReqNet is a two-phase neural approach that decides whether a software issue in a tracker describes security-related content using only the natural language of the issue description. The first phase learns word embeddings from a large corpus of vulnerability descriptions, open-source issue texts, and general articles; the second phase trains a convolutional neural network with multi-size n-gram kernels on labelled issues. The paper reports 96% accuracy on 1,032 unseen open-source issues and 71.6% accuracy on 69 industrial user stories from a commercial telecommunications project. If correct, this gives development teams an automated way to surface security-critical issues early, without needing structured or code-level information.","feed_headline":"Neural net flags security-related issues from text at 96% accuracy","feed_subtitle":"Two-phase model learns word meaning from vulnerability descriptions, then classifies open-source issues and industrial user stories.","key_machinery":"The load-bearing mechanism is the two-phase pipeline: an unsupervised Word2Vec skip-gram model that maps words from a combined corpus (vulnerability descriptions, issue texts, and general articles) into 100-dimensional vectors, followed by a supervised convolutional neural network whose convolutional layers use kernels of different n-gram widths (1-gram, 3-gram, 5-gram) to capture local word windows of varying length. The n-gram kernels let the network detect security-relevant phrases at different granularities, and max pooling collapses each feature map to a single value before a binary softmax layer produces the SR/non-SR decision.","core_discovery":"The central claim is that the semantics of security are learnable from text alone: a convolutional neural network operating purely on issue descriptions can separate security-related issues from non-security ones. The paper demonstrates this by pre-training skip-gram word embeddings on 52,908 vulnerability descriptions, over 52,000 open-source issue texts, and 10,000 general articles, then fine-tuning a supervised CNN on labelled issues. On a temporally separated test set of 1,032 unseen issues the best configuration, SecureReqNet (shallow), reaches 96% accuracy with an AUC of 0.984; on 69 industrial user stories the α-SecureReqNet configuration reaches 71.6% accuracy. The authors interpret the success as evidence that vulnerability corpora provide enough semantic context for a classifier to generalise to the more colloquial language of issue trackers.","pith_inferences":["The same two-phase recipe could be applied to other security-relevant text types, such as pull-request descriptions, commit messages, or security advisories, without architectural changes.","Because the industrial accuracy drops to 71.6% while open-source accuracy is 96%, the main obstacle is domain shift; a small amount of in-domain fine-tuning on industrial issues may close much of that gap.","The reliance on 'security' tags as ground truth suggests that the reported accuracy is an upper bound; a cleaner evaluation with expert-validated labels on both classes might lower the headline numbers."],"forward_implications":["Developers can receive automatic, real-time flags on incoming issues, helping security-sensitive bug reports get triaged before they are buried.","Teams using issue-tracker-based requirements can surface overlooked security-critical requirements during agile planning.","Automatically identified security issues can feed downstream tools for traceability, feature location, or assignment to security-experienced engineers.","The trained model and embeddings provide a reusable starting point for other security-related classification tasks on software text, a direction the authors explicitly propose for future work."],"supporting_citations":[{"why":"Supplies the Word2Vec skip-gram algorithm used to learn the semantic word embeddings that form the model's input representation.","marker":"[20]"},{"why":"Provides the n-gram kernel CNN architecture that SecureReqNet (shallow) adapts for binary security classification.","marker":"[12]"},{"why":"Supplies the AlexNet architecture that the deep variants adapt by replacing the final softmax with a binary classifier.","marker":"[14]"},{"why":"Establishes the prior art of identifying security requirements from natural language, which SecureReqNet extends to larger, more diverse issue-tracker data.","marker":"[22]"}],"fun_headline_variants":["SecureReqNet: CNN identifies security issues from text alone","96% accuracy flagging security issues via neural network","Two-phase CNN learns security semantics from vulnerability texts","AI model spots security-related issues in issue trackers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 'security' tag on open-source issues, with only 10% of non-security samples manually verified, is an accurate enough ground truth for what actually counts as security-related.","fun_headline_variants_meta":{"raw":{"variants":["SecureReqNet: CNN identifies security issues from text alone","96% accuracy flagging security issues via neural network","Two-phase CNN learns security semantics from vulnerability texts","AI model spots security-related issues in issue trackers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000426,"raw_usage":{"total_tokens":2190,"prompt_tokens":962,"completion_tokens":1228,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1165}},"tokens_in":578,"tokens_out":1228,"duration_ms":8352,"temperature":1.0,"reasoning_tokens":1165,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:43:35.016500+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask several security experts to independently label a random sample of issues that the model was not trained on, including both tagged and untagged issues; if the model's agreement with expert labels is substantially below 96%, the high accuracy reflects noise in the tag-based labels rather than true security-relatedness.","supporting_citations":[{"cited_title":"Mikolov, K","cited_arxiv_id":null,"evidence_quote":"Supplies the Word2Vec skip-gram algorithm used to learn the semantic word embeddings that form the model's input representation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the n-gram kernel CNN architecture that SecureReqNet (shallow) adapts for binary security classification."},{"cited_title":"Krizhevsky, I","cited_arxiv_id":null,"evidence_quote":"Supplies the AlexNet architecture that the deep variants adapt by replacing the final softmax with a binary classifier."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the prior art of identifying security requirements from natural language, which SecureReqNet extends to larger, more diverse issue-tracker data."}],"review_version":1}