Learning to Identify Security-Related Issues Using Convolutional Neural Networks

Carlos Bernal-C\'ardenas; Chris Shenefiel; Daniel McCrystal; David N. Palacio; Denys Poshyvanyk; Kevin Moran

REVIEW 4 major objections 4 minor 24 references

Learning to Identify Security-Related Issues Using Convolutional Neural Networks

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SecureReqNet claims a CNN can identify security-related issues from issue text alone, reaching 96% accuracy on open-source issues.

desk verdict The paper is a reasonable engineering contribution, but the 96% open-source figure is compromised by training word embeddings on the test set, and the missing baselines and lack of error bars further limit the claims. read the letter →

arxiv 1908.00614 v2 pith:DHPUMQC6 submitted 2019-08-01 cs.SE

David N. Palacio , Daniel McCrystal , Kevin Moran , Carlos Bernal-Cárdenas , Denys Poshyvanyk , Chris Shenefiel This is my paper · ORCID

classification cs.SE

keywords security-relatedissuesissuetrackingsystemsconvolutionalneuralnetworkswordembeddingsnaturallanguageprocessingsoftwaresecurityrequirementsidentificationdeeplearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SecureReqNet is a two-phase neural approach that decides whether a software issue in a tracker describes security-related content using only the natural language of the issue description. The first phase learns word embeddings from a large corpus of vulnerability descriptions, open-source issue texts, and general articles; the second phase trains a convolutional neural network with multi-size n-gram kernels on labelled issues. The paper reports 96% accuracy on 1,032 unseen open-source issues and 71.6% accuracy on 69 industrial user stories from a commercial telecommunications project. If correct, this gives development teams an automated way to surface security-critical issues early, without needing structured or code-level information.

What carries the argument

The load-bearing mechanism is the two-phase pipeline: an unsupervised Word2Vec skip-gram model that maps words from a combined corpus (vulnerability descriptions, issue texts, and general articles) into 100-dimensional vectors, followed by a supervised convolutional neural network whose convolutional layers use kernels of different n-gram widths (1-gram, 3-gram, 5-gram) to capture local word windows of varying length. The n-gram kernels let the network detect security-relevant phrases at different granularities, and max pooling collapses each feature map to a single value before a binary softmax layer produces the SR/non-SR decision.

What would settle it

Ask several security experts to independently label a random sample of issues that the model was not trained on, including both tagged and untagged issues; if the model's agreement with expert labels is substantially below 96%, the high accuracy reflects noise in the tag-based labels rather than true security-relatedness.

Watch

Extended reading notes

Core claim

The central claim is that the semantics of security are learnable from text alone: a convolutional neural network operating purely on issue descriptions can separate security-related issues from non-security ones. The paper demonstrates this by pre-training skip-gram word embeddings on 52,908 vulnerability descriptions, over 52,000 open-source issue texts, and 10,000 general articles, then fine-tuning a supervised CNN on labelled issues. On a temporally separated test set of 1,032 unseen issues the best configuration, SecureReqNet (shallow), reaches 96% accuracy with an AUC of 0.984; on 69 industrial user stories the α-SecureReqNet configuration reaches 71.6% accuracy. The authors interpret the success as evidence that vulnerability corpora provide enough semantic context for a classifier to generalise to the more colloquial language of issue trackers.

Load-bearing premise

The 'security' tag on open-source issues, with only 10% of non-security samples manually verified, is an accurate enough ground truth for what actually counts as security-related.

Editorial extensions

If this is right

Developers can receive automatic, real-time flags on incoming issues, helping security-sensitive bug reports get triaged before they are buried.
Teams using issue-tracker-based requirements can surface overlooked security-critical requirements during agile planning.
Automatically identified security issues can feed downstream tools for traceability, feature location, or assignment to security-experienced engineers.
The trained model and embeddings provide a reusable starting point for other security-related classification tasks on software text, a direction the authors explicitly propose for future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same two-phase recipe could be applied to other security-relevant text types, such as pull-request descriptions, commit messages, or security advisories, without architectural changes.
Because the industrial accuracy drops to 71.6% while open-source accuracy is 96%, the main obstacle is domain shift; a small amount of in-domain fine-tuning on industrial issues may close much of that gap.
The reliance on 'security' tags as ground truth suggests that the reported accuracy is an upper bound; a cleaner evaluation with expert-validated labels on both classes might lower the headline numbers.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Palacio et al. is an early application of word2vec + CNN to a real problem: flagging security-related issues in issue trackers. That's a useful triage aid, and the paper does the right things around it—released code and data, a temporal split that tries to mimic deployment, and a separate small industrial study. The combination of CVE text, issue text, and Wikipedia for embeddings is sensible.

The problem is the main result. The authors split the Embedding Dataset into train/validation/test, but they train the word2vec embeddings on the full Embedding Dataset. That includes the 1,032 test issues. So those test issues aren't 'unseen' in the way the paper claims—the embedding phase has already read their text. That's a transductive leak, and it can inflate the open-source accuracy (96%). The industrial evaluation isn't affected, since those 69 user stories are only in the test column, but 69 with no confidence intervals is a lot thinner than the abstract suggests.

The other gaps are the usual ones. No baselines—no bag-of-words, no logistic regression, no comparison to Han et al. or Riaz et al. That makes it hard to know how much the CNN adds. The labels are only as good as GitHub's 'security' tag, with a 10% manual check of non-SR text and no equivalent check on the SR side. And there are no error bars or statistical tests anywhere.

None of this is fatal to the idea. The leak is straightforward to fix by retraining embeddings only on the training+validation text, or on a corpus that excludes the test issues. With that fix, the accuracy may drop, but the approach would still be credible. The missing baselines are a matter of effort, not method.

So this is a paper for people who want to know whether neural text classification can be dropped into a security-requirements workflow. It's not a breakthrough, but it's a legitimate empirical artifact with real weaknesses that a referee can ask to have cleaned up. I'd send it to peer review rather than desk-reject it, and I'd expect a revision that fixes the embedding leak, adds baselines, and reports confidence intervals on the industrial set.

Referee Report

4 major / 4 minor

Summary. The paper presents SecureReqNet, a two-stage neural approach for automatically classifying software issue-tracker descriptions as security-related (SR) or non-SR. The first stage trains 100-dimensional skip-gram Word2Vec embeddings on a large corpus of CVE descriptions, GitHub/GitLab issues, and Wikipedia articles. The second stage uses these embeddings to vectorize issue texts and trains four CNN variants (shallow, deep, Alex-style, and an α-SecureReqNet hybrid) to perform binary classification. The authors evaluate the models on 1,032 temporally held-out open-source issues, reporting 96% accuracy for the best configuration, and on 69 industrial user stories from Cisco, reporting 71.6% accuracy. The paper claims a novel contribution in adapting CNNs for security-issue identification and provides an online appendix with code, data, and pre-trained models.

Significance. If validated, SecureReqNet would be a practical automation tool for agile security processes, helping teams identify security-critical issues, triage bug reports, and prioritize security requirements. The paper's strengths are its use of a balanced open-source test set, an explicit temporal-split design for the supervised stage, an industrial evaluation with a commercial partner, and the public release of code and data. However, the central performance claim is currently compromised by the fact that the unsupervised embedding model trains on the same test documents whose representations are later fed to the CNN, and the absence of baseline classifiers prevents the reader from attributing the observed accuracy to the proposed architecture rather than to the data or the label distribution.

major comments (4)

[Section III-A and Table I] The claim that test issues are 'unseen by the model' is not valid for the embedding stage. The Word2Vec model is trained on the entire Embedding Dataset, which includes the test documents (e.g., the GitHub SR row lists 4,575 documents in the Embedding Dataset and 458 in the Testing set). The resulting word vectors are used to vectorize the test issues before classification, so the feature representation for each test issue is learned from the text of that very issue. This transductive setup can inflate the reported 96% open-source accuracy by adapting the embedding space to the test corpus's vocabulary and wording. The authors should retrain the embeddings using only data available before the temporal cutoff of the test issues (or at least excluding test issues from the embedding corpus) and report the resulting accuracy.
[Section III-A] The ground-truth non-SR labels are defined by the absence of a 'security' tag, with manual verification of only a 10% random sample. The paper does not report the outcome of this verification, nor any inter-rater agreement statistics. If a non-negligible fraction of the tag-based non-SR issues actually contain security-related content, then the accuracy numbers, which treat the tag proxy as ground truth, will overstate the classifier's ability to identify true security-relatedness. The authors should provide verification statistics (e.g., precision/recall of the tag proxy) or use a more rigorously validated labeling procedure.
[Section IV] The evaluation reports only the raw performance of the four architectures and contains no comparison against standard baselines, such as majority-class prediction, a keyword/regular-expression classifier, TF-IDF with logistic regression, or the k-NN approach of Riaz et al. [22]. Without such comparisons, the paper cannot support the claim that the two-phase CNN architecture itself is responsible for the observed accuracy. At a minimum, the authors should add a conventional baseline and a paired significance test (e.g., McNemar's test) on the same test set.
[Section III-A and Table I] The data-split description is internally inconsistent: for GitHub (Non-SR), Table I lists an Embedding Dataset total of 47,483, but the Training, Validation, and Testing counts (33,238 + 9,497 + 458) sum to 43,193, not 47,483; similar discrepancies appear for the CVE row. This makes it impossible to verify the claimed 70/20/10 temporal split and undermines the reproducibility of the evaluation. The table and accompanying text should be corrected and clarified, including exactly how the test set was sampled from the full corpus.

minor comments (4)

[Section IV-A] The sentence 'We observed a test validation loss of 0.1313 and accuracy of 0.9736 compared to 0.0272 and 0.9936 for the α-SecureReqNet respectively' is confusing and does not directly match the values reported in Table II; please clarify which dataset (validation or test) and which configuration these numbers refer to.
[Figure 1] The caption states that the Word2Vec output is a softmax and that the network is trained with mean squared error, which is atypical for word2vec training; please specify the actual training objective (e.g., negative sampling or hierarchical softmax).
[Section III-A] There is a typo: 'generalizablility' should be 'generalizability'. Also, the paper uses 'requirements' and 'issues' interchangeably for the industrial user stories; the terminology should be unified.
[Section II-A] For reproducibility, the paper should report the Word2Vec hyperparameters (window size, negative samples, number of training epochs, and subsampling rate) in addition to the embedding dimension and dropout rate already given.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SecureReqNet is a standard supervised classifier trained on external labels and evaluated on held-out issues; the embedding leakage concern is a data-validity issue, not a circular derivation.

full rationale

The paper contains no derivation chain that reduces to its own inputs. SecureReqNet's CNN is trained on externally labeled data (CVE descriptions and 'security'-tagged open-source issues) and evaluated on temporally held-out issues, with accuracy computed against those same external labels. That is standard supervised learning, not circularity. The one arguable concern is that the Word2Vec embeddings are trained on the full Embedding Dataset, which is split into training/validation/test rows in Table I, so the 1,032 test issues' text is used in unsupervised embedding pretraining; the paper's phrase 'unseen by the model' (Section III-A) is therefore imprecise. However, this is a transductive-evaluation/data-leakage threat to the validity of the 96% open-source result, not a circularity reduction: the CNN's parameters are fit only on the Training Dataset, the embedding pretraining is label-free, and the predicted label is not by construction equal to the training label. No load-bearing self-citation is present: the adapted architectures (Han et al., Krizhevsky et al.) are external prior work, and no central premise depends on the present authors' own previous results. The industrial evaluation is unaffected by the embedding concern because industry documents appear only in the test column. Accordingly, no step satisfies the requirement of exhibiting an equation or fitted parameter that makes the prediction equivalent to its input by construction.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on standard machine learning components and the assumption that issue-tracker security labels are reliable. The only hand-set choices are hyperparameters; no new physical or conceptual entities are introduced.

free parameters (6)

word embedding dimension = 100
All input documents are vectorized into 100-dimensional embeddings; chosen by hand (Section III-B).
dropout rate = 0.2
Used during training to control overfitting; chosen without a tuning search (Section III-B).
early stopping patience = 100 epochs
Monitoring validation loss for 2000 epochs with patience 100; hand-set (Section III-B).
n-gram kernel sizes (shallow) = [1,3,5]
Adopted from Han et al. architecture; affects feature extraction (Sec II-B, Table II).
n-gram kernel sizes (alpha) = [7,5,3]
Used in alpha-SecureReqNet; hand-designed combination of AlexNet and Han et al. kernels (Sec II-B).
train/validation/test split = 70/20/10
Data split for training, validation, and testing; chosen for the evaluation (Sec III-A).

assumptions (4)

domain assumption CVE descriptions are inherently security-related and can serve as positive training examples.
Used in the Training Dataset to label 37,036 CVE descriptions as SR (Section III-A).
domain assumption Issues carrying a 'security' label or tag in GitLab and GitHub are security-related, and unlabeled issues that passed a 10% manual check are non-security.
Ground truth for training and evaluation depends on these labels (Section III-A).
domain assumption Word2Vec skip-gram embeddings trained on the combined corpus capture semantic information sufficient for a CNN to classify security-relatedness.
The entire two-phase design assumes these embeddings provide a useful feature space (Section II-A).
domain assumption Temporal separation of test issues from training issues eliminates data leakage that would inflate accuracy.
The authors ensure test issues are created after training issues; the validity of the evaluation relies on this (Section III-A).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning to Identify Security-Related Issues Using Convolutional Neural Networks." pith.science (2026). https://pith.science/paper/DHPUMQC6

@misc{pith2026190800614,
  author       = {Pith},
  title        = {Pith review of: Learning to Identify Security-Related Issues Using Convolutional Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DHPUMQC6}},
  note         = {Machine review of arXiv:1908.00614}
}

read the original abstract

Software security is becoming a high priority for both large companies and start-ups alike due to the increasing potential for harm that vulnerabilities and breaches carry with them. However, attaining robust security assurance while delivering features requires a precarious balancing act in the context of agile development practices. One path forward to help aid development teams in securing their software products is through the design and development of security-focused automation. Ergo, we present a novel approach, called SecureReqNet, for automatically identifying whether issues in software issue tracking systems describe security-related content. Our approach consists of a two-phase neural net architecture that operates purely on the natural language descriptions of issues. The first phase of our approach learns high dimensional word embeddings from hundreds of thousands of vulnerability descriptions listed in the CVE database and issue descriptions extracted from open source projects. The second phase then utilizes the semantic ontology represented by these embeddings to train a convolutional neural network capable of predicting whether a given issue is security-related. We evaluated SecureReqNet by applying it to identify security-related issues from a dataset of thousands of issues mined from popular projects on GitLab and GitHub. In addition, we also applied our approach to identify security-related requirements from a commercial software project developed by a major telecommunication company. Our preliminary results are encouraging, with SecureReqNet achieving an accuracy of 96% on open source issues and 71.6% on industrial requirements.

Figures

Figures reproduced from arXiv: 1908.00614 by the authors.

**Figure 2.** α-SecureReqNet CNN Architecture. an illustration of this architecture. The first convolutional layer has a kernel of 7-gram to reduce the input embedding matrix into 32 vector feature maps. Then, it applies a max pooling and a flattening function to derive a one column matrix. The second convolutional layer has a kernel of 5-gram followed by a max pooling and flattening function, which merges other 64 features into … view at source ↗

**Figure 3.** SecureReqNet (Shallow) Trained to Optimal Capacity. The binary [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages

[22]

M. Riaz, J. King, J. Slankas, and L. Williams. Hidden in plain sight: Automatically identifying security requirements from natural language artifacts. In (RE’14), pages 183–192. IEEE, 2014

work page 2014
[1]

https://blogs.cisco.com/security/the cisco secure development lifecycle an overview

The cisco secure development lifecycle: An overview. https://blogs.cisco.com/security/the cisco secure development lifecycle an overview

work page
[2]

Data of 143 million americans exposed in hack of credit reporting agency equifax https://www.washingtonpost.com/business/technology/ equifax-hack-hits-credit-histories-of-up-to-143-million-americans/ 2017/09/07/a4ae6f82-941a-11e7-b9bc-b2f7903bab0d story.html?utm term=.759b11d92381

work page 2017
[3]

Manifesto for agile software development https://agilemanifesto.org

work page
[4]

https://www.microsoft.com/ en-us/securityengineering/sdl

Microsoft secure development lifecycle. https://www.microsoft.com/ en-us/securityengineering/sdl

work page
[5]

https: //github.com/danaderp/SecureReqNet

Online appendix - learning to identify security related issues. https: //github.com/danaderp/SecureReqNet

work page
[6]

Breaux and A

T. Breaux and A. Ant ´on. Analyzing regulatory rules for privacy and security requirements. (TSE’08), 34(1):5–20, Jan. 2008

work page 2008
[7]

G. Chen, C. Chen, Z. Xing, and B. Xu. Learning a dual-language vector space for domain-speciﬁc cross-lingual question retrieval. In (ASE’16), pages 744–755, Sep. 2016

work page 2016

Show all 24 references

[8]

Fabian, S

B. Fabian, S. G ¨urses, M. Heisel, T. Santen, and H. Schmidt. A comparison of security requirements engineering methods. (RE’10), 15(1):7–40, Mar 2010

2010
[9]

X. Gu, H. Zhang, D. Zhang, and S. Kim. Deep api learning. In (FSE’16), pages 631–642, New York, NY , USA, 2016. ACM

2016
[10]

H. Guo, ¨O. Kafal, and M. P. Singh. Extraction and Formal Repre- sentation of Natural Language Requirements from Breach Reports. In (AIRE’18), 2018

2018
[11]

J. Guo, J. Cheng, and J. Cleland-Huang. Semantically enhanced software traceability using deep learning techniques. In (ICSE’17), pages 3–14, Piscataway, NJ, USA, 2017. IEEE Press

2017
[12]

Z. Han, X. Li, Z. Xing, H. Liu, and Z. Feng. Learning to predict severity of software vulnerability using only vulnerability description. In (ICSME’17), pages 125–136, Sep. 2017

2017
[13]

Howard and S

M. Howard and S. Lipner. The Security Development Lifecycle . Microsoft Press, Redmond, W A, USA, 2006

2006
[14]

Krizhevsky, I

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classiﬁcation with deep convolutional neural networks. (NIPS’13), pages 248–259
[15]

Li and V

F. Li and V . Paxson. A large-scale empirical study of security patches. In (CCS’17), pages 2201–2215, New York, NY , USA, 2017. ACM

2017
[16]

J. C. Maxwell, A. I. Antn, and P. Swire. A legal cross-references taxonomy for identifying conﬂicting software requirements. In (RE’11), pages 197–206, Aug 2011

2011
[17]

N. R. Mead and T. Stehney. Security quality requirements engineering (square) methodology. SIGSOFT Softw. Eng. Notes , 30(4):1–7, May
[18]

Mellado, C

D. Mellado, C. Blanco, L. E. S ´anchez, and E. Fern ´andez-Medina. A systematic review of security requirements engineering. (CSI’10), 32(4):153–165, June 2010

2010
[19]

Menzis and A

T. Menzis and A. Marcus. Automated severity assessment of software defect reports. In (ICSM’08), 2008

2008
[20]

Mikolov, K

T. Mikolov, K. Chen, G. S. Corrado, and J. Dean. Efﬁcient estimation of word representations in vector space, 2013

2013
[21]

R. N. Peclat and G. N. Ramos. Semantic analysis for identifying security concerns in software procurement edicts. (NGC’18), 36(1):21–40
[23]

Siponen, R

M. Siponen, R. Baskerville, and T. Kuivalainen. Integrating security into agile development methods. In Proceedings of the 38th Annual Hawaii International Conference on System Sciences , pages 185a–185a, Jan 2005

2005
[24]

B. Xu, D. Ye, Z. Xing, X. Xia, G. Chen, and S. Li. Predicting semanti- cally linkable knowledge in developer online forums via convolutional neural network. In (ASE’16), pages 51–62, Sep. 2016

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [22]

M. Riaz, J. King, J. Slankas, and L. Williams. Hidden in plain sight: Automatically identifying security requirements from natural language artifacts. In (RE’14), pages 183–192. IEEE, 2014

work page 2014

[2] [1]

https://blogs.cisco.com/security/the cisco secure development lifecycle an overview

The cisco secure development lifecycle: An overview. https://blogs.cisco.com/security/the cisco secure development lifecycle an overview

work page

[3] [2]

Data of 143 million americans exposed in hack of credit reporting agency equifax https://www.washingtonpost.com/business/technology/ equifax-hack-hits-credit-histories-of-up-to-143-million-americans/ 2017/09/07/a4ae6f82-941a-11e7-b9bc-b2f7903bab0d story.html?utm term=.759b11d92381

work page 2017

[4] [3]

Manifesto for agile software development https://agilemanifesto.org

work page

[5] [4]

https://www.microsoft.com/ en-us/securityengineering/sdl

Microsoft secure development lifecycle. https://www.microsoft.com/ en-us/securityengineering/sdl

work page

[6] [5]

https: //github.com/danaderp/SecureReqNet

Online appendix - learning to identify security related issues. https: //github.com/danaderp/SecureReqNet

work page

[7] [6]

Breaux and A

T. Breaux and A. Ant ´on. Analyzing regulatory rules for privacy and security requirements. (TSE’08), 34(1):5–20, Jan. 2008

work page 2008

[8] [7]

G. Chen, C. Chen, Z. Xing, and B. Xu. Learning a dual-language vector space for domain-speciﬁc cross-lingual question retrieval. In (ASE’16), pages 744–755, Sep. 2016

work page 2016