Pith. sign in

REVIEW 4 major objections 6 minor 111 references

VulRTex: A Reasoning-Guided Approach to Identify Vulnerabilities from Rich-Text Issue Report

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VulRTex claims that reasoning over screenshots and code snippets in issue reports outperforms text-only vulnerability identification, with gains of +11.0% F1 and +20.2% AUPRC on 973,572 reports.

desk verdict Solid engineering paper with a real contribution, but the headline F1/AUPRC gains are tuned on the test set, so treat the numbers as upper bounds. read the letter →

arxiv 2509.03875 v1 pith:KJIU2DYO submitted 2025-09-04 cs.SE

Ziyou Jiang , Mingyang Li , Guowei Yang , Lin Shi , Qing Wang This is my paper
classification cs.SE
keywords vulnerabilityidentificationissuereportsrich-textinformationreasoninggraphsretrieval-augmentedgenerationCWE-IDpredictionlargelanguagemodelsopen-sourcesoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many software employees file issue reports that include screenshots and code snippets showing how a vulnerability is triggered, but most automated identification reads only the text. VulRTex tries to establish that this rich-text information can be captured as reasoning graphs—chains of observations and analysis actions—and reused to guide a large language model in judging new reports. The paper argues that reports sharing a CWE-ID describe their triggering logic in similar ways, so retrieving a historical reasoning graph for a new report is a useful substitute for re-deriving the full analysis. On 973,572 issue reports with 4,002 vulnerability-related ones, VulRTex reports the best performance under class imbalance, with +11.0% F1, +20.2% AUPRC, and +10.5% Macro-F1 over the best baseline, at roughly half the time cost of other reasoning baselines. A field application identified 30 emerging vulnerabilities in 2024 reports across ten open-source projects, 11 of which later received CVE-IDs.

What carries the argument

The central object is the reasoning graph G, a directed graph of observations and actions that records, step by step, how an LLM agent explores a historical issue report's screenshots and code snippets to decide whether and how a vulnerability is triggered. Actions are drawn from a small tool set: screenshot OCR, AST-based code summarization, and a termination agent. A random-walk pruning algorithm with TF-IDF similarities selects the subgraph most relevant to a target report, and retrieval of these pruned graphs from the Vulnerability Reasoning Database supplies the guidance prompt that steers the LLM's final Yes/No and CWE-ID prediction. The design's work is to convert the implicit visual

What would settle it

A controlled experiment: replace each retrieved reasoning graph with a random historical graph of the same number of steps and measure F1 on the 1,601 target reports; if the score stays near 88.9%, retrieval is not doing the claimed work, and if it falls toward the no-guidance LLM's 72.8%, the retrieval premise is supported. A second check would use a held-out set of CWE-79 reports whose screenshots show triggering patterns absent from the historical database and test whether the +11.0% F1 gain survives.

Watch

Extended reading notes

Core claim

The paper's central claim is that vulnerability-related issue reports are best identified not from text alone but from the paths through their rich-text elements—page screenshots and code snippets—that together reveal how the vulnerability is triggered. VulRTex first asks an LLM agent to walk through historical reports, generating a reasoning graph whose nodes are observations and whose edges are tool actions such as screenshot analysis and code analysis. It then corrects possible factual errors in those graphs by matching paths against curated vulnerability knowledge. For a new target report, it prunes the historical graphs with a random-walk sampling step, retrieves the most similar ones b

Load-bearing premise

Vulnerability-related issue reports that share a CWE-ID describe their triggering logic similarly enough that reasoning graphs built from old reports' screenshots and code snippets can guide the analysis of new reports.

Editorial extensions

If this is right

  • If the claim holds, vulnerability screening can start from the moment an issue report is filed, using the same evidence human triagers see, rather than waiting for code-level analysis or public disclosure.
  • CWE-ID prediction is part of the same pipeline, so a flagged report arrives with a candidate weakness type for triage and prioritization.
  • Rich-text issue reports, about 39.1% of vulnerability-related reports in the paper's manual study, become usable signals instead of being dropped by text-only methods.
  • The retrieval design cuts reasoning cost roughly in half relative to full chain-of-thought or agent baselines, which matters at the scale of thousands of reports per day.
  • In deployment, the method identified vulnerabilities in 2024 issue reports before CVE assignment, giving maintainers a disclosure head start.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same-CWE commonality premise is right, the reasoning-database approach could be retargeted beyond GitHub issues to other vulnerability report channels where rich-text screenshots and snippets appear.
  • The TF-IDF retrieval layer is a deliberate simplicity trade; replacing it with learned semantic retrieval could either extend the method to reports that paraphrase the same triggering logic in different words, or confirm that vulnerability descriptions stay close to fixed keyword vocabularies.
  • A testable corollary is that the value of retrieval should drop when the database is filled with reasoning graphs from different CWE families; measuring that drop would isolate how much of the gain comes from shared triggering logic rather than from generic guidance.
  • The paper's correction module relies on external curated vulnerability databases, so the method's field ceiling is partly set by how quickly those databases are updated; a novel weakness reported before it appears in any dataset is the hardest case.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VulRTex, an LLM-based approach to identify vulnerability-related GitHub issue reports and predict their CWE-IDs by exploiting rich-text content (screenshots, code snippets). It builds a Vulnerability Reasoning Database from historical IRs by having an LLM generate reasoning graphs with factual-error correction from external VA datasets. At inference, VulRTex prunes and retrieves relevant historical reasoning graphs via TF-IDF-based random walking, generates a guidance prompt from them, and uses the LLM's output probability with a threshold to classify the target IR. The evaluation uses 973,572 IRs (4,002 positive) with a temporal 60/40 split, comparing against DL, LLM, and reasoning baselines. The authors report the best results with ChatGPT+VulRTex (F1 88.9%, AUPRC 80.4%, Macro-F1 83.7%), roughly 2x lower time cost than the reasoning baselines, and an application study identifying 30 emerging vulnerabilities, 11 of which received CVE-IDs.

Significance. If the reported gains are unbiased, VulRTex is a meaningful advance: it is among the first to bring LLM reasoning graphs and RAG to vulnerability identification from rich-text issue reports, and it targets a practically important problem. The paper has several strong points: a large-scale dataset, a temporal split, multiple baseline families, a component-wise ablation, an external factual-error correction mechanism, released artifacts, and a real-world application study with CVE validation. However, the central quantitative claim is weakened by the evaluation protocol: key hyperparameters are selected using the target test set, and no variance or significance information is reported despite 20 repeated runs. The magnitude of the headline improvements is therefore not yet established.

major comments (4)
  1. [§4.4, §6.1, Table 4] The headline comparison is not an unbiased estimate of performance on unseen IRs. Section 4.4 states that θ_out=0.55 is chosen as the value maximizing F1 on the target IRs, and Section 6.1 selects the 60% historical split and θ_sim=0.7 using the same target-set VI/CP metrics. The configuration is therefore fitted to the test labels, which inflates F1 and Macro-F1 and also biases the choice of retrieval setting used to compute AUPRC. Please move all hyperparameter selection to a validation split (or nested CV) and report results on a truly held-out target set, with sensitivity around the selected values.
  2. [§4.4, Table 4] Twenty repeated runs are averaged, but no standard deviation, confidence interval, or significance test is reported anywhere in Tables 4–5 or Figures 6–7. Because the LLM API is stochastic, the reader cannot tell whether margins such as +11.0 F1 or +20.2 AUPRC exceed run-to-run noise. Report mean±std (or CIs) and paired significance tests against the best baseline.
  3. [§3.2, §5.2] The retrieval premise—same-CWE IRs share vulnerability-triggering commonality—is not directly measured. The ablation with 'w/o Pruning' and the graph-pruning replacements show that the component matters, but they do not show that the retrieved graphs are actually relevant; a random-retrieval or perturbed-guidance control would isolate the value of relevant reasoning guidance from mere prompt augmentation. Adding retrieval relevance metrics (e.g., recall@k of same-CWE graphs) would strengthen the transfer argument.
  4. [§5.3, Table 6] The RQ3 application study only follows up on IRs flagged by VulRTex and reports 11/30 CVE assignments. Without a control sample of non-flagged IRs or a comparison to a cheaper baseline, this demonstrates feasibility but not a measurable precision advantage. Add a control set or explicitly restrict the claim to feasibility.
minor comments (6)
  1. [§3.1.1, Algorithm 1] The text contains reviewer-artifact passages: 'Referee-1 Q2 [parse]' in §3.1.1 and 'Referee-1 Q2 [V_r⊂V, E_r⊂E]' in the Algorithm 1 caption. These should be removed before publication.
  2. [Table 4] The ChatGPT+DS-Agent row reports AUROC 70.2, which is far below all other rows (90+). If this is a typo, fix it; if it is a real result, it needs an explanation because it is inconsistent with the surrounding scores.
  3. [§4.3] The sentence 'tuning the threshold θ_out within [0,1] with 0.5 as the interval' cannot be correct; specify the actual step (presumably 0.05).
  4. [§3.1.3 and references] The VA-dataset citation [42,102] is not aligned with Table 1, which lists five datasets but neither citation. Also, reference [20] gives the Rimedo-ts URL for the Carla project; the URL appears to be wrong.
  5. [§6.4] The threat-to-validity text says '60/4,003' but Table 3 reports 4,002 vulnerability-related IRs; reconcile the denominator.
  6. [Eqs. (3)–(4)] The threshold θ_sim is used both for factual-error knowledge retrieval and for relevant-graph retrieval. The paper does not discuss whether the same threshold is appropriate for both retrieval tasks; clarify or separate them.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline F1/AUPRC gains are fitted on the target test set via θ_out and hyper-parameter selection; method pipeline itself is not circular.

  1. fitted input called prediction [Section 4.4, Hyper-parameters (θ_out)]
    "Then, we set the threshold for determining the output of vulnerability, i.e., θ_out, within the range [0.0, 1.0], and choose the optimal F1 value when θ_out = 0.55 (the tuning of the θ_out may affect the trade-off between Precision and Recall, so we illustrate the curve in Section 5)."

    The headline F1 (88.9%, +11.0% over best baseline) is computed on the target test set at θ_out=0.55, a threshold selected to maximize F1 on that same test set. Since F1 is threshold-dependent, choosing the threshold that maximizes F1 on the test labels and then reporting that maximum as VulRTex's performance is fitting the metric to the evaluation data rather than predicting it on unseen IRs. The reported value is forced by the selection rule: F1_reported = max_θ_out F1(test, θ_out), not an unbiased estimate.

  2. fitted input called prediction [Section 4.1 STEP-4 and Section 6.1 (historical split and θ_sim)]
    "we split the vulnerability-related IRs by sorting the IRs in time order, then choose the first 60% of the IRs as the historical IR and the remaining 40% as the target IR (proportion setting is decided by the hyper-parameter tuning, which is shown in Section 6.1). ... To analyze the effect of the historical IR's proportion and threshold θ_sim for TF-IDF similarity, we conduct the analysis of hyper-parameters on VulRTex."

    The 'optimal' hyper-parameters (60% historical split, θ_sim=0.7) are selected using VulRTex's performance on the target IRs, which are the same IRs used for the final Table 4 evaluation. The reported +11.0% F1 and +20.2% AUPRC margins are thus the result of configuration search on the evaluation set, not predictions on unseen IRs. The split proportion and retrieval threshold are inputs fit to the test labels, so the performance estimate is inflated by construction.

full rationale

VulRTex's core pipeline is not circular: the vulnerability reasoning database is built only from the historical 60% of IRs, the target 40% is disjoint, and the RAG guidance is TF-IDF retrieval rather than the target label. The RQ3 prospective study on 2024 IRs provides external evidence. The significant circularity is in the evaluation protocol: θ_out is chosen on the target test set to maximize F1 (Section 4.4), and the historical proportion plus θ_sim are chosen on the same target-set metrics (Section 6.1; Section 4.1 attributes the split to that tuning). Thus the headline F1/AUPRC margins are optimistically biased and partly constructed by fitting to the evaluation data. The SCPatcher self-citation [42] is not load-bearing—it does not appear among the five VA datasets in Table 1—so it does not add to the score. Overall, partial circularity of the reported quantitative claim, not of the method itself.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central method relies on five stated assumptions: same-CWE commonality across reports, TF-IDF retrieval quality, OCR/AST extraction fidelity, VA datasets as gold knowledge, and label correctness. Four fitted hyperparameters (two thresholds, split proportion, temperature) are tuned on the evaluation data, which is the main hidden cost of the approach.

free parameters (4)
  • theta_out (output probability threshold) = 0.55
    Selected by choosing the optimal F1 on the target test set (Section 4.4, 5.1); this post-hoc tuning affects reported Precision/Recall/F1.
  • theta_sim (TF-IDF retrieval threshold) = 0.7
    Tuned in Section 6.1 on evaluation data; used for both golden-knowledge correction and reasoning-graph retrieval.
  • Historical/target split proportion = 60%
    Chosen via hyperparameter analysis in Section 6.1 from 30%-90% range based on performance, a test-set-fitting element.
  • LLM temperature = 0.3
    Set to ensure a stable probability distribution (Section 4.4); chosen by hand rather than derived from first principles.
assumptions (5)
  • domain assumption Vulnerability-related IRs with the same CWE-ID share commonalities in how they describe the vulnerability triggering logic.
    Stated in Section 1 and is the basis for expecting retrieved historical reasoning graphs to transfer to target IRs.
  • domain assumption TF-IDF similarity on OCR/CAST-parsed text is sufficient to retrieve relevant reasoning graphs and golden knowledge.
    Used in Equations 3 and 4 and throughout Section 3.2; a core modeling choice with no systematic sensitivity analysis beyond theta_sim.
  • domain assumption Tencent Cloud OCR and CAST accurately convert screenshots and code snippets into text that preserves vulnerability-relevant semantics.
    Invoked in Section 3.1.2 tool definitions; no validation of OCR/CAST error rates is provided.
  • domain assumption The five VA datasets (KB, BigVul, OWASP, Debian, VDISC) are correct golden knowledge for correcting LLM factual errors.
    Section 3.1.3 treats these datasets as ground truth; only a 100-IR manual check (21/21 corrected) is offered.
  • domain assumption CVE/NVD labels correctly identify vulnerability-related IRs and their CWE-IDs.
    Dataset construction (Section 4.1) relies on CVE-disclosed IRs as ground truth and manually refines 118 multi-CWE labels with high but not perfect agreement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VulRTex: A Reasoning-Guided Approach to Identify Vulnerabilities from Rich-Text Issue Report." pith.science (2026). https://pith.science/paper/KJIU2DYO

@misc{pith2026250903875,
  author       = {Pith},
  title        = {Pith review of: VulRTex: A Reasoning-Guided Approach to Identify Vulnerabilities from Rich-Text Issue Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJIU2DYO}},
  note         = {Machine review of arXiv:2509.03875}
}
read the original abstract

Software vulnerabilities exist in open-source software (OSS), and the developers who discover these vulnerabilities may submit issue reports (IRs) to describe their details. Security practitioners need to spend a lot of time manually identifying vulnerability-related IRs from the community, and the time gap may be exploited by attackers to harm the system. Previously, researchers have proposed automatic approaches to facilitate identifying these vulnerability-related IRs, but these works focus on textual descriptions but lack the comprehensive analysis of IR's rich-text information. In this paper, we propose VulRTex, a reasoning-guided approach to identify vulnerability-related IRs with their rich-text information. In particular, VulRTex first utilizes the reasoning ability of the Large Language Model (LLM) to prepare the Vulnerability Reasoning Database with historical IRs. Then, it retrieves the relevant cases from the prepared reasoning database to generate reasoning guidance, which guides LLM to identify vulnerabilities by reasoning analysis on target IRs' rich-text information. To evaluate the performance of VulRTex, we conduct experiments on 973,572 IRs, and the results show that VulRTex achieves the highest performance in identifying the vulnerability-related IRs and predicting CWE-IDs when the dataset is imbalanced, outperforming the best baseline with +11.0% F1, +20.2% AUPRC, and +10.5% Macro-F1, and 2x lower time cost than baseline reasoning approaches. Furthermore, VulRTex has been applied to identify 30 emerging vulnerabilities across 10 representative OSS projects in 2024's GitHub IRs, and 11 of them are successfully assigned CVE-IDs, which illustrates VulRTex's practicality.

Figures

Figures reproduced from arXiv: 2509.03875 by the authors.

Figure 1
Figure 1. The vulnerability-related IR with rich-text information, which has been assigned CVE-2019-15228. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The overview of our approach VulRTex. on this reasoning graph, we can determine whether this IR contains vulnerabilities and its relevant CWE-ID. Therefore, we believe that rich-text information can help identify the vulnerability-related IR. 3 APPROACH In this section, we introduce the details of VulRTex, and the overall framework is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The steps of identifying target IR’s (i.e., Fig. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Precision-Recall (PR) curves of VulRTex and baselines. 0.0 0.2 0.4 0.6 0.8 1.0 (a) PR Curve in rich-text IRs (b) PR Curve in plain-text IRs Recall 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Precision Precision and 80.4% (AUPRC) in vulnerability identification, and…
Figure 5
Figure 5. Figure 5: The case study of VulRTex on the motivation example (example in [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: The results of ablation study. and ChatGPT+DS-Agent, which can reflect the performance of these models under different output threshold 𝜃𝑜𝑢𝑡. In [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Effects of two hyper-parameters in VulRTex [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

111 extracted references · 65 canonical work pages

  1. [1]

    ISO/IEC 29147:2018: Security techniques - Vulnerability disclosure

    2018. ISO/IEC 29147:2018: Security techniques - Vulnerability disclosure. https://www.iso.org/standard/72311.html

    2018

  2. [2]

    XSS and CSRF in Blocks

    2019. XSS and CSRF in Blocks. https://github.com/daylightstudio/fuel-cms/issues/536

    2019

  3. [3]

    XSS in cmd.php for 1.2.5

    2020. XSS in cmd.php for 1.2.5. https://github.com/leenooks/phpldapadmin/issues/130

    2020

  4. [4]

    Bugzilla

    2023. Bugzilla. https://www.bugzilla.org/

    2023

  5. [5]

    Common vulnerabilities and exposures

    2023. Common vulnerabilities and exposures. https://cve.mitre.org/

    2023

  6. [6]

    Common weakness enumeration

    2023. Common weakness enumeration. https://cwe.mitre.org/

    2023

  7. [7]

    Fix C9K-230327

    2023. Fix C9K-230327. https://github.com/coreruleset/coreruleset/issues/3191

    2023

  8. [8]

    GHArchive

    2023. GHArchive. https://www.gharchive.org/. , Vol. 1, No. 1, Article . Publication date: September 2025. 22 • Ziyou Jiang, Mingyang Li, Guowei Yang, Lin Shi, and Qing Wang∗

    2023

Show all 111 references
  1. [9]

    VulRTex. 2024. Anonymized Repository. https://anonymous.4open.science/r/VulRTex-0F94

    2024

  2. [10]

    Facebook/React

    2024. Facebook/React. https://github.com/facebook/react

    2024

  3. [11]

    Gitstar Ranking

    2024. Gitstar Ranking. https://www.akto.io/cves/vendor/debian

    2024

  4. [12]

    Gitstar Ranking

    2024. Gitstar Ranking. https://gitstar-ranking.com/repositories

    2024

  5. [13]

    Gohugoio/Hugo

    2024. Gohugoio/Hugo. https://github.com/gohugoio/hugo

    2024

  6. [14]

    Honojs/Node-server

    2024. Honojs/Node-server. https://github.com/honojs/node-server

    2024

  7. [15]

    Hyprwm/Hyprland

    2024. Hyprwm/Hyprland. https://github.com/hyprwm/Hyprland

    2024

  8. [16]

    Jerryscript-project/Jerryscript

    2024. Jerryscript-project/Jerryscript. https://github.com/jerryscript-project/jerryscript

    2024

  9. [17]

    Kubernetes/Kubernetes

    2024. Kubernetes/Kubernetes. https://github.com/kubernetes/kubernetes

    2024

  10. [18]

    Mpdavis/Python-jose

    2024. Mpdavis/Python-jose. https://github.com/mpdavis/python-jose

    2024

  11. [19]

    Onosproject/Rimedo-ts

    2024. Onosproject/Rimedo-ts. https://github.com/onosproject/rimedo-ts

    2024

  12. [20]

    Onosproject/Rimedo-ts

    2024. Onosproject/Rimedo-ts. https://github.com/carla-simulator/carla

    2024

  13. [21]

    Xuxueli/Xxl-job

    2024. Xuxueli/Xxl-job. https://github.com/xuxueli/xxl-job

    2024

  14. [22]

    Zeeshan Afzal, Johan Garcia, Stefan Lindskog, and Anna Brunström. 2018. Slice Distance: An Insert-Only Levenshtein Distance with a Focus on Security Applications. In 9th IFIP International Conference on New Technologies, Mobility and Security, NTMS 2018 . IEEE, 1–5

    2018

  15. [23]

    Atlassian. 2023. Jira, Issue & Project Tracking Software. https://www.atlassian.com/software/jira

    2023

  16. [24]

    Leyla Bilge and Tudor Dumitras. 2012. Before we knew it: an empirical study of zero-day attacks in the real world. In the ACM Conference on Computer and Communications Security, CCS’12 . ACM, 833–844

    2012

  17. [25]

    Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2022. Deep Learning Based Vulnerability Detection: Are We There Yet? IEEE Trans. Software Eng. 48, 9 (2022), 3280–3296

    2022

  18. [26]

    Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David A. Wagner. 2023. DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defens...

    2023

  19. [27]

    Neeraj Cherakara, Finny Varghese, Sheena Shabana, Nivan Nelson, Abhiram Karukayil, Rohith Kulothungan, Mohammed Afil Farhan, Birthe Nesset, Meriam Moujahid, Tanvi Dinkar, Verena Rieser, and Oliver Lemon. 2023. FurChat: An Embodied Conversational Agent using LLMs, Combining Ope...

    2023

  20. [28]

    T. M. Corporation. 2011. Common Attack Pattern Enumeration and Classification (CAPEC). http://capec.mitre.org/

    2011

  21. [29]

    Jesse Davis and Mark Goadrich. 2006. The relationship between Precision-Recall and ROC curves. In Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006) (ACM International Conference Proceeding Series, Vol. 148) . ACM, 233–240

    2006

  22. [30]

    Engler, David Yu Chen, and Andy Chou

    Dawson R. Engler, David Yu Chen, and Andy Chou. 2001. Bugs as Deviant Behavior: A General Approach to Inferring Errors in Systems Code. In Proceedings of the 18th ACM Symposium on Operating System Principles, SOSP 2001 . ACM, 57–72

    2001

  23. [31]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In MSR ’20. ACM, 508–512

    2020

  24. [32]

    Michael Gegick, Pete Rotella, and Tao Xie. 2010. Identifying security bug reports via text mining: An industrial case study. InProceedings of the 7th International Working Conference on Mining Software Repositories, MSR 2010 (Co-located with ICSE) . IEEE Computer Society, 11–20

    2010

  25. [33]

    Seyed Mohammad Ghaffarian and Hamid Reza Shahriari. 2017. Software vulnerability analysis and discovery using machine-learning and data-mining techniques: A survey. Comput. Surveys 50, 4 (2017), 1–36

    2017

  26. [34]

    Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. 2024. DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning. CoRR abs/2402.17453 (2024). arXiv:2402.17453

    2024 arXiv

  27. [35]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. CoRR abs/2002.08909 (2020). arXiv:2002.08909

    2020 arXiv

  28. [36]

    Nima Shiri Harzevili, Alvine Boaye Belle, Junjie Wang, Song Wang, Zhen Ming (Jack) Jiang, and Nachiappan Nagappan. 2025. A Systematic Literature Review on Automated Software Vulnerability Detection Using Machine Learning. ACM Comput. Surv. 57, 3 (2025), 55:1–55:36

    2025

  29. [37]

    Allen D Householder, Garret Wassermann, Art Manion, and Chris King. 2017. The cert guide to coordinated vulnerability disclosure. Software Engineering Institute, Pittsburgh, PA (2017)

    2017

  30. [38]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. CoRR abs/2311.052...

    2023 arXiv

  31. [39]

    Hugging Face. 2023. meta-llama/Llama-2-13b-chat-hfd. https://huggingface.co/meta-llama/Llama-2-13b-chat-hf

    2023

  32. [40]

    Jiyong Jang, Abeer Agrawal, and David Brumley. 2012. ReDeBug: Finding Unpatched Code Clones in Entire OS Distributions. In IEEE Symposium on Security and Privacy, SP 2012, 21-23 May 2012, San Francisco, California, USA . IEEE Computer Society, 48–62. , Vol. 1, No. 1, Article ....

    2012

  33. [41]

    Tiantian Ji, Yue Wu, Chang Wang, Xi Zhang, and Zhongru Wang. 2018. The coming era of alphahacking?: A survey of automatic software vulnerability detection, exploitation and patching techniques. In 2018 IEEE third international conference on data science in cyberspace (DSC). IE...

    2018

  34. [42]

    Ziyou Jiang, Lin Shi, Guowei Yang, and Qing Wang. 2023. SCPatcher: Mining Crowd Security Discussions to Enrich Secure Coding Practices. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023 . IEEE, 358–370

    2023

  35. [43]

    Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig

    Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig

  36. [44]

    Gong Jie, Kuang Xiao-Hui, and Liu Qiang. 2016. Survey on software vulnerability analysis method based on machine learning. In 2016 IEEE first international conference on data science in cyberspace (DSC) . IEEE, 642–647

    2016

  37. [45]

    Dongkyu Kim, Byoungwook Kim, Donggeon Han, and Matous Eibich. 2024. AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipeline. CoRR abs/2410.20878 (2024). arXiv:2410.20878

    2024 arXiv

  38. [46]

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In NeurIPS’20

    2020

  39. [47]

    Zhen Li, Deqing Zou, Jing Tang, Zhihao Zhang, Mingqian Sun, and Hai Jin. 2019. A comparative study of deep learning-based vulnerability detection system. IEEE Access 7 (2019), 103184–103197

    2019

  40. [48]

    Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018. VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018 . The Internet Society

    2018

  41. [49]

    Hongliang Liang, Lei Wang, Dongyang Wu, and Jiuyun Xu. 2016. MLSA: A static bugs analysis tool based on LLVM IR. In 17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD

    2016

  42. [50]

    Guanjun Lin, Sheng Wen, Qing-Long Han, Jun Zhang, and Yang Xiang. 2020. Software Vulnerability Detection Using Deep Neural Networks: A Survey. Proc. IEEE 108, 10 (2020), 1825–1848

    2020

  43. [51]

    Guanjun Lin, Jun Zhang, Wei Luo, Lei Pan, and Yang Xiang. 2017. POSTER: Vulnerability Discovery with Function Representation Learning from Unlabeled Projects. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS ’17...

    2017

  44. [52]

    Tao Liu and Longtao Zhang. 2018. Application of logistic regression in web vulnerability scanning. In 2018 International Conference on Sensor Networks and Signal Processing (SNSP) . IEEE, 486–490

    2018

  45. [53]

    Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, and Dong Yu. 2023. LASER: LLM Agent with State-Space Exploration for Web Navigation. CoRR abs/2309.08172 (2023)

    2023 arXiv

  46. [54]

    Ruchika Malhotra. 2015. A systematic review of machine learning techniques for software fault prediction. Applied Soft Computing 27 (2015), 504–518

    2015

  47. [55]

    Mathieu and Alan E

    Richard G. Mathieu and Alan E. Turovlin. 2023. Lost in the middle - a pragmatic approach for ERP managers to prioritize known vulnerabilities by applying classification and regression trees (CART). Inf. Comput. Secur. 31, 5 (2023), 655–674

    2023

  48. [56]

    Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In1st International Conference on Learning Representations, ICLR 2013

    2013

  49. [57]

    MITRE. 2014. Adversarial Tactics, Techniques & Common Knowledge (ATT&CK). https://attack.mitre.org

    2014

  50. [58]

    MITRE. 2023. CWE-352: Cross-Site Request Forgery (CSRF) . https://cwe.mitre.org/data/definitions/352.html

    2023

  51. [59]

    MITRE. 2023. CWE-79: Improper Neutralization of Input During Web Page Generation (’Cross-site Scripting’) (4.13). https://cwe.mitre. org/data/definitions/79.html

    2023

  52. [60]

    MITRE. 2023. CWE-94: Improper Control of Generation of Code (’Code Injection’). https://cwe.mitre.org/data/definitions/79.html

    2023

  53. [61]

    Linyong Nan, Ellen Zhang, Weijin Zou, Yilun Zhao, Wenfei Zhou, and Arman Cohan. 2023. On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering. CoRR abs/2311.09721 (2023)

    2023 arXiv

  54. [62]

    nilsteampassnet. 2019. Stored XSS in log of Failed Logins. https://github.com/nilsteampassnet/TeamPass/issues/2688

    2019

  55. [63]

    Marwan Omar and Stavros Shiaeles. 2023. VulDetect: A novel technique for detecting software vulnerabilities using Language Models. In IEEE International Conference on Cyber Security and Resilience, CSR 2023. IEEE, 105–110. https://doi.org/10.1109/CSR57506.2023.10224924

    2023

  56. [64]

    OpenAI. 2023. Chatgpt: A language model for conversational AI. https://www.openai.com/research/chatgpt/

    2023

  57. [65]

    OpenAI. 2023. GPT-3-Models. https://platform.openai.com/docs/models/gpt-3

    2023

  58. [66]

    OpenAI. 2023. GPT-3.5-Models. https://platform.openai.com/docs/models/gpt-3-5

    2023

  59. [67]

    OWASP. 2023. Open web application security project. https://www.owasp.org/index.php/MainPage

    2023

  60. [68]

    Tosin Daniel Oyetoyan and Patrick Morrison. 2021. An improved text classification modelling approach to identify security messages in heterogeneous projects. Softw. Qual. J. 29, 2 (2021), 509–553

    2021

  61. [69]

    Page, Lawrence, Brin, Sergey, and Terry. 1999. The PageRank citation ranking: Bringing order to the web. stanford digital libraries working paper (1999). , Vol. 1, No. 1, Article . Publication date: September 2025. 24 • Ziyou Jiang, Mingyang Li, Guowei Yang, Lin Shi, and Qing Wang∗

    1999

  62. [70]

    Liuxuan Pan and Allan Tomlinson. 2016. A Systematic Review of Information Security Risk Assessment. International Journal of Safety and Security Engineering 6 (06 2016), 270–281

    2016

  63. [71]

    Shengyi Pan, Jiayuan Zhou, Filipe Roseiro Côgo, Xin Xia, Lingfeng Bao, Xing Hu, Shanping Li, and Ahmed E. Hassan. 2022. Automated unearthing of dangerous issue reports. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundatio...

    2022

  64. [72]

    Pérez, Jessica Díaz, Javier García Martin, and Bernardo Tabuenca

    Jorge E. Pérez, Jessica Díaz, Javier García Martin, and Bernardo Tabuenca. 2020. Systematic Literature Reviews in Software Engineering - Enhancement of the Study Selection Process Using Cohen’s Kappa Statistic. J. Syst. Softw. 168 (2020), 110657. https://doi.org/10.1016/ j.jss...

    2020

  65. [73]

    Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: online learning of social representations. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14 . ACM, 701–710

    2014

  66. [74]

    Fayola Peters, Thein Than Tun, Yijun Yu, and Bashar Nuseibeh. 2019. Text Filtering and Ranking for Security Bug Report Prediction. IEEE Trans. Software Eng. 45, 6 (2019), 615–631

    2019

  67. [75]

    Serena Elisa Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, and Cédric Dangremont. 2019. A manually-curated dataset of fixes to vulnerabilities of open-source software. In Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019 . IEEE ...

    2019

  68. [76]

    Hongjin Qian, Peitian Zhang, Zheng Liu, Kelong Mao, and Zhicheng Dou. 2024. MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery. CoRR abs/2409.05591 (2024). arXiv:2409.05591

    2024 arXiv

  69. [77]

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, ...

    2023 arXiv

  70. [78]

    Bonan Ruan, Jiahao Liu, Weibo Zhao, and Zhenkai Liang. 2024. VulZoo: A Comprehensive Vulnerability Intelligence Dataset. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24) . Association for Computing Machinery, 2334–2337

    2024

  71. [79]

    Russell, Louis Y

    Rebecca L. Russell, Louis Y. Kim, Lei H. Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul M. Ellingwood, and Marc W. McConley. 2018. Automated Vulnerability Detection in Source Code Using Deep Representation Learning. In 17th IEEE International Conference on Machine Le...

    2018

  72. [80]

    Gerard Salton and Christopher Buckley. 1988. Term-weighting Approaches in Automatic Text Retrieval. Information Processing & Management 24, 5 (1988), 513–523

    1988

  73. [81]

    Riccardo Scandariato, James Walden, Aram Hovsepyan, and Wouter Joosen. 2014. Predicting Vulnerable Software Components via Text Mining. IEEE Trans. Software Eng. 40, 10 (2014), 993–1006

    2014

  74. [82]

    Raphael Schumann, Wanrong Zhu, Weixi Feng, Tsu-Jui Fu, Stefan Riezler, and William Yang Wang. 2023. VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View. CoRR abs/2307.06082 (2023)

    2023 arXiv

  75. [83]

    Ensheng Shi, Yanlin Wang, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. 2021. CAST: Enhancing Code Summarization with Hierarchical Splitting and Reconstruction of Abstract Syntax Trees. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lang...

    2021

  76. [84]

    Lin Shi, Ziyou Jiang, Ye Yang, Xiao Chen, Yumin Zhang, Fangwen Mu, Hanzhi Jiang, and Qing Wang. 2021. ISPY: Automatic Issue- Solution Pair Extraction from Community Live Chats. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021. IEEE, 142–154

    2021

  77. [85]

    Williams, and Jason A

    Yonghee Shin, Andrew Meneely, Laurie A. Williams, and Jason A. Osborne. 2011. Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities. IEEE Trans. Software Eng. 37, 6 (2011), 772–787

    2011

  78. [86]

    Williams, and Tim Menzies

    Rui Shu, Tianpei Xia, Jianfeng Chen, Laurie A. Williams, and Tim Menzies. 2021. How to Better Distinguish Security Bug Reports (Using Dual Hyperparameter Optimization). Empir. Softw. Eng. 26, 3 (2021), 53

    2021

  79. [87]

    Andrea Stocco, Michael Weiss, Marco Calzana, and Paolo Tonella. 2020. Misbehaviour prediction for autonomous driving systems. In ICSE ’20: 42nd International Conference on Software Engineering . ACM, 359–371

    2020

  80. [88]

    Tencent. 2023. OCR, Tencent Cloud. https://www.tencentcloud.com/document/product/1045/49147

    2023

  81. [89]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation La...

    2023 arXiv

  82. [90]

    Jin Wang, Zishan Huang, Hengli Liu, Nianyi Yang, and Yinhao Xiao. 2023. DefectHunter: A Novel LLM-Driven Boosted-Conformer-based Code Vulnerability Detection Mechanism. CoRR abs/2309.15324 (2023)

    2023 arXiv

  83. [91]

    Kuan Wang, Yadong Lu, Michael Santacroce, Yeyun Gong, Chao Zhang, and Yelong Shen. 2023. Adapting LLM Agents Through Communication. CoRR abs/2310.01444 (2023). , Vol. 1, No. 1, Article . Publication date: September 2025. VulRTex: A Reasoning-Guided Approach to Identify Vulnera...

    2023 arXiv

  84. [92]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. 2022. Rationale-Augmented Ensembles in Language Models. CoRR abs/2207.00747 (2022)

    2022 arXiv

  85. [93]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS

    2022

  86. [94]

    Dumidu Wijayasekara, Milos Manic, and Miles McQueen. 2014. Vulnerability identification and classification via text mining bug databases. In IECON 2014 - 40th Annual Conference of the IEEE Industrial Electronics Society . IEEE, 3612–3618

    2014

  87. [95]

    Wright, and Miles McQueen

    Dumidu Wijayasekara, Milos Manic, Jason L. Wright, and Miles McQueen. 2012. Mining Bug Databases for Unidentified Software Vulnerabilities. In 2012 5th International Conference on Human System Interactions . IEEE, 89–96

    2012

  88. [96]

    Fang Wu, Jigang Wang, Jiqiang Liu, and Wei Wang. 2017. Vulnerability detection with deep learning. In2017 3rd IEEE international conference on computer and communications (ICCC) . IEEE, 1298–1302

    2017

  89. [97]

    Susheng Wu, Wenyan Song, Kaifeng Huang, Bihuan Chen, and Xin Peng. 2024. Identifying Affected Libraries and Their Ecosystems for Open Source Software Vulnerabilities. In ICSE’24. ACM, 162:1–162:12

    2024

  90. [98]

    Yueming Wu, Deqing Zou, Shihan Dou, Siru Yang, Wei Yang, Feng Cheng, Hong Liang, and Hai Jin. 2020. SCDetector: Software Functional Clone Detection Based on Semantic Tokens Analysis. In 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020. IEEE, 821–833

    2020

  91. [99]

    Fabian Yamaguchi, Christian Wressnegger, Hugo Gascon, and Konrad Rieck. 2013. Chucky: exposing missing checks in source code for vulnerability discovery. In 2013 ACM SIGSAC Conference on Computer and Communications Security, CCS’13 . ACM, 499–510

    2013

  92. [100]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023 . OpenReview.net

    2023

  93. [101]

    Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woo-Myoung Park. 2021. GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation. InFindings of the Association for Computational Linguistics: EMNLP 2021 . Association for Computational Linguistics, 2225–2239

    2021

  94. [102]

    Hao Yu, Xing Hu, Ge Li, Ying Li, Qianxiang Wang, and Tao Xie. 2022. Assessing and Improving an Evaluation Dataset for Detecting Semantic Code Clones via Deep Learning. ACM Trans. Softw. Eng. Methodol. 31, 4 (2022), 62:1–62:25

    2022

  95. [103]

    Prasanna

    Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. 2020. GraphSAINT: Graph Sampling Based Inductive Learning Method. In 8th International Conference on Learning Representations, ICLR 2020 . OpenReview.net

    2020

  96. [104]

    Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019. A novel neural source code representation based on abstract syntax tree. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019 . IEEE / ACM, 783–794

    2019

  97. [105]

    Sipeng Zheng, Jiazheng Liu, Yicheng Feng, and Zongqing Lu. 2023. Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds. CoRR abs/2310.13255 (2023)

    2023 arXiv

  98. [106]

    Lewis, Luca Buratti, Edward A

    Yunhui Zheng, Saurabh Pujar, Burn L. Lewis, Luca Buratti, Edward A. Epstein, Bo Yang, Jim Laredo, Alessandro Morari, and Zhong Su

  99. [107]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large Language Models are Human-Level Prompt Engineers. In The Eleventh International Conference on Learning Representations, ICLR 2023 . OpenReview.net

    2023

  100. [108]

    Yuhui Zhu, Guanjun Lin, Lipeng Song, and Jun Zhang. 2023. The application of neural network for software vulnerability detection: a review. Neural Comput. Appl. 35, 2 (2023), 1279–1301. , Vol. 1, No. 1, Article . Publication date: September 2025

    2023

  101. [2016]

    IEEE Computer Society, 407–412

  102. [2021]

    In ICSE (SEIP)’21

    D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis. In ICSE (SEIP)’21. IEEE, 111–120

  103. [2023]

    In EMNLP’23

    Active Retrieval Augmented Generation. In EMNLP’23. Association for Computational Linguistics, 7969–7992

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.