Pith. sign in

REVIEW 4 major objections 6 minor 111 references

VulRTex claims that reasoning over screenshots and code snippets in issue reports outperforms text-only vulnerability identification, with gains of +11.0% F1 and +20.2% AUPRC on 973,572 reports.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A retrieval-augmented LLM approach that identifies vulnerability-related issue reports and CWE types from screenshots and code snippets, improving F1 by 11 points and AUPRC by 20 points over baselines.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid engineering paper with a real contribution, but the headline F1/AUPRC gains are tuned on the test set, so treat the numbers as upper bounds. the 4 major comments →

arxiv 2509.03875 v1 pith:KJIU2DYO submitted 2025-09-04 cs.SE

VulRTex: A Reasoning-Guided Approach to Identify Vulnerabilities from Rich-Text Issue Report

Ziyou Jiang , Mingyang Li , Guowei Yang , Lin Shi , Qing Wang This is my paper
classification cs.SE
keywords vulnerability identificationissue reportsrich-text informationreasoning graphsretrieval-augmented generationCWE-ID predictionlarge language modelsopen-source software
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many software employees file issue reports that include screenshots and code snippets showing how a vulnerability is triggered, but most automated identification reads only the text. VulRTex tries to establish that this rich-text information can be captured as reasoning graphs—chains of observations and analysis actions—and reused to guide a large language model in judging new reports. The paper argues that reports sharing a CWE-ID describe their triggering logic in similar ways, so retrieving a historical reasoning graph for a new report is a useful substitute for re-deriving the full analysis. On 973,572 issue reports with 4,002 vulnerability-related ones, VulRTex reports the best performance under class imbalance, with +11.0% F1, +20.2% AUPRC, and +10.5% Macro-F1 over the best baseline, at roughly half the time cost of other reasoning baselines. A field application identified 30 emerging vulnerabilities in 2024 reports across ten open-source projects, 11 of which later received CVE-IDs.

Core claim

The paper's central claim is that vulnerability-related issue reports are best identified not from text alone but from the paths through their rich-text elements—page screenshots and code snippets—that together reveal how the vulnerability is triggered. VulRTex first asks an LLM agent to walk through historical reports, generating a reasoning graph whose nodes are observations and whose edges are tool actions such as screenshot analysis and code analysis. It then corrects possible factual errors in those graphs by matching paths against curated vulnerability knowledge. For a new target report, it prunes the historical graphs with a random-walk sampling step, retrieves the most similar ones b

What carries the argument

The central object is the reasoning graph G, a directed graph of observations and actions that records, step by step, how an LLM agent explores a historical issue report's screenshots and code snippets to decide whether and how a vulnerability is triggered. Actions are drawn from a small tool set: screenshot OCR, AST-based code summarization, and a termination agent. A random-walk pruning algorithm with TF-IDF similarities selects the subgraph most relevant to a target report, and retrieval of these pruned graphs from the Vulnerability Reasoning Database supplies the guidance prompt that steers the LLM's final Yes/No and CWE-ID prediction. The design's work is to convert the implicit visual

Load-bearing premise

Vulnerability-related issue reports that share a CWE-ID describe their triggering logic similarly enough that reasoning graphs built from old reports' screenshots and code snippets can guide the analysis of new reports.

What would settle it

A controlled experiment: replace each retrieved reasoning graph with a random historical graph of the same number of steps and measure F1 on the 1,601 target reports; if the score stays near 88.9%, retrieval is not doing the claimed work, and if it falls toward the no-guidance LLM's 72.8%, the retrieval premise is supported. A second check would use a held-out set of CWE-79 reports whose screenshots show triggering patterns absent from the historical database and test whether the +11.0% F1 gain survives.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, vulnerability screening can start from the moment an issue report is filed, using the same evidence human triagers see, rather than waiting for code-level analysis or public disclosure.
  • CWE-ID prediction is part of the same pipeline, so a flagged report arrives with a candidate weakness type for triage and prioritization.
  • Rich-text issue reports, about 39.1% of vulnerability-related reports in the paper's manual study, become usable signals instead of being dropped by text-only methods.
  • The retrieval design cuts reasoning cost roughly in half relative to full chain-of-thought or agent baselines, which matters at the scale of thousands of reports per day.
  • In deployment, the method identified vulnerabilities in 2024 issue reports before CVE assignment, giving maintainers a disclosure head start.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same-CWE commonality premise is right, the reasoning-database approach could be retargeted beyond GitHub issues to other vulnerability report channels where rich-text screenshots and snippets appear.
  • The TF-IDF retrieval layer is a deliberate simplicity trade; replacing it with learned semantic retrieval could either extend the method to reports that paraphrase the same triggering logic in different words, or confirm that vulnerability descriptions stay close to fixed keyword vocabularies.
  • A testable corollary is that the value of retrieval should drop when the database is filled with reasoning graphs from different CWE families; measuring that drop would isolate how much of the gain comes from shared triggering logic rather than from generic guidance.
  • The paper's correction module relies on external curated vulnerability databases, so the method's field ceiling is partly set by how quickly those databases are updated; a novel weakness reported before it appears in any dataset is the hardest case.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VulRTex, an LLM-based approach to identify vulnerability-related GitHub issue reports and predict their CWE-IDs by exploiting rich-text content (screenshots, code snippets). It builds a Vulnerability Reasoning Database from historical IRs by having an LLM generate reasoning graphs with factual-error correction from external VA datasets. At inference, VulRTex prunes and retrieves relevant historical reasoning graphs via TF-IDF-based random walking, generates a guidance prompt from them, and uses the LLM's output probability with a threshold to classify the target IR. The evaluation uses 973,572 IRs (4,002 positive) with a temporal 60/40 split, comparing against DL, LLM, and reasoning baselines. The authors report the best results with ChatGPT+VulRTex (F1 88.9%, AUPRC 80.4%, Macro-F1 83.7%), roughly 2x lower time cost than the reasoning baselines, and an application study identifying 30 emerging vulnerabilities, 11 of which received CVE-IDs.

Significance. If the reported gains are unbiased, VulRTex is a meaningful advance: it is among the first to bring LLM reasoning graphs and RAG to vulnerability identification from rich-text issue reports, and it targets a practically important problem. The paper has several strong points: a large-scale dataset, a temporal split, multiple baseline families, a component-wise ablation, an external factual-error correction mechanism, released artifacts, and a real-world application study with CVE validation. However, the central quantitative claim is weakened by the evaluation protocol: key hyperparameters are selected using the target test set, and no variance or significance information is reported despite 20 repeated runs. The magnitude of the headline improvements is therefore not yet established.

major comments (4)
  1. [§4.4, §6.1, Table 4] The headline comparison is not an unbiased estimate of performance on unseen IRs. Section 4.4 states that θ_out=0.55 is chosen as the value maximizing F1 on the target IRs, and Section 6.1 selects the 60% historical split and θ_sim=0.7 using the same target-set VI/CP metrics. The configuration is therefore fitted to the test labels, which inflates F1 and Macro-F1 and also biases the choice of retrieval setting used to compute AUPRC. Please move all hyperparameter selection to a validation split (or nested CV) and report results on a truly held-out target set, with sensitivity around the selected values.
  2. [§4.4, Table 4] Twenty repeated runs are averaged, but no standard deviation, confidence interval, or significance test is reported anywhere in Tables 4–5 or Figures 6–7. Because the LLM API is stochastic, the reader cannot tell whether margins such as +11.0 F1 or +20.2 AUPRC exceed run-to-run noise. Report mean±std (or CIs) and paired significance tests against the best baseline.
  3. [§3.2, §5.2] The retrieval premise—same-CWE IRs share vulnerability-triggering commonality—is not directly measured. The ablation with 'w/o Pruning' and the graph-pruning replacements show that the component matters, but they do not show that the retrieved graphs are actually relevant; a random-retrieval or perturbed-guidance control would isolate the value of relevant reasoning guidance from mere prompt augmentation. Adding retrieval relevance metrics (e.g., recall@k of same-CWE graphs) would strengthen the transfer argument.
  4. [§5.3, Table 6] The RQ3 application study only follows up on IRs flagged by VulRTex and reports 11/30 CVE assignments. Without a control sample of non-flagged IRs or a comparison to a cheaper baseline, this demonstrates feasibility but not a measurable precision advantage. Add a control set or explicitly restrict the claim to feasibility.
minor comments (6)
  1. [§3.1.1, Algorithm 1] The text contains reviewer-artifact passages: 'Referee-1 Q2 [parse]' in §3.1.1 and 'Referee-1 Q2 [V_r⊂V, E_r⊂E]' in the Algorithm 1 caption. These should be removed before publication.
  2. [Table 4] The ChatGPT+DS-Agent row reports AUROC 70.2, which is far below all other rows (90+). If this is a typo, fix it; if it is a real result, it needs an explanation because it is inconsistent with the surrounding scores.
  3. [§4.3] The sentence 'tuning the threshold θ_out within [0,1] with 0.5 as the interval' cannot be correct; specify the actual step (presumably 0.05).
  4. [§3.1.3 and references] The VA-dataset citation [42,102] is not aligned with Table 1, which lists five datasets but neither citation. Also, reference [20] gives the Rimedo-ts URL for the Carla project; the URL appears to be wrong.
  5. [§6.4] The threat-to-validity text says '60/4,003' but Table 3 reports 4,002 vulnerability-related IRs; reconcile the denominator.
  6. [Eqs. (3)–(4)] The threshold θ_sim is used both for factual-error knowledge retrieval and for relevant-graph retrieval. The paper does not discuss whether the same threshold is appropriate for both retrieval tasks; clarify or separate them.

Circularity Check

2 steps flagged

Headline F1/AUPRC gains are fitted on the target test set via θ_out and hyper-parameter selection; method pipeline itself is not circular.

specific steps
  1. fitted input called prediction [Section 4.4, Hyper-parameters (θ_out)]
    "Then, we set the threshold for determining the output of vulnerability, i.e., θ_out, within the range [0.0, 1.0], and choose the optimal F1 value when θ_out = 0.55 (the tuning of the θ_out may affect the trade-off between Precision and Recall, so we illustrate the curve in Section 5)."

    The headline F1 (88.9%, +11.0% over best baseline) is computed on the target test set at θ_out=0.55, a threshold selected to maximize F1 on that same test set. Since F1 is threshold-dependent, choosing the threshold that maximizes F1 on the test labels and then reporting that maximum as VulRTex's performance is fitting the metric to the evaluation data rather than predicting it on unseen IRs. The reported value is forced by the selection rule: F1_reported = max_θ_out F1(test, θ_out), not an unbiased estimate.

  2. fitted input called prediction [Section 4.1 STEP-4 and Section 6.1 (historical split and θ_sim)]
    "we split the vulnerability-related IRs by sorting the IRs in time order, then choose the first 60% of the IRs as the historical IR and the remaining 40% as the target IR (proportion setting is decided by the hyper-parameter tuning, which is shown in Section 6.1). ... To analyze the effect of the historical IR's proportion and threshold θ_sim for TF-IDF similarity, we conduct the analysis of hyper-parameters on VulRTex."

    The 'optimal' hyper-parameters (60% historical split, θ_sim=0.7) are selected using VulRTex's performance on the target IRs, which are the same IRs used for the final Table 4 evaluation. The reported +11.0% F1 and +20.2% AUPRC margins are thus the result of configuration search on the evaluation set, not predictions on unseen IRs. The split proportion and retrieval threshold are inputs fit to the test labels, so the performance estimate is inflated by construction.

full rationale

VulRTex's core pipeline is not circular: the vulnerability reasoning database is built only from the historical 60% of IRs, the target 40% is disjoint, and the RAG guidance is TF-IDF retrieval rather than the target label. The RQ3 prospective study on 2024 IRs provides external evidence. The significant circularity is in the evaluation protocol: θ_out is chosen on the target test set to maximize F1 (Section 4.4), and the historical proportion plus θ_sim are chosen on the same target-set metrics (Section 6.1; Section 4.1 attributes the split to that tuning). Thus the headline F1/AUPRC margins are optimistically biased and partly constructed by fitting to the evaluation data. The SCPatcher self-citation [42] is not load-bearing—it does not appear among the five VA datasets in Table 1—so it does not add to the score. Overall, partial circularity of the reported quantitative claim, not of the method itself.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central method relies on five stated assumptions: same-CWE commonality across reports, TF-IDF retrieval quality, OCR/AST extraction fidelity, VA datasets as gold knowledge, and label correctness. Four fitted hyperparameters (two thresholds, split proportion, temperature) are tuned on the evaluation data, which is the main hidden cost of the approach.

free parameters (4)
  • theta_out (output probability threshold) = 0.55
    Selected by choosing the optimal F1 on the target test set (Section 4.4, 5.1); this post-hoc tuning affects reported Precision/Recall/F1.
  • theta_sim (TF-IDF retrieval threshold) = 0.7
    Tuned in Section 6.1 on evaluation data; used for both golden-knowledge correction and reasoning-graph retrieval.
  • Historical/target split proportion = 60%
    Chosen via hyperparameter analysis in Section 6.1 from 30%-90% range based on performance, a test-set-fitting element.
  • LLM temperature = 0.3
    Set to ensure a stable probability distribution (Section 4.4); chosen by hand rather than derived from first principles.
axioms (5)
  • domain assumption Vulnerability-related IRs with the same CWE-ID share commonalities in how they describe the vulnerability triggering logic.
    Stated in Section 1 and is the basis for expecting retrieved historical reasoning graphs to transfer to target IRs.
  • domain assumption TF-IDF similarity on OCR/CAST-parsed text is sufficient to retrieve relevant reasoning graphs and golden knowledge.
    Used in Equations 3 and 4 and throughout Section 3.2; a core modeling choice with no systematic sensitivity analysis beyond theta_sim.
  • domain assumption Tencent Cloud OCR and CAST accurately convert screenshots and code snippets into text that preserves vulnerability-relevant semantics.
    Invoked in Section 3.1.2 tool definitions; no validation of OCR/CAST error rates is provided.
  • domain assumption The five VA datasets (KB, BigVul, OWASP, Debian, VDISC) are correct golden knowledge for correcting LLM factual errors.
    Section 3.1.3 treats these datasets as ground truth; only a 100-IR manual check (21/21 corrected) is offered.
  • domain assumption CVE/NVD labels correctly identify vulnerability-related IRs and their CWE-IDs.
    Dataset construction (Section 4.1) relies on CVE-disclosed IRs as ground truth and manually refines 118 multi-CWE labels with high but not perfect agreement.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of VulRTex: A Reasoning-Guided Approach to Identify Vulnerabilities from Rich-Text Issue Report." pith.science (2026). https://pith.science/paper/KJIU2DYO

@misc{pith2026250903875,
  author       = {Pith},
  title        = {Pith review of: VulRTex: A Reasoning-Guided Approach to Identify Vulnerabilities from Rich-Text Issue Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJIU2DYO}},
  note         = {Machine review of arXiv:2509.03875}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Software vulnerabilities exist in open-source software (OSS), and the developers who discover these vulnerabilities may submit issue reports (IRs) to describe their details. Security practitioners need to spend a lot of time manually identifying vulnerability-related IRs from the community, and the time gap may be exploited by attackers to harm the system. Previously, researchers have proposed automatic approaches to facilitate identifying these vulnerability-related IRs, but these works focus on textual descriptions but lack the comprehensive analysis of IR's rich-text information. In this paper, we propose VulRTex, a reasoning-guided approach to identify vulnerability-related IRs with their rich-text information. In particular, VulRTex first utilizes the reasoning ability of the Large Language Model (LLM) to prepare the Vulnerability Reasoning Database with historical IRs. Then, it retrieves the relevant cases from the prepared reasoning database to generate reasoning guidance, which guides LLM to identify vulnerabilities by reasoning analysis on target IRs' rich-text information. To evaluate the performance of VulRTex, we conduct experiments on 973,572 IRs, and the results show that VulRTex achieves the highest performance in identifying the vulnerability-related IRs and predicting CWE-IDs when the dataset is imbalanced, outperforming the best baseline with +11.0% F1, +20.2% AUPRC, and +10.5% Macro-F1, and 2x lower time cost than baseline reasoning approaches. Furthermore, VulRTex has been applied to identify 30 emerging vulnerabilities across 10 representative OSS projects in 2024's GitHub IRs, and 11 of them are successfully assigned CVE-IDs, which illustrates VulRTex's practicality.

Figures

Figures reproduced from arXiv: 2509.03875 by Guowei Yang, Lin Shi, Mingyang Li, Qing Wang, Ziyou Jiang.

Figure 1
Figure 1. Figure 1: The vulnerability-related IR with rich-text information, which has been assigned CVE-2019-15228. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overview of our approach VulRTex. on this reasoning graph, we can determine whether this IR contains vulnerabilities and its relevant CWE-ID. Therefore, we believe that rich-text information can help identify the vulnerability-related IR. 3 APPROACH In this section, we introduce the details of VulRTex, and the overall framework is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The steps of identifying target IR’s (i.e., Fig. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Precision-Recall (PR) curves of VulRTex and baselines. 0.0 0.2 0.4 0.6 0.8 1.0 (a) PR Curve in rich-text IRs (b) PR Curve in plain-text IRs Recall 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Precision Precision and 80.4% (AUPRC) in vulnerability identification, and 83.7% (Macro-F1) in CWE-ID Prediction, improving the best baseline with +11.0% (F1-Score), +0.7% (AUROC), +20.2% (AUPRC), and +10.5% (Macro… view at source ↗
Figure 5
Figure 5. Figure 5: The case study of VulRTex on the motivation example (example in [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The results of ablation study. and ChatGPT+DS-Agent, which can reflect the performance of these models under different output threshold 𝜃𝑜𝑢𝑡. In [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Effects of two hyper-parameters in VulRTex [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

111 extracted references · 65 canonical work pages

  1. [1]

    ISO/IEC 29147:2018: Security techniques - Vulnerability disclosure

    2018. ISO/IEC 29147:2018: Security techniques - Vulnerability disclosure. https://www.iso.org/standard/72311.html

    2018

  2. [2]

    XSS and CSRF in Blocks

    2019. XSS and CSRF in Blocks. https://github.com/daylightstudio/fuel-cms/issues/536

    2019

  3. [3]

    XSS in cmd.php for 1.2.5

    2020. XSS in cmd.php for 1.2.5. https://github.com/leenooks/phpldapadmin/issues/130

    2020

  4. [4]

    Bugzilla

    2023. Bugzilla. https://www.bugzilla.org/

    2023

  5. [5]

    Common vulnerabilities and exposures

    2023. Common vulnerabilities and exposures. https://cve.mitre.org/

    2023

  6. [6]

    Common weakness enumeration

    2023. Common weakness enumeration. https://cwe.mitre.org/

    2023

  7. [7]

    Fix C9K-230327

    2023. Fix C9K-230327. https://github.com/coreruleset/coreruleset/issues/3191

    2023

  8. [8]

    GHArchive

    2023. GHArchive. https://www.gharchive.org/. , Vol. 1, No. 1, Article . Publication date: September 2025. 22 • Ziyou Jiang, Mingyang Li, Guowei Yang, Lin Shi, and Qing Wang∗

    2023

  9. [9]

    VulRTex. 2024. Anonymized Repository. https://anonymous.4open.science/r/VulRTex-0F94

    2024

  10. [10]

    Facebook/React

    2024. Facebook/React. https://github.com/facebook/react

    2024

  11. [11]

    Gitstar Ranking

    2024. Gitstar Ranking. https://www.akto.io/cves/vendor/debian

    2024

  12. [12]

    Gitstar Ranking

    2024. Gitstar Ranking. https://gitstar-ranking.com/repositories

    2024

  13. [13]

    Gohugoio/Hugo

    2024. Gohugoio/Hugo. https://github.com/gohugoio/hugo

    2024

  14. [14]

    Honojs/Node-server

    2024. Honojs/Node-server. https://github.com/honojs/node-server

    2024

  15. [15]

    Hyprwm/Hyprland

    2024. Hyprwm/Hyprland. https://github.com/hyprwm/Hyprland

    2024

  16. [16]

    Jerryscript-project/Jerryscript

    2024. Jerryscript-project/Jerryscript. https://github.com/jerryscript-project/jerryscript

    2024

  17. [17]

    Kubernetes/Kubernetes

    2024. Kubernetes/Kubernetes. https://github.com/kubernetes/kubernetes

    2024

  18. [18]

    Mpdavis/Python-jose

    2024. Mpdavis/Python-jose. https://github.com/mpdavis/python-jose

    2024

  19. [19]

    Onosproject/Rimedo-ts

    2024. Onosproject/Rimedo-ts. https://github.com/onosproject/rimedo-ts

    2024

  20. [20]

    Onosproject/Rimedo-ts

    2024. Onosproject/Rimedo-ts. https://github.com/carla-simulator/carla

    2024

  21. [21]

    Xuxueli/Xxl-job

    2024. Xuxueli/Xxl-job. https://github.com/xuxueli/xxl-job

    2024

  22. [22]

    Zeeshan Afzal, Johan Garcia, Stefan Lindskog, and Anna Brunström. 2018. Slice Distance: An Insert-Only Levenshtein Distance with a Focus on Security Applications. In 9th IFIP International Conference on New Technologies, Mobility and Security, NTMS 2018 . IEEE, 1–5

    2018

  23. [23]

    Atlassian. 2023. Jira, Issue & Project Tracking Software. https://www.atlassian.com/software/jira

    2023

  24. [24]

    Leyla Bilge and Tudor Dumitras. 2012. Before we knew it: an empirical study of zero-day attacks in the real world. In the ACM Conference on Computer and Communications Security, CCS’12 . ACM, 833–844

    2012

  25. [25]

    Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2022. Deep Learning Based Vulnerability Detection: Are We There Yet? IEEE Trans. Software Eng. 48, 9 (2022), 3280–3296

    2022

  26. [26]

    Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David A. Wagner. 2023. DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2023 . ACM, 654–668

    2023

  27. [27]

    Neeraj Cherakara, Finny Varghese, Sheena Shabana, Nivan Nelson, Abhiram Karukayil, Rohith Kulothungan, Mohammed Afil Farhan, Birthe Nesset, Meriam Moujahid, Tanvi Dinkar, Verena Rieser, and Oliver Lemon. 2023. FurChat: An Embodied Conversational Agent using LLMs, Combining Open and Closed-Domain Dialogue with Facial Expressions. In Proceedings of the 24th...

    2023

  28. [28]

    T. M. Corporation. 2011. Common Attack Pattern Enumeration and Classification (CAPEC). http://capec.mitre.org/

    2011

  29. [29]

    Jesse Davis and Mark Goadrich. 2006. The relationship between Precision-Recall and ROC curves. In Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006) (ACM International Conference Proceeding Series, Vol. 148) . ACM, 233–240

    2006

  30. [30]

    Engler, David Yu Chen, and Andy Chou

    Dawson R. Engler, David Yu Chen, and Andy Chou. 2001. Bugs as Deviant Behavior: A General Approach to Inferring Errors in Systems Code. In Proceedings of the 18th ACM Symposium on Operating System Principles, SOSP 2001 . ACM, 57–72

    2001

  31. [31]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In MSR ’20. ACM, 508–512

    2020

  32. [32]

    Michael Gegick, Pete Rotella, and Tao Xie. 2010. Identifying security bug reports via text mining: An industrial case study. InProceedings of the 7th International Working Conference on Mining Software Repositories, MSR 2010 (Co-located with ICSE) . IEEE Computer Society, 11–20

    work page 2010

  33. [33]

    Seyed Mohammad Ghaffarian and Hamid Reza Shahriari. 2017. Software vulnerability analysis and discovery using machine-learning and data-mining techniques: A survey. Comput. Surveys 50, 4 (2017), 1–36

    work page 2017

  34. [34]

    Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. 2024. DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning. CoRR abs/2402.17453 (2024). arXiv:2402.17453

    Pith/arXiv arXiv 2024

  35. [35]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. CoRR abs/2002.08909 (2020). arXiv:2002.08909

    Pith/arXiv arXiv 2020

  36. [36]

    Nima Shiri Harzevili, Alvine Boaye Belle, Junjie Wang, Song Wang, Zhen Ming (Jack) Jiang, and Nachiappan Nagappan. 2025. A Systematic Literature Review on Automated Software Vulnerability Detection Using Machine Learning. ACM Comput. Surv. 57, 3 (2025), 55:1–55:36

    work page 2025

  37. [37]

    Allen D Householder, Garret Wassermann, Art Manion, and Chris King. 2017. The cert guide to coordinated vulnerability disclosure. Software Engineering Institute, Pittsburgh, PA (2017)

    work page 2017

  38. [38]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. CoRR abs/2311.05232 (2023)

    Pith/arXiv arXiv 2023

  39. [39]

    Hugging Face. 2023. meta-llama/Llama-2-13b-chat-hfd. https://huggingface.co/meta-llama/Llama-2-13b-chat-hf

    work page 2023

  40. [40]

    Jiyong Jang, Abeer Agrawal, and David Brumley. 2012. ReDeBug: Finding Unpatched Code Clones in Entire OS Distributions. In IEEE Symposium on Security and Privacy, SP 2012, 21-23 May 2012, San Francisco, California, USA . IEEE Computer Society, 48–62. , Vol. 1, No. 1, Article . Publication date: September 2025. VulRTex: A Reasoning-Guided Approach to Ident...

    work page 2012

  41. [41]

    Tiantian Ji, Yue Wu, Chang Wang, Xi Zhang, and Zhongru Wang. 2018. The coming era of alphahacking?: A survey of automatic software vulnerability detection, exploitation and patching techniques. In 2018 IEEE third international conference on data science in cyberspace (DSC). IEEE, 53–60

    work page 2018

  42. [42]

    Ziyou Jiang, Lin Shi, Guowei Yang, and Qing Wang. 2023. SCPatcher: Mining Crowd Security Discussions to Enrich Secure Coding Practices. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023 . IEEE, 358–370

    work page 2023

  43. [43]

    Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig

    Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig

    work page

  44. [44]

    Gong Jie, Kuang Xiao-Hui, and Liu Qiang. 2016. Survey on software vulnerability analysis method based on machine learning. In 2016 IEEE first international conference on data science in cyberspace (DSC) . IEEE, 642–647

    work page 2016

  45. [45]

    Dongkyu Kim, Byoungwook Kim, Donggeon Han, and Matous Eibich. 2024. AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipeline. CoRR abs/2410.20878 (2024). arXiv:2410.20878

    Pith/arXiv arXiv 2024

  46. [46]

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In NeurIPS’20

    work page 2020

  47. [47]

    Zhen Li, Deqing Zou, Jing Tang, Zhihao Zhang, Mingqian Sun, and Hai Jin. 2019. A comparative study of deep learning-based vulnerability detection system. IEEE Access 7 (2019), 103184–103197

    work page 2019

  48. [48]

    Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018. VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018 . The Internet Society

    work page 2018

  49. [49]

    Hongliang Liang, Lei Wang, Dongyang Wu, and Jiuyun Xu. 2016. MLSA: A static bugs analysis tool based on LLVM IR. In 17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD

    work page 2016

  50. [50]

    Guanjun Lin, Sheng Wen, Qing-Long Han, Jun Zhang, and Yang Xiang. 2020. Software Vulnerability Detection Using Deep Neural Networks: A Survey. Proc. IEEE 108, 10 (2020), 1825–1848

    work page 2020

  51. [51]

    Guanjun Lin, Jun Zhang, Wei Luo, Lei Pan, and Yang Xiang. 2017. POSTER: Vulnerability Discovery with Function Representation Learning from Unlabeled Projects. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS ’17). Association for Computing Machinery, 2539–2541

    work page 2017

  52. [52]

    Tao Liu and Longtao Zhang. 2018. Application of logistic regression in web vulnerability scanning. In 2018 International Conference on Sensor Networks and Signal Processing (SNSP) . IEEE, 486–490

    work page 2018

  53. [53]

    Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, and Dong Yu. 2023. LASER: LLM Agent with State-Space Exploration for Web Navigation. CoRR abs/2309.08172 (2023)

    Pith/arXiv arXiv 2023

  54. [54]

    Ruchika Malhotra. 2015. A systematic review of machine learning techniques for software fault prediction. Applied Soft Computing 27 (2015), 504–518

    work page 2015

  55. [55]

    Mathieu and Alan E

    Richard G. Mathieu and Alan E. Turovlin. 2023. Lost in the middle - a pragmatic approach for ERP managers to prioritize known vulnerabilities by applying classification and regression trees (CART). Inf. Comput. Secur. 31, 5 (2023), 655–674

    work page 2023

  56. [56]

    Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In1st International Conference on Learning Representations, ICLR 2013

    work page 2013

  57. [57]

    MITRE. 2014. Adversarial Tactics, Techniques & Common Knowledge (ATT&CK). https://attack.mitre.org

    work page 2014

  58. [58]

    MITRE. 2023. CWE-352: Cross-Site Request Forgery (CSRF) . https://cwe.mitre.org/data/definitions/352.html

    work page 2023

  59. [59]

    MITRE. 2023. CWE-79: Improper Neutralization of Input During Web Page Generation (’Cross-site Scripting’) (4.13). https://cwe.mitre. org/data/definitions/79.html

    work page 2023

  60. [60]

    MITRE. 2023. CWE-94: Improper Control of Generation of Code (’Code Injection’). https://cwe.mitre.org/data/definitions/79.html

    work page 2023

  61. [61]

    Linyong Nan, Ellen Zhang, Weijin Zou, Yilun Zhao, Wenfei Zhou, and Arman Cohan. 2023. On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering. CoRR abs/2311.09721 (2023)

    Pith/arXiv arXiv 2023

  62. [62]

    nilsteampassnet. 2019. Stored XSS in log of Failed Logins. https://github.com/nilsteampassnet/TeamPass/issues/2688

    work page 2019

  63. [63]

    Marwan Omar and Stavros Shiaeles. 2023. VulDetect: A novel technique for detecting software vulnerabilities using Language Models. In IEEE International Conference on Cyber Security and Resilience, CSR 2023. IEEE, 105–110. https://doi.org/10.1109/CSR57506.2023.10224924

    arXiv 2023

  64. [64]

    OpenAI. 2023. Chatgpt: A language model for conversational AI. https://www.openai.com/research/chatgpt/

    work page 2023

  65. [65]

    OpenAI. 2023. GPT-3-Models. https://platform.openai.com/docs/models/gpt-3

    work page 2023

  66. [66]

    OpenAI. 2023. GPT-3.5-Models. https://platform.openai.com/docs/models/gpt-3-5

    work page 2023

  67. [67]

    OWASP. 2023. Open web application security project. https://www.owasp.org/index.php/MainPage

    work page 2023

  68. [68]

    Tosin Daniel Oyetoyan and Patrick Morrison. 2021. An improved text classification modelling approach to identify security messages in heterogeneous projects. Softw. Qual. J. 29, 2 (2021), 509–553

    work page 2021

  69. [69]

    Page, Lawrence, Brin, Sergey, and Terry. 1999. The PageRank citation ranking: Bringing order to the web. stanford digital libraries working paper (1999). , Vol. 1, No. 1, Article . Publication date: September 2025. 24 • Ziyou Jiang, Mingyang Li, Guowei Yang, Lin Shi, and Qing Wang∗

    work page 1999

  70. [70]

    Liuxuan Pan and Allan Tomlinson. 2016. A Systematic Review of Information Security Risk Assessment. International Journal of Safety and Security Engineering 6 (06 2016), 270–281

    work page 2016

  71. [71]

    Shengyi Pan, Jiayuan Zhou, Filipe Roseiro Côgo, Xin Xia, Lingfeng Bao, Xing Hu, Shanping Li, and Ahmed E. Hassan. 2022. Automated unearthing of dangerous issue reports. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022 . ACM, 834–846

    work page 2022

  72. [72]

    Pérez, Jessica Díaz, Javier García Martin, and Bernardo Tabuenca

    Jorge E. Pérez, Jessica Díaz, Javier García Martin, and Bernardo Tabuenca. 2020. Systematic Literature Reviews in Software Engineering - Enhancement of the Study Selection Process Using Cohen’s Kappa Statistic. J. Syst. Softw. 168 (2020), 110657. https://doi.org/10.1016/ j.jss.2020.110657

    work page arXiv 2020

  73. [73]

    Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: online learning of social representations. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14 . ACM, 701–710

    work page 2014

  74. [74]

    Fayola Peters, Thein Than Tun, Yijun Yu, and Bashar Nuseibeh. 2019. Text Filtering and Ranking for Security Bug Report Prediction. IEEE Trans. Software Eng. 45, 6 (2019), 615–631

    work page 2019

  75. [75]

    Serena Elisa Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, and Cédric Dangremont. 2019. A manually-curated dataset of fixes to vulnerabilities of open-source software. In Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019 . IEEE / ACM, 383–387

    work page 2019

  76. [76]

    Hongjin Qian, Peitian Zhang, Zheng Liu, Kelong Mao, and Zhicheng Dou. 2024. MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery. CoRR abs/2409.05591 (2024). arXiv:2409.05591

    Pith/arXiv arXiv 2024

  77. [77]

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yux...

    Pith/arXiv arXiv 2023

  78. [78]

    Bonan Ruan, Jiahao Liu, Weibo Zhao, and Zhenkai Liang. 2024. VulZoo: A Comprehensive Vulnerability Intelligence Dataset. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24) . Association for Computing Machinery, 2334–2337

    work page 2024

  79. [79]

    Russell, Louis Y

    Rebecca L. Russell, Louis Y. Kim, Lei H. Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul M. Ellingwood, and Marc W. McConley. 2018. Automated Vulnerability Detection in Source Code Using Deep Representation Learning. In 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018 . IEEE, 757–762

    work page 2018

  80. [80]

    Gerard Salton and Christopher Buckley. 1988. Term-weighting Approaches in Automatic Text Retrieval. Information Processing & Management 24, 5 (1988), 513–523

    work page 1988

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.