Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

A 4-billion-parameter LLM trained with on-policy RL on repository-level code context reaches vulnerability-detection F1 of 70.45 and F1 of 58.05 on out-of-distribution vulnerabilities, rivaling a 150x larger model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 22:08 UTC pith:P6NSXLF2

load-bearing objection The RL recipe is real and the dataset work is useful, but the headline F1 numbers are unverified because the same GPT-4.1-mini judge is both the training reward and the test metric — treat the quantitative claims as provisional until independent evaluation. the 4 major comments →

arxiv 2511.11896 v3 pith:P6NSXLF2 submitted 2025-11-14 cs.CR cs.AIcs.SE

VULPO: Context-Aware Vulnerability Detection via On-Policy LLM Optimization

Youpeng Li , Fuxun Yu , Weiliang Qi , Xinda Wang This is my paper
classification cs.CR cs.AIcs.SE
keywords vulnerability detectioncontext-aware analysislarge language modelsreinforcement learningGRPOreward shapingLLM-as-a-judgecode property graphs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that real-world vulnerability detection fails not because small models lack capacity, but because existing training signals only check the final vulnerable/non-vulnerable label. It constructs ContextVul, a C/C++ dataset that pairs vulnerable functions with surrounding repository context and curated reasoning traces, then introduces VULPO, a two-stage pipeline: a light supervised cold start followed by on-policy reinforcement learning with multidimensional judge-based rewards for correctness, localization, and semantic relevance, plus difficulty-adaptive reward scaling. The central claim is that a 4B reasoning LLM trained this way substantially outperforms prompt-based and off-policy baselines, improving F1 by 85% over the base model and reaching performance comparable to a much larger frontier model. If true, the result matters because it suggests context-aware vulnerability detection can be learned by small, deployable models when the reward signal rewards the reasoning process, not just the answer.

Core claim

On its own terms, the paper establishes that the bottleneck in LLM-based vulnerability detection is the training signal rather than model size. VULPO scores each response on four dimensions — format, correctness judged against the CVE description, localization precision judged against the patch diff, and semantic relevance of the analysis — and then scales rewards so that detecting genuinely vulnerable code in harder samples counts more than easy true negatives. On a time-split held-out test set, the trained 4B model reaches pass@1 of 68.25, F1 of 70.45, and pass@8 of 97.87, and on a 31-vulnerability out-of-distribution set it reaches F1 of 58.05, beating all tested baselines of comparable o

What carries the argument

The load-bearing mechanism is vulnerability-adaptive policy optimization (VULPO), built on GRPO-style on-policy RL. A cold-start SFT stage initializes the 4B student on reasoning traces from a strong teacher model, filtered by a judge to retain only contextually correct analyses. In the RL stage, VULPO samples multiple responses per query and computes a group advantage from a multidimensional judge reward: prediction correctness (0.6/-0.6), localization precision (0/0.1/0.2), semantic relevance (0/0.1/0.2), plus format and reasoning-answer consistency penalties. Two scaling factors multiply the advantage: a label-level weight wl > 1 that increases the reward and penalty for vulnerable sample

Load-bearing premise

The whole result rests on the assumption that the judge LLM's rubric judgments — whether the model's reasoning matches the CVE description and the patch diff — are a faithful measure of genuine vulnerability detection quality, and that using that same judge as both the training reward and the test metric does not inflate the reported gains; the authors support this only with a 100-sample manual cross-check reported as 'highly consistent' without counts or inter-rater agreemen

What would settle it

Take a fresh set of 100+ hand-labeled vulnerable and fixed function pairs with repository context, score VULPO-4B's predictions with an independent judge or an execution-based oracle such as sanitizer or patch-verification signals, and compare its F1 and MCC against the base 4B model and the SFT baseline. If the margin over the base model collapses to within noise, or if VULPO's identified vulnerability locations do not align with the lines whose removal changes program behavior, the central claim fails. A simpler version: replace the judge model used during testing with a different judge appl

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, context-aware vulnerability detection is achievable at 4B scale, making repository-level scanning practical for deployment without frontier-model costs.
  • Multi-dimensional process rewards directly counteract the 'flawed reasoning but correct guess' failure mode, pushing models to locate and explain a vulnerability rather than merely label it.
  • The label-level scaling ablation implies that any future RL-based VD system must account for the asymmetry between true positives and true negatives or risk collapsing into false-negative shortcuts.
  • The cold-start epoch sensitivity — more SFT kills exploration — is a concrete tuning insight for applying on-policy RL to code-security tasks.
  • Pass@8 of 97.87 with pass@1 of 68.25 suggests the model is reliable when allowed multiple samples, which has direct implications for how such detectors would be used in practice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the same judge model supplies both the RL reward and the test metric, a portion of the measured improvement may reflect the model learning to satisfy that specific judge's rubric rather than genuine detection ability; this can be tested by re-scoring VULPO's outputs with an independent judge or an execution-based oracle.
  • The same recipe — cold-start SFT plus on-policy RL with rubric-grounded, difficulty-scaled rewards — could transfer to other code tasks where ground truth is available as a description and a diff, such as patch correctness, bug localization, or security review.
  • The large gap between pass@1 and pass@8 suggests that the practical deployment of such a model would require sampling or majority voting, and that reporting only pass@1 understates the model's usable reliability.
  • A direct extension would be to replace the closed judge with an open-source judge and measure whether the ranking of baselines shifts; if it does, the evaluation protocol needs an independent standard.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ContextVul, a C/C++ vulnerability-detection dataset that augments function-level samples from PrimeVul and SecVulEval with repository-level context extracted via CPG/cflow, plus teacher-generated reasoning traces. On top of this dataset, the authors propose VULPO, a two-stage pipeline: cold-start SFT on judge-filtered teacher reasoning, followed by GRPO-style on-policy RL with an LLM-judge multidimensional reward (correctness, localization, semantic relevance, consistency) and label/sample difficulty scaling. Experiments on 1,970 test samples and 31 OOD vulnerabilities report that a Qwen3-4B-based VULPO reaches F1 70.45 (vs 37.96 for Qwen3-4B) and OOD F1 58.05, matching or beating DeepSeek-V3.1. Ablations support the contribution of scaling and multi-dimensional rewards.

Significance. If the reported measurements are valid, the paper is significant: it provides a reusable context-augmented dataset, demonstrates on-policy RL for VD, and introduces a multi-dimensional judge-based reward that appears to substantially improve a small model. The time-ordered split, the OOD collection, and the ablation design are strengths. However, the central quantitative claims currently rest on a single judge that is also the training reward, with no independent/evaluator-agnostic confirmation and no statistical uncertainty quantification, so the magnitude of the claimed gains is not yet established.

major comments (4)
  1. [§4.3.2, §5] The same GPT-4.1-mini judge is used as the RL reward model and as the test-time scorer. Section 5 states "For reward evaluation during training and performance evaluation during testing, we select GPT-4.1-mini as the judge LLM"; the test metrics are defined by whether the judge says the target CVE is identified. VULPO is therefore directly optimized against the exact evaluator used to measure it, while the baselines are not. The only validation is a 100-sample manual check described as "highly consistent" without counts, per-dimension agreement, or inter-rater statistics. The reported F1 gains may reflect optimization against this judge's rubric rather than genuine detection. Please provide agreement statistics (e.g., Cohen's kappa) for the 100-sample check, and re-report head-to-head results using a different judge model or a human-labeled subset for the test metric.
  2. [Abstract, §5, Table 2] The abstract's headline "improving Pairwise Pass@1 by 203% relative to Qwen3-4B" is not defined in §5 and does not appear in Table 2. The closest reported metric, pass@1, improves from 53.31 to 68.25 (≈28%). Either define Pairwise Pass@1 and report it, or correct the abstract. This discrepancy affects the paper's central claim.
  3. [Table 4, OOD n=31] The OOD experiment contains 31 vulnerability/patch pairs. With n=31, pass@1 differences of 9 points (57.46 vs 48.39) have an approximate standard error of 12.7 percentage points for the difference (two proportions p≈0.5), so the observed gap is within sampling noise. No confidence intervals, bootstrap, or multiple seeds are reported anywhere in Tables 2-4. The claim of 23%/41% improvement over averages and "superior generalization ability" is not statistically supported. Please add interval estimates or use a larger OOD set.
  4. [Table 4, MCC] The reported MCC for VULPO is 14.93, which is outside the theoretical range [-1,1] for MCC. This indicates an error in the calculation or table entry. Because this same table is used to support OOD generalization, the corrected value must be reported and conclusions re-checked.
minor comments (5)
  1. [Abstract / Introduction] The abstract says "150% larger-scale" while the introduction says "150x larger-scale"; for DeepSeek-V3.1 vs a 4B model, the intended statement is "150 times", not "150%".
  2. [§4.3.2] "Figure 6.6 presents the prompt" should be "Figure 6".
  3. [§4.2] "model distillation at the code start stage" should be "cold start stage".
  4. [References] References [19] and [21] are the same paper, as are [25] and [38]; duplicate entries should be removed.
  5. [Reproducibility] The artifact is only promised "after paper acceptance"; for a paper whose claims depend on a custom dataset and judge prompts, releasing these at review time would substantially aid verification.

Circularity Check

1 steps flagged

Evaluation loop: GPT-4.1-mini is both the RL reward model and the test judge, so the reported F1/pass@1 gains are measured by the same oracle VULPO was optimized to satisfy.

specific steps
  1. fitted input called prediction [§4.3.2 LLM Judges as Reward Models; §4.3.5 Eq. (3); §5 Implementation and Metrics]
    "we employ publicly available vulnerability information obtained from NVD [42] as the ground-truth reference for evaluating the model’s predictions. ... For reward evaluation during training and performance evaluation during testing, we select GPT-4.1-mini as the judge LLM ... we calculate the performance metrics by evaluating whether the LLM identifies the target CVE in code."

    Eq. (3) maximizes r_i, GPT-4.1-mini's rubric scores from §4.3.2. §5 computes pass@1/F1/MCC from the same GPT-4.1-mini deciding whether the output 'identifies the target CVE,' using the same CVE-description/patch-diff ground truth fed to the reward judge. VULPO is thus optimized against the exact oracle that defines the reported metric, while baselines are not. The only external check is a 100-sample manual review called 'highly consistent' with no counts or agreement statistic, so the loop is unbroken; the headline F1 gains measure alignment with the reward judge, not an independent ground truth.

full rationale

The dataset construction (ContextVul) and the RL algorithm (VULPO with difficulty-adaptive scaling) are self-contained contributions and are not circular. No load-bearing self-citation chain is used; prior work by the same authors appears only for context extraction or related multi-agent methods. However, the central quantitative claim—VULPO-4B improving F1 by ~85% over Qwen3-4B and matching DeepSeek-V3.1—is evaluated with GPT-4.1-mini, the same LLM judge that provides the reward signal in Eq. (3). Since the judge prompt supplies CVE descriptions and patch diffs as ground truth, and VULPO's reward is precisely a rubric judgment of alignment with that ground truth, the test metric is not independent of the training objective. The only validation offered is a 100-sample manual check reported only as 'highly consistent,' with no counts, no inter-rater statistic, and no error bars. This is a partial but significant circularity in the evaluation loop: the model could still fail to optimize the judge or generalize poorly, and the ablations have some independent structure, but every headline comparison is confounded by construction. Hence the score is 7 rather than 6.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on a judge LLM whose rubric scores are treated as faithful measures of VD quality and are used on both the reward and evaluation side; on inherited label quality from PrimeVul/SecVulEval; on the completeness of CPG-based context extraction; and on a time-ordered split preventing reward-signal leakage. These are domain assumptions, not standard mathematics, and none is machine-checked or independently validated with reported quantitative evidence.

free parameters (5)
  • Label difficulty weight w_l = not disclosed (swept over 1.1, 1.3, 1.5 in RQ4)
    Multiplies the entire group advantage for vulnerable samples in Eq. 3 (§4.3.4). Ablations show it is the single most impactful component, yet the value used for the reported VULPO results is never stated.
  • Sample difficulty power-function constants A=1, B=2, exponent 7 = A=1, B=2, k=7
    Hand-chosen mapping from the correct-response ratio within a group to the sample weight w_s (§4.3.4, Figure 2); no sensitivity analysis is provided.
  • Reward magnitudes: correctness +0.6/-0.6, localization 0.2/0.1/0, semantic relevance 0.2/0.1/0, format -0.2 = as listed
    Hand-set in §4.3.3; the paper notes localization/semantic rewards mainly distinguish quality among correct responses, but the absolute values determine the reward landscape.
  • KL coefficient beta in Eq. 3 = 0
    Set to 0 for all VULPO runs (§5 Implementation) despite being part of the stated objective, removing the KL regularization term.
  • Group size G (responses per prompt) = 8
    Used for group advantage computation and for pass@1/pass@8/major@8 metrics (§5 Metrics).
axioms (5)
  • domain assumption GPT-4.1-mini judge evaluations (correctness, localization, semantic relevance) are a faithful proxy for true vulnerability detection quality.
    Invoked in §4.3.2 for RL rewards and in §5 Metrics for evaluation; supported only by a 100-sample manual cross-check described as 'highly consistent' with no reported counts or inter-rater agreement.
  • domain assumption The same judge model family can serve as both training reward and test evaluator without reward-hacking bias toward that evaluator.
    VULPO is RL-optimized against GPT-4.1-mini's rubric in §4.3.2, then the reported metrics are computed by the same judge in §5 — the paper provides no evidence that this alignment does not inflate the measured gains.
  • domain assumption The time-ordered 8:1:1 split of ContextVul by commit date prevents leakage of the CVE-description-based reward signal into the test set.
    §5 Dataset Partitioning: CVE descriptions, commit messages, and diffs are used both to compute training rewards and to score test predictions, so separation depends entirely on the split.
  • domain assumption PrimeVul and SecVulEval binary labels are accurate after the paper's cleaning (exclusion of multi-commit fixes and fixes touching multiple unrelated functions).
    §4.1 Data Preprocessing; label quality is inherited from the upstream datasets and their commit-based labeling process.
  • domain assumption CPG-based context extraction (Joern plus cflow-derived file selection, including callees, macros, type definitions, globals, includes) captures the repository-level semantics needed for vulnerability detection.
    §4.1 Context Extraction; context is limited to files related to the target file/call graph, and completeness is asserted but not empirically validated.

pith-pipeline@v1.3.0-alltime-deepseek · 22045 in / 21806 out tokens · 180404 ms · 2026-08-03T22:08:19.904768+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have recently shown strong potential in vulnerability detection (VD). However, accurately detecting vulnerabilities in real-world repositories requires reasoning over complex contextual interactions. Existing LLM-based VD approaches remain limited because current datasets lack complete contextual information and high-quality reasoning supervision, while existing optimization methods primarily rely on coarse outcome-centric supervision signals that fail to model the vulnerability reasoning process. To address these limitations, we first construct ContextVul, a new dataset that augments high-quality function-level vulnerability benchmarks with repository-level contextual information and curated vulnerability reasoning traces. Building upon ContextVul, we introduce a two-stage optimization framework consisting of lightweight cold-start supervised fine-tuning followed by vulnerability-adaptive on-policy optimization (VULPO). VULPO incorporates multidimensional rewards that jointly evaluate vulnerability identification, vulnerability-relevant localization, and causal reasoning quality, along with difficulty-adaptive reward scaling to mitigate reward hacking and improve RL effectiveness. Extensive experiments demonstrate the superiority of VULPO for context-aware VD. Our VULPO-4B, the first specialized vulnerability reasoning LLM, substantially outperforms existing VD baselines, improving Pairwise Pass@1 by 203% relative to Qwen3-4B and achieving competitive performance against a 150% larger-scale LLM, DeepSeek-V3.1.

Figures

Figures reproduced from arXiv: 2511.11896 by Fuxun Yu, Weiliang Qi, Xinda Wang, Youpeng Li.

Figure 1
Figure 1. Figure 1: System Overview 3. Motivation In context-aware VD, the quality of a model’s contextual reasoning directly determines the accuracy and reliability of its predictions. Therefore, its critical for the model to receive reward feedback concerning the vulnerability rea￾soning process during training. However, a critical flaw exists in current rule-based RL-based VD methods: They often focus solely on superficial… view at source ↗
Figure 6
Figure 6. Figure 6: presents the prompt for our judge LLM [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 2
Figure 2. Figure 2: Power Function To this end, we choose a power function to map the proportion of correct responses in a group (denoted as rc) to our sample difficulty weight ws. As can be seen from [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance by CWE types of vulnerabilities. In particular, VULPO’s unique ad￾vantages over other LLMs are clearly demonstrated by its deep understanding of core memory safety issues, achieving an impressive 84.9% on CWE-415 (Double Free) and 81.8% on CWE-119 (Improper Restriction of Operations within the Bounds of a Memory Buffer). Furthermore, its high recall of 81% for CWE-416 (Use After Free) and 68.9%… view at source ↗
Figure 5
Figure 5. Figure 5: Impact of Label Difficulty Weight on Model Performance [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Impact of Cold Start Stage on Model Performance [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Evaluation Judge Prompt for Vulnerable Sample. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics

    cs.SE 2026-04 unverdicted novelty 6.0

    GLMTest integrates code property graphs and GNNs with LLMs to steer test case generation toward targeted branches, raising branch accuracy from 27.4% to 50.2% on the TestGenEval benchmark.

Reference graph

Works this paper leans on

44 extracted references · 10 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Uncovering the limits of machine learn- ing for automatic vulnerability detection,

    N. Risse and M. B ¨ohme, “Uncovering the limits of machine learn- ing for automatic vulnerability detection,” in33rd USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, August 14-16, 2024, D. Balzarotti and W. Xu, Eds. USENIX Association, 2024

    2024

  2. [2]

    Llm-based vulnerability discovery through the lens of code metrics,

    F. Weissberg, L. Pirch, E. Imgrund, J. M ¨oller, T. Eisenhofer, and K. Rieck, “Llm-based vulnerability discovery through the lens of code metrics,”CoRR, vol. abs/2509.19117, 2025

    arXiv 2025

  3. [3]

    Are we learning the right features? A framework for evaluating dl-based software vul- nerability detection solutions,

    S. Das, S. T. Fabiha, S. Shafiq, and N. Medvidovic, “Are we learning the right features? A framework for evaluating dl-based software vul- nerability detection solutions,” in47th IEEE/ACM International Con- ference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 2025, pp. 2893–2904

    2025

  4. [4]

    Pre-training by predicting program dependencies for vulnerability analysis tasks,

    Z. Liu, Z. Tang, J. Zhang, X. Xia, and X. Yang, “Pre-training by predicting program dependencies for vulnerability analysis tasks,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20,

    2024

  5. [5]

    Automated software vulnerability detection with machine learning,

    J. A. Harer, L. Y . Kim, R. L. Russell, O. Ozdemir, L. R. Kosta, A. Rangamani, L. H. Hamilton, G. I. Centeno, J. R. Key, P. M. Ellingwood, M. W. McConley, J. M. Opper, P. Chin, and T. Lazovich, “Automated software vulnerability detection with machine learning,” CoRR, vol. abs/1803.04497, 2018

    Pith/arXiv arXiv 2018

  6. [6]

    Machine learning methods for software vulnerability detection,

    B. Chernis and R. M. Verma, “Machine learning methods for software vulnerability detection,” inProceedings of the Fourth ACM International Workshop on Security and Privacy Analytics, IWSPA@CODASPY 2018, Tempe, AZ, USA, March 19-21, 2018, R. M. Verma and M. Kantarcioglu, Eds. ACM, 2018, pp. 31–39

    2018

  7. [7]

    The rise of software vulnerability: Taxonomy of software vulnerabilities detection and machine learning approaches,

    H. Hanif, M. H. N. B. M. Nasir, M. F. A. Razak, A. Firdaus, and N. B. Anuar, “The rise of software vulnerability: Taxonomy of software vulnerabilities detection and machine learning approaches,”J. Netw. Comput. Appl., vol. 179, p. 103009, 2021

    2021

  8. [8]

    Deep learning based vulnerability detection: Are we there yet?

    S. Chakraborty, R. Krishna, Y . Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet?”IEEE Trans. Soft- ware Eng., vol. 48, no. 9, pp. 3280–3296, 2022

    2022

  9. [9]

    Vuldeepecker: A deep learning-based system for vulnerability de- tection,

    Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y . Zhong, “Vuldeepecker: A deep learning-based system for vulnerability de- tection,” in25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18- 21, 2018. The Internet Society, 2018

    2018

  10. [10]

    Diver- sevul: A new vulnerable source code dataset for deep learning based vulnerability detection,

    Y . Chen, Z. Ding, L. Alowain, X. Chen, and D. A. Wagner, “Diver- sevul: A new vulnerable source code dataset for deep learning based vulnerability detection,” inProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2023, Hong Kong, China, October 16-18, 2023. ACM, 2023, pp. 654–668

    2023

  11. [11]

    Llms cannot reliably identify and reason about security vulner- abilities (yet?): A comprehensive evaluation, framework, and bench- marks,

    S. Ullah, M. Han, S. Pujar, H. Pearce, A. K. Coskun, and G. Stringh- ini, “Llms cannot reliably identify and reason about security vulner- abilities (yet?): A comprehensive evaluation, framework, and bench- marks,” inIEEE Symposium on Security and Privacy, SP 2024, San Francisco, CA, USA, May 19-23, 2024. IEEE, 2024, pp. 862–880

    2024

  12. [12]

    APPATCH: automated adaptive prompting large language models for real-world software vulnerability patching,

    Y . Nong, H. Yang, L. Cheng, H. Hu, and H. Cai, “APPATCH: automated adaptive prompting large language models for real-world software vulnerability patching,” in34th USENIX Security Sympo- sium, USENIX Security 2025, Seattle, WA, USA, August 13-15, 2025, L. Bauer and G. Pellegrino, Eds. USENIX Association, 2025, pp. 4481–4500

    2025

  13. [13]

    Sv-trusteval-c: Evaluating structure and semantic reasoning in large language models for source code vulnerability analysis,

    Y . Li, P. Branco, A. M. Hoole, M. Marwah, H. M. Koduvely, G. Jour- dan, and S. Jou, “Sv-trusteval-c: Evaluating structure and semantic reasoning in large language models for source code vulnerability analysis,” inIEEE Symposium on Security and Privacy, SP 2025, San Francisco, CA, USA, May 12-15, 2025, M. Blanton, W. Enck, and C. Nita-Rotaru, Eds. IEEE, 2...

    2025

  14. [14]

    Llmxcpg: Context-aware vulnerability detection through code property graph- guided large language models,

    A. Lekssays, H. Mouhcine, K. Tran, T. Yu, and I. Khalil, “Llmxcpg: Context-aware vulnerability detection through code property graph- guided large language models,” in34th USENIX Security Sympo- sium, USENIX Security 2025, Seattle, WA, USA, August 13-15, 2025, L. Bauer and G. Pellegrino, Eds. USENIX Association, 2025, pp. 489–507

    2025

  15. [15]

    Direct preference optimization: Your language model is se- cretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is se- cretly a reward model,” inAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, A. Oh, T. Naumann...

    2023

  16. [16]

    ORPO: monolithic preference optimization without reference model,

    J. Hong, N. Lee, and J. Thorne, “ORPO: monolithic preference optimization without reference model,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, Y . Al- Onaizan, M. Bansal, and Y . Chen, Eds. Association for Compu- tational Linguistics, 2024, pp. 11 170–11 189

    2024

  17. [17]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y . K. Li, Y . Wu, and D. Guo, “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”CoRR, vol. abs/2402.03300, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,”CoRR, vol. abs/2501.12948, 2025

    Pith/arXiv arXiv 2025

  19. [20]

    Large language models for code: Security hardening and adversarial testing,

    J. He and M. T. Vechev, “Large language models for code: Security hardening and adversarial testing,” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen, Denmark, November 26-30, 2023, W. Meng, C. D. Jensen, C. Cremers, and E. Kirda, Eds. ACM, 2023, pp. 1865–1879

    2023

  20. [21]

    Vulnerability detection with code language models: How far are we?

    Y . Ding, Y . Fu, O. Ibrahim, C. Sitawarin, X. Chen, B. Alomair, D. A. Wagner, B. Ray, and Y . Chen, “Vulnerability detection with code language models: How far are we?” in47th IEEE/ACM Inter- national Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 2025, pp. 1729–1741

    2025

  21. [22]

    Everything you wanted to know about llm-based vulnerability de- tection but were afraid to ask,

    Y . Li, X. Li, H. Wu, M. Xu, Y . Zhang, X. Cheng, F. Xu, and S. Zhong, “Everything you wanted to know about llm-based vulnerability de- tection but were afraid to ask,”CoRR, vol. abs/2504.13474, 2025

    Pith/arXiv arXiv 2025

  22. [23]

    Improving LLM reasoning for vulnerability detection via group relative policy optimization,

    M. Simoni, A. Fontana, G. Rossolini, and A. Saracino, “Improving LLM reasoning for vulnerability detection via group relative policy optimization,”CoRR, vol. abs/2507.03051, 2025

    Pith/arXiv arXiv 2025

  23. [24]

    Secvuleval: Benchmarking llms for real-world C/C++ vulnerability detection,

    M. B. U. Ahmed, N. S. Harzevili, J. Shin, H. V . Pham, and S. Wang, “Secvuleval: Benchmarking llms for real-world C/C++ vulnerability detection,”CoRR, vol. abs/2505.19828, 2025

    Pith/arXiv arXiv 2025

  24. [26]

    Prompt- enhanced software vulnerability detection using chatgpt,

    C. Zhang, H. Liu, J. Zeng, K. Yang, Y . Li, and H. Li, “Prompt- enhanced software vulnerability detection using chatgpt,” inPro- ceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 2024, pp. 276– 277

    2024

  25. [27]

    Make agent defeat agent: Automatic detection of taint-style vulnerabilities in llm-based agents,

    F. Liu, Y . Zhang, J. Luo, J. Dai, T. Chen, L. Yuan, Z. Yu, Y . Shi, K. Li, C. Zhou, H. Chen, and M. Yang, “Make agent defeat agent: Automatic detection of taint-style vulnerabilities in llm-based agents,” in34th USENIX Security Symposium, USENIX Security 2025, Seattle, WA, USA, August 13-15, 2025, L. Bauer and G. Pellegrino, Eds. USENIX Association, 2025...

    2025

  26. [28]

    MA VUL: multi-agent vulner- ability detection via contextual reasoning and interactive refinement,

    Y . Li, K. Joshi, X. Wang, and E. Wong, “MA VUL: multi-agent vulner- ability detection via contextual reasoning and interactive refinement,” CoRR, vol. abs/2510.00317, 2025

    arXiv 2025

  27. [29]

    Let the trial begin: A mock- court approach to vulnerability detection using llm-based agents,

    R. Widyasari, M. Weyssow, I. C. Irsan, H. W. Ang, F. Liauw, E. L. Ouh, L. K. Shar, H. J. Kang, and D. Lo, “Let the trial begin: A mock- court approach to vulnerability detection using llm-based agents,” CoRR, vol. abs/2505.10961, 2025

    arXiv 2025

  28. [30]

    Large lan- guage model-powered smart contract vulnerability detection: New perspectives,

    S. Hu, T. Huang, F. Ilhan, S. F. Tekin, and L. Liu, “Large lan- guage model-powered smart contract vulnerability detection: New perspectives,” in5th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications, TPS-ISA 2023, Atlanta, GA, USA, November 1-4, 2023. IEEE, 2023, pp. 297–306

    2023

  29. [31]

    Security vulnerability detection with multitask self-instructed fine-tuning of large language models,

    A. Z. H. Yang, H. Tian, H. Ye, R. Martins, and C. Le Goues, “Security vulnerability detection with multitask self-instructed fine-tuning of large language models,”CoRR, vol. abs/2406.05892, 2024

    Pith/arXiv arXiv 2024

  30. [32]

    Instructpatentgpt: training patent language models to follow instructions with human feedback,

    J. Lee, “Instructpatentgpt: training patent language models to follow instructions with human feedback,”Artif. Intell. Law, vol. 33, no. 3, pp. 739–782, 2025

    2025

  31. [33]

    Boosting vul- nerability detection of llms via curriculum preference optimization with synthetic reasoning data,

    X. Wen, Y . Yang, C. Gao, Y . Xiao, and D. Ye, “Boosting vul- nerability detection of llms via curriculum preference optimization with synthetic reasoning data,” inFindings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Association for Computat...

    2025

  32. [34]

    R2vul: Learning to reason about software vulnerabilities with reinforcement learning and structured reasoning distillation,

    M. Weyssow, C. Yang, J. Chen, Y . Li, H. Huang, R. Widyasari, H. W. Ang, F. Liauw, E. L. Ouh, L. K. Shar, and D. Lo, “R2vul: Learning to reason about software vulnerabilities with reinforcement learning and structured reasoning distillation,”CoRR, vol. abs/2504.04699, 2025

    Pith/arXiv arXiv 2025

  33. [35]

    IPO: interior-point policy optimiza- tion under constraints,

    Y . Liu, J. Ding, and X. Liu, “IPO: interior-point policy optimiza- tion under constraints,” inThe Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New Yor...

    2020

  34. [36]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Ta...

    Pith/arXiv arXiv 2025

  35. [37]

    Reposvul: A repository-level high-quality vulnerability dataset,

    X. Wang, R. Hu, C. Gao, X. Wen, Y . Chen, and Q. Liao, “Reposvul: A repository-level high-quality vulnerability dataset,” inProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2024, Lis- bon, Portugal, April 14-20, 2024. ACM, 2024, pp. 472–483

    2024

  36. [38]

    Modeling and discovering vulnerabilities with code property graphs,

    F. Yamaguchi, N. Golde, D. Arp, and K. Rieck, “Modeling and discovering vulnerabilities with code property graphs,” in2014 IEEE Symposium on Security and Privacy, SP 2014, Berkeley, CA, USA, May 18-21, 2014. IEEE Computer Society, 2014, pp. 590–604

    2014

  37. [39]

    Clang: a c language family frontend for llvm,

    “Clang: a c language family frontend for llvm,” https://clang.llvm.org/, 2024

    2024

  38. [40]

    Cybergym: Evaluating AI agents’ cybersecurity capabilities with real-world vul- nerabilities at scale,

    Z. Wang, T. Shi, J. He, M. Cai, J. Zhang, and D. Song, “Cybergym: Evaluating AI agents’ cybersecurity capabilities with real-world vul- nerabilities at scale,”CoRR, vol. abs/2506.02548, 2025

    arXiv 2025

  39. [41]

    Secureagentbench: Benchmarking secure code generation under realistic vulnerability scenarios,

    J. Chen, H. Huang, Y . Lyu, J. An, J. Shi, C. Yang, T. Zhang, H. Tian, Y . Li, Z. Li, X. Zhou, X. Hu, and D. Lo, “Secureagentbench: Benchmarking secure code generation under realistic vulnerability scenarios,”CoRR, vol. abs/2509.22097, 2025

    Pith/arXiv arXiv 2025

  40. [42]

    Evaluation Judge Prompt for Vulnerable Sample <|system|> You are to act as a meticulous and impartial Code Security Expert and Evaluator

    https://nvd.nist.gov/, 2025. Evaluation Judge Prompt for Vulnerable Sample <|system|> You are to act as a meticulous and impartial Code Security Expert and Evaluator. <|user|> """1. GOAL Your primary goal is to assess the quality of an analysis of a vulnerable piece of code. You must evaluate the analysis against a provided set of ground truth information...

    2025

  41. [44]

    analysis

    INPUT FORMAT You will be provided with a JSON object containing two main keys: analysis and ground_truth_info ```json {"analysis": "<The full analysis, including its reasoning and answer.>", "ground_truth_info": { "is_vulnerable": true, "cve_description": "<The official CVE description of the vulnerability.>", "patch_commit_message": "<The developer's com...

  42. [45]

    For each dimension, you need to provide a brief justification and choose an option

    EVALUATION WORKFLOW AND OPTION RUBRIC You must follow these steps to evaluate the analysis and produce a final JSON output. For each dimension, you need to provide a brief justification and choose an option. Step 1: Analyze Ground Truth First, carefully review all the information in the ground_truth_info. This is your foundation for judgment. Step 2: Eval...

  43. [46]

    correctness

    OUTPUT FORMAT Your final output must be a single JSON object. Do not include any text or explanation outside of the JSON structure. The JSON must contain a key for each dimension's justification and option. ```json{ "correctness": { "justification": "<Your brief reason>", "option": <choose from ["CORRECT", "PARTIALLY CORRECT", "INCORRECT"]> }, "localizati...

  44. [2024]

    151:1–151:13

    ACM, 2024, pp. 151:1–151:13

    2024