Pith. sign in

REVIEW 4 major objections 4 minor 5 cited by

Everything You Wanted to Know About LLM-based Vulnerability Detection But Were Afraid to Ask

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A context-rich evaluation framework claims that LLMs are far better at vulnerability detection than prior benchmarks suggested, and that the three common beliefs against them are artifacts of missing context.

desk verdict Useful dataset and honest scaling analysis, but the core claim of overturning prior consensuses rests on scoring choices that manufacture the result. read the letter →

arxiv 2504.13474 v1 pith:JFMCL4L2 submitted 2025-04-18 cs.CR

Yue Li , Xiao Li , Hao Wu , Minghui Xu , Yue Zhang , Xiuzhen Cheng , Fengyuan Xu , Sheng Zhong This is my paper · ORCID
classification cs.CR
keywords LLMvulnerabilitydetectioncontext-richevaluationcodepropertygraphCWEclassificationpairedvulnerable-patchedbenchmarkrationaletest-timescalingfalsepositiveattribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that three widely repeated conclusions about LLM-based vulnerability detection—that models perform near random, that they cannot tell patched from vulnerable code, and that larger models do not improve—are artifacts of evaluating on isolated functions or files without surrounding context. To test this, it builds CORRECT, a context-rich evaluation that feeds models CWE descriptions, callee functions, type and global declarations, and slicing paths, and that scores the model's verdict together with its written rationale. Across 2,000 vulnerable–patched pairs spanning 99 CWE classes and 13 models from four model families, the best models reach about 67% accuracy, F1 above 0.7 on common CWE categories, and precision near 0.8, with a (1,0) pair proportion of 37% versus a 25% random baseline. The paper also argues that most false positives are reasoning errors—the model sees the patch but judges it insufficient—rather than failures to notice patches, and that scaling helps but with diminishing returns and recall loss. If the claims hold, prior benchmarks measured context deprivation as much as model capability.

What carries the argument

The load-bearing object is the context-rich paired prompt built by CORRECT's code-context pipeline: a vulnerable function and its patched counterpart share one merged context assembled from Code Property Graphs and slicing, containing callee functions and macros, global variables, type declarations, library imports, and slicing-path parameters, plus the CWE description and assumptions that restrict analysis to marked parameters and statements annotated '//potential'. That prompt is the mechanism that converts a guess about an isolated snippet into a grounded vulnerability-reasoning task, because it supplies the execution and data-flow facts that determine whether a patched code path is actually safe. The second mechanism is the two-mode assessment: Lenient Mode accepts any detection whose rationale names the ground truth, while Strict Mode feeds false alarms whose rationale does not mention the ground truth back into the model for up to four rounds of re-analysis, separating true misclassification from mere reasoning error.

What would settle it

Run the CORRECT prompt with the '//potential' markers and irrelevant-parameter annotations stripped out while keeping all callee, type, global, and slicing context; if SOTA accuracy drops back toward the 0.5–0.6 no-context range, the paper's central attribution of the gain to context fails. As a second check, replace the GPT-4o judge with an alternative judge or a majority of judges on the same rationales; if MATCH/MISMATCH flips on more than the reported 8% disagreement, the precision and F1 figures are not stable.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that context-deprived evaluation produces two systematic distortions: incorrect conclusions (a model flags patched code as vulnerable because a callee's real behavior is missing from the prompt) and incorrect rationales (a model gives the right label for a non-ground-truth reason). With context supplied, state-of-the-art models show real discrimination: DeepSeek-R1 reaches 67% accuracy and 37% (1,0) pair-wise detection in Strict Mode, SOTA precision approaches 0.8, and key CWE classes such as CWE-664 and CWE-682 reach F1 above 0.7. The paper interprets this as overturning all three consensuses: LLMs are not unreliable, they are not insensitive to patches, and a model-size scaling effect appears once context is present. It also reports new limitations that remain after context is fixed: rare CWE types are detected near random (e.g., CWE-697 max F1 0.400), recall stays around 0.5, and reasoning models overthink, turning correct answers wrong.

Load-bearing premise

The load-bearing premise is that telling the model which statements and parameters to focus on through '//potential' marks and 'irrelevant parameter' annotations is legitimate context rather than an answer-hiding hint; if those annotations, not the added callee, type, and global context, drive the gains, the comparison against the no-context baseline would not support overturning the consensuses.

Editorial extensions

If this is right

  • Prior function- and file-level benchmarks should be re-read as measuring context deprivation; their near-random scores are not evidence about model capability.
  • Future evaluations should supply callee, type, and global context and validate rationales, not just binary labels, or they will both under- and over-estimate models.
  • Detection systems should treat recall, not precision, as the binding constraint: SOTA precision is near 0.8 while recall stays around 0.5.
  • Scaling is a real but weak lever: fivefold more thinking tokens buy less than 0.05 accuracy, so test-time scaling alone is an uneconomical route to improvement.
  • Reasoning models need safeguards against over-thinking, since 'mis-corrected reasoning' converts correct answers into false alarms on patched code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the '//potential' markers and the explicit marking of unrelated parameters are ground-truth-derived hints, so a control that removes only those hints—while keeping callee, type, and global context—would show how much of the 67% accuracy is genuine context and how much is leakage.
  • Beyond the paper: the 92% LLM-as-a-judge agreement is measured on 50 sampled rationales; re-running the judge with ensembles or with a second, independently prompted judge would bound the noise in the reported precision and F1.
  • Beyond the paper: the common-versus-rare CWE gap suggests a testable extension—augmenting prompts with vulnerability-pattern examples for rare CWEs should close most of the gap if the bottleneck is knowledge rather than reasoning.
  • Beyond the paper: the paper's own RAG example (a retrieved integer-overflow patch leading the model to flag an unrelated input-validation issue) implies that naive semantic-similarity retrieval can hurt detection; grounding retrieval in vulnerability type, not code similarity, is the obvious follow-up.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper argues that three community consensuses about LLM-based vulnerability detection—that LLMs are unreliable, insensitive to code patches, and plateaued across model scales—are artifacts of context-free evaluations. To support this, it proposes CORRECT, a framework that augments vulnerable/patched function pairs with callee functions, type declarations, slicing information, and CWE descriptions, and that uses LLM-as-a-judge to evaluate rationales. The authors construct a 2,000-pair dataset spanning 99 CWEs, evaluate 13 LLMs, and report that with sufficient context, state-of-the-art models reach about 67% accuracy, precision around 0.8, and F1 above 0.7 on common CWE categories, overturning all three consensuses.

Significance. If the results held, the paper would be an important corrective to a large body of benchmarking work: it provides a substantial new dataset, evaluates a broad model zoo, and moves beyond binary labels to rationale-level assessment. The dataset release, the manual label audit, and the breadth of models are genuine strengths. However, the central comparison is confounded by construction: the context-rich condition includes ground-truth-derived hints (CWE descriptions and "//potential" markers), and the scoring procedure reclassifies false positives on patched code as true negatives and adds an interactive correction loop. These issues directly affect the headline numbers that overturn the consensuses, so the paper's central claim is not currently supported.

major comments (4)
  1. [§4.3, Table 2] Lenient Mode counts δ(Patched, 1, F) = 0 as True Negative, meaning that a model predicting HAS_VUL on patched code is scored as correct whenever the rationale is judged not to reference the ground-truth vulnerability. In any standard binary classification, HAS_VUL on patched code is a false positive regardless of the rationale's content. Because precision, recall, F1, accuracy, and the pairwise (1,1) proportion in Figures 4 and 5 are all computed from this reclassified contingency table, the paper's headline result that SOTA models achieve precision near 0.8 and a (1,1) rate near 10% (Finding #3) is inflated by the scoring rule itself. This directly undermines the evidence used to overturn Consensus #2.
  2. [§4.3, Appendix C] Strict Mode does not simply measure the model's ability to distinguish patched from vulnerable code; when a patched input is flagged as vulnerable with an unrelated rationale, the framework instructs the model to ignore previously inferred non-ground-truth vulnerabilities and re-asks it up to four times until it produces a non-alarm or exhausts the budget. Appendix C reports that this feedback altered the answer in 10.3% of NO_VUL cases at round 1 and in additional cases at later rounds. This interactive correction loop effectively tells the model that its previous answer was wrong, so the resulting precision and (1,0) proportions reflect the framework's ability to coax out a desired answer rather than the model's intrinsic discrimination. The paper's claim that LLMs can reliably distinguish patched from vulnerable code is therefore not established by the reported numbers.
  3. [§4.2, Prompt I] The context-rich prompt contains information derived from the ground truth that is absent from the "w/o context" baseline: the CWE description of the specific vulnerability, the "//potential" markers on slicing-path statements, and the explicit marking of unrelated parameters as irrelevant. The prompt also instructs the model to restrict its analysis to these marked statements. These hints can directly reveal where the vulnerability is located, so the observed improvement is not attributable to the added callee, type, and global context as claimed in Finding #1. To support the paper's central assertion that prior consensuses are artifacts of missing context, the authors would need to compare against a control condition that adds the same CWE and slicing information without the ground-truth-derived markers, and separately ablate the markers.
  4. [§5.2, Figure 4] The random-baseline comparisons used to support Finding #1 are also affected by the design. The pairwise (1,0) proportion is compared to a 25% random baseline, but under Lenient Mode a patched input that is flagged with an unrelated rationale is scored as correct (a true negative), which increases the (1,0) proportion by suppressing the (1,1) cell. Similarly, precision in Figure 4(f) is computed after Lenient/Strict reclassification. Thus the claim that models 'significantly surpass' the random baseline conflates model capability with the scoring revision. A scoring-only ablation would be needed to determine how much of the reported improvement comes from context versus from the evaluation rules.
minor comments (4)
  1. [§5.1] The sentence 'GPT-4o is n SOTA model' contains a typo and should read 'is a SOTA model.'
  2. [Figure 4 caption] The caption lists subfigures (a)-(d) as the 'w/o context' condition and (f), (h), (i), (j) as Strict Mode, but the figure contains subfigures (e) and (g) labeled 'w/ context, w/o revision' and 'Lenient Mode'; the caption should be updated to describe all panels consistently.
  3. [§5.2] The sentence 'most models achieved F1-scores ranging from 0.5 to 0.6 and 0.5 to 0.55 accuracy' is grammatically ambiguous; the two metric ranges should be presented in separate clauses.
  4. [Appendix G] The exclusion of r1-qn-7b from the main results is justified by abnormal outputs, but the paper should also report which performance metrics would change if the model were included, or at least state whether any of the headline findings depend on its exclusion.

Circularity Check

2 steps flagged · score 6.0 of 10

CORRECT's headline gains are partly manufactured: false positives on patched code are relabeled as true negatives and the prompt annotates the ground-truth vulnerability location as //potential.

  1. self definitional [Section 4.3, Table 2; RQ2 results in Section 5.3]
    "Notably, δ(Patched, 1, F) = 0 also yields a True Negative since the rationale doesn't reference the original vulnerability. ... While Lenient Mode counts these as True Negatives, Strict Mode implements a feedback loop providing corrective guidance (instructing the model to ignore previously inferred non-ground-truth vulnerabilities and re-analyze) until the model either (1) incorrectly identifies the patched ground-truth vulnerability as vulnerable, or (2) reaches max_feedback_rounds with no false alarm."

    A raw model output of HAS_VUL on patched code is a false positive under standard binary classification regardless of the rationale. Here that output is reclassified as a True Negative whenever the model's rationale is judged not to reference the ground-truth vulnerability, and Strict Mode then lets the model revise after corrective, ground-truth-derived guidance. Precision, accuracy, F1, and the (1,1) proportion in RQ2 are all computed from these redefined counts, so the reported 'precision close to 0.8' and '(1,1) proportion of approximately 10%' are inflated by construction. The claim that Consensus #2 is a misconception therefore rests on an outcome definition that removes the very false positives the consensus was about.

  2. other [Section 4.1 Code Context Build; Section 4.2 Prompt I; Appendix E]
    "CWE description: # including the ground-truth CWE and its description. Context-Rich Code: # including context and the code to be detected, the slicing path of vulnerabilities in the code will be marked with //potential. ... Assumptions restricting analysis to marked the slicing parameters and statements with the comment “//potential”; ... we explicitly mark unrelated function parameters (those without data/control-flow influence on vulnerable statements) as irrelevant"

    The 'context' supplied to the model is built from the ground-truth vulnerability: the prompt contains the ground-truth CWE description, and the code's vulnerability-relevant slicing path is explicitly tagged with //potential, while unrelated parameters are marked irrelevant. When the paper reports that context-rich evaluation raises accuracy/F1 and overturns Consensus #1, the improvement over the w/o-context baseline is not attributable solely to callee, type, or global context; the model is being pointed at the lines derived from the target vulnerability. The predicted label is therefore partially constructed from the ground-truth annotation in the input, making the claimed context-driven gain self-fulfilling.

full rationale

Two load-bearing pieces of evidence reduce by construction. First, in Section 4.3 and Table 2, the Lenient Mode revision function δ(Patched, 1, F) = 0 converts a raw false positive on patched code into a True Negative, and Strict Mode adds a corrective feedback loop that re-asks the model after telling it to ignore previously inferred non-ground-truth vulnerabilities. The RQ2 findings that SOTA precision reaches about 0.8 and that the (1,1) proportion drops to about 10% are computed from these redefined outcomes, so the claimed overturn of Consensus #2 is an artifact of the scoring definition rather than evidence of genuine patch discrimination. Second, the context-rich prompt itself leaks the ground-truth-derived answer location: it includes the ground-truth CWE description and marks the slicing path of the vulnerability with //potential, while marking unrelated parameters irrelevant. This makes the RQ1 and RQ3 gains attributed to context partly manufactured by input construction: the model is handed the location and type of the target vulnerability before it is asked to predict it. Because all three consensuses are declared overturned using these measurements, the central claim has substantial self-definitional circularity. The paper does not rely on load-bearing self-citations or imported uniqueness theorems, and there remains some independent content in raw model orderings and scaling trends, so the score is 6 rather than higher.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No fitted model parameters, but hand-chosen evaluation thresholds and several domain assumptions carry the central claim. The main circularity risk is that the 'context' includes ground-truth CWE descriptions and marked vulnerable statements.

free parameters (3)
  • max_feedback_rounds = 4
    Chosen from Table 6 because after round 2 fewer than 5% of cases change; affects Strict Mode precision and the central RQ2 results.
  • callee depth limit = 2 layers
    Section 4.1 restricts context to two layers of callees and says this is empirically validated; deeper call chains may be needed for some vulnerabilities.
  • CWE-1000 subset size = 400 pairs, about 50 per CWE
    The headline evaluation uses 400 of the 2,000 pairs; sampling choices affect all reported metrics and the scaling conclusions.
assumptions (5)
  • domain assumption Ground-truth labels from MoreFixes, PrimeVul, and ReposVul are accurate enough (98% label accuracy from a 50-pair manual audit).
    If underlying labels are wrong, all metrics shift. The paper cites known data-quality problems but audits only 50 of 2,000 pairs.
  • domain assumption GPT-4o as LLM-as-a-judge correctly determines rationale MATCH and FALSE_ALARM outcomes.
    Metrics in both Lenient and Strict modes depend on rationale correctness; judge errors propagate into F1, precision, and recall. Validation is only 91/99 sampled judgments.
  • ad hoc to paper Including CWE descriptions and marking '//potential' locations does not invalidate measuring vulnerability detection capability.
    Prompt I and Phase II use ground-truth-derived CWE and slicing paths; the paper treats this as context rather than leakage, but this assumption is load-bearing.
  • domain assumption Static context extraction with cflow call graphs, Joern CPGs, and slicing captures enough execution and data-flow context for real vulnerabilities.
    Section 4.1 relies on these tools; incomplete context could still cause the UO(I) and UO(II) failure modes the paper aims to remove.
  • ad hoc to paper Patched code flagged vulnerable with an unrelated rationale can be counted as a true negative in Lenient Mode and can be corrected by feedback in Strict Mode.
    Table 2 defines delta(Patched, 1, F) = 0 as True Negative; this reclassification affects precision and the finding that most false positives are reasoning errors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Everything You Wanted to Know About LLM-based Vulnerability Detection But Were Afraid to Ask." pith.science (2026). https://pith.science/paper/JFMCL4L2

@misc{pith2026250413474,
  author       = {Pith},
  title        = {Pith review of: Everything You Wanted to Know About LLM-based Vulnerability Detection But Were Afraid to Ask},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JFMCL4L2}},
  note         = {Machine review of arXiv:2504.13474}
}
read the original abstract

Large Language Models are a promising tool for automated vulnerability detection, thanks to their success in code generation and repair. However, despite widespread adoption, a critical question remains: Are LLMs truly effective at detecting real-world vulnerabilities? Current evaluations, which often assess models on isolated functions or files, ignore the broader execution and data-flow context essential for understanding vulnerabilities. This oversight leads to two types of misleading outcomes: incorrect conclusions and flawed rationales, collectively undermining the reliability of prior assessments. Therefore, in this paper, we challenge three widely held community beliefs: that LLMs are (i) unreliable, (ii) insensitive to code patches, and (iii) performance-plateaued across model scales. We argue that these beliefs are artifacts of context-deprived evaluations. To address this, we propose CORRECT (Context-Rich Reasoning Evaluation of Code with Trust), a new evaluation framework that systematically incorporates contextual information into LLM-based vulnerability detection. We construct a context-rich dataset of 2,000 vulnerable-patched program pairs spanning 99 CWEs and evaluate 13 LLMs across four model families. Our framework elicits both binary predictions and natural-language rationales, which are further validated using LLM-as-a-judge techniques. Our findings overturn existing misconceptions. When provided with sufficient context, SOTA LLMs achieve significantly improved performance (e.g., 0.7 F1-score on key CWEs), with 0.8 precision. We show that most false positives stem from reasoning errors rather than misclassification, and that while model and test-time scaling improve performance, they introduce diminishing returns and trade-offs in recall. Finally, we uncover new flaws in current LLM-based detection systems, such as limited generalization and overthinking biases.

Figures

Figures reproduced from arXiv: 2504.13474 by the authors.

Figure 1
Figure 1. Reasoning with LLMs. Compared to System 1 thinking, reasoning achieves higher accu￾racy by decomposing complex tasks into sequential steps. It also en￾hances interpretability, as the intermediate rationale provides trans￾parent insights into LLM’s decision-making process. Recent works further demonstrate that LLMs exhibit testing-time scaling proper￾ties [46]—allocating more computation time (e.g., additional reason… view at source ↗
Figure 2
Figure 2. (a) CVE-2017-7875, an out-of-bounds write vulnerability. (b) Essential callee context for (a). (c) A reasonable rationale [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The three stages of the CORRECT are as follows: The constructed functions and shared context are used in (b)-I; the rationales generated in (b)-I are evaluated for correctness by (b)-II, which employs LLM-as-a-judge. Finally, the assessment is conducted in (c). In (c)-Lenient Mode, only the key outcomes are displayed in this figure, with further details provided in §4.3. prepared in §4.1, including callee functions/… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Performance comparison across various evaluation settings. (a)-(d): Different metrics in the “w/o context, w/o revision” [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Pair-wise prediction proportion of all LLMs. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: F1-score comparison of 10 top-level CWEs. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Test-test scaling of o3-mini and r1-qn-14b. (a)–(d), (e)–(h), and (i)–(l) show accuracy, precision, and recall, respectively. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: CVE-2020-11039, an integer overflow vulnerability. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 10
Figure 10. Figure 10: Context-rich vulnerability assessment prompt. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 9
Figure 9. Figure 9: Distributions of context length, function length, [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 13
Figure 13. Figure 13: Prompt Length Distribution for Normal vs. Abnor [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: CVE-2022-39318, a division-by-zero vulnerability. [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: CVE-2022-1283, a null pointer dereference vulner [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: CVE-2012-6689, an improper access control vul [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments

    cs.CR 2026-07 conditional novelty 7.0 of 10

    Adaptive adversarial source-code comments evade four LLM vulnerability detectors at >90% success on 125 real null-pointer bugs without altering program behavior.

  2. DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection

    cs.CR 2026-07 conditional novelty 6.0 of 10

    DREA improves repository-level vulnerability detection by coupling an LLM planner that forms hypotheses with a cheap local explorer that gathers cross-file evidence, lifting paired accuracy from 19-26% to 30-42% at mu...

  3. VULPO: Context-Aware Vulnerability Detection via On-Policy LLM Optimization

    cs.CR 2025-11 conditional novelty 6.0 of 10

    A 4B LLM trained with judge-scored, difficulty-scaled on-policy RL reaches F1 70.45 on context-aware C/C++ vulnerability detection, comparable to DeepSeek-V3.1, with the caveat that the same judge provided the trainin...

  4. Mono: Is Your "Clean" Vulnerability Dataset Really Solvable? Exposing and Trapping Undecidable Patches and Beyond

    cs.CR 2025-06 conditional novelty 6.0 of 10

    Mono reports that 31% of MegaVul patches are non-security and about 16.7% of CVEs are 'undecidable', while its added context raises LLM vulnerability detection F1 by up to 15%.

  5. When LLMs Copy to Think: Uncovering Copy-Guided Attacks in Reasoning LLMs

    cs.CR 2025-07 conditional novelty 4.0 of 10

    Hidden strings in code exploit a reasoning model's tendency to copy tokens into its own thinking, enabling output length and result manipulation.

Reference graph

Works this paper leans on

66 extracted references · 24 canonical work pages · cited by 5 Pith papers

  1. [1]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157 (2024)

    arXiv 2024

  2. [2]

    Jafar Akhoundali, Sajad Rahim Nouri, Kristian Rietveld, and Olga Gadyatskaya

  3. [3]

    Angello Astorga, Siwakorn Srisakaokul, Xusheng Xiao, and Tao Xie. 2018. Pre- Infer: Automatic inference of preconditions via symbolic analysis. In 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). IEEE, 678–689

    work page 2018

  4. [4]

    Prajjwal Bhargava and Vincent Ng. 2022. Commonsense knowledge reasoning and generation with pre-trained language models: A survey. In Proceedings of the AAAI conference on artificial intelligence , Vol. 36. 12317–12325

    work page 2022

  5. [5]

    Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023. Uni- versal self-consistency for large language model generation. arXiv preprint arXiv:2311.17311 (2023)

    arXiv 2023

  6. [6]

    Roland Croft, M Ali Babar, and M Mehdi Kholoosi. 2023. Data quality for software vulnerability datasets. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 121–133

    2023

  7. [7]

    Cursor. 2025. Cursor - The AI Code Editor. https://www.cursor.com

    2025

  8. [8]

    CVE.org. 2025. CVE.org. https://www.cve.org/

    work page 2025

Show all 66 references
  1. [9]

    Hao Ding, Ziwei Fan, Ingo Guehring, Gaurav Gupta, Wooseok Ha, Jun Huan, Linbo Liu, Behrooz Omidvar-Tehrani, Shiqi Wang, and Hao Zhou. 2024. Reasoning and planning with large language models in code development. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Disc...

    2024

  2. [10]

    Yangruibo Ding, Saikat Chakraborty, Luca Buratti, Saurabh Pujar, Alessandro Morari, Gail Kaiser, and Baishakhi Ray. 2023. Concord: clone-aware contrastive learning for source code. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis . 26–38

    2023

  3. [11]

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability Detection with Code Language Models: How Far Are We?. In 2025 IEEE/ACM 47th International Conference on Software Engine...

    2024

  4. [12]

    Xueying Du, Geng Zheng, Kaixin Wang, Jiayi Feng, Wentai Deng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma, and Yiling Lou. 2024. Vul-rag: Enhanc- ing llm-based vulnerability detection via knowledge-level rag. arXiv preprint arXiv:2406.11147 (2024)

    2024 arXiv

  5. [13]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining ...

    2024

  6. [14]

    Zeyu Gao, Hao Wang, Yuchen Zhou, Wenyu Zhu, and Chao Zhang. 2023. How far have we gone in vulnerability detection using large language models. arXiv preprint arXiv:2311.12420 (2023)

    2023 arXiv

  7. [15]

    GitHub. 2025. GitHub Copilot · Your AI Pair Programmer. https://github.com/f eatures/copilot

    2025

  8. [16]

    José Gonçalves, Tiago Dias, Eva Maia, and Isabel Praça. 2024. Scope: Evaluating llms for software vulnerability detection. arXiv preprint arXiv:2407.14372 (2024)

    2024 arXiv

  9. [17]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  10. [18]

    Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. 2024. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.046...

    2024 arXiv

  11. [19]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 (2021)

    2021 arXiv

  12. [20]

    Jie Huang and Kevin Chen-Chuan Chang. 2022. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403 (2022)

    2022 arXiv

  13. [21]

    IBM. 2024. Cost of a Data Breach 2024. https://www.ibm.com/reports/data- breach

    2024

  14. [22]

    Yixin Ji, Juntao Li, Hai Ye, Kaixin Wu, Jia Xu, Linjian Mo, and Min Zhang. 2025. Test-time Computing: from System-1 Thinking to System-2 Thinking. arXiv preprint arXiv:2501.02497 (2025)

    2025 arXiv

  15. [23]

    Joern.io. 2024. Joern - The Bug Hunter’s Workbench. https://joern.io/

    2024

  16. [24]

    Daniel Kahneman. 2011. Thinking, fast and slow . macmillan

    2011

  17. [25]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  18. [26]

    Avishree Khare, Saikat Dutta, Ziyang Li, Alaia Solko-Breslin, Rajeev Alur, and Mayur Naik. 2023. Understanding the effectiveness of large language models in detecting security vulnerabilities. arXiv preprint arXiv:2311.16169 (2023)

    2023 arXiv

  19. [27]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing s...

    2020

  20. [28]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations

    2023

  21. [29]

    Yu Liu, Lang Gao, Mingxin Yang, Yu Xie, Ping Chen, Xiaojin Zhang, and Wei Chen. 2024. Vuldetectbench: Evaluating the deep capability of vulnerability detection with large language models. arXiv preprint arXiv:2406.07595 (2024)

    2024 arXiv

  22. [30]

    Wei Ma, Daoyuan Wu, Yuqiang Sun, Tianwen Wang, Shangqing Liu, Jian Zhang, Yue Xue, and Yang Liu. 2024. Combining fine-tuning and llm-based agents for in- tuitive smart contract auditing with justifications.arXiv preprint arXiv:2403.16073 (2024)

    2024 arXiv

  23. [31]

    Grégoire Menguy, Sébastien Bardin, Nadjib Lazaar, and Arnaud Gotlieb. 2022. Au- tomated program analysis: Revisiting precondition inference through constraint acquisition. In IJCAI-ECAI 22-31st International Joint Conference on Artificial Intelligence and the 25th European Con...

    2022

  24. [32]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 (2025)

    2025 arXiv

  25. [33]

    Yu Nong, Mohammed Aldeen, Long Cheng, Hongxin Hu, Feng Chen, and Haipeng Cai. 2024. Chain-of-thought prompting of large language models for discovering and fixing software vulnerabilities. arXiv preprint arXiv:2402.17230 (2024). Conference’17, July 2017, Washington, DC, USA Yu...

    2024 arXiv

  26. [34]

    National Institute of Standards and Technology (NIST). 2025. National Vulnera- bility Database (NVD). https://nvd.nist.gov/

    2025

  27. [35]

    OpenAI. 2025. o3-mini System Card. https://openai.com/index/o3-mini-system- card/

    2025

  28. [36]

    Saswat Padhi, Rahul Sharma, and Todd Millstein. 2016. Data-driven precondition inference with learned features. ACM SIGPLAN Notices 51, 6 (2016), 42–56

    2016

  29. [37]

    Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics 12...

    2024

  30. [38]

    GNU Project. 2021. GNU Cflow - Free Software Foundation. https://www.gnu. org/software/cflow/

    2021

  31. [39]

    Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher

  32. [40]

    Niklas Risse and Marcel Böhme. 2024. Top score on the wrong exam: On benchmarking in machine learning for vulnerability detection. arXiv preprint arXiv:2408.12986 (2024)

    2024 arXiv

  33. [41]

    Niklas Risse and Marcel Böhme. 2024. Uncovering the limits of machine learn- ing for automatic vulnerability detection. In 33rd USENIX Security Symposium (USENIX Security 24). 4247–4264

    2024

  34. [42]

    Dan Ristea, Vasilios Mavroudis, and Chris Hicks. 2024. Benchmarking OpenAI o1 in Cyber Security. arXiv preprint arXiv:2410.21939 (2024)

    2024 arXiv

  35. [43]

    Ze Sheng, Zhicheng Chen, Shuning Gu, Heqing Huang, Guofei Gu, and Jeff Huang

  36. [44]

    Ze Sheng, Zhicheng Chen, Shuning Gu, Heqing Huang, Guofei Gu, and Jeff Huang. 2025. LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights. arXiv e-prints (2025), arXiv–2502

    2025

  37. [45]

    Steven A Sloman. 1996. The empirical case for two systems of reasoning. Psy- chological bulletin 119, 1 (1996), 3

    1996

  38. [46]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test- time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 (2024)

    2024 arXiv

  39. [47]

    Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Hengbo Tong, Swarna Das, Earl T Barr, and Wei Le. 2024. To Err is Machine: Vulnerability Detection Challenges LLM Reasoning. arXiv preprint arXiv:2403.17218 (2024)

    2024 arXiv

  40. [48]

    2012.Human reasoning and cognitive science

    Keith Stenning and Michiel Van Lambalgen. 2012.Human reasoning and cognitive science. MIT Press

    2012

  41. [49]

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Yang Liu, and Yingjiu Li. 2024. Llm4vuln: A unified evaluation framework for decoupling and enhancing llms’ vulnerability reasoning. arXiv preprint arXiv:2401.16185 (2024)

    2024 arXiv

  42. [50]

    Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784 (2024)

    2024 arXiv

  43. [51]

    Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringhini. 2024. Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks. In 2024 IEEE Symposium on Security and Priv...

    2024

  44. [52]

    Guoqing Wang, Zeyu Sun, Zhihao Gong, Sixiang Ye, Yizhou Chen, Yifan Zhao, Qingyuan Liang, and Dan Hao. 2024. Do advanced language models elimi- nate the need for prompt engineering in software engineering? arXiv preprint arXiv:2411.02093 (2024)

    2024 arXiv

  45. [53]

    Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. 2024. Reposvul: A repository-level high-quality vulnerability dataset. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engi- neering: Companion Proceedings. 472–483

    2024

  46. [54]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  47. [55]

    Xin-Cheng Wen, Xinchen Wang, Yujia Chen, Ruida Hu, David Lo, and Cuiyun Gao. 2024. Vuleval: Towards repository-level evaluation of software vulnerability detection. arXiv preprint arXiv:2404.15596 (2024)

    2024 arXiv

  48. [56]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems 36 (2023), 11809–11822

    2023

  49. [57]

    Alperen Yildiz, Sin G Teo, Yiling Lou, Yebo Feng, Chong Wang, and Dinil M Divakaran. 2025. Benchmarking LLMs and LLM-based Agents in Practical Vul- nerability Detection for Code Repositories.arXiv preprint arXiv:2503.03586 (2025)

    2025 arXiv

  50. [58]

    Xin Yin, Chao Ni, and Shaohua Wang. 2024. Multitask-based evaluation of open- source llm on software vulnerability. IEEE Transactions on Software Engineering (2024)

    2024

  51. [59]

    Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. 2024. Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023 (2024)

    2024 arXiv

  52. [60]

    Arastoo Zibaeirad and Marco Vieira. 2024. VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patching. arXiv preprint arXiv:2409.10756 (2024)

    2024 arXiv

  53. [61]

    HAS_VUL" if any are found. If no vulnerabilities are detected, respond with

    Arastoo Zibaeirad and Marco Vieira. 2025. Reasoning with LLMs for Zero-Shot Vulnerability Detection. arXiv preprint arXiv:2503.17885 (2025). Everything You Wanted to Know About LLM-based Vulnerability Detection But Were Afraid to Ask Conference’17, July 2017, Washington, DC, U...

    2025 arXiv

  54. [65]

    Patch Ignored

    Division `BufferSize / NumberOfPackets`:-If `NumberOfPackets` is zero, this would result in a division by zero.-The code does not check if `NumberOfPackets` is zero before performing the division. 1+if (NumberOfPackets > 0)2+{3iso_packet_size = BufferSize / NumberOfPackets;4is...

    2022

  55. [66]

    Minimum Reasoning

    `ut8 bundle_length= *(ut8 *)(bin->entry_table+ off);`:-The `bin->entry_table` pointer is dereferenced without checking if it is `NULL`. This could lead to a NULL pointer dereference.9. `ut8 bundle_type= *(ut8 *)(bin->entry_table+ off);`:-The `bin->entry_table` pointer is deref...

    2022

  56. [2019]

    arXiv preprint arXiv:1906.02361 (2019)

    Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361 (2019)

    2019 arXiv

  57. [2024]

    InProceedings of the 20th International Conference on Predictive Models and Data Analytics in Software Engineering

    MoreFixes: A large-scale dataset of CVE fix commits mined through enhanced repository discovery. InProceedings of the 20th International Conference on Predictive Models and Data Analytics in Software Engineering . 42–51

  58. [2025]

    arXiv preprint arXiv:2502.07049 (2025)

    Large Language Models in Software Security: A Survey of Vulnerability Detection Techniques and Insights. arXiv preprint arXiv:2502.07049 (2025)

    2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.