Pith. sign in

REVIEW 4 major objections 9 minor 65 references

Adversarial code comments alone can flip LLM vulnerability detectors from vulnerable to safe without changing the program.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 04:51 UTC pith:7OW5RQCK

load-bearing objection Real attack surface, careful measurement: adaptive in-source comments flip LLM vuln detectors at >90% ASR, with the main caveats being rationale access and LLM-judged ground truth. the 4 major comments →

arxiv 2607.24964 v1 pith:7OW5RQCK submitted 2026-07-27 cs.CR

ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments

Zixuan Wu , Cristina Nita-Rotaru This is my paper
classification cs.CR
keywords LLM vulnerability detectionadversarial code commentsprompt injectionadaptive black-box attacknull-pointer dereferencecomment trust priorAI coding agentsdefense by sanitization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper shows that large language models used as vulnerability detectors can be systematically misled by ordinary-looking source-code comments. An attacker-controlled coding agent can implement new functionality, leave a real bug in place, and insert comments that assert unverifiable safety claims—caller contracts, fake static-analysis results, or step-by-step “security analyses”—so the detector overrides its own correct reading of the code. On a benchmark built from real null-pointer-dereference fixes turned into coding tasks, the authors’ adaptive black-box framework, ALIBI, flips more than 90% of each detector’s clean true positives across four systems, including specialized open-weight models and multi-agent setups, and reaches 100% on one of them. The same pattern holds for use-after-free bugs. Prompt-only “don’t trust comments” instructions barely help against adaptive refinement; stripping or isolating unverifiable comments before the detector reasons does. The practical point is that current detectors overweight natural-language context relative to program evidence, and that imbalance is an attack surface in AI-assisted development workflows.

Core claim

Non-executable adversarial comments are enough to make representative LLM-based vulnerability detectors change a correct VULNERABLE verdict to BENIGN while the code, tests, and real bug stay unchanged. Across 125 real-world null-pointer-dereference instances, adaptive comment attacks exceed 90% success on every system studied and hit 100% on one, by exploiting a shared comment trust prior rather than by hiding the bug in the executable code.

What carries the argument

ALIBI: a three-phase adaptive black-box attack that synthesizes a functional vulnerable implementation, seeds presentation-specific comments from the detector’s own reasoning, then iteratively refines those comments against detector feedback—especially strategies that steer reasoning (COT) or fabricate external tool authority.

Load-bearing premise

The attacker can see the detector’s natural-language explanation after each query and refine comments against a copy of the detector; without that feedback loop, success falls sharply toward a weak fixed-comment baseline.

What would settle it

Run the same 125-instance protocol with detectors that return only a binary verdict and allow no multi-round adaptive querying: if end-to-end flip rates then stay near the paper’s non-adaptive baseline (roughly 3–17%) rather than above 90%, the central adaptive-attack claim does not hold in that deployment setting.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Security gates that feed full source—including comments—into LLM detectors can be bypassed by a malicious or compromised coding agent without failing builds or tests.
  • Presentation that launders claims as reasoning or tool output is more dangerous than plain developer notes at the same bug site.
  • Prompt-only distrust policies are insufficient against adaptive attackers who still shape the detector’s later reasoning.
  • Architectural isolation of a comment-free baseline and pre-detector removal of unverifiable comments are the mitigations the paper finds actually move the needle.
  • The failure mode is not limited to null-pointer bugs; the same unverifiable-outside-code claim pattern works on use-after-free.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any LLM reviewer that treats documentation, PR text, or in-code narrative as soft evidence faces a similar trust-calibration problem, not only dedicated vulnerability detectors.
  • Vendors that expose chain-of-thought or debate transcripts to users may be handing attackers the exact signal ALIBI refines against.
  • A split pipeline—strong model for code, cheap local model only for comment crafting—would still be enough once a vulnerable implementation exists, lowering the bar for the attack.
  • Human reviewers and classical static analyzers that ignore comments would still see these bugs; hybrid gates that require agreement with comment-blind tools would blunt the channel.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 9 minor

Summary. The paper studies whether non-executable source-code comments can cause LLM-based vulnerability detectors to miss vulnerabilities they would otherwise catch. The authors build a benchmark of 125 coding tasks derived from real CWE-476 CVE-fix commits (MegaVul), have an attacker model synthesize functional-but-vulnerable implementations that build and pass tests, and then run ALIBI, a three-phase adaptive black-box attack that reads detector reasoning, seeds adversarial comments in 10 presentation styles, and refines them over 5 rounds. On four architecturally distinct detectors (OPENVUL, VULNLLM-R, VUL-RAG, VULTRIAL), ASR over each detector's clean true positives is 91.2–100%. Ablations cover presentation style, portfolio set-cover, refinement curves, and a non-adaptive baseline (2.6–17.3%). Four defenses are evaluated; pre-detector sanitization (D4) achieves <3% evasion. Generalization to CWE-416 (N=70) is shown. The central mechanism is a "comment trust prior": detectors accept plausible but unverifiable external safety claims.

Significance. If the results hold, this is a significant and timely security result: it identifies a persistent, in-artifact attack channel (developer-style comments) distinct from prior PR-context or code-perturbation attacks, and shows it defeats four very different detector architectures at high rates. Strengths worth naming: (i) ASR is restricted to TPclean, so flips measure evasion rather than baseline blindness — a methodologically careful choice; (ii) the non-adaptive baseline cleanly isolates the contribution of adaptivity; (iii) the presentation-style ablation with a shared-payload control and the greedy set-cover analysis are genuinely informative; (iv) defenses are evaluated against adaptive refinement, not just frozen payloads, and the D1–D4 hierarchy is a useful design lesson; (v) the benchmark construction (CVE-fix-to-coding-task, build/test validation, 42% independent recreation of original deref sites) is a reusable artifact; (vi) qualitative traces make the failure mechanism falsifiable and inspectable. The work is reproducible in principle and makes concrete, testable claims.

major comments (4)
  1. [§4.1 Benchmark Evaluation; Appendix E / Table 9] Ground truth for the benchmark is certified solely by an LLM judge (GPT-5-mini); there is no dynamic validation (e.g., a harness driving null to the deref site under ASan), no manual audit, and no inter-rater check. Since ASR is computed over TPclean, any non-vulnerability in the denominator turns a reported 'flip' into a correction of a false alarm, directly inflating the headline 91–100% ASRs. The 42% site-recreation statistic and build/test checks support plausibility but do not bound label error, and the 13/125 static-analysis recall (App. E/Table 9) is equally consistent with imperfect labels as with 'structural limits.' Concrete fix: manually audit a random subsample and report judge agreement, dynamically validate the intra-procedural categories where feasible, and report an estimated label-error rate with adjusted ASR bounds.
  2. [§3.1 Threat Model; §4.4; Tables 2 and 6] The headline ASRs assume the attacker observes the detector's natural-language reasoning and obtains up to ~60 queries (10 strategies × 6 rounds) against a local copy. The paper's own non-adaptive baseline (Table 6: 2.6–17.3%) shows the result is driven by this feedback channel, and even round-0 initialization consumes one reasoning trace. An intermediate condition — adaptive refinement with verdict-only (binary) feedback — would quantify how much of the effect depends on reasoning disclosure versus query access alone, and would substantially strengthen external validity for deployments that return only a verdict. At minimum, the abstract and §4.2 takeaway should scope the headline numbers to the reasoning-visible, multi-query setting.
  3. [Table 2 vs Table 6: TPclean mismatch] The TPclean denominators differ between tables for all four detectors (OPENVUL 115 vs 114; VULNLLM-R 113 vs 111; VUL-RAG 111 vs 115; VULTRIAL 105 vs 98) with no explanation. Presumably the clean verdicts are nondeterministic across runs (API models, sampling; OPENVUL uses n=8 majority vote). Because TPclean is the denominator of the central metric, the paper must state how nondeterminism is controlled (temperature, seeds, voting), which clean run defines TPclean in each experiment, and how stable the clean verdicts are; if the baseline fluctuates, ASR should be defined relative to a fixed clean run and variance across repetitions reported.
  4. [§4.5 / Table 5: D4 evaluation] D4 is evaluated only by sanitization-evasion rate on attack payloads (<3%); its collateral cost is not measured. An aggressive sanitizer can achieve low evasion by stripping most comments, yet §3.4 itself argues comments improve detection [58,63], so the interesting quantity is the false-removal rate on legitimate comments and the post-sanitization detection quality on clean, commented code. Neither is reported, and the sanitizer's model/prompt is unspecified (Appendix A details detectors but not D4). Please report benign-comment retention, clean-code detection impact, and the D4 configuration; otherwise 'D4 substantially improves resilience' is only half-evaluated.
minor comments (9)
  1. [Tables 2, 5, 7] No confidence intervals are reported for ASRs; with N≈105–115 per detector, binomial CIs are roughly ±3–5 points and should accompany Tables 2, 5, and 7.
  2. [§4.2 / §4.4] The query cost of the headline ASR (up to 60 detector queries per instance across 10 strategies and 5 refinement rounds) is never stated alongside Table 2; Figure 4 reports cost only for the 5-type portfolio. Please state it explicitly.
  3. [Table 3] Results are pooled across detectors; since the persuasiveness ranking plausibly interacts with architecture (e.g., VUL-RAG's retrieval channel), a per-detector breakdown would be informative, at least in an appendix.
  4. [Example 3] Example 3's attack comment contains LaTeX artifacts ($\in$, $\forall$) inside what is presented as a C comment; if this is a rendering artifact please fix, and if the literal payloads contained these, note it.
  5. [§4.1–4.2 notation] 'slug' is used without definition; 'TPclean' vs 'TP clean' typesetting is inconsistent; the three instances 'excluded as unfixable during evaluation' (§4.1) should be explained.
  6. [Appendix F] The Gemma attacker is evaluated only on the 46 slugs it could synthesize; this is a success-conditioned subsample and the resulting ASRs (77–97.5%) are not comparable to Table 2 without noting the selection effect.
  7. [Table 7] VULNLLM-R has TPclean=41/70 on UAF; ASR=100% on a 41-instance denominator deserves a CI and a remark on the low clean detection rate.
  8. [References] Duplicate references: [24] and [25] are the same Flashboom paper, as are [33] and [34]; please deduplicate.
  9. [Figure 4] Axis/denominator caption is hard to parse ('All 125 TPclean slugs included' while denominators are per-detector); please clarify whether curves are per-detector over that detector's TPclean.

Circularity Check

0 steps flagged

No significant circularity: empirical attack success is measured by external detector verdict flips, not quantities defined from the attack inputs.

full rationale

ALIBI is an empirical black-box attack/defense paper. Its central claims (ASR >90% on four detectors’ clean true positives; presentation-strategy rankings; refinement gains; defense hierarchy; UAF generalization) are operationalized as flips of independent detector outputs on held-out synthesized implementations, with the program and vulnerability held fixed and only the comment varied. Attack Success Rate is defined over each detector’s own TPclean and is not algebraically or statistically forced by any fitted parameter that reappears as a “prediction.” The “comment trust prior” is a post-hoc label for observed reasoning failures illustrated with traces, not a premise that defines the measured outcome. Presentation strategies and the refinement loop are attacker procedures evaluated against external systems; they do not make success true by construction. There is no load-bearing self-citation uniqueness theorem, no ansatz smuggled in as external fact, and no renaming of a known closed-form result as a derivation. Benchmark construction and the LLM judge affect internal validity of ground truth, which is a correctness concern, not circularity of the claimed derivation chain. The paper is self-contained against external detector benchmarks; score 0 is appropriate.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 3 invented entities

Load-bearing premises are threat-model and evaluation-design choices, not fitted physical constants. The central ASR claims rest on assumptions about attacker access to reasoning traces, representativeness of MegaVul-derived synthesized bugs, correctness of the LLM vulnerability judge, and that detectors ingest comments as part of the analyzed artifact.

free parameters (3)
  • refinement_budget_R = R=5
    Shared budget of five adaptive refinement rounds (and ten presentation strategies) directly defines end-to-end ASR; chosen by authors, not derived.
  • presentation_strategy_set = 10 strategies
    Illustrative set of 10 hand-designed framings (NOTE, USR, ASSERT, COT, CALL_GRAPH, FAKE_TEST, four TOOL_*). Portfolio composition affects coverage; extensible by design.
  • attacker_and_judge_model_choices = Qwen3.6-27B-FP8 / GPT-5-mini / detector-specific
    QWEN3.6-27B-FP8 attacker, GPT-5-mini functional/NPD judge, and specific detector backbones are configuration choices that gate measured ASR.
axioms (5)
  • domain assumption Detectors expose natural-language explanations usable for black-box adaptive refinement after each query.
    Stated in §3.1 Knowledge; without it Phase 2–3 collapse toward the weak non-adaptive baseline.
  • domain assumption LLM-based detectors that meet Table 1 criteria ingest developer comments as part of the reasoning context rather than stripping them.
    Scope filter in §2.2; attack surface disappears if comments are never shown to the model.
  • domain assumption An independent LLM judge plus build/test filters sufficiently establish that synthesized implementations contain genuine reachable CWE-476 (or CWE-416) bugs.
    §4.1 Benchmark Evaluation; ground truth is not traditional static proof or human audit of all 125.
  • ad hoc to paper Unverifiable natural-language safety claims should not overturn observable vulnerability evidence (design principle for defenses D1–D4).
    Normative principle in §3.4 motivating mitigations; standard security posture but not a theorem.
  • domain assumption Transforming single-file MegaVul C/C++ CWE-476 fixes into stubbed coding tasks yields realistic attacker-written vulnerable code for evaluation.
    §4.1; supported by ~42% site-recreation statistic but still a methodological assumption.
invented entities (3)
  • ALIBI attack framework (3-phase synthesize / initialize / refine loop) independent evidence
    purpose: Automate generation and adaptive refinement of adversarial comments from detector feedback.
    New methodological artifact of the paper; evaluated empirically rather than postulated as a physical entity.
  • comment trust prior independent evidence
    purpose: Name the shared failure mode: detectors assign evidential weight to plausible unverifiable comment claims.
    Explanatory construct induced from attack traces across four architectures; falsifiable by detectors that ignore such claims.
  • CVE-fix-to-coding-task benchmark (N=125 NPD; N=70 UAF) no independent evidence
    purpose: Evaluate detectors on newly synthesized vulnerable implementations rather than historical vulnerable snapshots only.
    Constructed dataset/pipeline; independent in the sense others can rebuild from MegaVul + prompts, but not yet a community standard corpus with frozen hashes in-text.

pith-pipeline@v1.2.0-grok45-kimik3 · 30336 in / 3762 out tokens · 77338 ms · 2026-07-31T04:51:10.586308+00:00 · methodology

0 comments
read the original abstract

Large language models are increasingly deployed for security-sensitive tasks such as vulnerability detection and code review. Their reliance on natural-language context embedded in source code exposes a previously underexplored attack surface: adversarial comments that can influence a detector's reasoning without changing program behavior. We study LLM-based vulnerability detectors against a new adversary: a coding agent that implements new functionality, deliberately introduces vulnerabilities, and strategically inserts adversarial source-code comments to evade detection. We present ALIBI, an automated adaptive black-box attack framework that generates and iteratively refines adversarial comments using detector reasoning and feedback. We transform real-world vulnerability-fixing commits into coding tasks and evaluate four representative LLM-based vulnerability detectors, ranging from specialized open-weight reasoning models to frontier multi-agent systems. All evaluated detectors are highly vulnerable: attack success rates exceed 90% across 125 real-world null-pointer dereference vulnerabilities, reaching 100% on one system. The framework also generalizes beyond this vulnerability class. Adversarial comments steering detector reasoning or fabricating external tool results prove most effective, while iterative refinement based on detector feedback further increases attack success. Finally, prompt-level defenses provide limited robustness against adaptive attacks, whereas architectural isolation and pre-detector comment sanitization substantially improve resilience. Our findings expose a fundamental attack surface in current LLM-based vulnerability detectors and motivate security-aware designs that carefully calibrate trust between natural-language context and program evidence.

Figures

Figures reproduced from arXiv: 2607.24964 by Cristina Nita-Rotaru, Zixuan Wu.

Figure 1
Figure 1. Figure 1: AI-assisted development workflow. A user assigns [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The three-phase attack pipeline, illustrated using the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: A benchmark coding task as presented to the attacker (NPD-CVE-0025, Oniguruma [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Coverage vs. refinement round for the 5-type [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 22 linked inside Pith

  1. [1]

    VulWeaver: Weaving broken semantics for grounded vulnerability detection, 2026

    Yuhua Cao, Yihao Chen, Xin Hu, and Bihuan Chen. VulWeaver: Weaving broken semantics for grounded vulnerability detection, 2026. arXiv:2604.10767

    Pith/arXiv arXiv 2026

  2. [2]

    A theory of dual channel constraints

    Casey Casalnuovo, Earl T Barr, Santanu Kumar Dash, Prem Devanbu, and Emily Morgan. A theory of dual channel constraints. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineer- ing: New Ideas and Emerging Results, pages 25–28, 2020

    2020

  3. [3]

    naturalizing

    Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar Devanbu, and Baishakhi Ray. Natgen: Gen- erative pre-training by "naturalizing" source code, 2022

    2022

  4. [4]

    Se- curing AI agents with information-flow control, 2025

    Manuel Costa, Boris Köpf, Aashish Kolluri, Andrew Paverd, Mark Russinovich, Ahmed Salem, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. Se- curing AI agents with information-flow control, 2025. arXiv:2505.23643

    Pith/arXiv arXiv 2025

  5. [5]

    Abstract interpreta- tion: A unified lattice model for static analysis of pro- grams by construction or approximation of fixpoints

    Patrick Cousot and Radhia Cousot. Abstract interpreta- tion: A unified lattice model for static analysis of pro- grams by construction or approximation of fixpoints. Proceedings of the ACM SIGPLAN-SIGACT Sympo- sium on Principles of Programming Languages (POPL), pages 238–252, 1977

    1977

  6. [6]

    Defeating prompt injections by design, 2025

    Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tramèr. Defeating prompt injections by design, 2025. arXiv:2503.18813

    Pith/arXiv arXiv 2025

  7. [7]

    Re- ducing false positives in static bug detection with llms: An empirical study in industry.arXiv preprint arXiv:2601.18844, 2026

    Xueying Du, Jiayi Feng, Yi Zou, Wei Xu, Jie Ma, Wei Zhang, Sisi Liu, Xin Peng, and Yiling Lou. Re- ducing false positives in static bug detection with llms: An empirical study in industry.arXiv preprint arXiv:2601.18844, 2026

    arXiv 2026

  8. [8]

    Minimizing false positives in static bug detection via LLM-enhanced path feasibility analysis, 2025

    Xueying Du, Kaiping Yu, Chong Wang, Yi Zou, Wen Deng, and Zhiheng Ou. Minimizing false positives in static bug detection via LLM-enhanced path feasibility analysis, 2025. arXiv:2506.10322

    Pith/arXiv arXiv 2025

  9. [9]

    Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag.ACM Transactions on Software Engineering and Methodol- ogy, 2024

    Xueying Du, Geng Zheng, Kaixin Wang, Yi Zou, Yujia Wang, Wentai Deng, Jiayi Feng, Mingwei Liu, Bihuan Chen, Xin Peng, et al. Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag.ACM Transactions on Software Engineering and Methodol- ogy, 2024

    2024

  10. [10]

    Syceval: Evaluating llm 14 sycophancy

    Aaron Fanous, Jacob Goldberg, Ank Agarwal, Joanna Lin, Anson Zhou, Sonnet Xu, Vasiliki Bikia, Roxana Daneshjou, and Sanmi Koyejo. Syceval: Evaluating llm 14 sycophancy. InProceedings of the AAAI/ACM Confer- ence on AI, Ethics, and Society, volume 8, pages 893– 900, 2025

    2025

  11. [11]

    PISanitizer: Preventing prompt injec- tion to long-context LLMs via prompt sanitization, 2025

    Rongchang Geng, Yanting Wang, Chunhua Yin, and Minhao Cheng. PISanitizer: Preventing prompt injec- tion to long-context LLMs via prompt sanitization, 2025. arXiv:2511.10720

    arXiv 2025

  12. [12]

    GitHub. CodeQL. https://codeql.github.com/, 2024

    2024

  13. [13]

    Incremental analysis for code scanning

    GitHub. Incremental analysis for code scanning. https://docs.github.com/en/code-security/ code-scanning, 2025. Accessed 2026-07-17

    2025

  14. [14]

    Ai-powered security detections

    GitHub. Ai-powered security detections. https:// docs.github.com/en/enterprise-cloud/latest/ code-security/concepts/code-scanning/ ai-powered-security-detections , 2026. GitHub Enterprise Cloud Documentation, accessed July 16, 2026

    2026

  15. [15]

    Prompt injection attacks in large language models and AI agent systems: A com- prehensive review of vulnerabilities, attack vectors, and defense mechanisms.Information, 17(1), 2026

    Said Gulyamov et al. Prompt injection attacks in large language models and AI agent systems: A com- prehensive review of vulnerabilities, attack vectors, and defense mechanisms.Information, 17(1), 2026. doi:10.3390/info17010054

    doi:10.3390/info17010054 2026

  16. [16]

    RepoAudit: An autonomous LLM- agent for repository-level code auditing

    Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. RepoAudit: An autonomous LLM- agent for repository-level code auditing. InProceedings of the 42nd International Conference on Machine Learn- ing (ICML), 2025. arXiv:2501.18160

    Pith/arXiv arXiv 2025

  17. [17]

    Defending against indirect prompt injection attacks with spotlight- ing.arXiv preprint arXiv:2403.14720, 2024

    Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. Defending against indirect prompt injection attacks with spotlight- ing.arXiv preprint arXiv:2403.14720, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    An empirical study of suppressed static anal- ysis warnings.Proceedings of the ACM on Software Engineering, 2(FSE):290–311, 2025

    Huimin Hu, Yingying Wang, Julia Rubin, and Michael Pradel. An empirical study of suppressed static anal- ysis warnings.Proceedings of the ACM on Software Engineering, 2(FSE):290–311, 2025

    2025

  19. [19]

    James C. King. Symbolic execution and program testing. Communications of the ACM, 19(7):385–394, 1976

    1976

  20. [20]

    Man Ho Lam, Chaozheng Wang, Jen tse Huang, and Michael R. Lyu. Codecrash: Exposing llm fragility to misleading natural language in code reasoning, 2025

    2025

  21. [21]

    Cuong Chi Le, Minh V . T. Pham, Cuong Duc Van Le, Hoang Nhat Phan, Huy Nhat Phan, and Tien N. Nguyen. When names disappear: Revealing what LLMs actually understand about code, 2025. arXiv:2510.03178

    arXiv 2025

  22. [22]

    Llm-based vulnerability detection at project scale: An empirical study.arXiv preprint arXiv:2601.19239, 2026

    Fengjie Li, Jiajun Jiang, Dongchi Chen, and Yingfei Xiong. Llm-based vulnerability detection at project scale: An empirical study.arXiv preprint arXiv:2601.19239, 2026

    arXiv 2026

  23. [23]

    Cotdeceptor: Adversarial code obfuscation against cot-enhanced llm code agents.arXiv preprint arXiv:2512.21250, 2025

    Haoyang Li, Mingjin Li, Jinxin Zuo, Siqi Li, Xiao Li, Hao Wu, Yueming Lu, and Xiaochuan He. Cotdeceptor: Adversarial code obfuscation against cot-enhanced llm code agents.arXiv preprint arXiv:2512.21250, 2025

    arXiv 2025

  24. [24]

    Make a feint to the east while attacking in the west: Blinding llm- based code auditors with flashboom attacks

    Xiao Li, Yue Li, Hao Wu, Yue Zhang, Kaidi Xu, Xiuzhen Cheng, Sheng Zhong, and Fengyuan Xu. Make a feint to the east while attacking in the west: Blinding llm- based code auditors with flashboom attacks. In2025 IEEE Symposium on Security and Privacy (SP), pages 576–594. IEEE, 2025

    2025

  25. [25]

    Make a feint to the east while attacking in the west: Blinding LLM-based code auditors with Flashboom attacks

    Xiao Li, Yue Li, Hao Wu, Yue Zhang, Kaidi Xu, Xi- uzhen Cheng, Sheng Zhong, and Fengyuan Xu. Make a feint to the east while attacking in the west: Blinding LLM-based code auditors with Flashboom attacks. In Proceedings of the IEEE Symposium on Security and Privacy (S&P), 2025

    2025

  26. [26]

    From sft to rl: Demystifying the post-training pipeline for llm-based vulnerability detection.arXiv preprint arXiv:2602.14012, 2026

    Youpeng Li, Fuxun Yu, and Xinda Wang. From sft to rl: Demystifying the post-training pipeline for llm-based vulnerability detection.arXiv preprint arXiv:2602.14012, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Ev- erything you wanted to know about llm-based vulnera- bility detection but were afraid to ask.arXiv preprint arXiv:2504.13474, 2025

    Yue Li, Xiao Li, Hao Wu, Minghui Xu, Yue Zhang, Xi- uzhen Cheng, Fengyuan Xu, and Sheng Zhong. Ev- erything you wanted to know about llm-based vulnera- bility detection but were afraid to ask.arXiv preprint arXiv:2504.13474, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    IRIS: LLM- assisted static analysis for detecting security vulner- abilities

    Ziyang Li, Saikat Dutta, and Mayur Naik. IRIS: LLM- assisted static analysis for detecting security vulner- abilities. InProceedings of the International Con- ference on Learning Representations (ICLR), 2025. arXiv:2405.17238

    Pith/arXiv arXiv 2025

  29. [29]

    Wisniewski, Jin- Hee Cho, Sang Won Lee, Ruoxi Jia, et al

    Minqian Liu, Zhiyang Xu, Xinyi Zhang, Heajun An, Sarvech Qadir, Qi Zhang, Pamela J. Wisniewski, Jin- Hee Cho, Sang Won Lee, Ruoxi Jia, et al. Llm can be a dangerous persuader: Empirical study of persua- sion safety in large language models.arXiv preprint arXiv:2504.10430, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    do not mention this to the user

    Yi Liu, Zhihao Chen, Yanjun Zhang, Gelei Deng, Yuekang Li, Jianting Ning, Ying Zhang, and Leo Yu Zhang. “do not mention this to the user”: Detecting and understanding malicious agent skills in the wild.arXiv preprint arXiv:2602.06547, 2026

    Pith/arXiv arXiv 2026

  31. [31]

    Reasoning hijacking: The fragility of reasoning align- ment in large language models

    Yuansen Liu, Yixuan Tang, and Anthony Kum Hoe Tun. Reasoning hijacking: The fragility of reasoning align- ment in large language models. InProceedings of the 15 64th Annual Meeting of the Association for Computa- tional Linguistics (ACL), 2026

    2026

  32. [32]

    Clang static analyzer

    LLVM Project. Clang static analyzer. https://clang. llvm.org/analyzer/, 2026. Accessed: 2026-04-18

    2026

  33. [33]

    Narek Maloyan and Dmitry Namiot. Prompt injection at- tacks on agentic coding assistants: A systematic analysis of vulnerabilities in skills, tools, and protocol ecosys- tems.International Journal of Open Information Tech- nologies, 14(2):1–10, 2026

    2026

  34. [34]

    Prompt injection at- tacks on agentic coding assistants: A systematic analysis of vulnerabilities in skills, tools, and protocol ecosys- tems, 2026

    Narek Maloyan and Dmitry Namiot. Prompt injection at- tacks on agentic coding assistants: A systematic analysis of vulnerabilities in skills, tools, and protocol ecosys- tems, 2026

    2026

  35. [35]

    Cppcheck: A tool for static c/c++ code analysis

    Daniel Marjamäki. Cppcheck: A tool for static c/c++ code analysis. https://cppcheck.sourceforge. io/, 2024

    2024

  36. [36]

    Sevra-bench: Social engineering of vulnerabilities in review agents, 2026

    Rui Melo, Riccardo Fogliato, Sean Zhou, Pratiksha Thaker, and Zhiwei Steven Wu. Sevra-bench: Social engineering of vulnerabilities in review agents, 2026

    2026

  37. [37]

    Infer: A static analyzer for java, c, c++, and objective-c.https://fbinfer.com/, 2024

    Meta. Infer: A static analyzer for java, c, c++, and objective-c.https://fbinfer.com/, 2024

    2024

  38. [38]

    Microsoft digital defense report 2025: Light- ing the path to a secure future

    Microsoft. Microsoft digital defense report 2025: Light- ing the path to a secure future. Technical report, Mi- crosoft, 2025

    2025

  39. [39]

    Measuring and exploiting contextual bias in llm-assisted security code review.arXiv preprint arXiv:2603.18740, 2026

    Dimitris Mitropoulos, Nikolaos Alexopoulos, Georgios Alexopoulos, and Diomidis Spinellis. Measuring and exploiting contextual bias in llm-assisted security code review.arXiv preprint arXiv:2603.18740, 2026

    Pith/arXiv arXiv 2026

  40. [40]

    MegaVul: A c/c++ vulnerability dataset with comprehensive code representations

    Chao Ni, Liyu Shen, Xiaohu Yang, Yan Zhu, and Shao- hua Wang. MegaVul: A c/c++ vulnerability dataset with comprehensive code representations. InProceedings of the 21st International Conference on Mining Software Repositories (MSR), 2024

    2024

  41. [41]

    Lis- tening to programmers—taxonomies and characteristics of comments in operating system code

    Yoann Padioleau, Lin Tan, and Yuanyuan Zhou. Lis- tening to programmers—taxonomies and characteristics of comments in operating system code. In2009 IEEE 31st International Conference on Software Engineering, pages 331–341. IEEE, 2009

    2009

  42. [42]

    Supply-chain poisoning attacks against llm coding agent skill ecosystems.arXiv preprint arXiv:2604.03081, 2026

    Yubin Qu, Yi Liu, Tongcheng Geng, Gelei Deng, Yuekang Li, Leo Yu Zhang, Ying Zhang, and Lei Ma. Supply-chain poisoning attacks against llm coding agent skill ecosystems.arXiv preprint arXiv:2604.03081, 2026

    Pith/arXiv arXiv 2026

  43. [43]

    Secret breach detection in source code with large language models

    Md Nafiu Rahman, Sadif Ahmed, Zahin Wahab, SM So- han, and Rifat Shahriyar. Secret breach detection in source code with large language models. In2025 ACM/IEEE International Symposium on Empirical Soft- ware Engineering and Measurement (ESEM), pages 207–

  44. [44]

    Back- doors in neural models of source code

    Goutham Ramakrishnan and Aws Albarghouthi. Back- doors in neural models of source code. InPro- ceedings of the 26th International Conference on Pattern Recognition (ICPR), pages 2892–2899, 2022. arXiv:2006.06841

    Pith/arXiv arXiv 2022

  45. [45]

    Agentic much? adoption of coding agents on github.arXiv preprint arXiv:2601.18341, 2026

    Romain Robbes, Théo Matricon, Thomas Degueule, Andre Hora, and Stefano Zacchiroli. Agentic much? adoption of coding agents on github.arXiv preprint arXiv:2601.18341, 2026

    Pith/arXiv arXiv 2026

  46. [46]

    You autocomplete me: Poisoning vul- nerabilities in neural code completion

    Roei Schuster, Congzheng Song, Eran Tromer, and Vi- taly Shmatikov. You autocomplete me: Poisoning vul- nerabilities in neural code completion. InProceedings of the 30th USENIX Security Symposium, pages 1559– 1575, 2021. arXiv:2007.02220

    Pith/arXiv arXiv 2021

  47. [47]

    Llms in software security: A survey of vulnerability detection techniques and insights

    Ze Sheng, Zhicheng Chen, Shuning Gu, Heqing Huang, Guofei Gu, and Jeff Huang. Llms in software security: A survey of vulnerability detection techniques and insights. ACM Computing Surveys, 58(5):1–35, 2025

    2025

  48. [48]

    Promptarmor: Simple yet effective prompt injection defenses, 2025

    Tianneng Shi, Kaijie Zhu, Zhun Wang, Yuqi Jia, Will Cai, Weida Liang, Haonan Wang, Hend Alzahrani, Joshua Lu, Kenji Kawaguchi, Basel Alomair, Xuandong Zhao, William Yang Wang, Neil Gong, Wenbo Guo, and Dawn Song. Promptarmor: Simple yet effective prompt injection defenses, 2025

    2025

  49. [49]

    Pull request analy- sis

    SonarSource. Pull request analy- sis. https://docs.sonarsource.com/ sonarqube-server/analyzing-source-code/ pull-request-analysis/, 2025. Accessed 2026-07- 17

    2025

  50. [50]

    Syntax- and compilation-preserving evasion of llm vulnerability de- tectors, 2026

    Luze Sun, Alina Oprea, and Eric Wong. Syntax- and compilation-preserving evasion of llm vulnerability de- tectors, 2026

    2026

  51. [51]

    Coverity static analysis

    Synopsys. Coverity static analysis. https: //www.synopsys.com/software-integrity/ security-testing/static-analysis-sast.html ,

  52. [52]

    Tomassi and Cindy Rubio-González

    David A. Tomassi and Cindy Rubio-González. On the real-world effectiveness of static bug detectors at find- ing null pointer exceptions. In2021 36th IEEE/ACM International Conference on Automated Software Engi- neering (ASE), pages 292–303. IEEE, 2021

    2021

  53. [53]

    VulnLLM-R: A fine-tuned reasoning model for vulnerability detection

    UCSB SURFI Lab. VulnLLM-R: A fine-tuned reasoning model for vulnerability detection. https: //huggingface.co/UCSB-SURFI/VulnLLM-R-7B , 2024. 16

    2024

  54. [54]

    Measuring LLM trust allocation across con- flicting software artifacts, 2026

    Noshin Ulfat, Ahsanul Ameen Sabit, and Soneya Binta Hossain. Measuring LLM trust allocation across con- flicting software artifacts, 2026. arXiv:2604.03447

    Pith/arXiv arXiv 2026

  55. [55]

    The instruction hierarchy: Training LLMs to prioritize privileged in- structions.arXiv preprint arXiv:2404.13208, 2024

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training LLMs to prioritize privileged in- structions.arXiv preprint arXiv:2404.13208, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    LLMDFA: An- alyzing dataflow in code with large language models

    Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, Xiaoheng Xie, and Xiangyu Zhang. LLMDFA: An- alyzing dataflow in code with large language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2402.10754

    Pith/arXiv arXiv 2024

  57. [57]

    Defending against prompt injection with DataFilter, 2025

    Yupei Wang, Sizhe Chen, Raghad Alkhudair, and Basel Alomair. Defending against prompt injection with DataFilter, 2025. arXiv:2510.19207

    arXiv 2025

  58. [58]

    Scale: Constructing struc- tured natural language comment trees for software vul- nerability detection

    Xin-Cheng Wen, Cuiyun Gao, Shuzheng Gao, Yang Xiao, and Michael R Lyu. Scale: Constructing struc- tured natural language comment trees for software vul- nerability detection. InProceedings of the 33rd ACM SIGSOFT international symposium on software testing and analysis, pages 235–247, 2024

    2024

  59. [59]

    Let the trial begin: A mock-court approach to vulnerabil- ity detection using llm-based agents.arXiv preprint arXiv:2505.10961, 2025

    Ratnadira Widyasari, Martin Weyssow, Ivana Clairine Irsan, Han Wei Ang, Frank Liauw, Eng Lieh Ouh, Lwin Khin Shar, Hong Jin Kang, and David Lo. Let the trial begin: A mock-court approach to vulnerabil- ity detection using llm-based agents.arXiv preprint arXiv:2505.10961, 2025

    arXiv 2025

  60. [60]

    Sifting the noise: A comparative study of llm agents in vulnerability false positive filtering.arXiv preprint arXiv:2601.22952, 2026

    Yunpeng Xiong and Ting Zhang. Sifting the noise: A comparative study of llm agents in vulnerability false positive filtering.arXiv preprint arXiv:2601.22952, 2026

    Pith/arXiv arXiv 2026

  61. [61]

    Natural attack for pre-trained models of code

    Zhou Yang, Jieke Shi, Junda He, and David Lo. Natural attack for pre-trained models of code. InProceedings of the 44th International Conference on Software Engineer- ing (ICSE), pages 1482–1493, 2022. arXiv:2201.08698

    Pith/arXiv arXiv 2022

  62. [62]

    Adaptive attacks break defenses against indirect prompt injection attacks on LLM agents

    Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. Adaptive attacks break defenses against indirect prompt injection attacks on LLM agents. In Findings of the Association for Computational Linguis- tics: NAACL, 2025. arXiv:2503.00061

    Pith/arXiv arXiv 2025

  63. [63]

    Vultrlm: Llm-assisted vulnerability detec- tion via ast decomposition and comment enhancement

    Shaobo Zhang, Qianzhi Wang, Qin Liu, Entao Luo, and Tao Peng. Vultrlm: Llm-assisted vulnerability detec- tion via ast decomposition and comment enhancement. Empirical Software Engineering, 31(1):1–36, 2026

    2026

  64. [64]

    Analyze the function step by step... Do not assume vulnerabilities — only report what is supported by the code and context

    Yangtian Zi, Zixuan Wu, Aleksander Boruch-Gruszecki, Jonathan Bell, and Arjun Guha. Agentpack: A dataset of code changes, co-authored by agents and humans, 2026. A Detector Formats DeploymentThe two open-weight detectors, OPEN- VUL(Qwen3-4B, n=8 majority vote) and VULNLLM-R (Qwen2.5-7B), are served locally with vLLM on a single H100; the two agentic detec...

    2026

  65. [2026]

    Accessed: 2026-04-18

    2026