Pith. sign in

REVIEW 3 major objections 5 minor 18 references

VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VulnGym claims repository-level vulnerability detection cannot be judged by labels alone, and shows that the best current coding agent detects fewer than a quarter of hard advisories and builds supporting traces that barely match ground tru

desk verdict A well-designed benchmark with real potential, but the headline numbers are not yet defensible: the leakage control likely fails and the artifact is unreleased. read the letter →

arxiv 2608.02001 v1 pith:FG75AS3I submitted 2026-08-03 cs.SE

Kexing Ji , Jiachen Liu , Enze Hu , Cuiyun Gao , Keke Lian , Yongheng Liu , Lei Zhang , Tian Dong
show 2 more authors
Hao Chen Wang Bin
This is my paper
classification cs.SE
keywords vulnerabilitydetectioncodingagentsrepository-levelbenchmarkLLMevaluationGitHubadvisoriestracesentrypointscriticaloperations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VulnGym aims to show that repository-level vulnerability detection by coding agents cannot be judged by labels alone: a detector can name the right function while missing the actual defect or citing irrelevant evidence. The paper builds 408 human-verified vulnerability entries from 184 GitHub advisories across 23 repositories, each annotated with an entry point, a critical operation, and an ordered trace between them. It then evaluates coding agents end-to-end and with oracle subtasks, and reports that the best configuration finds fewer than a quarter of hard advisories and produces traces that barely resemble ground truth. The point of the benchmark is to turn failure into diagnosis: to show whether an agent fails to find the entry point, misses the critical operation, or loses the chain of evidence connecting them.

What carries the argument

The atomic unit is the reachable-entry record, written EP → Trace → CO: an externally reachable entry point (API handler, event callback), a security-critical operation (unsafe sink, missing check), and an ordered, cross-file execution path connecting them. Each record is generated from data-flow and control-flow analysis and then human-verified in an iterative loop with a reported Cohen's kappa of 0.87. The EP-Trace-CO triple does the work: it turns a yes/no detection question into a line-level, path-level diagnosis, and it defines the T1-T4 evaluation tasks by withholding or providing components as oracles.

What would settle it

Ask any evaluated model to identify the GHSA ID, vulnerable file path, or patch lines for a random sample of the 184 advisories when given only the advisory title and description. If the model supplies correct details above chance, the benchmark's low recall numbers reflect memorization rather than repository exploration, and the empirical conclusion collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms: repository-level vulnerability detection is not a classification problem but an evidence-construction problem. An agent's output counts only when it hits both the externally reachable entry point and the security-critical operation, and its supporting trace must match the reference path. Under that standard, the strongest evaluated agent (OpenHands with DeepSeek-V4-Flash) achieves 22.58% advisory recall and 15.22% entry recall on the Hard subset, with 11.63% trace edit similarity. Oracle experiments show that even when one endpoint is supplied, agents localize critical operations more readily than entry points, and even with both endpoints given, trace construction

Load-bearing premise

The load-bearing premise is that the evaluated LLMs never saw the November 2025–April 2026 advisories or their fix commits; the paper selects the window to avoid leakage but runs no probe to confirm it.

Editorial extensions

If this is right

  • End-to-end vulnerability detection under this metric is far below deployment-ready: even the best agent leaves more than three quarters of hard advisories undetected.
  • Entry-point identification is a weaker capability than critical-operation localization; oracle experiments put the best EP file recall at 30.23% versus 51.16% for COs.
  • Trace construction is a separable bottleneck: with oracle endpoints, best edit similarity is 36.37% and trace node precision at most 26.71%, so agents recover nodes but include many wrong ones.
  • Model scaling helps but saturates; within one family only the 27B model scores nonzero on Hard, while a 4B model can beat a 9B model on Easy.
  • Agent scaffold choice matters as much as model choice: DeepSeek-V4-Flash ranges from 9.68% to 22.58% advisory recall across scaffolds, and the ordering of models reverses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark restricts agents to read-only commands, so it measures static navigation and reasoning only; permitting test execution or dynamic analysis might change the trace and endpoint picture, a testable extension the paper does not run.
  • The contamination hypothesis is testable: because all advisories fall in a window claimed to be after model training cutoffs, a direct probe of whether models recall advisory IDs or patch diffs would tell whether the low numbers are exploration failures or memorization effects.
  • The same EP-Trace-CO annotations could be reused as a training signal, e.g., to reward agents that terminate search at the critical operation or to supervise trace generation; the paper does not propose this but the data format invites it.
  • Since edit similarity uses Levenshtein distance on line lists, it may understate the quality of traces that are semantically correct but reordered; a variant metric tolerating small reorderings would give a more complete picture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces VulnGym, a repository-level vulnerability detection benchmark constructed from 184 GitHub advisories and 408 human-verified entries, each annotated with line-level entry points, critical operations, and vulnerability traces across 23 repositories. The benchmark defines an end-to-end detection task (T1) and three oracle-conditioned subtasks (T2–T4) that isolate entry-point identification, critical-operation localization, and trace construction. Using explicit line-level metrics, the authors evaluate four LLM families (Qwen3.5, DeepSeek-V4-Flash, GLM-5.2, MiniMax-M3) under three agentic scaffolds (Claude Code, OpenHands, MiniSWE). The main empirical finding is that current coding agents detect few Hard vulnerabilities: the best configuration (OpenHands + DeepSeek-V4-Flash) reaches 22.58% advisory recall, 15.22% entry recall, and 11.63% trace edit similarity on the Hard subset. Oracle-conditioned results further attribute failures chiefly to entry-point identification and incomplete trace construction.

Significance. If the contamination concern is resolved, VulnGym would be a valuable contribution: the line-level, human-verified trace annotations (inter-rater agreement κ=0.87) and the oracle-subtask design are more diagnostic than existing function-level or classification-only benchmarks. The metric definitions are explicit, the failure decomposition in Figure 5 is informative, and the cost-effectiveness analysis is useful. The claimed novelty—fine-grained evidence annotation for repository-level agent evaluation—is credible. However, the central empirical conclusion about agent limitations is only as strong as the evidence that the benchmark is unseen by the evaluated models, and that evidence is currently missing.

major comments (3)
  1. [Phase I: Dataset Construction (VulnGym/Benchmark Construction)] The sole leakage control is the selection of advisories 'published after the training cutoff of LLM, from November 2025 to April 2026.' The evaluated models are released in 2026 (Qwen3.5 in Feb 2026; DeepSeek-V4, MiniMax-M3, GLM-5.2 in Jun 2026), and the paper reports no model-specific training cutoffs and runs no contamination probe, despite citing LessLeak-Bench. Filtering on advisory publication date also cannot prevent the vulnerable repository snapshot or fix diff from appearing in pre-cutoff training corpora. Without a probe, the nonzero scores in Table 2 (e.g., 22.58% AR for OpenHands + DeepSeek-V4-Flash) could reflect memorized advisories/patches rather than repository exploration and reasoning. This is load-bearing for the paper's central conclusion. Please either report per-model cutoffs, apply a contamination test (e.g., n-gram or completion probes on advisory texts and fix di
  2. [Model Scaling across Difficulty Levels (Experiment)] The difficulty stratification is reported as 110/31/31 advisories and 278/56/46 entries for Easy/Medium/Hard. These sums are 172 advisories and 380 entries, not the stated 184 advisories and 408 entries. No explanation is given for the missing 12 advisories and 28 entries. Because Table 2 and Figure 4 are computed on the Hard/Medium subsets, this inconsistency directly affects the interpretation of the headline numbers. Please reconcile the counts or state the exclusion criteria used when forming the difficulty strata.
  3. [Table 1 / Related Work] Table 1 and the related-work text claim that VulnGym provides natural-language defect causes ('Cause'), and state that VulnGym fills the gap with 'natural-language defect causes.' However, the benchmark construction section defines only Entry Points, Critical Operations, and Traces; no Cause field appears in the entry schema, annotation pipeline, or evaluation metrics. Either add the Cause annotation to the dataset and describe how it was verified, or revise Table 1 and the related-work comparison to remove the claim.
minor comments (5)
  1. [Table 2 / Hard subset] Hard contains only 31 advisories and 46 entries; the top AR of 22.58% corresponds to 7 advisories. The paper should report raw counts and, ideally, confidence intervals or a small-sample caveat when comparing scaffolds and models.
  2. [Figure 5] The term 'relation-level failures' is not defined. The five failure categories sum to 347; please state the total number of relation-level outcomes (e.g., 9 configurations × 46 entries) and clarify how a 'relation' maps to an entry and a configuration.
  3. [Evaluation Framework / Metrics] The line matching tolerance k=5 and the normalized repository-relative path are used throughout but not fully specified. Please define the normalization procedure and provide a brief sensitivity analysis for k.
  4. [Experimental Setup] For reproducibility, report the exact prompts, temperature, max turns, API/model versions, and per-run budgets for each agent. Also state whether the benchmark dataset, agent trajectories, and evaluation code will be released.
  5. [Phase I: Dataset Construction] The phrase 'the training cutoff of LLM' is misleading because different models have different cutoffs. Please rephrase to indicate that the cutoff is assumed or per-model, and explain why the selected window is appropriate for all evaluated models.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity in VulnGym's construction or evaluation; ground truth is external and human-verified, and the residual contamination concern is a validity threat rather than a circularity.

full rationale

VulnGym's central claim—that coding agents achieve low recall on repository-level vulnerability detection—is not derived from its own inputs. Ground-truth advisories come from GitHub's external Advisory Database, and entries are verified by independent human reviewers (Cohen's kappa = 0.87); agents contribute no labels, so there is no self-confirming loop. The oracle subtasks (T2–T4) intentionally inject reference components, which is a diagnostic decomposition, not a prediction from fitted parameters. Difficulty bins and the k=5 line-matching tolerance are evaluation design choices, not fitted quantities that are later reported as findings. The only self-citation with overlapping authors (Lian et al. 2026) appears in the Introduction as one of several supporting references for the claim that existing benchmarks focus on preselected functions; that claim is independently supported by Yildiz et al. and the related-work table, so the self-citation is not load-bearing. The paper's leakage control relies on an assumed post-training-cutoff window (November 2025 to April 2026) without a contamination probe; if false, the headline numbers could be inflated by memorization. That is an external-validity/contamination risk, not circularity by construction. Accordingly, no circular step meets the evidentiary bar; the score of 2 reflects only the minor, non-load-bearing self-citation and the unprobed contamination assumption, not a reduction of the result to its own inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on three external premises: advisory veracity, snapshot reconstruction, and training-cutoff timing. The first two go through a manual verification loop (kappa=0.87) and are plausible; the third is untested and is the weakest. Two hand-set thresholds (k=5 line tolerance; trace-length bins) shape the headline numbers. No physical or formal entities are postulated; EP/CO/Trace are annotation constructs, not entities with independent falsifiable handles.

free parameters (2)
  • Line matching tolerance k=5 = 5 lines
    Hand-chosen acceptance window for file and line matching (Equations 4-6); directly controls how 'hit' entries are counted in ER, AR, FR, LR, and FP, so it shapes every headline number.
  • Difficulty stratification thresholds = Easy <=7, Medium =8, Hard >=9 trace steps
    Hand-chosen bins in the Model Scaling section; the scaling and difficulty conclusions (non-zero only on Easy for 2B/4B, zero on Hard) are stated relative to these bins.
assumptions (4)
  • domain assumption All evaluated and future LLMs have training data cutoffs before November 2025, so advisories chosen from that window are unseen by the models.
    Phase I: 'we select advisories published after the training cutoff of LLM, from November 2025 to April 2026.' No contamination probe is run; self-reported cutoffs are relied on.
  • domain assumption GitHub advisory descriptions correctly describe the vulnerability and its reachable triggering path.
    Phase I anchors Entry Point, Critical Operation, and Trace construction to advisory descriptions; if descriptions are imprecise, annotations inherit the error.
  • domain assumption Vulnerable repository snapshots are correctly reconstructed from advisory metadata and vulnerability databases.
    Phase I 'establish[es] its vulnerable snapshot using the detailed description and vulnerability databases'; 3,448 of 3,632 advisories were excluded, mostly on this basis.
  • domain assumption The automated data-flow and control-flow analysis produces candidate EPs, COs, and Traces complete enough for the review loop to converge to correct entries.
    Phase I constructs reachable-entry records automatically; Phase II corrects but does not re-derive from scratch, so initial candidate completeness limits final coverage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection." pith.science (2026). https://pith.science/paper/FG75AS3I

@misc{pith2026260802001,
  author       = {Pith},
  title        = {Pith review of: VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FG75AS3I}},
  note         = {Machine review of arXiv:2608.02001}
}
read the original abstract

Recent advances in LLM-based vulnerability detection have shown promising results, while coding agents further extend this capability from isolated code snippets to complete repositories. This shift requires agents to autonomously explore repositories and locate vulnerability-relevant code, instead of performing detection on preselected functions. However, existing benchmarks primarily focus on vulnerability classification over preselected code snippets, limiting their ability to evaluate coding agents in repository-level vulnerability detection. Moreover, without fine-grained vulnerability trace annotations, the capability limitations underlying the detection process remain difficult to explore. To address these limitations, we present \textbf{VulnGym}, a real-world repository-level benchmark for evaluating vulnerability detection by coding agents. VulnGym aligns reviewed GitHub advisories with their corresponding vulnerable version repositories. It contains 184 advisories and 408 vulnerability entries across 23 repositories, with each entry annotated with line-level entry points, critical operations, and vulnerability traces. Using this fine-grained ground truth, VulnGym defines an end-to-end detection task and three oracle-based subtasks to jointly evaluate vulnerability detection and diagnose limitations in code localization and evidence construction. Our evaluation indicates that current coding agents remain limited in both end-to-end repository-level vulnerability detection and the construction of accurate supporting traces.

Figures

Figures reproduced from arXiv: 2608.02001 by the authors.

Figure 1
Figure 1. Overview of the VulnGym benchmark construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Repository-level composition of VulnGym. The [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the four tasks in VulnGym. T1 per [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Scaling performance of Qwen3.5 models under Claude Code across the Easy, Medium, and Hard subsets. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Composition of relation-level T1 failures on Hard. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Turn-wise action composition of DeepSeek-V4-Flash on Hard under T1 across the three agentic scaffolds. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Efficiency–effectiveness trade-offs among frontier [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 3 linked inside Pith

  1. [2]

    https://github.com/open-webui/open-webui

    Open WebUI: User-Friendly AI Interface. https://github.com/open-webui/open-webui. GitHubreposi- tory. Accessed: 2026-07-28. Croft,R.;Newlands,D.;Chen,Z.;andBabar,M.A.2021.An empiricalstudyofrule-basedandlearning-basedapproaches for static application security testing. InProceedings of the 15th ACM/IEEE international symposium on empirical soft- ware engin...

    2026

  2. [4]

    https://github

    GitHub Advisory Database. https://github. com/advisories. Accessed: 2026-07-29. GitHub Advisory Database

    2026

  3. [6]

    InInternational Conference on Learning Representations, volume 2024, 23247–23275

    MetaGPT: Meta programming for a multi-agent collabora- tive framework. InInternational Conference on Learning Representations, volume 2024, 23247–23275. Islam, M. A.; Ali, M. E.; and Parvez, M. R

    2024

  4. [7]

    InIn- ternational conference on learning representations, volume 2025, 35735–35758

    IRIS: LLM-assisted static analysis for detecting security vulnerabilities. InIn- ternational conference on learning representations, volume 2025, 35735–35758. Lian, K.; Bin, W.; Zhang, L.; Chen, L.; Wang, J.; Zhao, Z.; Yang, Y.; Lin, M.; Duan, H.; Zhao, H.; et al

    2025

  5. [8]

    InFindings of the Association for Compu- tational Linguistics: ACL 2026, 31390–31405

    Ase: A repository-level benchmark for evaluating security in ai- generated code. InFindings of the Association for Compu- tational Linguistics: ACL 2026, 31390–31405. Lin, G.; Xiao, W.; Zhang, J.; and Xiang, Y

    2026

  6. [9]

    arXiv preprint arXiv:2406.07595

    Vuldetectbench: Evaluating the deep capa- bility of vulnerability detection with large language models. arXiv preprint arXiv:2406.07595. Ma, Y.; Tian, W.; Gao, X.; Sun, H.; and Li, L

    arXiv

  7. [10]

    https://www

    MiniMax M3: Frontier Coding, 1M Con- text,NativeMultimodality—AllinOneModel. https://www. minimax.io/blog/minimax-m3. Published June 1, 2026; ac- cessed July 29,

    2026

  8. [11]

    https://qwen.ai/blog?id=qwen3.5

    Qwen3.5: Towards Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5. PublishedFebru- ary 15, 2026; accessed July 29,

    2026

Show all 18 references
  1. [13]

    InProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Com- panion Proceedings, 472–483

    Reposvul: A repository-level high-quality vulner- ability dataset. InProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Com- panion Proceedings, 472–483. Wang, Z.; Shi, T.; He, J.; Cai, M.; Zhang, J.; and Song, D

    2024

  2. [14]

    Wen, X.-C.; Wang, X.; Chen, Y.; Hu, R.; Lo, D.; and Gao, C.2026

    CyberGym: Evaluating AI Agents’ Real- World Cybersecurity Capabilities at Scale.arXiv preprint arXiv:2506.02548. Wen, X.-C.; Wang, X.; Chen, Y.; Hu, R.; Lo, D.; and Gao, C.2026. FromFunctiontoRepository:TowardsRepository- LevelEvaluationofSoftwareVulnerabilityDetection.IEEE Tr...

    2026

  3. [15]

    arXiv preprint arXiv:2606.19348

    Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Yang, J.; Jimenez, C. E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K. R.; and Press, O

  4. [17]

    https: //z.ai/blog/glm-5.2

    GLM-5.2: Built for Long-Horizon Tasks. https: //z.ai/blog/glm-5.2. Published June 16, 2026; accessed July 29,

    2026

  5. [18]

    InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),13643– 13658

    Zhang,K.;Li,J.;Li,G.;Shi,X.;andJin,Z.2024.Codeagent: Enhancing code generation with tool-integrated agent sys- tems for real-world repo-level coding challenges. InPro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2024 arXiv

  6. [2022]

    InNeurIPS 2022 Foundation Models for Decision Making Workshop

    React: Synergizing reasoning and acting in language models. InNeurIPS 2022 Foundation Models for Decision Making Workshop. Yildiz, A.; Teo, S. G.; Lou, Y.; Feng, Y.; Wang, C.; and Divakaran, D. M

    2022

  7. [2023]

    Tamberg,K.;andBahsi,H.2025

    Reflexion: Language agents with verbal re- inforcement learning.Advances in neural information pro- cessing systems, 36: 8634–8652. Tamberg,K.;andBahsi,H.2025. Harnessinglargelanguage models for software vulnerability detection: A comprehen- sive benchmarking study.IEEE Access...

    2025

  8. [2024]

    Fan, J.; Li, Y.; Wang, S.; and Nguyen, T

    Vulnera- bilitydetectionwithcodelanguagemodels:Howfararewe? arXiv preprint arXiv:2403.18624. Fan, J.; Li, Y.; Wang, S.; and Nguyen, T. N

    arXiv

  9. [2025]

    Insert Prompt as Rich Text

    Open WebUI Vulnera- ble to Stored DOM XSS via Prompts When “Insert Prompt as Rich Text” Is Enabled, Resulting in ATO/RCE. https: //github.com/advisories/GHSA-w7xj-8fx7-wfch. GHSA- w7xj-8fx7-wfch; CVE-2025-64495. Accessed: 2026-07-28. Hong,S.;Zhuge,M.;Chen,J.;Zheng,X.;Cheng,Y.;...

    2025

  10. [2026]

    https: //www.cursor.com/

    Cursor: The AI Code Editor. https: //www.cursor.com/. Accessed: 2026-07-29. Baek, T. J

    2026

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.