Pith. sign in

REVIEW 2 major objections 7 minor 34 references

On Java, the train-test split is a first-class property of any vulnerability-detection score: moving from random to project-disjoint drops F1 by double digits, so project-disjoint should be the default headline split.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:47 UTC pith:K4B6HJJW

load-bearing objection Useful Java vuln-detection infrastructure: the package (splits + harness + contamination audit) is the real product, and the random→project F1 drop is cleanly shown. the 2 major comments →

arxiv 2607.02825 v1 pith:K4B6HJJW submitted 2026-07-02 cs.CR cs.AI

JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode

Norbert Sandor Szolnoki , Gabor Antal This is my paper
classification cs.CR cs.AI
keywords Java vulnerability detectionbenchmarkproject-disjoint splitdata leakagecontamination auditLLM evaluationmethod-level labelsCWE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper releases JavaVulBench, a Java vulnerability-detection benchmark of roughly 30,600 methods drawn from 1,740 CVEs across 700-plus projects, labeled at method and line level. It argues that reported detector scores are not comparable unless leakage is controlled: a random split lets near-duplicate methods from the same project appear in both train and test, which inflates F1. On their data, switching to a project-disjoint split reproduces the large F1 drops previously documented for C/C++, while temporal and CWE-family holds-out further stress generalization. The release ships five ready-made splits, a single evaluation harness that runs encoder classifiers and API or local LLMs under one schema, and a per-model contamination audit that separates potentially memorized CVEs from clean ones. A reader who builds or evaluates security tools should care because enterprise Java has been measured under optimistic conditions that do not reflect real project-to-project transfer.

Core claim

On Java, split strategy is a first-class property of any reported vulnerability-detection number. Switching from a random split to a project-disjoint split reproduces the large F1 drops seen on C/C++ (for example UniXcoder falls from 0.446 to 0.298 and GraphCodeBERT from 0.343 to 0.256), so project-disjoint should be treated as the default headline split and users should also quote clean-subset F1 after a pre-training contamination audit.

What carries the argument

JavaVulBench: a CVE-grounded corpus of about 30,600 labeled Java methods with five shipped splits (random, project-disjoint, temporal, deduplicated, unseen CWE-family), a unified LlmPrediction harness spanning encoder, Ollama, and OpenRouter backends, and a per-model contamination audit that partitions each test set into risky (pre-cutoff) and clean (post-cutoff) CVEs.

Load-bearing premise

The labels taken from developer fix commits (after filtering test, format, and refactor noise) correctly mark the vulnerable methods and lines, and the constructed same-file and same-project negatives are truly free of unlabeled bugs.

What would settle it

Have security experts re-label the project-disjoint test set independently; if the large F1 drop from random to project-disjoint disappears once fix-line approximation errors and contaminated negatives are corrected, the claim that split strategy alone drives the gap fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Project-disjoint, not random, becomes the default headline split for any Java vulnerability detector.
  • Papers reporting high Java F1 must also quote clean-subset F1 from a contamination audit so memorization is not mistaken for generalization.
  • New detectors can be dropped into the single-command harness and compared against the twelve reference baselines under identical sample IDs and metrics.
  • Temporal and CWE-family splits become standard stress tests for whether models transfer across time and vulnerability families.
  • Cross-language claims about leakage inflation can now be checked on enterprise Java as well as on C/C++.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If fix-line labels systematically miss semantic vulnerability sites, part of the measured F1 gap may be label noise rather than pure leakage; re-labeling with static-analysis oracles would separate the two.
  • The same risky-versus-clean audit discipline could be required for any LLM security benchmark whose test CVEs predate model cutoffs.
  • Stronger cross-project near-duplicate removal beyond Jaccard 0.8 might further enlarge the random-to-project gap and tighten the case against random splits.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The manuscript introduces JavaVulBench, a Java-focused vulnerability-detection benchmark of ~30,600 methods from 1,740 CVEs across 700+ projects, with method- and line-level labels, CVE publication dates, and five shipped split strategies (random, project-disjoint, temporal, deduplicated, CWE-family). It also contributes a unified multi-backend evaluation harness (HuggingFace encoders, Ollama local generative models, OpenRouter API LLMs) under a single LlmPrediction schema, twelve reference detectors, and a per-model pre-training contamination audit that partitions test CVEs into risky vs. clean subsets. Three validation studies support the central methodological claim that, on Java, split strategy is a first-class property of reported detection numbers: Table 1 shows large F1 drops from random to project-disjoint (e.g., UniXcoder 0.446→0.298; GraphCodeBERT 0.343→0.256; CodeBERT collapse), Tables 2–3 give full-set encoder and cost-matched API baselines, and Table 4 reports leakage-aware F1. Data, code, checkpoints, and a Dockerised reproduction pipeline are archived on Zenodo.

Significance. If the results hold, this is a practically useful infrastructure contribution for Java vulnerability detection research. Existing Java resources (Vul4J, VJBench, CWE-Bench-Java, CVEfixes-Java) do not jointly provide explicit negatives, line labels, multiple realistic splits, a multi-backend harness, and contamination auditing; Table 5 makes that gap concrete. Reproducing Chakraborty et al.’s random-vs-project leakage effect on Java is an important empirical service to the community and justifies treating project-disjoint as the default headline split. Strengths that should be credited explicitly include: the self-contained Zenodo artifact with fine-tuned checkpoints and one-command reproduction; the unified LlmPrediction interface that makes cross-backend comparison under identical sample_ids feasible; the shipped contamination audit with clean-subset reporting expectations; and the transparent cost-control design (200-sample stratified API probe). These are the right kind of deliverables for a benchmark/artifact paper.

major comments (2)
  1. [§2 Splits / §6 Limitations] §2 (Splits) and §6 (Limitations): one of the five advertised “realistic split strategies” is the deduplicated split, but the Limitations section states it is “currently equivalent to random at Jaccard 0.8.” That undercuts contribution (i) as written. Either strengthen near-duplicate detection (e.g., cross-project clone/AST/embedding-based filters) so the split is meaningfully distinct, or reframe the contribution as four primary realistic splits plus a near-duplicate diagnostic, and move the Jaccard-0.8 result into the main validation narrative rather than only the limitations.
  2. [§4.1 Table 1 / §4.2 Tables 2–3] §4.1 Table 1 and §4.2 Tables 2–3: the headline random→project F1 gaps and the API ranking rest on single-point estimates without uncertainty (no bootstrap CIs, no multi-seed fine-tunes, no probe resampling). CodeBERT’s F1=0.000 at the default 0.5 threshold is explained as majority-class collapse, and PR-AUC is correctly noted as still informative, but the paper’s recommendation that “project-disjoint should be the default headline split” would be more robust if the authors reported (i) threshold-swept or PR-AUC-selected operating points for the collapsed encoder and (ii) at least interval estimates for the 200-sample API probe. Without that, small absolute differences among API models (e.g., GPT-4o 0.419 vs Claude 0.416) cannot be interpreted as stable rankings.
minor comments (7)
  1. [Abstract] Abstract and title block: the YouTube demonstration URL is duplicated (once with escaped underscore and once plain). Clean to a single canonical link.
  2. [§3 Metrics] §3 Metrics: ECE is cited as “ECE [?]” with a missing bibliography entry. Add the standard Expected Calibration Error reference and complete the citation.
  3. [§2 Labelling] §2 Labelling: the negative-sampling ratios (same-file 3:1, same-project 2:1, random-project 1:1) are free design choices. A short sensitivity note (or appendix table) showing that the random→project F1 gap direction is stable under alternate ratios would help readers who worry that negatives drive the gap.
  4. [Table 1] Table 1: DeepSeek-Coder-1.3B has “–” for CWE-family while other models report numbers. State explicitly whether the model was not evaluated on that split or whether the run failed, so the table is self-contained.
  5. [Table 4] Table 4: several API models have “—” for Risky/Clean counts while still reporting F1 all / F1 clean. Either fill the partition counts for the probe or clarify that those rows are probe-only without full-split audit counts.
  6. [§6 / Table 5] §6 Positioning: the comparison table is helpful; a one-sentence note on whether JavaVulBench methods overlap with Vul4J/VJBench CVE sets would help users who want to combine repair and detection benchmarks.
  7. [Table 1 caption] Minor prose: “are distributed with the artifact” in the Table 1 caption is slightly awkward; “shipped with the artifact” matches the rest of the paper’s wording.

Circularity Check

0 steps flagged

No significant circularity: empirical artifact paper whose metrics and split-impact claims are measured on held-out data, not reduced to inputs by construction.

full rationale

JavaVulBench is a dataset-and-harness release, not a first-principles derivation. The load-bearing claim (project-disjoint vs random F1 drop on Java, Tables 1–2) is an empirical measurement of standard classification metrics (F1, MCC, PR-AUC, ECE, Brier) on held-out splits constructed from external CVE/fix-commit sources; the same labelling pipeline is applied uniformly, so the ranking is not forced by definition. The contamination audit partitions by external CVE publication dates versus stated model cutoffs and reports clean-subset F1 without redefining success. Citations to Chakraborty et al. [5] supply an external C/C++ precedent that is independently falsifiable; the authors’ own Zenodo archive [31] is the artifact itself, not a load-bearing uniqueness theorem or ansatz. No self-definitional loop, fitted-parameter-as-prediction, or renaming of a known result appears in the derivation chain. Score 0 is therefore the correct outcome.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central claim rests on standard software-mining and ML-evaluation assumptions plus a few design choices (negative sampling ratios, Jaccard threshold, temporal cut, 200-sample probe). No new physical entities are postulated; the free parameters are engineering thresholds that affect split composition and reported scores but are disclosed. Domain assumptions about CVE→fix-commit→vulnerable-method labeling are the main load-bearing premises.

free parameters (5)
  • Jaccard near-duplicate threshold = 0.8
    Deduplicated split uses Jaccard 0.8; authors note the split is currently equivalent to random at this setting, so the threshold is a free design choice that currently fails to change the partition.
  • Negative sampling ratios = 3:1 / 2:1 / 1:1
    Per vulnerable method: same-file 3:1, same-project 2:1, random-project 1:1. These hand-chosen ratios define class balance and difficulty of negatives.
  • Temporal split cutoff = train <2023; test ≥2023-07
    Train on CVEs before 2023; test from 2023-07 onward. Arbitrary but disclosed calendar cut that defines the time split.
  • API evaluation probe size and seed = N=200, seed=42
    Fixed stratified 200-sample subset per split with seed 42 for cost control; all API F1 numbers depend on this subsample.
  • Default classification threshold = 0.5
    F1 reported at default 0.5 confidence; CodeBERT collapses to majority class under this threshold on hard splits, so headline F1 is threshold-sensitive.
axioms (5)
  • domain assumption CVE records merged from NVD, GitHub Advisories, and CVEfixes, resolved to fix commits, correctly identify security-relevant vulnerable methods after noise filtering of test/format/refactor commits.
    §2 Mining/Labelling: entire ground truth depends on this mining pipeline.
  • domain assumption Developer-edited lines in fix commits are a sufficient approximation of vulnerable lines for localization metrics.
    §2 Labelling and §6 Limitations explicitly use fix lines as line labels.
  • domain assumption Constructed negatives (same file/project/random project) are non-vulnerable for binary detection evaluation.
    §2 Labelling; unlabeled latent vulnerabilities would inflate false-positive/true-negative counts.
  • domain assumption Model pre-training cutoffs used in the contamination audit correctly separate potentially memorized vs post-cutoff CVEs.
    §4.3 / Table 4: leakage-aware F1 depends on published cutoff dates as a proxy for memorization risk.
  • standard math Standard classification and calibration metrics (F1, MCC, PR-AUC, ECE, Brier) are appropriate summaries under class imbalance for comparing detectors.
    §3 Metrics; conventional ML evaluation assumptions.
invented entities (3)
  • JavaVulBench corpus + five shipped split strategies independent evidence
    purpose: Provide controlled-leakage Java vulnerability evaluation data at method and line granularity.
    Primary artifact contribution; not a physical postulate but a constructed dataset whose validity rests on mining/labeling assumptions above.
  • Unified LlmPrediction multi-backend harness independent evidence
    purpose: Run encoder, Ollama, and OpenRouter detectors under one schema and command for comparable metrics.
    Engineering interface enabling the reference baselines; falsifiable by re-running shipped scripts.
  • Per-model risky/clean contamination partition independent evidence
    purpose: Separate potentially memorized pre-cutoff CVEs from post-cutoff clean test items for leakage-aware F1.
    Evaluation mode introduced in the paper; independent handle is public CVE dates vs model cutoffs, not a new latent variable.

pith-pipeline@v1.1.0-grok45 · 14101 in / 4055 out tokens · 38249 ms · 2026-07-12T06:47:33.756297+00:00 · methodology

0 comments
read the original abstract

We release \textsc{JavaVulBench}, a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset contains $\sim$30{,}600 Java methods spanning 1{,}740 CVEs and 700+ projects, labelled at both method and line granularity, with per-CVE publication dates and five realistic split strategies: random, project-disjoint, temporal, deduplicated, and unseen CWE-family. The harness provides a single \texttt{LlmPrediction} schema across three backend families (encoder classifiers, local generative models served by Ollama, and API-served LLMs routed through OpenRouter) so that twelve reference detectors CodeBERT, GraphCodeBERT, UniXcoder, DeepSeek-Coder-1.3B, and eight API/open-weight LLMs (GPT-4o, GPT-4.1-mini, Claude Sonnet~4, DeepSeek-v3, DeepSeek-Coder-v2, Qwen-2.5-Coder-14B/7B, CodeLlama-13B) are evaluated under identical conditions from a single command. A pre-training contamination audit is shipped alongside every model so users can separate genuinely unseen test CVEs from potentially memorised ones. Data, code, and fine-tuned checkpoints are archived on Zenodo [31] and short demonstration video is available on YouTube (https://www.youtube.com/watch?v=nMTX\_hqkuoM) https://www.youtube.com/watch?v=nMTX_hqkuoM.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 1 canonical work pages

  1. [1]

    Guru Bhandari, Amara Naseer, and Leon Moonen. 2021. CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software. In 17th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE). ACM, 30–39. doi:10.1145/3475960.3475985

    doi:10.1145/3475960.3475985 2021

  2. [2]

    Glenn W. Brier. 1950. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review78, 1 (1950), 1–3

    1950

  3. [3]

    Max Brunsfeld. 2018. Tree-sitter: An Incremental Parsing System for Program- ming Tools. InStrange Loop Conference. https://tree-sitter.github.io/tree-sitter/

    2018

  4. [4]

    Díaz Ferreyra

    Quang-Cuong Bui, Riccardo Scandariato, and Nicolás E. Díaz Ferreyra. 2022. Vul4J: A Dataset of Reproducible Java Vulnerabilities Geared Towards the Study of Program Repair Techniques. In19th International Conference on Mining Software Repositories (MSR). ACM, 464–468. doi:10.1145/3524842.3528482

    doi:10.1145/3524842.3528482 2022

  5. [5]

    Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2022. Deep Learning Based Vulnerability Detection: Are We There Yet?IEEE Trans- actions on Software Engineering48, 9 (2022), 3280–3296. doi:10.1109/TSE.2021. 3087402

    doi:10.1109/tse.2021 2022

  6. [6]

    Cormack, Charles L

    Gordon V. Cormack, Charles L. A. Clarke, and Stefan Büttcher. 2009. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. In32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, 758–759. doi:10.1145/1571941.1572114

    doi:10.1145/1571941.1572114 2009

  7. [7]

    DeepSeek-AI. 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence.arXiv preprint arXiv:2406.11931(2024)

    Pith/arXiv arXiv 2024

  8. [8]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report.arXiv preprint arXiv:2412.19437(2024)

    Pith/arXiv arXiv 2024

  9. [9]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In17th In- ternational Conference on Mining Software Repositories (MSR). ACM, 508–512. doi:10.1145/3379597.3387501

    doi:10.1145/3379597.3387501 2020

  10. [10]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. InFindings of the Association for Computational Linguistics: EMNLP 2020. ACL, 1536–1547. doi:10.18653/v1/2020.findings-emnlp.139

    doi:10.18653/v1/2020.findings-emnlp.139 2020

  11. [11]

    Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for Datasets. Commun. ACM64, 12 (2021), 86–92. doi:10.1145/3458723

    doi:10.1145/3458723 2021

  12. [12]

    GitHub. 2024. GitHub Security Advisories Database. https://github.com/ advisories

    2024

  13. [13]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In60th Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 7212–7225. doi:10.18653/v1/2022.acl-long.499

    doi:10.18653/v1/2022.acl-long.499 2022

  14. [14]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Representations with Data Flow. InInternational Conference on Learning Representati...

    2021

  15. [15]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guant- ing Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang

  16. [16]

    DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196(2024)

    Pith/arXiv arXiv 2024

  17. [17]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations (ICLR)

    2022

  18. [18]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-Coder Technical Report.arXiv preprint arXiv:240...

    Pith/arXiv arXiv 2024

  19. [19]

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale Similarity Search with GPUs. InIEEE Transactions on Big Data, Vol. 7. IEEE, 535–547. doi:10. 1109/TBDATA.2019.2921572

    arXiv 2021

  20. [20]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in Neural Information Processing Systems (NeurIPS)

    2020

  21. [21]

    Ziyang Li, Saikat Dutta, and Mayur Naik. 2024. IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities.arXiv preprint arXiv:2405.17238(2024)

    Pith/arXiv arXiv 2024

  22. [22]

    Zhongxin Liu, Zhijie Tang, Junwei Zhang, Xin Xia, and Xiaohu Yang. 2024. Pre- training by Predicting Program Dependencies for Vulnerability Analysis Tasks. In46th IEEE/ACM International Conference on Software Engineering (ICSE). ACM, 1–13. doi:10.1145/3597503.3639142

    doi:10.1145/3597503.3639142 2024

  23. [23]

    B. W. Matthews. 1975. Comparison of the Predicted and Observed Secondary Structure of T4 Phage Lysozyme.Biochimica et Biophysica Acta (BBA) - Protein Structure405, 2 (1975), 442–451. doi:10.1016/0005-2795(75)90109-9

    doi:10.1016/0005-2795(75)90109-9 1975

  24. [24]

    National Institute of Standards and Technology. 2024. National Vulnerability Database. (2024). https://nvd.nist.gov/

    2024

  25. [25]

    Ollama. 2024. Ollama: Run Large Language Models Locally. https://ollama.com/

    2024

  26. [26]

    OpenAI. 2024. GPT-4o System Card.arXiv preprint arXiv:2410.21276(2024)

    Pith/arXiv arXiv 2024

  27. [27]

    OpenRouter. 2024. OpenRouter: A Unified Interface for LLMs. https://openrouter. ai/

    2024

  28. [28]

    OWASP Foundation. 2021. OWASP Top 10:2021 — The Ten Most Critical Web Application Security Risks. https://owasp.org/Top10/

    2021

  29. [29]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. In2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 3982–3992. doi:10.18653/v1/D19- 1410

    doi:10.18653/v1/d19- 2019

  30. [30]

    Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Frame- work: BM25 and Beyond.Foundations and Trends in Information Retrieval3, 4 (2009), 333–389. doi:10.1561/1500000019

    doi:10.1561/1500000019 2009

  31. [31]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itamar Gat, Xi- aoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, T...

    Pith/arXiv arXiv 2023

  32. [32]

    2026.JavaVulBench: A Java Vulner- ability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-A ware Evaluation Mode

    Norbert Sándor Szolnoki and Gábor Antal. 2026.JavaVulBench: A Java Vulner- ability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-A ware Evaluation Mode. doi:10.5281/zenodo.20110174

    work page doi:10.5281/zenodo.20110174 2026

  33. [33]

    Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, and Saheed Shah. 2023. How Effective Are Neural Networks for Fixing Security Vulnerabilities. In32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, 1282–1294. doi:10.1145/3597926. 3598135

    doi:10.1145/3597926 2023

  34. [34]

    Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective Vulnerability Identification by Learning Comprehensive Pro- gram Semantics via Graph Neural Networks. InAdvances in Neural Information Processing Systems (NeurIPS). 10197–10207

    2019