Pith. sign in

REVIEW 2 major objections 1 minor 14 references

Transformer models for software vulnerability detection show clear performance differences across HTML, Python, JavaScript, and PHP.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Comparative evaluation of three transformer models shows language-specific performance differences in vulnerability detection on CVEFixes with three-fold cross-validation.

T0 review reviewed 2026-06-27 challenge →

load-bearing objection This is a standard comparison of BERT, RoBERTa, and CodeBERT on CVEFixes across four languages that reports performance gaps but leaves open whether those gaps come from language traits or data skew. the 2 major comments →

arxiv 2606.10925 v1 pith:VO4KC4NB submitted 2026-06-09 cs.SE

Early Comparative Evaluation of Transformer Models for Multilingual Software Vulnerability Detection

Fiza Naseer , Javad Khan , Muhammad Yaqoob , Alexios Mylonas This is my paper
classification cs.SE
keywords software vulnerability detectiontransformer modelsBERTRoBERTaCodeBERTmultilingual code analysisCVEFixes datasetcross-validation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper evaluates BERT, RoBERTa, and CodeBERT for binary classification of vulnerable versus non-vulnerable code in four programming languages. It applies language-wise three-fold stratified cross-validation on the CVEFixes dataset to measure each model's effectiveness. The evaluation reveals marked variations in how well the models identify vulnerabilities depending on the language. A sympathetic reader would conclude that uniform application of these transformers falls short for real-world codebases that mix languages, pointing toward the need for modeling choices that respect language differences.

Core claim

This paper conducts an early comparative evaluation of BERT, RoBERTa, and CodeBERT for binary vulnerability detection across HTML, Python, JavaScript, and PHP. Using the CVEFixes dataset together with language-wise three-fold stratified cross-validation, the study finds clear performance differences across the languages and concludes that multilingual vulnerability detection requires more language-aware and robust transformer-based modelling strategies.

What carries the argument

Language-wise three-fold stratified cross-validation on the CVEFixes dataset, used to compare BERT, RoBERTa, and CodeBERT for binary vulnerability detection.

Load-bearing premise

The CVEFixes dataset supplies a representative and balanced sample of vulnerabilities across the four languages, and language-wise three-fold stratified cross-validation controls for distribution without introducing bias or leakage.

What would settle it

An independent replication that applies the same models and cross-validation procedure but obtains statistically indistinguishable performance metrics across all four languages would falsify the claim of clear performance differences.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Current transformer models cannot be applied uniformly when scanning code in multiple languages.
  • Language-specific adaptations or additional robustness measures become necessary for reliable multilingual detection.
  • Binary classification accuracy varies enough by language to affect practical deployment in mixed-language codebases.
  • Future modeling work must explicitly account for syntactic and semantic differences among programming languages.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The observed gaps may trace to mismatches between each model's pretraining corpus and the syntax of particular languages.
  • Development pipelines that scan mixed-language projects may need an initial language-identification step before model application.
  • Extending the evaluation to additional languages or larger balanced datasets would test whether the performance pattern persists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper presents an early comparative evaluation of BERT, RoBERTa, and CodeBERT for binary vulnerability detection across HTML, Python, JavaScript, and PHP. It uses the CVEFixes dataset with language-wise three-fold stratified cross-validation and reports clear performance differences across languages, concluding that multilingual vulnerability detection requires more language-aware and robust transformer-based modelling strategies.

Significance. If the performance differences are shown to arise from language properties rather than dataset artifacts, the work would provide a useful empirical baseline highlighting limitations of current transformers in multilingual security tasks and motivating targeted improvements in modeling strategies for software engineering applications.

major comments (2)
  1. The central claim that observed differences indicate the need for language-aware strategies depends on the CVEFixes subsets being comparably sized and representative per language and on the three-fold stratified CV preventing leakage. The manuscript provides no per-language sample counts, vulnerability-type distributions, or leakage audits (e.g., commit-level or AST-similarity checks), leaving open the possibility that differences reflect data skew or contamination instead.
  2. Abstract and methodology description: no quantitative results, error bars, per-language metrics, or ablation details are supplied, making it impossible to assess the magnitude or statistical significance of the reported performance differences.
minor comments (1)
  1. Abstract lacks any numerical results or table references, which reduces its utility as a standalone summary.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments highlighting important gaps in dataset transparency and result reporting. We address each major comment below and will incorporate revisions to improve the manuscript's clarity and rigor.

read point-by-point responses

  1. Referee: The central claim that observed differences indicate the need for language-aware strategies depends on the CVEFixes subsets being comparably sized and representative per language and on the three-fold stratified CV preventing leakage. The manuscript provides no per-language sample counts, vulnerability-type distributions, or leakage audits (e.g., commit-level or AST-similarity checks), leaving open the possibility that differences reflect data skew or contamination instead.

    Authors: We agree this is a valid concern for an early evaluation. In the revision we will add a table reporting exact sample counts per language (HTML, Python, JavaScript, PHP) from CVEFixes along with vulnerability-type distributions. The language-wise three-fold stratified CV was chosen to balance classes within each language, but we acknowledge the absence of explicit commit-level or AST-similarity leakage audits. We will expand the methodology to discuss potential contamination risks and note that full deduplication checks were not performed in this preliminary study; however, CVEFixes provides commit-linked fixes which we will reference to partially address this. revision: partial

  2. Referee: Abstract and methodology description: no quantitative results, error bars, per-language metrics, or ablation details are supplied, making it impossible to assess the magnitude or statistical significance of the reported performance differences.

    Authors: We accept that the current abstract and methodology lack the requested quantitative details. We will revise the abstract to include key per-language F1 scores (with standard deviations from the three folds) for each model and language. The methodology section will be updated to report all per-language metrics, include error bars on result figures, and describe any ablation experiments performed. These changes will allow readers to evaluate the magnitude and significance of the observed differences. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical comparison only

full rationale

The paper is a straightforward empirical study comparing BERT, RoBERTa, and CodeBERT on the CVEFixes dataset using language-wise 3-fold stratified cross-validation. No equations, derivations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the provided text. The central claim (performance differences across languages) is an observed experimental outcome, not a reduction to inputs by construction. This matches the default expectation for non-circular empirical ML papers.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The evaluation rests on standard machine-learning assumptions about data independence in cross-validation splits and representativeness of the CVEFixes dataset; no free parameters, invented entities, or ad-hoc axioms are introduced in the abstract.

axioms (1)
  • domain assumption Data points within each language subset are independent and identically distributed for the purpose of stratified cross-validation.
    Implicit in the choice of three-fold stratified cross-validation per language.

reviewed 2026-06-27 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Early Comparative Evaluation of Transformer Models for Multilingual Software Vulnerability Detection." pith.science (2026). https://pith.science/paper/VO4KC4NB

@misc{pith2026260610925,
  author       = {Pith},
  title        = {Pith review of: Early Comparative Evaluation of Transformer Models for Multilingual Software Vulnerability Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VO4KC4NB}},
  note         = {Machine review of arXiv:2606.10925}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Software vulnerability detection is increasingly important as modern applications combine multiple programming languages. This paper presents an early comparative evaluation of BERT, RoBERTa, and CodeBERT for binary vulnerability detection across HTML, Python, JavaScript, and PHP using the CVEFixes dataset and language-wise three-fold stratified cross-validation. The results show clear performance differences across languages, indicating that multilingual vulnerability detection requires more language-aware and robust transformer-based modelling strategies.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    CWE vulnerabilities,

    CWE, “CWE vulnerabilities,” Accessed: Nov. 18, 2025. [Online]. Available: https://cwe.mitre.org/

    2025

  2. [2]

    C3-vulmap: A dataset for privacy-aware vulnerability detection in healthcare systems,

    J. E. Ameh, A. Otebolaku, A. Shenfield, and A. Ikpehai, “C3-vulmap: A dataset for privacy-aware vulnerability detection in healthcare systems,” Electronics, vol. 14, no. 13, Art. no. 2703, 2025

    2025

  3. [3]

    DB-CBIL: A DistilBERT-based transformer hybrid model using CNN and BiLSTM for software vulnerability detection,

    A. Bahaa, A. E.-R. Kamal, H. Fahmy, and A. S. Ghoneim, “DB-CBIL: A DistilBERT-based transformer hybrid model using CNN and BiLSTM for software vulnerability detection,” IEEE Access, 2024

    2024

  4. [4]

    Multi-source cross-domain vulnerability detection based on code pre-trained model,

    Y. Cao and Y. Dong, “Multi-source cross-domain vulnerability detection based on code pre-trained model,” Information and Software Technology, Art. no. 107764, 2025

    2025

  5. [5]

    HLT: A hierarchical vulnerability detection model based on transformer,

    Y. Chen and Z. Liu, “HLT: A hierarchical vulnerability detection model based on transformer,” in Proc. 4th Int. Conf. Data Intelligence and Security (ICDIS), 2022, pp. 50– 54

    2022

  6. [6]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019, pp. 4171–4186

    2019

  7. [7]

    CodeBERT: A Pre-Trained Model for Programming and Natural Languages

    Z. Feng et al., “CodeBERT: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2002

  8. [8]

    sguard+: Machine learning guided rule-based automated vulnerability repair on smart contracts,

    C. Gao, W. Yang, J. Ye, Y. Xue, and J. Sun, “sguard+: Machine learning guided rule-based automated vulnerability repair on smart contracts,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 5, pp. 1–55, 2024

    2024

  9. [9]

    Software vulnerability analysis and discovery using machine -learning and data-mining techniques: A survey,

    S. M. Ghaffarian and H. R. Shahriari, “Software vulnerability analysis and discovery using machine -learning and data-mining techniques: A survey,” ACM Computing Surveys, vol. 50, no. 4, pp. 1–36, 2017

    2017

  10. [10]

    A systematic literature review on automated software vulnerability detection using machine learning,

    N. S. Harzevili, A. B. Belle, J. Wang, S. Wang, Z. M. Jiang, and N. Nagappan, “A systematic literature review on automated software vulnerability detection using machine learning,” ACM Computing Surveys, vol. 57, no. 3, 2025

    2025

  11. [11]

    Code vulnerability detection based on deep sequence and graph models: A survey,

    B. Wu and F. Zou, “Code vulnerability detection based on deep sequence and graph models: A survey,” Security and Communication Networks, vol. 2022, Art. no. 1176898, 2022

    2022

  12. [12]

    Vuldtransformer: Source code vulnerability detection via transformer,

    X. Zhang, F. Zhang, B. Zhao, B. Zhou, and B. Xiao, “Vuldtransformer: Source code vulnerability detection via transformer,” in Proc. 14th Asia-Pacific Symposium on Internetware, 2023, pp. 185– 193

    2023

  13. [13]

    CVEfixes: Automated collection of vulnerabilities and their fixes from open-source software,

    G. P. Bhandari, A. Naseer, and L. Moonen, “CVEfixes: Automated collection of vulnerabilities and their fixes from open-source software,” in Proc. 17th Int. Conf. Predictive Models and Data Analytics in Software Engineering (PROMISE ’21), 2021, pp. 1–10

    2021

  14. [14]

    A systematic literature Review for Transformer-based Software Vulnerability detection

    F. Naseer, J. A. Khan, M. Yaqoob, A. Mylonas, and I. Gambo, “A Systematic Literature Review for Transformer-Based Software Vulnerability Detection,” arXiv preprint arXiv:2604.24822, 2026. https://doi.org/10.48550/arXiv.2604.24822

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.24822 2026

This paper was first reviewed by grok-4.3 on June 27, 2026.