REVIEW 4 major objections 5 minor 16 references
ICVul: A Well-labeled C/C++ Vulnerability Dataset with Comprehensive Metadata and VCCs
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that ICVul is a well-labeled, balanced C/C++ vulnerability dataset: it labels only functions edited by fix commits, raising the vulnerable-function ratio to 41%, and adds VCCs and richer metadata.
desk verdict A useful new C/C++ vulnerability dataset artifact whose central 'well-labeled' claim is under-validated; it deserves review but needs a manual label check before being trusted as a benchmark. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The two load-bearing mechanisms are the SZZ algorithm and the ESC filter. SZZ reconstructs the vulnerability-contributing commit by blaming removed lines in a fix commit, or context lines when only lines are added, identifying when the vulnerable code entered the repository. ESC then removes fix commits that fail four checks: a commit that is itself blamed as a VCC of another fix, a commit spanning multiple CWE types, an unclear or merge-style commit message, or a commit touching an unusually high number of vulnerable functions, set at 100, the top 0.02%. Together these mechanisms decide which functions receive the vulnerable label and which commits are excluded from the dataset.
What would settle it
Take a random sample of ICVul fix commits and have two independent human analysts classify each changed function as vulnerable or not without seeing the label; if a substantial fraction are refactors, formatting changes, or helper updates unrelated to the flaw, the label assumption behind the 41% ratio breaks. Separately, manually compare ICVul's VCCs with repository blame or a known ground-truth history of when the vulnerable line entered; mismatches would show that the SZZ step is mislabeling introductions.
Extended reading notes
Core claim
The central discovery is a construction recipe that yields credible function-level labels: start from vulnerability records linked to public fix commits, restrict to C/C++ files, extract before and after versions of every function touched by the fix, trace each vulnerability to the commit that introduced it via SZZ, and drop suspect commits via four ESC rules. The result, ICVul, has 4,327 fix commits across 807 repositories and 146 CWE types, with 6,276 of 15,396 functions labeled vulnerable, a 41% positive ratio. Because every vulnerable function comes from a fix commit, the dataset avoids the 4-6% positive ratio of prior datasets while storing metadata in a relational-like schema that supports commit-, file-, and function-level analysis.
Load-bearing premise
The labels are only as trustworthy as the assumption that every function edited in a fixing commit is itself vulnerable and that the SZZ blame step correctly finds the commit that first introduced the flawed code.
Editorial extensions
If this is right
- Vulnerability detectors trained on ICVul face a much more balanced positive class, 41% instead of 4-6%, so reported accuracy should be less inflated by majority-class bias.
- The presence of VCCs makes the dataset usable for just-in-time prediction, where models learn to flag the commit that introduces a vulnerability rather than only the code fixed later.
- CWE-level labels and per-function metadata support multi-class classification of vulnerability types, not only binary vulnerable or benign prediction.
- Because the construction pipeline is re-runnable and public, the dataset can be refreshed when new vulnerability records and fix commits appear, keeping trained models current.
Reading between the lines
- My inference: because only functions touched by fix commits are labeled, functions containing vulnerable logic that a later patch leaves untouched remain unlabeled, so models trained on ICVul may miss such code in the wild.
- My inference: the four ESC rules are heuristics with an arbitrary threshold of 100 vulnerable functions per commit, so some mislabels may survive and some legitimate commits may be dropped.
- My inference: the 41% ratio is a property of the sampling strategy, not of underlying vulnerability prevalence, so calibration may still be needed when applying trained models to whole real-world codebases.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ICVul, a C/C++ vulnerability dataset constructed from NVD CVE records that link to GitHub fix commits. The construction pipeline filters CVEs, extracts fix commits, traces Vulnerability-Contributing Commits with SZZ, extracts repository/commit/file/function metadata, and applies a newly proposed ESC (Eliminate Suspicious Commit) filter. The authors claim that ICVul is well-labeled, well-balanced (with a 41% vulnerable-function ratio), enriched with VCC metadata, and better suited for ML-based vulnerability detection than prior datasets such as BigVul and MegaVul. The dataset and the construction framework are publicly available.
Significance. If the label-quality claims were substantiated, ICVul would be a useful community resource: it is publicly available, the construction pipeline is rerunnable, it includes VCCs as metadata, it uses a relational schema, and it provides function-level before/after pairs. These concrete artifacts are real strengths. However, the central contribution of the paper is data quality, and that claim is not yet established. The 41% balance ratio is a direct consequence of retaining only functions changed in fix commits, and no independent verification of function-level labels is reported. The resource is therefore potentially significant, but its key quality assertion currently rests on an unvalidated assumption.
major comments (4)
- [Section II(c) (Function-level extraction)] The central claim that ICVul is well-labeled rests on the unverified assumption that every function altered in a fixing commit is itself vulnerable. The manuscript stores before and after versions of all altered functions and labels them without any manual or independent validation. Fix commits routinely mix security-relevant changes with refactoring, formatting, test updates, or modifications to helper functions; all of these would be falsely labeled as vulnerable. The ESC filter described in Section II(d) removes only 9.6% of fix commits and is based on heuristics, not ground truth. To support the paper's headline claim, the authors should report a random-sample audit against independent manual judgment, ideally with multiple annotators and inter-annotator agreement, and quantify/correct the false-label rate.
- [Section III and Table I] The 41% vulnerable-function ratio is presented as an advantage ('the dataset achieves a much better balance ratio of 41%'), but it is a direct consequence of the construction rule: only functions changed in a fix commit are included, and all such functions are labeled vulnerable in their before-fix version. This ratio is not an independent quality property and should not be advertised as evidence of balance. Comparing it with datasets that include unchanged, non-vulnerable functions is an apples-to-oranges comparison. The authors should either compare under equivalent inclusion criteria or justify why the changed-function-only restriction itself improves label quality.
- [Section II(d) (Eliminate Suspicious Commit)] The threshold description is internally inconsistent. The text says that 100 vulnerable functions is 'the top 0.02%', but the filter is described as identifying 10 suspicious FCs out of 5,366 (about 0.19%). Either the percentile is incorrect or the calculation is unclear. More fundamentally, all four ESC heuristics are self-defined and unvalidated: the 9.6% commit-level removal and the 24.2% vulnerable-function-level removal are not shown to remove true noise rather than valid examples. The authors should evaluate the ESC rules on a labeled sample or compare them against an established noise-filtering baseline.
- [Section II(b) and Section V] The SZZ variant used here, which blames added and contextual lines when no lines are removed, can identify the fixing commit itself as the vulnerability-introducing commit. Section V concedes that SZZ has accuracy limitations but does not quantify the impact or validate the VCCs. Since VCC inclusion is a headline feature in the abstract and Table I, the authors should report the precision of VCC tracing on a sampled set or use a validated SZZ implementation, and state the resulting uncertainty in the metadata.
minor comments (5)
- [Abstract] 'supporting research in related field' should be 'related fields'.
- [Table I] The note that CrossVul and CVEFixes statistics include only C/C++ data appears only in the table caption; this important qualification should also appear in the main text near the comparison discussion.
- [Section II(a)] The transition from 13,733 CVE records to 4,723 CVEs and 5,366 fix commits needs explanation; the reader cannot tell how multiple fix commits per CVE are handled or why some CVEs yield more than one commit.
- [Figures 1 and 2] The figures are referenced but not described in enough detail; for example, the ESC block in Figure 1 should mirror the four steps exactly so that the filtering flow is unambiguous.
- [Section V] The limitations section should also acknowledge the changed-function labeling assumption, not only the SZZ accuracy concern; this would make the limitations more complete and honest.
Circularity Check
No significant circularity: ICVul's claims are construction reports, not predictions derived from inputs that encode them.
full rationale
The paper's central claims concern dataset construction and reported statistics, not a derived prediction that could reduce to its own inputs. The 41% vulnerable-function ratio is presented as a direct consequence of the inclusion rule that only functions changed in fix commits are collected (Section II-c, Section III), and the paper explicitly says 'by only collecting functions with changes in the commits to ensure label accuracy, the dataset achieves a much better balance ratio of 41%'. This is an artifact of the construction choice, but it is reported as a dataset property rather than validated as an independent finding, so it is a correctness/validity concern rather than a circular reduction. Similarly, the ESC filter in Section II-d uses self-defined heuristics to remove suspicious commits; the paper's claim that this 'ensuring credible data labels' is only as strong as those heuristics, which is an external-validation gap, not a circularity. The SZZ-based VCC tracing is cited to external prior work [5], and the paper's own Limitations section (Section V) acknowledges SZZ accuracy limitations with an external citation [12]. No load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling-via-citation appears. The unvalidated assumption that every function altered in a fixing commit is vulnerable is a real labeling risk, but it is not a step in which a claimed result is equivalent by construction to an input; it is an unverified empirical assumption. Therefore the appropriate finding is no significant circularity, score 0.
Assumptions & free parameters
free parameters (1)
- ESC Step 4 vulnerable-function count threshold =
100
assumptions (3)
- domain assumption SZZ algorithm accurately identifies vulnerability-contributing commits.
- domain assumption NVD CVE entries and their GitHub commit URLs correctly describe real vulnerabilities.
- ad hoc to paper All functions altered in a fix commit are exactly the vulnerable functions.
Cite this review
Pith. "Pith review of ICVul: A Well-labeled C/C++ Vulnerability Dataset with Comprehensive Metadata and VCCs." pith.science (2026). https://pith.science/paper/KEXAJVSC
@misc{pith2026250508503,
author = {Pith},
title = {Pith review of: ICVul: A Well-labeled C/C++ Vulnerability Dataset with Comprehensive Metadata and VCCs},
year = {2026},
howpublished = {\url{https://pith.science/paper/KEXAJVSC}},
note = {Machine review of arXiv:2505.08503}
}
read the original abstract
Machine learning-based software vulnerability detection requires high-quality datasets, which is essential for training effective models. To address challenges related to data label quality, diversity, and comprehensiveness, we constructed ICVul, a dataset emphasizing data quality and enriched with comprehensive metadata, including Vulnerability-Contributing Commits (VCCs). We began by filtering Common Vulnerabilities and Exposures from the NVD, retaining only those linked to GitHub fix commits. Then we extracted functions and files along with relevant metadata from these commits and used the SZZ algorithm to trace VCCs. To further enhance label reliability, we developed the ESC (Eliminate Suspicious Commit) technique, ensuring credible data labels. The dataset is stored in a relational-like database for improved usability and data integrity. Both ICVul and its construction framework are publicly accessible on GitHub, supporting research in related field.
Figures
Reference graph
Works this paper leans on
-
[1]
Deep learning based vulnerability detection: Are we there yet?
S. Chakraborty, R. Krishna, Y . Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet?”IEEE Transactions on Software Engineering, vol. 48, no. 9, pp. 3280–3296, 2021
2021
-
[2]
Data quality for software vulnerability datasets,
R. Croft, M. A. Babar, and M. M. Kholoosi, “Data quality for software vulnerability datasets,” in2023 IEEE/ACM 45th International Confer- ence on Software Engineering (ICSE). IEEE, 2023, pp. 121–133
2023
-
[3]
A c/c++ code vulnerability dataset with code changes and cve summaries,
J. Fan, Y . Li, S. Wang, and T. N. Nguyen, “A c/c++ code vulnerability dataset with code changes and cve summaries,” inProceedings of the 17th International Conference on Mining Software Repositories, 2020, pp. 508–512
work page 2020
-
[4]
Megavul: A c/c++ vulnerability dataset with comprehensive code representations,
C. Ni, L. Shen, X. Yang, Y . Zhu, and S. Wang, “Megavul: A c/c++ vulnerability dataset with comprehensive code representations,” in2024 IEEE/ACM 21st International Conference on Mining Software Reposi- tories (MSR). IEEE, 2024, pp. 738–742
work page 2024
-
[5]
The secret life of software vulnerabilities: A large-scale empirical study,
E. Iannone, R. Guadagni, F. Ferrucci, A. De Lucia, and F. Palomba, “The secret life of software vulnerabilities: A large-scale empirical study,” IEEE Transactions on Software Engineering, vol. 49, no. 1, pp. 44–63, 2022
work page 2022
-
[6]
A systematic survey of just-in-time software defect prediction,
Y . Zhao, K. Damevski, and H. Chen, “A systematic survey of just-in-time software defect prediction,”ACM Computing Surveys, vol. 55, no. 10, pp. 1–35, 2023
work page 2023
-
[7]
D. A. Wheeler, “Flawfinder.” [Online]. Available: https://github.com/ david-a-wheeler/flawfinder
-
[8]
Cvefixes: automated collec- tion of vulnerabilities and their fixes from open-source software,
G. Bhandari, A. Naseer, and L. Moonen, “Cvefixes: automated collec- tion of vulnerabilities and their fixes from open-source software,” in Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering, 2021, pp. 30–39
2021
Show all 16 references
-
[9]
An empirical study of deep learning models for vulnerability detection,
B. Steenhoek, M. M. Rahman, R. Jiles, and W. Le, “An empirical study of deep learning models for vulnerability detection,” inIEEE/ACM 45th International Conference on Software Engineering. IEEE, 2023, pp. 2237–2248
2023
-
[10]
Crossvul: a cross-language vulnerability dataset with commit data,
G. Nikitopoulos, K. Dritsa, P. Louridas, and D. Mitropoulos, “Crossvul: a cross-language vulnerability dataset with commit data,” inProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2...
2021
-
[11]
Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection,
Y . Chen, Z. Ding, L. Alowain, X. Chen, and D. Wagner, “Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection,” inProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, 2023, pp. 654–668
2023
-
[12]
Problems with szz and features: An empirical study of the state of practice of defect prediction data collection,
S. Herbold, A. Trautsch, F. Trautsch, and B. Ledel, “Problems with szz and features: An empirical study of the state of practice of defect prediction data collection,”Empirical Software Engineering, vol. 27, no. 2, p. 42, 2022
2022
-
[13]
Report on the static analysis tool exposition (sate) iv,
V . Okun, A. Delaitre, P. E. Blacket al., “Report on the static analysis tool exposition (sate) iv,”NIST Special Publication, vol. 500, p. 297, 2013
2013
-
[14]
Nist software assurance reference dataset,
National Institute of Standards and Technology, “Nist software assurance reference dataset,” Last accessed on October 08, 2024. [Online]. Available: https://samate.nist.gov/SARD
2024
-
[15]
D2a: A dataset built for ai-based vulnerability detection methods using differential analysis,
Y . Zheng, S. Pujar, B. Lewis, L. Buratti, E. Epstein, B. Yang, J. Laredo, A. Morari, and Z. Su, “D2a: A dataset built for ai-based vulnerability detection methods using differential analysis,” in2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Eng...
2021
-
[16]
DeVign: Effective Vulnerability identification by Learning Comprehensive Program Semantics via Graph Neural Networks,
Y . Zhou, S. Liu, J. K. Siow, X. Du, and Y . Liu, “DeVign: Effective Vulnerability identification by Learning Comprehensive Program Semantics via Graph Neural Networks,”arXiv (Cornell University), vol. 32, pp. 10 197–10 207, 9 2019. [Online]. Available: https://arxiv.org/pdf/1...
2019 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.