{"id":"46891cea-6d42-4fb7-9ed0-704b08805705","arxiv_id":"2505.08503","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper constructs ICVul, a C/C++ vulnerability dataset with VCCs, comprehensive metadata, and a heuristic filter (ESC) to improve label reliability.","lead":"ICVul is a new C/C++ vulnerability dataset built from NVD CVE fix commits, enriched with vulnerability-contributing commits and metadata, and it reports a 41% vulnerable-function ratio. The paper also introduces the ESC heuristic to filter out suspicious fix commits and is aimed at improving ML-based vulnerability detection.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'well-labeled' claim depends on the unvalidated assumption that every function changed in a fixing commit is itself vulnerable; this assumption is the most load-bearing risk and needs independent validation.","rationale":"The reader's weakest assumption is essentially the same as the concern here: Section II(c) labels all changed functions without independent verification, and Section II(b) relies on SZZ for VCCs. I agree that this is the load-bearing point. The paper is not internally inconsistent in a way that would force rejection; it is a dataset-construction paper whose central contribution is the labels and metadata, and those must be trusted for downstream ML training. Given that no manual validation is reported, and the ESC heuristics remove only a small fraction of suspicious commits, the 'well-labeled' statement is not yet supported. However, the authors do release the dataset and construction scripts, which makes the proposed sampling check feasible and gives the work real value even if labels need correction. A conditional acceptance is therefore appropriate: the paper should be accepted only if the authors either add a validation study of label precision or tone down the quality claims. Since the reader already reached CONDITIONAL, my analysis does not change the verdict, so I mark it UNCHANGED rather than proposing a different outcome.","tokens_in":5794,"tokens_out":2653,"duration_ms":29346,"concrete_test":"Randomly sample 100 fix commits from the released ICVul database, and for every function labeled 'vulnerable_before' in those commits, have two independent annotators (or a carefully prompted code-review model with a rubric) determine whether that function actually contains the security flaw being fixed. Compute the precision of the dataset's function-level vulnerable labels. If precision is materially below the level implied by 'well-labeled' (e.g., below 0.8), the central claim fails and the dataset must be re-released with only verified functions or with a clear confidence score per label.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ICVul is 'well-labeled' with a 41% vulnerable-function ratio. Section II(c) assigns labels by taking every function altered in a fix commit and storing before/after versions as vulnerable/patched, without checking whether each function actually contained the security flaw. Fix commits routinely mix the security-relevant hunk with refactoring, formatting, test updates, and changes to helper functions; all of those would be falsely labeled as vulnerable. The ESC filter described in Section II(d) removes only 462 of 5,366 fix commits (9.6%) and is based on heuristics (commit is a VCC of another FC, multiple CWEs, 'merge' keyword, >100 vulnerable functions), not ground truth. Moreover, the claimed threshold '100, which is the top 0.02%' is internally inconsistent with the stated counts, so the strength of that filter is unclear. The SZZ-based VCC tracing in Section II(b) is secondary metadata for this label claim, but it is also unreliable: when no lines are removed, the paper blames added and contextual lines, which can identify the fix itself as the vulnerability-introducing change. The paper's Limitations section (Section V) concedes that SZZ has accuracy limitations, but that concession does not address the more basic issue that function-level labels are derived from changed-function sets rather than from verified vulnerability locations. Until a sample of labels is checked against independent judgment, the dataset's core quality claim remains unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ICVul, a C/C++ vulnerability dataset constructed from NVD CVE records that link to GitHub fix commits. The construction pipeline filters CVEs, extracts fix commits, traces Vulnerability-Contributing Commits with SZZ, extracts repository/commit/file/function metadata, and applies a newly proposed ESC (Eliminate Suspicious Commit) filter. The authors claim that ICVul is well-labeled, well-balanced (with a 41% vulnerable-function ratio), enriched with VCC metadata, and better suited for ML-based vulnerability detection than prior datasets such as BigVul and MegaVul. The dataset and the construction framework are publicly available.","tokens_in":6100,"tokens_out":3967,"duration_ms":38707,"significance":"If the label-quality claims were substantiated, ICVul would be a useful community resource: it is publicly available, the construction pipeline is rerunnable, it includes VCCs as metadata, it uses a relational schema, and it provides function-level before/after pairs. These concrete artifacts are real strengths. However, the central contribution of the paper is data quality, and that claim is not yet established. The 41% balance ratio is a direct consequence of retaining only functions changed in fix commits, and no independent verification of function-level labels is reported. The resource is therefore potentially significant, but its key quality assertion currently rests on an unvalidated assumption.","major_comments":[{"comment":"The central claim that ICVul is well-labeled rests on the unverified assumption that every function altered in a fixing commit is itself vulnerable. The manuscript stores before and after versions of all altered functions and labels them without any manual or independent validation. Fix commits routinely mix security-relevant changes with refactoring, formatting, test updates, or modifications to helper functions; all of these would be falsely labeled as vulnerable. The ESC filter described in Section II(d) removes only 9.6% of fix commits and is based on heuristics, not ground truth. To support the paper's headline claim, the authors should report a random-sample audit against independent manual judgment, ideally with multiple annotators and inter-annotator agreement, and quantify/correct the false-label rate.","section":"Section II(c) (Function-level extraction)"},{"comment":"The 41% vulnerable-function ratio is presented as an advantage ('the dataset achieves a much better balance ratio of 41%'), but it is a direct consequence of the construction rule: only functions changed in a fix commit are included, and all such functions are labeled vulnerable in their before-fix version. This ratio is not an independent quality property and should not be advertised as evidence of balance. Comparing it with datasets that include unchanged, non-vulnerable functions is an apples-to-oranges comparison. The authors should either compare under equivalent inclusion criteria or justify why the changed-function-only restriction itself improves label quality.","section":"Section III and Table I"},{"comment":"The threshold description is internally inconsistent. The text says that 100 vulnerable functions is 'the top 0.02%', but the filter is described as identifying 10 suspicious FCs out of 5,366 (about 0.19%). Either the percentile is incorrect or the calculation is unclear. More fundamentally, all four ESC heuristics are self-defined and unvalidated: the 9.6% commit-level removal and the 24.2% vulnerable-function-level removal are not shown to remove true noise rather than valid examples. The authors should evaluate the ESC rules on a labeled sample or compare them against an established noise-filtering baseline.","section":"Section II(d) (Eliminate Suspicious Commit)"},{"comment":"The SZZ variant used here, which blames added and contextual lines when no lines are removed, can identify the fixing commit itself as the vulnerability-introducing commit. Section V concedes that SZZ has accuracy limitations but does not quantify the impact or validate the VCCs. Since VCC inclusion is a headline feature in the abstract and Table I, the authors should report the precision of VCC tracing on a sampled set or use a validated SZZ implementation, and state the resulting uncertainty in the metadata.","section":"Section II(b) and Section V"}],"minor_comments":[{"comment":"'supporting research in related field' should be 'related fields'.","section":"Abstract"},{"comment":"The note that CrossVul and CVEFixes statistics include only C/C++ data appears only in the table caption; this important qualification should also appear in the main text near the comparison discussion.","section":"Table I"},{"comment":"The transition from 13,733 CVE records to 4,723 CVEs and 5,366 fix commits needs explanation; the reader cannot tell how multiple fix commits per CVE are handled or why some CVEs yield more than one commit.","section":"Section II(a)"},{"comment":"The figures are referenced but not described in enough detail; for example, the ESC block in Figure 1 should mirror the four steps exactly so that the filtering flow is unambiguous.","section":"Figures 1 and 2"},{"comment":"The limitations section should also acknowledge the changed-function labeling assumption, not only the SZZ accuracy concern; this would make the limitations more complete and honest.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like a dataset artifact/engineering report than a validated research contribution; the requested label audit and the re-framing of the balance ratio should be mandatory before publication. The comparison table's 41% ratio is potentially misleading as presented and should be corrected prior to dissemination."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ICVul is a genuinely useful artifact in a crowded field: it packages VCC tracing, a relational schema, and commit filtering into a re-runnable pipeline, and the dataset fills a niche for C/C++ vulnerability research. The problem is the label-quality central claim: it rests on treating every function changed in a fix commit as vulnerable, which is not justified, and the ESC filtering is a small heuristic cleanup rather than independent validation.\n\nWhat's actually new is the combination rather than any single method. SZZ, NVD mining, and function extraction are all known techniques. The ESC heuristic is new but minor. The dataset includes VCC metadata and a database schema, which is more than BigVul, CVEFixes, or MegaVul offer, and the pipeline is public. That counts.\n\nThe 41% balance ratio is an artifact of including only functions changed in fix commits, not a quality achievement. The paper frames it as an advantage, which is misleading. More importantly, label quality is not independently validated. Fix commits routinely change helper functions, refactored code, or formatting; those functions get falsely labeled vulnerable. The Limitations section concedes SZZ inaccuracies but does not address this changed-function labeling problem. The ESC threshold also has an arithmetic inconsistency: with about 5,366 fix commits, the top 0.02% is roughly one commit, not the ten reported. The description of what percentile is being computed is ambiguous.\n\nThese are fixable issues. A manual validation sample, exact tool versions and parameters, and toned-down claims would address most of them. The dataset itself is still likely useful if treated as heuristically filtered rather than perfectly labeled.\n\nWho is this for? Researchers training ML detectors who need a moderately sized C/C++ dataset with richer metadata than existing alternatives. They should treat the labels as noisy and validate on their own. The paper deserves serious peer review, but my recommendation would be major revision, not acceptance: add a manual label check on a sample, pin down the ESC numbers, and soften 'well-labeled' to something like 'heuristically curated.'","headline":"A useful new C/C++ vulnerability dataset artifact whose central 'well-labeled' claim is under-validated; it deserves review but needs a manual label check before being trusted as a benchmark.","tokens_in":6605,"tokens_out":1665,"would_cite":false,"duration_ms":17913,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that ICVul is a well-labeled, balanced C/C++ vulnerability dataset: it labels only functions edited by fix commits, raising the vulnerable-function ratio to 41%, and adds VCCs and richer metadata.","keywords":["vulnerability dataset","C/C++ code","fix commits","VCC tracing","SZZ algorithm","data labeling","machine learning vulnerability detection","CVE mining"],"falsifier":"Take a random sample of ICVul fix commits and have two independent human analysts classify each changed function as vulnerable or not without seeing the label; if a substantial fraction are refactors, formatting changes, or helper updates unrelated to the flaw, the label assumption behind the 41% ratio breaks. Separately, manually compare ICVul's VCCs with repository blame or a known ground-truth history of when the vulnerable line entered; mismatches would show that the SZZ step is mislabeling introductions.","tokens_in":5617,"feed_emoji":"🛡️","tokens_out":5575,"duration_ms":51405,"temperature":0.7,"pith_summary":"ICVul is a new dataset for training machine-learning vulnerability detectors on C/C++ code. The paper claims it is better balanced and better labeled than earlier collections: instead of labeling a small fraction of all repository functions as vulnerable, it keeps only functions touched by a fix commit, so 41% of its 15,396 functions are labeled vulnerable. It also enriches each vulnerability with metadata, including the vulnerability-contributing commit found by the SZZ algorithm, and it filters out suspicious fix commits with an ESC technique. A careful reader would care because label quality and class balance directly shape how well deep-learning vulnerability detectors train in practice.","feed_headline":"C/C++ vulnerability dataset reaches 41% vulnerable-function ratio","feed_subtitle":"It labels only functions changed by fix commits, reaching 41% positive labels, and traces each flaw to its introducing commit.","key_machinery":"The two load-bearing mechanisms are the SZZ algorithm and the ESC filter. SZZ reconstructs the vulnerability-contributing commit by blaming removed lines in a fix commit, or context lines when only lines are added, identifying when the vulnerable code entered the repository. ESC then removes fix commits that fail four checks: a commit that is itself blamed as a VCC of another fix, a commit spanning multiple CWE types, an unclear or merge-style commit message, or a commit touching an unusually high number of vulnerable functions, set at 100, the top 0.02%. Together these mechanisms decide which functions receive the vulnerable label and which commits are excluded from the dataset.","core_discovery":"The central discovery is a construction recipe that yields credible function-level labels: start from vulnerability records linked to public fix commits, restrict to C/C++ files, extract before and after versions of every function touched by the fix, trace each vulnerability to the commit that introduced it via SZZ, and drop suspect commits via four ESC rules. The result, ICVul, has 4,327 fix commits across 807 repositories and 146 CWE types, with 6,276 of 15,396 functions labeled vulnerable, a 41% positive ratio. Because every vulnerable function comes from a fix commit, the dataset avoids the 4-6% positive ratio of prior datasets while storing metadata in a relational-like schema that supports commit-, file-, and function-level analysis.","pith_inferences":["My inference: because only functions touched by fix commits are labeled, functions containing vulnerable logic that a later patch leaves untouched remain unlabeled, so models trained on ICVul may miss such code in the wild.","My inference: the four ESC rules are heuristics with an arbitrary threshold of 100 vulnerable functions per commit, so some mislabels may survive and some legitimate commits may be dropped.","My inference: the 41% ratio is a property of the sampling strategy, not of underlying vulnerability prevalence, so calibration may still be needed when applying trained models to whole real-world codebases."],"forward_implications":["Vulnerability detectors trained on ICVul face a much more balanced positive class, 41% instead of 4-6%, so reported accuracy should be less inflated by majority-class bias.","The presence of VCCs makes the dataset usable for just-in-time prediction, where models learn to flag the commit that introduces a vulnerability rather than only the code fixed later.","CWE-level labels and per-function metadata support multi-class classification of vulnerability types, not only binary vulnerable or benign prediction.","Because the construction pipeline is re-runnable and public, the dataset can be refreshed when new vulnerability records and fix commits appear, keeping trained models current."],"supporting_citations":[{"why":"Supplies the main baseline whose 4% positive ratio ICVul aims to improve and whose label-quality limitations motivate the design.","marker":"[3]"},{"why":"Baseline dataset that also excludes noise but not at the commit level, positioning ICVul's ESC contribution.","marker":"[4]"},{"why":"Recommends SZZ for tracing vulnerability-contributing commits, the method adopted for VCC extraction.","marker":"[5]"},{"why":"Source of the relational-like schema inspiration and a comparison baseline with richer metadata.","marker":"[8]"},{"why":"Provides the file-extension allowlist used to filter irrelevant files during extraction.","marker":"[7]"},{"why":"Cited in the limitations to acknowledge that SZZ can be inaccurate, marking a bound on VCC quality.","marker":"[12]"},{"why":"Supports the paper's assertion that the dataset size is sufficient for training machine-learning and deep-learning models.","marker":"[9]"},{"why":"Motivates VCC inclusion by showing just-in-time defect prediction trains on introducing commits.","marker":"[6]"}],"fun_headline_variants":["ICVul: 41% of functions vulnerable in C/C++","ICVul: C/C++ dataset with 41% positive labels via fix commits","ICVul traces vulnerabilities to introducing commits with SZZ","ICVul: credible labels from SZZ and ESC at 41% ratio","ICVul: 4,327 fix commits yield 41% vulnerable-function labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The labels are only as trustworthy as the assumption that every function edited in a fixing commit is itself vulnerable and that the SZZ blame step correctly finds the commit that first introduced the flawed code.","fun_headline_variants_meta":{"raw":{"variants":["ICVul: 41% of functions vulnerable in C/C++","ICVul: C/C++ dataset with 41% positive labels via fix commits","ICVul traces vulnerabilities to introducing commits with SZZ","ICVul: credible labels from SZZ and ESC at 41% ratio","ICVul: 4,327 fix commits yield 41% vulnerable-function labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001509,"raw_usage":{"total_tokens":6002,"prompt_tokens":849,"completion_tokens":5153,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":5052}},"tokens_in":465,"tokens_out":5153,"duration_ms":37591,"temperature":1.0,"reasoning_tokens":5052,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:52:38.170672+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of ICVul fix commits and have two independent human analysts classify each changed function as vulnerable or not without seeing the label; if a substantial fraction are refactors, formatting changes, or helper updates unrelated to the flaw, the label assumption behind the 41% ratio breaks. Separately, manually compare ICVul's VCCs with repository blame or a known ground-truth history of when the vulnerable line entered; mismatches would show that the SZZ step is mislabeling introductions.","supporting_citations":[{"cited_title":"A c/c++ code vulnerability dataset with code changes and cve summaries,","cited_arxiv_id":null,"evidence_quote":"Supplies the main baseline whose 4% positive ratio ICVul aims to improve and whose label-quality limitations motivate the design."},{"cited_title":"Megavul: A c/c++ vulnerability dataset with comprehensive code representations,","cited_arxiv_id":null,"evidence_quote":"Baseline dataset that also excludes noise but not at the commit level, positioning ICVul's ESC contribution."},{"cited_title":"The secret life of software vulnerabilities: A large-scale empirical study,","cited_arxiv_id":null,"evidence_quote":"Recommends SZZ for tracing vulnerability-contributing commits, the method adopted for VCC extraction."},{"cited_title":"Flawfinder","cited_arxiv_id":null,"evidence_quote":"Provides the file-extension allowlist used to filter irrelevant files during extraction."},{"cited_title":"Problems with szz and features: An empirical study of the state of practice of defect prediction data collection,","cited_arxiv_id":null,"evidence_quote":"Cited in the limitations to acknowledge that SZZ can be inaccurate, marking a bound on VCC quality."},{"cited_title":"An empirical study of deep learning models for vulnerability detection,","cited_arxiv_id":null,"evidence_quote":"Supports the paper's assertion that the dataset size is sufficient for training machine-learning and deep-learning models."},{"cited_title":"A systematic survey of just-in-time software defect prediction,","cited_arxiv_id":null,"evidence_quote":"Motivates VCC inclusion by showing just-in-time defect prediction trains on introducing commits."}],"review_version":1}