{"id":"9f4438ac-7bb3-4db7-b9c4-596818cc1bde","arxiv_id":"2509.03876","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A benchmark of 1,128 CVEs shows 12 vulnerability-version identification tools all fall below 45% accuracy, and even the best ensembles stay under 60%.","lead":"Researchers evaluated 12 tools that try to identify which software versions are affected by known vulnerabilities, and found that the best one is correct less than 45% of the time. The study contributes a manually curated benchmark of 1,128 real C/C++ vulnerabilities and suggests that new approaches are needed, not just better tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark ground truth labels every version between inducing and fixing commits as vulnerable without per-version validation, biasing tracing tools and potentially distorting the 45% accuracy ceiling.","rationale":"The reader's weakest_assumption identifies exactly this issue: the ground truth uses the same SZZ continuous-range model as the tracing tools, so it may overstate their relative performance and distort the reported numbers. I agree that this is the most load-bearing concern. It is more fundamental than the ensemble overfitting or the numerical inconsistency, because it challenges the validity of the benchmark itself, which underpins every quantitative claim in the paper. The paper's manual annotation process validates the inducing commit and fixing commit but not each intermediate version, so the so-called 'manually validated' ground truth is only as good as the continuous-range assumption. This is not an internal inconsistency: the paper describes its method clearly and even discusses multi-branch difficulties, but it does not assess how often the continuous-range model is violated in its own dataset. The proposed concrete test is feasible and decisive: using Git ancestry on a sample of CVEs can independently verify whether each labeled affected version actually contains the vulnerability-introducing change and lacks the fix. If the test shows frequent violations, the reported accuracy numbers, especially the 45% ceiling and the tracing-vs-matching gap, would need to be recomputed. However, I do not think this warrants rejection or a verdict of unverdictable: the benchmark artifacts are released, the methodology is transparent, and the concern can be addressed with a supplementary validation study. The current CONDITIONAL verdict is appropriate, pending this validation. Therefore I do not change the reader's verdict.","tokens_in":25748,"tokens_out":4078,"duration_ms":41889,"concrete_test":"Sample 100 CVEs from the released benchmark. For each CVE, use the repository's Git history to check every release tag in the labeled affected range: a release is truly vulnerable iff it is a descendant of the vulnerability-inducing commit and not a descendant of the fixing commit (using git merge-base --is-ancestor). Count the CVEs where at least one labeled version violates this criterion. If more than 15% of sampled CVEs contain such a mislabeled version, the continuous-range ground truth is unreliable. Then recompute the vulnerability-level accuracy of the best tracing tool (VCCFinder) and the best matching tool (ReDeBug) using the corrected labels; if the tracing-vs-matching gap changes by more than 5 percentage points, the paradigm comparison is materially biased and the headline accuracy figures must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative result--no tool exceeds 45% vulnerability-level accuracy--rests entirely on the benchmark's ground truth. The construction (Section III-B.3) determines affected versions by identifying the vulnerability-inducing commit and the fixing commit, then labeling every release between them as vulnerable: 'All versions between the inducing-commit and patch-commit are labeled as vulnerable.' This is the same continuous-range model that tracing-based tools (V-SZZ and its variants) implement. The manual annotation process validates the inducing commit and the fixing commit; it does not independently validate each intermediate release. In real-world projects with multiple branches, backported security fixes, partial fixes, or refactorings that remove the vulnerable code before the official fix, this continuous-range labeling is incorrect: a release on a branch that never received the inducing commit would be labeled vulnerable, and a release on a branch that already received a backported fix would also be labeled vulnerable. The effect is twofold. First, tracing tools are rewarded for emitting the same interval as the ground truth, even for versions that never actually contained the vulnerable logic, inflating their absolute accuracy. Second, matching-based tools, which inspect each version's code for the vulnerable signature, are penalized for correctly reporting such versions as unaffected, systematically widening the reported gap between the two paradigms. If this ground-truth model is violated for a substantial fraction of the 1,128 CVEs, the headline 'no tool exceeds 45.0% accuracy' becomes an artifact of the evaluation's own assumption rather than a robust conclusion about real-world effectiveness. The paper itself acknowledges multi-branch development as a major challenge (Finding 8), yet the ground truth does not account for branch-specific propagation of the inducing commit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper constructs a new benchmark of 1,128 real-world C/C++ vulnerabilities with manually annotated affected-version ranges, and uses it to evaluate 12 tools (6 tracing-based, 6 matching-based) for vulnerability-affected version identification. The evaluation is multi-dimensional: vulnerability-level accuracy, version-level precision/recall/F1, root-cause analysis of false positives/negatives, sensitivity to patch type/scope/branching, and ensemble strategies. The central conclusion is that no individual tool exceeds 45% vulnerability-level accuracy, and that even tuned ensembles remain below 60%, indicating fundamental limitations in current architectures. The authors release their benchmark and replicated tool code.","tokens_in":26005,"tokens_out":4037,"duration_ms":41937,"significance":"If the benchmark and evaluation are accepted, this is a valuable reference point for a practically important but under-evaluated task. The study is larger in scope than prior evaluations, includes both tracing- and matching-based tools selected via a systematic literature review, and provides a useful stage-by-stage decomposition of tracing workflows. The double annotation with Cohen's Kappa 0.83 and the public release of data and code are explicit strengths. The negative result, if robust, would be an important caution for downstream N-day risk assessment and exploit-propagation work. However, the central quantitative claim depends heavily on the ground-truth model, which is not independent of the tracing tools being evaluated; this needs to be addressed before the headline numbers can be considered reliable.","major_comments":[{"comment":"The benchmark labels every released version between the vulnerability-inducing commit and the fixing commit as vulnerable, and the manual annotation validates the inducing and fixing commits, not each intermediate release (Section III-B.3: 'All versions between the inducing-commit and patch-commit are labeled as vulnerable'). This is exactly the interval model that V-SZZ and the other tracing-based tools implement. Tracing tools are therefore rewarded by construction for emitting intervals matching the ground truth, while matching-based tools, which inspect each version's code, are penalized when an intermediate release never contained the vulnerable logic (e.g., in multi-branch development or after backported fixes). Since 6 of the 9 projects in Table II are multi-branch, and Finding 8 reports large performance drops in multi-branch settings, this is not a hypothetical concern. The main","section":"III-B.3 and IV-A (Table III)"},{"comment":"The ensemble results are obtained by searching over many tool combinations and then reporting the best-performing configuration. There is no held-out set, no cross-validation, and no correction for multiple comparisons. The reported 55.0% vulnerability-level accuracy is therefore a selection optimum, not an unbiased estimate of what a practitioner should expect from that ensemble. The observation that LLM4SZZ+ appears in all leading combinations may also be a selection artifact. Please report out-of-sample performance (e.g., leave-one-project-out or a fixed pre-specified ensemble) and provide the distribution of ensemble accuracies over the search space.","section":"IV-D (Table VI)"},{"comment":"The threats-to-validity section states that manual labeling may introduce error and that it was mitigated by double annotation, but it does not address the issue that the label definition itself is the continuous-range model, which is not equivalent to per-version validation. This is distinct from annotator error and should be discussed explicitly. The paper's own RQ3 results show that multi-branch development is common and degrades tools, which increases the plausibility that continuous-range labels are wrong for a non-negligible fraction of intermediate versions. Please acknowledge this as a threat and provide evidence on how often the continuous-range assumption was checked or violated.","section":"V-D.2 (Internal Validity)"}],"minor_comments":[{"comment":"The text states the dataset covers '59,983 vulnerable versions,' but Table II's total is 59,187. Please reconcile.","section":"Section III-B.3 / Table II"},{"comment":"The sentence 'While V-SZZ incorporates cross-branch information when identifying affected versions, the general impact of single-branch versus multi-branch patches on tool performance has not been fully explored/determined' is duplicated with minor wording changes. Keep one.","section":"Section IV-C Setup"},{"comment":"Typo: 'accurace' should be 'accuracy'; 'Tools may yield both FNs and FNs' should be 'both FPs and FNs'.","section":"Section IV-A.2"},{"comment":"The tool is inconsistently named 'SEM-SZZ' in Table I and 'Sem-SZZ' in the text/references (e.g., ref [20]). Also, LLM4SZZ is cited as an arXiv preprint rather than a peer-reviewed publication; this should be flagged in the tool-selection description.","section":"Table I and References"},{"comment":"The manual root-cause analysis on 100 vulnerabilities reports counts such as '49 instances' and '16 of these 16 cases,' but no inter-rater agreement is reported for this qualitative coding. Given that the root-cause findings support the 'fundamental limitations' narrative, adding agreement statistics would strengthen the analysis.","section":"Section IV-B.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong empirical core and a useful released artifact, but the main quantitative result should not be published as is because the ground-truth interval model is the same as the tracing tools' output model. The authors can reasonably fix this with a focused sensitivity analysis or per-version validation on a sample. I would also ask the editor to ensure that the artifact URL and code are checked during the revision process, as the analysis depends heavily on the precise benchmark definitions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should look at this one if you track N-day analysis or SZZ variants. The central finding—no tool cracks 45% accuracy at the vulnerability level on a 1,128-CVE benchmark—is new and, in broad strokes, convincing. The benchmark is the largest of its kind, manually annotated with double coding and a Kappa of 0.83, and the authors actually release it. They also evaluate 12 tools from both the tracing and matching paradigms, which is substantially more than prior studies. The stage-wise root cause analysis is genuinely useful: showing that add-only patches and multi-file changes break most tools is concrete and actionable.\n\nThe soft spot is the ground truth. They label every version between the vulnerability-inducing commit and the fixing commit as vulnerable, following the V-SZZ annotation methodology. That is the same continuous-range model the tracing tools implement. So tracing tools get rewarded for emitting an interval, even for versions that never actually contained the vulnerable logic (e.g., branches that didn't get the inducing commit or already received a backported fix). Matching tools, which inspect each version's code, are penalized for correctly reporting such versions as unaffected. The manual validation checks the inducing and fixing commits; it does not independently validate each intermediate release. The paper even reports that multi-branch development is a major challenge (Finding 8), yet the ground truth ignores branch-specific propagation. This doesn't kill the central negative result—tools still look unreliable—but the reported 45% ceiling is not a clean measure of real-world accuracy, and the gap between tracing and matching is partly baked into the label definition.\n\nAlso, the ensemble gains in RQ4 are selected post hoc from the same dataset that produced the individual scores, so the headline 10.1% improvement is likely optimistic. No error bars or significance tests either. Minor nit: the text says 59,983 affected versions while Table II says 59,187.\n\nNone of this is fatal. The core message—that current tools are far from production-ready—holds up, and the dataset is a real contribution. But the quantitative claims need to be qualified, especially the tracing-versus-matching comparison.\n\nI'd send this to peer review. It deserves a serious referee who can push the authors on the ground-truth circularity and hold out some portion of the data to validate the ensemble selection. For your own reading group, it's worth discussing as an example of how a large benchmark can still inherit assumptions from the tools it evaluates.","headline":"Big new benchmark and a believable negative result, but the ground truth uses the same range model as the tracing tools, so the paradigm comparison is softer than it looks.","tokens_in":26618,"tokens_out":2175,"would_cite":true,"duration_ms":21036,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No affected-version tool tops 45% accuracy","keywords":["vulnerability-affected versions","SZZ algorithm","C/C++ vulnerabilities","benchmark construction","tool evaluation","tracing-based methods","matching-based methods","ensemble strategies"],"falsifier":"Select roughly 100 CVEs from the released benchmark where official security advisories or project logs document that the fix was backported to only some maintenance branches or applied as a partial fix. If the continuous-range labels, all versions between inducing and fixing commits, differ from the documented affected sets for a substantial fraction of those CVEs, the benchmark's ground truth is wrong for those cases. Alternatively, run a tracing tool on a known backport scenario and check whether it labels a version vulnerable that the advisory says is already patched.","tokens_in":1601,"feed_emoji":"🐛","tokens_out":4093,"duration_ms":62294,"temperature":0.7,"pith_summary":"This paper asks how well automated tools can identify which released versions of a C/C++ project are affected by a known vulnerability. The authors built a manually validated benchmark of 1,128 real-world CVEs and ran 12 representative tools on it. They find that no existing tool, whether it traces the vulnerability-inducing commit or matches vulnerable code signatures, correctly predicts the full set of affected versions more than 44.9% of the time, and the best ensemble strategy reaches only 55.0%. The paper argues that this ceiling is structural, not a matter of tuning: the tools share fundamental limitations in semantic reasoning, reliance on patch heuristics, and shallow verification of whether vulnerable logic is actually present. If true, this means security teams cannot currently rely on automated version identification alone and need fundamentally new approaches.","feed_headline":"No affected-version tool tops 45% accuracy","feed_subtitle":"A 1,128-CVE benchmark shows tracing and matching tools fall short; even ensembles stay below 60%.","key_machinery":"The central object is the benchmark: 1,128 CVEs from nine C/C++ projects, each annotated with a ground-truth set of affected versions following the V-SZZ methodology of labeling every version between the vulnerability-inducing commit and the fixing commit as vulnerable. The other key mechanism is the stage-based decomposition framework, which abstracts tracing workflows, statement selection, commit tracing, inducing-commit identification, affected-version inference, and matching workflows, signature construction, matching, into comparable steps. This decomposition is what lets the authors attribute false positives and false negatives to specific design choices rather than treating each tool","core_discovery":"The paper establishes that current automated vulnerability-affected version identification is far from reliable. Using a newly built benchmark of 1,128 C/C++ vulnerabilities with manually validated affected-version ranges, the authors evaluate 12 tools spanning tracing-based methods, which extend SZZ to find the vulnerability-inducing commit and map it to versions, and matching-based methods, which extract code signatures from the fix and search for them in historical versions. At the vulnerability level, where a prediction is correct only if it exactly matches the ground-truth set of affected versions, the best tool achieves 44.9% accuracy. At the version level, precision and recall are hig","pith_inferences":["The continuous-range ground truth, all versions between inducing and fixing commits are vulnerable, mirrors the internal assumption of tracing tools, so the benchmark may systematically favor tracing-based methods; annotating backported or partially fixed CVEs separately would test whether the relative ranking survives.","A concrete next experiment: take a sample of CVEs where official advisories document branch-specific backports, and check whether the benchmark labels match the advisories; if they disagree often, the reported accuracy numbers are distorted for a non-trivial fraction of the 1,128 cases.","The finding that multi-file patches inject noise suggests that patch modularization, partitioning a fix into semantic units, could be a more productive direction than improving individual matching or tracing algorithms.","The paper's released benchmark enables a direct test of its strongest claim: another group can re-run the same tools on a subset restricted to single-branch, deletion-based patches and check whether the 45% ceiling rises, which would confirm that the limit is tied to patch complexity rather than tool implementation quality."],"forward_implications":["Security teams should not treat any single current tool as authoritative for deciding which versions need patching, since the best tool is wrong about the full affected set more than half the time.","Ensemble strategies, especially majority voting across five tools, offer a practical near-term improvement, up to 55.0% accuracy, but cannot break the 60% ceiling, so the problem needs architectural innovation, not just combination.","Add-only patches, which make up 21.3% of the benchmark, break most tracing tools because those tools anchor on deleted lines; future tools must handle repairs that do not delete faulty code.","Multi-branch development is a major failure mode for both paradigms, so cross-branch patch reuse and branch-aware version inference should be a priority for new methods.","LLMs help most in early-stage statement selection, but LLM-based commit selection degrades performance; the best hybrid combines LLM selection with conventional heuristic commit tracing."],"supporting_citations":[{"why":"Supplies the V-SZZ annotation methodology used to label affected versions between the inducing and fixing commits.","marker":"[19]"},{"why":"VCCFinder is the best-performing standalone tool and defines the tracing-based approach the study evaluates.","marker":"[17]"},{"why":"The original SZZ algorithm that all tracing-based tools build upon.","marker":"[23]"},{"why":"LLM4SZZ is the LLM-based tool whose modular recomposition yields the best hybrid variant.","marker":"[22]"},{"why":"VISION exemplifies matching-based affected-version identification and motivates that paradigm's inclusion in the comparison.","marker":"[13]"},{"why":"Prior SZZ evaluation that the study contrasts with its broader multi-tool scope.","marker":"[30]"}],"fun_headline_variants":["Vuln version ID tools: best hits 45% accuracy","Affected-version tools top at 45% on 1,128-CVE test","Study: No affected-version tool beats 45% accuracy","Even ensembles fall short: <60% on CVE version ID","Benchmark reveals affected-version ID tools max 45%"],"cache_read_input_tokens":28288,"weakest_assumption_plain":"The benchmark labels every released version between the vulnerability-inducing commit and the fixing commit as vulnerable, and if many real-world CVEs are backported, partially fixed, or fixed differently across branches, this ground truth does not match reality and the reported accuracy numbers are distorted.","fun_headline_variants_meta":{"raw":{"variants":["Vuln version ID tools: best hits 45% accuracy","Affected-version tools top at 45% on 1,128-CVE test","Study: No affected-version tool beats 45% accuracy","Even ensembles fall short: <60% on CVE version ID","Benchmark reveals affected-version ID tools max 45%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1312,"prompt_tokens":739,"completion_tokens":573,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":493}},"tokens_in":483,"tokens_out":573,"duration_ms":5517,"temperature":1.0,"reasoning_tokens":493,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:34:11.551424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Select roughly 100 CVEs from the released benchmark where official security advisories or project logs document that the fix was backported to only some maintenance branches or applied as a partial fix. If the continuous-range labels, all versions between inducing and fixing commits, differ from the documented affected sets for a substantial fraction of those CVEs, the benchmark's ground truth is wrong for those cases. Alternatively, run a tracing tool on a known backport scenario and check whether it labels a version vulnerable that the advisory says is already patched.","supporting_citations":[{"cited_title":"V-szz: Automatic identification of version ranges affected by cve vulnerabilities,","cited_arxiv_id":null,"evidence_quote":"Supplies the V-SZZ annotation methodology used to label affected versions between the inducing and fixing commits."},{"cited_title":"Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits,","cited_arxiv_id":null,"evidence_quote":"VCCFinder is the best-performing standalone tool and defines the tracing-based approach the study evaluates."},{"cited_title":"LLM4SZZ: Enhancing SZZ Algorithm with Context-Enhanced Assessment on Large Language Models","cited_arxiv_id":"2504.01404","evidence_quote":"LLM4SZZ is the LLM-based tool whose modular recomposition yields the best hybrid variant."},{"cited_title":"Vision: Identifying affected library versions for open source software vulnerabilities,","cited_arxiv_id":null,"evidence_quote":"VISION exemplifies matching-based affected-version identification and motivates that paradigm's inclusion in the comparison."},{"cited_title":"Exploring and exploiting the correlations between bug-inducing and bug- fixing commits,","cited_arxiv_id":null,"evidence_quote":"Prior SZZ evaluation that the study contrasts with its broader multi-tool scope."}],"review_version":1}