{"id":"8bee5d51-89ed-410b-868c-1b80eb62e7b5","arxiv_id":"2501.14983","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM4VFD combines LLM-generated summaries of commit intent, issue/PR context, and retrieved historical fixes to detect vulnerability-fixing commits, reporting F1 gains of 68.1% to 145.4% over prior PLM baselines.","lead":"This paper introduces LLM4VFD, a framework that asks large language models to summarize a commit's intent, pull in related issue reports, and retrieve similar past vulnerability fixes before deciding whether the commit fixes a vulnerability. It reports large gains over prior commit-classification tools on a new post-2023 dataset and a small expert study.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contamination control filters by CVE publication date, not commit date, so BigVulFixes may include fixes LLMs memorized; the headline F1 gains may not reflect generalization.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the post-2023 selection is meant to prevent LLM pretraining leakage, but the paper does not verify that the selected commits are actually outside the models' training data. My reading sharpens this by noting that the selection criterion is CVE publication date, not commit date, so CVE publication lag alone can defeat the stated control even if the CVE date is correct. This matters because the central quantitative claim—68.1%–145.4% F1 improvement over PLM baselines—depends entirely on the integrity of the BigVulFixes evaluation. If memorization is present, both the absolute performance and the comparison to PLMs are suspect. The proposed date-filtered re-run is a direct, feasible test that would settle the issue. I also considered the separate concern that PLM baselines are evaluated off-the-shelf without fine-tuning or threshold calibration on BigVulFixes, which could also inflate the reported gap; that is a real secondary threat, but the contamination issue is more fundamental because it undermines the dataset's stated leakage control and affects all RQ1 comparisons simultaneously. The current CONDITIONAL verdict remains appropriate: the concern is addressable but unverified, and the proposed check should be required before the headline claim is accepted.","tokens_in":21401,"tokens_out":6991,"duration_ms":69538,"concrete_test":"For every commit in BigVulFixes, record the GitHub commit date and the model-specific knowledge-cutoff date; re-run RQ1 on the subset of commits whose commit date is later than every studied model's cutoff (e.g., after 2024-06-01). If the F1 advantage over the best PLM baseline and over vanilla LLMs persists on this uncontaminated subset, the central claim survives; if the gap shrinks or reverses, date-based contamination is driving the reported gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2.1 motivates the post-2023 cutoff by LLM knowledge-cutoff data leakage, but the filter is applied to CVE publication dates, and the actual vulnerability fix commits are collected from NVD references. A CVE published in 2024 can reference a commit made in 2022 or 2023, especially for silent fixes disclosed later; the paper never reports commit dates or verifies that the diffs and commit messages in BigVulFixes are absent from each LLM's pretraining corpus. Llama3.1, Qwen2, and DeepSeek-Coder-V2 have different knowledge cutoffs, and at least some of them plausibly include early-2024 GitHub data. If a non-trivial fraction of test commits were memorized, the vanilla LLM baseline would receive label information directly from pretraining, and the relative gains of LLM4VFD—and especially the gap over PLM baselines, which cannot benefit from post-cutoff memorization—could be inflated by memorization rather than by CCI/DA/HV reasoning. Because the BigVulFixes evaluation is the sole support for RQ1, this unverified date-based contamination control is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM4VFD, a framework for detecting vulnerability-fixing (VF) commits that combines three sources of LLM-distilled context: a Code Change Intention (CCI) summary, a Development Artifact (DA) summary derived from issue reports and pull requests, and a Historical Vulnerability (HV) retrieval component. These are merged in a Comprehensive Analysis and Vulnerability Fix Detection (CAVFD) prompt that asks an LLM to produce a yes/no vulnerability_fix judgment together with a free-text analysis. The authors introduce a new dataset, BigVulFixes, containing 1,689 VF commits and 26,468 non-VF commits sampled at 1:16, and they evaluate LLM4VFD on six LLMs from the Llama, Qwen, and DeepSeek families. They report F1 gains of 68.1%–145.4% over the best PLM baseline and 12.7%–105.6% over vanilla LLMs, plus an ablation study and a user study with ten security experts. The central claim is that combining CCI, DA, and HV context improves VF detection and provides useful explanations.","tokens_in":21673,"tokens_out":5910,"duration_ms":54285,"significance":"If the reported results withstand scrutiny, the paper would be a useful contribution to vulnerability fix detection: it is the first systematic LLM-based treatment of this task in the manuscript, it evaluates a multi-source prompting framework rather than a single prompt, and it releases a new post-2023 dataset and a replication package, which are concrete assets for the community. The multi-family, multi-size LLM evaluation and the ablation of each component are appropriate experimental structures. The user study, although flawed as a controlled experiment, is a reasonable first step toward assessing explainability utility. The main value is in demonstrating that artifacts beyond the raw diff—issue reports, pull requests, and historical fixes—can improve LLM-based VF detection, and that smaller LLMs benefit more from the framework. However, the reliability of the quantitative claims depends on resolving the contamination-control gap, the potential label leakage in the CCI prompt, and the absence of any uncertainty quantification.","major_comments":[{"comment":"The post-knowledge-cutoff control is applied to CVE publication dates, not to commit dates, so the claim that BigVulFixes is outside LLM pretraining data is not established. NVD references for a CVE published in 2024 can point to commits made in 2022 or 2023, exactly the silent-fix cases this paper targets; the paper never reports commit dates or checks whether the diffs and commit messages are absent from the training corpora of Llama3.1, Qwen2, and DeepSeek-Coder-V2. Because Table 1 is the sole support for RQ1, and vanilla LLM baselines can benefit from memorized fixes while PLM baselines cannot, the headline F1 gains over PLMs and vanilla LLMs could be inflated by memorization. The authors should filter by commit timestamps relative to each model's knowledge cutoff, run contamination probes, or report an analysis restricted to commits demonstrably after each model's training data collection window.","section":"Section 5.2.1 and Section 5.4"},{"comment":"The CCI prompt asks the LLM to categorize the 'Purpose of the Change' into categories that include 'fixing a vulnerability,' and that same CCI output is then inserted into the CAVFD prompt that asks for the final 'vulnerability_fix' label. The final decision can therefore be a restatement of a label that the same model already produced, rather than an independent judgment on the patch; this is a form of label leakage that differentially benefits LLM4VFD over the vanilla baseline. It also biases the HV retrieval, whose query is the CCI summary. To support the claim that CCI contributes reasoning rather than leaking the answer, the CCI prompt should avoid target-label vocabulary, or the evaluation should measure how often the CCI output contains the phrase 'fixing a vulnerability' before the CAVFD prediction and condition on its absence.","section":"Section 4.1, Figure 5 and Section 4.4, Figure 7"},{"comment":"All results are single point estimates with no variance, confidence intervals, or significance tests, despite stochastic LLM decoding and random non-VF sampling at a 1:16 ratio. The phrase 'significantly outperforms' is therefore not supported by the reported evidence. The authors should report repeated runs with different seeds or sample draws, and use paired tests such as bootstrap or McNemar for the RQ1 and RQ2 comparisons.","section":"Section 5.6, Table 1, Table 2"},{"comment":"The user study cannot support the claim that LLM4VFD's analysis improves efficiency, because there is no control condition in which participants see the raw commit without the generated analysis, no measurement of decision time or accuracy, and all 40 cases are positive VF examples. The yes/no questions are retrospective self-reports. The authors should compare conditions with and without the generated analysis on a mixed VF/NVF sample, ideally measuring decision time and verification accuracy.","section":"Section 5.7.3 and Section 6.3"}],"minor_comments":[{"comment":"The dataset is described as containing vulnerabilities 'after 2023,' but Section 6.3 discusses CVE-2023-48014, CVE-2023-37061, and CVE-2023-48657; please clarify whether BigVulFixes includes 2023 CVEs or whether the user study used a different sample.","section":"Section 5.2.1 versus Section 6.3"},{"comment":"The abstract says LLM4VFD comprises three components, while Section 4 describes four components including CAVFD; please align the naming convention.","section":"Abstract and Section 4"},{"comment":"Please clarify the grouping of the three PLM rows under 'CodeBERT' and the meaning of the asterisk on CoLeFunDa; the current layout makes it easy to misread the foundation model column.","section":"Table 1"},{"comment":"There are typographical inconsistencies, including 'CoLeFunDa' versus 'ColeFunda'/'CoLeFunda', 'VulCurator' versus 'Vulcurator', and 'an 15.4% reduction'; a careful proofreading pass is needed.","section":"Throughout"},{"comment":"The reference URLs for CVE-2023-37061, CVE-2023-48014, and CVE-2023-48657 point to CVE-2024-37061, CVE-2024-48014, and CVE-2024-48657, respectively; the identifiers should be corrected.","section":"References [44]-[46]"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution with a plausible central idea, and the main concerns are addressable within the scope of a revision: verifying the contamination control using commit dates, redesigning the CCI prompt to avoid leaking the target label, and adding uncertainty quantification. I would not reject, but the current version overstates the strength of the evidence, particularly because the BigVulFixes evaluation is the sole support for RQ1 and the date-based leakage control is not verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a solid engineering contribution with a genuinely useful new dataset (BigVulFixes), but the headline numbers need a robustness pass. The paper is the first to apply LLMs with CoT and ICL to vulnerability fix detection, and it does the obvious-but-not-yet-done thing of combining an LLM-generated change-intention summary, issue/PR context, and retrieved historical fixes. That combination, plus the human-readable explanation output, is a real step forward for practical screening. The evaluations across six LLMs, the ablation, and the user study add up to a reasonable first take on where the value sits. What the paper does well: it ships a new post-2023 dataset of 1,689 VF commits and 26,468 non-VF commits across seven languages, and it makes the replication package available. The ablation is informative: CCI matters most, DA and HV matter less but still help. The user study is small but the 80% helpfulness result is at least suggestive, and the failure analysis gives an honest picture of the limits. The soft spots, in order of seriousness. First, the contamination control is weaker than the paper claims. Section 5.2.1 says they restrict to vulnerabilities after 2023, but the filter is applied to CVE publication dates, not fix-commit dates. A CVE published in 2024 can reference a commit from 2022 or 2023, and the paper never checks whether the actual diffs and commit messages in BigVulFixes fall after each model's knowledge cutoff. For Llama3.1, Qwen2, and DeepSeek-Coder-V2, early-2024 GitHub data is plausibly in training. If a non-trivial fraction of test commits were memorized, the gap over PLMs—which cannot benefit from memorization—is inflated. This is load-bearing for RQ1. Second, the CCI component risks leaking the label. The CoT prompt explicitly asks the model to categorize the purpose as refactoring, feature enhancement, or fixing a vulnerability, and that summary is then fed directly into the final CAVFD prompt. The final answer may just be echoing the CCI's purpose category. The ablation showing CCI has the largest effect is consistent with this leak. Third, the user study has no control group, uses only positive examples, and reports no inter-rater reliability. That is a minor issue relative to the first two. None of this is fatal. The date issue can be addressed by reporting commit dates and running contamination probes; the CCI leak can be addressed by generating the summary without the final label or by testing a version where the purpose category is masked. The dataset and the framework deserve a serious referee. I would send this out, but I would expect the reviewers to push for those robustness checks before acceptance.","headline":"A useful dataset and a plausible LLM-based framework for vulnerability fix detection, but the headline gains rest on a leak-prone CCI design and an unverified date-based contamination control; both are fixable in revision.","tokens_in":794,"tokens_out":1899,"would_cite":true,"duration_ms":32635,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that feeding an LLM a distilled change intention, linked issue and pull-request context, and a retrieved similar historical fix detects vulnerability-fixing commits with F1 up to 0.54, beating prior tools by 68–145% and…","keywords":["vulnerability fix detection","large language models","chain-of-thought prompting","in-context learning","retrieval-augmented generation","development artifacts","tangled commits","BigVulFixes"],"falsifier":"Run a memorization probe: for each 2024 CVE fix commit in BigVulFixes, strip the fix-specific lines or ask the LLM to name the CVE from the raw diff; if the models can complete or identify the fixed vulnerability at high rates, the date-cutoff assumption fails and the F1 gains are suspect. Alternatively, rebuild the same evaluation with pre-2023 fixes and no historical-retrieval component; if the advantage over vanilla LLMs collapses, the post-2023 design is carrying the result.","tokens_in":21254,"feed_emoji":"🛡️","tokens_out":7790,"duration_ms":76191,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models, when fed three kinds of context—a distilled intention of the code change, linked issue reports and pull requests, and a retrieved similar historical vulnerability fix—can detect vulnerability fix commits much more effectively than prior specialized machine learning tools, and can say why in a form security experts find useful. The motivating problem is that vulnerability fixes are often disclosed only after a delay, so downstream users need to spot the fixing commit as soon as it lands; existing code-only methods miss tangled commits and silent fixes. If true, the result matters because it would give open-source maintainers a practical screening tool that both flags candidate fixes and explains its reasoning, and it would show that context, not just patch text, is the key signal for this task.","feed_headline":"Context-rich prompts lift vulnerability-fix detection by up to 145%","feed_subtitle":"Adding change intent, issue reports, and similar past fixes beats code-only tools and helps experts screen faster.","key_machinery":"The load-bearing device is a structured LLM-generated \"three-aspect summary\"—code change summary, purpose, and implications—produced by Chain-of-Thought prompts, applied both to the commit and to its linked issue/PR artifacts, and embedded to retrieve a similar historical vulnerability fix whose CVE description and summary are fed into a final Comparison-and-Analysis prompt. That machinery converts raw diffs into intention-level representations, which is what lets the model see through tangled commits, vague commit messages, and subtle one-line checks.","core_discovery":"The central claim is that vulnerability fix detection can be recast as a context-enriched LLM reading task: instead of classifying a raw diff, the framework first has the LLM write a three-aspect summary of the commit (what changed, why, and what it implies), summarizes any linked issue report or pull request the same way, retrieves the nearest historical vulnerability fix by embedding those summaries, and then asks the LLM to compare and decide. On a new post-2023 dataset of 1,689 vulnerability fixes and 26,468 non-fixes across seven languages, the best configuration reaches F1 0.54 and MCC 0.52, outperforming the best prior tool by 68.1–145.4% in F1 and the same LLM without the added context by 12.7–105.6%. The paper also reports that the generated analysis helped security experts understand commit intent in 95% of the 40 cases tested and improved their efficiency in identifying fixes in 80%.","pith_inferences":["Editorial inference: because the paper relies on calendar dates rather than contamination probing, the cleanest test of the result is a memorization check; until then, part of the gain could come from the LLMs having seen these CVEs.","Editorial inference: the three-aspect summary is a reusable abstraction; it could be applied to other commit classification problems such as refactoring detection or bug versus feature classification.","Editorial inference: the failure analysis shows the historical retrieval can actively mislead the model, so a stronger retriever with reranking should improve both precision and recall; the framework makes this a modular swap."],"forward_implications":["On the BigVulFixes test set, every LLM tested, from 7B to 236B parameters, beats the strongest prior approach on F1, MCC, and recall, so the gain is not tied to one model family.","Smaller models improve the most—on average 64% F1 gain versus 14.4% for larger ones—so the approach can make modest-size models competitive for this task.","Ablation shows all three components add value, with the intention summary contributing most to precision; removing it drops precision by 13–15%.","The explanations are usable: 10 security experts said the analysis helped them understand the intent in 95% of cases and improved screening efficiency in 80% of cases."],"supporting_citations":[{"why":"Baseline code-change-only vulnerability fix detector that LLM4VFD must beat; defines the task.","marker":"[72]"},{"why":"Baseline function-level silent fix identifier; included in the evaluation.","marker":"[71]"},{"why":"Best prior baseline that adds issue reports; target of the 68.1–145.4% F1 comparison.","marker":"[40]"},{"why":"Supplies Chain-of-Thought prompting used to generate the three-aspect summaries.","marker":"[64]"},{"why":"Supplies In-Context Learning mechanism used in the final detection prompt.","marker":"[13]"},{"why":"NVD is the source of CVE ground truth for BigVulFixes and the historical vulnerability database.","marker":"[43]"},{"why":"Supplies the sentence embedding model that vectorizes three-aspect summaries for retrieval.","marker":"[34]"},{"why":"Supplies the 1:16 sampling strategy used to build the imbalanced BigVulFixes dataset.","marker":"[16]"},{"why":"Defines tangled commits, the motivating failure mode for the intention component.","marker":"[4]"}],"fun_headline_variants":["Context-rich LLM framework boosts vulnerability-fix detection by 145%","LLM4VFD: adding intent and history lifts fix detection up to 145%","LLM explanations help experts spot fixes faster; detection up 145%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that choosing vulnerabilities whose CVE entries were published after 2023 keeps the test cases out of what the LLMs memorized during training, so the measured gains reflect reasoning rather than recall; the paper does not verify this with contamination checks.","fun_headline_variants_meta":{"raw":{"variants":["Context-rich LLM framework boosts vulnerability-fix detection by 145%","LLM4VFD: adding intent and history lifts fix detection up to 145%","LLM explanations help experts spot fixes faster; detection up 145%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000755,"raw_usage":{"total_tokens":3424,"prompt_tokens":1079,"completion_tokens":2345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":2279}},"tokens_in":695,"tokens_out":2345,"duration_ms":28889,"temperature":1.0,"reasoning_tokens":2279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:43:38.093987+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a memorization probe: for each 2024 CVE fix commit in BigVulFixes, strip the fix-specific lines or ask the LLM to name the CVE from the raw diff; if the models can complete or identify the fixed vulnerability at high rates, the date-cutoff assumption fails and the F1 gains are suspect. Alternatively, rebuild the same evaluation with pre-2023 fixes and no historical-retrieval component; if the advantage over vanilla LLMs collapses, the post-2023 design is carrying the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline code-change-only vulnerability fix detector that LLM4VFD must beat; defines the task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline function-level silent fix identifier; included in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Best prior baseline that adds issue reports; target of the 68.1–145.4% F1 comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Chain-of-Thought prompting used to generate the three-aspect summaries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NVD is the source of CVE ground truth for BigVulFixes and the historical vulnerability database."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 1:16 sampling strategy used to build the imbalanced BigVulFixes dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines tangled commits, the motivating failure mode for the intention component."}],"review_version":1}