{"id":"3eccabec-a1bd-44bc-a04f-71c479e7055b","arxiv_id":"2507.10729","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Just-in-time vulnerability prediction models that look strong on balanced datasets perform near-randomly on realistic, imbalanced commit streams.","lead":"This study tests eight just-in-time vulnerability prediction models on a new one-million-commit dataset from Linux and FFmpeg. It finds their performance collapses by up to 98% when evaluated on all commits instead of only on carefully balanced vulnerability-related commit pairs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The realistic setting's ground truth labels all VNCs as safe; latent vulnerabilities among these could heavily bias PR-AUC downward, so the reported 89-98% performance drop is not yet established.","rationale":"The paper makes a substantial empirical contribution: a large public dataset, replication of eight JIT-VP models, and a clear demonstration that class distribution matters for evaluation. The reader's CONDITIONAL verdict is well calibrated. The single most load-bearing concern is the assumption that all VNCs are safe, because the PR-AUC and MCC values that support the headline 89-98% drop are computed against this ground truth. The paper's Section V-B acknowledgment of latent vulnerabilities does not neutralize the concern; a small label-error rate in the negative class can have a large effect on precision-oriented metrics in highly imbalanced settings, and the paper supplies no sensitivity analysis. This is a correctness risk, not a consensus disagreement, and it directly affects the central quantitative claim. However, the concern does not falsify the qualitative direction of the finding, since even a corrected evaluation is likely to show meaningful degradation; it only means the precise magnitudes should be treated as provisional until the labels are validated. Table I also contains arithmetic inconsistencies in partition sums and the text reports conflicting imbalance ratios (1:217 vs 1:127), further supporting the need for a corrected and re-validated dataset before the numbers are taken at face value. These issues together leave the reader's CONDITIONAL verdict unchanged.","tokens_in":20780,"tokens_out":5989,"duration_ms":72141,"concrete_test":"Run a temporal validation on the Linux test split: collect commits dated after the September 2024 dataset cutoff that reference a security-fix commit (e.g., via a 'Fixes:' tag or a new CVE/NVD link), re-label those referenced VNCs as vulnerability-introducing, and recompute PR-AUC and MCC for the Table VII models with the corrected ground truth. If the corrected PR-AUC remains below 0.1, the concern is settled; if it rises substantially, the reported decline is partly an artifact of negative-label noise. As a complementary audit, have two annotators manually label a random sample of 1,000 VNCs to estimate the latent-vulnerability rate and bound its effect on the metrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-B defines VNCs as the residual commits that are neither VFCs nor VICs, and the realistic setting treats all VNCs as safe. Section V-B concedes that latent vulnerabilities may exist among VNCs, but dismisses the impact without quantification. This concession is load-bearing because the test sets contain 157,039 (Linux) and 37,778 (FFmpeg) VNCs against only 922 and 1,020 known VICs. In PR-AUC computation, a latent vulnerability that receives a high model score is counted as a false positive and does not contribute to recall, so even a small absolute number of mislabeled VNCs can substantially depress the measured curve when the positive class is extremely rare. For example, if 5% of Linux VNCs were actually vulnerability-introducing, the positive count would rise from 922 to roughly 8,900 and the imbalance ratio would shift from about 1:172 to about 1:17, a regime the paper itself treats as far less damaging. Thus the headline quantitative claim that PR-AUC declines by 89-98% could overstate the true capability gap and unfairly conclude that current JIT-VP models are unusable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that existing just-in-time vulnerability prediction (JIT-VP) evaluations are unrealistic because they train and test only on vulnerability-introducing commits (VICs) and vulnerability-fixing commits (VFCs), omitting the large majority of ordinary commits. The authors construct a dataset of over one million commits from FFmpeg and the Linux kernel, label the residual commits as vulnerability-neutral commits (VNCs), and evaluate eight JIT-VP models in an idealized setting (VICs vs. VFCs) and a realistic setting (VICs vs. VFCs plus VNCs). They report that average PR-AUC drops by roughly 89-98% in the realistic setting, that class imbalance is the primary cause, and that five common imbalance mitigation techniques (focal loss, random oversampling, random undersampling, SMOTE, OSS) fail to restore performance. The paper concludes that current JIT-VP models are not ready for practical deployment and calls for domain-specific imbalance solutions.","tokens_in":20991,"tokens_out":7696,"duration_ms":94026,"significance":"The dataset and replication package are substantial contributions, and the core question—whether JIT-VP results transfer from balanced VIC/VFC benchmarks to full commit streams—is important for the field. The paper is one of the first to quantify this gap at scale, and its negative results on off-the-shelf imbalance mitigation techniques are a useful caution for practitioners. However, the headline quantitative claim is overstated because PR-AUC is prevalence-dependent: comparing PR-AUC across settings with very different base rates conflates a change in class prevalence with a change in model skill. The residual VNC labeling also injects potentially large label noise into the realistic setting. The paper's strengths—public data, reproducible baselines, broad model coverage—make the underlying question worth pursuing, but the central claim needs reframing and additional analysis before it can be accepted at face value.","major_comments":[{"comment":"PR-AUC is a prevalence-dependent metric: for a random ranking, expected PR-AUC equals the positive-class prevalence. In the idealized test sets the positive prevalence is about 0.35, whereas in the realistic test sets it is approximately 0.025 for FFmpeg and 0.0058 for Linux. The reported drops from 0.833 to 0.091 and from 0.805 to 0.016 therefore partly reflect the change in base rate rather than a loss of ranking ability. Relative to the random baseline, the models are not 90% worse: the realistic FFmpeg PR-AUC of 0.091 is about 3.6 times the prevalence baseline, and the Linux value of 0.016 is about 2.8 times its baseline, while the idealized PR-AUCs are only about 2.4 times their baseline. The paper should re-express the central decline using prevalence-normalized quantities, such as lift over random, precision at fixed recall, or the ROC-AUC values already reported, and separate the ranking and threshold components of the drop.","section":"Section IV-B, Tables VI and VII"},{"comment":"Table I is internally inconsistent, which undermines the paper's stated imbalance ratios. For Linux, the training row sums to 802,161 rather than 800,426; the validation row sums to 35,933 rather than 35,317; and the testing row sums to 159,652 rather than 157,961. The 'All Commits' row also fails to sum: 9,715 + 8,996 + 1,069,933 = 1,088,644, not 1,081,882, and the partition totals for VFCs imply 8,794 VFCs, not 8,996. Separately, the vulnerable-to-safe ratios quoted in the text are inconsistent with the table: Section I states ratios of 1:17 and 1:217, while Section IV-B states 1:17 and 1:127; the test-set ratios computed from the printed counts are about 1:39 for FFmpeg and 1:172 for Linux. Since class imbalance is the paper's main explanatory mechanism, the authors must correct the table and reconcile all stated ratios.","section":"Table I"},{"comment":"The residual labeling of all non-VFC/non-VIC commits as 'vulnerability-neutral' is load-bearing for the realistic-setting results. The test sets contain 157,039 (Linux) and 37,778 (FFmpeg) VNCs against only 922 and 1,020 known VICs, so even a small percentage of mislabeled latent vulnerabilities among VNCs would change both the positive count and the shape of the PR curve. The paper concedes in Section V-B that latent vulnerabilities are possible but dismisses the impact with the claim that 'a small number of mislabeled instances is unlikely to substantially impact the overall empirical findings' without providing any quantification. The authors should provide a sensitivity analysis—for example, re-estimating PR-AUC and MCC under plausible latent-positive contamination rates (1%, 5%, 10%) using the model's score distribution—or otherwise bound the effect of label noise. Without such an analysis, the magnitude of the realistic-setting performance drop is not established.","section":"Sections III-B and V-B"},{"comment":"The RQ3 results contain an internal inconsistency that weakens the 'mitigation is ineffective' conclusion. In Table VIII, DeepJIT with random undersampling achieves PR-AUC 0.237 on FFmpeg and 0.160 on Linux, yet its MCC is 0.004 and 0.002 and its F1-score is 0.051 and 0.012. This pattern indicates that the model ranks positives reasonably well but produces useless default-threshold predictions. The paper reports these numbers together without explaining the discrepancy. Since PR-AUC and MCC/F1 measure different properties (ranking versus thresholded classification), the authors should specify the decision rule used for MCC/F1, discuss the calibration failure, and avoid presenting PR-AUC gains and threshold-based collapses as equally valid evidence for the same conclusion.","section":"Table VIII"}],"minor_comments":[{"comment":"There are typographical errors: 'vulnerability-neural' in Table I's header and 'vulnerability-introducting'/'vulnerability-neural' in Figure 1's caption should be 'vulnerability-neutral' and 'vulnerability-introducing'.","section":"Table I and Figure 1"},{"comment":"The statement that applying RUS to DeepJIT increases PR-AUC 'from 0.114 to 0.237' on FFmpeg does not match Table VII, which lists DeepJIT's no-sampling FFmpeg PR-AUC as 0.082; please correct the baseline value.","section":"Section V-A"},{"comment":"PR-AUC and MCC are described as 'two new evaluation metrics,' but both are standard in imbalanced classification; suggest rephrasing to 'additional evaluation metrics adopted for imbalanced settings.'","section":"Section III-C"},{"comment":"The selection of the two anchored VICs (VIC-1 and VIC-2) is underspecified; please state how these anchors were chosen and confirm that the resulting time-wise split is strictly chronological and free of leakage.","section":"Section III-B"},{"comment":"The focal loss experiments specify only gamma=2 and do not report the alpha_t class-balance coefficient; please provide the full focal loss configuration used for DeepJIT and SimCom.","section":"Section IV-C"},{"comment":"The paper reports that FFmpeg and Linux have approximately 117,000 and 1,300,000 commits but the curated dataset contains 88,178 and 993,704 commits; please quantify the loss from the filtering steps so readers can assess representativeness.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"This is a useful and timely empirical study with a strong open-science component. The main risk is that the headline PR-AUC drop is partly a base-rate artifact and partly a consequence of the unvalidated VNC residual labeling; both are fixable with additional analyses. I would be willing to accept a revised version that normalizes the performance comparison, corrects the dataset statistics, and provides a sensitivity analysis for label noise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2507.10729. The paper builds the largest public JIT-VP dataset to date (1M+ commits from Linux and FFmpeg), evaluates eight techniques under a balanced “idealized” setting and a full-commit-stream “realistic” setting, and finds that PR-AUC collapses from ~0.8 to below 0.1. It also tests five standard imbalance fixes and shows they don’t restore performance. The dataset and the empirical gap are real contributions, and the paper is honest about its threats to validity, including the latent-vulnerability issue.\n\nThe central finding is directionally convincing: models trained on balanced VIC/VFC data don’t transfer to a realistic commit stream. But I have three concerns, in order of severity.\n\nFirst, the VNC-as-safe assumption is load-bearing and unquantified. The paper concedes latent vulnerabilities may exist among VNCs (Section V-B) but dismisses the impact without numbers. The stress-test math checks out: even 1% of Linux test VNCs would nearly double the positive count (922 to ~1,900), and 5% would shift the ratio from ~1:170 to ~1:17. A mislabeled vulnerable commit scored high by the model counts as a false positive, so PR-AUC is depressed purely by label noise. The 89-98% drop could overstate the capability gap. This needs a sensitivity analysis, not a hand-wave.\n\nSecond, PR-AUC as reported is partly a prevalence artifact. Idealized PR-AUC of 0.8 on a balanced set is roughly 1.6x random; realistic PR-AUC of 0.016 on a ~1:170 set is roughly 2.7x random. By that measure, the models are doing relatively better in the realistic setting, not worse. The paper should report a normalized PR-AUC or at least the random baseline to make the comparison meaningful.\n\nThird, Table I has arithmetic inconsistencies: the partition sums don’t match the stated totals for several rows (FFmpeg training sums to ~46.2K vs 43.7K; Linux validation and testing are off as well). A dataset paper with internally inconsistent counts is hard to trust. Also no error bars or significance tests, and the imbalance-mitigation conclusion rests on a single configuration per method.\n\nThese are fixable. They don’t sink the paper’s main point that the field needs more realistic evaluation, but the headline magnitudes should be treated with caution until the label-noise sensitivity is quantified. I’d send it to review, and I’d ask for the sensitivity analysis and metric fix before accepting the numbers. Worth a reading-group slot.","headline":"Large-scale, honest empirical study showing JIT-VP models collapse on a realistic commit stream, but the headline 89-98% PR-AUC drop is inflated by the VNC-as-safe labeling assumption and the PR-AUC prevalence effect.","tokens_in":21572,"tokens_out":5158,"would_cite":true,"duration_ms":57451,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that just-in-time vulnerability prediction models lose 89–98% of their precision-recall performance when evaluated on a realistic, highly imbalanced commit stream rather than on balanced curated vulnerability data.","keywords":["just-in-time vulnerability prediction","software vulnerability prediction","class imbalance","commit-level prediction","benchmark evaluation","precision-recall AUC","vulnerability-introducing commits","V-SZZ"],"falsifier":"Take a random sample of the vulnerability-neutral commits from the released dataset, apply an independent oracle such as later 'Fixes:' references or manual security review, count how many are actually vulnerability-introducing, and recompute PR-AUC with corrected labels; if the corrected positive rate differs substantially from the paper's, the 89–98% decline is not robust to labeling error.","tokens_in":20559,"feed_emoji":"📉","tokens_out":12554,"duration_ms":124905,"temperature":0.7,"pith_summary":"The paper's central claim is that published evaluations of just-in-time vulnerability prediction (JIT-VP) are unrealistically favorable because they restrict the test set to vulnerability-introducing and vulnerability-fixing commits, leaving out the roughly 90–99% of commits that are neither. To test this, the authors built a dataset of 1,081,882 commits from FFmpeg and the Linux kernel and evaluated eight state-of-the-art predictors under the standard 'idealized' setting and a new 'realistic' setting that adds vulnerability-neutral commits. In the idealized setting the models average a precision-recall AUC (PR-AUC) of 0.833 on FFmpeg and 0.805 on Linux; in the realistic setting those averages fall to 0.091 and 0.016, drops of roughly 89% and 98%. The paper then shows that five common imbalance-mitigation techniques—focal loss, random oversampling, random undersampling, SMOTE, and OSS—fail to meaningfully recover the lost performance. The reason to care is that if this is right, current JIT-VP models are not ready for practical deployment, and the benchmark design itself, not the models, is driving the reported success.","feed_headline":"Realistic data cuts commit-level vulnerability prediction by up to 98%","feed_subtitle":"Balanced benchmarks hide a 89–98% precision-recall collapse; standard imbalance fixes don't recover it.","key_machinery":"The central mechanism is the 'realistic setting': an evaluation protocol in which the test set contains every commit from a project's history—vulnerability-introducing, vulnerability-fixing, and vulnerability-neutral—rather than only vulnerability-related commits. Its load-bearing component is the residual labeling rule: after vulnerability-fixing commits are collected from curated datasets and CVE/NVD records, and vulnerability-introducing commits are traced with V-SZZ and Linux 'Fixes:' tags, all remaining commits are labeled 'vulnerability-neutral' and treated as safe. That single design choice changes the class ratio from roughly balanced (1:0.89 and 1:0.88 vulnerable-to-safe) to severely imbalanced (about 1:17 and 1:127–1:217), and it is what turns PR-AUC from values above 0.8 into values near zero.","core_discovery":"On the paper's own terms, the discovery is that the strong results previously reported for JIT-VP are an artifact of excluding vulnerability-neutral commits (VNCs) from training and test sets. The authors label vulnerability-introducing commits as vulnerable, label vulnerability-fixing and vulnerability-neutral commits as safe, and evaluate eight existing predictors on a dataset of 1,081,882 commits. In the idealized setting, average PR-AUC is 0.833 (FFmpeg) and 0.805 (Linux), with per-model best values of 0.959 and 0.892; in the realistic setting, the averages collapse to 0.091 and 0.016, and even the best idealized model JITFine falls to 0.111 and 0.005. The paper attributes the collapse to the shift in the vulnerable-to-safe ratio from nearly balanced (1:0.89 and 1:0.88) to severely imbalanced (about 1:17 in FFmpeg and between 1:127 and 1:217 in Linux), which biases models toward predicting the majority safe class. It further reports that imbalance mitigation through focal loss, random sampling, SMOTE, or OSS does not restore performance, and in some cases makes it worse.","pith_inferences":["Beyond the paper: any commit-level prediction task that curates negatives to match known positives—such as defect prediction or fix prediction—may be overstating practical performance in the same way; the structure of the negative set, not the model alone, sets the score.","Beyond the paper: because the drop is driven by a change in the base rate, reweighting or resampling the same features cannot recover the lost precision; the results point toward adding vulnerability-specific signal (code semantics, patch context, or external knowledge) rather than further class-balance tuning.","Beyond the paper: a controlled sweep of the fraction of vulnerability-neutral commits (0%, 50%, 90%, 99%) could separate 'mere imbalance' from 'distributional shift'—a gradual decline would suggest threshold-based deployment is still viable, while an abrupt collapse would confirm that the models have not learned vulnerability indicators at all.","Beyond the paper: the paper treats vulnerability-fixing commits as safe; re-annotating VFCs as 'vulnerability-related but not vulnerable' in a follow-up experiment would test whether part of the precision loss is a false-alarm penalty on exactly the commits reviewers should inspect."],"forward_implications":["Published JIT-VP performance numbers from balanced benchmarks (PR-AUC above 0.8) should not be treated as deployable: under the realistic setting the same models average 0.091 and 0.016 PR-AUC.","Model rankings change under the realistic setting; JITFine, the best model in the idealized setting, is no longer best, so future evaluations should compare models on realistic, VNC-inclusive commit streams.","Standard imbalance fixes do not solve the problem; random undersampling gives only marginal and inconsistent gains, focal loss can drive MCC and F1 to zero, and SMOTE/OSS leave the models far below idealized performance.","ROC-AUC hides most of the damage, declining only 16–25% while PR-AUC falls 89–98%, so PR-AUC and MCC are the appropriate metrics for this setting.","The released dataset of over one million commits gives the community a public benchmark for evaluating JIT-VP under a realistic distribution."],"supporting_citations":[{"why":"Defines the JIT-VP task and the VIC/VFC balanced evaluation protocol the paper critiques.","marker":"[3]"},{"why":"Recent JIT-VP approach whose idealized-setting evaluation the paper replicates and contrasts with the realistic setting.","marker":"[4]"},{"why":"The best-performing baseline in the idealized setting; its realistic-setting collapse to PR-AUC 0.111 and 0.005 anchors the central result.","marker":"[8]"},{"why":"Focal loss, one of the five imbalance-mitigation techniques evaluated in RQ3.","marker":"[9]"},{"why":"SMOTE, one of the five imbalance-mitigation techniques evaluated in RQ3.","marker":"[10]"},{"why":"OSS, one of the five imbalance-mitigation techniques evaluated in RQ3.","marker":"[11]"},{"why":"Manually annotated Devign VFC dataset used as one seed source for vulnerability-fixing commits.","marker":"[29]"},{"why":"V-SZZ algorithm used to trace vulnerability-introducing commits from fixing commits; the labeling method behind the dataset.","marker":"[31]"},{"why":"Manually annotated VFC dataset used as another seed source for vulnerability-fixing commits.","marker":"[33]"},{"why":"Developer-informed labeling using Linux 'Fixes:' tags, used to identify vulnerability-introducing commits beyond V-SZZ.","marker":"[35]"}],"fun_headline_variants":["Realistic data slashes commit vulnerability prediction by 98%","JIT vulnerability prediction collapses under real-world imbalance","Balanced benchmarks hide 98% PR-AUC drop in vulnerability prediction","Standard imbalance fixes fail for commit vulnerability prediction","Vulnerability prediction metrics plummet when data imbalance is real"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole realistic setting rests on labeling every commit not identified as vulnerability-introducing or vulnerability-fixing as 'vulnerability-neutral' and therefore safe; if a non-trivial share of those commits actually contain latent vulnerabilities, the ground truth, the imbalance ratios, and the measured performance drops all change.","fun_headline_variants_meta":{"raw":{"variants":["Realistic data slashes commit vulnerability prediction by 98%","JIT vulnerability prediction collapses under real-world imbalance","Balanced benchmarks hide 98% PR-AUC drop in vulnerability prediction","Standard imbalance fixes fail for commit vulnerability prediction","Vulnerability prediction metrics plummet when data imbalance is real"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000359,"raw_usage":{"total_tokens":2010,"prompt_tokens":1076,"completion_tokens":934,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":854}},"tokens_in":692,"tokens_out":934,"duration_ms":8413,"temperature":1.0,"reasoning_tokens":854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:27:02.640846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the vulnerability-neutral commits from the released dataset, apply an independent oracle such as later 'Fixes:' references or manual security review, count how many are actually vulnerability-introducing, and recompute PR-AUC with corrected labels; if the corrected positive rate differs substantially from the paper's, the 89–98% decline is not robust to labeling error.","supporting_citations":[{"cited_title":"Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits,","cited_arxiv_id":null,"evidence_quote":"Defines the JIT-VP task and the VIC/VFC balanced evaluation protocol the paper critiques."},{"cited_title":"Smote for learning from imbalanced data: progress and challenges, marking the 15-year anniversary,","cited_arxiv_id":null,"evidence_quote":"SMOTE, one of the five imbalance-mitigation techniques evaluated in RQ3."},{"cited_title":"Addressing the curse of imbalanced training sets: one-sided selection,","cited_arxiv_id":null,"evidence_quote":"OSS, one of the five imbalance-mitigation techniques evaluated in RQ3."},{"cited_title":"Devign: Effective vul- nerability identification by learning comprehensive program semantics via graph neural networks,","cited_arxiv_id":null,"evidence_quote":"Manually annotated Devign VFC dataset used as one seed source for vulnerability-fixing commits."},{"cited_title":"V-szz: automatic identifi- cation of version ranges affected by cve vulnerabilities,","cited_arxiv_id":null,"evidence_quote":"V-SZZ algorithm used to trace vulnerability-introducing commits from fixing commits; the labeling method behind the dataset."},{"cited_title":"A large-scale empirical study on vulnerability distri- bution within projects and the lessons learned,","cited_arxiv_id":null,"evidence_quote":"Manually annotated VFC dataset used as another seed source for vulnerability-fixing commits."},{"cited_title":"Evaluating szz implementations: An empirical study on the linux kernel,","cited_arxiv_id":null,"evidence_quote":"Developer-informed labeling using Linux 'Fixes:' tags, used to identify vulnerability-introducing commits beyond V-SZZ."}],"review_version":1}