{"id":"1eaf9fc0-63e0-4a87-bf43-2c91ad03858a","arxiv_id":"2506.07390","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ReVD uses synthetic vulnerability reasoning data and curriculum preference optimization to boost LLM vulnerability detection accuracy by 12-22% over prior baselines on PrimeVul and SVEN.","lead":"A new training framework called ReVD improves how large language models spot software vulnerabilities by teaching them to explain why code is buggy, then fine-tuning them to prefer correct explanations over misleading ones. It reports large accuracy gains on two standard vulnerability benchmarks, but it leaves unclear how the training data was separated from the test data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No train/test split is ever defined; Algorithm 1's Eval Set E is unspecified and the COPO round count appears to be selected from test-set curves, so test labels may leak into BVD/T-SFT/COPO and inflate the reported SOTA.","rationale":"The most load-bearing condition for the SOTA claim is clean evaluation: the test sets of PrimeVul and SVEN must be disjoint from all training data and from any signal used to select hyperparameters. The paper never specifies this. Section 4 lists dataset characteristics but no split; Section 3.1 says Daug is generated from D with no indication D is training-only; Algorithm 1 uses an undefined Eval Set E to drive instance selection; and Section 5.3 uses accuracy curves (Figure 3) to pick the number of COPO rounds, with no statement that these curves are on a validation set rather than the test set. If any of these use test labels, the claimed improvements vanish or shrink. The reader's weakest_assumption focused on E; my concern extends the same leakage risk to the base training set D and to round-count selection, hence partial agreement. The proposed check—overlap analysis plus re-running COPO with a proper validation split—would settle the issue. If the split is clean, the framework is plausible; if not, the headline claim must be rejected. Since the reader already returned CONDITIONAL, my verdict remains UNCHANGED pending that check.","tokens_in":18234,"tokens_out":8415,"duration_ms":99224,"concrete_test":"From the published repository, reconstruct the exact split used for PrimeVul and SVEN, and verify that test-set commit hashes, function names, or normalized code snippets do not appear in Daug, D, or the COPO preference pairs; also confirm that E is a held-out validation split disjoint from the test set. If the repository lacks this information, request the split from the authors and re-run Algorithm 1 with E replaced by a proper validation split and with the COPO round count selected on that validation split rather than on test curves. If any overlap is found, or if the re-run accuracy drops materially below 58.05% on PrimeVul, the reported SOTA is inflated by leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that ReVD sets SOTA on PrimeVul/SVEN with 58.05%/63.72% accuracy (Table 1)—requires that test samples never appear in any training input or hyperparameter selection. The paper never states a train/validation/test partition for either dataset. Section 3.1 constructs Daug from \"the original vulnerability dataset D\" without saying D is training-only; Section 3.3's Algorithm 1 takes an \"Eval Set E\" and \"Training Set E\" (both named E, a typo) and uses per-type accuracy on E (Lines 4-6) to select training instances from D (Lines 8-13). The source and split of E are never given. Moreover, Section 5.3 chooses the COPO round count (3) from accuracy-versus-round curves (Figure 3) that appear to be computed on the test sets. If E overlaps the test set, or if D includes test samples, then T-SFT and COPO directly or indirectly consume test labels, and the reported 12.24%-22.77% improvements are inflated. The Limitations section only acknowledges model-size and token-length constraints, not this missing separation, so the omission is an unacknowledged threat to validity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReVD, a three-module framework for LLM-based vulnerability detection: (1) Bi-directional Vulnerability Data generation (BVD), which uses a generation model T to produce forward and backward reasoning traces from vulnerable/fixed code pairs; (2) Triplet Supervised Fine-Tuning (T-SFT), which fine-tunes a model with a sum of cross-entropy losses on pre-code, post-code, and code-diff inputs; and (3) Curriculum Online Preference Optimization (COPO), which iteratively applies Identity Preference Optimization (IPO) with instance-level selection based on per-vulnerability-type accuracy on an evaluation set E and task-level decomposition into three reasoning tasks. The authors report state-of-the-art results on PrimeVul and SVEN, e.g., 58.05% accuracy on PrimeVul and 63.72% on SVEN with Qwen2.5-Coder-7B-Instruct, outperforming GPT-4 and other baselines, and release code and data at a public repository.","tokens_in":18424,"tokens_out":4447,"duration_ms":58644,"significance":"If the reported evaluation protocol is sound, the contribution is significant: the paper introduces the first fully automated pipeline for synthesizing vulnerability reasoning data, shows that a 7B open-source model can surpass GPT-4 on vulnerability-detection benchmarks, and demonstrates that curriculum preference optimization can address class-imbalanced vulnerability types. The release of the reasoning dataset, preference dataset, and training code is a concrete strength that supports reproducibility. The ablations across three base models and two datasets give useful evidence for the effects of BVD, T-SFT, and COPO. However, the central empirical claim of state-of-the-art performance depends entirely on a train/validation/test separation that the manuscript never specifies; until that protocol is clarified or corrected, the magnitude of the reported improvements (12.24%-22.77%) cannot be assessed. The manual data-quality check of only 20 examples and the absence of variance reporting are secondary but real limitations.","major_comments":[{"comment":"The manuscript never defines a train/validation/test partition for PrimeVul and SVEN. Section 3.1 says BVD uses 'the original vulnerability dataset D' but does not state that D is disjoint from the test set; Algorithm 1 (line 2) lists an 'Eval Set E' and a 'Training Set E' without specifying where E comes from or how it was split; Section 4.2 gives only training hyperparameters. If E overlaps the test set, or if D includes test samples, the curriculum selection and the reported accuracy improvements leak test labels into training. The Limitations section (Sec. 7) does not acknowledge this risk. Please specify the exact partition, including any overlap with the official PrimeVul/SVEN splits, the random seed, and the sizes of D, E, and the test set, and report the per-type accuracy on E used by Algorithm 1 separately from test accuracy.","section":"§3.1, §3.3, §4.2"},{"comment":"The number of COPO rounds, C=3, is selected from accuracy and VP-Score curves in Figure 3, which are presented as model performance on PrimeVul and SVEN. If these curves are computed on the test sets, then the reported test results are the product of test-set-based model selection, which invalidates the claim that the final numbers are unbiased estimates of generalization. Please clarify whether Figure 3 uses a held-out validation set; if it uses test data, report results for a fixed C chosen a priori, or for C selected only on a validation split, and update the reported numbers accordingly.","section":"§5.3, Figure 3"},{"comment":"Eq. (1) is called a 'triplet loss,' but mathematically it is an unweighted sum of three independent cross-entropy losses on pre-code, post-code, and code-diff inputs. There is no term that ties the three outputs together or enforces consistency between forward and backward reasoning, so the Introduction's claim that T-SFT 'enables consistency checks between their forward and backward reasoning' is not supported by the loss itself. Please either rename the loss as a multi-task fine-tuning objective, or add an actual triplet/consistency term and describe it in the text.","section":"§3.2, Eq. (1)"},{"comment":"The proposed VP-Score metric is defined only as VP-S = (Correct pair - Wrong pair) / All pair, but the manuscript never defines what a 'pair' is or how Correct and Wrong pairs are counted across the PrimeVul and SVEN test sets. Because VP-Score is used in the headline improvements (e.g., the abstract reports an 18.15% VP-Score improvement), please give a formal definition that includes the aggregation unit, the handling of datasets without explicit pre/post pairs, and at least one worked example showing how the score is computed from model predictions.","section":"§4.1"}],"minor_comments":[{"comment":"Line 2 of Algorithm 1 uses the same symbol E for both 'Eval Set E' and 'Training Set E'; the notation in line 14, 'D(r) = (yw, yl, Sr 1 xr)', is malformed and should be rewritten with defined variables.","section":"Algorithm 1"},{"comment":"The manual analysis of the generated reasoning data covers only 20 examples, so the reported '95% accuracy' has a very wide confidence interval; please state the sample size and either enlarge the manual sample or report a confidence interval and acknowledge the small-sample limitation.","section":"Appendix G"},{"comment":"No variance or number of independent runs is reported for any method, despite the text claiming that ReVD 'significantly surpasses' baselines; please report means and standard deviations over at least three seeds for the trained models and, where feasible, for the prompting baselines.","section":"Table 1 and Figure 3"},{"comment":"The token-length analysis reports only accuracy and not F1 or VP-Score; since the paper argues accuracy alone can be misleading, please add the other metrics or explain why they are omitted in that analysis.","section":"§5.4 and Figure 4"},{"comment":"The phrase 'triplet loss' is used throughout the paper, but the actual objective in Eq. (1) is not a contrastive triplet loss; align the terminology in the abstract, introduction, and method sections with the mathematical content.","section":"§1 and §3.2"},{"comment":"The text in Section 4 refers to 'Appendix 5' when describing the vulnerability-type distribution; this should be Appendix B, and Figure 5 lacks error bars or sample counts for the CWE/GraphSPD distribution.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The core concern is the missing evaluation-protocol specification, not the method idea itself. If the authors can clarify the split and confirm that E and D are disjoint from the test sets, and if the COPO round count can be justified without test-set selection, the reported results may well be valid. I recommend asking for the exact configuration and released code paths to be checked during revision. The manuscript's scope fits the journal; the issue is a load-bearing one that is fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Best to know immediately: the paper has a real idea, but the evaluation as written leaks test-set information into model selection, so the headline numbers are not trustworthy yet.\n\nWhat's genuinely new: ReVD combines forward/backward reasoning for synthetic vulnerability data, a three-way supervised fine-tuning objective on pre-code, post-code, and diff, and an online preference optimization with instance- and task-level curricula. That combination is new for vulnerability detection, and the recipe is concrete. The authors release code and data, which is good practice. The gains over baselines are large (12-23% relative accuracy), and the ablations suggest each module contributes, though the contributions are interleaved.\n\nThe soft spots are substantial. Most importantly, the paper never defines a train/validation/test split. Section 3.1 constructs Daug from 'the original vulnerability dataset D' without saying D is training-only. Algorithm 1 takes an evaluation set E to score per-type accuracy for the curriculum, but E's origin is never given. And the number of COPO rounds (3) is picked from accuracy curves in Figure 3 that appear to be on the PrimeVul and SVEN test sets. That is test-set leakage in hyperparameter selection. If E overlaps the test set, or if D contains test samples, the reported improvements are inflated. The limitations section does not mention this. This is fixable: state the split, release E, and choose rounds on validation.\n\nSecondary issues: no error bars or variance across seeds; the VP-Score formula is under-specified (what counts as a 'pair', and are these percentages?); the manual data-quality check on 20 examples is thin; and the 'triplet' loss is simply a sum of three cross-entropy terms, not a metric-learning triplet. These are minor relative to the split problem.\n\nThe math itself looks standard—IPO equations are correct, and the citation of Chen et al. and Azar et al. is appropriate. I don't see evidence of fabrication or self-citation abuse.\n\nBottom line: this paper deserves a serious referee, but the referee should insist on a clear data partition and validation-based hyperparameter selection before the performance claims can be taken at face value. If the results survive that correction, the recipe is a useful contribution to the vulnerability-detection subfield.","headline":"Interesting training recipe for LLM vulnerability detection, but the missing train/test split definition and test-based round selection make the reported SOTA numbers unverifiable as they stand.","tokens_in":19015,"tokens_out":2774,"would_cite":false,"duration_ms":31965,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training pipeline that synthesizes forward and backward vulnerability-reasoning data and then applies curriculum preference optimization turns 7B–8B open-source LLMs into vulnerability detectors that outperform GPT-4 and all nine…","keywords":["vulnerability detection","large language models","synthetic reasoning data","preference optimization","curriculum learning","code security","PrimeVul","SVEN"],"falsifier":"Re-run ReVD on PrimeVul with E built only from training-split samples and with the COPO round count fixed before any test evaluation; if the per-type accuracy on E still predicts the reported gains and those gains persist, the curriculum is learning real patterns, whereas if the gains vanish, test information was doing the work.","tokens_in":17957,"feed_emoji":"🛡️","tokens_out":10234,"duration_ms":95239,"temperature":0.7,"pith_summary":"Software vulnerability detection is one place where LLMs still lag: they are trained on code semantics, not on the reasoning that connects a vulnerable line to its fix. This paper tries to close that gap with ReVD, a pipeline that first synthesizes bi-directional reasoning traces for vulnerable pre-code and fixed post-code pairs, then fine-tunes an LLM with a triplet loss tying together pre-code, post-code, and code diff, and finally runs curriculum online preference optimization that keeps re-training on the vulnerability types the model gets wrong. The core claim is that this turns 7B–8B open-source code LLMs into detectors that beat GPT-4 and all nine baselines on PrimeVul and SVEN, with relative accuracy improvements of 12.24%–22.77%. If true, it matters because high-quality labeled vulnerability data is scarce and expensive, so a fully automated reasoning-data synthesis route would make specialized vulnerability detectors much cheaper to build.","feed_headline":"7B LLMs beat GPT-4 at vulnerability detection","feed_subtitle":"Open-source 7B model hits 58% on PrimeVul and 64% on SVEN after synthetic-reasoning training.","key_machinery":"The load-bearing mechanism is the pairing of a vulnerable pre-code snippet with its fixed post-code and code diff, and the instruction to the generation model to produce two complementary reasoning traces: forward reasoning that explains how the vulnerability is triggered, and backward reasoning that explains why the fix blocks it. The triplet supervised fine-tuning loss trains the model on all three inputs—pre-code, post-code, and code-diff—so that vulnerability-relevant changes are emphasized over semantically similar surfaces. COPO then applies an Identity Preference Optimization (IPO) objective, where each round selects samples biased toward vulnerability types with low current accuracy (instance-level curriculum) and decomposes each preference instance into easy-to-hard tasks—locating the vulnerable line, tracing the trigger path, and interpreting root cause (task-level curriculum). This machinery is what lets a 7B model learn patterns rather than surface semantics.","core_discovery":"On its own terms, the discovery is that vulnerability detection improves when a model is forced to reason in both directions around a patch: forward from the vulnerable code to the trigger, and backward from the fixed code to why the repair prevents recurrence. ReVD operationalizes that idea in three modules: BVD synthesizes 28k reasoning samples from CVE/CWE/commit context; T-SFT fine-tunes the model with a triplet objective over pre-code, post-code, and code-diff; and COPO applies an online IPO objective with instance- and task-level curricula so each round focuses on vulnerability types the current model handles poorly. With Qwen2.5-Coder-7B-Instruct, ReVD reports 58.05% accuracy, 63.83% F1, and 16.09% VP-score on PrimeVul, and 63.72%, 69.49%, and 27.44% on SVEN, surpassing GPT-4 and all nine baselines. The paper also releases the first vulnerability reasoning dataset and the corresponding preference dataset.","pith_inferences":["A testable extension is to reuse the per-type accuracy signal from COPO at inference time, routing rare or long samples to more extensive reasoning; this could raise detection without extra training.","VP-Score, which credits models only when they flag the vulnerable pre-code and clear the fixed post-code in the same pair, could become a standard metric for patch-aware detectors.","The method's reliance on a separate evaluation set E suggests a calibration experiment: measure how COPO's gains change as E shrinks or shifts across vulnerability types.","If synthetic reasoning data is the principal driver, then feeding BVD with patch corpora beyond PrimeVul and SVEN should further improve accuracy without new human labels."],"forward_implications":["Open-source 7B and 8B LLMs can match or exceed GPT-4 and 70B-class models on vulnerability detection, removing the need for proprietary APIs.","Automatically synthesized reasoning data, checked on a random 20-sample human audit, can stand in for scarce human-written vulnerability explanations.","Curriculum preference optimization improves accuracy across three model families, so the gains are not tied to one base model's pretraining.","ReVD's released reasoning and preference datasets give other groups a ready-made resource for fine-tuning vulnerability detectors.","Because accuracy still drops for snippets over 2048 tokens, longer-code vulnerability detection remains an open boundary of the approach."],"supporting_citations":[{"why":"Provides the PrimeVul dataset and the chain-of-thought prompt template used for detection, and serves as the primary benchmark ReVD must beat.","marker":"Ding et al., 2024"},{"why":"Supplies the SVEN dataset, the second evaluation benchmark, whose vulnerable and fixed code pairs are manually vetted.","marker":"He and Vechev, 2023"},{"why":"Supplies the forward and backward reasoning design that BVD adapts for vulnerability interpretation.","marker":"Chen et al., 2024"},{"why":"Defines the IPO objective whose regularized log-likelihood gap COPO uses for preference optimization.","marker":"Azar et al., 2024"},{"why":"Provides GPT-4, the strongest closed-source baseline and the running example of a model that misses the divide-by-zero vulnerability.","marker":"OpenAI, 2023"},{"why":"Introduces Qwen2.5-Coder-7B-Instruct, the base model on which ReVD reaches its best reported results.","marker":"Hui et al., 2024"},{"why":"Provides Llama-3.1-8B-Instruct, one of the open-source base models ReVD is applied to and evaluated on.","marker":"Dubey et al., 2024"},{"why":"Provides StarCoder2-7B, another base model used in the cross-model generalization experiments.","marker":"Lozhkov et al., 2024"}],"fun_headline_variants":["Reasoning around patches lets a 7B model beat GPT-4 at finding bugs","Curriculum preference optimization with synthetic reasoning slashes LLM vulnerability gaps","Forward and backward reasoning on fixes lifts LLM vulnerability detection","Open-source method ReVD uses reasoning data to outdo GPT-4 on vulnerability spotting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that the evaluation set E used to choose which vulnerability types and samples to train on is a genuinely separate validation split, so that test labels never leak into curriculum selection; if E overlaps the test set, the reported improvements are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning around patches lets a 7B model beat GPT-4 at finding bugs","Curriculum preference optimization with synthetic reasoning slashes LLM vulnerability gaps","Forward and backward reasoning on fixes lifts LLM vulnerability detection","Open-source method ReVD uses reasoning data to outdo GPT-4 on vulnerability spotting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000872,"raw_usage":{"total_tokens":3801,"prompt_tokens":999,"completion_tokens":2802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":2722}},"tokens_in":615,"tokens_out":2802,"duration_ms":19646,"temperature":1.0,"reasoning_tokens":2722,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:35:07.131019+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run ReVD on PrimeVul with E built only from training-split samples and with the COPO round count fixed before any test evaluation; if the per-type accuracy on E still predicts the reported gains and those gains persist, the curriculum is learning real patterns, whereas if the gains vanish, test information was doing the work.","supporting_citations":[],"review_version":1}