{"id":"6e5902a5-3370-4274-ad14-a12c15818641","arxiv_id":"2504.13474","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"With context-rich prompts that include CWE hints and marked potential vulnerability sites, LLM detectors beat random baselines, but the evaluation may leak the answer.","lead":"This paper introduces CORRECT, a context-rich benchmark for testing whether LLMs can detect real-world code vulnerabilities, and reports that with extra context, top models reach about 67% accuracy and 0.8 precision on a 400-pair subset. It argues that earlier results showing LLMs were no better than random were artifacts of missing context, but the benchmark design partly reveals the vulnerabilities it asks models to find.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported precision and low false-alarm rate are manufactured by counting false positives as true negatives in Table 2, so the overturn of Consensus #2 is unsupported.","rationale":"The reader correctly identified both the leakage risk from CWE descriptions and //potential markers and the metric reclassification issue, but placed the weakest assumption on leakage. I focus on the metric definition because it is a logical flaw rather than an empirical confound: Table 2's lenient-mode rule counts a patched-code HAS_VUL prediction as True Negative when the rationale does not reference the ground-truth vulnerability. This directly inflates precision, F1, and accuracy, and the Strict Mode feedback loop further converts false alarms into non-alarms by telling the model to ignore its prior inference. These are load-bearing for the paper's central quantitative claims, especially the overturn of Consensus #2. The suggested test would settle the issue by recomputing metrics with standard counting. Because the reader's verdict is already REJECT and this concern reinforces that conclusion, the verdict remains unchanged.","tokens_in":26955,"tokens_out":4311,"duration_ms":42364,"concrete_test":"Recompute all RQ1/RQ2 metrics on the 400-pair evaluation set using standard binary classification: any HAS_VUL on a patched function counts as FP, regardless of rationale correctness, and remove the Strict Mode feedback loop and the δ(Patched, 1, F)=0 reclassification. If ds-r1's precision drops from the reported ~0.8 toward ~0.5 and the (1,1) proportion rises substantially, the claimed overturn of Consensus #2 is a metric artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3, Table 2 defines Lenient Mode with δ(Patched, 1, F) = 0: when the model outputs HAS_VUL on a patched function, the case is scored as True Negative if the rationale is judged not to reference the ground-truth vulnerability. This is not a harmless scoring convenience; it converts a false positive into a true negative. Under standard binary classification, any HAS_VUL prediction on patched code is a false positive regardless of rationale correctness. Because precision, F1, accuracy, and the (1,1) proportion are all computed from these reclassified counts, the headline results of precision near 0.8 and a (1,1) rate near 10% in RQ2/Consensus #2 are inflated by construction. Strict Mode makes the problem worse: when a patched input is flagged, the framework feeds back corrective guidance to ignore previously inferred non-ground-truth vulnerabilities and re-asks the model up to four rounds (§4.3, Appendix C). This interactive loop effectively tells the model its answer was wrong and lets it revise until it produces the desired output. Thus the evidence that LLMs can distinguish patched from vulnerable code is not evidence of genuine discrimination; it is an artifact of reclassifying disagreements as correct when the rationale is poor, plus an explicit correction loop. Since the paper's central claim that prior consensuses are misconceptions rests substantially on this precision/FPR evidence, the claim fails even if one sets aside the separate leakage concern about CWE descriptions and //potential markers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that three community consensuses about LLM-based vulnerability detection—that LLMs are unreliable, insensitive to code patches, and plateaued across model scales—are artifacts of context-free evaluations. To support this, it proposes CORRECT, a framework that augments vulnerable/patched function pairs with callee functions, type declarations, slicing information, and CWE descriptions, and that uses LLM-as-a-judge to evaluate rationales. The authors construct a 2,000-pair dataset spanning 99 CWEs, evaluate 13 LLMs, and report that with sufficient context, state-of-the-art models reach about 67% accuracy, precision around 0.8, and F1 above 0.7 on common CWE categories, overturning all three consensuses.","tokens_in":27283,"tokens_out":3133,"duration_ms":31069,"significance":"If the results held, the paper would be an important corrective to a large body of benchmarking work: it provides a substantial new dataset, evaluates a broad model zoo, and moves beyond binary labels to rationale-level assessment. The dataset release, the manual label audit, and the breadth of models are genuine strengths. However, the central comparison is confounded by construction: the context-rich condition includes ground-truth-derived hints (CWE descriptions and \"//potential\" markers), and the scoring procedure reclassifies false positives on patched code as true negatives and adds an interactive correction loop. These issues directly affect the headline numbers that overturn the consensuses, so the paper's central claim is not currently supported.","major_comments":[{"comment":"Lenient Mode counts δ(Patched, 1, F) = 0 as True Negative, meaning that a model predicting HAS_VUL on patched code is scored as correct whenever the rationale is judged not to reference the ground-truth vulnerability. In any standard binary classification, HAS_VUL on patched code is a false positive regardless of the rationale's content. Because precision, recall, F1, accuracy, and the pairwise (1,1) proportion in Figures 4 and 5 are all computed from this reclassified contingency table, the paper's headline result that SOTA models achieve precision near 0.8 and a (1,1) rate near 10% (Finding #3) is inflated by the scoring rule itself. This directly undermines the evidence used to overturn Consensus #2.","section":"§4.3, Table 2"},{"comment":"Strict Mode does not simply measure the model's ability to distinguish patched from vulnerable code; when a patched input is flagged as vulnerable with an unrelated rationale, the framework instructs the model to ignore previously inferred non-ground-truth vulnerabilities and re-asks it up to four times until it produces a non-alarm or exhausts the budget. Appendix C reports that this feedback altered the answer in 10.3% of NO_VUL cases at round 1 and in additional cases at later rounds. This interactive correction loop effectively tells the model that its previous answer was wrong, so the resulting precision and (1,0) proportions reflect the framework's ability to coax out a desired answer rather than the model's intrinsic discrimination. The paper's claim that LLMs can reliably distinguish patched from vulnerable code is therefore not established by the reported numbers.","section":"§4.3, Appendix C"},{"comment":"The context-rich prompt contains information derived from the ground truth that is absent from the \"w/o context\" baseline: the CWE description of the specific vulnerability, the \"//potential\" markers on slicing-path statements, and the explicit marking of unrelated parameters as irrelevant. The prompt also instructs the model to restrict its analysis to these marked statements. These hints can directly reveal where the vulnerability is located, so the observed improvement is not attributable to the added callee, type, and global context as claimed in Finding #1. To support the paper's central assertion that prior consensuses are artifacts of missing context, the authors would need to compare against a control condition that adds the same CWE and slicing information without the ground-truth-derived markers, and separately ablate the markers.","section":"§4.2, Prompt I"},{"comment":"The random-baseline comparisons used to support Finding #1 are also affected by the design. The pairwise (1,0) proportion is compared to a 25% random baseline, but under Lenient Mode a patched input that is flagged with an unrelated rationale is scored as correct (a true negative), which increases the (1,0) proportion by suppressing the (1,1) cell. Similarly, precision in Figure 4(f) is computed after Lenient/Strict reclassification. Thus the claim that models 'significantly surpass' the random baseline conflates model capability with the scoring revision. A scoring-only ablation would be needed to determine how much of the reported improvement comes from context versus from the evaluation rules.","section":"§5.2, Figure 4"}],"minor_comments":[{"comment":"The sentence 'GPT-4o is n SOTA model' contains a typo and should read 'is a SOTA model.'","section":"§5.1"},{"comment":"The caption lists subfigures (a)-(d) as the 'w/o context' condition and (f), (h), (i), (j) as Strict Mode, but the figure contains subfigures (e) and (g) labeled 'w/ context, w/o revision' and 'Lenient Mode'; the caption should be updated to describe all panels consistently.","section":"Figure 4 caption"},{"comment":"The sentence 'most models achieved F1-scores ranging from 0.5 to 0.6 and 0.5 to 0.55 accuracy' is grammatically ambiguous; the two metric ranges should be presented in separate clauses.","section":"§5.2"},{"comment":"The exclusion of r1-qn-7b from the main results is justified by abnormal outputs, but the paper should also report which performance metrics would change if the model were included, or at least state whether any of the headline findings depend on its exclusion.","section":"Appendix G"}],"recommendation":"reject","confidential_remarks":"The paper addresses an important question and the dataset may be a useful community resource, but the core empirical claims are not supported as presented. The scoring rule in Table 2 and the feedback loop in Strict Mode manufacture the precision and (1,1)-rate results, and Prompt I's ground-truth-derived markers confound the context comparison. Correcting these issues requires re-running the evaluation with proper controls and standard scoring, which is beyond a routine revision. I would be open to considering a substantially revised version that separates the effect of genuine context from the effect of leakage and scoring artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe short version: this paper has a genuinely useful dataset and a sensible instinct—benchmarking on isolated functions understates what LLMs can do with context—but its headline claim about overturning the three consensuses is not supported by its own evaluation. The main evidence for Consensus #2, precision near 0.8, is an artifact of scoring.\n\nWhat is actually new: CORRECT ships a 2,000-pair vulnerable-patched dataset with shared context, spanning 99 CWEs, and it evaluates rationales with a judge model. The per-CWE breakdown of F1 and the taxonomy of false positives (patch ignored vs. patch deemed insufficient) are useful. The test-time scaling analysis across o3-mini and R1-distill is also suggestive. The scale of data collection deserves credit.\n\nThe soft spots are in the evaluation design. First, the \"context-rich\" condition bundles several additions at once: callee functions, type declarations, slicing parameters, the ground-truth CWE description, and comments that mark the vulnerable statements with //potential. The no-context baseline gets none of those. So any gain cannot be attributed to context alone; the CWE description and markers are, for all practical purposes, hints derived from the ground truth.\n\nSecond, and more load-bearing, the Lenient Mode scoring in Table 2 counts a patched function flagged as HAS_VUL as a True Negative if the rationale doesn't reference the ground-truth vulnerability. In ordinary binary classification that is a false positive. Because precision, F1, accuracy, and the (1,1) proportion are all computed from those reclassified counts, the reported precision near 0.8 is baked into the definition. This is not a minor presentation issue—it is the main evidence for RQ2.\n\nThird, Strict Mode goes further: when a patched input is flagged, the framework feeds back guidance to ignore previously inferred non-ground-truth vulnerabilities and re-asks the model up to four times. That is an explicit correction loop. It tells the model its answer was wrong and lets it revise until it agrees. Whatever this measures, it is not a clean measure of the model's ability to distinguish patched from vulnerable code.\n\nThe scaling story (RQ3) is less affected by these confounds because it compares models under the same conditions, and the error attribution in Table 5 is interesting. But the paper's central claims about the three consensuses rest on RQ1 and RQ2, and those are not sound as written.\n\nI would send this to reviewers: the dataset and framework are worth engaging with, and the paper could be salvageable with ablations, honest metrics, and a much narrower set of claims. As written, it overclaims. The kind of reader who picks it up should see it as a cautionary example of how evaluation choices can manufacture results, rather than as evidence that LLMs are actually good at vulnerability detection.\n\nBest,","headline":"Useful dataset and honest scaling analysis, but the core claim of overturning prior consensuses rests on scoring choices that manufacture the result.","tokens_in":27770,"tokens_out":2911,"would_cite":false,"duration_ms":25994,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A context-rich evaluation framework claims that LLMs are far better at vulnerability detection than prior benchmarks suggested, and that the three common beliefs against them are artifacts of missing context.","keywords":["LLM vulnerability detection","context-rich evaluation","code property graph","CWE classification","paired vulnerable-patched benchmark","rationale evaluation","test-time scaling","false positive attribution"],"falsifier":"Run the CORRECT prompt with the '//potential' markers and irrelevant-parameter annotations stripped out while keeping all callee, type, global, and slicing context; if SOTA accuracy drops back toward the 0.5–0.6 no-context range, the paper's central attribution of the gain to context fails. As a second check, replace the GPT-4o judge with an alternative judge or a majority of judges on the same rationales; if MATCH/MISMATCH flips on more than the reported 8% disagreement, the precision and F1 figures are not stable.","tokens_in":26751,"feed_emoji":"🛡️","tokens_out":7484,"duration_ms":61772,"temperature":0.7,"pith_summary":"The paper claims that three widely repeated conclusions about LLM-based vulnerability detection—that models perform near random, that they cannot tell patched from vulnerable code, and that larger models do not improve—are artifacts of evaluating on isolated functions or files without surrounding context. To test this, it builds CORRECT, a context-rich evaluation that feeds models CWE descriptions, callee functions, type and global declarations, and slicing paths, and that scores the model's verdict together with its written rationale. Across 2,000 vulnerable–patched pairs spanning 99 CWE classes and 13 models from four model families, the best models reach about 67% accuracy, F1 above 0.7 on common CWE categories, and precision near 0.8, with a (1,0) pair proportion of 37% versus a 25% random baseline. The paper also argues that most false positives are reasoning errors—the model sees the patch but judges it insufficient—rather than failures to notice patches, and that scaling helps but with diminishing returns and recall loss. If the claims hold, prior benchmarks measured context deprivation as much as model capability.","feed_headline":"With context, top LLMs hit 0.8 precision on vulnerability detection","feed_subtitle":"A 2,000-pair benchmark spanning 99 CWE classes says function-level tests underrated model capability.","key_machinery":"The load-bearing object is the context-rich paired prompt built by CORRECT's code-context pipeline: a vulnerable function and its patched counterpart share one merged context assembled from Code Property Graphs and slicing, containing callee functions and macros, global variables, type declarations, library imports, and slicing-path parameters, plus the CWE description and assumptions that restrict analysis to marked parameters and statements annotated '//potential'. That prompt is the mechanism that converts a guess about an isolated snippet into a grounded vulnerability-reasoning task, because it supplies the execution and data-flow facts that determine whether a patched code path is actually safe. The second mechanism is the two-mode assessment: Lenient Mode accepts any detection whose rationale names the ground truth, while Strict Mode feeds false alarms whose rationale does not mention the ground truth back into the model for up to four rounds of re-analysis, separating true misclassification from mere reasoning error.","core_discovery":"On the paper's own terms, the central discovery is that context-deprived evaluation produces two systematic distortions: incorrect conclusions (a model flags patched code as vulnerable because a callee's real behavior is missing from the prompt) and incorrect rationales (a model gives the right label for a non-ground-truth reason). With context supplied, state-of-the-art models show real discrimination: DeepSeek-R1 reaches 67% accuracy and 37% (1,0) pair-wise detection in Strict Mode, SOTA precision approaches 0.8, and key CWE classes such as CWE-664 and CWE-682 reach F1 above 0.7. The paper interprets this as overturning all three consensuses: LLMs are not unreliable, they are not insensitive to patches, and a model-size scaling effect appears once context is present. It also reports new limitations that remain after context is fixed: rare CWE types are detected near random (e.g., CWE-697 max F1 0.400), recall stays around 0.5, and reasoning models overthink, turning correct answers wrong.","pith_inferences":["Beyond the paper: the '//potential' markers and the explicit marking of unrelated parameters are ground-truth-derived hints, so a control that removes only those hints—while keeping callee, type, and global context—would show how much of the 67% accuracy is genuine context and how much is leakage.","Beyond the paper: the 92% LLM-as-a-judge agreement is measured on 50 sampled rationales; re-running the judge with ensembles or with a second, independently prompted judge would bound the noise in the reported precision and F1.","Beyond the paper: the common-versus-rare CWE gap suggests a testable extension—augmenting prompts with vulnerability-pattern examples for rare CWEs should close most of the gap if the bottleneck is knowledge rather than reasoning.","Beyond the paper: the paper's own RAG example (a retrieved integer-overflow patch leading the model to flag an unrelated input-validation issue) implies that naive semantic-similarity retrieval can hurt detection; grounding retrieval in vulnerability type, not code similarity, is the obvious follow-up."],"forward_implications":["Prior function- and file-level benchmarks should be re-read as measuring context deprivation; their near-random scores are not evidence about model capability.","Future evaluations should supply callee, type, and global context and validate rationales, not just binary labels, or they will both under- and over-estimate models.","Detection systems should treat recall, not precision, as the binding constraint: SOTA precision is near 0.8 while recall stays around 0.5.","Scaling is a real but weak lever: fivefold more thinking tokens buy less than 0.05 accuracy, so test-time scaling alone is an uneconomical route to improvement.","Reasoning models need safeguards against over-thinking, since 'mis-corrected reasoning' converts correct answers into false alarms on patched code."],"supporting_citations":[{"why":"Supplies a large share of the real-world vulnerable–patched code pairs used to build the 2,000-pair dataset.","marker":"[2]"},{"why":"Provides the paired evaluation methodology (vulnerable and patched versions) and the prior near-random results used as baselines for the first two consensuses.","marker":"[11]"},{"why":"The earlier evaluation that concluded LLMs cannot reliably reason about vulnerabilities; its 13% (1,0) proportion is the Consensus #1 baseline.","marker":"[51]"},{"why":"The prior large-scale study reporting balanced accuracy 0.5–0.55 with no scaling effect; its prompt design motivates including CWE descriptions.","marker":"[47]"},{"why":"The prior study reporting precision between 39% and 57% across 7B–70B models; CORRECT's precision near 0.8 is the direct contrast.","marker":"[60]"},{"why":"The prior measurement that F1 stayed near 0.6 from small models to GPT-4, taken as evidence for the plateaued consensus.","marker":"[26]"},{"why":"The prior few-shot evaluation reporting F1 of 0.084–0.128 across 6.7B–34B models, another plateau baseline for the scaling question.","marker":"[58]"},{"why":"The state-of-the-art reasoning model whose results (67% accuracy, ~0.8 precision) carry the claim that capability was underestimated.","marker":"[17]"},{"why":"Supplies the sequential test-time scaling technique (appending 'Wait') and the evidence that outputs degrade beyond ~4k tokens.","marker":"[32]"},{"why":"Supplies the parallel test-time scaling method (Universal Self-Consistency) used for majority voting over 3, 5, and 8 rationales.","marker":"[5]"}],"fun_headline_variants":["Context boosts LLM vulnerability detection to 0.8 precision","LLMs not unreliable: context-rich eval shows 0.7 F1 on top CWEs","Why function-level tests underrate LLM vulnerability detectors","New benchmark: with context, LLMs hit 0.8 precision on vulnerabilities","Overthrowing three myths about LLM vulnerability detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that telling the model which statements and parameters to focus on through '//potential' marks and 'irrelevant parameter' annotations is legitimate context rather than an answer-hiding hint; if those annotations, not the added callee, type, and global context, drive the gains, the comparison against the no-context baseline would not support overturning the consensuses.","fun_headline_variants_meta":{"raw":{"variants":["Context boosts LLM vulnerability detection to 0.8 precision","LLMs not unreliable: context-rich eval shows 0.7 F1 on top CWEs","Why function-level tests underrate LLM vulnerability detectors","New benchmark: with context, LLMs hit 0.8 precision on vulnerabilities","Overthrowing three myths about LLM vulnerability detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000464,"raw_usage":{"total_tokens":2397,"prompt_tokens":1100,"completion_tokens":1297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":716,"completion_tokens_details":{"reasoning_tokens":1204}},"tokens_in":716,"tokens_out":1297,"duration_ms":8945,"temperature":1.0,"reasoning_tokens":1204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:07:37.213390+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the CORRECT prompt with the '//potential' markers and irrelevant-parameter annotations stripped out while keeping all callee, type, global, and slicing context; if SOTA accuracy drops back toward the 0.5–0.6 no-context range, the paper's central attribution of the gain to context fails. As a second check, replace the GPT-4o judge with an alternative judge or a majority of judges on the same rationales; if MATCH/MISMATCH flips on more than the reported 8% disagreement, the precision and F1 figures are not stable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The earlier evaluation that concluded LLMs cannot reliably reason about vulnerabilities; its 13% (1,0) proportion is the Consensus #1 baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior few-shot evaluation reporting F1 of 0.084–0.128 across 6.7B–34B models, another plateau baseline for the scaling question."}],"review_version":1}