{"id":"66465f36-5ffb-4d2f-92f5-4d30aafafaf5","arxiv_id":"2504.16877","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"PacVD adds hand-built summaries of security-sensitive API calls to LLM prompts and reports F1 improvements of a few points, though the gains are not validated against a majority-class baseline.","lead":"Researchers tested whether adding program-analysis summaries of security-sensitive API calls to code snippets helps large language models spot memory safety flaws. They found modest F1 gains around 60 to 69 percent, but the best scores stay below a simple 'always vulnerable' classifier, so the claimed benefit is not clearly established.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All reported F1 scores fall below a trivial all-vulnerable classifier (70.6%), so the claimed 'significant enhancement' is unsupported without significance testing.","rationale":"The reader's REJECT verdict is well supported, and I agree with it. However, the reader's explicitly stated 'weakest_assumption' concerns the sufficiency of the primitive API list (Section 3.2.1). That is a legitimate limitation, but the more immediate threat to the central claim is the evaluation protocol: the absence of a trivial majority baseline, significance tests, and held-out few-shot examples. With 340/623 vulnerable samples, an all-vulnerable predictor reaches F1 = 70.6%, above every reported result, including the best PacVD configuration (69.09%). The paper's 'w/o API' rows are the right comparison for the abstraction effect, but point estimates alone do not support 'significantly enhances' when the differences are small, the LLM outputs are stochastic, and no paired statistical test is reported. The few-shot leakage risk further weakens the empirical support. These issues do not require the API-list assumption to fail; they invalidate the evidence for the headline claim as it stands. Therefore the existing REJECT verdict should remain unchanged.","tokens_in":27111,"tokens_out":3924,"duration_ms":38833,"concrete_test":"Recompute the trivial all-vulnerable baseline on the same 623 samples: F1 = 2 * 340 / 623 = 70.6%. Then, for each model and prompt strategy, run a paired evaluation in which PacVD at API Level 3 and the 'w/o API' baseline are applied to the same samples, and report McNemar's test p-value, 95% bootstrap confidence intervals for the F1 difference, and MCC. If the best reported F1 remains below 70.6%, or if the 95% CI for the A3-vs-no-API difference includes zero for the headline DeepSeek-R1 + CoT row, the central enhancement claim fails. As a secondary check, re-run the FSR and FSC conditions with few-shot examples selected from a held-out set separated at CVE or commit level, and confirm the GitHub repository link in Section 9 contains the dataset and code used for the tables.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that primitive API abstraction 'significantly enhances' LLM-based vulnerability detection. The evaluation does not establish this. The dataset (Section 4.2, Table 3) has 340 vulnerable and 283 non-vulnerable samples out of 623 total. A classifier that labels every sample vulnerable achieves precision 340/623 = 54.6%, recall 100%, and F1 = 70.6%. Every reported F1 in Tables 4 and 5 is below this value, including the headline DeepSeek-R1 + CoT result at API Level 3 (F1 = 69.09%). The paper never reports this majority-class baseline, and the 'up to 12.77%' improvement is computed relative to the weakest baseline in each row (e.g., accuracy 53.42 vs. 47.37 in Table 5), which is cherry-picking. No significance tests, confidence intervals, or repeated runs are reported, so the 5-8 point F1 gaps between API abstraction levels and the 'w/o API' rows could be noise rather than signal. In addition, the few-shot examples are drawn 'from our dataset' (Section 3.3.2) without a train/test split, creating a leakage risk that would inflate the few-shot conditions, and the Section 9 data availability link cannot be verified as populated from the manuscript. These are load-bearing problems: they change the conclusion from 'significant enhancement' to 'unverified marginal difference'.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PacVD, a context-enhanced vulnerability detection approach that combines static program analysis with large language models (LLMs). The method extracts primitive-API information (fuzzy branches, concrete branches, call counts, and key variables) from a target function and its callees up to three call layers, and feeds this abstraction together with the source code to ChatGPT-4o, DeepSeek, and CodeLLaMA under several prompting strategies. The authors report four findings: (1) API abstraction improves detection accuracy and reduces false positives; (2) different vulnerability types benefit from different abstraction levels; (3) larger models prefer higher-level abstractions while CodeLLaMA benefits from more detail; and (4) reasoning-oriented prompting matches higher abstraction levels. The evaluation uses a 623-sample C/C++ dataset (340 vulnerable, 283 non-vulnerable) drawn from NVD-New and reports accuracy, precision, recall, F1, and MCC across six prompts and four abstraction levels, with the best F1 being 69.09% for DeepSeek-R1 + CoT at API Level 3.","tokens_in":27246,"tokens_out":4624,"duration_ms":42618,"significance":"If the central claim were established, the idea of using program-analysis-derived API context to improve LLM vulnerability detection would be a useful contribution to the software engineering literature. The paper covers a broad experimental space: three model families, six prompting strategies, four abstraction levels, and a real CVE-derived dataset. The abstraction scheme itself is concrete and could be replicated from the description. However, the evaluation as presented does not support the headline claim. The dataset is imbalanced, and a trivial classifier that always predicts 'vulnerable' achieves F1 = 70.6% on this data; every configuration in Tables 4 and 5 reports an F1 below that value. No majority-class baseline, no significance tests, no confidence intervals, and no repeated runs are reported, and the few-shot examples are drawn from the same dataset without a train/test split. Because the load-bearing evidence is missing or contradicted by the data, the current result does not constitute a verified empirical finding.","major_comments":[{"comment":"The evaluation omits the trivial majority-class baseline. The dataset contains 340 vulnerable and 283 non-vulnerable samples; a classifier that labels every sample as vulnerable has precision 340/623 = 54.6%, recall 100%, and F1 = 70.6%. All reported F1 scores in Tables 4 and 5, including the headline DeepSeek-R1 + CoT result at API Level 3 (69.09%), are below this value. The paper's central claim that the approach 'significantly enhances' detection is therefore not supported by the reported numbers; no configuration in the paper outperforms a predictor that ignores the code entirely. The authors should add this baseline and report statistical tests (e.g., paired tests with multiple runs or bootstrap confidence intervals) before any enhancement claim can be evaluated.","section":"Section 4.2 (Table 3) and Tables 4-5"},{"comment":"The claimed improvement of 'up to 12.77% in accuracy' is computed against the weakest baseline in Table 5 (accuracy 47.37 for SS-C vs. 53.42 for PacVD with DeepSeek-R1 + CoT), while the best baseline in the same row, RS-C, achieves 53.26% accuracy, a difference of only 0.16 percentage points. Many of the reported F1 gaps between the proposed method and baselines are between 1 and 6 points, and no significance testing or repeated sampling is described. These differences are within the range that could be produced by sampling noise or prompt randomness, so the conclusion that PacVD 'consistently outperforms' baselines is not established.","section":"Section 5.4 (Table 5) and Section 1"},{"comment":"The few-shot learning prompts use examples that are 'randomly selected from our dataset' with no indication that they are drawn from a separate training set or excluded from the test samples. This creates a leakage risk because a few-shot example may contain the very function being evaluated or a near duplicate from the same CVE, which would artificially inflate the few-shot conditions. The authors should specify how the few-shot examples were selected, confirm that they are disjoint from the test instances, and rerun the affected configurations with a clean split.","section":"Section 3.3.2"},{"comment":"The primitive API list is explicitly chosen because those APIs are the locus of the targeted vulnerability types (malloc/free for memory errors, open/close for resource leaks, etc.). Since the dataset is restricted to those CWE families, the abstraction directly injects the vocabulary that encodes the expected root cause. The observed gains relative to the no-API condition therefore partially reflect that the hand-picked features are label-aligned by construction, rather than demonstrating a general capability of the abstraction methodology. The claims should be scoped to the memory-safety CWE families considered, and the method should be tested with an expanded API set or with control abstractions built from non-security-sensitive APIs to separate genuine contextual value from direct feature-label correspondence.","section":"Section 3.2.1 (Table 1)"}],"minor_comments":[{"comment":"The model configuration section mentions 'DeepSeek V2.5', but the experiments report results for DeepSeek-V3 and DeepSeek-R1; the text should be updated to name the exact models used.","section":"Section 4.3 vs. Tables 4-5"},{"comment":"The notation for prompt strategies is inconsistent: the table note uses 'CT', 'FR', and 'FC', while the text and figure legends use 'CoT', 'FSR', and 'FSC'. Please unify the abbreviations.","section":"Table 4, Figure 2, Figure 4"},{"comment":"The model is called 'CodeLLaMA-34b' in Sections 4.3 and 5.3 but 'CodeLLaMA-32b' in Section 5.4 and Table 5. This needs to be corrected.","section":"Section 5.4 and Table 5"},{"comment":"The per-CWE F1 scores are presented on small subsets without reporting the number of samples per CWE or any uncertainty measure. Adding sample counts and confidence bounds would help the reader judge whether the per-type gains are meaningful.","section":"Section 5.2 and Figure 3"},{"comment":"The data availability link (https://github.com/DoeSEResearch/PacVD.git) cannot be verified from the manuscript. Please provide a DOI or an accessible repository snapshot and indicate whether the dataset and code are publicly available at the time of publication.","section":"Section 9"}],"recommendation":"reject","confidential_remarks":"The core problem is not stylistic but empirical: the best reported F1 is below the F1 of the trivial all-positive classifier on the paper's own dataset, and the few-shot setup risks leakage. These issues are load-bearing because the paper's contribution is an empirical claim of significant enhancement. If the authors rerun the study with a proper majority-class baseline, a clean train/test split for few-shot examples, and statistical testing, and if the enhanced configurations then exceed the trivial baseline, the underlying idea may deserve a future submission. As it stands, the evaluation cannot support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core idea is worth a look: instead of dumping whole call chains into an LLM, summarize callees via four levels of primitive-API abstraction (fuzzy branches, concrete branches, call counts, key variables) and compare how different models and prompts react. That granularity comparison is a legitimate new empirical question, and the methodology is clearly described. If the data and code are ever released, the abstraction pipeline is reproducible in principle. The authors also build sensibly on prior work by Zhang, Steenhoek, and others on LLM vulnerability detection context.\n\nThe problem is the evaluation. The dataset has 340 vulnerable and 283 non-vulnerable samples. A classifier that labels everything vulnerable gets precision 340/623 = 54.6%, recall 100%, and F1 = 70.6%. Every F1 in Tables 4 and 5 is below that, including the headline DeepSeek-R1+CoT result at 69.09%. The paper never reports this majority-class baseline, and the \"up to 12.77% improvement\" is computed against the weakest baseline in each row. That is cherry-picking. There are no significance tests, confidence intervals, or repeated runs, so the 5–8 point F1 differences between abstraction levels and the no-API condition could be noise.\n\nAlso, the few-shot examples are drawn from the same dataset without a train/test split, which risks leakage, and the data availability link appears to be a placeholder. The primitive API list is chosen because those APIs are the locus of the targeted vulnerability types, which introduces some circularity: the abstraction encodes the vulnerability patterns it claims to help detect. That is not fatal, but it tempers how surprised we should be that A1–A3 sometimes beat no-API.\n\nTo be clear, the relative ranking of abstraction levels and the model-by-prompt interactions could still be real, and the paper's practical suggestion—choose abstraction level based on model capability—is reasonable. But as it stands, the absolute performance is below a trivial predictor, and the central claim of \"significant enhancement\" is unsupported. This needs major rework: add the majority-class baseline, run repeated trials or bootstrap CIs, and use a proper held-out split for few-shot examples.\n\nWho is this for? Researchers working on LLM-based vulnerability detection, particularly those interested in context augmentation. It deserves a serious referee in the sense that the question is worth pursuing, but the current manuscript should not be accepted. I would encourage the authors to revise and resubmit with the missing baselines and statistical grounding.","headline":"A sensible abstraction-granularity study is undermined by a missing majority-class baseline: every reported F1 falls below an always-vulnerable classifier, so the central claim of significant enhancement is unverified.","tokens_in":27918,"tokens_out":2106,"would_cite":false,"duration_ms":19923,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feeding large language models compact program-analysis summaries of security-sensitive APIs such as malloc, free, open, and close significantly improves their ability to detect inter-procedural memory-safety vulnerabilities.","keywords":["vulnerability detection","large language models","primitive API abstraction","program analysis","memory safety","inter-procedural vulnerabilities","prompt engineering","context enhancement"],"falsifier":"Construct or collect an inter-procedural memory-safety vulnerability whose root cause flows through a custom function not on the primitive API list—for example, a hand-rolled memory pool or a reference-count macro—then run PacVD with and without the abstraction. If a strong LLM detects it from raw callee code but not from the API abstraction, while no listed API appears in the call chain, the claim that primitive APIs are an adequate context lens for memory-safety vulnerabilities fails for that class.","tokens_in":26765,"feed_emoji":"🛡️","tokens_out":8800,"duration_ms":75190,"temperature":0.7,"pith_summary":"Vulnerability detectors built on large language models often miss inter-procedural memory bugs because they see only the patched function. This paper claims that adding a compact program-analysis summary of security-sensitive primitive APIs—such as malloc, free, open, and close—as context exposes the root cause without flooding the model with irrelevant repository code. In experiments across several LLMs, prompt strategies, and 623 C/C++ vulnerability samples, every API-abstraction level improved detection over no context, and the level that adds concrete branch conditions plus call counts gave the best overall balance, with top configurations beating raw-callee baselines by as much as 12.77 accuracy points. The paper further finds that the ideal abstraction depth depends on the model and the vulnerability class: reasoning-oriented models prefer high-level summaries, code-specialized models need more detail, and resource-management bugs need richer context than boundary errors. If these findings hold, choosing what context to extract and how abstractly to present it becomes a central design decision for any LLM-based code analysis.","feed_headline":"API context boosts LLM vulnerability detection by up to 12.8%","feed_subtitle":"Program-analysis notes on malloc and free give models the context they need to catch memory bugs.","key_machinery":"The central object is the primitive API abstraction: a compact, program-analysis-derived description of how security-sensitive application programming interfaces are used across a target function and its callees. It is defined along four dimensions—fuzzy branch coverage, concrete branch conditions, call counts, and the key variables operated on—which are combined into four abstraction levels. This object carries the argument because it is what converts noisy inter-procedural code into a small prompt that keeps the root-cause information the LLM needs; the empirical comparison across abstraction levels is precisely a test of which combination of these dimensions gives the best signal-to-noise ratio.","core_discovery":"The paper's central claim is that primitive API abstraction—a program-analysis summary of what security-sensitive API operations a function and its callees perform—substantially improves LLM-based vulnerability detection. For each target function, the method analyzes callees up to three call layers deep and summarizes four dimensions of API usage: whether a primitive API is invoked in all, some, or none of the control-flow branches (fuzzy branches); the exact branch conditions under which it is invoked (concrete branches); how many times it is invoked; and which variables it operates on. Different combinations of these dimensions form abstraction levels A1 through A4, and these summaries are appended to the source code in the prompt. The paper reports that all abstraction levels beat the no-context baseline, with A3 (concrete branches plus call counts) the most consistently effective, reaching F1 scores near 69 percent, while adding variable-level information at A4 yields diminishing returns. It also reports that resource-management vulnerabilities need the richer abstraction levels, boundary-related memory errors can be caught with minimal abstraction, and different models require different abstraction depths, with a reasoning-oriented model performing robustly across levels and a code-specialized model peaking at the more detailed level.","pith_inferences":["Editorial extension: if the mechanism is noise filtering, the same primitive API abstraction should transfer to other LLM code-analysis tasks, such as bug localization or API-misuse detection; a direct test would use the exact A1–A4 summaries as prompts in those tasks and compare against raw code.","Editorial extension: the diminishing returns at A4 raise a testable question the paper leaves open—whether the drop comes from alias-analysis imprecision in key-variable extraction or from exceeding a general information-saturation point; replacing static variable association with a lightweight taint summary would separate the two.","Editorial extension: the primitive list is C/POSIX resource oriented, so extending it to non-memory classes such as locks, integer conversions, and authentication checks would reveal whether the method generalizes beyond memory-safety CWEs to other vulnerability families.","Editorial extension: because abstraction level and prompt strategy interact, an adaptive selector that chooses level and prompt per model and per CWE could plausibly outperform the fixed A3 default, a consequence the paper suggests in its discussion but does not implement."],"forward_implications":["Bare patched functions are a weak input: even the coarsest API summary improved F1 by roughly 5–8 points over the no-context baseline, so LLM detectors should include some abstracted callee context.","When a single configuration must be defaulted, concrete branch conditions plus call counts (A3) is the best-documented choice: it is the most consistently effective level across models, prompt strategies, and CWE types.","Abstraction level and model type should be chosen jointly: reasoning-oriented models work well with high-level context, while code-specialized models peak at more detailed abstraction; richer prompts such as chain-of-thought only pay off when the API context is already rich.","The approach improves the precision–recall profile where it matters operationally: it keeps high recall while increasing precision relative to raw-callee baselines, meaning fewer false positives without losing detections.","Resource-management CWEs such as double free, resource leak, and uncontrolled consumption push the needed abstraction level upward, while boundary-error CWEs such as out-of-bounds read and write are detectable even at the coarsest level."],"supporting_citations":[{"why":"Supplies the pre-selected list of primitive APIs whose usage summaries the whole abstraction is built on.","marker":"[39]"},{"why":"Documents that inter-procedural vulnerabilities span roughly 2.8 call layers, justifying the three-layer analysis scope.","marker":"[25]"},{"why":"Provides the repository-context baseline method of raw callee selection strategies that PacVD is compared against.","marker":"[50]"},{"why":"Shows LLMs miss context-dependent vulnerabilities unless given data-flow and API information, the gap the paper targets.","marker":"[56]"},{"why":"Finds LLM-only vulnerability detection near random without added context, motivating context enhancement.","marker":"[42]"},{"why":"Constructs the original vulnerability dataset lineage that the study expands and also grounds the security-operation API inspiration.","marker":"[28]"},{"why":"Expands the NVD-New dataset with further vulnerable and non-vulnerable samples from open-source projects.","marker":"[59]"}],"fun_headline_variants":["Program analysis context sharpens LLM vulnerability spotting","API summaries give LLMs the edge on memory bugs","Context-aware prompts lift LLM vuln detection F1 to 69%","Granular code context unlocks better LLM security scans","Three-layer call analysis boosts LLM bug catch rates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a hand-picked list of security-sensitive primitive APIs (malloc, free, open, close, and similar) plus summaries of their branch conditions and call counts is enough to expose the root cause of the targeted vulnerabilities; if the dangerous behavior flows through an API outside that list, or through semantics the summaries discard, the added context contributes nothing.","fun_headline_variants_meta":{"raw":{"variants":["Program analysis context sharpens LLM vulnerability spotting","API summaries give LLMs the edge on memory bugs","Context-aware prompts lift LLM vuln detection F1 to 69%","Granular code context unlocks better LLM security scans","Three-layer call analysis boosts LLM bug catch rates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1421,"prompt_tokens":1029,"completion_tokens":392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":312}},"tokens_in":645,"tokens_out":392,"duration_ms":3669,"temperature":1.0,"reasoning_tokens":312,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:53:31.148717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or collect an inter-procedural memory-safety vulnerability whose root cause flows through a custom function not on the primitive API list—for example, a hand-rolled memory pool or a reference-count macro—then run PacVD with and without the abstraction. If a strong LLM detects it from raw callee code but not from the API abstraction, while no listed API appears in the call chain, the claim that primitive APIs are an adequate context lens for memory-safety vulnerabilities fails for that class.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pre-selected list of primitive APIs whose usage summaries the whole abstraction is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that inter-procedural vulnerabilities span roughly 2.8 call layers, justifying the three-layer analysis scope."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows LLMs miss context-dependent vulnerabilities unless given data-flow and API information, the gap the paper targets."}],"review_version":1}