{"id":"e943c263-082f-42ce-af00-630b495681ba","arxiv_id":"2506.17798","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An LLM-driven dependency vulnerability detector reports 83.8% precision, 73.8% recall, and 78.5% F1 on 55 Java projects, outperforming two SCA baselines.","lead":"SAVANT uses large language models and proof-of-vulnerability tests to decide whether a known vulnerable library API is actually exploitable in a Java application. On 55 real-world projects it reports higher precision and recall than two existing software composition analysis tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark labels are likely co-derived from the same proof-of-vulnerability tests that SAVANT uses as seeds, so the headline metrics may measure reproduction of Test Mimicry labels rather than true exploitability.","rationale":"I read the paper in good faith: the architecture is coherent, the table arithmetic is internally consistent (31 TP, 6 FP, 7 TN, 11 FN reproduce precision 0.838, recall 0.738, accuracy 0.691, F1 0.785), and the authors are transparent about thresholds, model choices, and failures of baseline tools. The central claim, however, requires that the 55 benchmark labels reflect actual exploitability and that SAVANT's PoV-test seeds are not the same artifacts used to create those labels. The reader's weakest assumption captures exactly this, and my review found no reason to set it aside. I considered alternative concerns such as the accuracy metric being below the all-vulnerable majority baseline (76.4% vs. 69.1%) and the unfairness of counting baseline crashes as misses, but these affect how strongly the result can be advertised, not whether the result measures what it claims. The deeper risk is that the benchmark's test-mimicry construction and SAVANT's PoV-seeded retrieval are two ends of the same pipeline, so the reported performance may be circular. The manual audit in Section 5.1 is a partial defense, but it is not blind: the auditors know the benchmark labels and start from the same known vulnerable APIs. Since no code, prompts, or raw per-project evidence are released, independent verification is currently impossible. The condition I would impose is therefore independent re-labeling and a PoV-removal sanity check. If the independent labels agree and metrics hold, the paper's central claim is supported; if not, the reported numbers are uninterpretable. This matches the reader's CONDITIONAL verdict, so I do not change it.","tokens_in":19956,"tokens_out":13231,"duration_ms":120910,"concrete_test":"Construct independent ground-truth labels for a stratified sample of at least 20 of the 55 projects (ideally all 55) without consulting the Test Mimicry labels or SAVANT's outputs: for each pinned project commit, have security engineers independently write or manually trace executable PoV-based tests and data-flow paths to determine exploitability, reporting inter-rater agreement. Recompute Table 3 against these independent labels. If label agreement is low, or SAVANT's F1 drops by more than about 5 points, the reported effectiveness is an artifact of benchmark co-derivation; if labels agree and metrics hold, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SAVANT's detection loop is seeded by proof-of-vulnerability (PoV) tests (Section 4.3.1, Equation 5), and the final query in Section 4.3.3 includes the PoV test. The benchmark used for ground truth (Section 5.1, [34]) is Kang et al.'s Test Mimicry benchmark, whose labels are constructed by adapting library PoV tests to downstream projects. Thus the same type of artifact that underlies the label is given to SAVANT as input. High agreement (83.8% precision, 73.8% recall) may then only show that SAVANT is good at retrieving code resembling a PoV test, not that those projects are 'actually impacted' as the abstract claims. The manual audit in Section 5.1 mitigates this only partially: it traces from the same known vulnerable APIs and is performed by authors who know the benchmark labels, so confirmation bias remains. This is a circularity in the evaluation, not in the algorithm's internal logic, and it makes the central empirical claim unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAVANT, a two-phase LLM-based system for detecting whether Java applications are actually impacted by known vulnerabilities in their third-party library dependencies. Phase I segments application source code with an AST-based heuristic and embeds the segments; Phase II retrieves candidate code blocks using vulnerable API signatures and proof-of-vulnerability (PoV) test cases as seeds, iteratively expands context via LLM reflection, and asks an LLM to decide whether each candidate is exploitable. The evaluation on 55 Java projects from the Test Mimicry benchmark reports that SAVANT achieves 83.8% precision, 73.8% recall, 69.0% accuracy, and 78.5% F1, outperforming Eclipse Steady and VAScanner, with an additional ablation study across six LLMs, two embedding models, and several code-segment size thresholds.","tokens_in":20133,"tokens_out":15205,"duration_ms":126536,"significance":"If the empirical claims held, SAVANT would be a practically useful advance for software composition analysis, since it targets the well-known false-positive problem of version-based scanners and the scalability limitations of full call-graph reachability analysis. The motivating example (Kylin's safe use of BCryptPasswordEncoder) is well chosen, and the paper's architecture is clearly described. The per-project results in Table 2 are detailed enough to allow independent recomputation of SAVANT's own metrics: I confirmed precision 0.838, recall 0.738, accuracy 0.691, and F1 0.785 from the table, which is a strength. The ablation across LLMs and embedding models is also useful and gives a sense of robustness across model choices. However, the evaluation's ground-truth construction overlaps with the method's own seed signal, and the baseline comparison table is not internally consistent with the per-project results; these issues make the central 'outperforms state-of-the-art SCA tools' claim unverified as presented.","major_comments":[{"comment":"The evaluation is vulnerable to circularity. SAVANT's candidate identification in Eq. (5) is seeded by the proof-of-vulnerability test ftest, and the final reachability query in Section 4.3.3 also includes ftest. The ground-truth labels, however, come from Kang et al.'s Test Mimicry benchmark [34], whose labels are constructed by adapting library PoV tests to downstream projects. Thus the same type of artifact that underlies the benchmark labels is given to SAVANT as input. High agreement may therefore show that SAVANT retrieves code resembling a PoV test rather than that the 31 flagged projects are truly exploitable. The manual audit in Section 5.1, which traces from known vulnerable APIs and is performed by authors who know the benchmark labels, does not break this circularity. Please provide an evaluation on ground truth obtained without using PoV tests (for example, independent exploitability validation or a reachability-based labeling procedure), and include an ablation that removes the PoV seed while keeping only vulnerable API signatures as input.","section":"§4.3.1, §4.3.3, §5.1"},{"comment":"The headline comparison cannot be reproduced from the published per-project results. Using the stated convention of Table 2 (✓ secure, ✗ insecure, - failed) and excluding failed projects from the denominator, I recompute Eclipse Steady's precision as 15/19 = 0.789 and recall as 15/42 = 0.357, whereas Table 3 reports 0.700 precision and 0.241 recall. For VAScanner, the same exercise gives precision 13/15 = 0.867 and recall 13/42 = 0.310, not the reported 0.73 and 0.262. Other treatments of the '-' entries yield other values, but none match Table 3. The exact denominators, the treatment of failed runs, and the counting rule must be specified and the two tables reconciled before the claimed advantage over existing SCA tools can be assessed.","section":"Table 2 and Table 3"},{"comment":"The core similarity threshold τ is never disclosed. Eq. (5) defines candidate selection by the condition Sc > τ, and Section 6.1 admits that parameter sensitivity has not been analyzed and is left to future work. Because τ directly controls which code blocks enter the pipeline, the reported 83.8% precision and 78.5% F1 are not reproducible without this value. In addition, no repeated LLM runs or confidence intervals are reported; given the known non-determinism of LLM-based analysis (e.g., SecLLMHolmes [64]), please report the value of τ, provide a sensitivity sweep over τ, and give repeated-run statistics for the headline metrics.","section":"§4.3.1, Eq. (5), §6.1"}],"minor_comments":[{"comment":"The figure contains typos: 'Candidata Identification', 'Relection-based Context Retrival', and 'Semantic-guilded Reachability Analysis' should be corrected.","section":"Figure 1"},{"comment":"There are several language issues: the abstract says 'we proposed SAVANT' where 'we propose' is intended; Section 4.3.2 says 'SAVANT querys' instead of 'queries'; and the Section 4.3.3 heading reads 'Sematic-Guided Reachability Analysis'.","section":"§4.3.2, §4.3.3, Abstract"},{"comment":"The notation in Eq. (5) is inconsistent with the database definition in Section 4.2.2: D stores tuples di = (bi, vi), but Ccandidate is written as ci = (vi, bi, vapi) ∈ D. Please align the tuple definitions.","section":"§4.3.1, Eq. (5)"},{"comment":"The caption does not state how '-' (tool failed to run) entries were counted when computing the metrics in Table 3; even after the numbers are corrected, the counting rule should be explicit in the caption.","section":"Table 2"},{"comment":"The legend distinguishes embedding models only by bar color ('green bars' vs 'purple bars'), which is not legible in grayscale; adding direct labels or hatching would improve readability.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the benchmark-label overlap with the PoV seed signal; if the authors cannot obtain or construct an independent ground truth, the claims should be substantially softened to 'agreement with Test Mimicry labels' rather than 'actually impacted by vulnerable APIs.' The disagreement between Table 2 and Table 3 is a factual inconsistency that the editor should ensure is resolved before further review. The paper also provides no artifact or code link, which matters here because the method relies on proprietary LLM APIs and an undisclosed threshold."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the system is worth knowing about, but the evaluation as written does not support the headline claim. The architecture is a sensible combination of known ingredients: PoV test seeds, semantic embeddings, and LLM reflection for reachability-style analysis. That specific pipeline is new relative to the cited work, and the paper addresses a genuine industrial pain point. The ablation across LLMs, embedding models, and segment sizes is more thorough than most papers in this space, and the reported metrics do recompute correctly from Table 2.\n\nThe soft spot is the circularity between the benchmark and the method's input. The ground truth comes from Kang et al.'s Test Mimicry benchmark ([34]), whose labels are constructed by adapting library proof-of-vulnerability tests to downstream projects. SAVANT is seeded with those same PoV tests (Section 4.3.1, Equation 5) and includes the PoV test in the final query. So high precision and recall may only show that SAVANT is good at finding code that resembles a PoV test, not that those projects are actually exploitable. The manual audit in Section 5.1 mitigates this only partially: it traces from the same known vulnerable APIs, and the authors know the benchmark labels. I don't think this is a hidden fraud; it is a benchmark-design trap that the authors walked into, and they did disclose the dataset source and their audit process.\n\nOther issues are less severe but still real. No code or prompts are released, the similarity threshold tau is undisclosed, and there are no repeated-run confidence intervals for LLM nondeterminism. The baseline comparison also counts analysis failures as misses: Eclipse Steady failed on 17 projects, VAScanner on 35, while SAVANT analyzed all 55. Treating 'failed to run' as 'did not detect' inflates the relative performance. The threats-to-validity section is honest about parameter sensitivity and ground truth, but it does not acknowledge the structural circularity.\n\nThese are fixable with a re-designed independent label set, released artifacts, and a baseline comparison that separates 'no finding' from 'could not analyze.' As it stands, I would not trust the 83.8% precision / 78.5% F1 as evidence of real exploitability detection. The idea is promising enough that a serious referee should see it: the paper deserves a round of revision, not a desk reject. I would read it again if the authors address the circularity and release the artifacts. If you work on SCA or LLM-based vulnerability detection, this is worth your attention; otherwise, wait for the revised version.","headline":"SAVANT is a plausible LLM-based SCA pipeline with a real circularity problem: the benchmark labels and the method's seeds come from the same proof-of-vulnerability artifacts, so the headline numbers likely measure retrieval of PoV-like code, not true exploitability.","tokens_in":20679,"tokens_out":1819,"would_cite":false,"duration_ms":20348,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAVANT claims that LLM reflection over proof-of-vulnerability tests can decide whether a known vulnerable library API is actually exploitable in each application, reporting 83.8% precision and 73.8% recall on 55 Java projects.","keywords":["software composition analysis","vulnerable API detection","proof-of-vulnerability test","LLM-based code analysis","semantic reachability","AST code segmentation","Java dependency security","supply chain security"],"falsifier":"Have an independent team produce ground-truth labels for the same 55 applications without seeing SAVANT's outputs or the PoV test functions, then rerun SAVANT with PoV tests withheld from the retrieval stage; if recall drops sharply, the PoV seeds were leaking the answer rather than providing generalizable semantics.","tokens_in":19735,"feed_emoji":"🛡️","tokens_out":5942,"duration_ms":54719,"temperature":0.7,"pith_summary":"The paper tries to establish that software composition analysis can move from 'which library version is vulnerable' to 'is this application actually exploitable through a vulnerable library API.' It proposes SAVANT, a two-phase pipeline that first segments an application's Java source into AST-aligned code blocks and embeds them, then uses the vulnerable API's signature and its proof-of-vulnerability test as seeds to retrieve relevant code, iteratively asks an LLM to notice missing context, and finally asks the LLM to judge whether the vulnerable API is reachable and exploitable. On a third-party benchmark of 55 real-world Java applications, SAVANT reports 83.8% precision, 73.8% recall, 69.0% accuracy, and 78.5% F1, outperforming two existing SCA tools, which fail to analyze 17 and 35 projects respectively. The practical stake is that a correct SAVANT would let development teams ignore harmless version matches and focus fixes on genuinely reachable vulnerabilities.","feed_headline":"Semantic reachability finds real library hits at 83.8% precision","feed_subtitle":"SAVANT pairs proof-of-vulnerability tests with LLM context analysis to cut false alerts across 55 Java apps","key_machinery":"The load-bearing mechanism is the reflection loop over semantic retrieval. SAVANT maintains a database of AST-segmented code blocks paired with embeddings; for each candidate call site it loops between Q, an LLM reflection query that reports whether the current code context is sufficient and why not; P, an LLM code-inference query that names the missing code snippet and structural scope; and Search, embedding similarity with scope filtering. The proof-of-vulnerability test functions as the semantic anchor that tells the loop what 'triggered' looks like, replacing the work a call-graph reachability analysis would do.","core_discovery":"SAVANT's central claim is that proof-of-vulnerability test cases, which security patches ship to show how a vulnerability is triggered, encode the semantic conditions of exploitability, and that an LLM can use those conditions to perform reachability analysis without a complete call graph. The pipeline starts from the vulnerable API signature and the PoV test, retrieves semantically similar source blocks by embedding cosine similarity, gates them with an LLM grader to confirm real API invocations, then iteratively reflects: the LLM says whether its context is sufficient, generates a query for missing code and scope constraints, retrieves those blocks, and repeats until context is complete or no new blocks are found. The final decision is disjunctive: the application is labeled vulnerable if any context-complete candidate makes the LLM answer 'vulnerable', and secure only if every candidate is judged safe. On the evaluation benchmark SAVANT identifies 31 of 42 vulnerable projects correctly plus 6 false positives, and the authors report F1 between 0.72 and 0.87 across six LLMs.","pith_inferences":["If SAVANT's advantage comes from PoV tests, then a corollary not tested in the paper is that libraries without PoV tests will see degraded recall; a natural extension is to synthesize PoV-like triggers from CVE descriptions or patch diffs.","The disjunctive final rule (any vulnerable candidate wins) explains the precision/recall balance and implies that a single hallucinated 'vulnerable' answer overrides many secure judgments; measuring per-candidate agreement across repeated LLM runs would quantify this fragility.","The method could be turned into a differential analyzer: run SAVANT on two consecutive commits of a project to isolate which dependency upgrade actually removes exploitable reachability, giving developers an upgrade-priority signal.","Because ground truth is the shaky plank, a strong extension is to build a new benchmark of projects whose exploitability is confirmed by executable tests rather than manual audit, then rerun the comparison."],"forward_implications":["Version-based scanners can be triaged: projects whose vulnerable dependency versions are flagged can be filtered by whether the specific API is invoked and whether context shows exploitable conditions, removing false positives like the safe BCryptPasswordEncoder use in Apache Kylin.","Call-graph-free reachability becomes feasible for large codebases; SAVANT analyzed all 55 projects while the baseline tools each failed on a large subset due to memory and classpath limitations.","The method's accuracy is coupled to LLM semantic ability and context completeness: across six models F1 spans 0.64 to 0.87, and segment sizes of 2,000 to 2,500 tokens give the best results.","The authors state the approach is theoretically language-agnostic, so the same two-phase pipeline could be applied to other ecosystems, though the paper only evaluates Java."],"supporting_citations":[{"why":"Supplies the 55-project benchmark and the ground-truth labels SAVANT is evaluated against.","marker":"[34]"},{"why":"Eclipse Steady, the reachability-analysis baseline that failed on 17 projects.","marker":"[22]"},{"why":"VAScanner, the call-graph baseline that failed on 35 projects.","marker":"[73]"},{"why":"Provides the self-reflective retrieval-augmented generation idea adapted into SAVANT's reflection loop.","marker":"[14]"},{"why":"Supplies the embedding models used for semantic-preserving code retrieval.","marker":"[50]"},{"why":"Documents performance degradation with longer LLM sequences, motivating AST-based code segmentation.","marker":"[23]"}],"fun_headline_variants":["LLM-guided reachability catches real library exploits","SAVANT: PoV tests + LLM to find reachable vulnerabilities","LLM reads API context to trace real vulnerability impacts","Semantic-guided reachability: LLM outperforms SCA tools","Proof-of-vulnerability tests drive LLM dependency scanner"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 55-project ground-truth labels, taken from the third-party benchmark plus the authors' manual audit, are correct and independent of the proof-of-vulnerability tests SAVANT is seeded with; if a label was derived from the same PoV test, the reported precision and recall would not transfer to fresh projects.","fun_headline_variants_meta":{"raw":{"variants":["LLM-guided reachability catches real library exploits","SAVANT: PoV tests + LLM to find reachable vulnerabilities","LLM reads API context to trace real vulnerability impacts","Semantic-guided reachability: LLM outperforms SCA tools","Proof-of-vulnerability tests drive LLM dependency scanner"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0006,"raw_usage":{"total_tokens":2814,"prompt_tokens":965,"completion_tokens":1849,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1765}},"tokens_in":581,"tokens_out":1849,"duration_ms":12334,"temperature":1.0,"reasoning_tokens":1765,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:01:03.740671+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent team produce ground-truth labels for the same 55 applications without seeing SAVANT's outputs or the PoV test functions, then rerun SAVANT with PoV tests withheld from the retrieval stage; if recall drops sharply, the PoV seeds were leaking the answer rather than providing generalizable semantics.","supporting_citations":[{"cited_title":"Large language model-powered smart contract vulnerability detection: New perspectives","cited_arxiv_id":null,"evidence_quote":"Supplies the 55-project benchmark and the ground-truth labels SAVANT is evaluated against."},{"cited_title":"Bugtreemodel.java","cited_arxiv_id":null,"evidence_quote":"Eclipse Steady, the reachability-analysis baseline that failed on 17 projects."},{"cited_title":"Automated Program Repair in the Era of Large Pre- trained Language Models","cited_arxiv_id":null,"evidence_quote":"VAScanner, the call-graph baseline that failed on 35 projects."},{"cited_title":"Kyli- nUserService.java - Apache Kylin","cited_arxiv_id":null,"evidence_quote":"Provides the self-reflective retrieval-augmented generation idea adapted into SAVANT's reflection loop."},{"cited_title":"Embedding models: Openai docu- mentation","cited_arxiv_id":null,"evidence_quote":"Supplies the embedding models used for semantic-preserving code retrieval."},{"cited_title":"Llmeffichecker: Understanding and testing ef- ficiency degradation of large language models","cited_arxiv_id":null,"evidence_quote":"Documents performance degradation with longer LLM sequences, motivating AST-based code segmentation."}],"review_version":1}