{"id":"a34ca3be-e31b-4d32-a562-8ec72b6f1278","arxiv_id":"1909.00973","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper claims that merging static and dynamic call graphs improves vulnerability reachability analysis in software composition analysis, but the evaluation does not directly measure false positive reduction.","lead":"This paper from Veracode describes a system that combines static and dynamic call graphs to find library vulnerabilities in Java applications. It reports that adding runtime call data discovers many more vulnerable code paths, but the evidence is based on only four projects with no ground-truth validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Union of static and dynamic call graphs is never shown to reduce false positives; added sinks may be test-only, leaving the central claim unsupported.","rationale":"The reader's weakest_assumption correctly identifies that dynamic edges are assumed to be feasible and that the union brings the graph closer to the ideal. I agree, and I sharpen the concern by noting the test-versus-production mismatch: even if dynamic edges are feasible in the test run, they may not be feasible in the deployed application. The paper's own Section IV.C emphasizes that dynamic graphs never produce infeasible edges, but that is only relative to the specific test execution. For SCA, false positive elimination must reason about production reachability, and tests are not production. The evaluation uses sink counts only, which is a recall-oriented metric; a higher sink count could mean more true positives or more false positives. The proposed concrete test—checking whether the added sinks are reachable from production entry points—would settle whether the union helps or hurts. This is a load-bearing threat because the entire claimed benefit of the dynamic composition rests on the relevance of test executions to production reachability. Therefore, the verdict of REJECT remains appropriate, as the central claim is unsupported by the presented evidence.","tokens_in":6956,"tokens_out":4650,"duration_ms":48346,"concrete_test":"For helios and java-apns, randomly sample 50 of the newly discovered dynamic sinks and trace each sink's call path from a production entry point (e.g., a main method, servlet, or message handler) while excluding paths that require a test runner such as JUnit or TestNG. If the majority of sampled sinks are reachable only through test entry points, then the dynamic edges are not production-relevant, and the union does not improve false positive elimination; it introduces test-only false positives. This would directly falsify the paper's central claim. If, instead, most sampled sinks are reachable from production entry points, the concern would be alleviated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims that combining static and dynamic call graphs 'improve[s] the performance of false positive elimination,' but the evaluation in Section VI reports only vertex/edge counts and sink counts—surrogate metrics that measure recall, not false positive reduction. The composition step in Section IV.D takes the union of the static graph (an over-approximation) and the dynamic graph (a under-approximation built from test executions, per Section IV.C). Because dynamic edges are observed under test, they are feasible only for the test execution, not necessarily for production executions. The paper explicitly identifies framework entry points such as JUnit (Section IV.E.3) but never filters paths that are only reachable through test runners or test fixtures. Consequently, the dynamic graph can add edges that are infeasible in the deployed application, so the union can increase false positives instead of eliminating them. The ideal call graph defined in Section IV.D as 'the union of dynamic call graphs across all possible executions' is ambiguous: if test executions count as 'possible,' the ideal is not aligned with production reachability, which is what SCA false positive elimination requires. No ground truth is provided to show that the newly discovered sinks (e.g., helios going from 1 to 3,616) are truly reachable in production. Without a precision measurement or ground-truth comparison, the central claim that false positive elimination improves is not merely unproven; the direction of the evidence could even contradict it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the architecture of a commercial Software Composition Analysis (SCA) product covering dependency discovery, call-graph-based checking of whether vulnerability-specific sinks are reachable, and automated remediation. Its central claim, stated in the abstract, is that combining call graphs derived from static and dynamic analysis improves the performance of false positive elimination. The evaluation in Section VI examines four Java projects and reports that dynamic call graphs add on average 824% more vertices and 361% more edges, and that the number of reachable vulnerable-method sinks rises sharply in two of the four projects (e.g., helios from 1 to 3,616).","tokens_in":7140,"tokens_out":2596,"duration_ms":25142,"significance":"If the central claim were substantiated, the result would be practically valuable: SCA tools could discover many more genuinely reachable vulnerable methods and thereby reduce false positives. The paper has concrete strengths: a modular composition of precomputed library call chains, hand-curated vulnerability-specific sinks, a clearly stated architecture, and a microbenchmark-based evaluation of dynamic dependency discovery covering 18 package managers. However, the main evaluation does not measure the claimed outcome, false positive elimination, and the theoretical justification for why the union should approach the ideal call graph is not tied to production-relevant executions. The significance of the paper is therefore limited by a mismatch between the claim and the evidence.","major_comments":[{"comment":"The evaluation reports only vertex counts, edge counts, and sink counts; it does not measure false positive elimination, which is the central claim of the paper. The abstract promises improvements in 'the performance of false positive elimination,' but no precision, false-positive-rate, or ground-truth comparison is presented anywhere in Section VI. The reported numbers are surrogate metrics that mostly reflect recall, so the primary claim is unsupported by the evidence.","section":"Section VI, Table I"},{"comment":"Dynamic call graphs are constructed from test executions, and Section IV.C explicitly notes that only flows observed during tests are reported. The composition step in Section IV.D takes the union of the static graph (an over-approximation) and the dynamic graph (an under-approximation for the test runs). Because dynamic edges are feasible only for the specific test executions, and the paper never filters or labels edges that are reachable only through test runners, test fixtures, or framework inversion-of-control (Section IV.E.3, JUnit), the union can add edges that are infeasible in the deployed application. This means the composition can increase, rather than eliminate, false positives; the paper provides no argument or measurement to rule out this scenario.","section":"Section IV.C and IV.D"},{"comment":"The ideal call graph is defined as 'the union of dynamic call graphs across all possible executions of a program.' This definition is ambiguous: if test executions are counted as 'possible,' the ideal graph is not aligned with production reachability, which is what SCA false-positive elimination requires. If test executions are excluded, then the dynamic graph used in the composition is not a subset of the ideal graph, contradicting the intuition in Figure 1 that the union brings the analysis closer to the ideal. The ambiguity needs to be resolved before the theoretical motivation for the union can be accepted.","section":"Section IV.D, Figure 1"},{"comment":"The increase from 1 static sink to 3,616 dynamic sinks in helios is presented as a benefit, but without any validation that those 3,616 sinks are reachable in a deployed configuration, the same number could equally represent a false-positive explosion. A sample manual inspection, a precision calculation, or any external ground truth is required to determine whether this result demonstrates improvement.","section":"Section VI, helios row"}],"minor_comments":[{"comment":"Several references contain typos or malformed names, e.g., 'V on Krogh' [25] and 'Ondej Lhotk' [28].","section":"References"},{"comment":"The sentence 'chains comprising only feasible edges may themselves prove be infeasible' contains a grammatical error ('prove be'); it should read 'prove to be infeasible.'","section":"Section IV.B, paragraph 3"},{"comment":"The claim that 'commercial SCA products report that 70-80% of dependencies are never referenced in application code' lacks a citation; please provide a source for this statistic.","section":"Section IV.A, paragraph 2"},{"comment":"The comparison with [26] identifies 'hand-curated, vulnerability-specific sinks' as a key innovation, but the discussion does not clarify how these sinks differ from the code-centric sinks in the closely related work; a brief concrete differentiation would strengthen the novelty claim.","section":"Section II"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this as an industrial experience report, not a research paper. The dependency-discovery part is solid: integrating with package managers and measuring the result on 41 microbenchmarks across 18 package managers is the strongest evidence in the paper, and the 204% improvement claim there is credible. The modular precomputation of vulnerability-specific call chains per library, merged into the combined graph with a suffix-matching step, is a practical idea worth stealing, and the authors are honest about its limitations (e.g., Section IV.F on intra-library chains).\n\nThe central problem is the false-positive-elimination claim. The abstract says combining static and dynamic call graphs improves false positive elimination, but the evaluation in Section VI reports only vertices, edges, and sink counts. Sink counts are a recall metric, not a precision metric. A dynamic call graph from test executions can add edges that are feasible only in the test harness — the authors even mention JUnit framework entry points without filtering them out. Unioning a static overapproximation with a dynamic underapproximation does not guarantee the result is closer to the ideal call graph; it just guarantees you see more sinks, some of which may be test-only. The helios jump from 1 to 3,616 sinks sounds dramatic, but without ground truth you cannot tell whether those 3,616 sinks are reachable in production or are artifacts of a test runner pulling in third-party code. The paper does not compare against the closest prior work [26], so we have no baseline for false positive reduction either.\n\nThese are not minor blemishes. The central claim is load-bearing and the evaluation does not measure it. That said, the paper is not incoherent: the architectural description is clear, the related work is properly situated, and the dependency-discovery evaluation stands on its own. The stress-test concern holds up.\n\nWho gets value from this? Practitioners building SCA tooling will find the modular call-chain composition worth copying. Researchers will find the evaluation insufficient but the problem statement useful. I would send it to peer review because the architecture is relevant and the dependency evaluation is solid, but I would expect major revision: either soften the abstract to claim 'sink recall improves,' or supply a precision/ground-truth study on reachable vs. test-only sinks. As it stands, the central claim overreaches the evidence.","headline":"A useful industrial SCA architecture paper whose headline claim about false-positive elimination is not supported by the evaluation, which measures sink counts rather than precision.","tokens_in":7711,"tokens_out":1953,"would_cite":false,"duration_ms":22075,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that Software Composition Analysis finds more genuinely reachable vulnerable methods by merging static call graphs with call graphs recorded from instrumented test runs, because the union approximates the ideal call graph…","keywords":["software composition analysis","call graph","static analysis","dynamic analysis","reachability analysis","vulnerability detection","false positive elimination","Java"],"falsifier":"Take one reported case, helios, whose reachable vulnerable-method sinks grow from 1 to 3,616 after the merge, and build a ground-truth list of which of those 3,616 sinks can actually be reached from an application entry point on any real execution path. If most of the added sinks are not truly reachable in representative executions, the central claim that the union improves false-positive elimination is refuted.","tokens_in":6713,"feed_emoji":"🔍","tokens_out":11843,"duration_ms":107670,"temperature":0.7,"pith_summary":"Software Composition Analysis (SCA) audits the open-source libraries an application depends on, and one of its key tasks is deciding whether a known vulnerable library method is actually reachable from application code. This paper argues that static call graphs alone are wrong in both directions: they contain infeasible edges that cause false positives, and they miss dynamic framework and reflection flows that cause false negatives. Its proposed fix is to take the union of the static call graph with a call graph recorded by instrumenting and running the project's tests, then check reachability of hand-curated, vulnerability-specific sinks in the merged graph. Across four Java projects the dynamic graph added on average 824% more vertices and 361% more edges, and in one project, helios, the number of reachable vulnerable-method sinks grew from 1 to 3,616. If the union really is closer to the ideal call graph, SCA tools would report many fewer missed vulnerabilities and would target remediation more accurately.","feed_headline":"Merged call graphs expose 3,600+ new vulnerable sinks","feed_subtitle":"Dynamic graphs added 824% more vertices and 361% more edges, revealing many more reachable sinks.","key_machinery":"The load-bearing object is the combined call graph $G_c = G'_s \\cup G'_d$. $G'_s$ is built from static analysis by adding Class Hierarchy Analysis edges, Rapid Type Analysis refinements, and a 'soundy' reflection pass, then restricting the graph to methods reachable from first-party entry points. $G'_d$ is built by instrumenting test runs and keeping only the transitive closure of framework entry points, so all its edges are observed and feasible and its entry points remain first-party. Library-side vulnerable method call chains $CC$ — paths from a library entry point to a hand-curated sink method — are precomputed once per library and merged into $G'_s$ only as suffixes of existing edges, which preserves first-party reachability and makes sink reachability a set-membership check. The union $G_c$ is what lets downstream checks see paths that span static and dynamic edges, including flows that appear only under framework inversion of control.","core_discovery":"Stated as the authors would put it: statically constructed call graphs are soundy approximations, not exact models, and dynamically observed call graphs are always feasible but incomplete; therefore taking the union of the two moves the analysis toward the ideal call graph, which is the union of all dynamic graphs over every possible execution. The paper's concrete demonstration is that on four real Maven-based Java projects, the dynamic call graph contributes 824% more vertices and 361% more edges on average, with most of the extra edges coming from third-party dependencies. The reachability payoff appears in the sink counts: helios goes from 1 to 3,616 reachable vulnerable-method sinks, and java-apns from 58 to 859, while the other two projects show no change. The practical enabler is that vulnerable method call chains are precomputed per library and merged modularly into the application graph, so the analysis can be repeated in CI pipelines without rebuilding library call graphs on demand.","pith_inferences":["Labeling each newly discovered sink with exploitability ground truth in a follow-up study would separate true precision gains from added coverage, and would quantify how much of the union's benefit is genuine.","Comparing test-derived dynamic graphs with dynamic graphs traced from production traffic would directly test the central assumption: the union should help most when test coverage resembles production usage.","If helios-scale increases generalize, static-only SCA rankings systematically understate exposure in framework-heavy applications, so prioritization of dependency upgrades would need to weight dynamic reachability evidence.","The hand-curated sink method itself is a scaling bottleneck; mining sink methods from CVE patch diffs or commit-level fixes would let the same union machinery cover new vulnerabilities without manual curation."],"forward_implications":["SCA reachability checks will report vulnerabilities inside framework-driven and reflection-heavy flows that static-only scans cannot see, because the dynamic half supplies feasible edges for those flows.","Because library call chains are precomputed and only merged into the application graph, the combined reachability analysis can run in minutes, making it practical inside CI/CD pipelines rather than a one-off deep audit.","Automated remediation can use the dynamic edges to avoid flagging library upgrades as breaking when the only path to a changed method was an infeasible static call chain.","For projects with little or no test coverage, the dynamic half contributes little, so the benefit of the union is conditional on the project already having a runnable, reasonably covering test suite."],"supporting_citations":[{"why":"establishes the ideal call graph as the union of dynamic graphs over all executions, with the static graph as a superset, which motivates taking the union.","marker":"[28]"},{"why":"documents that about 25.5% of statically constructed call chains are infeasible, the false-positive problem this work targets.","marker":"[16]"},{"why":"is the closest prior SCA approach that also mixes static and dynamic analysis; the paper distinguishes its curated sinks and modular construction from it.","marker":"[26]"},{"why":"supplies the application-only call graph construction idea that makes precomputed, modular library call chains feasible.","marker":"[31]"},{"why":"provides the Class Hierarchy Analysis pass used to add static call graph edges for possible receiver classes.","marker":"[4]"},{"why":"provides the Rapid Type Analysis pass that prunes receiver classes using instantiation information.","marker":"[5]"},{"why":"supplies the soundiness argument that justifies treating reflection and other hard features as unsound-but-practical in static graph construction.","marker":"[19]"},{"why":"is the paper's earlier remediation analysis whose semantic diffs are augmented with the combined call graphs.","marker":"[1]"}],"fun_headline_variants":["Merged call graphs reveal 3,600+ new vulnerable sinks in one project","Dynamic call graphs add 824% more vertices, exposing far more reachable sinks","Union of static and dynamic analysis boosts vulnerability discovery","Modular call graph merging sharpens false positive elimination","Combining call graph types uncovers thousands of additional vulnerable methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's central bet is that running the project's test suite gives a picture of real executions that can be merged with static edges to approximate how the software actually runs; if the tests do not resemble production usage, the extra reachable sinks it reports are not real false-positive eliminations.","fun_headline_variants_meta":{"raw":{"variants":["Merged call graphs reveal 3,600+ new vulnerable sinks in one project","Dynamic call graphs add 824% more vertices, exposing far more reachable sinks","Union of static and dynamic analysis boosts vulnerability discovery","Modular call graph merging sharpens false positive elimination","Combining call graph types uncovers thousands of additional vulnerable methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1683,"prompt_tokens":859,"completion_tokens":824,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":734}},"tokens_in":475,"tokens_out":824,"duration_ms":8679,"temperature":1.0,"reasoning_tokens":734,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:30:32.359773+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one reported case, helios, whose reachable vulnerable-method sinks grow from 1 to 3,616 after the merge, and build a ground-truth list of which of those 3,616 sinks can actually be reached from an application entry point on any real execution path. If most of the added sinks are not truly reachable in representative executions, the central claim that the union improves false-positive elimination is refuted.","supporting_citations":[{"cited_title":"Comparing call graphs","cited_arxiv_id":null,"evidence_quote":"establishes the ideal call graph as the union of dynamic graphs over all executions, with the static graph as a superset, which motivates taking the union."},{"cited_title":"Static and dynamic analysis of call chains in Java","cited_arxiv_id":null,"evidence_quote":"documents that about 25.5% of statically constructed call chains are infeasible, the false-positive problem this work targets."},{"cited_title":"Beyond metadata: Code-centric and usage-based analysis of known vulnerabili- ties in open-source software","cited_arxiv_id":null,"evidence_quote":"is the closest prior SCA approach that also mixes static and dynamic analysis; the paper distinguishes its curated sinks and modular construction from it."},{"cited_title":"Application-only call graph construction","cited_arxiv_id":null,"evidence_quote":"supplies the application-only call graph construction idea that makes precomputed, modular library call chains feasible."},{"cited_title":"Optimization of object-oriented programs using static class hierarchy analysis","cited_arxiv_id":null,"evidence_quote":"provides the Class Hierarchy Analysis pass used to add static call graph edges for possible receiver classes."},{"cited_title":"Practical virtual method call resolution for Java","cited_arxiv_id":null,"evidence_quote":"provides the Rapid Type Analysis pass that prunes receiver classes using instantiation information."},{"cited_title":"In defense of soundiness: a manifesto","cited_arxiv_id":null,"evidence_quote":"supplies the soundiness argument that justifies treating reflection and other hard features as unsound-but-practical in static graph construction."},{"cited_title":"Efficient static checking of library updates","cited_arxiv_id":null,"evidence_quote":"is the paper's earlier remediation analysis whose semantic diffs are augmented with the combined call graphs."}],"review_version":1}