Pith. sign in

REVIEW 4 major objections 4 minor 31 references

The Dynamics of Software Composition Analysis

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read The paper claims that Software Composition Analysis finds more genuinely reachable vulnerable methods by merging static call graphs with call graphs recorded from instrumented test runs, because the union approximates the ideal call graph…

desk verdict A useful industrial SCA architecture paper whose headline claim about false-positive elimination is not supported by the evaluation, which measures sink counts rather than precision. read the letter →

arxiv 1909.00973 v2 pith:C3MMN3YW submitted 2019-09-03 cs.SE cs.PL

classification cs.SEcs.PL
keywords softwarecompositionanalysiscallgraphstaticdynamicreachabilityvulnerabilitydetectionfalsepositiveeliminationJava
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Software Composition Analysis (SCA) audits the open-source libraries an application depends on, and one of its key tasks is deciding whether a known vulnerable library method is actually reachable from application code. This paper argues that static call graphs alone are wrong in both directions: they contain infeasible edges that cause false positives, and they miss dynamic framework and reflection flows that cause false negatives. Its proposed fix is to take the union of the static call graph with a call graph recorded by instrumenting and running the project's tests, then check reachability of hand-curated, vulnerability-specific sinks in the merged graph. Across four Java projects the dynamic graph added on average 824% more vertices and 361% more edges, and in one project, helios, the number of reachable vulnerable-method sinks grew from 1 to 3,616. If the union really is closer to the ideal call graph, SCA tools would report many fewer missed vulnerabilities and would target remediation more accurately.

What carries the argument

The load-bearing object is the combined call graph $G_c = G'_s \cup G'_d$. $G'_s$ is built from static analysis by adding Class Hierarchy Analysis edges, Rapid Type Analysis refinements, and a 'soundy' reflection pass, then restricting the graph to methods reachable from first-party entry points. $G'_d$ is built by instrumenting test runs and keeping only the transitive closure of framework entry points, so all its edges are observed and feasible and its entry points remain first-party. Library-side vulnerable method call chains $CC$ — paths from a library entry point to a hand-curated sink method — are precomputed once per library and merged into $G'_s$ only as suffixes of existing edges, which preserves first-party reachability and makes sink reachability a set-membership check. The union $G_c$ is what lets downstream checks see paths that span static and dynamic edges, including flows that appear only under framework inversion of control.

What would settle it

Take one reported case, helios, whose reachable vulnerable-method sinks grow from 1 to 3,616 after the merge, and build a ground-truth list of which of those 3,616 sinks can actually be reached from an application entry point on any real execution path. If most of the added sinks are not truly reachable in representative executions, the central claim that the union improves false-positive elimination is refuted.

Watch

Extended reading notes

Core claim

Stated as the authors would put it: statically constructed call graphs are soundy approximations, not exact models, and dynamically observed call graphs are always feasible but incomplete; therefore taking the union of the two moves the analysis toward the ideal call graph, which is the union of all dynamic graphs over every possible execution. The paper's concrete demonstration is that on four real Maven-based Java projects, the dynamic call graph contributes 824% more vertices and 361% more edges on average, with most of the extra edges coming from third-party dependencies. The reachability payoff appears in the sink counts: helios goes from 1 to 3,616 reachable vulnerable-method sinks, and java-apns from 58 to 859, while the other two projects show no change. The practical enabler is that vulnerable method call chains are precomputed per library and merged modularly into the application graph, so the analysis can be repeated in CI pipelines without rebuilding library call graphs on demand.

Load-bearing premise

The method's central bet is that running the project's test suite gives a picture of real executions that can be merged with static edges to approximate how the software actually runs; if the tests do not resemble production usage, the extra reachable sinks it reports are not real false-positive eliminations.

Editorial extensions

If this is right

  • SCA reachability checks will report vulnerabilities inside framework-driven and reflection-heavy flows that static-only scans cannot see, because the dynamic half supplies feasible edges for those flows.
  • Because library call chains are precomputed and only merged into the application graph, the combined reachability analysis can run in minutes, making it practical inside CI/CD pipelines rather than a one-off deep audit.
  • Automated remediation can use the dynamic edges to avoid flagging library upgrades as breaking when the only path to a changed method was an infeasible static call chain.
  • For projects with little or no test coverage, the dynamic half contributes little, so the benefit of the union is conditional on the project already having a runnable, reasonably covering test suite.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Labeling each newly discovered sink with exploitability ground truth in a follow-up study would separate true precision gains from added coverage, and would quantify how much of the union's benefit is genuine.
  • Comparing test-derived dynamic graphs with dynamic graphs traced from production traffic would directly test the central assumption: the union should help most when test coverage resembles production usage.
  • If helios-scale increases generalize, static-only SCA rankings systematically understate exposure in framework-heavy applications, so prioritization of dependency upgrades would need to weight dynamic reachability evidence.
  • The hand-curated sink method itself is a scaling bottleneck; mining sink methods from CVE patch diffs or commit-level fixes would let the same union machinery cover new vulnerabilities without manual curation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper describes the architecture of a commercial Software Composition Analysis (SCA) product covering dependency discovery, call-graph-based checking of whether vulnerability-specific sinks are reachable, and automated remediation. Its central claim, stated in the abstract, is that combining call graphs derived from static and dynamic analysis improves the performance of false positive elimination. The evaluation in Section VI examines four Java projects and reports that dynamic call graphs add on average 824% more vertices and 361% more edges, and that the number of reachable vulnerable-method sinks rises sharply in two of the four projects (e.g., helios from 1 to 3,616).

Significance. If the central claim were substantiated, the result would be practically valuable: SCA tools could discover many more genuinely reachable vulnerable methods and thereby reduce false positives. The paper has concrete strengths: a modular composition of precomputed library call chains, hand-curated vulnerability-specific sinks, a clearly stated architecture, and a microbenchmark-based evaluation of dynamic dependency discovery covering 18 package managers. However, the main evaluation does not measure the claimed outcome, false positive elimination, and the theoretical justification for why the union should approach the ideal call graph is not tied to production-relevant executions. The significance of the paper is therefore limited by a mismatch between the claim and the evidence.

major comments (4)
  1. [Section VI, Table I] The evaluation reports only vertex counts, edge counts, and sink counts; it does not measure false positive elimination, which is the central claim of the paper. The abstract promises improvements in 'the performance of false positive elimination,' but no precision, false-positive-rate, or ground-truth comparison is presented anywhere in Section VI. The reported numbers are surrogate metrics that mostly reflect recall, so the primary claim is unsupported by the evidence.
  2. [Section IV.C and IV.D] Dynamic call graphs are constructed from test executions, and Section IV.C explicitly notes that only flows observed during tests are reported. The composition step in Section IV.D takes the union of the static graph (an over-approximation) and the dynamic graph (an under-approximation for the test runs). Because dynamic edges are feasible only for the specific test executions, and the paper never filters or labels edges that are reachable only through test runners, test fixtures, or framework inversion-of-control (Section IV.E.3, JUnit), the union can add edges that are infeasible in the deployed application. This means the composition can increase, rather than eliminate, false positives; the paper provides no argument or measurement to rule out this scenario.
  3. [Section IV.D, Figure 1] The ideal call graph is defined as 'the union of dynamic call graphs across all possible executions of a program.' This definition is ambiguous: if test executions are counted as 'possible,' the ideal graph is not aligned with production reachability, which is what SCA false-positive elimination requires. If test executions are excluded, then the dynamic graph used in the composition is not a subset of the ideal graph, contradicting the intuition in Figure 1 that the union brings the analysis closer to the ideal. The ambiguity needs to be resolved before the theoretical motivation for the union can be accepted.
  4. [Section VI, helios row] The increase from 1 static sink to 3,616 dynamic sinks in helios is presented as a benefit, but without any validation that those 3,616 sinks are reachable in a deployed configuration, the same number could equally represent a false-positive explosion. A sample manual inspection, a precision calculation, or any external ground truth is required to determine whether this result demonstrates improvement.
minor comments (4)
  1. [References] Several references contain typos or malformed names, e.g., 'V on Krogh' [25] and 'Ondej Lhotk' [28].
  2. [Section IV.B, paragraph 3] The sentence 'chains comprising only feasible edges may themselves prove be infeasible' contains a grammatical error ('prove be'); it should read 'prove to be infeasible.'
  3. [Section IV.A, paragraph 2] The claim that 'commercial SCA products report that 70-80% of dependencies are never referenced in application code' lacks a citation; please provide a source for this statistic.
  4. [Section II] The comparison with [26] identifies 'hand-curated, vulnerability-specific sinks' as a key innovation, but the discussion does not clarify how these sinks differ from the code-centric sinks in the closely related work; a brief concrete differentiation would strengthen the novelty claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claim of improved false positive elimination rests on unvalidated metrics, not on a derivation that reduces to its own inputs.

full rationale

The paper does not present a predictive derivation; it describes a call-graph construction pipeline and reports measurements. The only definitional relation is that dynamic call graphs contain only observed edges (Section IV.C), which is a standard property of dynamic analysis and not a circular reduction. The evaluation compares static-sink counts to dynamic-sink counts (Table I); these counts are direct outputs of the pipeline, not fitted parameters later called predictions. The hand-curated sinks and manually identified framework entry points are analyst choices, not quantities fitted to the reported outcome, so no fitted-input-called-prediction pattern applies. No load-bearing uniqueness theorem or prior self-citation is invoked: references [1]-[3] support auxiliary components (remediation and dataset mining), and the closest prior work [26] is an external comparison point, not a self-citation chain. The abstract's wording that the combination 'improve[s] the performance of false positive elimination' is not supported by the sink-count metric, which measures recall rather than false-positive reduction, and the generalizability of test-observed dynamic edges to production executions is an external-validity concern. These are correctness and evaluation-design risks, not circularity under the definitions used here.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no fitted numeric parameters and no new physical or conceptual entities. Its central claim rests on domain assumptions about dynamic graph feasibility, sink accuracy, manual framework identification, and the representativeness of four projects. These assumptions are engineering judgments, not free constants.

assumptions (5)
  • domain assumption Dynamic call graph edges are always feasible, and taking the union of static and dynamic call graphs moves the analysis closer to the ideal call graph.
    Invoked in Sections IV.C and IV.D as the justification for why the union improves false positive elimination. If test executions cover paths not used in production, or miss production paths, the premise fails.
  • domain assumption Hand-curated, vulnerability-specific sinks are accurate method-level root causes of the vulnerabilities being checked.
    Stated in Section IV.E. Any error in sink curation propagates directly into the reachability result.
  • domain assumption Framework entry points can be identified manually and special-cased for common frameworks such as JUnit and TestNG.
    Section IV.E3: 'We identify frameworks manually here as there is no good way to differentiate them without more context.'
  • domain assumption The four Maven Java projects (helios, immutables, java-apns, retrofit) are representative of real-world applications using third-party libraries.
    Section VI provides no selection criteria, diversity argument, or justification of representativeness.
  • domain assumption Reachability of a vulnerable-method sink is an appropriate proxy for the library being used in a vulnerable way.
    Section IV.E acknowledges this is weaker than exploitability, but the evaluation treats increased sink reachability as an unqualified improvement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Dynamics of Software Composition Analysis." pith.science (2026). https://pith.science/paper/C3MMN3YW

@misc{pith2026190900973,
  author       = {Pith},
  title        = {Pith review of: The Dynamics of Software Composition Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C3MMN3YW}},
  note         = {Machine review of arXiv:1909.00973}
}
read the original abstract

Developers today use significant amounts of open source code, surfacing the need for ways to automatically audit and upgrade library dependencies, and giving rise to the subfield of Software Composition Analysis (SCA). SCA products are concerned with three tasks: discovering dependencies, checking the reachability of vulnerable code for false positive elimination, and automated remediation. The latter two tasks rely on call graphs of application and library code to check whether vulnerability-specific sinks identified in libraries are used by applications. However, statically-constructed call graphs introduce both false positives and false negatives on real-world projects. In this paper, we develop a novel, modular means of combining call graphs derived from both static and dynamic analysis to improve the performance of false positive elimination. Our experiments indicate significant performance improvements.

Figures

Figures reproduced from arXiv: 1909.00973 by the authors.

Figure 2
Figure 2. Merging vulnerable method call chains possible edges due to reflective calls, the approach is soundy [19] Thus we have G0 s = (Vs, RT A(CHA(Es))). We define the set of first-party entry points Ve as the set of methods without callers, i.e. {m1 | ∃m1 ∈ V s, ∀(m2, m3) ∈ Es, m1 6= m3}. A and D in the diagram are examples. Methods of Ve must be first-party, as third-party methods must be called by a first￾party method t… view at source ↗
Figure 1
Figure 1. Combined static and dynamic call graphs and dynamic call graphs, the intuition being that it brings us closer to the ideal call graph. In the following sections, we explain how this is done in a modular fashion, preserving the properties of the graph that allow a fast reachability check. E. Vulnerable Methods We consider a vulnerable library to be possibly used in an application if a vulnerability-specific sink is r… view at source ↗
Figure 3
Figure 3. Example of a composed call graph. The boundary lines represent (from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages

  1. [26]

    Beyond metadata: Code-centric and usage-based analysis of known vulnerabili- ties in open-source software

    Ponta, Serena Elisa and Plate, Henrik and Sabetta, Antonino, “Beyond metadata: Code-centric and usage-based analysis of known vulnerabili- ties in open-source software”, 2018

  2. [1]

    Efficient static checking of library updates

    Foo, Darius and Chua, Hendy and Yeo, Jason and Ang, Ming Yi and Sharma, Asankhaya, “Efficient static checking of library updates”, 2018

  3. [2]

    SGL: A domain-specific language for large-scale analysis of open- source code

    Foo, Darius and Ang, Ming Yi and Yeo, Jason and Sharma, Asankhaya, “SGL: A domain-specific language for large-scale analysis of open- source code”, 2018

  4. [3]

    Automated identification of security issues from commit messages and bug reports

    Zhou, Yaqin and Sharma, Asankhaya, “Automated identification of security issues from commit messages and bug reports”, 2017

  5. [4]

    Optimization of object-oriented programs using static class hierarchy analysis

    Dean, Jeffrey and Grove, David and Chambers, Craig, “Optimization of object-oriented programs using static class hierarchy analysis”, 1995

  6. [5]

    Practical virtual method call resolution for Java

    Sundarespan, Vijay and Hendren, Laurie and Razafimahefa, Chrislain and Vall´ee-Rai, Raja and Lam, Patrick and Gagnon, Etienne and Godin, Charles, “Practical virtual method call resolution for Java”, 2000

  7. [6]

    Semantic Versioning versus Breaking Changes: A Study of the Maven Repository

    S. Raemaekers and A. van Deursen and J. Visser, “Semantic Versioning versus Breaking Changes: A Study of the Maven Repository”, 2014

  8. [7]

    Do developers update their library dependencies?

    Kula, Raula Gaikovina and German, Daniel M and Ouni, Ali and Ishio, Takashi and Inoue, Katsuro, “Do developers update their library dependencies?”

Show all 31 references
  1. [8]

    Can automated pull requests encourage software developers to upgrade out-of-date dependencies?

    Mirhosseini, Samim and Parnin, Chris, “Can automated pull requests encourage software developers to upgrade out-of-date dependencies?”, 2017

  2. [9]

    Con- tinuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices

    Shahin, Mojtaba and Babar, Muhammad Ali and Zhu, Liming, “Con- tinuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices”, 2017

  3. [10]

    API-Evolution Support with Diff-CatchUp

    Z. Xing and E. Stroulia, “API-Evolution Support with Diff-CatchUp”, 2007

  4. [11]

    CatchUp!: Capturing and Replay- ing Refactorings to Support API Evolution

    Henkel, Johannes and Diwan, Amer, “CatchUp!: Capturing and Replay- ing Refactorings to Support API Evolution”, 2005

  5. [12]

    Do the dependency conflicts in my project matter?

    Wang, Ying and Wen, Ming and Liu, Zhenwei and Wu, Rongxin and Wang, Rui and Yang, Bo and Yu, Hai and Zhu, Zhiliang and Cheung, Shing-Chi, “Do the dependency conflicts in my project matter?”, 2018

  6. [13]

    Deckard: Scalable and accurate tree-based detection of code clones

    Jiang, Lingxiao and Misherghi, Ghassan and Su, Zhendong and Glondu, Stephane, “Deckard: Scalable and accurate tree-based detection of code clones”, 2007

  7. [14]

    A Practical Approach to the Automatic Classification of Security-Relevant Commits

    Sabetta, Antonino and Bezzi, Michele, “A Practical Approach to the Automatic Classification of Security-Relevant Commits”, 2018

  8. [15]

    A framework for call graph construction algorithms

    Grove, David and Chambers, Craig, “A framework for call graph construction algorithms”, 2001

  9. [16]

    Static and dynamic analysis of call chains in Java

    Rountev, Atanas and Kagan, Scott and Gibas, Michael, “Static and dynamic analysis of call chains in Java”, 2004

  10. [17]

    Modelgen: mining explicit information flow specifications from concrete executions

    Clapp, Lazaro and Anand, Saswat and Aiken, Alex, “Modelgen: mining explicit information flow specifications from concrete executions”, 2015

  11. [18]

    Thou shalt not depend on me: Analysing the use of outdated javascript libraries on the web

    Lauinger, Tobias and Chaabane, Abdelberi and Arshad, Sajjad and Robertson, William and Wilson, Christo and Kirda, Engin, “Thou shalt not depend on me: Analysing the use of outdated javascript libraries on the web”, 2018

  12. [19]

    In defense of soundiness: a manifesto

    Livshits, Benjamin and Sridharan, Manu and Smaragdakis, Yannis and Lhot ´ak, Ond ˇrej and Amaral, J Nelson and Chang, Bor-Yuh Evan and Guyer, Samuel Z and Khedker, Uday P and Møller, Anders and Vardoulakis, Dimitrios, “In defense of soundiness: a manifesto”, 2015

  13. [20]

    More sound static handling of Java reflection

    Smaragdakis, Yannis and Balatsouras, George and Kastrinis, George and Bravenboer, Martin, “More sound static handling of Java reflection”, 2015

  14. [21]

    Blended anal- ysis for performance understanding of framework-based applications

    Dufour, Bruno and Ryder, Barbara G and Sevitsky, Gary, “Blended anal- ysis for performance understanding of framework-based applications”, 2007

  15. [22]

    A scalable technique for characterizing the usage of temporaries in framework- intensive Java applications

    Dufour, Bruno and Ryder, Barbara G and Sevitsky, Gary, “A scalable technique for characterizing the usage of temporaries in framework- intensive Java applications”, 2008

  16. [23]

    OW ASP Top Ten Project

    “OW ASP Top Ten Project”

  17. [24]

    On the Impact of Micro-Packages: An Empirical Study of the npm JavaScript Ecosystem

    Kula, Raula Gaikovina and Ouni, Ali and German, Daniel M and Inoue, Katsuro, “On the Impact of Micro-Packages: An Empirical Study of the npm JavaScript Ecosystem”, 2017

  18. [25]

    Code reuse in open source software

    Haefliger, Stefan and V on Krogh, Georg and Spaeth, Sebastian, “Code reuse in open source software”, 2008

  19. [27]

    Vulnerability Detection in Open Source Software: The Cure and the Cause

    Millar, Stuart. “Vulnerability Detection in Open Source Software: The Cure and the Cause.” (2017)

  20. [28]

    Comparing call graphs

    Lhotk, Ond. “Comparing call graphs.” 2007

  21. [29]

    Infeasible paths in object-oriented programs

    Tip, Frank. “Infeasible paths in object-oriented programs.” (2015)

  22. [30]

    Trin-trin: Whos calling? a pin-based dynamic call graph extraction framework

    Jalan, Rohit, and Arun Kejariwal. “Trin-trin: Whos calling? a pin-based dynamic call graph extraction framework.”

  23. [31]

    Application-only call graph construction

    Ali, Karim, and Ondej Lhotk. “Application-only call graph construction.” 2012

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.