REVIEW 4 major objections 4 minor 31 references
The Dynamics of Software Composition Analysis
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that Software Composition Analysis finds more genuinely reachable vulnerable methods by merging static call graphs with call graphs recorded from instrumented test runs, because the union approximates the ideal call graph…
desk verdict A useful industrial SCA architecture paper whose headline claim about false-positive elimination is not supported by the evaluation, which measures sink counts rather than precision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the combined call graph $G_c = G'_s \cup G'_d$. $G'_s$ is built from static analysis by adding Class Hierarchy Analysis edges, Rapid Type Analysis refinements, and a 'soundy' reflection pass, then restricting the graph to methods reachable from first-party entry points. $G'_d$ is built by instrumenting test runs and keeping only the transitive closure of framework entry points, so all its edges are observed and feasible and its entry points remain first-party. Library-side vulnerable method call chains $CC$ — paths from a library entry point to a hand-curated sink method — are precomputed once per library and merged into $G'_s$ only as suffixes of existing edges, which preserves first-party reachability and makes sink reachability a set-membership check. The union $G_c$ is what lets downstream checks see paths that span static and dynamic edges, including flows that appear only under framework inversion of control.
What would settle it
Take one reported case, helios, whose reachable vulnerable-method sinks grow from 1 to 3,616 after the merge, and build a ground-truth list of which of those 3,616 sinks can actually be reached from an application entry point on any real execution path. If most of the added sinks are not truly reachable in representative executions, the central claim that the union improves false-positive elimination is refuted.
Extended reading notes
Core claim
Stated as the authors would put it: statically constructed call graphs are soundy approximations, not exact models, and dynamically observed call graphs are always feasible but incomplete; therefore taking the union of the two moves the analysis toward the ideal call graph, which is the union of all dynamic graphs over every possible execution. The paper's concrete demonstration is that on four real Maven-based Java projects, the dynamic call graph contributes 824% more vertices and 361% more edges on average, with most of the extra edges coming from third-party dependencies. The reachability payoff appears in the sink counts: helios goes from 1 to 3,616 reachable vulnerable-method sinks, and java-apns from 58 to 859, while the other two projects show no change. The practical enabler is that vulnerable method call chains are precomputed per library and merged modularly into the application graph, so the analysis can be repeated in CI pipelines without rebuilding library call graphs on demand.
Load-bearing premise
The method's central bet is that running the project's test suite gives a picture of real executions that can be merged with static edges to approximate how the software actually runs; if the tests do not resemble production usage, the extra reachable sinks it reports are not real false-positive eliminations.
Editorial extensions
If this is right
- SCA reachability checks will report vulnerabilities inside framework-driven and reflection-heavy flows that static-only scans cannot see, because the dynamic half supplies feasible edges for those flows.
- Because library call chains are precomputed and only merged into the application graph, the combined reachability analysis can run in minutes, making it practical inside CI/CD pipelines rather than a one-off deep audit.
- Automated remediation can use the dynamic edges to avoid flagging library upgrades as breaking when the only path to a changed method was an infeasible static call chain.
- For projects with little or no test coverage, the dynamic half contributes little, so the benefit of the union is conditional on the project already having a runnable, reasonably covering test suite.
Reading between the lines
- Labeling each newly discovered sink with exploitability ground truth in a follow-up study would separate true precision gains from added coverage, and would quantify how much of the union's benefit is genuine.
- Comparing test-derived dynamic graphs with dynamic graphs traced from production traffic would directly test the central assumption: the union should help most when test coverage resembles production usage.
- If helios-scale increases generalize, static-only SCA rankings systematically understate exposure in framework-heavy applications, so prioritization of dependency upgrades would need to weight dynamic reachability evidence.
- The hand-curated sink method itself is a scaling bottleneck; mining sink methods from CVE patch diffs or commit-level fixes would let the same union machinery cover new vulnerabilities without manual curation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes the architecture of a commercial Software Composition Analysis (SCA) product covering dependency discovery, call-graph-based checking of whether vulnerability-specific sinks are reachable, and automated remediation. Its central claim, stated in the abstract, is that combining call graphs derived from static and dynamic analysis improves the performance of false positive elimination. The evaluation in Section VI examines four Java projects and reports that dynamic call graphs add on average 824% more vertices and 361% more edges, and that the number of reachable vulnerable-method sinks rises sharply in two of the four projects (e.g., helios from 1 to 3,616).
Significance. If the central claim were substantiated, the result would be practically valuable: SCA tools could discover many more genuinely reachable vulnerable methods and thereby reduce false positives. The paper has concrete strengths: a modular composition of precomputed library call chains, hand-curated vulnerability-specific sinks, a clearly stated architecture, and a microbenchmark-based evaluation of dynamic dependency discovery covering 18 package managers. However, the main evaluation does not measure the claimed outcome, false positive elimination, and the theoretical justification for why the union should approach the ideal call graph is not tied to production-relevant executions. The significance of the paper is therefore limited by a mismatch between the claim and the evidence.
major comments (4)
- [Section VI, Table I] The evaluation reports only vertex counts, edge counts, and sink counts; it does not measure false positive elimination, which is the central claim of the paper. The abstract promises improvements in 'the performance of false positive elimination,' but no precision, false-positive-rate, or ground-truth comparison is presented anywhere in Section VI. The reported numbers are surrogate metrics that mostly reflect recall, so the primary claim is unsupported by the evidence.
- [Section IV.C and IV.D] Dynamic call graphs are constructed from test executions, and Section IV.C explicitly notes that only flows observed during tests are reported. The composition step in Section IV.D takes the union of the static graph (an over-approximation) and the dynamic graph (an under-approximation for the test runs). Because dynamic edges are feasible only for the specific test executions, and the paper never filters or labels edges that are reachable only through test runners, test fixtures, or framework inversion-of-control (Section IV.E.3, JUnit), the union can add edges that are infeasible in the deployed application. This means the composition can increase, rather than eliminate, false positives; the paper provides no argument or measurement to rule out this scenario.
- [Section IV.D, Figure 1] The ideal call graph is defined as 'the union of dynamic call graphs across all possible executions of a program.' This definition is ambiguous: if test executions are counted as 'possible,' the ideal graph is not aligned with production reachability, which is what SCA false-positive elimination requires. If test executions are excluded, then the dynamic graph used in the composition is not a subset of the ideal graph, contradicting the intuition in Figure 1 that the union brings the analysis closer to the ideal. The ambiguity needs to be resolved before the theoretical motivation for the union can be accepted.
- [Section VI, helios row] The increase from 1 static sink to 3,616 dynamic sinks in helios is presented as a benefit, but without any validation that those 3,616 sinks are reachable in a deployed configuration, the same number could equally represent a false-positive explosion. A sample manual inspection, a precision calculation, or any external ground truth is required to determine whether this result demonstrates improvement.
minor comments (4)
- [References] Several references contain typos or malformed names, e.g., 'V on Krogh' [25] and 'Ondej Lhotk' [28].
- [Section IV.B, paragraph 3] The sentence 'chains comprising only feasible edges may themselves prove be infeasible' contains a grammatical error ('prove be'); it should read 'prove to be infeasible.'
- [Section IV.A, paragraph 2] The claim that 'commercial SCA products report that 70-80% of dependencies are never referenced in application code' lacks a citation; please provide a source for this statistic.
- [Section II] The comparison with [26] identifies 'hand-curated, vulnerability-specific sinks' as a key innovation, but the discussion does not clarify how these sinks differ from the code-centric sinks in the closely related work; a brief concrete differentiation would strengthen the novelty claim.
Circularity Check
No significant circularity: the claim of improved false positive elimination rests on unvalidated metrics, not on a derivation that reduces to its own inputs.
full rationale
The paper does not present a predictive derivation; it describes a call-graph construction pipeline and reports measurements. The only definitional relation is that dynamic call graphs contain only observed edges (Section IV.C), which is a standard property of dynamic analysis and not a circular reduction. The evaluation compares static-sink counts to dynamic-sink counts (Table I); these counts are direct outputs of the pipeline, not fitted parameters later called predictions. The hand-curated sinks and manually identified framework entry points are analyst choices, not quantities fitted to the reported outcome, so no fitted-input-called-prediction pattern applies. No load-bearing uniqueness theorem or prior self-citation is invoked: references [1]-[3] support auxiliary components (remediation and dataset mining), and the closest prior work [26] is an external comparison point, not a self-citation chain. The abstract's wording that the combination 'improve[s] the performance of false positive elimination' is not supported by the sink-count metric, which measures recall rather than false-positive reduction, and the generalizability of test-observed dynamic edges to production executions is an external-validity concern. These are correctness and evaluation-design risks, not circularity under the definitions used here.
Assumptions & free parameters
assumptions (5)
- domain assumption Dynamic call graph edges are always feasible, and taking the union of static and dynamic call graphs moves the analysis closer to the ideal call graph.
- domain assumption Hand-curated, vulnerability-specific sinks are accurate method-level root causes of the vulnerabilities being checked.
- domain assumption Framework entry points can be identified manually and special-cased for common frameworks such as JUnit and TestNG.
- domain assumption The four Maven Java projects (helios, immutables, java-apns, retrofit) are representative of real-world applications using third-party libraries.
- domain assumption Reachability of a vulnerable-method sink is an appropriate proxy for the library being used in a vulnerable way.
Cite this review
Pith. "Pith review of The Dynamics of Software Composition Analysis." pith.science (2026). https://pith.science/paper/C3MMN3YW
@misc{pith2026190900973,
author = {Pith},
title = {Pith review of: The Dynamics of Software Composition Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/C3MMN3YW}},
note = {Machine review of arXiv:1909.00973}
}
read the original abstract
Developers today use significant amounts of open source code, surfacing the need for ways to automatically audit and upgrade library dependencies, and giving rise to the subfield of Software Composition Analysis (SCA). SCA products are concerned with three tasks: discovering dependencies, checking the reachability of vulnerable code for false positive elimination, and automated remediation. The latter two tasks rely on call graphs of application and library code to check whether vulnerability-specific sinks identified in libraries are used by applications. However, statically-constructed call graphs introduce both false positives and false negatives on real-world projects. In this paper, we develop a novel, modular means of combining call graphs derived from both static and dynamic analysis to improve the performance of false positive elimination. Our experiments indicate significant performance improvements.
Figures
Reference graph
Works this paper leans on
-
[26]
Ponta, Serena Elisa and Plate, Henrik and Sabetta, Antonino, “Beyond metadata: Code-centric and usage-based analysis of known vulnerabili- ties in open-source software”, 2018
work page 2018
-
[1]
Efficient static checking of library updates
Foo, Darius and Chua, Hendy and Yeo, Jason and Ang, Ming Yi and Sharma, Asankhaya, “Efficient static checking of library updates”, 2018
work page 2018
-
[2]
SGL: A domain-specific language for large-scale analysis of open- source code
Foo, Darius and Ang, Ming Yi and Yeo, Jason and Sharma, Asankhaya, “SGL: A domain-specific language for large-scale analysis of open- source code”, 2018
work page 2018
-
[3]
Automated identification of security issues from commit messages and bug reports
Zhou, Yaqin and Sharma, Asankhaya, “Automated identification of security issues from commit messages and bug reports”, 2017
work page 2017
-
[4]
Optimization of object-oriented programs using static class hierarchy analysis
Dean, Jeffrey and Grove, David and Chambers, Craig, “Optimization of object-oriented programs using static class hierarchy analysis”, 1995
work page 1995
-
[5]
Practical virtual method call resolution for Java
Sundarespan, Vijay and Hendren, Laurie and Razafimahefa, Chrislain and Vall´ee-Rai, Raja and Lam, Patrick and Gagnon, Etienne and Godin, Charles, “Practical virtual method call resolution for Java”, 2000
work page 2000
-
[6]
Semantic Versioning versus Breaking Changes: A Study of the Maven Repository
S. Raemaekers and A. van Deursen and J. Visser, “Semantic Versioning versus Breaking Changes: A Study of the Maven Repository”, 2014
work page 2014
-
[7]
Do developers update their library dependencies?
Kula, Raula Gaikovina and German, Daniel M and Ouni, Ali and Ishio, Takashi and Inoue, Katsuro, “Do developers update their library dependencies?”
Show all 31 references
-
[8]
Can automated pull requests encourage software developers to upgrade out-of-date dependencies?
Mirhosseini, Samim and Parnin, Chris, “Can automated pull requests encourage software developers to upgrade out-of-date dependencies?”, 2017
2017
-
[9]
Con- tinuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices
Shahin, Mojtaba and Babar, Muhammad Ali and Zhu, Liming, “Con- tinuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices”, 2017
2017
-
[10]
API-Evolution Support with Diff-CatchUp
Z. Xing and E. Stroulia, “API-Evolution Support with Diff-CatchUp”, 2007
2007
-
[11]
CatchUp!: Capturing and Replay- ing Refactorings to Support API Evolution
Henkel, Johannes and Diwan, Amer, “CatchUp!: Capturing and Replay- ing Refactorings to Support API Evolution”, 2005
2005
-
[12]
Do the dependency conflicts in my project matter?
Wang, Ying and Wen, Ming and Liu, Zhenwei and Wu, Rongxin and Wang, Rui and Yang, Bo and Yu, Hai and Zhu, Zhiliang and Cheung, Shing-Chi, “Do the dependency conflicts in my project matter?”, 2018
2018
-
[13]
Deckard: Scalable and accurate tree-based detection of code clones
Jiang, Lingxiao and Misherghi, Ghassan and Su, Zhendong and Glondu, Stephane, “Deckard: Scalable and accurate tree-based detection of code clones”, 2007
2007
-
[14]
A Practical Approach to the Automatic Classification of Security-Relevant Commits
Sabetta, Antonino and Bezzi, Michele, “A Practical Approach to the Automatic Classification of Security-Relevant Commits”, 2018
2018
-
[15]
A framework for call graph construction algorithms
Grove, David and Chambers, Craig, “A framework for call graph construction algorithms”, 2001
2001
-
[16]
Static and dynamic analysis of call chains in Java
Rountev, Atanas and Kagan, Scott and Gibas, Michael, “Static and dynamic analysis of call chains in Java”, 2004
2004
-
[17]
Modelgen: mining explicit information flow specifications from concrete executions
Clapp, Lazaro and Anand, Saswat and Aiken, Alex, “Modelgen: mining explicit information flow specifications from concrete executions”, 2015
2015
-
[18]
Thou shalt not depend on me: Analysing the use of outdated javascript libraries on the web
Lauinger, Tobias and Chaabane, Abdelberi and Arshad, Sajjad and Robertson, William and Wilson, Christo and Kirda, Engin, “Thou shalt not depend on me: Analysing the use of outdated javascript libraries on the web”, 2018
2018
-
[19]
In defense of soundiness: a manifesto
Livshits, Benjamin and Sridharan, Manu and Smaragdakis, Yannis and Lhot ´ak, Ond ˇrej and Amaral, J Nelson and Chang, Bor-Yuh Evan and Guyer, Samuel Z and Khedker, Uday P and Møller, Anders and Vardoulakis, Dimitrios, “In defense of soundiness: a manifesto”, 2015
2015
-
[20]
More sound static handling of Java reflection
Smaragdakis, Yannis and Balatsouras, George and Kastrinis, George and Bravenboer, Martin, “More sound static handling of Java reflection”, 2015
2015
-
[21]
Blended anal- ysis for performance understanding of framework-based applications
Dufour, Bruno and Ryder, Barbara G and Sevitsky, Gary, “Blended anal- ysis for performance understanding of framework-based applications”, 2007
2007
-
[22]
A scalable technique for characterizing the usage of temporaries in framework- intensive Java applications
Dufour, Bruno and Ryder, Barbara G and Sevitsky, Gary, “A scalable technique for characterizing the usage of temporaries in framework- intensive Java applications”, 2008
2008
-
[23]
OW ASP Top Ten Project
“OW ASP Top Ten Project”
-
[24]
On the Impact of Micro-Packages: An Empirical Study of the npm JavaScript Ecosystem
Kula, Raula Gaikovina and Ouni, Ali and German, Daniel M and Inoue, Katsuro, “On the Impact of Micro-Packages: An Empirical Study of the npm JavaScript Ecosystem”, 2017
2017
-
[25]
Code reuse in open source software
Haefliger, Stefan and V on Krogh, Georg and Spaeth, Sebastian, “Code reuse in open source software”, 2008
2008
-
[27]
Vulnerability Detection in Open Source Software: The Cure and the Cause
Millar, Stuart. “Vulnerability Detection in Open Source Software: The Cure and the Cause.” (2017)
2017
-
[28]
Comparing call graphs
Lhotk, Ond. “Comparing call graphs.” 2007
2007
-
[29]
Infeasible paths in object-oriented programs
Tip, Frank. “Infeasible paths in object-oriented programs.” (2015)
2015
-
[30]
Trin-trin: Whos calling? a pin-based dynamic call graph extraction framework
Jalan, Rohit, and Arun Kejariwal. “Trin-trin: Whos calling? a pin-based dynamic call graph extraction framework.”
-
[31]
Application-only call graph construction
Ali, Karim, and Ondej Lhotk. “Application-only call graph construction.” 2012
2012
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.