{"id":"0c3a3fe7-91e5-428e-9c94-f45cba822f3e","arxiv_id":"2506.06247","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A language-agnostic data-flow engine for Joern that over-approximates external library calls and filters spurious flows at query time, enabling incremental annotation without full re-analysis.","lead":"The authors built and released a new taint-tracking engine inside the open-source Joern platform that analyzes Java, Python, and JavaScript code without needing the source of external libraries. The system over-approximates data flows through unknown library calls, then filters out false paths at query time so users can add library annotations without rebuilding the graph.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'safely overapproximate' claim in Section 3.1 is contradicted by Section 4's admission that aliasing and the heap are not tracked: true flows through aliases may have no path in the DDG, so added summaries cannot discover them, yielding false negatives.","rationale":"The paper's central architectural idea is a stable data-dependence graph in which unknown callees are over-approximated and user-supplied semantics filter at query time. That idea is plausible, implemented in a publicly available tool, and the incremental-semantics evaluation is a genuine strength. However, the guarantee that 'adding such summaries is not required to discover additional flows' depends on every real data flow being representable as a path in the DDG before any summary is applied. Section 4 explicitly concedes that aliasing, heap-allocated data structures, and global state are not modeled, with only weak updates for assignments. A true taint flow that passes through an alias or a heap object mutated through another reference is therefore absent from the DDG, so no query-time semantics can recover it. This is a false negative, not a false positive, and it is exactly the kind of unsoundness that the word 'safely overapproximate' in Section 3.1 rules out. The reader's weakest assumption identified this same aliasing/heap gap, and I agree that it is the most load-bearing issue. The paper discloses the limitation honestly, which is why the verdict should remain conditional rather than reject: the authors should either temper the over-approximation claim, or better, add an artifact test demonstrating whether alias-induced flows are found or missed. The proposed concrete test would settle whether the concern lands by checking a minimal alias case against the released engine.","tokens_in":11120,"tokens_out":5472,"duration_ms":55724,"concrete_test":"Construct a minimal program in one supported language, e.g., Java: class A { B b; } A a1 = new A(); A a2 = a1; a1.b = source(); sink(a2.b); (or the equivalent in Python/JavaScript). Run the released Joern engine with default operator semantics and no hand-written library summaries. If the engine reports no flow from source to sink, the Section 3.1 'safely overapproximate' claim fails for alias-induced flows. As a follow-up, add a summary that makes the field write taint the receiver object; if the flow is still missed, this confirms that added summaries cannot recover flows absent from the underlying DDG.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.1 states that treating all callees as external with unknown semantics and over-approximating data dependencies at each call site 'safely overapproximate[s] the data flow', and that adding summaries is not required to discover additional flows. This is load-bearing for the central claim: query-time semantics can only filter paths that already exist in the stable DDG. Section 4, however, admits that 'aliasing and the heap of data structures are not tracked' and that assignments propagate flow 'only via weak updates'. Under weak updates, a write through one alias (e.g., a.x = source) need not create a data-dependence edge to a later read through another alias (e.g., sink(b.x)) even when a and b refer to the same object. The true flow then has no explicit DDG path, so no amount of added summaries can discover it; the analysis returns a false negative. For such programs, the engine is not a may analysis over real executions, and 'safely overapproximate' is false. This is not merely imprecision (extra false positives) but unsoundness (missing true flows), and it directly undercuts the claim that missing library annotations are the only source of over-approximation. The evaluation benchmarks (Securibench Micro, Thorat, securibench-micro.js) are small and may not exercise heap-alias flows, so the empirical results do not provide evidence against this gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a design and implementation of a taint-analysis engine for the Joern code-analysis platform. The engine builds a whole-program data-dependence graph (DDG) and, for calls to procedures whose code is unavailable or unannotated, over-approximates data flow by assuming every input may taint every output. Users can provide semantic summaries for such external calls at query time; these summaries filter invalid paths without requiring the DDG to be recomputed. The evaluation compares Joern (with and without user semantics) against Semgrep and CodeQL on Java (Securibench Micro), Python (Thorat), and JavaScript (securibench-micro.js) benchmarks, and measures scalability on Defects4j and BugsInPy. The paper reports that Joern finds the most true positives at the cost of additional false positives, and contributes two new benchmark datasets and an open-source implementation.","tokens_in":11367,"tokens_out":11370,"duration_ms":109164,"significance":"The incremental-annotation property is potentially valuable: if the DDG is a sound over-approximation of all real data flows, then users can add library semantics and immediately get fewer false positives without re-analysis, which is attractive for CI pipelines. The paper also contributes reproducible benchmarks and integrates with a widely used open-source platform. However, the evaluation does not fully establish the claimed over-approximation; Section 4 concedes that aliasing and heap-allocated data structures are not tracked (weak updates only), so true flows can be absent from the DDG. The k-limit in Algorithm 1 is another source of possible false negatives. These gaps are not merely theoretical, and the benchmark suite may not exercise them, so the comparative results should be read with caution.","major_comments":[{"comment":"The paper's core design claim is that treating all callees as external with unknown semantics and over-approximating data dependencies at each call site 'safely overapproximate[s] the data flow' (Section 3.1), so that 'adding such summaries is not required to discover additional flows but helps eliminate false positives.' This is only correct if every real data flow has a corresponding path in the DDG. Section 4 states that 'aliasing and the heap of data structures are not tracked' and that assignments propagate flow 'only via weak updates.' Under weak updates, a write through one alias need not create a dependence edge to a later read through another alias, so a true taint path can be absent from the graph and no amount of added summaries can recover it. The engine is therefore not a may analysis over real executions for such flows; it is incomplete. The limitation is described as 'imprecision,' but it is actually a source of false negatives. Please either restrict the over-approximation claim to flows representable in the modeled graph, or extend the analysis to handle heap and aliasing soundly, or provide experimental evidence that the evaluated benchmarks (and the intended real-world workload) do not contain such flows.","section":"3.1, 4"},{"comment":"The maximum call depth k_max terminates task generation at line 3-4 of Algorithm 1. When the depth limit is reached, no new tasks are created from a partial result; the text says 'flows will be over-approximated for dependent callers of this result,' but in fact sources that are more than k_max call edges away from the sink are never reached. For example, with k_max=1, a source in main() cannot be matched to a sink in a method called by main through an intermediate call, because the task from the parameter of the intermediate method to the call site in main is not created. This is an additional source of false negatives and contradicts the description of the analysis as a 'may analysis' (Section 3) and the 'safely overapproximate' phrasing. The paper should state explicitly that k_max introduces incompleteness, and should quantify the effect of k on recall/precision (e.g., in Figure 2) as part of the accuracy discussion.","section":"3.2, Algorithm 1"},{"comment":"The maximum call depth k=5 is chosen by scanning k over exactly the same three benchmarks (Securibench Micro, Thorat, securibench-micro.js) on which the final results in Table 1 are reported, with no held-out data or cross-validation. Any monotone relationship between k and the J/F1 metrics makes the reported numbers optimistically biased. The paper should either report the full k-sweep curves for all benchmarks and select k on a separate tuning set, or present results for a range of k values and discuss the sensitivity of the comparative claims to this choice.","section":"5.3, Table 1"},{"comment":"The Joern_SEM configuration uses 'manually specified semantics' but the paper does not describe which library calls were annotated, how many annotations were written, or whether they were authored before or after examining the expected outcomes of the benchmarks. If these semantics were tuned to the test set, the precision improvement (e.g., Securibench Micro false positives decreasing from 36 to 17) is not evidence of a general benefit. The paper should document the semantics, their provenance, and ideally validate on an independent set of programs.","section":"5.4, 5.5"}],"minor_comments":[{"comment":"The phrase 'the possible data dependency between result at the call to bar and its occurrence at the call to Sink.addValue is indicated by a path' is unclear; consider rewriting for readability.","section":"Section 2"},{"comment":"The labels 'DDG' appear on multiple edges without distinguishing edge types; a legend or distinct arrow styles would improve clarity.","section":"Figure 1"},{"comment":"'All experiments [3] were performed on a platform' attaches citation [3] (a dataset DOI) to a hardware description; rephrase to avoid implying the hardware is part of the dataset. Also state the versions and taint-mode configurations of Semgrep and CodeQL.","section":"Section 5.1"},{"comment":"The result table R* is described as a set, but line 6 uses R*[s0] as a map; clarify whether results are keyed by start node and define what 'prepend known path' means precisely.","section":"Algorithm 2"},{"comment":"For the new Thorat and securibench-micro.js datasets, state how ground truth was established and whether the labels were independently validated; this is important since the datasets are a contribution of the paper.","section":"Section 5.2"},{"comment":"Runtime and memory values for Semgrep are shown without any spread; state whether these are single runs or include variance as for the other tools.","section":"Table 1"},{"comment":"The conclusion says semantics can be added 'without having to re-analyze the dependencies,' but the evaluation does not directly demonstrate the incremental workflow; a small experiment timing a query with and without a new semantic would strengthen this claim.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the conference's scope, and the open-source release plus the two new benchmarks are concrete contributions. The main risk is that the evaluation is used to support a stronger soundness claim than the implementation warrants. The self-contradiction between 'safely overapproximate' (Section 3.1) and the admitted aliasing/heap gaps (Section 4) needs to be resolved, and the k-selection methodology should be improved. A careful revision that narrows the claims or extends the analysis is needed before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about arXiv:2506.06247 is that the core design is genuinely neat: treat every callee as external with unknown semantics, over-approximate data dependencies at each call site, and then apply library semantics as edge filters during query-time task solving. That means you can add or refine semantics without rebuilding the data-dependence graph. They implement it in Joern, ship it open-source, and also release two benchmark sets (a completed Thorat for Python and a new securibench-micro.js). That is a real, reproducible contribution.\n\nThe evaluation is honest and useful. They compare against Semgrep and CodeQL on three taint benchmarks, report TP/FP/FN, F1 and Youden's J, and include runtime and memory. Joern finds more true positives, at the cost of extra false positives. That is a reasonable trade-off and the numbers support it.\n\nThe soft spots are in the soundness framing. Section 3.1 says the approach 'safely overapproximate[s] the data flow', but Section 4 admits aliasing and the heap are not tracked, and assignments use weak updates. The stress-test is right: a real taint path that goes through an alias (a.x = source; later sink(b.x) where a and b alias) may have no explicit DDG edge, so no added semantics can recover it. That is a false negative, not just imprecision. So 'safely overapproximate' should be scoped to flows represented in the DDG, not real executions. The limitation section already says this, but the main text's phrasing overclaims.\n\nTwo evaluation details also deserve attention. The max call depth k=5 is selected on the same benchmarks used for the headline numbers (Section 5.3), and the Joern_SEM semantics are manually written for the library calls appearing in exactly those benchmarks. That makes the improvement of Joern_SEM over plain Joern partly a curve-fit. A separate tuning set or a blinded semantics-writing protocol would firm up the claim. These are not fatal, but they should be addressed in revision.\n\nWho should read this: anyone building or evaluating language-agnostic taint engines, and the Joern community. It deserves a serious referee; the design is non-obvious and the system is public and testable. I'd suggest the reviewers push on the soundness language and the parameter/semantics fitting, but I'd be surprised if the central idea does not survive.\n\nRecommendation: send to peer review, with a request for revisions on the over-approximation claim and the evaluation methodology.","headline":"A genuinely useful stable-DDG design with query-time semantics, but the 'safely overapproximate' claim is too strong given the admitted aliasing and heap gaps.","tokens_in":11952,"tokens_out":2629,"would_cite":true,"duration_ms":21680,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A taint-tracking engine can detect vulnerabilities without library source code by over-approximating unknown calls, and its users sharpen results later by adding semantics without re-analysis.","keywords":["static analysis","taint analysis","data dependence graph","code property graph","library semantics","partial program analysis","vulnerability detection","language-agnostic analysis"],"falsifier":"Run the engine on a small program where taint is written into a field of a heap object through one reference and later read through a different alias, with the tainted value reaching a sink at runtime; if the engine reports no flow, the over-approximation claim is false for flows the representation does not encode.","tokens_in":10864,"feed_emoji":"🔍","tokens_out":9648,"duration_ms":75833,"temperature":0.7,"pith_summary":"Taint analysis normally needs to know what external libraries do with their inputs. This paper builds a whole-program explicit data-dependence graph that treats every unknown callee as potentially passing every input to every output, over-approximating flows so that no library semantics are required to find candidate vulnerabilities. Because semantic annotations are applied only when answering a query, adding them later removes false positives without rebuilding or re-running the analysis. The evaluation on Java, Python, and JavaScript benchmarks indicates that this over-approximate partial-program analysis finds more vulnerabilities than two widely used multi-language tools, at the price of more false positives.","feed_headline":"Over-approximating unknown calls still finds the most flaws","feed_subtitle":"Treat every external library call as opaque, then drop false paths as semantics arrive—no re-analysis needed.","key_machinery":"The central object is the explicit data-dependence graph (DDG), a graph whose edges go from nodes that define a variable to nodes that use it without an intervening redefinition. The engine constructs this graph on top of a code property graph, treating all callees as external and giving each call site over-approximated argument-to-argument data flows; a small set of special indices lets users write annotations such as receiver-to-receiver and receiver-to-return flows to state which flows survive. At query time, a demand-driven backward or forward traversal solves per-method tasks on worker threads, checks each parent edge against the supplied semantics, caches results, and stops at a maximum call depth to guarantee termination. The stable graph plus query-time semantics is what lets annotations be added without recalculation.","core_discovery":"On the paper's own terms, the central discovery is that a whole-program data-dependence graph can be kept stable while knowledge of library semantics grows. Every callee is treated as external with unknown semantics, so each call site over-approximates data dependencies by assuming any input may taint any output. The resulting graph contains invalid paths, but those paths are filtered at query time using user-supplied flow semantics for specific methods, so the graph itself never has to be recomputed when semantics are added. The paper states that adding such summaries is not required to discover additional flows but helps eliminate false positives. The evaluated engine is reported to identify the most vulnerabilities among the compared tools, with additional false positives, and to scale well enough for partial-program analysis of modern programs.","pith_inferences":["A natural consequence is that the tool's recall is fixed by what the graph encodes: any real flow that depends on aliasing or heap shape can only appear if the representation gains those edges, since summaries alone cannot invent new paths.","The incremental-semantics property suggests a practical workflow in which heuristic or learned summaries are generated programmatically and loaded on the fly until query results stop changing, without re-running the full analysis.","The evaluation's comparison suggests that partial-program analysis may be the right deployment mode for continuous pipelines, but the missing alias and heap sensitivity means results should be treated as candidate flows for triage rather than proof of absence.","A natural experiment would measure how much semantic annotation is needed on a large real-world codebase to match the precision of whole-program analysis, quantifying the precision gap this paper reports."],"forward_implications":["With no user-supplied semantics at all, the engine still reports flows from sources to sinks, so partial-program taint analysis can run before library dependencies are available or understood.","Adding or refining library semantics reduces false positives on a later query without re-analyzing the program, which suits continuous integration pipelines with tight analysis budgets.","The maximum call depth is a widening step: paths deeper than the limit are over-approximated, trading precision for guaranteed termination.","Whole-program analysis costs considerably more runtime for only a small precision gain, so the partial-program setting is the intended operating point.","Because the same graph construction and query machinery works across Java, Python, and JavaScript, the approach is language-agnostic in principle and demonstrated on those three languages."],"supporting_citations":[{"why":"Supplies system dependence graphs and summary edges for transitive dependencies, the representation this design avoids baking into the graph.","marker":"[13]"},{"why":"Defines the program dependence graph that the data-dependence representation is based on.","marker":"[8]"},{"why":"Introduces code property graphs, the underlying graph model of the platform this engine extends.","marker":"[29]"},{"why":"Supplies the demand-driven, task-based interprocedural query strategy adapted to the work-queue traversal.","marker":"[7]"},{"why":"Presents the IFDS exploded-supergraph formulation whose hard-coded semantics this approach contrasts with stable-graph over-approximation.","marker":"[23]"},{"why":"Provides the Java taint benchmark on which precision and recall are measured.","marker":"[20]"},{"why":"Provides the JavaScript taint benchmark contributed with this paper.","marker":"[1]"},{"why":"Provides the Python taint benchmark completed and adopted for evaluation.","marker":"[2]"},{"why":"Supplies real-world Java programs used to test scalability.","marker":"[16]"},{"why":"Supplies real-world Python programs used to test scalability.","marker":"[28]"}],"fun_headline_variants":["Over-approximate library calls, refine taint at query time","Stable taint graph accepts semantics later without recompute","Language-agnostic taint analysis scales past unknown callees","Defer call summaries to keep taint graphs fast"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach only catches taint that travels through explicit variable definitions and uses; if taint moves through aliased objects or shared mutable state, the graph has no edge for that path and the tool will miss the flow, a limitation the paper states openly.","fun_headline_variants_meta":{"raw":{"variants":["Over-approximate library calls, refine taint at query time","Stable taint graph accepts semantics later without recompute","Language-agnostic taint analysis scales past unknown callees","Defer call summaries to keep taint graphs fast"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000543,"raw_usage":{"total_tokens":2537,"prompt_tokens":817,"completion_tokens":1720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":1651}},"tokens_in":433,"tokens_out":1720,"duration_ms":13621,"temperature":1.0,"reasoning_tokens":1651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:57:57.623646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the engine on a small program where taint is written into a field of a heap object through one reference and later read through a different alias, with the tainted value reaching a sink at runtime; if the engine reports no flow, the over-approximation claim is false for flows the representation does not encode.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the demand-driven, task-based interprocedural query strategy adapted to the work-queue traversal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Java taint benchmark on which precision and recall are measured."},{"cited_title":"2025.securibench-micro.js Dataset","cited_arxiv_id":null,"evidence_quote":"Provides the JavaScript taint benchmark contributed with this paper."},{"cited_title":"2025.Thorat Dataset","cited_arxiv_id":null,"evidence_quote":"Provides the Python taint benchmark completed and adopted for evaluation."}],"review_version":1}