{"id":"1d2336d3-010e-4cfc-b30a-91f1e6c8d2e7","arxiv_id":"2506.18191","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GRAPHIA uses graph neural network link prediction on whole-program graphs to rank candidate callees for unresolved JavaScript call sites, placing the correct one in the top 5 for 72% of cases.","lead":"This paper trains a graph neural network to rank which function a JavaScript call site points to, aiming to recover call relations that static analysis misses. For 42% of unresolved call sites it puts the correct function first, which could cut manual effort in code analysis.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 42%/72% headline is measured on call edges CodeQL already resolved (RQ1), not on unresolved call sites; the unresolved-case claim therefore rests on RQ2's unquantified dynamic ground truth.","rationale":"I read the paper in good faith: the approach is plausible, the dataset is substantial, the replication package is a real asset, and the ablations give some support to the code representation and node features. My concern is not that the architecture is wrong or the results are fabricated; it is that the experiment producing the headline number measures the wrong quantity. RQ1 evaluates how well GRAPHIA reproduces CodeQL's existing call edges, which is a useful approximate-static-analysis result but not a demonstration of resolving statically-unresolved call sites. The abstract explicitly claims the 42% figure applies to statically-unresolved call sites, but Section IV-C's test set contains only edges CodeQL already found. The only genuinely unresolved evaluation is RQ2, which depends on dynamic edges collected from unit tests. The reader's weakest assumption already flagged the dynamic ground truth as unquantified and potentially biased; I agree, but I would broaden the concern: even the static main experiment is not an unresolved-call-site benchmark. This is a correctness-of-claim issue rather than a disagreement with community consensus, and it can be settled by re-running the evaluation on a test set of truly unresolved call sites. Until that is done, the paper should not be read as showing that GRAPHIA resolves missed call edges with the stated accuracy. I keep the conditional verdict because a focused revision could supply the missing measurement.","tokens_in":17213,"tokens_out":7758,"duration_ms":83453,"concrete_test":"Using the released replication package, enumerate every RQ1 test call site and determine whether it has at least one incident edge in the CodeQL-derived call graph; report the fraction of RQ1 test call sites with no CodeQL edge. If that fraction is zero, the 42%/72% figures cannot be cited as evidence about unresolved call sites. The decisive follow-up is then to recompute rank-0 and top-5 accuracy only on call sites whose true edge appears in the dynamic ground truth but not in the static edge set, reporting per-package test coverage alongside the ranking.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section IV-C states that the RQ1 test set 'exclusively comprises call edges derived from the output of the static analysis tool,' and its summary reports GRAPHIA 'identifying statically-known edges at rank 0 in 42% of cases.' The abstract and introduction, however, present the same 42%/72% numbers as applying to 'statically-unresolved call sites' and 'unresolved cases.' These are different objects: a call edge that CodeQL produced is not a false negative, so ranking it top-1 among all function definitions demonstrates retrieval fidelity to CodeQL, not recovery of missed call relations. The only experiment aimed at genuinely unresolved sites is RQ2/Section IV-D, which uses dynamic edges from unit tests; that experiment covers ten libraries, reports no coverage metric, and the authors concede that unit tests 'might not execute the entire program functionality.' Thus the central claim rests on a metric mismatch plus an unquantified, potentially biased partial ground truth. Even if the GNN ranking mechanism works perfectly, the headline numbers do not establish the stated capability on unresolved call sites.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"GRAPHIA is a GNN-based link prediction approach that builds a whole-program graph from a pruned AST augmented with semantic identifier nodes, and trains on static call edges from CodeQL, optionally combined with dynamic edges from instrumented unit tests. For each call site, GRAPHIA ranks all candidate function definitions in the project; the paper claims that it places the correct callee first in over 42% and within the top five in 72% of statically unresolved call sites, thereby augmenting JavaScript call graph construction. The evaluation covers 50 npm libraries, four research questions, two ablations, a transfer-learning study, and is accompanied by a replication package.","tokens_in":17479,"tokens_out":7251,"duration_ms":69447,"significance":"If the central claim were properly supported, this would be a useful new ML-based augmentation path for JavaScript call graph analysis, and the paper's large-scale dataset, ranking-based evaluation, ablations, and publicly released replication package are genuine strengths. The focus on ranking rather than ROC is well motivated, and the idea of learning from imperfect static and dynamic labels is worth pursuing. However, the current evidence does not establish the headline capability: the main quantitative claims are computed on static edges that the baseline tool already resolved, and the only experiment on truly unresolved edges lacks a baseline and coverage control. The paper is therefore promising but requires substantial revision before its central claim can be accepted.","major_comments":[{"comment":"Section IV-C states that the RQ1 test set 'exclusively comprises call edges derived from the output of the static analysis tool,' meaning the test edges are exactly the edges CodeQL resolved; ranking them among all candidate function definitions measures retrieval fidelity to CodeQL, not recovery of missed call edges. The abstract's claim that 'for more than 42% of the statically-unresolved call sites, GRAPHIA correctly predicts the right target function' is therefore not supported by RQ1, and the introduction uses the same numbers for 'unresolved cases.' The only experiment targeting genuinely unresolved call sites is RQ2, and its limitations are discussed separately.","section":"IV-C and Abstract/Introduction"},{"comment":"RQ2 selects 'ten large npm libraries with substantial test coverage' but reports no coverage metric, and the authors concede that unit tests 'might not execute the entire program functionality.' The experiment evaluates only ten libraries without a baseline such as name-based or lexical matching, and without statistical significance tests. As a result, the dynamic ground truth may be biased toward easy, test-covered patterns, so the reported rank accuracy (e.g., 68% for express) does not establish the rate at which GRAPHIA would resolve real-world unresolved call sites.","section":"IV-D"},{"comment":"In Section III-D, the dataset is split into 80/10/10 subsets at the level of individual call edges, and the GNN is trained and tested on the same whole-program graph. Under this transductive, edge-level split, test edges share call-site nodes with training edges, so the model has already observed each test call site's local and global context during training. This can inflate the ranking metrics in RQ1; the paper should split by call sites or by files, or report the results under such a split.","section":"III-D / IV-C"},{"comment":"The paper frames the contribution as improving the recall of JavaScript call graph construction, but it never measures the end-to-end effect on a call graph, e.g., how many true edges are added and how many false positives are introduced when the top-k candidates are accepted for unresolved call sites. The ranking metric alone, even where valid, does not quantify recall improvement; an experiment that augments CodeQL's output with GRAPHIA's top-k predictions and compares precision/recall against a non-learning baseline would make the practical claim testable.","section":"I and IV-F"}],"minor_comments":[{"comment":"Replace 'statically-unresolved call sites' with 'held-out static edges' in the sentence reporting the 42%/72% results, because those results come from RQ1.","section":"Abstract"},{"comment":"The formal definition 'Ri = Rank(P i)' does not define a position in a ranked list; please formalize the ranking computation, including the treatment of ties and the candidate space for each call site.","section":"IV-E"},{"comment":"Clarify the learning-rate schedule and whether the batch size of 32,768 counts training edges, graph nodes, or call-site–candidate pairs.","section":"III-D"},{"comment":"RQ5 is a case study on the express package only; the text should state this explicitly rather than generalizing to GRAPHIA's overall versatility.","section":"IV-G"},{"comment":"Add a legend or otherwise identify which curve belongs to each package; without it, the claimed variance across libraries cannot be inspected.","section":"Figure 6"},{"comment":"Specify the exact weighting formula for the aggregate statistics (e.g., weight by number of call edges) so the reader can reproduce the weighted averages.","section":"IV-C"}],"recommendation":"major_revision","confidential_remarks":"The metric mismatch between the abstract's 'unresolved call sites' and RQ1's held-out static edges is serious but repairable. The paper has a strong replication package and a large dataset; with a proper baseline in RQ2, a call-site-level split in RQ1, and an end-to-end precision/recall measurement, the central claim could be substantiated. I recommend major revision rather than rejection. The related-work section may also under-cite recent ML-based call-graph construction papers, which could affect the novelty framing; the authors should verify that their 'first to apply GNN link prediction to full multi-file program graphs' statement is accurate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the 42%/72% figures in the abstract are measured on held-out CodeQL edges, not on unresolved call sites. Section IV-C says the test set 'exclusively comprises call edges derived from the output of the static analysis tool.' The advertised central claim—resolving statically-unresolved call sites—therefore rests on RQ2, a much smaller experiment over ten libraries with unquantified unit-test coverage. The authors concede the tests 'might not execute the entire program functionality.' That is a real mismatch, and it needs fixing before the headline numbers are cited.\n\nWhat is genuinely new: this is the first work I know to apply GNN link prediction to whole multi-file program graphs rather than to local embeddings or edge-pruning. The semantic identifier nodes that densify the AST are a sensible representation choice, and the ablations show both structure and features matter. The dataset is substantial (50 libraries, 163K call edges), and the replication package with graphs and scripts is real evidence. The citation pattern is fine; the related work includes the pruning and embedding baselines you'd expect.\n\nSoft spots, in rough order of importance. RQ1 is a fidelity-to-CodeQL check, not evidence of resolving true misses; the abstract and intro overstate it. RQ2 lacks a baseline—name matching or simple scoping would tell us how much of the ranking is just lexical similarity. The random edge split in a transductive setting can leak structural information; a temporal or cross-project split would be more convincing. And there are no significance tests; per-library variance is high (qs near zero, express at 68%), so the weighted averages hide a lot. These are fixable in revision, not fatal.\n\nThis paper is for static-analysis researchers and ML-for-code folks who care about whole-program representations. The core idea is sound and the work deserves serious peer review. I'd conditionally accept with required revisions: align the abstract with what RQ1 actually measures, add a name-matching baseline for RQ2, report test coverage for the ten libraries, and give per-library confidence intervals. Do not take the 42%/72% as proof of recovering true misses until RQ2 is tightened.","headline":"A useful new application of GNN link prediction to whole-program graphs, but the headline numbers are about reproducing CodeQL; the evidence for finding genuinely missed edges is thinner and needs stronger evaluation.","tokens_in":17996,"tokens_out":3267,"would_cite":true,"duration_ms":30654,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A GNN can rank the correct callee for unresolved JavaScript call sites, reaching top-5 accuracy in 72% of cases.","keywords":["call graph construction","JavaScript static analysis","graph neural networks","link prediction","program graphs","dynamic call edges","call graph recall","semantic edges"],"falsifier":"Manually resolve a random sample of statically unresolved call sites in the ten tested libraries by full runtime tracing or code inspection, then rerun GRAPHIA's ranking against that ground truth; if top-5 accuracy on that sample falls well below 72%, the novel-edge claim is an artifact of test coverage.","tokens_in":17038,"feed_emoji":"🕸️","tokens_out":5635,"duration_ms":53790,"temperature":0.7,"pith_summary":"Static call graph tools for JavaScript leave many call sites unresolved because dynamic property access, higher-order functions, and cross-file definitions defeat purely static analysis. This paper claims that a graph neural network can take over those unresolved call sites: if the whole program is represented as a graph, link prediction can rank the correct callee among every function definition in the project. GRAPHIA ranks the right target first in over 42% of unresolved cases and within the top five in 72%, so an analyst could check a short list instead of searching the entire codebase. It also shows the model can learn from static edges alone or from a mix of static and dynamic edges, including dynamic edges gathered from other projects. Reducing false negatives in call graph construction matters because missing edges hide security-relevant flows from downstream interprocedural analyses.","feed_headline":"GNN recovers missing JavaScript call edges: top-5 in 72%","feed_subtitle":"Whole-program graphs rank candidate targets for call sites static tools miss, making unresolved calls easier for analysts to check.","key_machinery":"The load-bearing object is the augmented program graph: a pruned abstract syntax tree whose nodes are connected by structural edges and by semantic edges that link every occurrence of the same identifier to a dedicated semantic node. These semantic edges shorten the path between a call site and a far-away callee, letting a graph neural network with gated message passing propagate information across file boundaries and long-distance relations. Node features are the AST node type, identifier name, and argument or parameter counts. The same graph carries the training signal as call-edge labels, so link prediction on this representation is what turns code structure into candidate rankings.","core_discovery":"The paper's central claim is that JavaScript call graph completion can be framed as link prediction over the entire program, and that a graph neural network can handle a meaningful share of that task. GRAPHIA builds a whole-program graph from a pruned abstract syntax tree, adds semantic nodes connected to every use of the same identifier, and labels call edges with static edges from existing analysis tools plus dynamic edges from instrumented unit tests. At inference time, each statically unresolved call site is scored against all candidate function definitions, and the output is a ranked list rather than a binary edge decision. On 50 npm libraries with 163K call edges, the correct callee is the top-ranked candidate in more than 42% of unresolved cases and appears in the top five in 72%. The authors take these numbers as evidence that learning-based link prediction can complement existing static call graph tools by reducing their false negatives.","pith_inferences":["A natural testable extension is to plug the top-k candidate edges into a downstream taint or alias analysis and measure how many new security findings appear; the current paper stops at edge rankings, so the end-to-end payoff is not yet measured.","The same whole-program link prediction recipe could transfer to Python, Ruby, or PHP, where dynamic dispatch and higher-order functions create similar unresolved-call problems, though the semantic-edge design would need to adapt to each language's scoping rules.","If the ranking signal relies heavily on identifier names, minified or renamed production code may degrade accuracy; an explicit test would compare GRAPHIA's ranks on a library before and after minification.","Semantic edges act as a learned substitute for pointer analysis: instead of computing points-to sets, the model short-circuits name and structure relations, so the approach's ceiling likely depends on how often those shortcuts are sufficient."],"forward_implications":["Static call graph tools can be augmented without retraining from scratch: GRAPHIA learns from the tool's own static edges and then proposes candidates for the call sites the tool leaves open.","Analysts would inspect a ranked shortlist of possible callees for each unresolved call site instead of the whole project, with top-5 accuracy of 72% bounding the manual effort.","Dynamic edges gathered from unit tests of one set of libraries transfer to others: transfer experiments improved rank-0 accuracy for mathjs, formula-parser, express, and js-yaml by 18% to 92%.","The proposed code representation carries most of the performance: removing the syntactic and semantic structure drops rank-0 accuracy by 61% for formula-parser and by large margins for several other libraries.","The paper's own discussion points toward a human-in-the-loop pipeline in which analysts vet the top-ranked candidates, especially because graph neural network outputs are hard to explain."],"supporting_citations":[{"why":"Provides the baseline false-positive pruning approach and the feature-selection criteria GRAPHIA adapts for node features.","marker":"[6]"},{"why":"Prior neural call graph pruner using structural and transformer features; GRAPHIA differs by generating missing edges rather than pruning false ones.","marker":"[10]"},{"why":"Comparative study reporting WALA's recall and false-edge rate, motivating the need to recover missed edges.","marker":"[5]"},{"why":"Shows static JavaScript call graph tools trade completeness for precision, explaining the large share of unresolved call sites.","marker":"[11]"},{"why":"The static analysis tool used to extract the statically labeled call edges that serve as training and test ground truth.","marker":"[24]"},{"why":"Babel-based instrumentation supplies the dynamic call edges extracted from unit test executions, used both as novel-edge ground truth and as a training signal.","marker":"[25]"},{"why":"Introduces GNN link prediction with training and test edge removal, the methodological template GRAPHIA extends to whole-program graphs.","marker":"[23]"},{"why":"Provides the gated graph neural network architecture used for message passing on the program graphs.","marker":"[28]"},{"why":"Analyzes root causes of missing JavaScript call graph edges, motivating the difficult language constructs addressed here.","marker":"[12]"}],"fun_headline_variants":["GNN recovers missing JS call edges: top-5 in 72%","Neural link prediction: 72% of missed JS calls in top 5","GRAPHIA: GNN ranks unresolved JS calls, top-1 42%","JavaScript call graphs improved: GNN finds 72% of missed calls","First GNN for full program graphs boosts JS call discovery"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the edges recovered by running each library's unit tests are a fair and representative sample of the edges that static analysis misses; if the tests cover only easy call patterns, the measured novel-edge rankings overstate how often GRAPHIA would resolve real-world unresolved call sites.","fun_headline_variants_meta":{"raw":{"variants":["GNN recovers missing JS call edges: top-5 in 72%","Neural link prediction: 72% of missed JS calls in top 5","GRAPHIA: GNN ranks unresolved JS calls, top-1 42%","JavaScript call graphs improved: GNN finds 72% of missed calls","First GNN for full program graphs boosts JS call discovery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":3132,"prompt_tokens":1046,"completion_tokens":2086,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":1987}},"tokens_in":662,"tokens_out":2086,"duration_ms":12963,"temperature":1.0,"reasoning_tokens":1987,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:53:24.248606+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually resolve a random sample of statically unresolved call sites in the ten tested libraries by full runtime tracing or code inspection, then rerun GRAPHIA's ranking against that ground truth; if top-5 accuracy on that sample falls well below 72%, the novel-edge claim is an artifact of test coverage.","supporting_citations":[{"cited_title":"Striking a balance: pruning false-positives from static call graphs,","cited_arxiv_id":null,"evidence_quote":"Provides the baseline false-positive pruning approach and the feature-selection criteria GRAPHIA adapts for node features."},{"cited_title":"Autopruner: transformer-based call graph pruning,","cited_arxiv_id":null,"evidence_quote":"Prior neural call graph pruner using structural and transformer features; GRAPHIA differs by generating missing edges rather than pruning false ones."},{"cited_title":"Is javascript call graph extraction solved yet? a comparative study of static and dynamic tools,","cited_arxiv_id":null,"evidence_quote":"Comparative study reporting WALA's recall and false-edge rate, motivating the need to recover missed edges."},{"cited_title":"Static JavaScript call graphs: A comparative study,","cited_arxiv_id":null,"evidence_quote":"Shows static JavaScript call graph tools trade completeness for precision, explaining the large share of unresolved call sites."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The static analysis tool used to extract the statically labeled call edges that serve as training and test ground truth."},{"cited_title":"Link prediction based on graph neural networks,","cited_arxiv_id":null,"evidence_quote":"Introduces GNN link prediction with training and test edge removal, the methodological template GRAPHIA extends to whole-program graphs."},{"cited_title":"Automatic Root Cause Quantification for Missing Edges in JavaScript Call Graphs (Extended Version)","cited_arxiv_id":"2205.06780","evidence_quote":"Analyzes root causes of missing JavaScript call graph edges, motivating the difficult language constructs addressed here."}],"review_version":2}