{"id":"9032824d-02a9-404f-b02a-eb6aa16baf21","arxiv_id":"2509.05980","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"GRACE combines a multi-level code graph, hybrid text-structure retrieval, and graph fusion to improve repository-level code completion over vanilla and graph-based RAG baselines.","lead":"GRACE is a system for repository-level code completion that models the whole codebase as a layered graph, retrieves relevant subgraphs using both text and structure, and fuses them into the LLM prompt. The authors report substantial gains on Python and Java completion benchmarks over graph-based RAG baselines, but the draft is missing key method details and contains garbled tables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GNN-based structural encoder and reranker are specified with no training objective, loss, or graph readout; without a defined mechanism, the claimed structural-retrieval gains are unsubstantiated and the central result is not reproducible as written.","rationale":"Reading the paper in good faith, GRACE's design is plausible: multi-level graphs, hybrid retrieval, fusion, and serialization are all reasonable and the authors provide an ablation study, complexity analysis, and a code link. The strongest evidence for the central claim is the large reported margins over RepoFuse/RLCoder on public CrossCodeEval and the modified RepoEval-Updated. However, the mechanism for the central novelty—GNN-based structural similarity—is unspecified at the exact point where it must be load-bearing. The paper never defines graph embeddings from node embeddings, never gives a loss or training signal for the GNN or reranker, and the abstract's GAT reranker is absent from the method. This is not a stylistic quibble: Table 3's ablation attributes concrete numerical gains to the hybrid retriever, and if the structural path is untrained or randomly initialized, the attribution collapses. The reader's CONDITIONAL verdict is appropriate; my check would either locate the missing training procedure in the released code or demonstrate that the structural path is a fixed feature pipeline, in which case the paper needs substantial revision. Secondary concerns (unreleased RepoEval-Updated, garbled Table 2 numbers, missing error bars) are real but subordinate; they affect reproducibility rather than mechanism. I recommend no change to the reader's verdict.","tokens_in":20142,"tokens_out":7251,"duration_ms":75579,"concrete_test":"Open the released code at https://anonymous.4open.science/r/grace_icse-C3D5 and locate the structural encoder, reranker, and GNN modules. Search for any training loop (loss.backward(), optimizer.step(), or stored checkpoints) that updates W_alpha, b_alpha, or GNN weights. If no such training exists, rerun the CrossCodeEval-Python experiment from Table 3 with the structural retrieval path replaced by a fixed Laplacian-PE baseline (no learned GNN) and compare F1 to the reported 82.76. If F1 does not drop materially, the learned structural component is not load-bearing; if it does drop, the paper must specify the missing training details before the result can be credited to graph structure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that GRACE's hybrid graph retriever significantly outperforms SOTA—rests on the structural retrieval path and the graph-aware reranker. Section 2.2.1 defines node embeddings h_v = [v_c; v_s] via node embedding + Laplacian positional encoding (Eq. 2), but Section 2.2.2 then uses 'graph embeddings h_Gq' without ever specifying a graph-level readout/pooling from h_v. The GNN encoders in Eq. 3–4 and the learnable parameters W_alpha, b_alpha in §2.2.3 are never accompanied by a training objective, loss function, optimizer, or any fitting procedure. Algorithm 1's 'GNN + Laplacian' does not resolve this: it is not clear what is trained, on what data, or with what signal. The abstract additionally promises a 'graph attention network-based re-ranker' that never appears in the method section. RQ2 attributes a 3.79-point F1 drop (82.76→78.97, CrossCodeEval-Python) to replacing HGR with BM25, but if the structural path is a fixed feature pipeline with random weights, that drop is attributable to dense semantic embeddings, not to learned structure. The central scientific claim therefore lacks a mechanism as written, and the reported ablation cannot be interpreted without knowing whether the structural components were trained.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GRACE, a repository-level code completion framework that constructs a multi-level, multi-semantic code graph, retrieves relevant context through a hybrid graph retriever combining semantic and structural similarity, reranks candidates, and fuses retrieved subgraphs with the query graph before serializing into an LLM prompt. The central claim is that GRACE significantly outperforms state-of-the-art RAG and graph-RAG baselines on RepoEval-Updated and CrossCodeEval across three backbone LLMs, with headline gains of +8.19% EM and +7.51% ES over RepoFuse on DeepSeek-V3 and top placement in 35/48 metric–language–dataset combinations.","tokens_in":20495,"tokens_out":9571,"duration_ms":96720,"significance":"If the reported results hold, the contribution is potentially valuable: the multi-level graph construction and graph-to-text serialization are clearly motivated, the experimental design includes external baselines and multiple LLMs, the code is released, and the complexity analysis is a useful addition. The evaluation is also not circular: alpha is tuned on validation data and comparisons are against independent methods. However, the central mechanism—the structural encoder and the graph-aware reranker—is under-specified to the point of being unreproducible as written, and the main results tables contain internal inconsistencies. These issues currently prevent the paper from supporting its claims.","major_comments":[{"comment":"The structural retrieval path and reranker are not defined at the level required to support the central claim. Eq. (2) defines node-level h_v, but StructSim in §2.2.2 uses graph embeddings h_Gq/h_Gc without any graph-level readout or pooling. The GNN encoders in Eqs. (3)–(4) appear only in §2.3.2 and are not described (layers, message passing, training objective, loss, optimizer, or supervision data). The parameters W_alpha and b_alpha in §2.2.3 are called learnable but no training procedure is given. The abstract additionally promises a 'graph attention network-based re-ranker' that never appears in §2.2.3 or Algorithm 1. Without these specifications, the structural retrieval mechanism and the RQ2 ablation (82.76→78.97 under BM25) cannot be interpreted: the drop could be due to the semantic embedding difference alone. The released code does not cure an under-specified method description","section":"§2.2.1–2.2.3, Algorithm 1 (Eqs. (2)–(4))"},{"comment":"Table 3's caption states that metrics are macro-averaged over Code and Identifier tasks, but every value equals the Code-only column of Table 2 (e.g., CrossCodeEval-Python F1 82.76, EM 31.38; Java F1 83.38, EM 26.72; RepoEval-Python F1 79.96, EM 29.52; RepoEval-Java F1 79.93, EM 23.85). Macro-averaging the Python F1 with the Identifier F1 of 74.63 would give 78.70, not 82.76. Also, RQ2 states that replacing HGR with BM25 cuts F1 by 4–6% and cites 82.76→78.97, which is a 3.79-point drop. The ablation analysis needs to be re-reported with the correct aggregation and consistent arithmetic.","section":"Table 3 and §3.3 (RQ2)"},{"comment":"Several cells appear duplicated across supposedly different conditions. For example, the RepoEval-Updated Java GRACE row contains the Qwen block '23.2 59.85 80.96 78.31 38.47 62.83 68.69 70.08', which is nearly identical to the Python GRACE row's Qwen block '29.20 57.45 80.96 78.31 38.47 62.83 68.69 70.08', including identical Recall/F1 values across different languages. Such anomalies make it impossible to verify the '35/48' claim or the headline 8.19% EM / 7.51% ES averages. Please supply a clean, machine-readable table and state the aggregation formula used for the headline numbers.","section":"Table 2"},{"comment":"The paper evaluates on a modified benchmark, RepoEval-Updated, but does not release it. The described modification (removing repositories created before 2022-03-31 and adding new Python/Java projects up to 2023-01-01) is not accompanied by a repository list, split definitions, or a version hash. Since this modified dataset is one of only two benchmarks supporting the central comparison, and no results on the original RepoEval are reported, readers cannot reproduce the evaluation or audit the claim that leakage was mitigated. The dataset should be released, or the evaluation should also be run on the original public RepoEval.","section":"§3.1 (RepoEval-Updated)"}],"minor_comments":[{"comment":"Typos and inconsistent naming: 'Vallina RAG' in Table 2, 'Hybird Graph Retriver' in Figure 2, and inconsistent use of 'reranker' vs 're-ranker'.","section":"Throughout"},{"comment":"Both figures appear as garbled unicode/font artifacts in the manuscript text, with no readable axes or curves. They should be regenerated. The text claims 'six Qwen variants' in Figure 3 but the figure is unreadable.","section":"Figures 3 and 4"},{"comment":"The adaptive fusion weight alpha is said to be tuned on a held-out validation split, but no split sizes or validation details are given. The relationship between this tuned alpha and the 'learnable' W_alpha/b_alpha is also unclear.","section":"§3.1"},{"comment":"The word 'significantly' is used repeatedly, but no confidence intervals, standard deviations, or significance tests are reported. Please either add statistical analysis or temper the wording.","section":"§3.2"},{"comment":"The headline gains of +8.19% EM and +7.51% ES are reported only as averages. Please include per-dataset or per-language breakdowns so the reader can verify the claim.","section":"Abstract/§3.2"},{"comment":"The CCS Concepts block still contains the placeholder 'Do Not Use This Code' and must be filled in before publication.","section":"Front matter"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands. As written, the structural encoder and graph-aware reranker have no training procedure or graph-level readout, and the abstract promises a component that never appears in the method. The duplication and caption inconsistencies in Tables 2–3 further undermine the quantitative claims. I do not see evidence of circularity or deliberate misreporting—the comparisons are against external baselines and the code is released—but the paper as currently written is not reproducible. A major revision that fully specifies the structural components, corrects the tables, and releases the modified benchmark could make the paper acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: GRACE is a plausible, well-scoped engineering system for repository-level code completion, and the idea of combining a multi-level code graph with hybrid graph/text retrieval and graph fusion is a real extension over GraphCoder, RepoHyper, and RepoFuse. The paper deserves a serious referee. But the draft under-specifies the exact piece that is supposed to make it work, so I would not accept it without major revision.\n\nWhat's new: the assembly--unifying folder, call, inheritance, AST/CFG/DFG graphs into one heterogeneous graph, retrieving with a hybrid of text and graph similarity, then fusing retrieved subgraphs with the query and serializing the result as triples--is not done in the cited prior works. The complexity analysis is a nice touch, and releasing code is the right instinct.\n\nWhere it falls short:\n\nThe structural encoder and reranker are not defined well enough to reproduce. Section 2.2.1 gives node representations h_v = [v_c; v_s], then Section 2.2.2 uses graph embeddings h_Gq and h_Gc without saying how you pool nodes to a graph vector. The learnable parameters W_alpha, b_alpha in the reranker and the GNN in Algorithm 1 have no training objective, loss, or fitting procedure anywhere in the paper. On top of that, the abstract promises a 'graph attention network-based re-ranker' that never appears in the method--Section 2.2.3 is just a weighted sum with an adaptive scalar. For a method whose headline claim is that structural retrieval helps, this is a load-bearing gap: RQ2 attributes a 3-4 point F1 drop to replacing the hybrid graph retriever with BM25, but if the structural path is a fixed feature pipeline, that drop might be from the dense semantic embeddings, not from learned structure. The authors need to say what is trained, on what data, and with what signal--or show that the untrained pipeline works that way.\n\nTable 2 is also hard to trust as printed. The 'Code/Identifier' columns look misaligned, and spot checks don't match the 'top in 35/48' claim as cleanly as stated. I'd want a clean table and a per-cell breakdown.\n\nNo error bars or significance tests; maybe acceptable with greedy decoding, but worth a line. The modified RepoEval-Updated dataset isn't released, which blocks reproducibility; the anonymous code link may or may not contain everything.\n\nNet: the central idea is sensible and the experimental direction is right, but the paper currently overclaims relative to what is specified. A serious referee would need to verify the structural-components story, and the authors should release the dataset and clean up the tables. If that is fixed, this could be useful for anyone building RAG for code.\n\nRecommendation: send to peer review, with the expectation of major revision. I'd bring it to reading group once it's cleaned up.","headline":"A plausible graph-based RAG system for repository-level code completion, but the current draft doesn't specify its central learned structural components, so the headline gains can't be assessed as written.","tokens_in":20988,"tokens_out":4920,"would_cite":false,"duration_ms":51656,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GRACE's claim: repository-level code completion improves when the whole repository is modeled as a layered code graph, retrieved by structure and text, and fused with the incomplete code before the LLM sees it — beating the strongest graph","keywords":["code completion","repository-level code completion","retrieval-augmented generation","code graph","graph neural network","reranking","graph fusion","large language models"],"falsifier":"Randomize (or freeze) the parameters of the structural encoder and graph-aware reranker while keeping the semantic retrieval path intact, and re-run the CrossCodeEval Python experiments: if EM and F1 do not fall by the ablation-attributed margins, the learned structural components are not what produces the reported gains. A second check: re-run GRACE and baselines on the original, unmodified RepoEval splits; if the margin over baselines shrinks sharply, part of the headline result depends on the unreleased RepoEval-Updated refresh.","tokens_in":20057,"feed_emoji":"📈","tokens_out":15052,"duration_ms":131188,"temperature":0.7,"pith_summary":"GRACE claims that repository-level code completion is limited not mainly by model capability but by what RAG systems feed the model: text-only retrieval that ignores how code relates, and naïve concatenation that flattens those relations away. The fix the paper proposes is to model the whole repository as a layered, multi-relation graph — file structure and cross-file dependencies, call and inheritance graphs, ASTs, control and data flow — and keep that structure visible at every stage: retrieval scores combine textual and topological similarity, a graph-aware reranker reorders candidates, and a fusion stage attaches retrieved subgraphs to the incomplete code by cross-attention before serialization into the prompt. If the paper is right, structural context is a lever for code completion that operates alongside parameter scaling and becomes more valuable as models grow. The evidence is first place in 35 of 48 metric-language-dataset cells across two benchmarks and three backbone LLMs, and, with DeepSeek-V3, average gains of +8.19% exact match and +7.51% edit similarity over the strongest graph-based RAG baseline.","feed_headline":"8.19% EM gain: one graph beats text-only RAG for code completion","feed_subtitle":"GRACE fuses a layered repo graph into LLM context, beating the strongest graph RAG baseline on every dataset.","key_machinery":"The central object is the multi-level, multi-semantic code graph — a heterogeneous graph G = (V, E, T_v, T_e) with repository-level (folders, files), module-level (classes, functions, types), and function-level (AST, control-flow, data-flow) entities unified by cross-level edges such as file→function definition links and function→AST roots. Two mechanisms carry the argument. The hybrid retriever's dual-path encoding represents each node as h_v = [v_c; v_s], concatenating a semantic code embedding with a Laplacian positional encoding from eigenvectors of the normalized Laplacian, making structural similarity searchable. The fusion stage computes cross-attention A = softmax(H_q H_rᵀ / √d) betw","core_discovery":"The paper's discovery, on its own terms: a repository should be treated as a graph database, not a text archive, and every pipeline stage should preserve that graph. GRACE builds a unified graph spanning repository, module, and function levels — folder structure, cross-file dependencies, call and inheritance graphs, ASTs, control/data flow — joined by cross-level edges. Retrieval runs semantic and structural paths in parallel, merges and reranks candidates, then fuses the top-k subgraphs with the query graph by cross-attention. The empirical claim: with DeepSeek-V3, GRACE beats the strongest graph-based RAG baseline by 8.19% EM and 7.51% ES and leads 35 of 48 metric-language-dataset cells.","pith_inferences":["Because the paper gives no training procedure for the structural encoder or reranker, a replication that trains them explicitly, or replaces them with untrained versions, would isolate whether the reported structural gains come from learned matching or from the graph representation itself.","The gain widening with backbone scale suggests a testable substitution: whether GRACE at 7B parameters matches vanilla RAG at 32B, which would make structure a cheaper alternative to scale in constrained deployments.","The graph construction is language-agnostic in principle (AST, call, inheritance, and data-flow edges exist in most languages), so porting GRACE to C, Go, or Rust is a direct test of whether the mechanism is structural rather than benchmark-specific.","Since the fused graph is serialized before the LLM sees it, the serialization format is itself a control variable: re-ordering or re-labeling the same edges should measurably change completion accuracy, quantifying how much of the gain is presentation versus content."],"forward_implications":["Replacing the hybrid retriever with BM25 costs 4–6 F1 points, so structural retrieval carries signal that text similarity alone misses.","Removing the fusion stage is the single largest drop (up to –6.2 F1 and –5.4 EM), so wiring retrieved subgraphs into the query graph matters more than retrieval quality alone.","The gain widens with backbone size: from Qwen2.5-Coder-14B to DeepSeek-V3 the margin over the best non-graph competitor grows from 2.3 to 5.4 EM points on Python/CrossCodeEval, and scaling Qwen from 0.6B to 32B raises F1 by 56.3%.","Retrieval depth saturates at k = 3; adding more retrieved subgraphs beyond that mostly adds redundant edges rather than new cues.","AST-only graphs land 2–3% behind the full system, showing call, data-flow, and control-flow edges each contribute complementary structure."],"supporting_citations":[{"why":"Defines repository-level code completion with iterative retrieval-generation RAG and supplies the original RepoEval benchmark that GRACE's updated dataset refreshes.","marker":"[53]"},{"why":"RepoFuse is the strongest graph-based RAG baseline; its fused dual-context retrieval is the direct target of the headline 8.19% EM / 7.51% ES comparisons.","marker":"[24]"},{"why":"GraphCoder contributes the code-context-graph retrieval baseline whose coarse-to-fine retrieval GRACE's hybrid graph retriever extends and outperforms.","marker":"[30]"},{"why":"CrossCodeEval provides the cross-file benchmark (Python, Java, TypeScript, C) that yields half the experimental evaluation.","marker":"[13]"},{"why":"RLCoder is the reinforcement-learning repository-level baseline against which GRACE reports +7.94% EM / +6.83% ES with DeepSeek-V3.","marker":"[48]"},{"why":"CodeT5+ supplies the codet5p-110m-embedding model used for semantic retrieval encoding and node feature extraction.","marker":"[47]"},{"why":"DeepSeek-V3 is the backbone LLM for the headline results, supporting the claim that larger models exploit graph-structured cues better.","marker":"[26]"}],"fun_headline_variants":["GRACE: graph-guided RAG lifts code completion by 8.19% EM","Treat code as a graph: GRACE beats text RAG by 8.19% EM","Graph-first code completion: GRACE gains 8.19% EM over RAG","GRACE fuses repo graphs to outscore text RAG by 8.19% EM","Repo code completion: graph fusion tops RAG by 8.19% EM"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The structural-retrieval and reranking gains presuppose that the graph neural components actually perform learned structural matching, but the paper never specifies how they are trained — no loss, no fitting procedure, no graph-level readout — so if those parameters are untrained or arbitrary, the structure-based improvement has no demonstrated mechanism.","fun_headline_variants_meta":{"raw":{"variants":["GRACE: graph-guided RAG lifts code completion by 8.19% EM","Treat code as a graph: GRACE beats text RAG by 8.19% EM","Graph-first code completion: GRACE gains 8.19% EM over RAG","GRACE fuses repo graphs to outscore text RAG by 8.19% EM","Repo code completion: graph fusion tops RAG by 8.19% EM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001052,"raw_usage":{"total_tokens":4296,"prompt_tokens":830,"completion_tokens":3466,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":3362}},"tokens_in":574,"tokens_out":3466,"duration_ms":27305,"temperature":1.0,"reasoning_tokens":3362,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:41:41.256762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomize (or freeze) the parameters of the structural encoder and graph-aware reranker while keeping the semantic retrieval path intact, and re-run the CrossCodeEval Python experiments: if EM and F1 do not fall by the ablation-attributed margins, the learned structural components are not what produces the reported gains. A second check: re-run GRACE and baselines on the original, unmodified RepoEval splits; if the margin over baselines shrinks sharply, part of the headline result depends on the unreleased RepoEval-Updated refresh.","supporting_citations":[],"review_version":1}