{"id":"e26cf9af-507b-4e33-9531-2d9f79c85c7b","arxiv_id":"2505.16901","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A graph-integrated open-source LLM with agentless RAG resolves 43% of SWE-bench Lite issues, best among open-weight models.","lead":"This paper presents Code Graph Models, which feed repository code graphs into an open-source LLM through a specialized adapter and graph-aware attention, plus an agentless retrieval pipeline. On SWE-bench Lite the system resolves 43% of issues, the highest reported for open-weight models as of May 2025.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 43% headline rests on an evaluation definition (Appendix C.5.3) that differs from the official SWE-bench test-pass criterion; if literally applied, the number is not comparable to the leaderboard it is ranked against.","rationale":"The reader's weakest assumption concerns the quality of the code graph construction. That is a reasonable secondary concern, but it does not threaten the headline number itself: even if the graph edges are imperfect, the 43% could still be a valid empirical result for the whole system. The more load-bearing issue is whether the 43% number was produced by the official SWE-bench evaluation at all. Appendix C.5.3 explicitly substitutes a gold-patch-superset condition for the official test-based condition. That is exactly the kind of missing support the reviewing rules require flagging. If the literal definition was used, the result is not a SWE-bench Lite resolution rate and Table 1's ranking is invalid. If it was a paraphrase, the paper still has not documented use of the official harness, and the released code should be checked. This is not an accusation of misconduct; it is a precise ambiguity in the paper's own evaluation statement. The concrete test settles it: rerunning the official harness on the released checkpoint either validates 43% or refutes it. The reader's graph-quality worry can be tested separately (e.g., an SWE-bench ablation with the graph mask removed), but the metric question is more fundamental because it determines whether the central number is benchmark-comparable at all. No other concern, such as missing error bars or benchmark-specific fine-tuning, is as directly load-bearing: error bars affect confidence intervals but not the validity of the protocol, and training on the SWE-bench train split is a disclosed and standard practice. Therefore the verdict should remain conditional, with the condition being verification of the official evaluation metric.","tokens_in":27622,"tokens_out":6857,"duration_ms":57726,"concrete_test":"Run the released CGM-SWE-PY checkpoint (CodeFuse-CGM-72B) through the official SWE-bench Lite evaluation harness on all 300 instances, using the standard FAIL_TO_PASS/PASS_TO_PASS test execution on generated patches. Then compare the official pass count to the reported 43.00% (129/300). If the official count differs by more than a couple of instances, the headline claim is not established under the benchmark's own protocol; if it matches, the Appendix C.5.3 sentence is a documentation error and the headline stands.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is the 43.00% resolution rate on SWE-bench Lite. That number is only meaningful if the official SWE-bench evaluation protocol is used, but Appendix C.5.3 defines a resolved patch as one that 'correctly addresses the issue and is a superset of the ground-truth edits.' This is not the SWE-bench criterion: the official harness applies the generated patch and checks that all FAIL_TO_PASS tests pass and all PASS_TO_PASS tests still pass. The gold patch is not part of the evaluation condition. If the authors literally used a gold-patch-superset filter, their 43% is computed under a different metric and is not comparable to the other entries in Table 1, which are official SWE-bench scores. Even if the sentence is only a loose paraphrase, the paper does not state anywhere that the official SWE-bench harness was used, and the wording creates a direct ambiguity about the headline result. This is a load-bearing, checkable condition: it sits upstream of every ranking claim, the 'first among open-weight models' statement, and the 12.33% margin over Moatless+DeepSeek-V3.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Code Graph Models (CGMs), which inject repository-level code graphs into decoder-only LLMs through two mechanisms: semantic integration (a CodeT5+ encoder and an MLP adapter map node text into 'node tokens') and structural integration (a graph-aware attention mask over node tokens). Training uses subgraph reconstruction pre-training followed by noisy fine-tuning on issue-patch pairs. The authors also build an agentless Graph RAG framework with Rewriter, Retriever, Reranker, and Reader, and report a 43.00% resolution rate on SWE-bench Lite with Qwen2.5-72B, claiming first place among open-weight models, plus results on SWE-bench Verified, SWE-bench-java Verified, and code completion benchmarks (CrossCodeEval, ComplexCodeEval). The paper includes ablations of RAG modules, semantic/structural integration, and pretraining, as well as cost and backbone generalization experiments.","tokens_in":27881,"tokens_out":7299,"duration_ms":58421,"significance":"If the 43.00% SWE-bench Lite result is validated under the official SWE-bench harness, this is a notable result: it would show that an agentless, open-weight system can compete with closed-source agent-based systems, and the graph-aware attention design is a plausible architectural contribution for repository-level code understanding. The paper is transparent in several ways: code and model weights are released, the RAG modules are described with prompts in appendices, and ablations isolate the graph mask, reconstruction pretraining, and each RAG component. The significance is conditional, however, because the central number and the ranking claims depend on an evaluation-protocol clarification and on statistical robustness that the current text does not provide.","major_comments":[{"comment":"The definition of Resolve Rate in Appendix C.5.3 ('a patch is considered resolved if it correctly addresses the issue and is a superset of the ground-truth edits') is not the official SWE-bench evaluation criterion. The official harness applies the model-generated patch and requires the FAIL_TO_PASS tests to pass and the PASS_TO_PASS tests to remain passing; the gold patch is not part of the evaluation condition. If the C.5.3 sentence is literal, the 43.00% headline in the abstract and Table 1 is computed under a different metric and is not comparable with the official leaderboard scores listed in the same table, so the ranking claims (first among open-weight models, +12.33% over Moatless+DeepSeek-V3) are not supported. The paper never states explicitly that the official SWE-bench harness was used; the statements in Section 5.1 and C.5.2 about 'developer-written unit tests' and 'Docker executable environment' do not establish this. Please clarify whether the official harness was used, report the exact evaluation command and version, or rerun the evaluation under the official protocol.","section":"Appendix C.5.3 (and Section 5.1, Table 1)"},{"comment":"All reported resolution rates in Table 1 are single point estimates from a stochastic pipeline, since Appendix C.3 sets temperature=0.7, top_p=0.8, and repetition_penalty=1.1. The central comparative claims, such as the 12.33% margin over Moatless+DeepSeek-V3 and the 2.33% margin over Agentless+Claude-3.5-Sonnet, are presented without variance, repeated runs, or confidence intervals. Given that the differences are small relative to typical run-to-run variation in LLM sampling, please report at least 3-5 independent runs with means and standard deviations (or bootstrap intervals) for the main benchmark results, and state whether the same seeds were used across compared systems.","section":"Table 1 and Appendix C.3"},{"comment":"The ComplexCodeEval results in Table 3 are based on only 100 randomly selected samples per language, but the paper does not report the sampling seed, the selection procedure, or any confidence intervals. With n=100, the reported improvements (for example, CGM-Multi Python EM 43.00 vs. Qwen2.5-72B 28.00) are not statistically grounded as presented. Please provide the exact selection protocol, run-to-run variance, and ideally evaluate on the full benchmark or a substantially larger stratified sample before claiming that CGM 'greatly outperforms' baselines on ComplexCodeEval.","section":"Appendix C.6.2 and Table 3"},{"comment":"The comparison in Table 3 is confounded by fine-tuning: CGM-Multi-72B has been pre-trained and fine-tuned on code completion data, while the baseline models (Mistral-Large-123B, DeepSeek-V2.5-236B, Qwen2.5-72B) are used off-the-shelf with FIM prompting and one-hop expansion. The reported gains on CrossCodeEval and ComplexCodeEval may therefore reflect the additional fine-tuning rather than the graph integration. A cleaner baseline, such as Qwen2.5-72B fine-tuned on the same data with flattened graph text but no graph-aware mask, should be reported for the 72B-scale completion results; the 7B-scale ablation in Table 11 partially addresses this, but Table 3 does not.","section":"Table 3 and Appendix C.6.4"}],"minor_comments":[{"comment":"There are typos in this section: 'esults in further enhancements' should be 'results in further enhancements', and 'GGM-Freeze' should be 'CGM-Freeze'.","section":"Appendix C.7.2"},{"comment":"The table title contains a typo: 'funcitons' should be 'functions'.","section":"Table 13"},{"comment":"The abbreviation 'R3' is used in the table before being defined; please define it explicitly as Rewriter, Retriever, and Reranker.","section":"Table 10"},{"comment":"The sentence in Section 5.1 that 'All benchmarks utilize developer-written unit tests to verify the correctness of model-generated patches' is difficult to reconcile with the ground-truth-superset definition in C.5.3; these two statements should be aligned.","section":"Section 5.1 and Appendix C.5.3"},{"comment":"In Table 1(b), the SWE-Agent row using Qwen2.5-Coder-32B is marked 'No' in the Agent column even though SWE-Agent is an agent-based method; please check the annotation and correct it.","section":"Table 1(b)"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the evaluation-protocol ambiguity in Appendix C.5.3. If the authors cannot confirm that the 43.00% SWE-bench Lite number was computed with the official SWE-bench harness, the paper's central claim and all rankings in Table 1 should not be published in their current form. The other concerns (variance, ComplexCodeEval sampling, baseline confound in Table 3) are fixable but need to be addressed before the paper can be considered acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a serious systems paper with a genuinely new integration, but the headline number has an evaluation-protocol ambiguity that sits upstream of every ranking claim. Read Appendix C.5.3 before trusting Table 1.\n\nThe core idea is worth taking seriously. Compressing 512-token chunks into single node tokens and gating attention between those tokens with the code graph's adjacency matrix is a clean way to give a decoder-only LLM repository structure without blowing up the context. The agentless Graph RAG is a natural extension, and the paper does the right empirical work: three issue-fixing benchmarks, two completion benchmarks, ablations that isolate the graph mask, the reranker, and the reconstruction pretraining, plus released code and model weights. The ablations in Table 11 give real evidence that the graph mask and reconstruction pretraining carry weight. This is not a toy.\n\nNow the soft spots, in order of severity. The metric definition in Appendix C.5.3 says a patch is resolved if it \"correctly addresses the issue and is a superset of the ground-truth edits.\" That is not the SWE-bench criterion. The official harness applies the patch and checks FAIL_TO_PASS and PASS_TO_PASS tests; the gold patch is not part of the evaluation condition. If the authors literally used a gold-patch-superset filter, the 43.00% is computed under a different metric and cannot be ranked against the other entries in Table 1. If the sentence is just a loose paraphrase, the paper never says the official SWE-bench harness was used, which is a strange omission for a claim that hinges on comparability. This has to be resolved before the headline means anything.\n\nEverything else is smaller. Results are single-run with no variance, which matters for a 2-3% edge over the nearest competitors. ComplexCodeEval uses a 100-sample subsample per language, so those numbers are roughly 10% error bars. The reranker is doing a lot of work—removing it drops performance by 24.67%—and the reranker is an LLM prompted with file skeletons, so some of the credit belongs to retrieval, not the graph architecture. The graph's over-approximation of dynamic calls is a known tradeoff, but the ablation evidence suggests the mask still helps even with noisy edges.\n\nWho should read this: anyone working on repository-level code intelligence, especially people building open-weight alternatives to agent pipelines. The paper deserves a serious referee; the architecture is novel enough and the evaluation broad enough to merit the time. But the referee needs to pin the authors down on the exact evaluation protocol for the headline result. My recommendation: send it to peer review with a strong request to clarify or re-run the SWE-bench evaluation under the official harness, and to report variance across runs.","headline":"Real architectural contribution and broad evaluation, but the 43% headline rests on a non-standard 'resolved' definition that must be clarified before the leaderboard claim is credible.","tokens_in":28442,"tokens_out":1128,"would_cite":true,"duration_ms":9854,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a 43.00% resolution rate on SWE-bench Lite can be achieved with an open-weight, agentless system by integrating repository dependency structure directly into the LLM's attention, beating the prior best open-weight…","keywords":["code graph model","graph-aware attention","repository-level software engineering","SWE-bench Lite","agentless issue fixing","retrieval-augmented generation","open-source LLM","code completion"],"falsifier":"Replace the code graph in the pipeline with a random graph matched in node count and degree distribution and rerun SWE-bench Lite under identical training; if the resolution rate does not drop materially, the paper's structural-integration claim is not doing the work. A simpler check: measure oracle-file recall of the Retriever on the 300 Lite issues and compare against the reported 94% recall; if recall collapses on issues where the fix touches a file the static graph lacks edges for, that is where the method would fail.","tokens_in":27455,"feed_emoji":"⚙️","tokens_out":6661,"duration_ms":49113,"temperature":0.7,"pith_summary":"This paper asks whether open-source language models can resolve real-world repository-level issues without agent loops or closed-source models, and claims the answer is yes. The proposed Code Graph Model (CGM) builds a graph of code entities and their dependencies, feeds each node's text through an encoder and adapter, and lets the LLM attend only along graph edges via a modified attention mask. Wrapped in a four-module, agentless RAG pipeline and trained with LoRA, CGM-SWE-PY reaches a 43.00% resolution rate on SWE-bench Lite with Qwen2.5-72B. That result would place an open-weight, agentless system ahead of several agent-based systems that rely on Claude-3.5-Sonnet, narrowing the gap to closed-source approaches.","feed_headline":"Open-source graph model resolves 43% of SWE-bench Lite issues","feed_subtitle":"A graph-fed open LLM folds repository dependencies into attention, matching agent-based systems with no tool calls.","key_machinery":"The carrying mechanism is the graph-aware attention mask combined with compressed node tokens. The code graph itself, built from AST parsing plus lightweight symbol resolution with CHA for inheritance and over-approximation for dynamic calls, defines which nodes exchange messages: each 512-token chunk of node text is encoded and compressed to one token, and attention among these node tokens is restricted to adjacent nodes in the graph, mimicking spatial GNN message passing while the text tokens keep causal attention. The encoder-adapter pair (CodeT5+ with LoRA, plus a two-layer GELU MLP) maps semantic content into the LLM's input space, and the whole encoder-adapter-decoder is trained with LoRA in two phases.","core_discovery":"The central claim is that an open-weight LLM can compete with closed-source agent systems on repository-level issue resolution if the repository's structure is made explicit to the model. CGM encodes each repository as a directed graph with seven node types and five edge types (contains, calls, imports, extends, implements), then integrates it into the LLM in two ways: semantically, node text is encoded by CodeT5+ and projected into the LLM's embedding space by a two-layer MLP adapter, compressing each 512-token chunk into a single node token; structurally, the causal attention mask is replaced between node tokens by the graph's adjacency matrix, so message passing follows code dependencies. After subgraph-reconstruction pre-training and noisy fine-tuning on issue-patch pairs, the model plus a Rewriter-Retriever-Reranker pipeline reaches 43.00% on SWE-bench Lite, ranking first among open-weight methods and second among open-source-systems, surpassing the previous best open-source model-based method by 12.33%.","pith_inferences":["The method does not test whether the static graph itself, rather than the retrieval pipeline, carries the gain; a likely extension is to compare CGM against a flat-text reader given identical retrieved files, which would isolate the value of structural attention from file selection.","Because graph-aware attention is limited to one-hop neighbors per layer, deeper dependency chains must be composed across layers; whether the architecture actually does this is untested, and a layer-wise probe on multi-hop issues would clarify the limit.","The 512-token-to-one-node-token compression implies a strong assumption about information retention; a practical test is to increase chunk size or add per-chunk residual tokens and see if hard cases that depend on local variables benefit.","If the backbone-generalization trend holds, the architecture is a recipe rather than a single model: releasing stronger open-weight base models or better static analyzers should push the 43% upward without redesigning the graph integration."],"forward_implications":["Open-weight, agentless systems become a viable default for repository-level bug fixing, removing the need for closed APIs and reducing privacy exposure when handling proprietary code.","The four-module RAG replaces the ten-step agentless pipeline (e.g., Agentless) with fewer moving parts while slightly improving resolution on the same benchmark, so pipeline complexity is not a prerequisite for strong results.","Graph-integrated fine-tuning transfers across backbone sizes (7B, 32B, 70B, 72B), so the approach can be run in resource-constrained settings and improved by swapping in a better base model.","On code-completion benchmarks (CrossCodeEval, ComplexCodeEval), CGM-Multi-72B outperforms larger baselines such as Mistral-Large-123B and DeepSeek-V2.5-236B, suggesting structural context matters more than raw parameter count for repository-level understanding."],"supporting_citations":[{"why":"Provides the SWE-bench Lite and Verified benchmarks whose resolution rates define the paper's central numerical claims.","marker":"Jimenez et al., 2024"},{"why":"The Agentless pipeline that CGM compares against and whose file-skeleton stage inspires the Reranker's second stage.","marker":"Xia et al., 2024"},{"why":"Moatless-tools with DeepSeek-V3, the previous best open-source-model method that CGM-SWE-PY surpasses by 12.33%.","marker":"Aor, 2024"},{"why":"Qwen2.5-72B-Instruct, the base LLM decoder that CGM fine-tunes and the model used for Rewriter and Reranker.","marker":"Yang et al., 2024a"},{"why":"CodeT5+ encoder that produces the semantic node representations fed through the adapter.","marker":"Wang et al., 2023"},{"why":"LoRA, the low-rank adaptation used to fine-tune encoder and decoder without full parameter updates.","marker":"Hu et al., 2021"},{"why":"GraphCodeBERT, the attention-mask approach to structural integration that CGM extends to decoder-only LLMs at repository scale.","marker":"Guo et al., 2020"},{"why":"DeepSeek-Coder's topological sorting of repository files, reused in CGM's graph-to-code reconstruction pretraining.","marker":"Zhu et al., 2024"},{"why":"The open-source CGE-Large embedding model used for semantic search in the Retriever.","marker":"CodeFuse, 2024"}],"fun_headline_variants":["Graph-fed open LLM resolves 43% of SWE-bench Lite issues","Open LLM reads repo graph, beats agents on SWE-bench","CGM: open LLM with repo graph tackles SWE-bench Lite","Graph-informed open LLM matches agent systems on SWE-bench","Repository graph in attention: open LLM hits 43% on SWE-bench"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The graph built from static analysis must contain the dependency edges that matter for an issue; if key call edges are missed or the over-approximation adds enough wrong edges, the structural signal that the ablation credits could be noise.","fun_headline_variants_meta":{"raw":{"variants":["Graph-fed open LLM resolves 43% of SWE-bench Lite issues","Open LLM reads repo graph, beats agents on SWE-bench","CGM: open LLM with repo graph tackles SWE-bench Lite","Graph-informed open LLM matches agent systems on SWE-bench","Repository graph in attention: open LLM hits 43% on SWE-bench"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001184,"raw_usage":{"total_tokens":4893,"prompt_tokens":955,"completion_tokens":3938,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":3837}},"tokens_in":571,"tokens_out":3938,"duration_ms":16336,"temperature":1.0,"reasoning_tokens":3837,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:53:20.661519+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the code graph in the pipeline with a random graph matched in node count and degree distribution and rerun SWE-bench Lite under identical training; if the resolution rate does not drop materially, the paper's structural-integration claim is not doing the work. A simpler check: measure oracle-file recall of the Retriever on the 300 Lite issues and compare against the reported 94% recall; if recall collapses on issues where the fix touches a file the static graph lacks edges for, that is where the method would fail.","supporting_citations":[],"review_version":1}