{"id":"a1d2eb70-ca6b-484c-bc30-484ddec15888","arxiv_id":"2411.17621","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"CodeGraphNet, a GraphCodeBERT-plus-GCN embedding with a DeepTree classifier, is claimed to detect five CWE vulnerability classes at 98% accuracy, but its own unseen-data results are much lower (76-87%) and baseline comparisons are not apples-to-apples.","lead":"This paper combines GraphCodeBERT with a graph convolution layer to embed C/C++ code, then classifies code into five vulnerability types with a hybrid decision tree and neural network model, reporting 98% accuracy. The claim is weakened by a large drop to 76-87% accuracy on an additional unseen dataset and by comparisons to model numbers from other datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 98% claim likely reflects test-set leakage: the train/test split described in Table I precedes the random balancing step, so balanced test rows are not independent of training data, and no deduplication is reported.","rationale":"The reader's weakest assumption identified test-set leakage as the key threat, and that is also the most load-bearing concern here. The paper's own Table III provides direct internal evidence: the unseen-dataset accuracies (0.76-0.87) are dramatically lower than the claimed 0.98, and the paper offers no explanation other than dataset characteristics. Given that the methodology describes random balancing and data augmentation before any mention of a held-out split, and never describes deduplication, the most parsimonious explanation for the gap is that Table II's test rows overlap with training data in content or were generated under the balancing procedure. A second, independent problem is internal inconsistency in the central claim: the abstract says DeepTree outperforms state-of-the-art approaches, but Table II shows CodeGraphNet+BERT outperforming DeepTree on essentially every metric. The paper's justification for choosing DeepTree is pragmatic (simpler to build), not performance-based. I agree with REJECT because both the leakage concern and the self-contradicting comparison undermine the headline result, and no code or hyperparameters are released to resolve the ambiguity. The concrete test I propose is the single check that would settle whether the leakage concern lands: a leak-free re-run with balancing confined to training and clone filtering across the split, compared against the Table III range.","tokens_in":15257,"tokens_out":1647,"duration_ms":13452,"concrete_test":"Re-run the CodeGraphNet+DeepTree pipeline on VDISC with a leak-free protocol: (1) apply any balancing or augmentation only to the training portion of the 80/20 split, (2) remove near-duplicate functions across the train/test boundary using a code-clone detector or 0.9 token-overlap filter, and (3) report per-class accuracy on the original untouched test partition. If accuracy falls below roughly 0.90 (comparable to the Table III unseen-data range), the 98% headline claim is not a valid generalization result.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's headline claim, CodeGraphNet plus DeepTree reaching 98% accuracy (Abstract; Table II), rests on the VDISC split described in Section IV and Table I. The paper states it applied 'random balancing procedure and data augmentation approach' after forming the dataset, but does not say the balancing was fit only on training folds or applied to a held-out test set. If augmentation or minority-class duplication was performed before the 80/20 split, or if the same vulnerable code patterns appear in both partitions, then the test set is not a clean out-of-sample evaluation, and 0.98 accuracy is inflated. The paper itself concedes in Section VI-D and Table III that on a separate unseen dataset accuracy drops to 0.76-0.87, an 11-22 point gap. That gap is consistent with leakage inflating Table II, and it directly undermines the abstract's 'improved vulnerability identification with 98% accuracy' as a generalization claim. The methodology section also never describes deduplication, project-level splitting, or clone filtering for the VDISC-based data. Additionally, the promoted DeepTree is not the best entry in Table II: BERT with CodeGraphNet reports AUC 0.99, accuracy 0.99, F1 0.97, surpassing DeepTree's 0.97/0.98/0.96. The paper's own stated reason for preferring DeepTree is implementation simplicity, not superiority, which contradicts the abstract's 'outperforms state-of-the-art approaches'. These two issues together mean the central claim is not supported by the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CodeGraphNet, a feature-embedding method that combines GraphCodeBERT with a graph convolutional network over a line-adjacency graph, and DeepTree, a hybrid decision-tree/neural-network classifier, for five-class CWE vulnerability detection and LIME-based vulnerable-line highlighting. The authors report 98% accuracy for CodeGraphNet plus DeepTree on a balanced VDISC-derived dataset, claim that DeepTree outperforms state-of-the-art approaches, and present qualitative line-level results on Stack Overflow code. The central claims are the 98% accuracy figure and the superiority of DeepTree.","tokens_in":15623,"tokens_out":5100,"duration_ms":46064,"significance":"If the reported 98% accuracy were a valid out-of-sample generalization result and the line-level highlighting were quantitatively validated, the work would be a useful practical contribution to software vulnerability detection. The paper has some positive elements: it uses a publicly documented dataset, provides a public repository link, and describes a reproducible architecture. However, the main quantitative claims are not supported by the reported evaluation: the training/test protocol is not shown to be leak-free, the paper's own Table II contradicts the 'outperforms' claim, and the separate unseen-dataset results in Table III are 11-22 points lower than the headline figure. The significance of the contribution therefore cannot be assessed from the current evidence.","major_comments":[{"comment":"The evaluation protocol does not rule out test-set leakage. The text states that a 'random balancing procedure and data augmentation approach' was applied to the dataset before the balanced samples were used for training and independent testing, and Table I reports an 80/20 split of the already balanced set. The paper never states that balancing or augmentation was applied only to training folds, nor does it describe any deduplication, project-level splitting, or clone filtering. If augmented or duplicated samples appear in both partitions, the Table II test results are not a clean out-of-sample estimate, and the 0.98 accuracy is inflated. The drop to 0.76-0.87 accuracy on the unseen dataset in Table III is consistent with this concern and directly undermines the abstract's claim of 98% accuracy as a generalization result.","section":"Section IV, Table I"},{"comment":"The claim that DeepTree 'outperforms state-of-the-art approaches' is contradicted by the paper's own results. In Table II, the CodeGraphNet+BERT row reports AUC 0.99, accuracy 0.99, precision 0.99, recall 0.98, F1 0.97, and MCC 0.99, whereas the promoted CodeGraphNet+DeepTree row reports AUC 0.97, accuracy 0.98, precision 0.95, recall 0.96, and F1 0.96. The text justifies the choice of DeepTree on the grounds that BERT is an LLM and 'quite challenging to build', not on superior performance. The abstract and Section VI-E overstate what the data show, and Table IV's cross-paper comparison cannot establish superiority because the compared models were evaluated on different datasets.","section":"Section VI-B, Table II"},{"comment":"The proposed graph does not implement the claimed 'where-the-value-comes-from' relationship. The adjacency matrix in Eq. (4) is defined from edges E = {(i, i+1)}, i.e., edges only between consecutive lines of code. The Introduction and the motivating example in Section III claim that the method captures data dependencies, function calls, and contextual relationships, but the actual graph is a simple line-order chain. The GCN in Eq. (6) therefore aggregates only sequential line neighbors, and the reported performance gains cannot be attributed to the semantically richer graph structure that the paper advertises.","section":"Section V-A, Eqs. (2)-(4)"},{"comment":"The vulnerable-line-highlighting contribution (RQ2) is evaluated only qualitatively. The paper shows examples from Stack Overflow and states that experts reviewed the highlighted lines, but it reports no line-level precision, recall, or F1 and no comparison with existing line-level tools such as LineVul. Without quantitative line-level evaluation, the 'vulnerable lines highlighting' claim is not supported beyond anecdotal demonstration.","section":"Section VI-C, Fig. 6"}],"minor_comments":[{"comment":"The text refers to 'Table IV' when presenting the unseen-dataset validation results, but the results appear in Table III. This cross-reference error should be corrected.","section":"Section VI-D"},{"comment":"Several rows contain suspiciously identical values across different feature extractors; for example, the Decision Tree row for CodeBERT is identical to the Decision Tree row for GraphCodeBERT (0.58/0.49/0.42/0.39/0.34/0.27/0.24/3.55/1.81), and the DeepTree rows for CodeBERT and GraphCodeBERT are also identical. If this is not a copy-paste error, the paper should explain why different embeddings produce exactly the same metrics.","section":"Table II"},{"comment":"The discussion states that integrating the classifier with different embedding techniques produces accuracy 'outperforming the results seen in Table II', but Table III reports lower accuracy values than Table II for the CodeGraphNet rows. This sentence appears to reverse the actual comparison and should be rewritten.","section":"Section VII"},{"comment":"Reference [43] is cited for the Google Colab Pro+ platform, but [43] is the LIME paper. The platform citation should be a different reference or removed.","section":"Section VI-A"},{"comment":"There are numerous typos and formatting issues, including 'GrapCodeNet' in Section VI-D, the malformed '0.4 8' entry in the CodeGraphNet SVM row of Table II, and inconsistent use of 'GrapCodeNet' versus 'CodeGraphNet'. A careful proofread is needed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"For the editor: the central quantitative claim is not supported by a valid out-of-sample evaluation, and the paper's own Table II contradicts the headline superiority claim. These are load-bearing issues that would require re-running the empirical study with a leakage-free protocol, new line-level evaluation, and a substantial rewrite of the conclusions, rather than a local revision. I do not see a path to acceptance within the manuscript's current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know that the central claim here—98% accuracy with CodeGraphNet plus DeepTree—does not survive contact with the paper's own evidence. The authors do some things right: they compare several embeddings and classifiers, ask focused RQs, and include an unseen-dataset validation, which is exactly the kind of extra check you want. The idea of layering a GCN on GraphCodeBERT is a reasonable direction, and they deserve credit for putting a Zenodo link in the data availability statement, even if the contents aren't verified in the text.\n\nThe soft spots are load-bearing. First, Table II shows CodeGraphNet+BERT beating their promoted DeepTree on every headline metric (0.99 vs 0.98 accuracy, 0.99 vs 0.97 AUC). The abstract's \"DeepTree outperforms state-of-the-art\" is false by their own numbers; the text even admits BERT is better but says DeepTree is easier to build. Second, the unseen-dataset results in Table III drop to 0.76–0.87 accuracy. That 11–22 point gap is more consistent with train/test leakage than with the paper's CWE-overlap explanation. The methodology describes random balancing and augmentation before the split but never mentions deduplication or project-level partitioning, so the 98% figure is suspect. Third, the GCN math is muddled: equation (5) transforms the pooled 768-d embedding, then equation (6) multiplies by the n×n adjacency matrix. Those dimensions don't match, and the paper never clarifies how per-line features exist after mean-pooling the entire snippet. That's not a minor typo; it's a missing piece of the method. Fourth, the SOTA comparison in Table IV uses numbers from other papers on different datasets, so it's not a controlled benchmark.\n\nThis is a paper with good intentions but a flawed evaluation. The extra validation and the LIME-based highlighter are worth something, but the main result is not reproducible or internally consistent. I would desk reject this version. If the authors fix the split, add dedup, clarify the architecture, and compare on a common benchmark, it could become a worthwhile study—but not as-is.","headline":"Promising setup, but the headline 98% accuracy is undercut by the paper's own tables, a likely leakage problem, and a GCN architecture that does not line up dimensionally.","tokens_in":16168,"tokens_out":3405,"would_cite":false,"duration_ms":32108,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph-augmented embedding and a tree-neural hybrid classify five vulnerability classes at 98% accuracy and mark the vulnerable lines.","keywords":["software vulnerability detection","multi-class CWE classification","feature embedding","GraphCodeBERT","graph convolutional network","DeepTree","code-line localization","LIME"],"falsifier":"Take the same five CWE classes, remove any samples that are near-duplicates or come from the same project as training code, split by project rather than randomly, and rerun CodeGraphNet plus DeepTree; if the accuracy lands at the 0.76–0.87 level of the paper's own unseen-dataset table instead of 0.98, the headline claim is an artifact of data overlap.","tokens_in":15060,"feed_emoji":"🐛","tokens_out":6675,"duration_ms":58577,"temperature":0.7,"pith_summary":"The paper tries to establish that software vulnerabilities can be classified into specific weakness families and traced to exact source lines much more reliably when the embedding of code is built as a graph rather than as plain text. It proposes CodeGraphNet, which pools GraphCodeBERT token embeddings and passes them through graph-convolutional layers over a line-sequence graph, and pairs it with DeepTree, a hybrid of a decision tree and a neural network. On a balanced public dataset of C/C++ snippets spanning five CWE classes, the combination is reported to reach 98% accuracy and 0.97 AUC, beating the other embeddings and classifiers tested and prior detectors. A separate unseen-dataset check reported by the paper drops to 0.76–0.87 accuracy, so the generalization claim depends on how representative that balanced split is.","feed_headline":"Graph code embeddings spot five bug classes at 98%","feed_subtitle":"A graph-augmented embedding and hybrid classifier also highlight the exact lines to fix.","key_machinery":"CodeGraphNet is the central object: a feature embedding that builds a directed line-sequence graph $G=(V,E)$ with one node per line of code and edges $(i,i+1)$, forms the adjacency matrix $A$, and applies graph convolution $x'' = A x'$ after a linear transformation $x' = W h_{\\text{code}} + b$ of the averaged GraphCodeBERT embedding $h_{\\text{code}} = \\frac{1}{T}\\sum_{t=1}^T h_t$, followed by ReLU and mean pooling. DeepTree is the classifier that carries the result: a decision tree trained on CodeGraphNet features produces class-probability vectors that become the inputs of a neural network, combining interpretable tree splits with learned nonlinear combination. The LIME explainer turns DeepTree's local behavior into per-line vulnerability weights. Together these pieces let the model see both semantic context from the pretrained transformer and line-to-line structure from graph propagation, which is why the paper says it can separate five CWE classes and localize defects.","core_discovery":"On the paper's own account, CodeGraphNet is the key step. GraphCodeBERT encodes each snippet into 768-dimensional token embeddings, these are averaged per snippet, and a line-sequential directed-graph adjacency matrix propagates line-level features through a graph convolutional network with ReLU; the final per-sample vector is the mean of the transformed node features. That representation is then fed to DeepTree, in which a decision tree first predicts class probabilities and those probabilities become the input features of a neural network trained with Adam and sparse categorical cross-entropy. The paper reports that this pipeline reaches 0.98 accuracy, 0.97 AUC, and 0.96 F1 on the held-out split of its main comparison table, outperforming LSA, GloVe, FastText, CodeBERT, and GraphCodeBERT embeddings across ten classifiers, and that LIME-based highlighting marks vulnerable lines in real-world code examples. It also reports that on an unseen dataset, per-class accuracy ranges from 0.76 to 0.87, which it attributes to overlapping CWE patterns and the limited set of classes.","pith_inferences":["The reported drop from 0.98 on the balanced split to 0.76–0.87 on the unseen dataset is the number to watch: it suggests that near-duplicate code in the random split, not the embedding design, may be carrying much of the measured gain.","The line-sequence adjacency matrix used here records only that line $i$ precedes line $i+1$; true data-flow edges, where a value defined on one line is used on another, would test whether graph propagation or merely ordering explains the improvement.","Because only five classes are used and the dataset is balanced by random resampling, the practical gain over prior work may be smaller on naturally imbalanced, many-class vulnerability corpora; a project-level evaluation would settle this.","The LIME-based highlighter gives local weights, not a causal explanation; using it to drive automated repair would require additional validation that the highlighted lines are sufficient for fixing the vulnerability."],"forward_implications":["Vulnerability detectors could report the exact lines needing repair, not just the vulnerable function, because the same embedding drives both classification and line-level explanation.","The DeepTree pattern—decision-tree probabilities as neural-network features—can be lifted to other code-classification problems wherever structured embeddings are available.","Code embeddings built from graph propagation over pretrained code models should become the default comparison point for new detectors, since the paper reports that they beat transformer-only and NLP-only embeddings across ten classifiers.","Tool builders could prioritize CWE-119, CWE-120, CWE-469, and CWE-476 with high precision if the 0.98 figure holds on project-separated data."],"supporting_citations":[{"why":"Supplies the pretrained code-representation model whose pooled token embeddings feed CodeGraphNet.","marker":"[26]"},{"why":"Provides the CodeBERT transformer embedding used as a comparative feature extractor.","marker":"[27]"},{"why":"Supplies the graph-neural-network vulnerability detector that motivates and is compared against the proposed graph-based approach.","marker":"[19]"},{"why":"Provides the line-level transformer vulnerability predictor used as a comparison baseline for line explanation.","marker":"[28]"},{"why":"Supplies the motivating example and the explainer-equipped tool whose omissions the paper contrasts with its own line highlighting.","marker":"[29]"},{"why":"Provides the public vulnerability dataset with CWE labels used to train and test the models.","marker":"[30]"},{"why":"Supplies the LIME local-interpretability method used to turn DeepTree predictions into per-line vulnerability weights.","marker":"[43]"},{"why":"Supplies a value-flow graph neural network multi-class vulnerability predictor used as a comparative baseline.","marker":"[45]"}],"fun_headline_variants":["CodeGraphNet: graph embeddings hit 98% accuracy on bug detection","Graph code embeddings plus hybrid tree-net spots defects at 98%","Bug spotting: graph embeddings + tree network reaches 98% accuracy","Graph-augmented embeddings lift software defect detection to 98%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 98% result assumes the balanced random train/test split does not leak near-duplicate vulnerable code into both sides, so the test measures true generalization rather than memorization.","fun_headline_variants_meta":{"raw":{"variants":["CodeGraphNet: graph embeddings hit 98% accuracy on bug detection","Graph code embeddings plus hybrid tree-net spots defects at 98%","Bug spotting: graph embeddings + tree network reaches 98% accuracy","Graph-augmented embeddings lift software defect detection to 98%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2581,"prompt_tokens":1046,"completion_tokens":1535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":1459}},"tokens_in":662,"tokens_out":1535,"duration_ms":10701,"temperature":1.0,"reasoning_tokens":1459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:56:22.890504+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same five CWE classes, remove any samples that are near-duplicates or come from the same project as training code, split by project rather than randomly, and rerun CodeGraphNet plus DeepTree; if the accuracy lands at the 0.76–0.87 level of the paper's own unseen-dataset table instead of 0.98, the headline claim is an artifact of data overlap.","supporting_citations":[{"cited_title":"GraphCodeBERT: Pre-training Code Representations with Data Flow,","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained code-representation model whose pooled token embeddings feed CodeGraphNet."},{"cited_title":"AIBugHunter: A Practi- cal tool for predicting, classifying and repairing software vulnerabili- ties,","cited_arxiv_id":null,"evidence_quote":"Supplies the motivating example and the explainer-equipped tool whose omissions the paper contrasts with its own line highlighting."},{"cited_title":"Draper VDISC Dataset- Vulnerability Detection in Source Code","cited_arxiv_id":null,"evidence_quote":"Provides the public vulnerability dataset with CWE labels used to train and test the models."},{"cited_title":"Why Should I Trust You? Explaining the Predictions of Any Classifier,","cited_arxiv_id":null,"evidence_quote":"Supplies the LIME local-interpretability method used to turn DeepTree predictions into per-line vulnerability weights."},{"cited_title":"Multi-class vulnerability prediction using value flow and graph neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies a value-flow graph neural network multi-class vulnerability predictor used as a comparative baseline."}],"review_version":1}