{"id":"7baaa111-a334-434f-9a1d-30e840e2c484","arxiv_id":"2506.14470","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Adding control-flow and data-flow edges to abstract syntax trees helps some graph neural networks but not others, and the best model (GMN) works well even on plain syntax trees.","lead":"This preprint compares several graph-based code representations, plain syntax trees plus enriched versions with control-flow and data-flow edges, across four graph neural networks for detecting code clones. It reports that simpler syntax trees often match or beat heavily enriched graphs, and that the model architecture matters more than extra graph edges.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pair-level split shares all 8,876 code fragments across train/test, so reported GMN gains may reflect fragment leakage rather than architecture.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing concern: the train/test split is at the clone-pair level without guaranteeing disjoint code fragments, and the text/table strongly suggest all 8,876 fragments are shared across splits. This directly threatens the central empirical claim that GMN is superior with plain AST. My independent reading of the manuscript confirms Section IV.F's sparse description of the split and Table I's fragment count appearing only on the training row. The paper provides no statement of fragment-level deduplication or disjoint partitioning. Without fragment-disjoint splits, the reported F1 values, precision, and recall are all potentially inflated, and the comparative claims about which architecture benefits from which representation are unreliable. Other issues, such as the lack of error bars, the non-functional code repository, and the small differences (0.001 F1) among some GMN variants, are secondary; they would affect interpretability and reproducibility, but the split issue is the one that, if confirmed, invalidates the quantitative basis of the paper's main conclusion. I therefore agree with the reader's REJECT verdict. A conditional acceptance might be possible if the authors can rerun with a proper fragment-disjoint split and show the qualitative findings survive, but as submitted the evidence is not sufficient.","tokens_in":12652,"tokens_out":3235,"duration_ms":36138,"concrete_test":"Inspect the released dataset/split construction code to determine whether the 80/10/10 split is applied to clone pairs or to code-fragment IDs. If any fragment ID appears in both training and test sets, rerun the full experiment with a fragment-disjoint split: randomly partition the 8,876 fragments into 80/10/10, then form training/validation/test pairs using only fragments within each partition (and discard cross-partition pairs). Recompute the Table II metrics for all four GNNs and all representations. If GMN's F1 with AST falls below GCN/GAT with AST+CFG+DFG, or if the relative ranking changes, the original conclusion is an artifact of leakage.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that GMN outperforms other GNNs even with the standard AST, reducing the need for enriched structures. That claim rests on the reported F1/precision/recall numbers, but the experimental protocol appears to split at the clone-pair level rather than the code-fragment level. Section IV.F states only that \"The dataset is divided into training (80%), validation (10%), and testing (10%) sets,\" and the sentence before Table I says \"the 8,876 code fragments were paired to form labeled positive and negative clone pairs used across training, validation, and test sets.\" Table I lists 8,876 code fragments for the training row and does not list fragment counts for validation or testing. This strongly implies that the same Java method (fragment) can appear in both training and test pairs. Since GNNs are trained directly on graph structures, a model that has seen a fragment during training can memorize its node/edge patterns, inflating pair-level performance on test pairs that reuse those fragments. The effect is especially dangerous for GMN, which performs cross-graph matching and could benefit disproportionately from recognizing already-seen fragment structures. The paper never states that fragments were partitioned disjointly, nor that duplicate methods across splits were removed. Therefore the absolute F1 values and the headline conclusion \"GMN outperforms others even with standard AST\" are not trustworthy as reported. This is load-bearing because the entire empirical contribution is the comparison of representations and architectures; if the split leaks fragments, every comparison in Tables II and Figure 5 is potentially biased.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of AST-based hybrid graph representations (AST alone, AST+CFG, AST+DFG, AST+CFG+DFG, and FA-AST combinations) for code clone detection, evaluated across four GNN architectures (GCN, GAT, GGNN, GMN) on BigCloneBench. It reports precision, recall, F1, and computational-overhead metrics, and its headline claims are that AST+CFG+DFG consistently helps convolution- and attention-based models, FA-AST edges often hurt performance, and GMN outperforms the other architectures even with a plain AST, reducing the need for enriched structures.","tokens_in":12907,"tokens_out":4881,"duration_ms":51750,"significance":"If the experimental evidence were reliable, this would be a useful systematic benchmark: it compares representations and architectures under a shared implementation, quantifies overhead, and provides an open-source resource for follow-up work. However, the current evidence has a likely train/test fragment-leakage problem, no repeated-run or significance analysis, and a simplified CFG/DFG construction, so the headline conclusions are not currently supported. The research question is worth answering, but the manuscript needs a substantially more rigorous evaluation before its claims can be accepted.","major_comments":[{"comment":"The train/test split is described only as \"The dataset is divided into training (80%), validation (10%), and testing (10%) sets,\" and the sentence before Table I states that \"the 8,876 code fragments were paired to form labeled positive and negative clone pairs used across training, validation, and test sets.\" Table I lists 8,876 code fragments in the training row and gives no fragment counts for validation or testing. This indicates that the same code fragment can appear in both training and test pairs. Since GNNs are trained directly on graph structures, a model such as GMN that performs cross-graph matching can benefit from having seen a fragment's structure during training, inflating the reported F1 values and the \"GMN outperforms\" conclusion. The authors must either split at the fragment level, assigning all pairs containing a given fragment to the same split, or explicitly report and justify any overlap, and then rerun all experiments under a fragment-disjoint split.","section":"IV.F and Table I"},{"comment":"All results in Table II are reported as single precision/recall/F1 values with no multiple seeds, confidence intervals, or significance tests, despite several headline differences being very small—for example, GMN's F1 on AST is 0.953 and on AST+FA+CFG is 0.954, and GGNN's AST, AST+CFG, and AST+DFG all round to 0.900. The claims that \"FA often degrades performance\" and that \"GMN outperforms others\" depend on differences that may be within run-to-run noise. The authors should repeat each configuration with several seeds, report variance, and apply an appropriate multiple-comparison procedure when ranking representations and architectures.","section":"V.A and Table II"},{"comment":"The CFG and DFG are not constructed with standard program-analysis algorithms but by heuristic AST traversal: control edges are added based on \"execution order of statements\" and data edges by a backward scan for a \"nearest dominating definition along the AST traversal path.\" This is a particular simplified edge-augmentation scheme, not a standard CFG or DFG, so the conclusions about \"AST+CFG+DFG\" may not generalize to the flow graphs used in the cited prior work (e.g., [4], [10]). The authors should either use or cite an established CFG/DFG extractor and validate the constructions, or explicitly reposition the findings as being about their heuristic edge additions rather than about CFG/DFG.","section":"IV.C"}],"minor_comments":[{"comment":"The sentence \"we conducted an empirical evaluation across four GNNs architectures: GMN, GCN, GAT and GMN\" lists GMN twice; the second occurrence should be GGNN.","section":"V.A.1"},{"comment":"The bottom panel label \"GMT\" appears to be a typo for \"GMN\".","section":"Figure 2"},{"comment":"Table I is visually misaligned: the \"Average Number of Line of Codes\" and pair counts are not clearly associated with the correct rows, and the validation/test fragment counts are missing.","section":"Table I"},{"comment":"The classification threshold sigma in Eq. (1) is called \"fixed\" but its value is never reported; please state it or give the specific reference where it is defined.","section":"IV.A"},{"comment":"Reference [16] is the same paper as [15] (Li et al., Gated Graph Sequence Neural Networks); the duplicate should be merged, and the typo \"SwA V\" in Section III should be \"SwAV.\"","section":"References"}],"recommendation":"major_revision","confidential_remarks":"I agree with the reader that the fragment-level split issue is the decisive concern; the study needs a full re-run with a fragment-disjoint split before it can be considered. I would not reject outright because the research question and experimental framework are worth pursuing, but the current manuscript is not publishable as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the ablation matrix is genuinely new: nobody has compared AST, AST+CFG, AST+DFG, AST+CFG+DFG, and the FA variants across GCN, GAT, GGNN, and GMN under a single fixed protocol on BigCloneBench. The overhead analysis (generation time, storage, density, inference cost) is a practical addition that most clone-detection papers skip. Second, the experimental split looks broken. The paper says the 8,876 code fragments were paired into train, validation, and test sets, and Table I only lists fragment counts for the training row. That means the same Java method can appear in both training and test pairs. If so, every number in Table II is inflated, and the headline 'GMN outperforms others even with standard AST' may just be GMN memorizing fragments it already saw.\n\nCredit where due: the writing is clear, the research questions are sensible, and the finding that FA often degrades performance (especially for GGNN) is the kind of nuance that could be useful. The computational overhead results are concrete and reproducible in principle.\n\nThe soft spots beyond the split are real but secondary. There are no repeated runs or significance tests; several F1 differences the text treats as meaningful are 0.001, which is noise. The novelty claim that 'no studies focus on the impact of AST-based representations' is contradicted by their own citations [21], [22], though the specific hybrid matrix is new. The GitHub link in the abstract contains spaces and likely does not work, so the promised open-source resource is not verifiable.\n\nThe load-bearing issue is the split. If the authors redo the experiment with a fragment-disjoint split and report means and standard deviations across multiple seeds, the direction of their main finding might survive—GMN's cross-attention could well help on genuinely unseen fragments, and FA's overhead might not pay off. But as submitted, the evidence does not support the central claim.\n\nWho is this for? Researchers working on code clone detection or empirically evaluating graph representations. I would bring it to a reading group as a cautionary tale about data leakage, but not as a source of trustworthy performance numbers.\n\nShould it be peer-reviewed? Yes. A serious editor should send it to referees rather than desk-reject: the comparison matrix is valuable and the flaws are fixable. Referees should ask for a corrected split, statistical rigor, and a working repository link. My own verdict on the current version is reject, but it's the kind of reject that comes with a clear path to a useful paper.","headline":"Useful ablation matrix with a broken train/test split; the GMN-is-best finding is not trustworthy as reported.","tokens_in":13472,"tokens_out":3733,"would_cite":false,"duration_ms":35211,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that for code clone detection, the GNN architecture matters more than enriching the AST with semantic edges, since a graph-matching model (GMN) is most accurate even on plain AST.","keywords":["code clone detection","graph neural networks","code representation","hybrid graphs","abstract syntax tree","GMN","BigCloneBench","empirical study"],"falsifier":"Re-run the comparison on the released splits after checking for overlapping code-fragment IDs between training and test pairs, then retrain all four GNNs on a fragment-disjoint split; if GMN's plain-AST precision (reported as 0.986) and its F1 margin over GCN and GAT drop materially, the conclusion that architecture outweighs enrichment does not survive contamination-free evaluation.","tokens_in":12436,"feed_emoji":"📊","tokens_out":4402,"duration_ms":44049,"temperature":0.7,"pith_summary":"This paper runs a controlled comparison of AST-based code representations—plain AST, AST plus control-flow edges, data-flow edges, and flow-augmented ASTs—across four graph neural network architectures for clone detection, using a balanced and filtered version of BigCloneBench. The central result is that no single representation is best: GCN and GAT improve when CFG and DFG edges are added, FA-AST generally hurts performance, especially for GGNN, and the Graph Matching Network (GMN) achieves the highest precision and near-best recall even on plain AST. The authors conclude that cross-code similarity computation in the model architecture contributes more than the extra semantic edges. The paper also quantifies overhead: DFG generation costs about 21 times more than AST alone, and FA edges roughly double storage and graph density.","feed_headline":"Model design beats graph enrichment for clone detection","feed_subtitle":"A BigCloneBench study finds GMN matches or beats enriched ASTs on plain AST alone, at far lower cost.","key_machinery":"The central object is the Graph Matching Network (GMN), a GNN whose propagation layer shares messages between the two input graphs so that each node's representation is conditioned on the other graph's nodes; the paper implements this layer following Wang et al. The comparison is carried by four propagation architectures (GCN, GAT, GGNN, GMN) under identical embeddings, pooling, and training, with five edge-type combinations built on the AST. The efficiency analysis uses graph density, generation cost, storage cost, and inference time to quantify what the extra edges buy.","core_discovery":"The paper claims that the marginal value of enriching an AST with semantic edges depends on the GNN architecture, and that for the strongest model, GMN, enrichment is largely unnecessary. Across BigCloneBench, AST+CFG+DFG consistently raises F1 for GCN and GAT, but FA-AST edges—the flow-augmented structure introduced by Wang et al.—tend to add structural noise and dilute performance, most clearly for GGNN. GMN, whose graph-matching propagation compares node neighborhoods across the two code fragments, reaches its best precision with plain AST and an F1 within 0.001 of the best hybrid, indicating that the architecture's similarity computation matters more than semantic graph augmentation.","pith_inferences":["Our inference: if the train/test split is not fragment-disjoint, the reported absolute F1 scores and GMN's margin over other models are likely inflated; re-testing on a leakage-free split is needed before taking the architecture-over-representation conclusion as quantitative.","Our inference: the result suggests a testable design principle—direct cross-code comparison mechanisms are a substitute for handcrafted semantic edges, so a transformer with cross-attention over ASTs might match GMN's performance and would extend the finding beyond GNNs.","Our inference: graph density alone is a poor predictor of detection quality, since FA raises density to 0.0202 yet often lowers F1; future representation design should target edge-type informativeness per unit of density.","Our inference: the 21x generation cost for DFG is dominated by backward data-dependency tracking, so a batched or incremental extraction algorithm could materially improve the efficiency trade-off without changing detection accuracy."],"forward_implications":["For practitioners, plain AST with GMN is a lightweight default: near-optimal accuracy with lower storage and faster generation than enriched graphs.","For GCN and GAT users, AST+CFG+DFG is the best-value hybrid because CFG and DFG add complementary semantic signal without the overhead of FA edges.","FA-AST's high density and storage cost (roughly double AST) rarely justify its performance, especially for recurrent architectures like GGNN.","Architecture search may matter more than representation engineering: a model with the capacity to match structures can substitute for handcrafted semantic edges.","Resource-constrained clone detectors can skip DFG generation if the roughly 21-fold generation cost is instead spent on a matching-capable architecture."],"supporting_citations":[{"why":"Supplies the FA-AST construction, the GMN propagation implementation used in the experiments, and the binarized similarity-threshold practice.","marker":"[5]"},{"why":"Provides the AST+CFG+DFG hybrid construction methodology that the paper adapts for control- and data-flow edge generation.","marker":"[4]"},{"why":"Defines the Graph Matching Network architecture that the paper identifies as the best-performing model.","marker":"[16]"},{"why":"Defines the GCN baseline whose performance improves when CFG and DFG edges are added.","marker":"[13]"},{"why":"Defines the GAT baseline whose performance improves with hybrid representations, supporting the architecture-dependent conclusion.","marker":"[14]"},{"why":"The original BigCloneBench benchmark from which the evaluation dataset is derived.","marker":"[24]"},{"why":"Provides the balanced, filtered version of BigCloneBench that the paper merges with the original to restore clone-type labels.","marker":"[25]"}],"fun_headline_variants":["GMN beats enriched ASTs with plain syntax alone","Semantic graph enrichment fails to boost most clone detectors","Architecture trumps graph enrichment for clone detection","Flow-augmented ASTs add noise, not accuracy","Plain AST suffices when the GNN matches well"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 80/10/10 split keeps code fragments disjoint between training and test; if the same function appears in both sides, the model has seen test structure during training, and the reported F1 scores—especially GMN's—are inflated.","fun_headline_variants_meta":{"raw":{"variants":["GMN beats enriched ASTs with plain syntax alone","Semantic graph enrichment fails to boost most clone detectors","Architecture trumps graph enrichment for clone detection","Flow-augmented ASTs add noise, not accuracy","Plain AST suffices when the GNN matches well"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00064,"raw_usage":{"total_tokens":2948,"prompt_tokens":945,"completion_tokens":2003,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":1927}},"tokens_in":561,"tokens_out":2003,"duration_ms":16250,"temperature":1.0,"reasoning_tokens":1927,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:17:19.862362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison on the released splits after checking for overlapping code-fragment IDs between training and test pairs, then retrain all four GNNs on a fragment-disjoint split; if GMN's plain-AST precision (reported as 0.986) and its F1 margin over GCN and GAT drop materially, the conclusion that architecture outweighs enrichment does not survive contamination-free evaluation.","supporting_citations":[{"cited_title":"Detecting code clones with graph neural network and flow-augmented abstract syntax tree,","cited_arxiv_id":null,"evidence_quote":"Supplies the FA-AST construction, the GMN propagation implementation used in the experiments, and the binarized similarity-threshold practice."},{"cited_title":"Learning graph-based code representations for source-level functional similarity detection,","cited_arxiv_id":null,"evidence_quote":"Provides the AST+CFG+DFG hybrid construction methodology that the paper adapts for control- and data-flow edge generation."},{"cited_title":"Gated graph sequence neural networks,","cited_arxiv_id":null,"evidence_quote":"Defines the Graph Matching Network architecture that the paper identifies as the best-performing model."},{"cited_title":"Towards a big data curated benchmark of inter-project code clones,","cited_arxiv_id":null,"evidence_quote":"The original BigCloneBench benchmark from which the evaluation dataset is derived."},{"cited_title":"Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code","cited_arxiv_id":null,"evidence_quote":"Provides the balanced, filtered version of BigCloneBench that the paper merges with the original to restore clone-type labels."}],"review_version":1}