{"id":"cdcf3cf0-b5db-43f8-a6bb-f58418dc3cec","arxiv_id":"2505.00990","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A relational graph convolutional network with graph type conversion ranks deleted lines in bug-fixing commits and is reported to beat prior methods on Recall@1-3 and MFR, with no error bars or code release.","lead":"This paper presents RC-Detection, a graph neural network method that ranks deleted code lines in bug-fixing commits to identify which deleted line most likely caused the bug. It reports improved ranking accuracy over an existing neural SZZ method on Java open-source data, while supplying no released code and reporting no statistical error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset arithmetic in Table 1 is internally inconsistent, and the ground-truth oracle adopted from Tang et al. is reused with the same inductive biases, so the reported gains lack an independent check.","rationale":"This stress-test agrees with the reader's target: the label oracle is the weakest point in the evidential chain, because every Recall@N/MFR number is computed against it and the strongest baseline, NEURAL-SZZ, shares the same labeling protocol. I highlight the concrete compounding issue of the dataset arithmetic: the paper's abstract and Section 6.2 claim 675 commits and 87 projects, while Table 1 adds to 1,489 bug-fixing commits. A manuscript with internally inconsistent dataset counts cannot substantiate the claimed improvements until the count is reconciled, because it is impossible to know which corpus produced Table 2. This does not change the verdict: as the reader says, the method is coherent and the reported gains are plausible, but the absence of the artifact, significance tests, and independent labels means accept would be premature. I do not find the graph construction or RGCN component to have an evident technical flaw. The threat is empirical verification, not internal inconsistency. Hence CONDITIONAL, matching the reader's verdict, with agreement marked 'partial' because the reader emphasizes label quality while I add the dataset-count discrepancy as a separate load-bearing arithmetic failure.","tokens_in":20147,"tokens_out":1607,"duration_ms":14294,"concrete_test":"Run the full RC-Detection evaluation on a precisely enumerated dataset: list the 87 projects, give the exact number of bug-fix commits used (675 or the Table 1 union), and report per-fold and per-dataset results. Then compare RC-Detection and NEURAL-SZZ on a held-out manually labeled sample of at least 100 bug-fix commits, with labels produced independently of Tang et al.'s pipeline; if the Recall@1 gap over NEURAL-SZZ drops below the reported 4.107% or the MFR advantage reverses, the headline claim fails its central test.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that RC-Detection outperforms NEURAL-SZZ on root-cause line ranking. Two load-bearing problems undercut this. First, the dataset description does not add up: the abstract and Section 6.2 say the study uses 675 bug-fix commits across 87 projects, but Table 1 lists 241 + 957 + 291 = 1,489 bug-fixing commits and 351 + 957 + 378 = 1,686 bug-inducing commits. Neither number matches the stated 675 commits. If the model was evaluated on the full Table 1 union, the reported numbers (Tables 2-7) cannot be audited against the claimed 675-commit corpus, and if it was evaluated on a 675-commit subset, the paper never specifies which one. Second, Section 4.2 states that true labels were obtained by 'adopting the processing methods used by Tang et al.' The strongest comparison baseline is NEURAL-SZZ, which uses Tang et al.'s own labels and pipeline. Any systematic bias in Tang et al.'s labeling (e.g., which deleted lines are marked as root causes, how they handled noise/refactorings) is then baked into the evaluation and is preferential to methods tuned on that same labeling scheme. The paper has no independent oracle, no manual validation sample, and no error analysis of the labels. Sections 6.1 and 6.3 acknowledge only generic threats; they do not address label-quality bias or dataset-count inconsistency. Thus the relative improvements of 4.107%, 5.113%, 4.289%, and 24.536% are not yet established against an unbiased oracle.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RC-Detection, a relational graph convolutional network (RGCN) approach that ranks deleted code lines in bug-fixing commits as candidate root causes of the introduced bug. The pipeline constructs a heterogeneous graph from AST, CFG, DDG, call graph, and class-member-reference relationships between added and deleted lines, converts it into a homogeneous graph with type vectors, embeds code statements with CodeBERT, applies a two-layer RGCN, and ranks deletion nodes with RankNet trained by focal loss. The evaluation compares RC-Detection with NEURAL-SZZ and several machine-learning and deep-learning baselines on three Java datasets, reporting improvements in Recall@1, Recall@2, Recall@3, and MFR, with the headline numbers being 0.811, 0.884, 0.924, and 1.830 and relative gains of 4.107%, 5.113%, 4.289%, and 24.536% over the state of the art.","tokens_in":20477,"tokens_out":6707,"duration_ms":64621,"significance":"If the reported results are trustworthy, RC-Detection would be a useful contribution to SZZ noise reduction and just-in-time defect prediction: the model is clearly described, the graph type conversion is a sensible response to feature-dimension mismatch in heterogeneous code graphs, and the ablations over embedding models, GNN variants, layer counts, regularization, and loss functions are extensive. The paper also specifies its cross-validation strategy and states an intention to release code. However, the evaluation is not currently convincing: the dataset arithmetic is internally inconsistent, the ground-truth labels are inherited from the same pipeline family as the main baseline without independent validation, the headline numbers are point estimates without significance tests, and hyperparameters are selected on the same cross-validation folds used for reporting. Because these issues affect every reported table, the claimed improvements are not yet established.","major_comments":[{"comment":"The dataset description is internally inconsistent. The abstract and Section 6.2 state that the study uses 675 bug-fixing commits and 87 projects, but Table 1 lists 241 + 957 + 291 = 1,489 bug-fixing commits and 351 + 957 + 378 = 1,686 bug-inducing commits, and the project counts in the three blocks do not obviously sum to 87. If the evaluation used the full Table 1 union, the '675 commits' claim is wrong; if it used a 675-commit subset, the selection procedure is never specified. Tables 2 through 7 are aggregate rankings, so this discrepancy is load-bearing for every reported result. Please reconcile these counts and precisely describe the evaluation corpus.","section":"Abstract / Section 6.2 / Table 1"},{"comment":"The ground-truth labels are obtained by 'adopting the processing methods used by Tang et al.' (Section 4.2), and the strongest baseline, NEURAL-SZZ, is exactly Tang et al.'s method (Section 4.4). There is no independent oracle, no manual validation sample, and no label-error analysis in the manuscript, and Section 6.1 discusses only the accuracy of reproducing the compared methods, not the quality or bias of the adopted labels. If the Tang et al. labeling pipeline encodes particular heuristics about which deleted lines are root causes, models adapted to that same pipeline have a built-in advantage, and the measured gains may not transfer to independently labeled data. Please add an external validation of a sample of labels and report agreement against an independent oracle.","section":"Section 4.2 / Section 4.4 / Section 6.1"},{"comment":"The headline comparison in Table 2 consists of single point estimates with no confidence intervals or significance tests, so the 0.032 gap in Recall@1 between RC-Detection and NEURAL-SZZ is not shown to be outside cross-validation noise. Moreover, Section 5.4 selects the number of RGCN layers and the regularization setting by comparing Recall@1 on the same ten-fold cross-validation folds that are later used to report final performance; this is a tuning-on-test procedure that biases the reported numbers. Please report per-fold distributions with significance tests and perform hyperparameter selection on a separate validation split.","section":"Section 5.1 / Table 2 / Section 5.4"},{"comment":"The relative-improvement claims contain internal inconsistencies. Section 5.1 states that RC-Detection 'outperforms Neural SZZ by 69.8%, 52.1%, in MFR,' which contradicts the abstract and Section 5.1 Conclusion, both of which report a 24.536% MFR improvement. In Section 5.4, the model with two layers is described as best on 'Recall@1 and Recall@2' with scores 0.811 and 0.924, but 0.924 is the Recall@3 column in Table 5. Please correct these numbers so that the reported gains are auditable.","section":"Section 5.1 Results / Section 5.4 Results"}],"minor_comments":[{"comment":"The project count should be made consistent with Table 1: if '120 more projects' is additional to the five named entries in the second block, the table implies 135 projects, not 87 as stated in the abstract and Section 6.2.","section":"Section 6.2 / Table 1"},{"comment":"The heading 'Deletion Nodes Rankling Layer' should read 'Ranking Layer,' and the document should be checked for similar typos, including 'DTection' in Section 4.1 and 'bus' in Section 3.4.","section":"Section 3.4.3"},{"comment":"The promise to release code and datasets is not yet fulfilled; please provide a working artifact link or clearly state the planned availability so that the experiments can be reproduced.","section":"Introduction / Section 8"},{"comment":"The sentence 'Considering Neural SZZ using heterogeneous graph neural network, the substantial improvement...' is grammatically incomplete and should be rephrased.","section":"Section 5.1"},{"comment":"The sentence beginning 'Compared to the SOTA methods reported in Section 5.1, RC-Detection with BERT model outperforms them...' is confusing because the proposed method already includes CodeBERT; please clarify which configuration is being compared with which baseline.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is within the journal's scope and worth pursuing, but the evaluation must be rebuilt around a correctly counted dataset and an independently validated label set. I would ask the editor to require the authors to provide the dataset-processing scripts, a manual label-validation sample, and per-fold results with significance tests before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a legitimate step forward on NEURAL-SZZ, not a breakthrough. The architecture is coherent, and the ablation study tells you where the gains come from. But the headline numbers are undermined by a dataset-count inconsistency and by an evaluation that reuses the baseline's own ground-truth labels as the oracle.\n\nWhat's actually new is the graph type conversion component. It collapses the heterogeneous graph (deleted/added nodes, control/data/call/mapping edges) into a homogeneous graph while appending type identifiers as vectors, which lets a relational GCN with basis decomposition do message passing without the over-parameterization seen in NEURAL-SZZ's heterogeneous attention. That's a sensible design and a real (if narrow) contribution. The paper also runs a full set of ablations—BERT variants, GNN layers, layer count, regularization, loss functions—which is more than most SE papers do.\n\nNow the soft spots, in proportion. First, Table 1 does not add up. The abstract and §6.2 say 675 bug-fix commits over 87 projects, but the table rows sum to 1,489 bug-fixing commits and 1,686 bug-inducing commits. The paper never explains which corpus produced Tables 2–7. That's not a fatal flaw by itself, but it's the kind of arithmetic that makes a referee doubt the rest.\n\nSecond, and more load-bearing: §4.2 says ground-truth labels were obtained by 'adopting the processing methods used by Tang et al.' The strongest baseline is NEURAL-SZZ, which uses the same pipeline. Any systematic bias in Tang et al.'s labeling—what counts as a root-cause deletion line, how refactorings and noise are handled—is baked into the test set and is preferential to methods tuned on that same labeling. There is no independent oracle, no manual validation sample, no error analysis. (The limitations sections acknowledge generic threats but not this one.)\n\nThird, there are no error bars or significance tests anywhere. The Recall@1 gain over NEURAL-SZZ is 4.1%; without variance estimates, that could be noise. Compounding that, §5.4 selects layer count and regularization using the same cross-validation folds that are later reported as results, which is tuning-on-test.\n\nTo be fair, the authors state an intent to release code and data; the GitHub link exists. That improves the odds of an auditable artifact, but it's a promise, not a delivery.\n\nWho should read this: anyone working on SZZ variants, JIT defect prediction labeling, or GNNs on commit graphs. The graph conversion trick is worth borrowing.\n\nMy recommendation: it deserves a serious referee, but with a request for the artifact, a reconciliation of the dataset numbers, statistical testing, and at least a small manual label-quality check. If those are addressed, the contribution is a solid incremental result; as it stands, the headline claims are conditional.","headline":"A coherent incremental GNN method for root-cause line ranking, but the evaluation rests on the same label oracle as its main baseline and the dataset count does not add up.","tokens_in":21036,"tokens_out":3442,"would_cite":false,"duration_ms":33894,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RC-Detection claims that ranking deleted lines in a bug-fixing commit with a relational graph convolutional network identifies the true root-cause line better than prior SZZ-based methods.","keywords":["root cause detection","SZZ algorithm","relational graph convolutional network","changed code lines","Just-In-Time defect prediction","heterogeneous graph","code ranking"],"falsifier":"Take a random sample of the 675 bug-fixing commits, have developers or independent annotators mark the true root-cause deleted line from the bug report, and recompute Recall@1, Recall@2, Recall@3, and MFR on that manually labeled subset. If RC-Detection's Recall@1 does not remain above the NEURAL-SZZ baseline on that subset, the central claim of improved root-cause identification fails.","tokens_in":19930,"feed_emoji":"🐛","tokens_out":6167,"duration_ms":57986,"temperature":0.7,"pith_summary":"This paper tries to show that the deleted lines truly responsible for a bug can be ranked above the noise of a bug-fixing commit by modeling the changed code as a graph. Its method, RC-Detection, builds a heterogeneous graph whose nodes are added and deleted lines and whose edges are control-flow, data-dependency, call, and line-mapping relations, then converts it to a homogeneous graph whose type information is preserved as extra vectors. A relational graph convolutional network propagates semantic information through this graph, and a pairwise ranker orders deleted lines by root-cause probability. On three Java datasets totaling 87 projects and 675 bug-fix commits, the paper reports Recall@1, Recall@2, and Recall@3 of 0.811, 0.884, and 0.924, and a mean first rank of 1.830, all better than the prior state of the art. If this holds, Just-In-Time defect prediction gets a sharper, line-level signal about which deletion actually caused the bug.","feed_headline":"Root-cause lines ranked first in 81% of commits by RC-Detection","feed_subtitle":"On 675 Java bug-fix commits, the RGCN approach beats SZZ baselines and cuts mean first rank to 1.83.","key_machinery":"The mechanism is the relational graph convolutional network (RGCN), defined as a message-passing layer that updates each node's hidden state by summing, for every edge relation, a relation-specific weight matrix applied to neighboring node states plus a self-connection. Basis decomposition keeps the parameter count manageable by expressing each relation weight matrix as a linear combination of shared basis matrices. Before the RGCN layer, the graph type conversion component merges node and edge feature matrices into one homogeneous graph while appending type vectors, which preserves the identity of different node and edge kinds even after unification. This combination is what carries the argument: without per-relation aggregation the semantic relationships among changed lines are not exploited, and without the type-preserving conversion the heterogeneous feature dimensions cannot be integrated without over-parameterization.","core_discovery":"The paper's central claim is that root-cause identification in bug-fixing commits reduces to ranking deleted code lines by semantic relatedness to the other changed lines, and that relational graph convolutional networks are the right machinery for that ranking. Given a bug-fixing commit, RC-Detection extracts deleted and added lines as nodes, connects them with edges drawn from control-flow graphs, data-dependency graphs, call graphs, and class-member reference graphs, and adds line-mapping edges between corresponding old and new lines. It then merges the heterogeneous node and edge features into one homogeneous graph while encoding node and edge type as additional vectors, embeds each code statement with CodeBERT, and applies an RGCN whose relation-specific weights are regularized by basis decomposition. Deleted lines are assigned probabilities and ranked with RankNet. The paper reports Recall@1 of 0.811, Recall@2 of 0.884, Recall@3 of 0.924, and MFR of 1.830, which it states are improvements of 4.107%, 5.113%, 4.289%, and 24.536% over the strongest baseline, and concludes that the type-conversion plus relation-aware aggregation is what earns the gain.","pith_inferences":["Because the metrics are computed against labels produced by the same SZZ-style pipeline that the main baseline uses, part of the reported edge may reflect agreement with that pipeline's inductive biases rather than with independently verified root causes; a manual-oracle study would settle this.","The graph type conversion step is generic enough to apply to other heterogeneous program graphs, such as combining abstract syntax trees, dataflow, and call graphs at method level, so it could be tested for fault localization or vulnerability patch discovery.","If the ranking signal is genuinely semantic, RC-Detection should transfer to non-Java languages with the same graph construction, so the absence of such data is a concrete testable next step.","Embedding the ranker into a commit-time review tool that highlights the top-ranked deleted line as a suspected root cause is a plausible productization, given the reported mean first rank below two."],"forward_implications":["Root-cause identification can be treated as a learned ranking problem on code graphs rather than a filtering problem, so the top-ranked deleted line can directly steer SZZ-style labeling of bug-inducing changes.","JIT defect prediction can consume line-level root-cause signals instead of whole-commit labels, which should reduce the false positives and false negatives caused by noisy bug-inducing change labels.","The reported MFR of 1.830 means the true root-cause deleted line appears, on average, within the first two ranked positions, which is within reach of a developer's manual review.","The ablation results imply that relation-type awareness is the active ingredient: replacing RGCN with non-relational graph convolutions consistently lowers Recall@1, so keeping distinct relation types matters.","Retaining both added and deleted lines as graph nodes is necessary, because lines of both kinds carry semantic evidence about which deletion caused the bug."],"supporting_citations":[{"why":"Supplies the ground-truth label-generation pipeline adopted in Section 4.2 and the NEURAL-SZZ baseline that RC-Detection must beat.","marker":"[15]"},{"why":"Defines the relational graph convolutional network whose per-relation weight matrices carry the message-passing in the RGCN layer.","marker":"[17]"},{"why":"Provides CodeBERT, the pretrained code-language model used to embed each changed line into a fixed-length node vector.","marker":"[24]"},{"why":"One of the three Java bug-fix and bug-inducing commit datasets used for evaluation.","marker":"[18]"},{"why":"Second dataset source, contributing regression commits from Java project histories.","marker":"[19]"},{"why":"Third dataset source, used to extend the evaluation across more open-source projects.","marker":"[20]"},{"why":"Defines RankNet, the pairwise ranking model that orders deleted nodes by root-cause probability.","marker":"[32]"},{"why":"Focal loss, the class-imbalance loss used to train the ranker.","marker":"[33]"}],"fun_headline_variants":["RGCN spots root-cause lines 81% of the time in bug-fix commits","RC-Detection: RGCN ranks root-cause lines, MFR 1.8","Root-cause detection MFR cut 24.5% with RGCN","RGCN finds root-cause lines in bug-fixes: 81% Recall@1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the root-cause labels produced by the adopted SZZ-style tracing pipeline are accurate; if those labels are noisy or carry the same biases as the main baseline, the reported improvements are relative to that pipeline rather than to the true root cause of each bug.","fun_headline_variants_meta":{"raw":{"variants":["RGCN spots root-cause lines 81% of the time in bug-fix commits","RC-Detection: RGCN ranks root-cause lines, MFR 1.8","Root-cause detection MFR cut 24.5% with RGCN","RGCN finds root-cause lines in bug-fixes: 81% Recall@1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000894,"raw_usage":{"total_tokens":3912,"prompt_tokens":1060,"completion_tokens":2852,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":2763}},"tokens_in":676,"tokens_out":2852,"duration_ms":18628,"temperature":1.0,"reasoning_tokens":2763,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:29:50.212868+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 675 bug-fixing commits, have developers or independent annotators mark the true root-cause deleted line from the bug report, and recompute Recall@1, Recall@2, Recall@3, and MFR on that manually labeled subset. If RC-Detection's Recall@1 does not remain above the NEURAL-SZZ baseline on that subset, the central claim of improved root-cause identification fails.","supporting_citations":[{"cited_title":"Neural szz algorithm","cited_arxiv_id":null,"evidence_quote":"Supplies the ground-truth label-generation pipeline adopted in Section 4.2 and the NEURAL-SZZ baseline that RC-Detection must beat."},{"cited_title":"Kipf, Peter Bloem, Rianne van·den Berg, Ivan Titov, and Max Welling","cited_arxiv_id":null,"evidence_quote":"Defines the relational graph convolutional network whose per-relation weight matrices carry the message-passing in the RGCN layer."},{"cited_title":"Codebert: A pre-trained model for programming and natural languages","cited_arxiv_id":null,"evidence_quote":"Provides CodeBERT, the pretrained code-language model used to embed each changed line into a fixed-length node vector."},{"cited_title":"Exploring and exploiting the correlations between bug-inducing and bug-fixing commits","cited_arxiv_id":null,"evidence_quote":"One of the three Java bug-fix and bug-inducing commit datasets used for evaluation."},{"cited_title":"Regminer: Towards constructing a large regression dataset from code evolution history","cited_arxiv_id":null,"evidence_quote":"Second dataset source, contributing regression commits from Java project histories."},{"cited_title":"Revisiting and improving szz implementations","cited_arxiv_id":null,"evidence_quote":"Third dataset source, used to extend the evaluation across more open-source projects."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines RankNet, the pairwise ranking model that orders deleted nodes by root-cause probability."}],"review_version":1}