{"id":"5534331a-c585-4a83-a731-f6b1cf5730e0","arxiv_id":"2411.17101","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A multi-objective feature fusion plus MLP/GRU pipeline for software fault localization is reported, but the central performance claims are not supported by the presented data.","lead":"This paper combines multi-objective feature selection with MLP and GRU classifiers to find software faults, using spectrum, mutation, and text features. The authors claim large accuracy and speed gains on Defects4J and PROMISE, but the headline numbers do not match their own tables and key baselines are missing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 11 vs Table 12 Top-1 counts are internally inconsistent by 6–13 faults per model, so the paper's headline accuracy numbers cannot be verified from its own data.","rationale":"The reader's stated weakest assumption concerns the missing multi-objective objective vector in Section 3.3. That is a real problem, but the most load-bearing issue is more direct: the reported results that constitute the paper's central claim do not survive a trivial consistency check. Table 12's per-subset Top-1 counts sum to values that differ from Table 11 by 6 to 13 faults for every model. Top-N counts are necessarily additive across disjoint project subsets, so this is not an aggregation subtlety like MAR or MFR; it is an internal contradiction. If the underlying raw rankings are unavailable or the contradiction persists after recomputation, then no amount of specifying the objective vector can salvage the headline accuracy and improvement percentages. I therefore keep the reader's REJECT verdict unchanged, but shift the emphasized concern from the undefined objective vector to the self-inconsistent empirical tables. Agreement is partial because the reader did also note the Table 11/Table 12 mismatch in the rationale, but did not make it the weakest-assumption centerpiece.","tokens_in":15030,"tokens_out":4293,"duration_ms":39564,"concrete_test":"Ask the authors to release the raw per-fault prediction rankings for all 434 Defects4J faults and all compared models. From that data, recompute Top-1/Top-3/Top-5, MAR, MFR, and AUC at project level, then aggregate; verify that project-level values in Table 12 sum exactly to the overall values in Table 11. Independently recompute the stated '7.67%' improvement over DeepFL and the '94.2%' improvement over traditional FL methods from the raw ranks. If the raw data cannot be provided or the recomputation reproduces the inconsistency, the empirical claims should be treated as unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the Defects4J accuracy comparison in Section 5.5. Table 11 reports overall Top-1 counts of 74, 83, 195, 214, and 221 for Tarantula, Dstar, DeepFL, MLP-FL, and RNN-FL, respectively. The same paper's per-subset Table 12 gives Top-1 sums of 80, 89, 208, 226, and 234 when the six project rows are added. These differences (6, 6, 13, 12, and 13) are far too large to be rounding artifacts, and no explanation is given, such as excluded faults or different dataset versions. Since RNN-FL's claimed 221 Top-1 count and its 7.67% improvement over DeepFL are derived from numbers that are mutually contradictory, the key empirical result is unverifiable from the reported data. The undefined objective vector in Section 3.3 is a secondary mechanism-level concern, but it cannot be cured if the headline data do not even add up.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a software fault localization pipeline that combines multi-objective feature selection (NSGA-II, MOPSO, MODE) with feature fusion (voting and weighting) and two deep learning models (MLP-FL and RNN-FL, the latter based on GRU). The features are spectrum-based, mutation-based, and text-based. The models are evaluated on Defects4J (434 faults) and PROMISE, and the claims are that the pipeline reduces processing time by 78.2%, improves localization accuracy by 94.2% over traditional methods, outperforms DeepFL by 7.67% on Defects4J, and gains 4.6% on cross-project PROMISE tests.","tokens_in":15283,"tokens_out":4713,"duration_ms":38023,"significance":"If the results were valid, the paper would contribute a practical combination of multi-objective feature selection with deep learning for fault localization, and the use of public benchmarks (Defects4J, PROMISE) with established baselines (Tarantula, Dstar, DeepFL) is a strength. However, the load-bearing empirical claims are not verifiable from the reported tables: the Defects4J Top-1 totals in Table 11 disagree with the sum of the per-subset rows in Table 12, the 78.2% time reduction is not reproduced by any comparison in Table 9, and the stated 7.67% and 4.6% improvements do not follow from Tables 11 and 14. In addition, the central mechanism of the paper, the multi-objective objective vector in Section 3.3, is never explicitly defined. Because the central claims are unsupported by the paper's own data, the contribution cannot currently be assessed.","major_comments":[{"comment":"The Defects4J Top-1 counts are internally inconsistent. Summing the per-subset Top-1 rows in Table 12 gives 80, 89, 208, 226, and 234 for Tarantula, Dstar, DeepFL, MLP-FL, and RNN-FL, respectively, while Table 11 reports overall Top-1 counts of 74, 83, 195, 214, and 221. The differences (6, 6, 13, 12, 13) are too large to be rounding artifacts and no explanation is given (e.g., excluded faults or different dataset versions). Since the headline accuracy numbers, including the claimed advantage over DeepFL, are derived from these counts, the central empirical result is unverifiable from the reported data.","section":"Section 5.5, Tables 11 and 12"},{"comment":"The objective vector optimized by NSGA-II, MOPSO, and MODE is never defined. Equations (6), (7), and (8) define accuracy, stability, and time separately, but the paper does not state which combination of these quantities constitutes the multi-objective function, how the objectives are aggregated or compared in Pareto dominance, or how stability (a standard deviation across datasets or parameter settings, Eq. 7) is computed for a single feature subset on a single dataset. Without a well-defined objective vector, the feature selection stage is not actually specified as a multi-objective optimization, and the downstream accuracy and runtime results cannot be attributed to the proposed mechanism.","section":"Section 3.3"},{"comment":"Equation (6) defines accuracy as Acc = N_correct / N_total, but the text immediately states that N_correct represents 'the number of true positives' and N_total represents 'the number of true negatives.' This is a mislabeling: if N_total were true negatives, the ratio would not be an accuracy; if N_total is intended to be the total number of statements or faults, the notation conflicts with the confusion matrix in Table 1. Because accuracy is both an optimization objective and an evaluation metric, this ambiguity undermines the quantitative interpretation of the experiments.","section":"Equation (6) and Section 3.3"},{"comment":"The claimed 78.2% processing-time reduction is not reproducible from Table 9. For the MOPSO columns, the summed time over all six Defects4J subsets is 455 s for MLP and 555 s for RNN, while the corresponding baseline columns ('--') sum to 3299 s and 4158 s, giving reductions of about 86.2% and 86.7%. For MODE, the reductions are about 76.4% (MLP) and 77.3% (RNN). No row or average in the table yields 78.2%. The meaning of the baseline columns is also unclear, since the table header does not define what '--' represents beyond 'no multi-objective algorithm.' The abstract's central efficiency claim therefore does not follow from the presented data.","section":"Abstract, Section 5.4, Table 9"},{"comment":"The claimed improvements of 7.67% over state-of-the-art deep learning on Defects4J and 4.6% on PROMISE are not derivable from the tables. In Table 11, RNN-FL's Top-1 of 221 versus DeepFL's 195 is a 13.3% relative improvement, while the MAR values differ by about 13.5%; no calculation using the reported metrics yields 7.67%. In Table 14, the best Top-1 improvement over DeepFL is 191 vs. 183 (about 4.4%) for MLP-FL, and the Top-5 improvement is 271 vs. 259 (about 4.6%), but the paper does not state which metric the 4.6% refers to. These percentages are presented in the abstract and conclusion as headline results, but they are not supported by the experimental tables.","section":"Abstract, Section 6, Tables 11 and 14"},{"comment":"The paper does not specify whether feature selection and hyperparameter choices are nested inside the ten-fold cross-validation folds. The text states that the datasets were randomly divided into 10 parts (Section 5.1), but it does not say whether the multi-objective feature selection is performed on the training fold only or on the full dataset before splitting. If feature selection uses information from the test fold, the reported Top-N and AUC results would be optimistically biased. This is a standard methodological requirement for evaluating feature-selection pipelines, and the omission is consequential for the validity of all reported accuracy comparisons.","section":"Section 5.1 and Section 5.4"}],"minor_comments":[{"comment":"The abstract refers to a 'gated recurrent network (GRN)' while Section 4.2.2 consistently uses 'GRU' (gated recurrent unit). The notation should be unified.","section":"Abstract and Section 4.2.2"},{"comment":"In the sentence after Equation (14), the symbol 'g' is used for the complete hidden layer, but the equation and surrounding text use 'H'. This should be corrected to 'H'.","section":"Equation (14)"},{"comment":"The text states that Top-1, Top-3, and Top-5 'represent the number of faults correctly predicted by the model,' but these metrics actually count faults whose faulty statement is ranked within the top N positions. The wording should be clarified.","section":"Section 5.3, Table 8"},{"comment":"The header of Table 9 uses '--' for the two rightmost columns without defining the baseline condition. Please define what these columns represent, e.g., MLP or RNN without multi-objective feature selection.","section":"Section 5.4, Table 9"},{"comment":"The mutation probability in Equation (3) uses scalar fitness values f_i, f_max, and f_min, but the algorithms are described as multi-objective with vector-valued objectives. The paper should explain how a scalar fitness is derived from the objective vector for this equation.","section":"Equation (3)"},{"comment":"Hyperparameters for the three multi-objective algorithms and the two deep learning models are listed, but the paper gives no justification for these choices or sensitivity analysis. At minimum, a statement that these are standard settings would help.","section":"Section 5.2, Tables 3-7"}],"recommendation":"reject","confidential_remarks":"The internal numerical inconsistencies (Tables 11 vs. 12, the 78.2% claim vs. Table 9) are severe. Even with full revision, the authors would need to re-run and re-report the experiments to make the central claims verifiable. The undefined objective vector in Section 3.3 is a fundamental specification gap. I do not see a path to acceptance within the scope of a normal revision, although a substantially rewritten manuscript with corrected data and a fully specified method could be reconsidered as a new submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a standard feature-fusion fault-localization paper whose headline numbers do not survive contact with its own tables. The stress-test note is right. Table 12's per-subset Top-1 sums exceed Table 11's totals by 6, 6, 13, 12, and 13 faults across the five models, and nothing in the text explains exclusions or version differences. That alone makes the central accuracy claims unverifiable.\n\nWhat is actually new: the particular combination of spectrum, mutation, and text features, selected by NSGA-II/MOPSO/MODE and fused with voting and weighting, then fed to MLP and GRU classifiers. Each component is established, and the paper is honest that this is an incremental combination. The methods sections are clearly written, and the experiments use standard benchmarks (Defects4J, PROMISE) with common metrics (Top-N, MAR, MFR, AUC). Table 8, the feature-type comparison, is internally coherent and shows the expected result that combining feature types helps.\n\nThe soft spots are load-bearing. First, the objective vector for the MOO stage is never defined. Section 3.3 gives formulas for accuracy, stability, and time, but never states which combination NSGA-II, MOPSO, or MODE optimizes, and stability as defined (standard deviation over datasets) does not apply to a single feature subset. That means the core mechanism cannot be audited. Second, the headline percentages are not backed by the tables: the 94.2% improvement over traditional methods does not follow from Table 11 (RNN-FL 221 vs Tarantula 74 is about 199%), and 7.67% is not the Top-1 gap between RNN-FL and DeepFL in that table. The conclusion's 'over 10% improvement in Top-1' is also inconsistent with the 26-fault difference over DeepFL. Third, Eq. 6 defines N_total as true negatives, which is wrong. Fourth, there is no ablation that isolates the contribution of multi-objective fusion, and no code or data are provided. Hyperparameters are not nested inside the cross-validation folds, so selection leakage is possible.\n\nThe stress-test conclusion holds: the paper's key empirical result cannot be verified from its own data. This is reject-level, not polish-level. I would not cite it, and I would not spend referee time on it until the authors fix the arithmetic and define the objective vector. My recommendation: desk reject, with the option to resubmit a corrected version.","headline":"The paper's feature-fusion pipeline is a plausible incremental combination, but its headline accuracy numbers are contradicted by its own tables and the multi-objective core is underspecified.","tokens_in":15814,"tokens_out":3883,"would_cite":false,"duration_ms":34523,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fusing spectrum, mutation, and text features via multi-objective selection improves fault localization: RNN-FL ranks 221 of 434 faulty statements first, beating DeepFL by 7.67% and cutting runtime by 78.2%.","keywords":["software fault localization","multi-objective optimization","feature fusion","spectrum-based features","mutation-based features","text-based features","deep learning","Defects4J"],"falsifier":"Run the MLP-FL and RNN-FL pipelines on Defects4J with (i) the multi-objective-selected fused feature subsets, (ii) the unselected full feature set, and (iii) random subsets of equal size, holding network and hyperparameters fixed; if (ii) or (iii) matches or beats the reported Top-1 and AUC values, the multi-objective selection is not what drives the 7.67% gain. A reader could also inspect Section 3.3 and the experiment scripts to confirm that Eqs. 6–8 define a real objective vector used by NSGA-II, MOPSO, or MODE rather than three disconnected metrics; if only accuracy is optimized, the method is single-objective in effect.","tokens_in":14797,"feed_emoji":"🐞","tokens_out":7161,"duration_ms":61382,"temperature":0.7,"pith_summary":"This paper argues that fault localization improves when feature selection is treated as a multi-objective optimization problem over three complementary feature families: spectrum-based, mutation-based, and text-based. It proposes a pipeline that uses NSGA-II, MOPSO, or MODE to pick compact feature subsets, fuses them by voting and weighting, and feeds the result into an MLP or a gated recurrent network. On 434 Defects4J faults, the best model, RNN-FL, ranks the faulty statement first in 221 cases, beats the DeepFL baseline by 7.67% in average accuracy, and cuts processing time by 78.2% relative to single-objective selection. The same design transfers to the PROMISE dataset with a 4.6% cross-project gain. If correct, this would show that the bottleneck in learning-based fault localization is not just the classifier but the diversity and selection of features fed to it.","feed_headline":"Fused features lift fault localization past DeepFL by 7.67%","feed_subtitle":"Scoring spectrum, mutation, and text features together and selecting them with NSGA-II/MOPSO lifts Top-1 accuracy on 434 real bugs.","key_machinery":"The load-bearing mechanism is feature selection as a multi-objective optimization problem over a binary-coded chromosome, where each bit marks whether a spectrum, mutation, or text feature survives. NSGA-II, MOPSO, and MODE search for Pareto-optimal subsets; the surviving subsets are then fused by voting and weighting to yield an ordered feature vector, which is embedded into MLP-FL and RNN-FL, the latter built around gated recurrent units. The paper computes three objectives, accuracy, stability, and runtime, and uses crowding distance and adaptive mutation to guide the search; the multi-objective stage is what the authors credit with cutting runtime by 78.2% while improving localization accuracy.","core_discovery":"The central claim is that fusing three fault-related feature sets—dynamic spectrum features, dynamic mutation features, and static text features—through multi-objective feature selection and fusion produces fault-localization models that are both more accurate and faster than single-feature or unoptimized deep learning approaches. In the paper's own terms, RNN-FL attains 221 Top-1 hits out of 434, versus DeepFL's 195, with an average accuracy improvement of 7.67%, AUC of 0.928, and lower MAR and MFR; MLP-FL also surpasses DeepFL. The authors claim this holds on Defects4J and generalizes to cross-project PROMISE tests with a 4.6% improvement over DeepFL, and that the multi-objective selection stage reduces processing time by 78.2% compared with single-objective methods.","pith_inferences":["Editorial inference: the reported speedup suggests the real cost in deep fault localization is feature dimensionality, not classifier training; a fair test would compare DeepFL with the same fused input as MLP-FL and RNN-FL to isolate the effect of the feature-selection stage from the effect of changing the network.","Editorial inference: if the method is right, the same multi-objective feature-fusion wrapper could be applied to other deep localizers, such as graph-based or transformer-based models, since the selection step is decoupled from the network architecture.","Editorial inference: the paper's own validity section concedes that public fault datasets are limited in volume and quality, so the reported percentages should be read as bounded by the datasets used; larger and more diverse benchmarks could either confirm or shrink the gains.","Editorial inference: because the objective vector combining accuracy, stability, and time is never explicitly stated, a direct next step would be to define a concrete Pareto objective and report the actual fronts, which would make the multi-objective mechanism testable rather than inferential."],"forward_implications":["If multi-objective feature fusion works as claimed, learning-based fault localizers can add static text features without losing dynamic information; the paper's RQ1 results show combined features beat each single-feature family in Top-1, Top-3, and Top-5 for both MLP and RNN.","If RNN-FL keeps its 7.67% average accuracy edge over DeepFL and its 78.2% time reduction, time-constrained fault localization pipelines should prefer fused, selected feature subsets over exhaustive feature engineering.","The PROMISE cross-project result of 4.6% over DeepFL implies that the selected features transfer beyond the projects they were tuned on, which matters for fault localization on unseen codebases.","The paper's ranking of single-feature methods, spectrum-based over mutation-based over text-based, gives a concrete ordering that future feature engineering can build on, while showing that text features still add value on top of dynamic ones."],"supporting_citations":[{"why":"Defects4J benchmark of 434 real Java faults is the primary dataset for the main accuracy and runtime results.","marker":"[25]"},{"why":"DeepFL is the state-of-the-art deep learning baseline that the proposed MLP-FL and RNN-FL models are compared against and exceed by 7.67%.","marker":"[7]"},{"why":"Tarantula is the traditional spectrum-based fault localization baseline used in the accuracy comparisons.","marker":"[2]"},{"why":"The Dstar method serves as the other traditional baseline in the Defects4J and PROMISE comparisons.","marker":"[3]"},{"why":"The PROMISE dataset supplies the cross-project fault data for the generalizability experiments.","marker":"[26]"},{"why":"The voting method for fusing feature subsets is one of the two fusion steps in the multi-objective feature fusion pipeline.","marker":"[24]"},{"why":"The survey frames the two challenges the paper addresses, limited feature diversity and low localization precision.","marker":"[1]"}],"fun_headline_variants":["Fused features beat DeepFL by 7.67%","Multi-objective fusion lifts fault localization","Deep learning + feature fusion finds bugs faster","Fault localization: 78.2% quicker, 7.67% more accurate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the multi-objective algorithms are actually optimizing a clearly defined combination of accuracy, stability, and time; the paper gives separate formulas (Eqs. 6–8) but never states the combined objective vector, so if that unspecified objective does not exist, the feature selection stage and its claimed gains lose their stated mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Fused features beat DeepFL by 7.67%","Multi-objective fusion lifts fault localization","Deep learning + feature fusion finds bugs faster","Fault localization: 78.2% quicker, 7.67% more accurate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00015,"raw_usage":{"total_tokens":1198,"prompt_tokens":947,"completion_tokens":251,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":183}},"tokens_in":563,"tokens_out":251,"duration_ms":2657,"temperature":1.0,"reasoning_tokens":183,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:30:47.949766+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the MLP-FL and RNN-FL pipelines on Defects4J with (i) the multi-objective-selected fused feature subsets, (ii) the unselected full feature set, and (iii) random subsets of equal size, holding network and hyperparameters fixed; if (ii) or (iii) matches or beats the reported Top-1 and AUC values, the multi-objective selection is not what drives the 7.67% gain. A reader could also inspect Section 3.3 and the experiment scripts to confirm that Eqs. 6–8 define a real objective vector used by NSGA-II, MOPSO, or MODE rather than three disconnected metrics; if only accuracy is optimized, the method is single-objective in effect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defects4J benchmark of 434 real Java faults is the primary dataset for the main accuracy and runtime results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeepFL is the state-of-the-art deep learning baseline that the proposed MLP-FL and RNN-FL models are compared against and exceed by 7.67%."},{"cited_title":"Jones and Mary Jean Harrold","cited_arxiv_id":null,"evidence_quote":"Tarantula is the traditional spectrum-based fault localization baseline used in the accuracy comparisons."},{"cited_title":"Eric Wong, Vidroha Debroy, Ruizhi Gao, and Yihao Li","cited_arxiv_id":null,"evidence_quote":"The Dstar method serves as the other traditional baseline in the Defects4J and PROMISE comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The PROMISE dataset supplies the cross-project fault data for the generalizability experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The voting method for fusing feature subsets is one of the two fusion steps in the multi-objective feature fusion pipeline."},{"cited_title":"Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa","cited_arxiv_id":null,"evidence_quote":"The survey frames the two challenges the paper addresses, limited feature diversity and low localization precision."}],"review_version":1}