{"id":"71b249ad-8b6f-402e-8f94-83853821a6be","arxiv_id":"2412.11304","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An empirical comparison of four deep learning fault localisation tools on 22 faults shows low accuracy under a single ground truth, with measured recall rising when alternative equivalent fixes are included.","lead":"This paper tests four automated tools that try to locate faults in deep learning models, using 22 real and artificially created bugs. It finds all tools perform poorly when judged against a single official fix, and better when many alternative valid fixes are counted, with DeepFD scoring highest.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.31-to-0.61 recall gain is likely an artifact of scoring each tool against the best of 1-291 search-generated alternative ground truths; without a random-output baseline or a control for alternative-set size, the headline does not demonstrate better localisation.","rationale":"Reader's verdict of CONDITIONAL is appropriate. I read the paper as an empirical comparison whose headline claim is that DFD is most effective and that FL tools are underestimated by single-GT evaluation. Both claims rest on the post-neutrality numbers. The weakest assumption is indeed the max-over-alternative-GT scoring, which the reader identified. The BFS search generates a large number of alternative GTs with a weak neutrality criterion; matching any one of them is not equivalent to localising the actual fault. Since the per-fault alternative counts vary widely (Table 9) and DFD outputs multiple fault types, the maximum-recall protocol creates a selection bias that grows with k. A random baseline is the decisive control: if it achieves comparable recall, the paper's main quantitative contribution is not supported. The paper is transparent about its benchmark provenance and provides a replication package, and it explicitly acknowledges the DFD benchmark overlap, which is a secondary caveat; I do not think the study should be rejected, but the headline numbers should be reported with a random baseline or with per-fault alternative-set-size controls. Thus no change to the reader's CONDITIONAL verdict is needed.","tokens_in":22060,"tokens_out":6465,"duration_ms":59460,"concrete_test":"Run a random-baseline control under the exact post-neutrality protocol: for each of the 22 issues, sample a random output of the same length as each tool's actual output from the union of fault-type labels used in the benchmark, then score it by maximum RC across that issue's alternative GTs exactly as in Table 10. Repeat 1,000 times and average. If the random baseline's mean post-neutrality RC approaches DFD's 0.61 (e.g., within 0.15), the reported improvement is an artifact of the max-over-alternatives scoring rather than evidence of localisation skill. A complementary single-check variant: score DFD using one randomly chosen alternative GT per fault instead of the maximum; if recall falls back toward 0.31, selection bias is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central effectiveness claim (Abstract; Section 5.2, Tables 10-11) is based on taking, for each fault, the maximum Recall (RC) across all alternative ground truths found by the BFS neutrality analysis (Algorithm 1, Table 9). The alternative set per fault ranges from 1 to 291 nodes. A tool is credited if its output fault-type list intersects any one of these alternatives. This is a max-over-k protocol: as k grows, the probability that a tool output containing one or more common fault types (e.g., DFD outputs 1-4 types per issue in Table 4) matches at least one alternative increases, even if the tool has no actual localisation skill. The paper does not compare against a random-output baseline, does not report per-issue k in the scoring, and does not show that each BFS neighbour is a genuine root-cause repair rather than an arbitrary neutral hyperparameter mutation (neutrality is mean-accuracy equivalence over 10 trainings, a weak criterion). The jump from average RC 0.31 to 0.61 can therefore be explained by the size of the alternative sets, especially for M1/M2 with 240/291 alternatives. Precision inherits the same optimism because the reported PR and F3 values correspond to the same best-matching GT choices. The DFD-vs-benchmark overlap acknowledged in Section 5.1 compounds the ranking issue, but the max-over-alternatives protocol is the more fundamental threat because it can inflate every tool's score and is the basis for the headline numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical evaluation of four state-of-the-art fault localisation tools for deep learning programs (DeepFD, DeepDiagnosis, Neuralint, and Umlaut) on a benchmark of 22 faulty models, 13 with artificially injected faults and 9 real faults curated from bug-reporting platforms. The evaluation uses precision, recall, and F3, first against a single ground truth and then against a set of alternative ground truths produced by a BFS-based neutrality analysis. The main finding is that under a single ground truth all tools achieve low effectiveness (maximum average recall of 0.31), but when alternative ground truths are considered, recall increases substantially, with DeepFD achieving the highest average recall of 0.61. The paper also investigates the stability, efficiency, and actionability of the tools' outputs.","tokens_in":22438,"tokens_out":5314,"duration_ms":44936,"significance":"This is the first third-party systematic comparison of these four DL fault localisation tools, and the authors contribute a curated benchmark and an augmented dataset with extended ground truths. The idea of considering multiple equivalent repairs in FL evaluation is novel and important for the community. The paper ships a replication package, which is a strength. However, the central effectiveness claim depends on a methodological choice—taking the maximum recall over up to 291 alternative ground truths—that is not adequately justified and can inflate scores regardless of tool quality. Without a random-output baseline or a well-defined aggregation rule, the quantitative ranking of tools is not trustworthy as it stands. The qualitative analyses of stability, efficiency, and output actionability are useful and less affected by this issue.","major_comments":[{"comment":"The post-neutrality effectiveness scores are computed by taking, for each fault, the maximum RC across all alternative ground truths (the number of alternatives ranges from 1 to 291 per fault, as shown in Table 9), and then averaging these maxima. This is a max-over-k protocol: the score is an upper bound that increases with the number of alternatives and with the number of fault types in a tool's output. A tool that outputs several common fault types will match at least one of the many search-generated alternatives purely by chance. To support the headline claim that recall grows from 0.31 to 0.61, the paper must report per-fault alternative counts, include a random-output control (or a control that outputs a fixed set of common fault types), and report an expected recall (e.g., the average over all alternative GTs) rather than the maximum. Without such a baseline, the reported improvement cannot be attributed to the tools' localisation skill.","section":"§5.2, Tables 10–11"},{"comment":"The neutrality criterion is defined as 'equivalent or higher fitness' based on the mean accuracy of ten trained instances, with no statistical test or tolerance threshold. As a result, many nodes in the neutrality graph may have accuracy that is not meaningfully different from the buggy model, and they may not represent genuine repairs of the root-cause fault. The paper should validate that the alternative GTs indeed repair the fault as intended, e.g., by requiring a statistically significant improvement over the buggy model (as is done for the 'blue' nodes) or by reporting a sensitivity analysis with stricter neutrality thresholds. Currently, the alternative GTs used in Table 10 may include incidental hyperparameter changes that are unrelated to the actual fault, making the FL evaluation overly permissive.","section":"§3, Algorithm 1 and footnote 1"},{"comment":"The paper states that it reports 'the maximum RC observed across all ground truth variants and the average PR and F3 calculated on these GTs,' but it is not specified which GT is used to compute PR and F3 when multiple alternatives yield the same maximum recall. If PR and F3 are computed for the GT that maximises RC, then precision is also optimistically biased, and the values such as M1-DFD RC=0.67, PR=0.5, F3=0.65 are not directly comparable to the pre-neutrality metrics. The definition of the aggregated PR and F3 must be made explicit, and the same rule must be applied consistently to all tools, preferably by fixing an aggregation rule (e.g., average over all GTs) rather than a per-fault maximum.","section":"§5.2, Table 10"},{"comment":"The preprocessing step that removes 'softmax' suggestions from DeepDiagnosis, Umlaut, and Neuralint, and overfitting warnings from Umlaut, is a subjective filter that can change the ranking. For example, in Table 7, Umlaut's output for M2 includes 'ACH(7), HLR', and the ACH match is retained; but for many other issues the softmax suggestion is the only one, and removing it makes the output empty. The paper should justify the filtering with a reproducible criterion and show how sensitive the results are to this decision. Currently, the filtering is applied post hoc and may favour tools that happen to avoid such messages.","section":"§5.1, Tables 4–7"},{"comment":"The Wilcoxon signed-rank tests comparing DeepFD with DeepDiagnosis and with Umlaut are reported with p-values (0.002 and 0.023, respectively), but the paper does not specify what data were used (per-issue RC? F3? before or after neutrality?) or how many issues were included. Moreover, the difference between DeepFD and Neuralint is reported as not statistically significant (p=0.066), which weakens the abstract's claim that 'DeepFD is the most effective tool'. The authors should clarify the statistical procedure and moderate the ranking claim accordingly.","section":"§5.2, statistical tests"},{"comment":"The real-fault section of the benchmark is derived from the evaluation dataset of DeepFD, and DeepFD's detectable fault types were chosen based on that dataset's fault distribution. This is acknowledged in Section 5.1, but the consequence is that DeepFD has an inherent advantage on the real-fault subset, which is where its improvement over other tools is largest (e.g., RF Avg. RC after neutrality: DFD 0.50 vs. NL 0.22). To support the general claim about tool effectiveness, the authors should either re-analyse the two benchmark halves separately as the primary result, or obtain an independent set of real faults. At minimum, the headline average recall of 0.61 should be accompanied by a clear caveat about this overlap.","section":"§4.2 and §5.1"}],"minor_comments":[{"comment":"There are grammatical issues in the abstract, e.g., 'increases also the need' and 'such figures increase when considering alternative, equivalent patches'. These should be corrected.","section":"Abstract"},{"comment":"The definition of the post-neutrality aggregation (maximum RC, and average PR/F3 'calculated on these GTs') should be introduced in the evaluation metrics section, not only in Section 5.2, to avoid ambiguity.","section":"§4.3"},{"comment":"The 'Improvement' column shows values such as 0.000 for M1–M3, which seems inconsistent with the text saying improvements are marginal; consider adding more digits or explaining the rounding.","section":"Table 9"},{"comment":"The greyed-out issues (R1, R3, D2, D9) are not visually distinct in the rendered table; consider a footnote or other notation to indicate where alternative GTs were not found.","section":"Table 10"},{"comment":"The construct-validity section should explicitly discuss the threats posed by the max-over-alternatives aggregation and the manual filtering of tool outputs, as these are the most significant measurement choices.","section":"§6.1"},{"comment":"The actionability analysis is qualitative; adding a small table summarising how often each tool provides layer-level information and actionable fix suggestions would make it easier to compare tools.","section":"§5.5"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper addresses an important problem and provides a reusable benchmark and replication package. However, the main effectiveness result is built on an optimistic max-over-alternatives metric without a random baseline, and the real-fault subset overlaps with DeepFD's own evaluation data. These are fixable within a revision: the authors can re-run the evaluation with a defined aggregation rule and a random-output control, and can separate the analysis by benchmark half. I recommend major revision rather than rejection because the underlying study is valuable and the quantitative claims can be made sound with targeted rework. I would also note that the statistical reporting needs to be more complete for the paper to be archival."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read, and worth a serious referee, but the stress-test is right about the numbers. The jump from 0.31 to 0.61 recall is a max over up to 291 alternative ground truths, not a typical performance figure. Without a random-output baseline or per-issue k, that gain does not demonstrate better localisation.\n\nWhat's actually new: this is the first third-party comparison of DeepFD, DeepDiagnosis, Neuralint, and Umlaut. The benchmark combines real faults and DeepCrime mutants. They run tools 20 times, report stability and runtime, analyse output actionability, and are upfront about the DeepFD benchmark overlap. The neutrality analysis idea—expanding the ground truth to equivalent patches—is a genuinely useful direction, and the paper makes a plausible qualitative case that single-GT evaluation understates tools.\n\nThe soft spots are real, in proportion. First, the max-RC protocol is an optimistic bound. With 240–291 alternative GTs for M1/M2, any tool that outputs a few common fault types will match at least one alternative by chance. The paper does not provide a random baseline or control for alternative-set size. Second, the neutrality criterion is weak: mean accuracy over 10 trainings means many arbitrary hyperparameter mutations count as 'equivalent fixes'. That inflates the alternative set. Third, the abstract presents 0.61/0.41 as plain averages, with no caveat that they are maximums. Finally, the real-fault section comes from DeepFD's own evaluation data; the disclosure helps, but it means the ranking on real faults is not independent.\n\nAll that said, the qualitative conclusion probably holds: single ground truth is too strict for this task, and tools like DeepFD and Neuralint are better than a naive reading of the original numbers suggests. The paper's reporting is transparent enough that a reader can see the protocol and judge it.\n\nSend it to review. A good referee should ask for a random baseline, per-issue alternative counts, a less optimistic metric (e.g., union-based or expected recall), and a headline that does not overstate the max. With those changes, the neutrality-analysis contribution would be solid.","headline":"Useful first third-party comparison of DL fault-localisation tools, but the headline recall gain from neutrality analysis is an optimistic max-over-alternatives bound, not a typical performance figure.","tokens_in":22929,"tokens_out":4890,"would_cite":true,"duration_ms":45632,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Single-ground-truth evaluation undervalues DL fault localisation tools; counting equivalent fixes lifts best recall from 0.31 to 0.61.","keywords":["deep learning fault localisation","empirical study","neutrality analysis","alternative ground truths","DNN debugging","mutation testing","DeepFD"],"falsifier":"Re-run the same comparison but give every fault the same number of alternative fixes, for example ten randomly sampled neutral neighbours, and require each claimed match to reproduce a performance gain on a held-out test set; if the best tool's recall no longer approaches 0.61, the reported improvement is an artefact of the variable and large alternative sets.","tokens_in":21861,"feed_emoji":"🔧","tokens_out":5313,"duration_ms":42840,"temperature":0.7,"pith_summary":"This paper argues that the standard way of judging deep-learning fault localisation tools, checking whether the tool's reported fault matches one human-defined fix, undervalues the tools. On a benchmark of 22 faulty models, 9 real and 13 mutant-generated, the four state-of-the-art tools reach at most 0.31 average recall and 0.23 precision against that single ground truth. When the authors run a neutrality analysis that searches for many alternative fixes with equal or better model accuracy, the picture changes: the best tool, DeepFD, reaches 0.61 average recall and 0.41 precision. The paper's central claim is that evaluation protocols should count all equivalent patches, not just the one a human happened to write.","feed_headline":"Neutrality analysis lifts best DL fault-localiser recall to 0.61","feed_subtitle":"Judging tools against all equivalent fixes, not one human patch, changes the verdict.","key_machinery":"The neutrality analysis is the mechanism responsible for the central result. It is a breadth-first search that starts from a faulty model, mutates one hyperparameter at a time (activation function, initialisers, loss function, optimiser, learning rate, epochs, batch size, layer structure, or neuron count), retrains each neighbour ten times, and keeps neighbours whose mean accuracy is equal to or better than the parent's accuracy. Models that match or beat the known human fix are added as alternative ground truths. The paper also uses recall, precision, and $F_3$ (weighting recall three times more than precision) to score tools against the original and the extended ground-truth sets.","core_discovery":"On its own terms, the paper claims to show that existing DL fault localisation tools are more effective than a single-ground-truth evaluation suggests, and that the conventional evaluation is therefore misleading. Using a breadth-first search over hyperparameter mutations to build a neutrality network, the authors find between 1 and 291 alternative fixes per fault, with an average of 69 for artificial faults and 28 for real faults. Recalculating performance against these alternative fixes raises DeepFD's average recall from 0.31 to 0.61 and precision from 0.19 to 0.41, with Neuralint also improving substantially, while DeepDiagnosis and Umlaut improve much less. The paper concludes that any empirical evaluation of DL fault localisation should include some form of neutrality analysis, while acknowledging that even with it the recall values remain far from perfect.","pith_inferences":["Because the number of alternative fixes varies from 1 to 291 across faults, part of the recall gain may simply reflect the size of the alternative set; a fairer comparison would fix the number of alternatives per fault or weight them.","The neutrality-analysis methodology could transfer to other evaluation problems in deep learning, such as repair, hyperparameter optimisation, or test adequacy, wherever 'correct' has multiple equivalent outcomes.","A testable prediction follows: if tools are given credit for matching any one of many mutants, tool ordering may change on larger benchmarks; a stricter metric that requires a tool to propose the specific repaired configuration would likely keep recall lower.","One could directly validate the alternative fixes by checking on held-out data whether each matched 'equivalent' patch actually restores performance, rather than relying on mean training accuracy from the search."],"forward_implications":["Judging DL fault localisation tools against a single human fix underestimates them and can mark correct alternative suggestions as wrong.","Future DL fault-localisation evaluations should report results both on the original ground truth and on an extended ground truth obtained by neutrality analysis.","DeepFD is the most effective tool in this comparison, but also the slowest because it retrains the model 20 times; Neuralint is the efficient static alternative.","Even with alternative fixes counted, the best average recall is 0.61, so DL fault localisation remains an open problem.","Tool rankings are largely stable across the two evaluation modes, with DeepFD significantly ahead of DeepDiagnosis and Umlaut."],"supporting_citations":[{"why":"Supplies the curated benchmark of real and artificial faulty models and the mutation operators reused for neutrality analysis.","marker":"[16]"},{"why":"Describes DeepFD, the tool that emerges most effective, and provides the original set of real faults later curated into the benchmark.","marker":"[3]"},{"why":"Describes Neuralint, the static-analysis tool under comparison.","marker":"[4]"},{"why":"Describes Umlaut, the rule-based tool under comparison.","marker":"[5]"},{"why":"Describes DeepDiagnosis, the dynamic-analysis tool under comparison.","marker":"[2]"},{"why":"Provides the notion of software neutrality on which the alternative-ground-truth search is built.","marker":"[15]"},{"why":"DeepCrime generated the artificially injected faults in the benchmark.","marker":"[13]"},{"why":"The taxonomy of real DL faults defines the fault types used to label the ground truth and tool outputs.","marker":"[14]"}],"fun_headline_variants":["Neutrality analysis boosts best DL fault-localiser recall","Single-patch evaluation underrates DL fault localisers","Equivalence-aware scoring lifts DL fault localisation results","All-fix evaluation changes DL fault localiser verdicts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The large recall increase rests on counting a tool as successful when its reported fault matches at least one of the automatically generated alternative fixes, where 'equivalent' is defined by mean accuracy from ten retrainings.","fun_headline_variants_meta":{"raw":{"variants":["Neutrality analysis boosts best DL fault-localiser recall","Single-patch evaluation underrates DL fault localisers","Equivalence-aware scoring lifts DL fault localisation results","All-fix evaluation changes DL fault localiser verdicts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001824,"raw_usage":{"total_tokens":7150,"prompt_tokens":898,"completion_tokens":6252,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":6187}},"tokens_in":514,"tokens_out":6252,"duration_ms":39937,"temperature":1.0,"reasoning_tokens":6187,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:04:33.418121+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same comparison but give every fault the same number of alternative fixes, for example ten randomly sampled neutral neighbours, and require each claimed match to reproduce a performance gain on a held-out test set; if the best tool's recall no longer approaches 0.61, the reported improvement is an artefact of the variable and large alternative sets.","supporting_citations":[{"cited_title":"Repairing dnn architecture: Are we there yet?","cited_arxiv_id":null,"evidence_quote":"Supplies the curated benchmark of real and artificial faulty models and the mutation operators reused for neutrality analysis."},{"cited_title":"Umlaut: Debugging deep learning programs using program structure and model behavior,","cited_arxiv_id":null,"evidence_quote":"Describes Umlaut, the rule-based tool under comparison."},{"cited_title":"DeepDiagnosis: automatically diagnosing faults and recommending actionable fixes in deep learning programs,","cited_arxiv_id":null,"evidence_quote":"Describes DeepDiagnosis, the dynamic-analysis tool under comparison."},{"cited_title":"Taxonomy of real faults in deep learning systems,","cited_arxiv_id":null,"evidence_quote":"The taxonomy of real DL faults defines the fault types used to label the ground truth and tool outputs."}],"review_version":1}