Pith. sign in

REVIEW 4 major objections 3 minor 84 references

Fault Localisation and Repair for DL Systems: An Empirical Study with LLMs

T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prompted GPT-4 outperforms dedicated tools at localising and repairing faults in deep learning models on a curated benchmark of 34 faults, reaching average FL recall of 0.91 versus 0.61 for the best existing tool and a median repair…

desk verdict Solid empirical study; GPT-4's advantage is real but the headline numbers are inflated by training-data leakage and a best-match ground-truth protocol. read the letter →

arxiv 2506.03396 v1 pith:AHZNAHZW submitted 2025-06-03 cs.SE

Jinhan Kim , Nargiz Humbatova , Gunel Jahangirova , Shin Yoo , Paolo Tonella This is my paper
classification cs.SE
keywords deeplearningfaultlocalisationrepairlargelanguagemodelsGPT-4neutralityanalysishyperparameteroptimisationmutationsempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models, specifically GPT-4, are more effective than existing specialised tools at both localising and repairing faults in deep learning models. It builds a benchmark of 25 artificial and 9 real-world DL faults, and it argues that previous evaluations unfairly penalised tools by assuming only one correct fix exists. Using a neutrality analysis to find alternative valid patches, the paper reports GPT-4 achieves 0.91 average recall in fault localisation versus 0.61 for the best baseline, and a median repair Improvement Rate of 1.0 versus 0.55. If correct, LLM prompting becomes a practical, fast and stable alternative to rule-based diagnosis and search-based repair for DL systems.

What carries the argument

The approach rests on a custom prompt that asks a GPT model to review DL source code and output numbered fault causes (for localisation) or to output replacement hyperparameter values in JSON format (for repair). The evaluation is carried with a curated benchmark whose artificial faults come from DeepCrime mutation operators and whose real faults are reproduced from the DeepFD dataset. A neutrality analysis, implemented as a breadth-first search over single-hyperparameter mutations, discovers alternative ground-truth patches, and the paper recomputes localisation and repair metrics against this extended ground truth. The repair effectiveness metric is the Improvement Rate, the ratio of the patch's performance gain to the ground-truth patch's gain.

What would settle it

Run GPT-4 and the best baseline tools on a fresh sample of DL faults taken from developer posts or commits after GPT-4's training cutoff, applying the same repair effectiveness and localisation metrics; if GPT-4's recall and improvement rate drop to near-baseline levels on that sample, the reported advantage is an artifact of the curated benchmark rather than a general property of LLMs.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs, especially GPT-4, substantially outperform state-of-the-art fault localisation and repair tools for DL models on a curated benchmark, and that this advantage holds across artificial and real faults. For fault localisation, GPT-4 attains an average recall of 1.0 on artificial faults and 0.79 on real faults after neutrality analysis, far exceeding the best existing tool DeepFD at 0.68 and 0.50 respectively. For repair, GPT-4 and GPT-4T achieve a median Improvement Rate of 1.0, meaning their patches reach the same performance as the ground-truth fix, while the best baseline (Random and HEBO) reaches only 0.55. The paper also shows that GPTs are the most stable repair technique across runs, with notably lower standard deviation than hyperparameter optimisation methods, and that their patches are less complex than those of search-based tools.

Load-bearing premise

The benchmark's artificial and reproduced faults are representative of real developer-made DL faults, so the measured gap between GPT-4 and existing tools would transfer to practice.

Editorial extensions

If this is right

  • If these results generalise, a single prompted GPT-4 call can serve as a fast first-line fault localisation and repair tool for DL models, with average runtime around 15 seconds per fault compared to minutes or hours for training-based tools.
  • Evaluation practice for DL fault localisation and repair should treat the ground truth as a set of acceptable patches rather than a single fix, because alternative valid fixes substantially change measured tool performance.
  • Random search remains a surprisingly strong repair baseline, outperforming a dedicated SE repair tool and matching state-of-the-art hyperparameter optimisers on the studied benchmark.
  • LLM-generated patches are closer in complexity to the ground truth than search-based patches, but still more complex, pointing toward patch minimisation as an open need.
  • GPT stability is high across repeated runs even with non-zero temperature, making LLM recommendations usable in practice despite inherent nondeterminism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gap may partly reflect data leakage, since GPT-4 was trained on public code repositories; a fair test would evaluate on DL faults introduced after the model's training cutoff.
  • The benchmark filtering thresholds of 10-15 percentage points accuracy drop and 1.5-5 times loss growth likely exclude faults that are trivial for rule-based tools, so the comparison may favour any method that handles medium-difficulty faults.
  • Since basic prompting already works, retrieval-augmented generation or few-shot examples with known DL faults could push LLM performance further, though the paper does not test this.
  • The results suggest that LLM repair would benefit from an explicit patch-minimisation step, since GPT-4 often finds valid patches that are more complex than the ground-truth fix yet still simpler than search-based patches.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper presents an empirical comparison of fault localisation (FL) and repair techniques for deep learning (DL) systems against LLMs (GPT-3.5, GPT-4, GPT-4T). The benchmark contains 25 artificial faults generated with the public DeepCrime mutation tool and 9 real faults from StackOverflow. The authors introduce a neutrality analysis that expands the ground truth with alternative patches, and evaluate four FL tools, three repair tools, Random search, and GPTs. The central claims are that GPT-4 achieves far higher FL recall (0.91 after neutrality analysis vs. 0.61 for the best baseline) and far higher repair effectiveness (median IR 1.0 vs. 0.55) than existing tools, and that LLMs are a promising avenue for DL fault handling.

Significance. If the results are robust, this is a valuable empirical contribution: it is the first study to compare multiple state-of-the-art DL FL/repair tools with LLMs on a common benchmark, and it makes a methodological proposal (neutrality analysis for alternative ground truths) that could influence future evaluations. The authors provide a public replication package and detailed per-fault tables, which are clear strengths. However, the central quantitative claims are threatened by training-data contamination of the LLM evaluation, by a circularity in the expanded-ground-truth scoring, and by the very small and hand-curated benchmark. These issues need to be resolved before the headline conclusions can be taken as evidence of generalizable LLM superiority.

major comments (4)
  1. [Section 9.2, 3.2, 3.3, Table 9] The statement in Section 9.2 that artificial faults 'guarantee their absence from the training set' is not supported: the artificial faults are derived from the public DeepCrime replication package (Section 3.2) and the real faults are sourced from StackOverflow (Section 3.3), both plausible components of GPT-4's training corpus. The near-identical reported losses for all UnityEyes faults (0.032) and all Reuters faults (0.81) in Table 9 are more consistent with retrieval of canonical configurations than with per-fault reasoning. To support the claimed generalization, the authors should run a contamination-controlled experiment, e.g., with a held-out set of faults not derivable from public code, or show that perturbing non-essential details of the subject programs degrades GPT's performance in a way consistent with genuine reasoning rather than memorization.
  2. [Section 6.2, Table 6] The RQ1.2 evaluation matches each tool's output with the most similar alternative GT among those discovered by the neutrality analysis (stated in the Table 6 caption). This makes the ground truth a function of the tool's output and can inflate reported recall for any tool; the reported recall of 0.91 for GPT-4 is partly an artifact of this matching. The paper should report the primary analysis against the fixed, pre-specified original GT (as in RQ1.1), and treat the neutrality analysis strictly as a sensitivity analysis, or pre-register the rule for expanding the GT before examining tool outputs.
  3. [Section 3.2, Table 2, Table 3] The benchmark construction relies on hand-chosen thresholds (10 to 15 percentage points accuracy drop, 1.5 to 5 times loss) and, when multiple mutants remain, random selection of one mutant per operator and dataset. The benchmark also excludes two AU/UD faults because of time constraints and contains only nine real faults. No sensitivity analysis is provided to show that the relative ranking of tools is stable under reasonable variations of these thresholds and sampling choices. Without such analysis, the benchmark may represent a narrow difficulty band, and the measured gap between LLMs and existing tools may not transfer to realistic DL fault distributions.
  4. [Appendix A, Section 5.3.2] The final paragraph of Appendix A states that the authors 'filter out these misleading suggestions' from the output of UMLAUT and DeepDiagnosis (e.g., recommendations to apply softmax when it is already the activation function). This filtering is applied manually and appears to be applied only to the baseline tools, while GPT-4's output is mapped to fault types by one author without an equivalent explicit filtering step. The evaluation protocol should define the filtering rule a priori and apply it symmetrically to all tools; otherwise the comparison is at risk of subjective bias.
minor comments (3)
  1. [Section 5.3.2] The mapping of GPT-4 and Neuralint outputs to fault types is performed by a single author; reporting inter-annotator agreement or providing the mapping rules would improve reproducibility.
  2. [Section 3.3] The description of the real-fault benchmark says the DeepFD dataset contains 58 buggy models but only nine survive filtering; the filtering criteria are summarized but the per-fault reasons (e.g., irreproducibility, no significant improvement) are not tabulated, which would help readers assess selection bias.
  3. [Figure 3] The boxplots of IR values aggregate all faults but do not show the number of statistically significant patches per tool; adding counts or sample sizes below each boxplot would make the comparison easier to interpret.

Circularity Check

1 steps flagged · score 4.0 of 10

Neutrality analysis defines recall as the maximum over alternative ground truths, making post-analysis FL gains definitional; the core LLM superiority claim is independent and survives.

  1. fitted input called prediction [Section 6.2 (RQ1.2), Table 6; see also Section 4.1 and Table 7]
    "This table shows the highest average RC achieved across all GT variants, along with the average PR and F3 values calculated for the corresponding GTs. This means that we match the output of each tool with the most similar GT among the available ones."

    The reported 'recall after neutrality analysis' is, by the paper's own definition, the highest recall over all available GT variants. Because the original GT is included among the variants, this quantity is at least the pre-neutrality recall for every tool; hence the improvement is a mathematical consequence of taking a maximum over a superset. The evaluation additionally matches each tool's output to the most similar GT, i.e., the ground truth is fitted to the tool's output before the score is computed. The RQ1.2 conclusion that 'all FL tools exhibit improved performance' therefore restates the definition of the metric rather than an empirical finding. The central LLM-vs-baseline claim survives this step (GPT-4 pre-neutrality RC 0.84 vs. DeepFD 0.31), so the circularity is partial.

full rationale

The paper's central claim, that GPT-4 outperforms existing DL FL/repair tools, is backed by a concrete benchmark and does not reduce to a fitted parameter. The main derivation chain (Table 5, Table 7, Table 9, Figure 3) is self-contained: repair IR and FL recall are computed from tool outputs against fixed ground truths. The only notable circular element is the neutrality-analysis evaluation (Section 4.1, RQ1.2): 'recall after neutrality analysis' is defined as the highest recall across all alternative GTs, and the paper explicitly matches each tool's output to the most similar GT. Since the original GT is among the variants, the reported improvements (e.g., DeepFD recall 0.31 -> 0.61) are definitional consequences of taking a maximum over a superset, and the RQ1.2 answer 'all FL tools exhibit improved performance' restates this. This is a form of fitted-input-called-prediction in the evaluation protocol, but it is not the load-bearing support for the LLM advantage: before neutrality analysis GPT-4 recall is 0.84 vs. DeepFD 0.31, and the repair results are independent of the neutrality expansion. The data-leakage concern raised in Section 9.2 ('guaranteeing their absence from the training set') is an overstatement, since the artificial faults come from the public DeepCrime replication package, but this is a validity threat rather than a circular derivation, and the paper does disclose the leakage risk. Self-citations to DeepCrime [28,29] and the taxonomy [26,27] support benchmark construction but are externally published and independently falsifiable, so they are not load-bearing circularity. Overall score 4 reflects one definitional evaluation step with a surviving independent central claim.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the benchmark's representativeness and on several hand-chosen thresholds and evaluation choices. The neutrality analysis introduces an ad hoc definition of neutrality and a best-matching-GT evaluation protocol. There are no invented physical entities or fitted parameters in the traditional sense, but the benchmark selection and evaluation metric choices act as free parameters that influence all reported results.

free parameters (6)
  • accuracy drop thresholds = 10-15%pt
    Mutants with accuracy drops below 10%pt or less than 15%pt worse than the original are excluded; these hand-chosen bounds select faults of intermediate difficulty.
  • loss ratio thresholds for regression mutants = 1.5 to 5 times original loss
    Hand-chosen range to keep regression mutants neither too easy nor too hard to repair.
  • top_k in neutrality BFS = 5
    Limits the number of neighbouring models enqueued at each BFS step; chosen as a computational budget without sensitivity analysis.
  • time budget for neutrality analysis = 48 hours
    Stopping criterion for the BFS search for alternative ground truths.
  • F-beta parameter beta = 3
    Weights recall three times more than precision in FL evaluation, based on the assumption that recall matters more in fault localisation.
  • repair operator set size = 12
    Top 12 fault types by prevalence from a taxonomy, selected as the search space for random, HPO, and GPT repair.
assumptions (5)
  • domain assumption DeepCrime mutation operators produce faults representative of real-world DL faults
    The artificial fault part of the benchmark relies on DeepCrime's mutation taxonomy, which is based on analysis of real DL faults. The paper cites this taxonomy but does not independently validate the representativeness of the chosen mutants.
  • domain assumption The statistical mutation killing criterion correctly identifies faults
    Mutants are included only if they exhibit a statistically significant drop in test accuracy (p<0.05, non-negligible effect size). This criterion is adopted from prior work.
  • domain assumption Accuracy and loss are the appropriate fitness measures for repair and neutrality
    All repair evaluation and neutrality analysis use test accuracy (or regression loss) as the fitness function, ignoring other potential quality criteria.
  • ad hoc to paper A model is neutral to another if the mean of ten trained instances is not significantly worse
    The neutrality definition and the ten-training-run protocol are introduced for this study and directly shape the set of alternative ground truths.
  • domain assumption Manual mapping of tool and LLM outputs to fault types is correct
    One author mapped the natural language outputs of Neuralint and GPT-4 to fault types. No inter-rater reliability or automated validation is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fault Localisation and Repair for DL Systems: An Empirical Study with LLMs." pith.science (2026). https://pith.science/paper/AHZNAHZW

@misc{pith2026250603396,
  author       = {Pith},
  title        = {Pith review of: Fault Localisation and Repair for DL Systems: An Empirical Study with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AHZNAHZW}},
  note         = {Machine review of arXiv:2506.03396}
}
read the original abstract

Numerous Fault Localisation (FL) and repair techniques have been proposed to address faults in Deep Learning (DL) models. However, their effectiveness in practical applications remains uncertain due to the reliance on pre-defined rules. This paper presents a comprehensive evaluation of state-of-the-art FL and repair techniques, examining their advantages and limitations. Moreover, we introduce a novel approach that harnesses the power of Large Language Models (LLMs) in localising and repairing DL faults. Our evaluation, conducted on a carefully designed benchmark, reveals the strengths and weaknesses of current FL and repair techniques. We emphasise the importance of enhanced accuracy and the need for more rigorous assessment methods that employ multiple ground truth patches. Notably, LLMs exhibit remarkable performance in both FL and repair tasks. For instance, the GPT-4 model achieves 44% and 82% improvements in FL and repair tasks respectively, compared to the second-best tool, demonstrating the potential of LLMs in this domain. Our study sheds light on the current state of FL and repair techniques and suggests that LLMs could be a promising avenue for future advancements.

Figures

Figures reproduced from arXiv: 2506.03396 by the authors.

Figure 1
Figure 1. An example neutrality graph for the known patch (black node) of D4; green (resp. blue) nodes improve [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Average execution time and average performance ( [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗
Figure 3
Figure 3. IR values from all faults in the benchmark, broken down by the combinations of repair technique and [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Complexity of statistically significant patches. The [PITH_FULL_IMAGE:figures/full_fig_p026_4.png]
Figure 5
Figure 5. Figure 5: Asymmetric Jaccard of statistically significant patches. The [PITH_FULL_IMAGE:figures/full_fig_p027_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

84 extracted references · 46 canonical work pages

  1. [1]

    The Keras CIFAR10 Dataset of Colour Images

    2000. The Keras CIFAR10 Dataset of Colour Images. (2000). Available at https://keras.io/api/datasets/cifar10/

    2000

  2. [2]

    DeepCrime Replication Package

    2020. DeepCrime Replication Package. https://zenodo.org/record/4772465

    work page arXiv 2020

  3. [3]

    Keras Reuters Dataset

    2021. Keras Reuters Dataset. Available at https://keras.io/api/datasets/reuters/

    2021

  4. [5]

    James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization.Journal of machine learning research13, 2 (2012)

    2012

  5. [6]

    Eric Brochu, Vlad M Cora, and Nando De Freitas. 2010. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning.arXiv preprint arXiv:1012.2599(2010)

    arXiv 2010

  6. [7]

    Tom B Brown. 2020. Language models are few-shot learners.arXiv preprint arXiv:2005.14165(2020). , Vol. 1, No. 1, Article . Publication date: September 2018. Fault Localisation and Repair for DL Systems: An Empirical Study with LLMs 31

    arXiv 2020

  7. [8]

    Jialun Cao, Meiziniu Li, Xiao Chen, Ming Wen, Yongqiang Tian, Bo Wu, and Shing-Chi Cheung. 2021. Replication package of DeepFD. https://github.com/ArabelaTso/DeepFD

    2021

  8. [9]

    Jialun Cao, Meiziniu Li, Xiao Chen, Ming Wen, Yongqiang Tian, Bo Wu, and Shing-Chi Cheung. 2022. DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs. InProceedings of the 44th International Conference on Software Engineering(Pittsburgh, Pennsylvania)(ICSE ’22). Association for Computing Machinery, New York, NY, USA, 573–585. http...

    arXiv 2022

Show all 84 references
  1. [10]

    Jialun Cao, Meiziniu Li, Xiao Chen, Ming Wen, Yongqiang Tian, Bo Wu, and Shing-Chi Cheung. 2022. DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs. InProceedings of the 44th International Conference on Software Engineering (ICSE 2022). ACM

    2022

  2. [11]

    Jialun Cao, Meiziniu Li, Ming Wen, and Shing-chi Cheung. 2023. A study on prompt design, advantages and limitations of chatgpt for deep learning program repair.arXiv preprint arXiv:2304.08191(2023)

    2023 arXiv

  3. [12]

    Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. 2015. Deepdriving: Learning affordance for direct perception in autonomous driving. InProceedings of the IEEE International Conference on Computer Vision. 2722–2730

    2015

  4. [13]

    Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. 2017. Multi-view 3d object detection network for autonomous driving. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1907–1915

    2017

  5. [14]

    Marc Claesen and Bart De Moor. 2015. Hyperparameter search in machine learning.arXiv preprint arXiv:1502.02127 (2015)

    2015 arXiv

  6. [15]

    Alexander I Cowen-Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexan- dre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, et al . 2022. HEBO: Pushing The Limits of Sample-Efficient Hyper-parameter Optimisation.Journal of Artificial Intell...

    2022

  7. [16]

    George E Dahl, Tara N Sainath, and Geoffrey E Hinton. 2013. Improving deep neural networks for LVCSR using rectified linear units and dropout. In2013 IEEE international conference on acoustics, speech and signal processing. IEEE, 8609–8613

    2013

  8. [17]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating large language models in class-level code generation. InProceedings of the IEEE/ACM 46th International Conference on Software Engine...

    2024

  9. [18]

    Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and efficient hyperparameter optimization at scale. InInternational Conference on Machine Learning. PMLR, 1437–1446

    2018

  10. [19]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineerin...

    2023

  11. [20]

    Matthias Feurer and Frank Hutter. 2019. Hyperparameter optimization. InAutomated machine learning. Springer, Cham, 3–33

    2019

  12. [21]

    Isa Fulford and Andrew Ng. [n. d.]. ChatGPT Prompt Engineering for Developers. Available at https://www. deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/

  13. [22]

    Patrick Henriksen, Francesco Leofante, and Alessio Lomuscio. 2022. Repairing misclassifications in neural networks using limited data. InProceedings of the 37th ACM/SIGAPP Symposium on Applied Computing. 1031–1038

    2022

  14. [23]

    Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu

    Abram Hindle, Earl T. Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012. On the Naturalness of Software. InProceedings of the 34th International Conference on Software Engineering(Zurich, Switzerland)(ICSE ’12). IEEE Press, Piscataway, NJ, USA, 837–847

    2012

  15. [24]

    Hinton, L

    G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury. 2012. Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups.IEEE Signal Processing Magazine29...

    2012

  16. [25]

    Q. Hu, L. Ma, X. Xie, B. Yu, Y. Liu, and J. Zhao. 2019. DeepMutation++: A Mutation Testing Framework for Deep Learning Systems. In2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1158–1161

    2019

  17. [26]

    Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2020. Taxonomy of Real Faults in Deep Learning Systems. InThe proceedings of the 42nd IEEE/ACM International Conference on Software Engineering (ICSE 2020). 1110–1121

    2020

  18. [27]

    Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2020. Taxonomy of real faults in deep learning systems. InProceedings of the 41st International Conference on Software Engineering, ICSE

    2020

  19. [29]

    Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. 2021. DeepCrime: Mutation Testing of Deep Learning Systems Based on Real Faults. InProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing , Vol. 1, No. 1, Article . Publication date: September 201...

    2021

  20. [30]

    Nargiz Humbatova, Jinhan Kim, Gunel Jahangirova, Shin Yoo, and Paolo Tonella. 2024. An Empirical Study of Fault Localisation Techniques for Deep Learning.arXiv preprint arXiv:2412.11304(2024)

    2024 arXiv

  21. [31]

    Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. 2019. A comprehensive study on deep learning bug characteristics. InProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineeri...

    2019

  22. [32]

    Gunel Jahangirova and Paolo Tonella. 2020. An Empirical Evaluation of Mutation Operators for Deep Learning Systems. In2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). IEEE, 74–84

    2020

  23. [33]

    Gunel Jahangirova and Paolo Tonella. 2020. An Empirical Evaluation of Mutation Operators for Deep Learning Systems. InIEEE International Conference on Software Testing, Verification and Validation (ICST’20). IEEE, 12 pages. https://doi.org/10.1109/ICST46399.2020.00018

    2020

  24. [34]

    Kevin Jamieson and Ameet Talwalkar. 2016. Non-stochastic best arm identification and hyperparameter optimization. InArtificial intelligence and statistics. PMLR, 240–248

    2016

  25. [35]

    Yue Jia and Mark Harman. 2011. An analysis and survey of the development of mutation testing.IEEE transactions on software engineering37, 5 (2011), 649–678

    2011

  26. [36]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation.arXiv preprint arXiv:2406.00515(2024)

    2024 arXiv

  27. [37]

    Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. Inferfix: End-to-end program repair with llms. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Softwa...

    2023

  28. [38]

    Donald R Jones. 2001. A taxonomy of global optimization methods based on response surfaces.Journal of global optimization21, 4 (2001), 345–383

    2001

  29. [39]

    Jinhan Kim, Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella, and Shin Yoo. 2023. Repairing DNN Architecture: Are We There Yet?. In2023 IEEE Conference on Software Testing, Verification and Validation (ICST). https://doi.org/10. 1109/ICST57152.2023.00030

    2023

  30. [40]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980 (2014)

    2014 arXiv

  31. [41]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017. ImageNet Classification with Deep Convolutional Neural Networks.Commun. ACM60, 6 (May 2017), 84–90. https://doi.org/10.1145/3065386

    2017 doi

  32. [42]

    Yann LeCun. 1998. The MNIST Database of Handwritten Digits. (1998). Available at http://yann.lecun.com/exdb/mnist/

    1998

  33. [43]

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 919–931

    2023

  34. [44]

    Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2017. Hyperband: A novel bandit-based approach to hyperparameter optimization.The Journal of Machine Learning Research18, 1 (2017), 6765–6816

    2017

  35. [45]

    Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. 2018. Tune: A Research Platform for Distributed Model Selection and Training.arXiv preprint arXiv:1807.05118(2018)

    2018 arXiv

  36. [46]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems36 (2024)

    2024

  37. [47]

    Kaibo Liu, Yiyang Liu, Zhenpeng Chen, Jie M Zhang, Yudong Han, Yun Ma, Ge Li, and Gang Huang. 2024. LLM-Powered Test Case Generation for Detecting Tricky Bugs.arXiv preprint arXiv:2404.10304(2024)

    2024 arXiv

  38. [48]

    Yangtao Liu, Hengyuan Liu, Zezhong Yang, Zheng Li, and Yong Liu. 2024. Empirical Evaluation of Large Language Models for Novice Program Fault Localization. In2024 IEEE 24th International Conference on Software Quality, Reliability and Security (QRS). IEEE, 180–191

    2024

  39. [49]

    Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, et al

  40. [50]

    Gábor Melis, Chris Dyer, and Phil Blunsom. 2017. On the state of the art of evaluation in neural language models. arXiv preprint arXiv:1707.05589(2017)

    2017 arXiv

  41. [51]

    2017.Design and analysis of experiments

    Douglas C Montgomery. 2017.Design and analysis of experiments. John wiley & sons

    2017

  42. [52]

    Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, et al. 2024. L2ceval: Evaluating language-to-code generation capabilities of large language models. Transactions of the Association for Computational Lin...

    2024

  43. [53]

    Amin Nikanjam, Houssem Ben Braiek, Mohammad Mehdi Morovati, and Foutse Khomh. [n. d.]. Replication package of Neuralint. Available at https://github.com/neuralint/neuralint

  44. [54]

    Amin Nikanjam, Houssem Ben Braiek, Mohammad Mehdi Morovati, and Foutse Khomh. 2021. Automatic fault detection for deep learning programs using graph transformations.ACM Transactions on Software Engineering and Methodology (TOSEM)31, 1 (2021), 1–27

    2021

  45. [55]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  46. [56]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2023. LLM is Like a Box of Chocolates: the Non- determinism of ChatGPT in Code Generation.arXiv preprint arXiv:2308.02828(2023)

    2023 arXiv

  47. [57]

    Joseph Renzullo, Westley Weimer, Melanie Moses, and Stephanie Forrest. 2018. Neutrality and Epistasis in Program Space. InProceedings of the 4th International Workshop on Genetic Improvement Workshop(Gothenburg, Sweden)(GI ’18). Association for Computing Machinery, New York, N...

    2018

  48. [58]

    Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing machine learning based systems: a systematic mapping.Empir. Softw. Eng.25, 6 (2020), 5193–5254. https: //doi.org/10.1007/s10664-020-09881-0

    2020 doi

  49. [59]

    2002.Flexibility and efficiency enhancements for constrained global design optimization with kriging approximations

    Michael James Sasena. 2002.Flexibility and efficiency enhancements for constrained global design optimization with kriging approximations. University of Michigan

    2002

  50. [60]

    Eldon Schoop, Forrest Huang, and Bjoern Hartmann. 2021. Replication package of UMLAUT. Available at https: //github.com/BerkeleyHCI/umlaut

    2021

  51. [61]

    Eldon Schoop, Forrest Huang, and Bjoern Hartmann. 2021. Umlaut: Debugging deep learning programs using program structure and model behavior. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–16. , Vol. 1, No. 1, Article . Publication date: Septe...

    2021

  52. [62]

    Weijun Shen, Jun Wan, and Zhenyu Chen. 2018. Munn: Mutation analysis of neural networks. In2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE, 108–115

    2018

  53. [63]

    Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical bayesian optimization of machine learning algorithms.Advances in neural information processing systems25 (2012)

    2012

  54. [64]

    Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. 2015. Scalable bayesian optimization using deep neural networks. InInternational conference on machine learning. PMLR, 2171–2180

    2015

  55. [65]

    Jeongju Sohn, Sungmin Kang, and Shin Yoo. 2022. Arachne: Search Based Repair of Deep Neural Networks.ACM Transactions on Software Engineering Methodologyto appear (2022)

    2022

  56. [66]

    Matthew Sotoudeh and Aditya V Thakur. 2021. Provable repair of deep neural networks. InProceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 588–603

    2021

  57. [67]

    Bing Sun, Jun Sun, Long H Pham, and Jie Shi. 2022. Causality-based neural network repair. InProceedings of the 44th International Conference on Software Engineering. 338–349

    2022

  58. [68]

    Shogo Tokui, Susumu Tokumoto, Akihito Yoshii, Fuyuki Ishikawa, Takao Nakagawa, Kazuki Munakata, and Shinji Kikuchi. 2022. NeuRecover: Regression-Controlled Repair of Deep Neural Networks with Training History.arXiv preprint arXiv:2203.00191(2022)

    2022 arXiv

  59. [69]

    Muhammad Usman, Divya Gopinath, Youcheng Sun, Yannic Noller, and Corina S Păsăreanu. 2021. NN repair: Constraint-Based Repair of Neural Network Classifiers. InInternational Conference on Computer Aided Verification. Springer, 3–25

    2021

  60. [70]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision.IEEE Transactions on Software Engineering(2024)

    2024

  61. [71]

    Mohammad Wardat, Breno Dantas Cruz, Wei Le, and Hridesh Rajan. 2021. Replication package of DeepDiagnosis. Available at https://github.com/deepdiagnosis/icse2022

    2021

  62. [72]

    Mohammad Wardat, Breno Dantas Cruz, Wei Le, and Hridesh Rajan. 2022. DeepDiagnosis: automatically diagnosing faults and recommending actionable fixes in deep learning programs. InProceedings of the 44th International Conference on Software Engineering. 561–572

    2022

  63. [73]

    Wardat, W

    M. Wardat, W. Le, and H. Rajan. 2021. DeepLocalize: Fault Localization for Deep Neural Networks. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE Computer Society, Los Alamitos, CA, USA, 251–262. https://doi.org/10.1109/ICSE43902.2021.00034

    2021

  64. [74]

    Erroll Wood, Tadas Baltrušaitis, Louis-Philippe Morency, Peter Robinson, and Andreas Bulling. 2016. Learning an Appearance-Based Gaze Estimator from One Million Synthesised Images(ETRA ’16). Association for Computing Machinery, New York, NY, USA, 131–138. https://doi.org/10.11...

    2016

  65. [75]

    Huanhuan Wu, Zheng Li, Zhanqi Cui, and Jianbin Liu. 2022. GenMuNN: A Mutation-based approach to repair deep neural network models.International Journal of Modeling, Simulation, and Scientific Computing(2022), 2341008

    2022

  66. [76]

    Yonghao Wu, Zheng Li, Jie M Zhang, Mike Papadakis, Mark Harman, and Yong Liu. 2023. Large language models in fault localisation.arXiv preprint arXiv:2308.15276(2023)

    2023 arXiv

  67. [77]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1482–1494

    2023

  68. [78]

    Aidan ZH Yang, Claire Le Goues, Ruben Martins, and Vincent Hellendoorn. 2024. Large language models for test-free fault localization. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–12

    2024

  69. [79]

    Shengcheng Yu, Chunrong Fang, Yuchen Ling, Chentian Wu, and Zhenyu Chen. 2023. Llm for test script generation and migration: Challenges, capabilities, and opportunities. In2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS). IEEE, 206–217

    2023

  70. [80]

    Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. 2018. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search.arXiv preprint arXiv:1807.06906(2018)

    2018 arXiv

  71. [81]

    Zhang and W

    H. Zhang and W. K. Chan. 2019. Apricot: A Weight-Adaptation Approach to Fixing Deep Learning Models. In2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 376–387

    2019

  72. [82]

    J. M. Zhang, M. Harman, L. Ma, and Y. Liu. 5555. Machine Learning Testing: Survey, Landscapes and Horizons.IEEE Transactions on Software Engineering01 (feb 5555), 1–1. https://doi.org/10.1109/TSE.2019.2962027

    2019

  73. [83]

    Quanjun Zhang, Chunrong Fang, Yang Xie, YuXiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. 2024. A systematic literature review on large language models for automated program repair.arXiv preprint arXiv:2405.01466(2024)

    2024

  74. [84]

    Xiaoyu Zhang, Juan Zhai, Shiqing Ma, and Chao Shen. 2021. AUTOTRAINER: An Automatic DNN Training Problem Detection and Repair System. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 359–371

    2021

  75. [85]

    Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018. An Empirical Study on TensorFlow Program Bugs. InProceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (Amsterdam, Netherlands)(ISSTA 2018). ACM, New York, NY, ...

    2018

  76. [2018]

    In2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE)

    Deepmutation: Mutation testing of deep learning systems. In2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 100–111

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.