Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An analysis of 38,742 issue reports and 19 interviews claims that deep-learning frameworks fail to support large language models in five specific ways, with fragile environment setup and silent execution instability topping the list.

desk verdict A solid, genuinely useful empirical taxonomy of LLM-centric DL framework issues, but the 'LLM-centric' claim needs a non-LLM baseline and a couple of arithmetic fixes before it is fully convincing. read the letter →

arxiv 2506.13114 v2 pith:VUGR6FM3 submitted 2025-06-16 cs.SE

Yanzhou Mu , Rong Wang , Juan Zhai , Chunrong Fang , Xiang Chen , Jiacong Wu , An Guo , Jiawei Shen
show 2 more authors
Bingzhuo Li Zhenyu Chen
This is my paper
classification cs.SE
keywords deeplearningframeworkslargelanguagemodelsissuereportanalysistaxonomyconstructionempiricalsoftwareengineeringLLM-centricbugsframeworkusabilitydistributedtrainingstability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the difficulties deep-learning (DL) frameworks face in supporting large language models (LLMs) are systematic enough to be captured in a single map. Drawing on 38,742 issue reports from PyTorch, MindSpore, TensorFlow, and eight LLM toolkits, plus interviews with 11 LLM users and eight framework developers, the authors construct a taxonomy of LLM-centric bugs, user questions, and requirements. They then claim that five key challenges, above all fragile environment setup and silent execution instability, are what most limit DL framework support for LLM workflows. If the claim holds, the result is a prioritized, lifecycle-aware list of framework weaknesses that developers and testing researchers can act on directly.

What carries the argument

The central object is the LLM-centric Taxonomy of Challenges in DL Frameworks. It is produced by a two-stage screening pipeline: first a validated keyword filter (72 to 84 keywords per framework) selects candidate issue reports, then manual inspection removes false positives and labels each report as a question, requirement, or bug; a bottom-up clustering step groups the labeled reports into themes and sub-themes. The taxonomy carries the argument because every downstream finding, interview protocol, and recommendation is anchored in its categories and their frequencies. The interview guideline is the second mechanism: it takes the taxonomy to practitioners, validates and extends it, and attaches priority rankings that the issue reports alone cannot provide.

What would settle it

A replication that applies the published keyword lists to a set of issues independently known to be LLM-related, such as reports maintainers have explicitly tagged as LLM-specific, and finds that a large fraction are missed, or a re-annotation of a random sample by a fresh team that yields substantially different type or theme assignments, would settle whether the taxonomy is representative and stable.

Watch

Extended reading notes

Core claim

The paper's central claim is that a taxonomy with three issue types (questions, requirements, bugs), 20 themes, and 75 sub-themes captures a comprehensive set of LLM-centric challenges in DL frameworks, and that five key challenges, especially fragile environment setup and silent execution instability, limit framework support for LLMs. The taxonomy is built bottom-up from manually labeled issue reports and then refined and enriched through interviews, which added categories such as training-strategy confusion (A.5), memory fragmentation and leakage (C.2.iv), and hybrid parallelism misconfiguration (C.3.iv). The paper further argues that users and developers perceive these challenges differently: users prioritize configuration clarity, observability, and reproducibility, while developers prioritize operator portability, memory alignment, and runtime robustness. These divergences are presented as evidence that framework support for LLMs is not just a scaling problem but a distinct systems-engineering problem.

Load-bearing premise

The load-bearing premise is that the keyword-filtered issue reports from the selected repositories form a representative sample of LLM-related framework challenges; if the keyword list misses common LLM terms, if silent usability problems are underreported, or if the spot-check precision is optimistic, the taxonomy's completeness and the reported frequencies shift.

Editorial extensions

If this is right

  • Framework developers get a prioritized defect list: installation and configuration confusion is 27.31% of all user questions (A.1), and unexpected behavior without diagnostic cues is 30.07% (A.3).
  • Observability and maintainability failures dominate at scale, accounting for 39.29% of bug reports across C.6 to C.10, so tooling and documentation should be treated as core reliability concerns rather than peripheral ones.
  • Silent failures, such as lost gradients, invisible checkpoint mismatches, and non-deterministic execution, form the most dangerous bug class and need runtime consistency checks rather than better error messages alone.
  • User and developer priorities diverge enough that framework roadmaps should be validated against both groups: users ask for workflow transparency while developers focus on low-level runtime robustness.
  • The five proposed optimizations (simplified setup, high-level LLM abstractions, consistency checks, lightweight observability, and documentation and tooling investment) are concrete starting points for improving framework support.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the taxonomy's sub-theme frequencies can serve as a triage oracle for issue trackers: a new report matching a high-frequency LLM-centric sub-theme is more likely to be a genuine signal amid the rising volume of AI-generated noise.
  • The 19-person interview sample is small relative to the 38,742-report dataset, so a natural extension is a large-scale survey or instrumented telemetry collection to test whether the five key challenges remain on top outside the studied repositories.
  • The taxonomy has a shelf life tied to the rapid evolution of LLM tooling; as new model families, hardware, and parallelism schemes appear, the keyword list and sub-themes will need periodic refresh, and the paper's pipeline provides a reusable template for that refresh.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper presents an empirical study of challenges that DL frameworks face in supporting large language models. The authors collect and analyze 38,742 issue reports from PyTorch, TensorFlow, MindSpore, and eight associated LLM toolkits (e.g., vLLM, DeepSpeed, Megatron), manually label them, and construct a three-level taxonomy with three issue types (questions, requirements, bugs), 20 themes, and 75 sub-themes. They then interview 11 LLM users and 8 DL framework developers to validate and refine the taxonomy, and derive five key findings and five optimization recommendations. The paper claims that the taxonomy captures a comprehensive set of LLM-centric challenges and that five key challenges, especially fragile environment setup and silent execution instability, limit DL framework support for LLMs.

Significance. If the central claims hold, the paper provides a useful, large-scale empirical map of LLM-related issues in DL frameworks, with practical implications for framework developers, testing researchers, and tool designers. The study's strengths include a large annotated dataset, explicit inter-annotator agreement (Fleiss' kappa 0.717-0.757), a described two-stage precision-checking procedure, and a publicly promised data repository. However, the 'LLM-centric' qualifier is not yet established because no non-LLM baseline is analyzed, and several reported summary percentages are miscomputed. These issues affect the paper's main quantitative claims but are addressable within the manuscript's scope.

major comments (3)
  1. [Section 4.1 and Section 8 (Table 3)] The central qualifier 'LLM-centric' is not supported by the evidence because the analysis lacks a non-LLM baseline. The keyword filter includes generic terms such as 'oom', 'cuda out of memory', 'quantization', 'bfloat16', 'tf.function', and 'graph mode', which also match small-scale or non-LLM deep learning workflows, and the dataset is dominated by LLM-specific toolkits: vLLM alone contributes roughly 35% of the 38,742 reports, and the eight toolkits together contribute about 75%, while the three core frameworks contribute only about 25%. The reported 92.9% precision in Section 4.1 only indicates that a spot-checked 10% of retained issues were judged LLM-related by annotators; it does not show that these issues are LLM-specific or that the observed theme frequencies would differ in a matched sample of general DL issues. The assertion in Section 8 (Table 3) that LLM issues differ 'qualitatively in frequency, impact, and diagnosability' from small-model issues is asserted without a comparative sample. Without a non-LLM baseline, the taxonomy's theme frequencies and the five key challenges cannot be uniquely attributed to LLM workloads.
  2. [Section 4.4.3, Bug Summary and Finding 3] Several quantitative claims about bug theme percentages are incorrect. The summary states that 'observability and maintainability weaknesses (39.29%, C.6–C.10) dominate at scale', but the per-theme percentages in the same section sum to 34.25% (C.6 4.01% + C.7 6.47% + C.8 1.14% + C.9 19.72% + C.10 2.91%). This error also appears in Finding 3 and in Section 6.1's reference to '39.29% of bugs (C.6–C.10)'. Additionally, C.1 is reported as 18.21% of bug-related issues in the theme description in Section 4.4.3 and in the Bug Summary, but Figure 3 reports 6,268 reports for C.1, which is 24.38% of the 25,709 bugs; the 18.21% figure is actually the percentage for requirement theme B.5. These quantitative errors are load-bearing for the ranking claims in Finding 3 and for the derivation of the observability/maintainability challenge.
  3. [Section 6.1, first key challenge] The derivation of the 'Environment Setup Fragility' challenge misstates Finding 1. The text says that 'Finding 1 identifies installation and runtime errors (27.31%, A.1) as the most frequently reported user questions', but Section 4.4.1 reports that A.3 (Unexpected Behavior and Unintuitive Design) is the largest question theme at 30.07%, with A.1 tied with A.4 at 27.31%. This contradiction undermines the evidential basis for the paper's first key challenge and should be corrected for internal consistency.
minor comments (7)
  1. [Table 1] The toolkit name is misspelled as 'DeepSeed'; it should be 'DeepSpeed'.
  2. [Section 5.1] The sentence 'This group includes eight LLM users from leading Internet companies' should read 'eight DL framework developers', because the paragraph is describing the developer interviewee group.
  3. [Figure 2] The word 'captures' appears as a stray token in the B.5 sub-theme list, and should be removed.
  4. [Section 4.4.4] The text references 'Fig. ??' when discussing detailed differences between questions, requirements, and bugs; this unresolved reference should be fixed.
  5. [Section 5.3.1 vs Section 4.4.2/Figure 2] The interview section describes the newly added sub-themes as B.1.iv 'Multi-platform Interoperability' and B.4.iii 'Example Drift and Documentation Mismatch', but the final taxonomy in Figure 2 and Section 4.4.2 names them 'Local Development and Lightweight Setup Support' and 'Minimal Working Examples'; the paper should reconcile these names or explicitly describe the renaming.
  6. [Section 4.4.1] The A.1 sub-themes are initially described with their pre-rename names, 'System Support and Dependency Management' and 'Configuration and Parameter Usage', even though the paper later renames them to 'System Environment and Dependency Setup' and 'Runtime Configuration and Launch Parameters'; using final names consistently would improve clarity.
  7. [Section 4.2] The paper reports Fleiss' kappa values but does not state the number of reports or the sampling method used for the agreement computation, which limits reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the taxonomy is derived from external issue reports and interview feedback, and the few self-citations are not load-bearing.

full rationale

The core contribution is an empirically grounded taxonomy built bottom-up from issue reports collected from PyTorch, MindSpore, TensorFlow, and eight LLM toolkits, labeled by seven annotators, and refined through interviews with 11 users and 8 developers. The derivation chain is inductive rather than deductive: themes and sub-themes are clustered from labeled reports, and the resulting structure is then presented as a summary of that data. I found no equation or definition that equates an output with an input. The keyword filter is a data-selection step, not a fitted parameter that later reappears as a 'prediction'; the reported precision checks describe quality control rather than a circular reduction. The interview stage asks external participants to assess and extend a taxonomy shown to them; while this is a validation design that could in principle be self-confirming if participants merely echoed presented categories, the paper reports concrete additions and renamings proposed by interviewees, so the taxonomy is not defined in terms of the interview responses. The self-citations [14, 15] appear only in a list of prior work on DL-framework testing and are not used to justify the taxonomy's structure or the paper's empirical claims. Concerns such as the absence of a non-LLM baseline for the 'LLM-centric' qualifier, or the possible incompleteness of the keyword list, are legitimate correctness and generalization risks, but they are not circularity. The study is self-contained against external data sources and does not reduce its main findings to its own assumptions by construction.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The quantitative claims depend on representative issue sampling, reliable manual labeling, and generalizable interviewees; each is an assumption about the world rather than a result derived in the paper. No mathematical derivation and no new physical or computational entities are involved.

assumptions (5)
  • domain assumption Issue reports from GitHub and Gitee are a faithful proxy for real-world LLM framework challenges.
    Section 4.1 treats user-submitted issues as direct reflections of obstacles in building, training, and deploying LLMs; this conflates reporting behavior with actual challenge frequency.
  • domain assumption The keyword-based filtering plus spot-check precision preserves a representative sample.
    Section 4.1 uses 72 to 84 keywords per framework, an 80% precision threshold, and 92.9% spot-check precision to retain 38,742 reports; missed keywords or false exclusions would alter every reported distribution.
  • domain assumption Volunteer labeling with Fleiss kappa above 0.7 yields reliable ground truth.
    Section 4.2 relies on seven volunteers and reports kappa values of 0.717 to 0.757; the resulting categories are treated as objective labels rather than researcher constructs.
  • domain assumption The interview sample of 11 users and 8 developers represents the broader LLM community.
    Section 5.1 and Section 7 acknowledge that most users are students or early-career researchers; the priority rankings and interview findings inherit this limitation.
  • domain assumption The pilot study of 7,000 reports (18.07%) is sufficient to define the tag set.
    Section 4.2 uses the pilot to fix categories before full annotation; if the pilot mis-sampled, later labeling was anchored to an incomplete category set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis." pith.science (2026). https://pith.science/paper/VUGR6FM3

@misc{pith2026250613114,
  author       = {Pith},
  title        = {Pith review of: Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VUGR6FM3}},
  note         = {Machine review of arXiv:2506.13114}
}
read the original abstract

Large language models (LLMs) have driven significant progress across a wide range of real-world applications. Realizing such models requires substantial system-level support. Deep learning (DL) frameworks provide this foundation by enabling efficient model construction, distributed execution, and optimized deployment. The large parameter scale and extended execution cycles impose exacting demands on deep learning frameworks, particularly in terms of scalability, stability, and efficiency. Therefore, poor usability, limited functionality, and subtle bugs in DL frameworks may hinder development efficiency and cause severe failures or resource waste. However, a fundamental question has not been thoroughly investigated in previous studies, i.e., what challenges do DL frameworks face in supporting LLMs? To answer this question, we analyze issue reports from three major DL frameworks (i.e., MindSpore, PyTorch, and TensorFlow) and eight associated LLM toolkits such as Megatron. Based on a manual review of these reports, we construct a taxonomy that captures LLM-centric framework bugs, user requirements, and user questions. We then refine and enrich this taxonomy through interviews with 11 LLM users and eight DL framework developers. Based on the constructed taxonomy and findings summarized from interviews, our study further reveals key technical challenges and mismatches between LLM user needs and developer priorities.

Figures

Figures reproduced from arXiv: 2506.13114 by the authors.

Figure 1
Figure 1. Workflow of Our Empirical Study In Stage I, we investigate the following research question: “RQ1: What types of LLM-centric challenges are exposed among DL frameworks and LLM toolkits?” We aim to identify the common challenges that LLM users frequently encounter and report to framework communities, including common questions, requirements about new functionality, and bugs that threaten their LLM development. To answ… view at source ↗
Figure 2
Figure 2. LLM-centric Taxonomy of Challenges in DL Frameworks [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Statistics of Different Themes of our Taxonomy [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: A sample post in Model Behavior and Output Verification (A.1.ii) [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: A sample post in Model Behavior and Output Verification (A.3.iii) [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: A sample post in Deployment and Toolchain Optimization (B.1.ii) [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: A sample post in Training Configuration and Scalability Support (B.5.ii) [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: A sample post in Scheduling and Parameter Management Errors (C.2.i) [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: A sample post in Precision-Induced Accuracy Degradation (C.4.ii) [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: A sample post in Annotation and Metadata Inconsistencies (C.10.ii) [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: LLM-Centric Challenges Summarized from our Taxonomy [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: LLM development lifecycle 5.3 Result Analysis: Interviews with Users and Developers 5.3.1 Interview Results from Users We present interview findings from LLM users, focusing on their perspectives on the requirement, ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Art…
Figure 13
Figure 13. Figure 13: LLM-centric Challenges summarized from the Interview Results [PITH_FULL_IMAGE:figures/full_fig_p032_13.png]
Figure 14
Figure 14. Figure 14: Implications summarized from Findings in RQ1 and RQ2 [PITH_FULL_IMAGE:figures/full_fig_p033_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments

    cs.SE 2025-07 conditional novelty 6.0 of 10

    SetupBench, a 93-instance environment-bootstrap benchmark, finds coding agents succeed on only 34.4-62.4% of setup tasks, with database configuration and repo setup being the hardest.

Reference graph

Works this paper leans on

163 extracted references · 69 canonical work pages · cited by 1 Pith paper

  1. [1]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, ”arXiv preprint arXiv:2109.00859, 2021

    arXiv 2021

  2. [2]

    Chatgpt for good? on opportunities and challenges of large language models for education,

    E. Kasneci, K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier et al., “Chatgpt for good? on opportunities and challenges of large language models for education, ” Learning and individual differences , vol. 103, p. 102274, 2023

    2023

  3. [3]

    Large language models in healthcare and medical domain: A review,

    Z. A. Nazi and W. Peng, “Large language models in healthcare and medical domain: A review, ” inInformatics, vol. 11, no. 3. MDPI, 2024, p. 57

    2024

  4. [4]

    Revolutionizing finance with llms: An overview of applications and insights,

    H. Zhao, Z. Liu, Z. Wu, Y. Li, T. Yang, P. Shu, S. Xu, H. Dai, L. Zhao, G. Maiet al., “Revolutionizing finance with llms: An overview of applications and insights, ”arXiv preprint arXiv:2401.11641, 2024

    arXiv 2024

  5. [5]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report, ” 2023. [Online]. Available: https://openai.com/research/gpt-4

    2023

  6. [6]

    CVE-2024-3568: Remote Code Execution Vulnerability in Hugging Face Transformers,

    “CVE-2024-3568: Remote Code Execution Vulnerability in Hugging Face Transformers, ” https://avd.aliyun.com/detail? id=AVD-2024-3568, 2024, last accessed: January 28, 2025

    2024

  7. [7]

    Hugging face model hub statistics,

    “Hugging face model hub statistics, ” https://huggingface.co/models, 2023, as of 2023, over 90% of text-generation models on Hugging Face provide PyTorch weights

    2023

  8. [8]

    Tiktok owner sacks intern for sabotaging ai project,

    “Tiktok owner sacks intern for sabotaging ai project, ” https://www.bbc.com/news/articles/c7v62gg49zro, 2024, ac- cessed: 2024-10-21

    2024

Show all 163 references
  1. [9]

    Cradle: cross-backend validation to detect and localize bugs in deep learning libraries,

    H. V. Pham, T. Lutellier, W. Qi, and L. Tan, “Cradle: cross-backend validation to detect and localize bugs in deep learning libraries, ” in2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 2019, pp. 1027–1038

    2019

  2. [10]

    Deep learning library testing via effective model generation,

    Z. Wang, M. Yan, J. Chen, S. Liu, and D. Zhang, “Deep learning library testing via effective model generation, ” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020, pp. 788–799

    2020

  3. [11]

    Audee: Automated testing for deep learning frameworks,

    Q. Guo, X. Xie, Y. Li, X. Zhang, Y. Liu, X. Li, and C. Shen, “Audee: Automated testing for deep learning frameworks, ” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering , 2020, pp. 486–498

    2020

  4. [12]

    Comet: Coverage-guided model generation for deep learning library testing,

    M. Li, J. Cao, Y. Tian, T. O. Li, M. Wen*, and S.-C. Cheung*, “Comet: Coverage-guided model generation for deep learning library testing, ”ACM Transactions on Software Engineering and Methodology , 2023

    2023

  5. [13]

    Free lunch for testing: Fuzzing deep-learning libraries from open source,

    A. Wei, Y. Deng, C. Yang, and L. Zhang, “Free lunch for testing: Fuzzing deep-learning libraries from open source, ” 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) , pp. 995–1007, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:246016475

    2022

  6. [14]

    Devmut: Testing deep learning framework via developer expertise-based mutation,

    Y. Mu, J. Zhai, C. Fang, X. Chen, Z. Cao, P. Yang, Y. Zou, T. Zheng, and Z. Chen, “Devmut: Testing deep learning framework via developer expertise-based mutation, ” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 1533–1544

    2024

  7. [15]

    Improving deep learning framework testing with model-level metamorphic testing,

    Y. Mu, J. Zhai, C. Fang, X. Chen, Z. Cao, P. Yang, K. Zhao, A. Guo, and Z. Chen, “Improving deep learning framework testing with model-level metamorphic testing, ”Proceedings of the ACM on Software Engineering , vol. 2, no. ISSTA, pp. 2158–2180, 2025

    2025

  8. [16]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,

    Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models, ” inProceedings of the 32nd ACM SIGSOFT international symposium on software testing and analysis, 2023, pp. 423–435

    2023

  9. [17]

    Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt,

    Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, “Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt, ”arXiv preprint arXiv:2304.02014, 2023

    2023 arXiv

  10. [18]

    Toward understanding deep learning framework bugs,

    J. Chen, Y. Liang, Q. Shen, J. Jiang, and S. Li, “Toward understanding deep learning framework bugs, ”ACM Transactions on Software Engineering and Methodology , vol. 32, no. 6, pp. 1–31, 2023

    2023

  11. [19]

    The symptoms, causes, and repairs of bugs inside a deep learning library,

    L. Jia, H. Zhong, X. Wang, L. Huang, and X. Lu, “The symptoms, causes, and repairs of bugs inside a deep learning library, ”Journal of Systems and Software , vol. 177, p. 110935, 2021. ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 1. Publication date: 2025. Understa...

    2021

  12. [20]

    Silent bugs in deep learning frameworks: an empirical study of keras and tensorflow,

    F. Tambon, A. Nikanjam, L. An, F. Khomh, and G. Antoniol, “Silent bugs in deep learning frameworks: an empirical study of keras and tensorflow, ”Empirical Software Engineering, vol. 29, no. 1, p. 10, 2024

    2024

  13. [21]

    Investigating and detecting silent bugs in pytorch programs,

    S. Hong, H. Sun, X. Gao, and S. H. Tan, “Investigating and detecting silent bugs in pytorch programs, ” in2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2024, pp. 272–283

    2024

  14. [22]

    An empirical study on program failures of deep learning jobs,

    R. Zhang, W. Xiao, H. Zhang, Y. Liu, H. Lin, and M. Yang, “An empirical study on program failures of deep learning jobs, ” inProceedings of the ACM/IEEE 42nd international conference on software engineering , 2020, pp. 1159–1170

    2020

  15. [23]

    What do programmers discuss about deep learning frameworks,

    J. Han, E. Shihab, Z. Wan, S. Deng, and X. Xia, “What do programmers discuss about deep learning frameworks, ” Empirical Software Engineering, vol. 25, pp. 2694–2747, 2020

    2020

  16. [24]

    Automatic unit test generation for machine learning libraries: How far are we?

    S. Wang, N. Shrestha, A. K. Subburaman, J. Wang, and M. Wei, “Automatic unit test generation for machine learning libraries: How far are we?” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 1548–1560

    2021

  17. [25]

    Evaluating spectrum-based fault localization on deep learning libraries,

    M. Yan, J. Chen, T. Jiang, J. Jiang, and Z. Wang, “Evaluating spectrum-based fault localization on deep learning libraries, ”IEEE Transactions on Software Engineering , 2025

    2025

  18. [26]

    N-gram statistics for natural language understanding and text processing,

    C. Y. Suen, “N-gram statistics for natural language understanding and text processing, ”IEEE transactions on pattern analysis and machine intelligence , no. 2, pp. 164–172, 1979

    1979

  19. [27]

    A tutorial on hidden markov models and selected applications in speech recognition,

    L. R. Rabiner, “A tutorial on hidden markov models and selected applications in speech recognition, ”Proceedings of the IEEE, vol. 77, no. 2, pp. 257–286, 1989

    1989

  20. [28]

    A critical review of recurrent neural networks for sequence learning,

    Z. C. Lipton, “A critical review of recurrent neural networks for sequence learning, ”arXiv Preprint, CoRR, abs/1506.00019, 2015

    2015 arXiv

  21. [29]

    Learning to forget: Continual prediction with lstm,

    F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with lstm, ”Neural computation, vol. 12, no. 10, pp. 2451–2471, 2000

    2000

  22. [30]

    Attention is all you need,

    A. Vaswani, “Attention is all you need, ”Advances in Neural Information Processing Systems , 2017

    2017

  23. [31]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Donget al., “A survey of large language models, ”arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  24. [32]

    Improving language understanding by generative pre-training,

    A. Radford, “Improving language understanding by generative pre-training, ” 2018

    2018

  25. [33]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners, ”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  26. [34]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners, ”Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

    1901

  27. [35]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding, ” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  28. [36]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer, ”Journal of machine learning research , vol. 21, no. 140, pp. 1–67, 2020

    2020

  29. [37]

    Text generation: A systematic literature review of tasks, evaluation, and challenges,

    J. Becker, J. P. Wahle, B. Gipp, and T. Ruas, “Text generation: A systematic literature review of tasks, evaluation, and challenges, ”arXiv preprint arXiv:2405.15604, 2024

    2024 arXiv

  30. [38]

    Xlnet: Generalized autoregressive pretraining for language understanding,

    Z. Yang, Z. Dai, Y. Yang, M. Schuster, C. Quirk, X. He, H. Liu, J. Hu, L. Deng, and A. Smola, “Xlnet: Generalized autoregressive pretraining for language understanding, ” inProceedings of NeurIPS 2019. NeurIPS, 2019, pp. 5754–5764

    2019

  31. [39]

    Reading wikipedia to answer open-domain questions,

    D. Chen, A. Fisch, J. Weston, and A. Bordes, “Reading wikipedia to answer open-domain questions, ” inProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, 2017, pp. 1870–1879

    2017

  32. [40]

    Convolutional neural networks for sentence classification,

    A. Rakhlin, “Convolutional neural networks for sentence classification, ” GitHub, vol. 6, p. 25, 2016

    2016

  33. [41]

    Google’s neural machine translation system: Bridging the gap between human and machine translation,

    Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al., “Google’s neural machine translation system: Bridging the gap between human and machine translation, ” arXiv preprint arXiv:1609.08144, 2016

    2016 arXiv

  34. [42]

    Textrank: Bringing order into text,

    R. Mihalcea and P. Tarau, “Textrank: Bringing order into text, ” in Proceedings of the 2004 conference on empirical methods in natural language processing , 2004, pp. 404–411

    2004

  35. [43]

    Get to the point: Summarization with pointer-generator networks,

    A. See, P. J. Liu, and C. D. Manning, “Get to the point: Summarization with pointer-generator networks, ”arXiv preprint arXiv:1704.04368, 2017

    2017 arXiv

  36. [44]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition, ” inProceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  37. [45]

    Learning both weights and connections for efficient neural networks,

    S. Han, H. Mao, and W. J. Dally, “Learning both weights and connections for efficient neural networks, ” inProceedings of the 28th International Conference on Neural Information Processing Systems (NIPS) . MIT Press, 2015, pp. 1135–1143

    2015

  38. [46]

    Tensorflow model optimization toolkit,

    T. M. O. Team, “Tensorflow model optimization toolkit, ” Google, 2019, accessed: 2025-01-28. [Online]. Available: https://www.tensorflow.org/model_optimization ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 1. Publication date: 2025. 1:44 Mu, et al

    2019

  39. [47]

    Pruning convolutional neural networks for resource efficient inference,

    P. Molchanov, A. Ashukha, and D. Vetrov, “Pruning convolutional neural networks for resource efficient inference, ” in Proceedings of the 30th Conference on Neural Information Processing Systems (NeurIPS) . MIT Press, 2016, pp. 1382–1390. [Online]. Available: https://arxiv.org...

    2016 arXiv

  40. [48]

    Knowledge distillation: A survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey, ”International Journal of Computer Vision , vol. 129, no. 6, pp. 1789–1819, 2021

    2021

  41. [49]

    Mixed precision training,

    P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh et al., “Mixed precision training, ”arXiv preprint arXiv:1710.03740, 2017

    2017 arXiv

  42. [50]

    Optimization of deep learning inference on edge devices,

    E. Kristiani, C.-T. Yang, and K. L. P. Nguyen, “Optimization of deep learning inference on edge devices, ” in 2020 International Conference on Pervasive Artificial Intelligence (ICPAI) . IEEE, 2020, pp. 264–267

    2020

  43. [51]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation,

    K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation, ” inProceedings of the 2014 Conference on Empirical Methods in Natural Language Process...

    2014

  44. [52]

    Neural machine translation by jointly learning to align and translate,

    D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate, ” in International Conference on Learning Representations (ICLR) , 2015

    2015

  45. [53]

    Glove: Global vectors for word representation,

    J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation, ” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, 2014, pp. 1532–1543

    2014

  46. [54]

    Enriching word vectors with subword information,

    P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information, ”Transactions of the Association for Computational Linguistics , vol. 5, pp. 135–146, 2017

    2017

  47. [55]

    K. D. Cooper and L. Torczon, Engineering a compiler. Morgan Kaufmann, 2022

    2022

  48. [56]

    V. A. Alfred, S. L. Monica, and D. U. Jeffrey, Compilers principles, techniques & tools . pearson Education, 2007

    2007

  49. [57]

    Beyond data and model parallelism for deep neural networks

    Z. Jia, M. Zaharia, and A. Aiken, “Beyond data and model parallelism for deep neural networks. ” Proceedings of Machine Learning and Systems , vol. 1, pp. 1–13, 2019

    2019

  50. [58]

    Large scale distributed deep networks,

    J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yanget al., “Large scale distributed deep networks, ”Advances in neural information processing systems , vol. 25, 2012

    2012

  51. [59]

    Parallel and distributed deep learning,

    V. Hegde and S. Usmani, “Parallel and distributed deep learning, ”May, vol. 31, pp. 1–8, 2016

    2016

  52. [60]

    Scaling distributed machine learning with the parameter server,

    M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su, “Scaling distributed machine learning with the parameter server, ” in11th USENIX Symposium on operating systems design and implementation (OSDI 14), 2014, pp. 583–598

    2014

  53. [61]

    Memory-efficient backpropagation through time,

    A. Gruslys, R. Munos, I. Danihelka, M. Lanctot, and A. Graves, “Memory-efficient backpropagation through time, ” Advances in neural information processing systems , vol. 29, 2016

    2016

  54. [62]

    Mindspore lite: A lightweight ai inference framework for edge devices,

    M. Team, “Mindspore lite: A lightweight ai inference framework for edge devices, ” Huawei, White Paper, 2020, accessed: 2025-01-28. [Online]. Available: https://www.mindspore.cn/

    2020

  55. [63]

    Deepdriving: Learning affordance for direct perception in autonomous driving,

    C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: Learning affordance for direct perception in autonomous driving, ” inProceedings of the IEEE international conference on computer vision , 2015, pp. 2722–2730

    2015

  56. [64]

    Neural network libraries: A deep learning framework designed from engineers’ perspectives,

    T. Narihira, J. Alonsogarcia, F. Cardinaux, A. Hayakawa, M. Ishii, K. Iwaki, T. Kemp, Y. Kobayashi, L. Mauch, A. Nakamura et al., “Neural network libraries: A deep learning framework designed from engineers’ perspectives, ” arXiv preprint arXiv:2102.06725, 2021

    2021 arXiv

  57. [65]

    A comprehensive study on challenges in deploying deep learning based software,

    Z. Chen, Y. Cao, Y. Liu, H. Wang, T. Xie, and X. Liu, “A comprehensive study on challenges in deploying deep learning based software, ” inProceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineer...

    2020

  58. [66]

    Stable architectures for deep neural networks,

    E. Haber and L. Ruthotto, “Stable architectures for deep neural networks, ”Inverse problems, vol. 34, no. 1, p. 014004, 2017

    2017

  59. [67]

    Ai slop: Github devs swamped by fake bug reports made by artificial morons,

    “Ai slop: Github devs swamped by fake bug reports made by artificial morons, ” https://www.theregister.com/2024/12/ 10/ai_slop_bug_reports/, 2024, accessed: 2025-05-22

    2024

  60. [68]

    Machine learning and deep learning frameworks and libraries for large-scale data mining: a survey,

    G. Nguyen, S. Dlugolinsky, M. Bobák, V. Tran, Á. López García, I. Heredia, P. Malík, and L. Hluch`y, “Machine learning and deep learning frameworks and libraries for large-scale data mining: a survey, ”Artificial Intelligence Review, vol. 52, pp. 77–124, 2019

    2019

  61. [69]

    Pytorch,

    “Pytorch, ” https://pytorch.org/, accessed: 2025-08-15

    2025

  62. [70]

    Tensorflow,

    “Tensorflow, ” https://www.tensorflow.org/?hl=zh-cn, accessed: 2025-08-15

    2025

  63. [71]

    Mindspore,

    “Mindspore, ” https://www.mindspore.cn/, accessed: 2025-08-15

    2025

  64. [72]

    Deepspeed,

    “Deepspeed, ” https://www.deepspeed.ai/, accessed: 2025-08-15

    2025

  65. [73]

    Megatron-lm,

    “Megatron-lm, ” https://github.com/NVIDIA/Megatron-LM, accessed: 2025-08-15

    2025

  66. [74]

    “vllm, ” https://docs.vllm.ai/en/latest/, accessed: 2025-08-15

    2025

  67. [75]

    Tensorrt-llm,

    “Tensorrt-llm, ” https://nvidia.github.io/TensorRT-LLM/latest/index.html, accessed: 2025-08-15

    2025

  68. [76]

    Colossai,

    “Colossai, ” https://colossalai.org/, accessed: 2025-08-15. ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 1. Publication date: 2025. Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis 1:45

    2025

  69. [77]

    Mindspeed,

    “Mindspeed, ” https://gitee.com/ascend/MindSpeed, accessed: 2025-08-15

    2025

  70. [78]

    Mindnlp,

    “Mindnlp, ” https://gitee.com/mindspore-lab/mindnlp, accessed: 2025-08-15

    2025

  71. [79]

    Mindformers,

    “Mindformers, ” https://gitee.com/mindspore/mindformers, accessed: 2025-08-15

    2025

  72. [80]

    Data avaiable,

    “Data avaiable, ” Accessed: 2025, https://github.com/Anony-coding/LLMChallengeEmpirical

    2025

  73. [81]

    Predicting defects for eclipse,

    T. Zimmermann, N. Nagappan, H. Gall, E. Giger, and B. Murphy, “Predicting defects for eclipse, ” inProceedings of the 3rd international workshop on Predictor models in software engineering . IEEE Computer Society, 2007, pp. 9–15

    2007

  74. [82]

    Automatic topic naming to support cross-project analysis of software maintenance activities,

    A. Hindle, M. W. Godfrey, and R. C. Holt, “Automatic topic naming to support cross-project analysis of software maintenance activities, ” inProceedings of the 2008 International Conference on Software Maintenance . IEEE, 2008, pp. 3–12

    2008

  75. [83]

    Failures and fixes: A study of software system incident response,

    J. Sillito and E. Kutomi, “Failures and fixes: A study of software system incident response, ” in2020 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2020, pp. 185–195

    2020

  76. [84]

    Bug taxonomies: Use them to generate better tests,

    G. Vijayaraghavan and C. Kaner, “Bug taxonomies: Use them to generate better tests, ”Star East, vol. 2003, pp. 1–40, 2003

    2003

  77. [85]

    https://github.com/vllm-project/vllm/issues/129

  78. [86]

    https://github.com/mindspore-ai/mindspore/issues/253

  79. [87]

    https://github.com/NVIDIA/Megatron-LM/issues/944

  80. [88]

    https://github.com/deepspeedai/DeepSpeed/issues/4725

  81. [89]

    https://github.com/deepspeedai/DeepSpeed/issues/6793

  82. [90]

    https://github.com/pytorch/pytorch/issues/22107

  83. [91]

    https://github.com/tensorflow/tensorflow/issues/46111

  84. [92]

    https://github.com/NVIDIA/Megatron-LM/issues/909

  85. [93]

    https://github.com/NVIDIA/TensorRT-LLM/issues/3253

  86. [94]

    https://github.com/pytorch/pytorch/issues/83230

  87. [95]

    https://github.com/pytorch/pytorch/issues/116966

  88. [96]

    https://github.com/pytorch/pytorch/issues/126719

  89. [97]

    https://github.com/intel/intel-extension-for-tensorflow/issues/81

  90. [98]

    https://github.com/tensorflow/tensorflow/issues/67291

  91. [99]

    https://github.com/pytorch/pytorch/pull/98327

  92. [100]

    https://github.com/huggingface/accelerate/issues/3522

  93. [101]

    https://github.com/huggingface/datasets/issues/6623

  94. [102]

    https://github.com/vllm-project/vllm/issues/231

  95. [103]

    https://github.com/huggingface/transformers/issues/25147

  96. [104]

    https://github.com/huggingface/diffusers/issues/8907

  97. [105]

    https://github.com/huggingface/transformers/issues/34610

  98. [106]

    https://github.com/Lightning-AI/pytorch-lightning/pull/9722

  99. [107]

    https://github.com/deepspeedai/DeepSpeed/issues/5498

  100. [108]

    https://github.com/NVIDIA/TensorRT-LLM/discussions/52

  101. [109]

    https://github.com/huggingface/transformers/issues/9742

  102. [110]

    https://github.com/huggingface/peft/issues/638

  103. [111]

    https://github.com/pytorch/pytorch/issues/122924

  104. [112]

    https://github.com/tensorflow/tensorflow/issues/63362

  105. [113]

    https://github.com/microsoft/DeepSpeed/issues/6906

  106. [114]

    https://github.com/huggingface/transformers/issues/24558

  107. [115]

    https://github.com/deepspeedai/DeepSpeed/issues/4036

  108. [116]

    https://github.com/deepspeedai/DeepSpeed/issues/5636

  109. [117]

    https://github.com/pytorch/pytorch/issues/113245

  110. [118]

    https://github.com/tensorflow/tensorflow/issues/41539

  111. [119]

    https://github.com/pytorch/pytorch/issues/50803

  112. [120]

    https://github.com/pytorch/pytorch/issues/40497

  113. [121]

    https://github.com/tensorflow/tensorflow/issues/43719

  114. [122]

    https://github.com/huggingface/diffusers/issues/5004

  115. [123]

    https://github.com/pytorch/pytorch/issues/88621

  116. [124]

    https://github.com/huggingface/transformers/issues/26498

  117. [125]

    https://github.com/pytorch/pytorch/issues/104026

  118. [126]

    https://github.com/microsoft/DeepSpeed/issues/5898

  119. [127]

    https://github.com/volcengine/verl/issues/839

  120. [128]

    ACM Trans

    https://github.com/pytorch/pytorch/issues/75147. ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 1. Publication date: 2025. 1:46 Mu, et al

    2025

  121. [129]

    https://github.com/oobabooga/text-generation-webui/issues/3851

  122. [130]

    https://github.com/huggingface/transformers/issues/26952

  123. [131]

    https://github.com/microsoft/DeepSpeed/issues/6836

  124. [132]

    https://github.com/NVIDIA/Megatron-LM/issues/1154

  125. [133]

    https://github.com/huggingface/transformers/issues/35270

  126. [134]

    https://github.com/huggingface/transformers/issues/34080

  127. [135]

    https://github.com/deepspeedai/DeepSpeed/issues/5631

  128. [136]

    https://github.com/microsoft/DeepSpeed/issues/5484

  129. [137]

    https://github.com/huggingface/transformers/issues/33645

  130. [138]

    https://github.com/pytorch/pytorch/issues/89254

  131. [139]

    https://github.com/huggingface/transformers/issues/30153

  132. [140]

    https://github.com/NVIDIA/Megatron-LM/issues/855

  133. [141]

    https://github.com/huggingface/transformers/issues/23790

  134. [142]

    https://github.com/huggingface/transformers/issues/29942

  135. [143]

    https://github.com/NVIDIA/Megatron-LM/issues/1134

  136. [144]

    https://github.com/pytorch/pytorch/issues/38115

  137. [145]

    https://github.com/mlflow/mlflow/issues/12926

  138. [146]

    https://github.com/n8n-io/n8n/issues/15692

  139. [147]

    https://github.com/BerriAI/litellm/issues/8029

  140. [148]

    https://github.com/crewAIInc/crewAI/issues/2862

  141. [149]

    https://github.com/huggingface/transformers/issues/27469

  142. [150]

    https://github.com/huggingface/transformers/issues/32203

  143. [151]

    https://github.com/NVIDIA/Megatron-LM/issues/1224

  144. [152]

    Nvivo qualitative software,

    Lumivero, “Nvivo qualitative software, ” 2024. [Online]. Available: https://lumivero.com/products/nvivo/

    2024

  145. [153]

    Pytorch,

    “Pytorch, ” Accessed: 2019, https://pytorch.org/

    2019

  146. [154]

    Theano: A Python framework for fast computation of mathematical expressions,

    T. T. D. Team, “Theano: A Python framework for fast computation of mathematical expressions, ” 2016, accessed: 2025-01-29. [Online]. Available: http://deeplearning.net/software/theano/

    2016

  147. [155]

    Evosuite: automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software, ” inProceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering , 2011, pp. 416–419

    2011

  148. [156]

    Randoop: feedback-directed random testing for java,

    C. Pacheco and M. D. Ernst, “Randoop: feedback-directed random testing for java, ” inCompanion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion , 2007, pp. 815–816

    2007

  149. [157]

    Feedback-directed random test generation,

    C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, “Feedback-directed random test generation, ” in29th International Conference on Software Engineering (ICSE’07) . IEEE, 2007, pp. 75–84

    2007

  150. [158]

    Fault diagnosis of digital systems-a review,

    R. Bennetts and D. Lewin, “Fault diagnosis of digital systems-a review, ”Computer, vol. 4, no. 4, pp. 12–20, 1971

    1971

  151. [159]

    Hints on test data selection: Help for the practicing programmer,

    R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Hints on test data selection: Help for the practicing programmer, ” Computer, vol. 11, pp. 34–41, 1978. [Online]. Available: https://api.semanticscholar.org/CorpusID:342522

    1978

  152. [160]

    An empirical study of fault triggers in deep learning frameworks,

    X. Du, Y. Sui, Z. Liu, and J. Ai, “An empirical study of fault triggers in deep learning frameworks, ”IEEE Transactions on Dependable and Secure Computing , vol. 20, no. 4, pp. 2696–2712, 2022

    2022

  153. [161]

    An empirical study on tensorflow program bugs,

    Y. Zhang, Y. Chen, S.-C. Cheung, Y. Xiong, and L. Zhang, “An empirical study on tensorflow program bugs, ” in Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis , 2018, pp. 129–140

    2018

  154. [162]

    Keras: The python deep learning library,

    “Keras: The python deep learning library, ” https://github.com/fchollet/keras, 2015, accessed: 2025-01-28

    2015

  155. [163]

    An empirical study on performance bugs in deep learning frameworks,

    T. Makkouk, D. J. Kim, and T.-H. P. Chen, “An empirical study on performance bugs in deep learning frameworks, ” in 2022 ieee international conference on software maintenance and evolution (icsme) . IEEE, 2022, pp. 35–46. Received 20 February 2007; revised 12 March 2009; accep...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.