Pith. sign in

REVIEW 3 cited by

Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2112.13314 v2 pith:WPRMD24M submitted 2021-12-26 cs.SE cs.LG

Florian Tambon , Amin Nikanjam , Le An , Foutse Khomh , Giuliano Antoniol This is my paper · ORCID
classification cs.SEcs.LG
keywords bugssilentimpactprogramsframeworksissuestheyusers
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Deep Learning (DL) frameworks are now widely used, simplifying the creation of complex models as well as their integration to various applications even to non DL experts. However, like any other programs, they are prone to bugs. This paper deals with the subcategory of bugs named silent bugs: they lead to wrong behavior but they do not cause system crashes or hangs, nor show an error message to the user. Such bugs are even more dangerous in DL applications and frameworks due to the "black-box" and stochastic nature of the systems (the end user can not understand how the model makes decisions). This paper presents the first empirical study of Keras and TensorFlow silent bugs, and their impact on users' programs. We extracted closed issues related to Keras from the TensorFlow GitHub repository. Out of the 1,168 issues that we gathered, 77 were reproducible silent bugs affecting users' programs. We categorized the bugs based on the effects on the users' programs and the components where the issues occurred, using information from the issue reports. We then derived a threat level for each of the issues, based on the impact they had on the users' programs. To assess the relevance of identified categories and the impact scale, we conducted an online survey with 103 DL developers. The participants generally agreed with the significant impact of silent bugs in DL libraries and acknowledged our findings (i.e., categories of silent bugs and the proposed impact scale). Finally, leveraging our analysis, we provide a set of guidelines to facilitate safeguarding against such bugs in DL frameworks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Verifying Computational Graphs in Production-Grade Distributed Machine Learning Frameworks

    cs.LG 2025-09 conditional novelty 7.0 of 10

    Scalify verifies semantic equivalence of baseline and distributed ML computational graphs using equality saturation and relational reasoning, finding real silent errors in production frameworks.

  2. Deep Learning Framework Testing via Model Mutation: How Far Are We?

    cs.SE 2025-06 conditional novelty 6.0 of 10

    Existing mutation-based deep learning framework testing tools largely miss high-priority defects such as performance and memory issues, and targeted optimizations to mutation settings can uncover new developer-confirmed bugs.

  3. Fault Localization in Deep Learning-based Software: A System-level Approach

    cs.SE 2024-11 conditional novelty 6.0 of 10

    FL4Deep localizes faults across the full deep learning pipeline using a knowledge graph built from static and dynamic code information, outperforming four prior tools on three of six fault types.

Pith tools