Pith. sign in

REVIEW 4 major objections 5 minor 64 references

DaiFu: In-Situ Crash Recovery for Deep Learning Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DaiFu claims that deep learning crashes can be recovered in seconds by catching the exception in the live program, applying a code or configuration patch, and resuming from the failing statement—without restart or checkpoint replay.

desk verdict Real engineering contribution: DaiFu's cell-based DSU for active training functions is novel and mostly works, but the crash coverage claim is broader than the mechanism supports—loop-header exceptions like DataLoader failures fall outside it, and the evaluation numbers need tightening. read the letter →

arxiv 2507.01628 v1 pith:U7R7Z7RI submitted 2025-07-02 cs.SE

Zilong He , Pengfei Chen , Hongyu Zhang , Xiaoyun Li , Guangba Yu , Hongyang Chen , Zibin Zheng This is my paper · ORCID
classification cs.SE
keywords deeplearningsystemscrashrecoveryin-situdynamicsoftwareupdatingexceptioninterceptioncheckpoint-retryfunctiondecompositionprogramvaccination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DaiFu proposes a third way to recover from crashes in deep learning systems, alongside restart-from-scratch and checkpoint-retry: intercept the crash while the program is still alive, let the developer patch code or configuration in the running program, and resume from the exact statement that failed. The paper's claim is that for the many crashes that manifest as an exception without corrupting program state, this in-situ recovery restores a crashed training run in 0.28 to 3.97 seconds across ResNet50, ViT, Swin, GPT-2, and LLaMA-7B runs—over 1327x faster than the baselines, including the open-source checkpointing method CheckFreq—with runtime overhead under 0.40%. If this holds, the dominant cost of routine DL development crashes, which is minutes to hours of re-execution and GPU waste, becomes seconds, and crash handling becomes an interactive debugging step rather than a long wait. Adopting it requires importing the library and decorating one entry function with a single line.

What carries the argument

The load-bearing mechanism is program vaccination: an ahead-of-time source transformation that decomposes the vaccinated entry function into cells and then reconstructs it. A cell is a function that either contains no loops or contains loops whose bodies are calls to other cells wrapped in crash barriers; a crash barrier is an exception handler that prevents an exception from escaping a loop and hands control to the runtime engine. This decomposition makes unfinished-procedure synthesis tractable because the restart location never falls in a loop with a control-flow back edge to a skipped statement. Function reconstruction preserves semantics by redirecting variable lookups into a shared context dictionary and translating break, continue, and return into return indicators that the caller interprets. At recovery time the runtime offers three update interfaces—pass to retry the failing statement, surgery to replace code with new code, and action to execute a snippet in the live context—and the context manager re-executes only the synthesized unfinished remainder.

What would settle it

Take a crash whose failing statement follows a mutating operation that re-execution cannot undo—for example a branch that deletes a file or clears a model state and then raises—and run DaiFu on it; if the recovered program either cannot reach completion or yields outputs different from a clean restart, the idempotence premise fails. The paper's own failed case, API Misuse [3], already exhibits exactly this pattern.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that crash recovery for deep learning systems does not have to be a restart: most crashes arrive as Python exceptions while the program's context—its executed code locations and live variables—is still intact, so recovery can be done by updating the live program instead of replaying from a checkpoint. To make this work, DaiFu vaccinates the long-running entry function ahead of time, decomposing it into cells so each loop body becomes a call to a separate cell wrapped in a crash barrier, and reconstructing the program so all cells share one namespace and break, continue, and return keep their meaning. When a crash is intercepted, the developer uses pass, surgery, or action to retry the failing statement, replace code, or execute new code in the program's context; the context manager then synthesizes the unfinished procedure from the restart location and resumes execution. The paper positions this as the first use of dynamic software updating for in-situ crash recovery of DL systems, and reports that it recovers 31 of 32 benchmark crash cases across seven scenarios while passing correctness tests that compare recovered outcomes with restart-from-scratch outcomes.

Load-bearing premise

The load-bearing premise is that the crash arrives as an interceptable Python exception while the program's context is still intact, and that the slice of code between the crash point and the re-execution start has no irreversible side effects, so re-running the patched code from that point reproduces a correct state.

Editorial extensions

If this is right

  • Restore time for the common exception-based crash class drops from minutes or hours to a few seconds, so developers can treat a crash as an interactive debugging event instead of a long re-run.
  • Because normal-execution overhead stays under 0.40% and integration is two lines, in-situ recovery can be left permanently enabled during development, avoiding the checkpoint-frequency tuning that prior methods require.
  • Distributed training is supported: fixes applied to one crashing process are recorded, broadcast, and replayed on the other crashed processes, so all workers rejoin the synchronization barrier and continue together.
  • Recovery preserves training semantics: in correctness tests the outcomes of DaiFu-recovered runs are not statistically different from restart-from-scratch runs, so the dynamic update does not silently change model behavior.
  • Crashes that corrupt state irreversibly, are terminated externally (e.g., kill -9), or whose pre-crash segment is non-idempotent remain outside DaiFu's scope, so the method is complementary to checkpoint-restart rather than a full replacement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence is that crashes can be classified by recoverability—interceptable or not, state-preserving or not—and future tools could inspect the crash site and route each case to in-situ recovery, checkpoint replay, or restart automatically.
  • The same vaccination machinery could support live experimentation beyond crash recovery, letting developers change hyperparameters, data pipelines, or model code mid-run without restarting.
  • A testable extension is to log undo information for mutating operations (file writes, variable-scope creation, in-place tensor changes) so non-idempotent pre-crash segments can be rolled back; this would target exactly the failed API Misuse [3] case.
  • A further experiment could decompose the 1327x speedup into what comes from skipping data loading and warm-up versus skipping repeated training iterations, since DaiFu eliminates both.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents DaiFu, a source-to-source transformation framework that 'vaccinates' Python DL programs by decomposing long-running entry functions into cells, wrapping cell calls in crash barriers, and redirecting variable access to a shared namespace. On a crash, DaiFu catches the exception, lets the developer update code or state via pass/surgery/action interfaces, and synthesizes an 'unfinished procedure' to resume execution in place. The evaluation uses 32 crash cases across 7 categories and reports restore times of 0.28-3.97s, a speedup of 1327x (with the abstract also stating 1372x) relative to baselines, overhead below 0.40%, and 31/32 recovery success. Limitations acknowledged in §6.3 include uninterceptable crashes (kill -9) and non-idempotent faults.

Significance. If fully validated, the in-situ recovery paradigm is a meaningful alternative to checkpoint-retry for a class of exception-based crashes. The key technical challenge—enabling DSU of active long-running functions—is addressed with a decomposition that appears plausible, and the reported overhead is very low. The benchmark construction and the artifact link are strengths, as is the honest failure analysis of one API Misuse case. However, the supported crash class is narrower than the paper claims: exceptions raised by loop/iterator headers are not recoverable by the described mechanism, and the headline speedup is not internally consistent. These issues must be resolved before the general claims can be accepted.

major comments (4)
  1. [§3.2, Algorithm 1, Table 2] Algorithm 1 (lines 6-12) places a crash barrier only around calls to decomposed cells; the for/while headers themselves remain in the parent cell outside the try. Consequently, if an exception is raised while evaluating the loop header (e.g., a DataLoader's __next__ during 'for batch in data:'), the exception propagates to the enclosing cell's barrier, and the restart location is the loop header. Table 2 explicitly classifies this as illegal 'loop structure 2' because the back edge points to a skipped statement. The paper gives no mechanism for preserving and resuming partially consumed iterators. Therefore a substantial class of interceptable, context-preserving crashes (transient DataLoader failures, streaming input timeouts) is outside DaiFu's supported mechanism, yet the abstract and §3.3 describe recovery from intercepted exceptions generally. §6.3 lists only kill -9 and non-idempotent faults as limitations and does not mention this gap, so the claimed coverage is overstated.
  2. [§5.2, Table 6, Abstract] The reported speedup is internally inconsistent. The abstract in the full text says '1327×' while the displayed abstract block at the top of the submission says '1372×'. More importantly, Table 6 does not reproduce either value: the CheckFreq-to-DaiFu restore-time ratios are 1571× (ResNet50), 1246× (ViT-L/16), 1273× (Swin-B), and 1210× (LLaMA-7B), whose average is about 1325×; the Restart-to-DaiFu ratios are much larger for every model, and GPT2 has no CheckFreq value due to the RuntimeError noted in the table. The paper should state exactly which models and baselines are included in the speedup computation and report the per-comparison ratios, including how the GPT2 row is handled.
  3. [§5.4, Table 8] The applicability experiment uses 'detailed recovery actions generated based on the patch associated with each collected crash case' (§5.4). This pre-arranges the fix for every crash, so the 31/32 success rate measures whether DaiFu can apply a known patch in situ, not whether a developer can diagnose and recover an unseen crash. Since the paper's motivation (§2.2) emphasizes developer interaction and debugging support, the benchmark should include at least some cases where the recovery action is not derived from the ground-truth patch, or the claim should be restricted to 'recovery actions are applicable when the fix is known'. This threat to external validity is not discussed in §6.4.
  4. [§5.5] RQ4 states that statistical significance is tested with a generalized linear model following prior work, but no test statistic, p-value, or confidence interval is reported; the sentence 'DaiFu passes all the correctness tests' is therefore not verifiable from the manuscript. Please report the model results, the number of replicates per case, and the exact outcome metric (e.g., accuracy difference) for each benchmark system.
minor comments (5)
  1. [§5.1] The phrase 'the number of training epoches' contains a typo; 'epoches' should be 'epochs'.
  2. [Figure 6] The axis label 'The Training Iteration that the Crash Occurs' appears to label the x-axis, but the caption says the comparison is shown 'under different scales'; please label the axes clearly and explain the scaling.
  3. [§5.2] The definition of restore time as ending at 'the end of its original crashing iteration' should be stated earlier and justified, since it affects the comparison with CheckFreq, whose checkpointing may allow resume before the iteration end.
  4. [§6.1 and §6.2] Section 6.1 says DaiFu 'does not interfere with the libraries used', but §6.2 immediately lists incompatibilities with PyTorch JIT and TensorFlow static graph mode; please reconcile the wording to reflect the conditional compatibility.
  5. [References [3] and [4]] The text uses references [3] and [4] to name crash cases (e.g., 'API Misuse [4]' and 'API Misuse [3]'); if these are benchmark dataset entries rather than bibliographic citations, please clarify the notation in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the recovery mechanism is evaluated by direct measurement, and the notable caveats are coverage/validity issues, not circular derivation.

full rationale

The paper contains no derivation chain that reduces to its own inputs. The central quantities (restore time, overhead) are direct measurements in Table 6; the reported speedup is simply the quotient of measured baseline and DaiFu restore times, not a fitted prediction. The benchmark in Section 5.4 uses ground-truth patches to construct recovery actions ('Detailed recovery actions are generated based on the patch associated with each collected crash case'), but the success predicate is whether the vaccinated program reaches its end after in-situ resume; that outcome is not forced by the patch, as the failed non-idempotent case demonstrates, so the evaluation has independent content. Self-citations ([22], [39], [62]) appear only in background or motivation passages and are not load-bearing for the framework's design. The main caveats are external-validity and coverage concerns rather than circularity: recovery actions presume that a correct fix is already known, and the cell decomposition in Algorithm 1 leaves a possible gap for exceptions raised by loop headers (e.g., iterator failures), because crash barriers wrap only cell calls. These are correctness/coverage risks, not cases where a prediction or first-principles result is equivalent to its own input by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numerical parameters are fitted to data in this paper; the experimental configuration (training epochs, iteration counts) is chosen for tractability and does not affect the qualitative claim. The main assumptions are that crashes manifest as interceptable exceptions and that the crashed region can be re-executed safely, both stated by the authors. No new physical or conceptual entities are postulated beyond the software abstractions of the framework itself.

assumptions (3)
  • domain assumption A DL system crash that needs recovery will manifest as an interceptable Python exception rather than a hard kill or native crash.
    Section 2.2 states many crashes manifest as exceptions thrown across the function call stack and can be intercepted. Section 6.3 acknowledges crashes like kill -9 cannot be intercepted.
  • domain assumption The AST-based function decomposition and reconstruction preserves the original program semantics.
    The transformation rewrites variables to a shared namespace and converts break/continue/return to indicator returns; correctness is tested only on the benchmark subset in RQ4, not formally proven.
  • domain assumption The unfinished procedure synthesis can be performed for any crash location within a cell, and the re-executed segment is idempotent.
    Section 3.2 and Table 2 describe synthesis rules; Section 5.4.3 and Section 6.3 acknowledge that non-idempotent faults break the approach.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DaiFu: In-Situ Crash Recovery for Deep Learning Systems." pith.science (2026). https://pith.science/paper/U7R7Z7RI

@misc{pith2026250701628,
  author       = {Pith},
  title        = {Pith review of: DaiFu: In-Situ Crash Recovery for Deep Learning Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U7R7Z7RI}},
  note         = {Machine review of arXiv:2507.01628}
}
read the original abstract

Deep learning (DL) systems have been widely adopted in many areas, and are becoming even more popular with the emergence of large language models. However, due to the complex software stacks involved in their development and execution, crashes are unavoidable and common. Crashes severely waste computing resources and hinder development productivity, so efficient crash recovery is crucial. Existing solutions, such as checkpoint-retry, are too heavyweight for fast recovery from crashes caused by minor programming errors or transient runtime errors. Therefore, we present DaiFu, an in-situ recovery framework for DL systems. Through a lightweight code transformation to a given DL system, DaiFu augments it to intercept crashes in situ and enables dynamic and instant updates to its program running context (e.g., code, configurations, and other data) for agile crash recovery. Our evaluation shows that DaiFu helps reduce the restore time for crash recovery, achieving a 1372x speedup compared with state-of-the-art solutions. Meanwhile, the overhead of DaiFu is negligible (under 0.40%). We also construct a benchmark spanning 7 distinct crash scenarios in DL systems, and show the effectiveness of DaiFu in diverse situations.

Figures

Figures reproduced from arXiv: 2507.01628 by the authors.

Figure 1
Figure 1. Three crash recovery paradigms for DL systems [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An analysis of crashes in DL systems from Sense [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. The overview of DaiFu. Algorithm 1: Function Decomposition Input: The AST of given function 𝑓 _𝑎𝑠𝑡 Output: The AST of vaccinated function 𝑣_𝑎𝑠𝑡; The cell tree 𝑐_𝑡𝑟𝑒𝑒 1 Initialize 𝑐_𝑡𝑟𝑒𝑒 2 𝑣_𝑎𝑠𝑡, 𝑐_𝑡𝑟𝑒𝑒 ←RecursiveExtract(𝑓 _𝑎𝑠𝑡, 𝑐_𝑡𝑟𝑒𝑒) 3 Function RecursiveExtract(𝑛𝑜𝑑𝑒, 𝑐_𝑡𝑟𝑒𝑒): 4 foreach child∈𝑛𝑜𝑑𝑒.𝑐ℎ𝑖𝑙𝑑𝑟𝑒𝑛𝑠 do 5 𝑐ℎ𝑖𝑙𝑑, 𝑐_𝑡𝑟𝑒𝑒 ←RecursiveExtract(𝑐ℎ𝑖𝑙𝑑, 𝑐_𝑡𝑟𝑒𝑒) 6 if child.type ∈ { ′ 𝐹𝑜𝑟′ , ′𝑊 ℎ𝑖𝑙𝑒′ , ′ 𝐹𝑢𝑛𝑐𝐷𝑒 𝑓 ′ } th… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: An example of program vaccination, with arrows indicating how the instrumented code enables in-situ recovery. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The restore time for crashes occurring at different [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The software updates in the Success Case I. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 32 canonical work pages

  1. [3]

    defect4ML-043

    2025. defect4ML-043. https://github.com/tensorflow/models/commit/9d96e9f

    work page 2025

  2. [1]

    2025. CUDA. https://developer.nvidia.com/cuda-toolkit

    work page 2025

  3. [2]

    DeepSpeed

    2025. DeepSpeed. https://www.deepspeed.ai/

    work page 2025

  4. [4]

    ggnn.pytorch

    2025. ggnn.pytorch. https://github.com/chingyaoc/ggnn.pytorch/commit/ 9c58ca6

    work page 2025

  5. [5]

    2025. NCCL. https://developer.nvidia.com/nccl

    work page 2025

  6. [6]

    The NVIDIA Data Loading Library (DALI)

    2025. The NVIDIA Data Loading Library (DALI). https://docs.nvidia.com/ deeplearning/dali/user-guide/docs/index.html

    work page 2025

  7. [7]

    2025. Pyright. https://github.com/Microsoft/pyright

    work page 2025

  8. [8]

    PyTorch Examples

    2025. PyTorch Examples. https://github.com/pytorch/examples

    work page 2025

Show all 64 references
  1. [9]

    PyTorch JIT

    2025. PyTorch JIT. https://pytorch.org/docs/stable/jit.html

    2025

  2. [10]

    DaiFu Artifact

    2025. DaiFu Artifact. https://anonymous.4open.science/r/DaiFu

    2025

  3. [11]

    Babiker Hussien Ahmed, Sai Peck Lee, Moon Ting Su, and Abubakar Zakari. 2020. Dynamic software updating: a systematic mapping study. IET Softw. 14, 5 (2020), 468–481. doi:10.1049/IET-SEN.2019.0201

    2020

  4. [12]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang,...

    2024 doi arXiv

  5. [14]

    Haibo Chen, Jie Yu, Rong Chen, Binyu Zang, and Pen-Chung Yew. 2007. POLUS: A POwerful Live Updating System. In 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007 . IEEE Computer Society, 271–281. doi:10.1109/ICSE.2007.65

    2007 doi

  6. [15]

    Yu Chen, Zhenming Liu, Bin Ren, and Xin Jin. 2020. On Efficient Constructions of Checkpoints. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119) . PMLR, 1627–16...

    2020

  7. [16]

    Yangtao Deng, Xiang Shi, Zhuo Jiang, Xingjian Zhang, Lei Zhang, Zhang Zhang, Bo Li, Zuquan Song, Hang Zhu, Gaohong Liu, et al. 2025. Minder: Faulty Machine Detection for Large-scale Distributed Model Training. In22nd USENIX Symposium on Networked Systems Design and Implementat...

    2025

  8. [17]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recogn...

    2021

  9. [18]

    Assaf Eisenman, Kiran Kumar Matam, Steven Ingram, Dheevatsa Mudigere, Raghuraman Krishnamoorthi, Krishnakumar Nair, Misha Smelyanskiy, and Mu- rali Annavaram. 2022. Check-N-Run: a Checkpointing System for Training Deep Learning Recommendation Models. In 19th USENIX Symposium o...

    2022

  10. [19]

    Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. 2019. Open- WebText Corpus. http://Skylion007.github.io/OpenWebTextCorpus

    2019

  11. [20]

    Gulavani, Nipun Kwatra, Ramachandran Ramjee, and Muthian Sivathanu

    Tanmaey Gupta, Sanjeev Krishnan, Rituraj Kumar, Abhishek Vijeev, Bhargav S. Gulavani, Nipun Kwatra, Ramachandran Ramjee, and Muthian Sivathanu. 2024. Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures. In Proceedings of the Nineteenth Euro...

    2024

  12. [21]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 . IEEE Computer Society, 770–778. doi:10.1109/CVPR.2016.90

    2016 doi

  13. [22]

    Zilong He, Pengfei Chen, Yu Luo, Qiuyu Yan, Hongyang Chen, Guangba Yu, and Fangyuan Li. 2022. Graph based Incident Extraction and Diagnosis in Large- Scale Online Systems. In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA...

    2022

  14. [23]

    Hicks and Scott Nettles

    Michael W. Hicks and Scott Nettles. 2005. Dynamic software updating. ACM Trans. Program. Lang. Syst. 27, 6 (2005), 1049–1096. doi:10.1145/1108970.1108971

    2005

  15. [24]

    Gísli Hjálmtýsson and Robert Gray. 1998. Dynamic C++ Classes - A Lightweight Mechanism to Update Code in a Running Program. In 1998 USENIX Annual Technical Conference, New Orleans, Louisiana, USA, June 15-19, 1998 . USENIX Association. https://www.usenix.org/conference/1998-us...

    1998

  16. [25]

    Qinghao Hu, Peng Sun, Shengen Yan, Yonggang Wen, and Tianwei Zhang. 2021. Characterization and prediction of deep learning workloads in large-scale GPU datacenters. In International Conference for High Performance Computing, Net- working, Storage and Analysis, SC 2021, St. Lou...

    2021

  17. [26]

    Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xiaolin Wang, Yingwei Luo, Yonggang Wen, and Tianwei Zhang. 2024. Characterization of Large Language Model Development in the Datacenter. In 21st USENIX Symposium on Networked Sy...

    2024

  18. [27]

    Haochen Huang, Chengcheng Xiang, Li Zhong, and Yuanyuan Zhou. 2021. PYLIVE: On-the-Fly Code Change for Python-based Online Services. In 2021 USENIX Annual Technical Conference, USENIX ATC 2021, July 14-16, 2021. USENIX Association, 349–363. https://www.usenix.org/conference/at...

    2021

  19. [29]

    Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. 2021. DeepCrime: mutation testing of deep learning systems based on real faults. In ISSTA ’21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, Denmark, July 11-17, 2021 . ACM, 67–...

    2021

  20. [31]

    Gunel Jahangirova and Paolo Tonella. 2020. An Empirical Evaluation of Mutation Operators for Deep Learning Systems. In 13th IEEE International Conference on Software Testing, Validation and Verification, ICST 2020, Porto, Portugal, October 24-28, 2020. IEEE, 74–84. doi:10.1109...

    2020

  21. [32]

    Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wen- cong Xiao, and Fan Yang. 2019. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. In 2019 USENIX Annual Technical Confer- ence, USENIX ATC 2019, Renton, W A, USA, July 10-12, ...

    2019

  22. [33]

    Jinhan Kim, Nargiz Humbatova, Gunel Jahangirova, Paolo Tonella, and Shin Yoo. 2023. Repairing DNN Architecture: Are We There Yet?. InIEEE Conference on Software Testing, Verification and Validation, ICST 2023, Dublin, Ireland, April 16-20, 2023. IEEE, 234–245. doi:10.1109/ICST...

    2023

  23. [34]

    Taeyoon Kim, Suyeon Jeong, Jongseop Lee, Soobee Lee, and Myeongjae Jeon

  24. [35]

    Sampo Kuutti, Richard Bowden, Yaochu Jin, Phil Barber, and Saber Fallah. 2021. A Survey of Deep Learning Applications to Autonomous Vehicle Control. IEEE Trans. Intell. Transp. Syst. 22, 2 (2021), 712–733. doi:10.1109/TITS.2019.2962338

    2021

  25. [36]

    Linyi Li, Yuhao Zhang, Luyao Ren, Yingfei Xiong, and Tao Xie. 2023. Reliability Assurance for Deep Neural Network Architectures Against Numerical Defects. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE...

    2023

  26. [37]

    Yunkai Liang, Yun Lin, Xuezhi Song, Jun Sun, Zhiyong Feng, and Jin Song Dong

  27. [38]

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-...

    2021

  28. [39]

    In 44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2022, Pittsburgh, PA, USA, May 22-24,

    gDefects4DL: A Dataset of General Real-World Deep Learning Program Defects. In 44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2022, Pittsburgh, PA, USA, May 22-24,

    2022

  29. [40]

    doi:10.1145/3510454.3516826

    ACM/IEEE, 90–94. doi:10.1145/3510454.3516826

  30. [41]

    Mohammad Mehdi Morovati, Amin Nikanjam, Foutse Khomh, and Zhen Ming (Jack) Jiang. 2023. Bugs in machine learning-based systems: a faultload benchmark. Empir. Softw. Eng. 28, 3 (2023), 62. doi:10.1007/s10664-023-10291-1

    2023 doi

  31. [42]

    Chuan Luo, Pu Zhao, Bo Qiao, Youjiang Wu, Hongyu Zhang, Wei Wu, Weihai Lu, Yingnong Dang, Saravanakumar Rajmohan, Qingwei Lin, and Dongmei Zhang. 2021. NTAM: Neighborhood-Temporal Attention Model for Disk Failure Prediction in Cloud Platforms. In WWW ’21: The Web Conference 20...

    2021

  32. [43]

    Jayashree Mohan, Amar Phanishayee, and Vijay Chidambaram. 2021. CheckFreq: Frequent, Fine-Grained DNN Checkpointing. In 19th USENIX Conference on File and Storage Technologies, FAST 2021, February 23-25, 2021 . USENIX Association, 203–216. https://www.usenix.org/conference/fas...

    2021

  33. [44]

    Angela Nicoara, Gustavo Alonso, and Timothy Roscoe. 2008. Controlled, system- atic, and efficient code replacement for running java programs. In Proceedings of the 2008 EuroSys Conference, Glasgow, Scotland, UK, April 1-4, 2008. ACM, 233–246. doi:10.1145/1352592.1352617

    2008

  34. [45]

    Hicks, Gareth Paul Stoyle, and Manuel Oriol

    Iulian Neamtiu, Michael W. Hicks, Gareth Paul Stoyle, and Manuel Oriol. 2006. Practical dynamic software updating for C. In Proceedings of the ACM SIGPLAN 2006 Conference on Programming Language Design and Implementation, Ottawa, Ontario, Canada, June 11-14, 2006 , Michael I. ...

    2006

  35. [46]

    John Ashworth Nelder and Robert WM Wedderburn. 1972. Generalized linear models. Journal of the Royal Statistical Society: Series A (General) 135, 3 (1972), 370–384

    1972

  36. [47]

    Luís Pina, Luís Veiga, and Michael W. Hicks. 2014. Rubah: DSU for Java on a stock JVM. In Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications, OOPSLA 2014, part of SPLASH 2014, Portland, OR, USA, October 20-24, 2...

    2014

  37. [48]

    Amin Nikanjam, Houssem Ben Braiek, Mohammad Mehdi Morovati, and Foutse Khomh. 2022. Automatic Fault Detection for Deep Learning Programs Using Graph Transformations. ACM Trans. Softw. Eng. Methodol. 31, 1 (2022), 14:1–14:27. doi:10.1145/3470006

    2022 doi

  38. [49]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Des- maison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, L...

    2019

  39. [50]

    Bernstein, Alexander C

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis. 115, 3 (2015), ...

    2015 doi

  40. [51]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9

    2019

  41. [52]

    Florian Rommel, Christian Dietrich, Daniel Friesel, Marcel Köppen, Christoph Borchert, Michael Müller, Olaf Spinczyk, and Daniel Lohmann. 2020. From Global to Local Quiescence: Wait-Free Code Patching of Multi-Threaded Processes. In 14th USENIX Symposium on Operating Systems D...

    2020

  42. [53]

    Wei Tang and Min Zhang. 2018. PyReload: Dynamic Updating of Python Programs by Reloading. In 25th Asia-Pacific Software Engineering Conference, APSEC 2018, Nara, Japan, December 4-7, 2018 . IEEE, 229–238. doi:10.1109/APSEC.2018.00037

    2018

  43. [54]

    Eldon Schoop, Forrest Huang, and Bjoern Hartmann. 2021. UMLAUT: Debugging Deep Learning Programs using Program Structure and Model Behavior. InCHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama, Japan, May 8-13, 2021 . ACM, 310:1–310:16. do...

    2021

  44. [55]

    Hicks, and Kathryn S

    Suriya Subramanian, Michael W. Hicks, and Kathryn S. McKinley. 2009. Dynamic software updates: a VM-centric approach. In Proceedings of the 2009 ACM SIG- PLAN Conference on Programming Language Design and Implementation, PLDI 2009, Dublin, Ireland, June 15-21, 2009 . ACM, 1–12...

    2009

  45. [56]

    Chengcheng Wan, Shicheng Liu, Sophie Xie, Yuhan Liu, Henry Hoffmann, Michael Maire, and Shan Lu. 2024. Keeper: Automated Testing and Fixing of Machine Learning Software. ACM Trans. Softw. Eng. Methodol. 33, 7 (2024), 167:1–167:33. doi:10.1145/3672451

    2024 doi

  46. [57]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2025. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_ alpaca

    2025

  47. [58]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lam- ple. 2023. LLaMA: Open and Efficient Foundation ...

    2023 doi arXiv

  48. [60]

    Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xinwei Fu, T. S. Eugene Ng, and Yida Wang. 2023. GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany,...

    2023

  49. [61]

    Mohammad Wardat, Breno Dantas Cruz, Wei Le, and Hridesh Rajan. 2022. Deep- Diagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA,...

    2022

  50. [62]

    Ru Zhang, Wencong Xiao, Hongyu Zhang, Yu Liu, Haoxiang Lin, and Mao Yang

  51. [63]

    Dangwei Wu, Beijun Shen, Yuting Chen, He Jiang, and Lei Qiao. 2021. Tensfa: Detecting and Repairing Tensor Shape Faults in Deep Learning Systems. In 32nd IEEE International Symposium on Software Reliability Engineering, ISSRE 2021, Wuhan, China, October 25-28, 2021 . IEEE, 11–...

    2021

  52. [64]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith St...

    2016 arXiv

  53. [67]

    Xiaoyu Zhang, Juan Zhai, Shiqing Ma, and Chao Shen. 2021. AUTOTRAINER: An Automatic DNN Training Problem Detection and Repair System. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021 . IEEE, 359–371. doi:10.1109/ICSE43...

    2021

  54. [68]

    Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018. An empirical study on TensorFlow program bugs. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, Amsterdam, The Netherlands, July 16-21, 201...

    2018

  55. [2020]

    In ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020

    An empirical study on program failures of deep learning jobs. In ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020. ACM, 1159–1170. doi:10.1145/3377811.3380362

    2020

  56. [2022]

    In 2022 USENIX Annual Technical Conference, USENIX ATC 2022, Carlsbad, CA, USA, July 11-13, 2022

    Sibylla: To Retry or Not To Retry on Deep Learning Job Failure. In 2022 USENIX Annual Technical Conference, USENIX ATC 2022, Carlsbad, CA, USA, July 11-13, 2022. USENIX Association, 263–270. https://www.usenix.org/conference/ atc22/presentation/kim-taeyoon

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.