Pith. sign in

REVIEW 3 major objections 4 minor 52 references

Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training invariants inferred from small example runs can catch silent deep-learning training errors within a single iteration.

desk verdict Solid systems paper with real practical value; the 18/20 detection rate deserves scrutiny because of the opaque invariant pruning, but the independent bug finds and honest evaluation make it worth reviewing. read the letter →

arxiv 2506.14813 v1 pith:67S2C6KI submitted 2025-06-06 cs.LG cs.AI

Yuxuan Jiang , Ziming Zhou , Boyu Xu , Beijie Liu , Runhui Xu , Peng Huang This is my paper
classification cs.LGcs.AI
keywords silenterrorstraininginvariantsdeeplearninginvariantinferenceproactivecheckingdistributedpreconditionruntimeverification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that most silent deep-learning training errors can be caught early by checking a small set of training invariants: rules about how model weights, API calls, and tensor shapes should behave during correct training. It presents TRAINCHECK, which infers such invariants automatically from a few clean example pipelines, deduces the conditions under which each rule applies, and then checks the rules continuously while a target job trains. On a reproduced suite of 20 real-world silent errors, the approach detects 18 within a single training iteration, and it uncovered 6 previously unknown library bugs, 3 already fixed. If this holds, proactive invariant checking would give deep-learning engineers a practical safety net that catches errors high-level loss and accuracy monitoring miss.

What carries the argument

The load-bearing mechanism is preconditioned invariant inference over runtime traces. TRAINCHECK instruments a training program, extracts high-level events and variable states, then generates and validates hypotheses from five relation templates: Consistent, EventContain, APISequence, APIArg, and APIOutput. The precondition deduction algorithm scans passing and failing examples to find conditions, from types CONSTANT, CONSISTENT, UNEQUAL, and EXIST, that cleanly separate them; when a single conjunction is unsafe, it splits passing examples into subgroups and combines the resulting preconditions disjunctively. This precondition machinery is what makes invariants transferable across pipelines and keeps false positives low.

What would settle it

Take 100 correct training programs that span frameworks, parallelism strategies, and optimizers beyond the 63 programs studied, and apply TRAINCHECK's inferred invariants unchanged; if any invariant fires on a known-correct pipeline, or if running the same 20 reproduced silent errors detects fewer than 18 within one iteration, the central claim is weakened.

Watch

Extended reading notes

Core claim

TRAINCHECK's central claim is that silent training errors can be detected as violations of transferable training invariants, where each invariant is a concrete rule about a relation between variables or events, guarded by a precondition that says when the rule applies. The paper argues that non-determinism in training is an artifact of checking at too high a level: by observing the right level of behavior, invariants can be simple and deterministic. It demonstrates this by inferring invariants from small-scale example pipelines (at most 4 GPUs and 100 iterations) and using them to detect 18 of 20 reproduced real-world silent errors within one training iteration, while also uncovering 6 unknown bugs in popular training libraries. The invariants are expressed through five relation templates, and each invariant's precondition is deduced from observed traces so that the check is precise enough to avoid false alarms.

Load-bearing premise

The framework's power rests on the premise that invariants learned from a handful of clean, small-scale example pipelines hold on every correct training pipeline, so that any violation reliably signals a real error.

Editorial extensions

If this is right

  • Training jobs can be monitored with low overhead, typically under 2% in selective mode and at most about 1.6x slowdown, while catching errors in the same iteration they manifest.
  • Invariants inferred once from high-quality examples can be reused across different frameworks and libraries, since transferable invariants with preconditions generalize beyond the pipelines they were learned from.
  • High-level monitoring based on loss, accuracy, and gradient norms can be complemented by invariant checks that localize root causes in 10 of 18 detected cases and narrow the problem area in the rest.
  • Detection coverage depends on how well the example pipelines cover the semantics involved; errors in under-represented features, such as DeepSpeed's mixture-of-experts, can be missed under random sampling.
  • The framework can serve as a proactive deployment-time safety net rather than only a development-time test, because checks run continuously with the training task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The transferability result suggests a plausible division of labor: library maintainers could curate and publish invariant sets for their frameworks, while users simply run a verifier against their own pipelines, without writing checks by hand.
  • Because invariants are learned from observed traces and filtered by whether a precondition can be deduced, the approach is inherently limited to errors that manifest as violations of the five supported relation templates; legitimate pipeline diversity is the main remaining false-positive risk.
  • A natural next experiment is to apply TRAINCHECK to a large corpus of unmodified production training jobs and measure how often reported violations correspond to confirmed bugs, which would quantify precision beyond the 63-program false-positive study.
  • The framework's design implies a testable extension: adapt the same preconditioned-invariant idea to track Python primitive variables, which would target errors like incorrect total step counts that currently escape detection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies silent errors in deep learning training and presents TRAINCHECK, a framework that automatically infers deterministic training invariants (five relation templates with preconditions) from small-scale example pipelines and checks them at runtime. The authors collect and reproduce 20 real-world silent training errors, report that TRAINCHECK detects 18 within one iteration, find 6 previously unknown library bugs (3 fixed), and report low false-positive rates on 63 programs, transferability across pipelines, and low runtime overhead. The paper also honestly documents its scope: Python-only, no local-primitive tracking, no JIT-compiled code paths, and two missed benchmark cases.

Significance. If correct, the paper makes a substantial contribution to ML systems reliability: it demonstrates that a small set of transferable, deterministic invariants inferred from clean example pipelines can catch a diverse range of silent training errors with high precision and little overhead, while providing actionable diagnosis hints. The independent discovery of six new library bugs, the 63-program false-positive study, and the open-source release are strong evidence of practical value. The main weakness is that the invariant-selection step used for the headline detection rate is not described reproducibly, which currently makes the 18/20 figure impossible to verify independently.

major comments (3)
  1. [§5.6] The selection of the final 100 invariants from roughly 3,000 inferred invariants is not specified concretely, so the 18/20 detection result in §5.1 is not reproducible and may be inflated if the pruning was influenced by the 20 target errors. The text says the pruning 'focuses on the most critical APIs and variables... prioritizing those less tested or more likely to change,' but it gives no algorithm, no threshold, and no final invariant list. Please provide a fixed, public selection policy or the exact invariant list used for Table 4, and add an ablation comparing detection under the stated policy with detection under random or purely coverage-based selection. The independent findings in §5.2 mitigate but do not eliminate this concern, because the paper does not report which invariants fired for the six new bugs or whether those invariants overlap with the ones used for the 18/20 benchmark.
  2. [§3.6 and §3.7] A precondition is defined as 'safe' when it separates passing from failing examples in the same traces used to infer it, and §3.7 discards invariants without a deducible precondition. Because there is no held-out split at inference time, preconditions can overfit to the small-scale example pipelines, which would inflate detection rates. The 63-program false-positive study in §5.3 provides useful external evidence, but it does not apply the exact invariant-selection policy used for the 20-error benchmark. Please report a held-out evaluation of precondition safety (for example, splitting traces into inference and validation folds) and, at minimum, report the false-positive results using the precise §5.6 selection policy.
  3. [§3.2, Table 2] The five relation templates are described as 'representative relations often violated in these silent errors,' but the paper does not state whether this relation set was fixed before or after assembling the 20-error benchmark. If the relation set was adjusted based on the benchmark cases, the 18/20 detection rate partly reflects configuration fitting rather than a general property of the approach. Please document the timeline and process for choosing the relation templates, or show that each relation is justified independently of the benchmark, for example by grounding them in the 88-error study and by reporting which relations were responsible for the six new bugs in §5.2.
minor comments (4)
  1. [Table 4] The detection row uses '0', '1', and '✖' without explaining that '0' means detected in the same iteration as the root-cause trigger and '1' means one iteration later; please state this explicitly in the caption or legend.
  2. [§5.2] Please include GitHub issue links or IDs and the specific invariants that fired for each of the six new bugs; this is the strongest external validation and is currently not fully reproducible from the text.
  3. [§5.7] The overhead evaluation deploys 100 randomly sampled invariants, whereas §5.6 uses a pruned, apparently non-random set; please clarify whether selective-instrumentation overhead depends on which invariants are chosen.
  4. [§5.3] The false-positive rate is reported as a percentage, but the denominator is not defined in the text or figure caption; please state whether it is the fraction of checked invariants, of pipelines, or of training iterations that produce false alarms.

Circularity Check

2 steps flagged · score 4.0 of 10

Detection pipeline is largely self-contained, but the transferability evaluation in §5.4 is circular by construction; the relation vocabulary is also benchmark-informed, leaving the headline 18/20 only partially out-of-sample.

  1. fitted input called prediction [§5.4 Invariant Transferability, Figure 8]
    "To evaluate this transferability, we apply a set of valid invariants to all 63 collected pipelines. These invariants are inferred using a 5/6-input setup across all classes and exclude any that triggered false positives in §5.3."

    The invariant set used for the transferability claim is explicitly filtered by requiring no false positives on the same 63-pipeline collection that is then used to measure transferability. The next sentence in the paper says that for each invariant the authors count how many pipelines it can be applied to without raising a false alarm; since invariants that did raise false alarms on those pipelines were excluded beforehand, the resulting 'broad transferability' numbers in Figure 8 are guaranteed by construction for the retained set. This is a circular evaluation of the transferability claim: the validation set is used both to select the invariants and to report their applicability. The honest out-of-sample false-positive signal is Figure 7, which is computed before this filtering.

  2. other [§3.2 Invariant Representation, Table 2]
    "Through analyzing silent training errors in practice, we identify and provide a set of representative relations often violated in these silent errors."

    The five supported relation templates are selected by studying the same class of silent training errors later used as the 20-error benchmark in §5.1. This is a benchmark-informed design choice rather than a by-construction reduction: the invariants themselves are still inferred automatically from separate clean example pipelines, and the six newly found bugs provide independent evidence. However, the 18/20 detection rate is not fully out-of-sample with respect to the relation vocabulary, because that vocabulary was chosen to cover the error types represented in the benchmark. This lowers the strength of the headline number as an independent validation of the framework's expressiveness, though it is not a self-citation chain or a fitted-parameter rename.

full rationale

The core derivation of TRAINCHECK's detection claim is not circular: invariants are inferred from official example pipelines (PyTorch GCN, Autocast, DDP examples, Megatron-DeepSpeed GPT, and Transformers trainer examples), not from the 20 reproduced errors, and the six previously unknown bugs (three fixed) were found by applying the invariants to unseen issue reports. That independent discovery is strong evidence that the invariants carry content beyond the evaluation set. The main circular step is confined to §5.4, where the transferability statistic is computed on the same 63-program collection used to filter the invariant set, making the reported 'without raising a false alarm' applicability true by construction for the retained invariants. A secondary concern is that the five relation templates were explicitly identified by analyzing silent errors in practice, meaning the 20-error benchmark partially re-tests the design vocabulary; this is a common and transparent design choice, not a logical reduction. The precondition safety definition in §3.6 ('safe if it provides a clean separation' on the same traces) is an overfitting risk rather than a circular derivation, and the paper acknowledges it does not guarantee the weakest precondition. The invariant-pruning step in §5.6 (3,000 to 100 invariants, 'prioritizing those less tested or more likely to change') is under-specified, but the paper does not state that it used the benchmark errors for pruning, so it is a reproducibility gap, not evidence of circularity. No load-bearing self-citation exists; the Oathkeeper citation [28] is related work only. Overall, the central 18/20 detection claim has independent support, while a secondary transferability metric and the benchmark-informed relation vocabulary justify a moderate score of 4 rather than 0-2.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on hand-selected relation templates, a subjective invariant-pruning step, and domain assumptions about the representativeness of example pipelines and the sufficiency of model/optimizer-level checks. No new physical or mathematical entities are introduced.

free parameters (3)
  • Relation templates = 5 templates (Consistent, EventContain, APISequence, APIArg, APIOutput)
    Hand-selected by analyzing the 20 studied silent errors; the choice of these relations, rather than a formal derivation, determines what TRAINCHECK can detect (§3.2).
  • Invariant pruning criteria = 100 of ~3,000 invariants retained
    The pruning chooses 'critical' APIs and variables, 'prioritizing those less tested or more likely to change'; selection criteria are subjective and may be informed by the evaluation errors (§5.6).
  • Precondition condition types = CONSTANT, CONSISTENT, UNEQUAL, EXIST
    The grammar of preconditions is fixed by the authors, restricting expressiveness; used in §3.6 for precondition deduction.
assumptions (4)
  • ad hoc to paper The example pipelines used for invariant inference (PyTorch examples, Megatron-DeepSpeed GPT, Transformers trainer examples) are bug-free and representative of correct DL training.
    If an example pipeline contains a hidden bug or is not representative, the inferred invariants will be incorrect or too specific; this is load-bearing for the transferability claim (§5.1).
  • domain assumption Correct DL training can be validated by checking only the five relation templates over model/optimizer state and framework API events.
    The two missed cases (TF-33455, TF-29903) show this scope boundary; the paper acknowledges it in §5.1 and §6.
  • domain assumption Non-determinism in DL training disappears at the chosen level of abstraction, so deterministic invariant rules can be used.
    Stated in §1: 'Non-determinism is an artifact of checking at too high of a level.' This is the basis for using strict relations rather than probabilistic predicates.
  • domain assumption Hashing tensors preserves the equality relationships needed for invariant checking.
    The Instrumentor logs hashes instead of values (§4.1); this prevents detecting numerical instabilities, as noted in §6, and assumes no collision or equality loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks." pith.science (2026). https://pith.science/paper/67S2C6KI

@misc{pith2026250614813,
  author       = {Pith},
  title        = {Pith review of: Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/67S2C6KI}},
  note         = {Machine review of arXiv:2506.14813}
}
read the original abstract

Training deep learning (DL) models is a complex process, making it prone to silent errors that are challenging to detect and diagnose. This paper presents TRAINCHECK, a framework that takes a proactive checking approach to address silent training errors. TRAINCHECK automatically infers invariants tailored for DL training. It uses these invariants to proactively detect silent errors during the training process while providing debugging help. To evaluate TRAINCHECK, we reproduce 20 real-world silent training errors with diverse root causes. TRAINCHECK successfully detects 18 errors within a single training iteration. It also uncovers 6 unknown bugs in popular training libraries that lead to silent errors.

Figures

Figures reproduced from arXiv: 2506.14813 by the authors.

Figure 1
Figure 1. shows a real-world silent training error in Hug￾gingFace’s training of BLOOM-176B [4]. DeepSpeed’s BF16Optimizer was used in this training task, which had a logic bug in gradient clipping. This bug did not trigger any exception but caused parts of the model to diverge silently across GPUs, a problem that went undetected for 10 days. ∗A shorter version of this paper is to appear in the Proceedings of the 19th USENIX … view at source ↗
Figure 2
Figure 2. Root cause locations and types of the studied errors. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. shows the workflow of TRAINCHECK, which op￾erates in two phases. In the offline phase, it automatically infers a set of training invariants from high-quality training pipelines. In the online phase, these invariants are deployed to a given training job to check for violations during training. TRAINCHECK reports invariant violations with contextual information to assist confirmation and investigation. TRAINCHECK comp… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The invariant with preconditions TRAINCHECK infers for Bloom-176B error (§ 2.2), and the simplified trace it emits and uses. continuously validates a training task against the invariants. When Instrumentor is used in the online stage, it performs selective instrumentat…
Figure 5
Figure 5. Figure 5: Deduce precondition in under-constrained situations. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Root cause types of errors we reproduce for evaluation. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: False positive rates across four program classes, broken [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Invariant applicability across all collected pipelines. The [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Detection rate as the number of input pipelines increases [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 12
Figure 12. Figure 12: Inference time vs. trace size (normalized to ResNet-18 [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 51 canonical work pages

  1. [1]

    TensorFlow: a system for large-scale machine learning

    Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, San- jay Ghemawat, Geoffrey Irving, et al. TensorFlow: a system for large-scale machine learning. InProceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, OSDI ’16, page 265–283, Savannah, GA, USA, November 2016. USENIX Asso- ciation

    work page 2016

  2. [2]

    PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compila- tion

    Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael V oznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compila- tion. In29th ACM International Conference on Archi- tectural Support for Programming Languages and Op- e...

    work page 2024

  3. [3]

    Varuna: scal- able, low-cost training of massive deep learning models

    Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ra- machandran Ramjee, and Nipun Kwatra. Varuna: scal- able, low-cost training of massive deep learning models. InProceedings of the Seventeenth European Confer- ence on Computer Systems, EuroSys ’22, page 472–487, Rennes, France, April 2022. Association for Computing Machinery

    work page 2022

  4. [4]

    BLOOM: Megatron-DeepSpeed

    Stas Bekman. BLOOM: Megatron-DeepSpeed. https: //huggingface.co/blog/bloom-megatron-deepspeed, 2022

    work page 2022

  5. [5]

    Weights and Biases

    Lukas Biewald. Weights and Biases. https://www. wandb.com/, 2020

    work page 2020

  6. [6]

    Chronicles of Big- Science TR11-176B-ML Training

    BigScience Workshop. Chronicles of Big- Science TR11-176B-ML Training. https: //github.com/bigscience-workshop/bigscience/ blob/master/train/tr11-176B-ml/chronicles.md# 2022-04-30-hanging-at-eval, 2022

    work page 2022

  7. [7]

    Dropout with Theano

    Rishi Bommasani. Dropout with Theano. https://rishy.github.io/ml/2016/10/12/ dropout-with-theano/, 2016

    work page 2016

  8. [8]

    Hudson, Ehsan Adeli, and et al

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, and et al. On the opportunities and risks of foundation mod- els, 2022

    work page 2022

Show all 52 references
  1. [9]

    Toward understanding deep learning framework bugs.ACM Trans

    Junjie Chen, Yihua Liang, Qingchao Shen, Jiajun Jiang, and Shuochuan Li. Toward understanding deep learning framework bugs.ACM Trans. Softw. Eng. Methodol., 32(6), September 2023

    2023

  2. [10]

    CodeParrot Clean Train Dataset, 2021

    CodeParrot. CodeParrot Clean Train Dataset, 2021

    2021

  3. [11]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: fast and memory- efficient exact attention with IO-awareness. InPro- ceedings of the 36th International Conference on Neu- ral Information Processing Systems, NIPS ’22, page 16344–16359, New...

    2022

  4. [12]

    DeepSpeed GitHub Is- sues

    DeepSpeed Contributors. DeepSpeed GitHub Is- sues. https://github.com/microsoft/DeepSpeed/ issues, 2024

    2024

  5. [13]

    Un- derstanding software-2.0: A study of machine learning library usage and evolution.ACM Trans

    Malinda Dilhara, Ameya Ketkar, and Danny Dig. Un- derstanding software-2.0: A study of machine learning library usage and evolution.ACM Trans. Softw. Eng. Methodol., 30(4), July 2021

    2021

  6. [14]

    The Llama 3 Herd of Models, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, et al. The Llama 3 Herd of Models, 2024

    2024

  7. [15]

    Bugs as deviant behavior: a general approach to inferring errors in systems code

    Dawson Engler, David Yu Chen, Seth Hallem, Andy Chou, and Benjamin Chelf. Bugs as deviant behavior: a general approach to inferring errors in systems code. InProceedings of the Eighteenth ACM Symposium on Operating Systems Principles, SOSP ’01, page 57–72, Banff, Alberta, Cana...

    2001

  8. [16]

    Ernst, Jake Cockrell, William G

    Michael D. Ernst, Jake Cockrell, William G. Griswold, and David Notkin. Dynamically Discovering Likely Program Invariants to Support Program Evolution. In Proceedings of the 21st International Conference on Software Engineering, ICSE ’99, page 213–224, Los Angeles, California,...

    1999

  9. [17]

    A guide to deep learning in healthcare.Na- ture Medicine, 25(1):24–29, January 2019

    Andre Esteva, Alexandre Robicquet, Bharath Ramsun- dar, V olodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. A guide to deep learning in healthcare.Na- ture Medicine, 25(1):24–29, January 2019

    2019

  10. [18]

    Audee: automated testing for deep learning frameworks

    Qianyu Guo, Xiaofei Xie, Yi Li, Xiaoyu Zhang, Yang Liu, Xiaohong Li, and Chao Shen. Audee: automated testing for deep learning frameworks. InProceedings of the 35th IEEE/ACM International Conference on Auto- mated Software Engineering, ASE ’20, pages 486–498, Virtual Event, Au...

    2020

  11. [19]

    Sudheendra Hangal and Monica S. Lam. Tracking down software bugs using automatic anomaly detection. In Proceedings of the 24th International Conference on Software Engineering, ICSE ’02, page 291–301, Or- lando, Florida, May 2002. Association for Computing Machinery

    2002

  12. [20]

    Dataloader not ran- domly sampling in PyTorch

    Monica Heddneck. Dataloader not ran- domly sampling in PyTorch. https: //stackoverflow.com/questions/50124712/ dataloader-not-randomly-sampling-in-pytorch , 2018

    2018

  13. [21]

    Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates

    Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowdhury. Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates. InProceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 382–395, Koblenz, Germany, October 2023....

    2023

  14. [22]

    A static an- alyzer for detecting tensor shape errors in deep neural network training code

    Ho Young Jhoo, Sehoon Kim, Woosung Song, Kyuyeon Park, DongKwon Lee, and Kwangkeun Yi. A static an- alyzer for detecting tensor shape errors in deep neural network training code. InProceedings of the ACM/IEEE 44th International Conference on Software Engineer- ing: Companion P...

    2022

  15. [23]

    Deep learning.Nature, 521(7553):436–444, May 2015

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.Nature, 521(7553):436–444, May 2015

    2015

  16. [24]

    Reliability Assurance for Deep Neural Network Architectures against Numerical Defects

    Linyi Li, Yuhao Zhang, Luyao Ren, Yingfei Xiong, and Tao Xie. Reliability Assurance for Deep Neural Network Architectures against Numerical Defects. InProceed- ings of the 45th International Conference on Software Engineering, ICSE ’23, page 1827–1839. IEEE Press, May 2023

    2023

  17. [25]

    Universal Checkpointing: Efficient and Flex- ible Checkpointing for Large Scale Distributed Training, 2024

    Xinyu Lian, Sam Ade Jacobs, Lev Kurilenko, Masahiro Tanaka, Stas Bekman, Olatunji Ruwase, and Minjia Zhang. Universal Checkpointing: Efficient and Flex- ible Checkpointing for Large Scale Distributed Training, 2024

    2024

  18. [26]

    NN- Smith: Generating Diverse and Valid Test Cases for Deep Learning Compilers

    Jiawei Liu, Jinkun Lin, Fabian Ruffy, Cheng Tan, Jinyang Li, Aurojit Panda, and Lingming Zhang. NN- Smith: Generating Diverse and Valid Test Cases for Deep Learning Compilers. InProceedings of the 28th ACM International Conference on Architectural Support for Programming Langu...

    2023

  19. [27]

    NeuRI: Diversifying DNN Generation via In- ductive Rule Inference

    Jiawei Liu, Jinjun Peng, Yuyao Wang, and Lingming Zhang. NeuRI: Diversifying DNN Generation via In- ductive Rule Inference. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineer- ing, ESEC/FSE ’23, pa...

    2023

  20. [28]

    Demystify- ing and Checking Silent Semantic Violations in Large Distributed Systems

    Chang Lou, Yuzhuo Jing, and Peng Huang. Demystify- ing and Checking Silent Semantic Violations in Large Distributed Systems. In16th USENIX Symposium on Operating Systems Design and Implementation, OSDI ’22, pages 91–107, Carlsbad, CA, July 2022. USENIX Association

    2022

  21. [29]

    Sakallah

    Haojun Ma, Aman Goel, Jean-Baptiste Jeannin, Manos Kapritsos, Baris Kasikci, and Karem A. Sakallah. I4: incremental inference of inductive invariants for verifi- cation of distributed protocols. InProceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP ’19,...

    2019

  22. [30]

    Efficient large-scale language model training on GPU clusters using megatron-LM

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on GPU clusters u...

    2021

  23. [31]

    DeepXplore: Automated Whitebox Testing of Deep Learning Systems

    Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. InProceedings of the 26th Sym- posium on Operating Systems Principles, SOSP ’17, page 137–145, Shanghai, China, October 2017. ACM

    2017

  24. [32]

    CRADLE: Cross-Backend Validation to De- tect and Localize Bugs in Deep Learning Libraries

    Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, and Lin Tan. CRADLE: Cross-Backend Validation to De- tect and Localize Bugs in Deep Learning Libraries. In 2019 IEEE/ACM 41st International Conference on Soft- ware Engineering, ICSE ’19, pages 1027–1038, Mon- treal, QC, Canada, M...

    2019

  25. [33]

    PyTorch Discussion Forum

    PyTorch Community. PyTorch Discussion Forum. https://discuss.pytorch.org/, 2024

    2024

  26. [34]

    PyTorch Examples

    PyTorch Contributors. PyTorch Examples. https:// github.com/pytorch/examples, 2024

    2024

  27. [35]

    PyTorch GitHub Issues

    PyTorch Contributors. PyTorch GitHub Issues. https: //github.com/pytorch/pytorch/issues, 2024. 18

    2024

  28. [36]

    A Bug That Plagues Thousands of Open-Source ML Projects

    Tanel Pärnamaa. A Bug That Plagues Thousands of Open-Source ML Projects. https://tanelp.github.io/posts/ a-bug-that-plagues-thousands-of-open-source-ml-projects/ , 2019

    2019

  29. [37]

    Tanel Pärnamaa. Using PyTorch & Numpy: A Bug That Plagues Thousands of Open-Source ML Projects.https: //www.reddit.com/r/MachineLearning/comments/ mocpgj/p_using_pytorch_numpy_a_bug_that_plagues/, 2021

    2021

  30. [38]

    Language Models are Un- supervised Multitask Learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Un- supervised Multitask Learners. OpenAI, 2019

    2019

  31. [39]

    DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20,...

    2020

  32. [40]

    StackOverflow - Questions and Answers on PyTorch

    StackOverflow Contributors. StackOverflow - Questions and Answers on PyTorch. https://stackoverflow.com/ questions/tagged/pytorch, 2024

    2024

  33. [41]

    Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs

    John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs. In20th USENIX Symposium on Networked Systems Design and Implementati...

    2023

  34. [42]

    DeepTest: automated testing of deep-neural-network- driven autonomous cars

    Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. DeepTest: automated testing of deep-neural-network- driven autonomous cars. InProceedings of the 40th In- ternational Conference on Software Engineering, ICSE ’18, page 303–314, Gothenburg, Sweden, June 2018. As- sociation ...

    2018

  35. [43]

    Deep learning library testing via ef- fective model generation

    Zan Wang, Ming Yan, Junjie Chen, Shuang Liu, and Dongdi Zhang. Deep learning library testing via ef- fective model generation. InProceedings of the 28th ACM Joint Meeting on European Software Engineer- ing Conference and Symposium on the Foundations of Software Engineering, ES...

    2020

  36. [44]

    BloombergGPT: A Large Language Model for Finance, 2023

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravol- ski, Mark Dredze, Sebastian Gehrmann, Prabhan- jan Kambadur, David Rosenberg, and Gideon Mann. BloombergGPT: A Large Language Model for Finance, 2023

    2023

  37. [45]

    Unawareness of Deep Learning Mistakes

    Yuxin Wu. Unawareness of Deep Learning Mistakes. https://ppwwyyxx.com/blog/2017/ Unawareness-Of-Deep-Learning-Mistakes/, 2017

    2017

  38. [46]

    Fight Against Silent Bugs in Deep Learning Libraries

    Yuxin Wu. Fight Against Silent Bugs in Deep Learning Libraries. https://ppwwyyxx.com/blog/2020/ Fight-Against-Silent-Bugs-in-Deep-Learning-Libraries/ , 2020

    2020

  39. [47]

    DuoAI: Fast, Automated Inference of Inductive In- variants for Verifying Distributed Protocols

    Jianan Yao, Runzhou Tao, Ronghui Gu, and Jason Nieh. DuoAI: Fast, Automated Inference of Inductive In- variants for Verifying Distributed Protocols. In16th USENIX Symposium on Operating Systems Design and Implementation, OSDI ’22, pages 485–501, Carlsbad, CA, July 2022. USENIX...

    2022

  40. [48]

    DistAI: Data-Driven Automated Invariant Learning for Distributed Protocols

    Jianan Yao, Runzhou Tao, Ronghui Gu, Jason Nieh, Suman Jana, and Gabriel Ryan. DistAI: Data-Driven Automated Invariant Learning for Distributed Protocols. In15th USENIX Symposium on Operating Systems De- sign and Implementation, OSDI ’21, pages 405–421, Virtual Event, USA, Jul...

    2021

  41. [49]

    Hyper-Parameter Optimization: A Review of Algorithms and Applications, 2020

    Tong Yu and Hong Zhu. Hyper-Parameter Optimization: A Review of Algorithms and Applications, 2020

    2020

  42. [50]

    An empirical study on program failures of deep learning jobs

    Ru Zhang, Wencong Xiao, Hongyu Zhang, Yu Liu, Haoxiang Lin, and Mao Yang. An empirical study on program failures of deep learning jobs. InProceed- ings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE ’20, page 1159–1170, Seoul, South Korea, May 2020...

    2020

  43. [51]

    OPT: Open Pre-trained Trans- former Language Models, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoye...

    2022

  44. [52]

    An Empirical Study of Common Chal- lenges in Developing Deep Learning Applications

    Tianyi Zhang, Cuiyun Gao, Lei Ma, Michael Lyu, and Miryung Kim. An Empirical Study of Common Chal- lenges in Developing Deep Learning Applications. In 2019 IEEE 30th International Symposium on Software Reliability Engineering, ISSRE ’19, pages 104–115, Berlin, Germany, October...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.