Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis

Abhinav Goel; George K. Thiruvathukal; James C. Davis; Nikita Ravi

REVIEW 4 major objections 5 minor 62 references

Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A case study shows that three extra practices make deep learning models reproducible in practice.

desk verdict A genuine case-study contribution with honest disclosure, but the headline accuracy gains are built on single runs and post-hoc recipe selection, so the causal claim needs more support. read the letter →

arxiv 2505.03165 v1 pith:RBPC44SM submitted 2025-05-06 cs.LG cs.SE

Nikita Ravi , Abhinav Goel , James C. Davis , George K. Thiruvathukal This is my paper · ORCID

classification cs.LGcs.SE

keywords reproducibilitydeeplearningsoftwarecasestudysensitivityanalysisTRUNKhierarchicalneuralnetworktrainingpipelinetransparencydependencymanifest

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles the reproducibility problem in deep learning: even when source code and data are shared, results often cannot be recreated. Using the hierarchical TRUNK classifier as a case study, it shows that the standard reproducibility guidelines are insufficient. It argues that adding three practices — a dependency manifest listing only primary GPU-compatible packages, full disclosure of the data-processing and training pipeline, and a hyperparameter sensitivity analysis — substantially improves reproducibility. When these were applied, reproduced accuracy on EMNIST rose from 63.62% to 84.30%, and on CIFAR-10 from 67.61% to 81.53%. A sympathetic reader would care because if the claim holds, researchers who follow this extended checklist can save the weeks of effort currently needed to interpret incomplete instructions and can trust that a published model can actually be re-trained.

What carries the argument

The central object is TRUNK, the Tree-Based Unidirectional Neural Network, a hierarchical computer-vision model structured as a tree of small networks whose exact tree structure is built from the data using a 'grouping volatility' threshold. The key mechanism is the sensitivity analysis over this grouping-volatility hyperparameter: it reveals that small changes in the threshold drastically change both the tree geometry and the final accuracy, so an investigator can adjust for non-determinism and recover the published structure and performance. The manifest listing only primary GPU-compatible dependencies and the PyYAML configuration files disclosing the exact training and data-processing pipeline are the other two load-bearing mechanisms; together they turn an un-reproducible repository into one whose environment, pipeline, and hyperparameter choices are inspectable and adjustable.

What would settle it

Run the same three-step procedure on a diverse set of published deep learning repositories — for example, a transformer, a GAN, and a medical-imaging model — and measure how often the extended guidelines raise the reproduced accuracy closer to the published values relative to the original guidelines; if the gains are small or inconsistent outside hierarchical computer-vision models, the claimed general effectiveness of the extensions is falsified.

Watch

Extended reading notes

Core claim

The central claim is that the current reproducibility guidelines for deep learning software, extended with (1) a manifest listing only primary GPU-compatible dependencies, (2) full disclosure of the data-processing and training pipeline, and (3) sensitivity analysis of hyperparameters, improve the reproducibility of deep learning software. The paper demonstrates this by taking the Tree-Based Unidirectional Neural Network (TRUNK), a hierarchical classifier whose structure adapts to the dataset, and attempting to reproduce its published results from source code alone. It found that simply following the provided scripts and official manifests failed: the EMNIST accuracy reproduced at 63.62% versus the published 85.77%, and CIFAR-10 could not be trained at all because hyperparameters and pipeline details were missing. The authors then re-implemented the tree-building component, fixed the manifest, documented the pipeline in configuration files, and conducted a sensitivity analysis over the grouping-volatility hyperparameter. With these changes, EMNIST accuracy rose to 84.30% and CIFAR-10 accuracy rose from 67.61% to 81.53% across a series of revised training recipes, coming much closer to the original 85.77% and 91.99% respectively. The paper interprets these numbers as evidence that the extended guidelines diagnose and mitigate the specific reproducibility failures exposed by this case study.

Load-bearing premise

The central assumption is that TRUNK is a representative deep learning system, so lessons from reproducing it transfer to other models, pipelines, and domains; the paper itself notes this external threat, and the ground truth for "reproduced" is the original authors' own reported numbers and weights.

Editorial extensions

If this is right

If these extended guidelines are adopted, published deep learning repositories would carry a minimal GPU-compatible manifest, a configuration-file description of the training and data-processing pipeline, and a sensitivity analysis for sensitive hyperparameters, making re-training a deterministic expectation rather than a research project.
For hierarchical or adaptive neural networks like TRUNK, reproducibility requires disclosing not just the trained weights and training scripts but the structure-building algorithm itself, since the tree geometry is part of the model's behavior.
The roughly 14% accuracy gain from revising the CIFAR-10 training recipe shows that undisclosed pipeline choices cause errors of the same magnitude as algorithmic novelty, so documenting them is as important as sharing source code.
Sensitivity analysis of a single sensitive hyperparameter can serve as a practical substitute for full determinism: it shows which parameter to nudge when hardware- or software-induced randomness changes the result.
The paper's checklist, combining existing best practices with the three extensions, is concrete enough to be used as a template for other deep learning reproducibility studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same methodology could be tested prospectively on a randomly chosen set of published deep learning models to see whether the three extensions reliably predict which repositories reproduce; the paper only demonstrates them retrospectively on one model chosen for its complexity.
Because the sensitivity analysis targeted a single hyperparameter, grouping volatility, one might infer that other adaptive architectures harbor similar high-leverage parameters; identifying such parameters could be systematized into a reproducibility audit routine.
The finding that a 0.03 change in grouping volatility swings accuracy by up to 7% suggests that published accuracy claims for such models should be accompanied by a confidence interval over the sensitive parameter, not just a single number.
The co-authorship of a TRUNK author means the investigation had privileged access to the intended pipeline; a non-affiliated team attempting the same reproduction would test whether the documented recipe is sufficient on its own, which is the stronger version of the reproducibility claim.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick read: this is a genuine case-study contribution—nobody has audited TRUNK's reproducibility before, and the observation that the same seed on the same GPU model can yield different tree structures and a ~1.8-point accuracy spread is a useful data point. The authors also disclose the conflict of interest plainly in Section 3.6 and admit in Section 4.3 that full end-to-end reproduction was not achieved. That honesty counts.

What it does well: the empirical documentation of environment-setup failures (the manifest including unused packages), the inference verification of pre-trained weights, the grouping-volatility sensitivity sweeps, and the batch-size ablation are concrete and reproducible. The proposed guideline extensions—minimal GPU-compatible manifests, full pipeline disclosure, sensitivity analysis—are reasonable, though not especially novel; they largely restate existing recommendations with a case-study wrapper.

Where it softens: the central quantitative evidence for the extensions is thin. All key accuracy numbers are single runs. In Section 4.5/Figure 8 the paper itself shows a same-seed, same-GPU-type run-to-run difference of 81.53 vs 79.72 (about 1.8 points), and adjacent recipes in Table 7 differ by similar amounts. The 13.92-point TR1→TR6 gain comes from six recipes varied on multiple axes simultaneously, with TR6 selected post hoc. That is a selection result, not an independent confirmation. The SVHN tables also disagree: Table 5 reports 98.22 from original scripts, Table 6 reports 90.24 after the 'improved' pipeline, and the paper never explains the 8-point drop. Given the paper's own variance data, the stated conclusion that the extensions improve reproducibility is not yet empirically pinned down.

None of this invalidates the qualitative suggestions or the case-study narrative. But the paper should either run multi-seed experiments with variance, or pre-register the recipe-selection rule, or explicitly frame the recipes as exploratory. And it should address the SVHN discrepancy.

My sense: worth a serious referee. It's a well-structured empirical case study on a relevant topic, with new data and an honest limitations section. It needs major-to-moderate revision before the quantitative conclusions can stand. If I were the editor, I would send it to review, but I would push for multi-seed results and a revised conclusion.

For whom: people working on ML reproducibility and MLOps tooling; also as a teaching example in a software-engineering-for-ML course. I'd maybe bring it to reading group, but I wouldn't cite the accuracy gains.

Referee Report

4 major / 5 minor

Summary. The paper reports a case-study investigation of reproducibility practices for deep learning software, using the Tree-Based Unidirectional Neural Network (TRUNK) as the target system. The authors replicate the original environment, verify pre-trained weights, attempt to retrain TRUNK on EMNIST, CIFAR-10, and SVHN, and then introduce modifications: a minimal GPU-compatible dependency manifest, an end-to-end training implementation, a sensitivity analysis of the grouping-volatility hyperparameter, and explicit configuration files for the data-processing and training pipelines. They report accuracy improvements on EMNIST (63.62% to 84.30%) and CIFAR-10 (67.61% under TR1 to 81.53% under TR6) and conclude that the current reproducibility guidelines, extended with their three additions, improve the reproducibility of deep learning software. The paper also discloses that one co-author is a principal author of TRUNK and that the authors collaborated with that author during the investigation.

Significance. If the empirical claims were robust, the paper would be a useful contribution to the software-engineering reproducibility literature: it ships source code and configuration files, documents environment-setup failures and fixes, provides a sensitivity analysis for a non-standard hierarchical architecture, and explicitly discloses positionality. The suggested extensions—minimal GPU-compatible manifests, sensitivity analysis, and transparent training/data-processing pipelines—are reasonable and actionable. However, the quantitative evidence for the central claim is currently fragile because all reported accuracy numbers come from single runs without variance estimates, the CIFAR-10 improvement is based on post hoc selection among several recipes, and the SVHN results show an unexplained accuracy drop after the proposed changes. The significance of the paper therefore depends on whether the authors can provide multi-seed evidence and reconcile the inconsistent SVHN numbers.

major comments (4)

[§4.5, Figure 8, Table 7] The claim that the proposed pipeline changes improve accuracy is not supported by the reported measurements, because every accuracy number in Table 7 is a single run and the recipes were selected after observing results. Section 4.5 and Figure 8 show that two runs with the same seed, code, and pipeline on different A100 GPUs differed by 1.81 percentage points (81.53% versus 79.72%), which is the same order as several adjacent differences in Table 7. Without multiple seeds, confidence intervals, or a pre-registered recipe-selection rule, the 13.92-point TR1-to-TR6 improvement cannot be attributed to the guideline changes rather than to random variation and best-of-many selection. The authors should report per-recipe variance across seeds and either pre-specify the recipe search or frame the comparison as exploratory.
[Table 6, §4.4, SVHN results] The SVHN results contradict the paper's narrative. Table 5 reports 98.22% accuracy after training with the original scripts, while Table 6 reports 90.24% after the supposedly improved end-to-end pipeline, and the original pre-trained weights give 96.75%. This is an unexplained 8-point drop relative to the original scripts and a 6.5-point drop relative to the pre-trained weights, yet Section 4.4 discusses only EMNIST and CIFAR-10 and never addresses SVHN. The authors need to explain why the enhanced, more transparent pipeline performs worse on SVHN, or the conclusion that these practices improve reproducibility is directly undermined.
[§4.3, §4.4, Table 6] The EMNIST comparison is not an apples-to-apples test of the proposed guidelines. Section 4.3 reports 63.62% using the original scripts with a pre-defined tree, while Section 4.4 reports 84.30% after integrating the tree-building code; however, Section 4.4 also states that the reproduced tree structure still differs from the original (Figure 5). Because the paper itself acknowledges that structural reproducibility was not achieved, reporting only the accuracy gain overstates what the end-to-end implementation contributes to reproducibility. The authors should report structural agreement explicitly and separate the effects of the new implementation from the effects of the guideline extensions.
[§3.6, §4.2] The ground truth used to judge reproducibility—the pre-trained weights and reported accuracies from TRUNK [15,16]—was produced by the same research group that includes a co-author of this paper, and Section 3.6 discloses active collaboration with the TRUNK principal author during the study. This is an independence risk for the verification claims in Section 4.2, because the 'original results' being reproduced are not independent of the investigating team. The authors should either obtain an independent external replication of the pre-trained-weight verification or explicitly bound the conclusions as a self-study, noting that the case-study findings may reflect insider knowledge rather than the sufficiency of the published guidelines.

minor comments (5)

[Listing 3, Listing 4] The YAML listings appear corrupted in the manuscript text (for example, 'tr an sf or m' and 'o pt im ize r'), which makes the configuration examples difficult to read; please ensure the final PDF renders them with proper spacing and indentation.
[Table 5] The entry '–.–' for CIFAR-10 in Table 5 should be explicitly explained in the caption, indicating that training was not completed because hyperparameters were missing, rather than leaving the notation undeclared.
[§4.5] The sentence 'the same seed, training regime, and data processing pipeline were used from for Training Regime 6' contains a typo ('from for') and should be corrected to 'used for Training Regime 6'.
[§4.6.2] The text refers to 'PyY AML' configuration files; this appears to be a typo for 'PyYAML', which should be fixed throughout the section.
[Table 7] The 'RA.' abbreviation for Random Augmentation appears with an inconsistent period, and the column ordering in the table makes it hard to map the checkmarks to the abbreviation list; please reformat the header and legend for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's quantitative results are empirical measurements from running the case study, not consequences of its guideline definitions or self-citations.

full rationale

The paper makes no first-principles derivation of its results. Its central claim is that certain reproducibility practices (primary GPU-compatible dependency manifest, disclosed data-processing/training pipeline, sensitivity analysis) help diagnose and improve reproducibility, and this claim is supported by executed experiments (Tables 4–7, Figures 6–11). These accuracies are measurements produced by training and evaluating the TRUNK network, not quantities forced by the definitions of the guidelines. The self-referential relationship is explicitly disclosed in Section 3.6: 'One of the contributors to this paper also served as the principal author of the Tree-Based Unidirectional Neural Network (TRUNK)', and the authors state they 'collaborated with this author to gain insights and validate findings'. This is a real independence concern, but it is not circularity because the ground-truth comparison is made by running the provided pre-trained weights (Section 4.2: 'We found that the pre-trained weights correspond to the results reported in the paper'), which is an independently executable artifact rather than a fitted value of this paper. Likewise, the paper's own disclosures undermine any reading that the results are true by construction: Section 4.3 admits 'we were unable to achieve end-to-end reproduction of the training of the TRUNK network', Table 6 shows SVHN accuracy dropping from 98.22% to 90.24% after the 'improved' pipeline, and Section 6 notes 'the selected case study, TRUNK, may not represent all DL algorithms'. These are validity and generalizability limitations, not evidence that any prediction reduces to its input. The main weaknesses—single-run measurements, post-hoc selection of training recipe TR6, and the non-independent provenance of the case study and its reported baseline—are correctness and independence risks, which fall outside the circularity criteria defined for this analysis. No circular step can be exhibited via the paper's own equations or definitions; therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical case study, so the ledger holds no formal derivation axioms. The supporting premises are domain assumptions about evaluation metrics, the trustworthiness of the original TRUNK weights, the generalizability of a single case study, and the dominance of one hyperparameter in the sensitivity analysis. The paper's headline accuracy improvements depend on hand-selected training recipes and a chosen sensitivity sweep, which act as implicit free parameters.

free parameters (2)

CIFAR-10 training recipe TR6 (from Table 7) = LR 2.36e-4, BS 500, GV 0.88, CutOut + RandomAug, acc 81.53%
The authors varied six recipes until accuracy improved; the 13.92% gain over TR1 is a post hoc selection, not a pre-registered prediction, and is specific to this hand-chosen combination.
Optimal grouping volatility for CIFAR-10 = 1.02 within sweep 0.60 to 1.20 in steps of 0.03
Section 4.5 selects GV 1.02 because it produced the highest observed accuracy (67.61%); the 'optimal' value is read off the fitted curve, not predicted from theory.

assumptions (4)

domain assumption Accuracy on a held-out test set is a valid measure of successful reproduction of a deep learning model.
Used throughout Sections 4.2 to 4.6 to compare original and reproduced results; the paper does not consider other fidelity metrics such as output distributions or calibration.
domain assumption The pre-trained weights and reported accuracies from the original TRUNK papers [15,16] are correct and constitute the benchmark for a successful reproduction.
Section 4.2 treats inference accuracy of these weights as the ground truth; because a TRUNK author is a co-author (Section 3.6), this benchmark is not independent of the investigating team.
domain assumption TRUNK is representative of deep learning software, so conclusions from this single case study generalize to other DL models.
Section 3.2 selects TRUNK for its complexity; Section 6 External acknowledges it 'may not represent all DL algorithms', so generalizability is an assumed premise for answering RQ1 and RQ2.
domain assumption Grouping volatility is the dominant hyperparameter controlling TRUNK's tree structure; other hyperparameters are held constant during the sensitivity analysis.
Section 4.5 sweeps only grouping volatility and attributes tree-shape and accuracy changes to it; if initialization or other hyperparameters dominate, the sensitivity conclusions could differ.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis." pith.science (2026). https://pith.science/paper/RBPC44SM

@misc{pith2026250503165,
  author       = {Pith},
  title        = {Pith review of: Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RBPC44SM}},
  note         = {Machine review of arXiv:2505.03165}
}

read the original abstract

The field of deep learning has witnessed significant breakthroughs, spanning various applications, and fundamentally transforming current software capabilities. However, alongside these advancements, there have been increasing concerns about reproducing the results of these deep learning methods. This is significant because reproducibility is the foundation of reliability and validity in software development, particularly in the rapidly evolving domain of deep learning. The difficulty of reproducibility may arise due to several reasons, including having differences from the original execution environment, incompatible software libraries, proprietary data and source code, lack of transparency, and the stochastic nature in some software. A study conducted by the Nature journal reveals that more than 70% of researchers failed to reproduce other researchers experiments and over 50% failed to reproduce their own experiments. Irreproducibility of deep learning poses significant challenges for researchers and practitioners. To address these concerns, this paper presents a systematic approach at analyzing and improving the reproducibility of deep learning models by demonstrating these guidelines using a case study. We illustrate the patterns and anti-patterns involved with these guidelines for improving the reproducibility of deep learning models. These guidelines encompass establishing a methodology to replicate the original software environment, implementing end-to-end training and testing algorithms, disclosing architectural designs, and enhancing transparency in data processing and training pipelines. We also conduct a sensitivity analysis to understand the model performance across diverse conditions. By implementing these strategies, we aim to bridge the gap between research and practice, so that innovations in deep learning can be effectively reproduced and deployed within software.

Figures

Figures reproduced from arXiv: 2505.03165 by the authors.

**Figure 2.** (a) Monolithic Architectures vs (b) TRUNK [16]. We will use TRUNK, a type of hierarchical neural network, as a case study to demonstrate our guidelines [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Experiment Methodology used to Test the Reproducibility of TRUNK [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (10 more)

**Figure 4.** Figure 4: We choose TRUNK for our case study analysis due to its complexity of the network architecture design varying by dataset (a) EMNIST (b) CIFAR-10 (c) [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Comparison of the EMNIST (a) Tree Structure Developed by [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Ablation study analyzing the sensitivity of the grouping volatility hyperparameter for CIFAR-10. Each data point color represents a specific tree built. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Analyzing the sensitivity of the grouping volatility parameter by changing the original implementation to include layer normalization. Each data point [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: Analyzing how using different NVIDIA A100 GPUs and the same hyperparameters can change the structure and performance of a model, with (a) Structure of the Tree from the First Execution of the Re-implemented TRUNK software with an accuracy of 81.53% and (b) the Reproduc…

**Figure 9.** Figure 9: Differences in the TRUNK tree structures between (a) the structure achieving the highest accuracy and (b) the reported tree structure by the original authors of TRUNK. nal authors [15, 16] achieved. The original authors of TRUNK set grouping volatility = 1, but in our …

**Figure 10.** Figure 10: A slight change in a hyperparameter—such as adjusting the group [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

**Figure 11.** Figure 11: Ablation over the batch sizes with revised data augmentations [56] and the same tree network as the authors for the CIFAR-10 dataset. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

**Figure 12.** Figure 12: Example of different augmentation techniques to illustrate the transformations undergone by the data sample: [Left to Right, Top to Bottom] (a) The original image before augmentations (b) The image after normalization (c) The image after applying random crop (d) The …

**Figure 13.** Figure 13: Documenting the links to the Pre-Trained Weights and a summary [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 32 canonical work pages

[1]

Challenges and practices of deep learning model reengi- neering: A case study on computer vision,

W. Jiang, V . Banna, N. Vivek, A. Goel, N. Synovic, G. K. Thiruvathukal, and J. C. Davis, “Challenges and practices of deep learning model reengi- neering: A case study on computer vision,”Empirical Software Engineer- ing, vol. 29, no. 6, pp. 142, 2024. Springer

work page 2024
[2]

A Partial Replication of MaskFormer in TensorFlow on TPUs for the TensorFlow Model Garden

V . Purohit, W. Jiang, A. R. Ravikiran, J. C. Davis, A partial replication of maskformer in tensorflow on tpus for the tensorflow model garden (2024). arXiv:2404.18801

work page Pith review arXiv 2024
[3]

Hutson, Missing data hinder replication of artificial intelligence studies

M. Hutson, Missing data hinder replication of artificial intelligence studies. URL https : / / www.science.org / content / article / missing - data - hinder - replication - artificial - intelligence - studies

work page
[4]

Lemay, K

A. Lemay, K. Hoebel, C. P. Bridge, B. Befano, S. De Sanjos ´e, D. Ege- men, A. C. Rodriguez, M. Schiffman, J. P. Campbell, J. Kalpathy-Cramer, Improving the repeatability of deep learning models with monte carlo dropout, NPJ Digit Med 5 (1) (2022) 174

work page 2022
[5]

Artrith, K

N. Artrith, K. T. Butler, F.-X. Coudert, S. Han, O. Isayev, A. Jain, A. Walsh, Best practices in machine learning for chemistry (2021). URL https://www .nature.com/articles/s41557- 021- 00716- z#citeas

work page 2021
[6]

B. Chen, M. Wen, Y . Shi, D. Lin, G. K. Rajbahadur, Z. M. Jiang, To- wards training reproducible deep learning models, CoRR abs/2202.02326 (2022). arXiv:2202.02326. URL https://arxiv.org/abs/2202.02326

work page Pith review arXiv 2022
[7]

Haibe-Kains, G

B. Haibe-Kains, G. A. Adam, A. Hosny, F. Khodakarami, T. Shrad- dha, R. Kusko, S.-A. Sansone, W. Tong, R. D. Wolfinger, C. E. Mason, W. Jones, J. Dopazo, C. Furlanello, L. Waldron, B. Wang, C. McIn- tosh, A. Goldenberg, A. Kundaje, C. S. Greene, T. Broderick, M. M. Hoffman, J. T. Leek, K. Korthauer, W. Huber, A. Brazma, J. Pineau, R. Tibshirani, T. Hastie...

2020
[8]

Isdahl, O

R. Isdahl, O. E. Gundersen, Out-of-the-box reproducibility: A sur- vey of machine learning platforms, in: 2019 15th International Con- ference on eScience (eScience), 2019, pp. 86–95. doi : 10.1109 / eScience.2019.00017

arXiv 2019

Show all 62 references

[9]

Pineau, P

J. Pineau, P. Vincent-Lamarre, K. Sinha, V . Larivi `ere, A. Beygelzimer, F. d’Alch´e-Buc, E. B. Fox, H. Larochelle, Improving reproducibility in machine learning research (A report from the neurips 2019 reproducibility program), CoRR abs/2003.12206 (2020). arXiv:2003.12206. U...

2020 arXiv
[10]

Semmelrock, S

H. Semmelrock, S. Kopeinik, D. Theiler, T. Ross-Hellauer, D. Kowald, Reproducibility in machine learning-driven research (2023). arXiv: 2307.10320

2023 arXiv
[11]

Baker, 1,500 scientists lift the lid on reproducibility (2016)

M. Baker, 1,500 scientists lift the lid on reproducibility (2016). URL https://www.nature.com/articles/533452a

2016
[12]

Gibney, Could machine learning fuel a reproducibility crisis in sci- ence? (2022)

E. Gibney, Could machine learning fuel a reproducibility crisis in sci- ence? (2022). URL https://www.nature.com/articles/d41586-022-02035-w

2022
[13]

O. E. Gundersen, S. Kjensmo, State of the art: Reproducibility in artifi- cial intelligence, Proceedings of the AAAI Conference on Artificial Intel- ligence 32 (1) (Apr. 2018). doi:10.1609/aaai.v32i1.11503. URL https : / / ojs.aaai.org / index.php / AAAI / article / view / 11503

2018 doi
[14]

Bouthillier, C

X. Bouthillier, C. Laurent, P. Vincent, Unreproducible research is reproducible, in: K. Chaudhuri, R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning, V ol. 97 of Proceedings of Machine Learning Research, PMLR, 2019, pp. 725–734. URL ...

2019
[15]

A. Goel, S. Aghajanzadeh, C. Tung, S.-H. Chen, G. K. Thiruvathukal, Y .- H. Lu, Modular Neural Networks for Low-Power Image Classification on Embedded Devices, ACM Transactions on Design Automation of Elec- tronic Systems 26 (1) (2020) 1:1–1:35. doi:10.1145/3408062. URL https:...

2020 doi
[16]

A. Goel, C. Tung, N. Eliopoulos, G. K. Thiruvathukal, A. Wang, Y .- H. Lu, J. C. Davis, Tree-based unidirectional neural networks for low- power computer vision, IEEE Design & Test 40 (3) (2023) 53–61. doi: 10.1109/MDAT.2022.3217016

2023
[17]

Reusing deep learning models: Challenges and di- rections in software engineering,

J. C. Davis, P. Jajal, W. Jiang, T. R. Schorlemmer, N. Synovic, and G. K. Thiruvathukal, “Reusing deep learning models: Challenges and di- rections in software engineering,” in Proceedings of the 2023 IEEE John Vincent Atanasoff International Symposium on Modern Computing (JVA...

2023
[18]

Frakes, K

W. Frakes, K. Kang, Software reuse research: status and future, IEEE Transactions on Software Engineering 31 (7) (2005) 529–536. doi: 10.1109/TSE.2005.85

2005 doi
[19]

C. W. Krueger, Software reuse, ACM Comput. Surv. 24 (2) (1992) 131–183. doi:10.1145/130844.130856. URL https://doi.org/10.1145/130844.130856

1992
[20]

Amershi, A

S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagap- pan, B. Nushi, T. Zimmermann, Software engineering for machine learn- ing: a case study, in: Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE-...

2019
[21]

S. S. Alahmari, D. B. Goldgof, P. R. Mouton, L. O. Hall, Challenges for the repeatability of deep learning models, IEEE Access 8 (2020) 211860– 211868. doi:10.1109/ACCESS.2020.3039833

2020
[22]

Karpathy, Software 2.0 (2017)

A. Karpathy, Software 2.0 (2017). URL https : / / karpathy .medium.com / software - 2 - 0 - a64152b37c35

2017
[23]

Pouchard, Y

L. Pouchard, Y . Lin, H. Van Dam, Replicating machine learning experi- ments in materials science (2020). arXiv:10.3233/APC200105

2020 doi
[24]

Ahmed, J

H. Ahmed, J. Lofstead, Managing randomness to enable reproducible machine learning, in: Proceedings of the 5th International Workshop on Practical Reproducible Evaluation of Computer Systems, P-RECS ’22, Association for Computing Machinery, New York, NY , USA, 2022, p. 15–20. ...

2022
[25]

Antunes, D

B. Antunes, D. R. C. Hill, Reproducibility, energy e fficiency and per- formance of pseudorandom number generators in machine learning: a comparative study of python, numpy, tensorflow, and pytorch implemen- tations (2024). arXiv:2401.17345

2024 arXiv
[26]

URL https : / / docs.conda.io / projects / conda / en / latest / user-guide/getting-started .html

Anaconda, Getting started with conda. URL https : / / docs.conda.io / projects / conda / en / latest / user-guide/getting-started .html
[27]

URL https://www.docker.com/

Docker, Docker (2013). URL https://www.docker.com/

2013
[28]

Boettiger, An introduction to docker for reproducible research, SIGOPS Oper

C. Boettiger, An introduction to docker for reproducible research, SIGOPS Oper. Syst. Rev. 49 (1) (2015) 71–79. doi : 10.1145 / 2723872.2723882. URL https://doi.org/10.1145/2723872.2723882

2015
[29]

Ocean, Code ocean

C. Ocean, Code ocean. URL https://codeocean.com/
[30]

URL https://elifesciences .org/labs/bdbeac92/gigantum- a- simple- way- to- create- and- share- reproducible- data- science-and-research

Gigantum, Gigantum – a simple way to create and share reproducible data science and research (2018). URL https://elifesciences .org/labs/bdbeac92/gigantum- a- simple- way- to- create- and- share- reproducible- data- science-and-research

2018
[31]

URL https://colab.research.google.com/

Google, Colab research. URL https://colab.research.google.com/
[32]

H. V . Pham, S. Quan, J. Wang, T. Lutellier, J. Rosenthal, L. Tan, Y . Yu, N. Nagappan, Problems and opportunities in training deep learning soft- ware systems: An analysis of variance (2020)

2020
[33]

URL https : / / docs.nvidia.com / deeplearning / performance / dl-performance-matrix-multiplication/index .html

NVIDIA, Matrix multiplication background user’s guide (2023). URL https : / / docs.nvidia.com / deeplearning / performance / dl-performance-matrix-multiplication/index .html

2023
[34]

URL https : / / pytorch .org / docs / stable / notes / randomness.html

PyTorch, Reproducibility. URL https : / / pytorch .org / docs / stable / notes / randomness.html
[35]

Goldberg, What every computer scientist should know about floating- point arithmetic, ACM Comput

D. Goldberg, What every computer scientist should know about floating- point arithmetic, ACM Comput. Surv. 23 (1) (1991) 5–48.doi:10.1145/ 103162.103163. URL https://doi.org/10.1145/103162.103163

1991
[36]

Stojnic, Ml code completeness checklist (2020)

R. Stojnic, Ml code completeness checklist (2020). URL https : / / medium .com / paperswithcode / ml - code - completeness-checklist-e9127b168501 15

2020
[37]

Matsubara, torchdistill: A modular, configuration-driven framework for knowledge distillation, CoRR abs /2011.12913 (2020)

Y . Matsubara, torchdistill: A modular, configuration-driven framework for knowledge distillation, CoRR abs /2011.12913 (2020). arXiv : 2011.12913. URL https://arxiv.org/abs/2011.12913

2020 arXiv
[38]

Krizhevsky, I

A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, Commun. ACM 60 (6) (2017) 84–90. doi:10.1145/3065386. URL https://doi.org/10.1145/3065386

2017 doi
[39]

OpenAI, A. Josh, A. Steven, Gpt-4 technical report (2024). arXiv: 2303.08774

2024 arXiv
[40]

Hamblen, Update: Chatgpt runs 10k nvidia training gpus with poten- tial for thousands more (2023)

M. Hamblen, Update: Chatgpt runs 10k nvidia training gpus with poten- tial for thousands more (2023). URL https://www .fierceelectronics.com/sensors/chatgpt- runs-10k-nvidia-training-gpus-potential-thousands-more

2023
[41]

Singh, U

S. Singh, U. Ahuja, M. Kumar, K. Kumar, M. Sachdeva, Face mask detec- tion using YOLOv3 and faster R-CNN models: COVID-19 environment, Multimedia Tools and Applications 80 (13) (2021) 19753–19768

2021
[42]

D. Maji, S. Nagori, M. Mathew, D. Poddar, Yolo-pose: Enhancing yolo for multi person pose estimation using object keypoint similarity loss (2022). arXiv:2204.06806

2022 arXiv
[43]

Cordts, M

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benen- son, U. Franke, S. Roth, B. Schiele, The cityscapes dataset for seman- tic urban scene understanding, CoRR abs /1604.01685 (2016). arXiv: 1604.01685. URL http://arxiv.org/abs/1604.01685

2016 arXiv
[44]

X. Zhu, M. Bain, B-cnn: branch convolutional neural network for hierar- chical classification, arXiv preprint arXiv:1709.09890 (2017)

2017 arXiv
[45]

Z. Yan, H. Zhang, R. Piramuthu, V . Jagadeesh, D. DeCoste, W. Di, Y . Yu, Hd-cnn: Hierarchical deep convolutional neural networks for large scale visual recognition, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015

2015
[46]

D. Roy, P. Panda, K. Roy, Tree-CNN: A Hierarchical Deep Convolutional Neural Network for Incremental Learning, arXiv:1802.05800 [cs, eess, stat]ArXiv: 1802.05800 (Feb. 2018). URL http://arxiv.org/abs/1802.05800

2018 arXiv
[47]

Simonyan, A

K. Simonyan, A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv:1409.1556 [cs]ArXiv: 1409.1556 (Sep. 2014). URL http://arxiv.org/abs/1409.1556

2014 arXiv
[48]

Cohen, S

G. Cohen, S. Afshar, J. Tapson, A. van Schaik, EMNIST: an extension of MNIST to handwritten letters (Feb. 2017). URL https://arxiv.org/abs/1702.05373v2

2017 arXiv
[49]

Krizhevsky, Learning multiple layers of features from tiny images, Tech

A. Krizhevsky, Learning multiple layers of features from tiny images, Tech. rep., University of Toronto (2009). URL https://www .cs.toronto.edu/~kriz/learning- features- 2009-TR.pdf

2009
[50]

Netzer, T

Y . Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y . Ng, Reading digits in natural images with unsupervised feature learning (2011)

2011
[51]

K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, arXiv:1512.03385 [cs]ArXiv: 1512.03385 (Dec. 2015). URL http://arxiv.org/abs/1512.03385

2015 arXiv
[52]

Krizhevsky, I

A. Krizhevsky, I. Sutskever, G. E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, Advances in Neural Information Processing Systems 25 (2012) 1097–1105. URL https : / / proceedings.neurips.cc / paper / 2012 / hash / c399862d3b9d6b76c8436e924a68c45b-Ab...

2012
[53]

J. L. Ba, J. R. Kiros, G. E. Hinton, Layer normalization (2016). arXiv: 1607.06450

2016 arXiv
[54]

Touvron, M

H. Touvron, M. Cord, H. J ´egou, Deit iii: Revenge of the vit (2022). arXiv:2204.07118

2022 arXiv
[55]

Steiner, A

A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, L. Beyer, How to train your vit? data, augmentation, and regularization in vision transformers (2022). arXiv:2106.10270

2022 arXiv
[56]

URL https://github.com/kuangliu/pytorch-cifar

kuangliu, Train cifar10 with pytorch (2020). URL https://github.com/kuangliu/pytorch-cifar

2020
[57]

Io ffe, C

S. Io ffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, CoRR abs /1502.03167 (2015). arXiv:1502.03167. URL http://arxiv.org/abs/1502.03167

2015 arXiv
[58]

E. D. Cubuk, B. Zoph, J. Shlens, Q. V . Le, Randaugment: Practi- cal automated data augmentation with a reduced search space (2019). arXiv:1909.13719

2019 arXiv
[59]

DeVries, G

T. DeVries, G. W. Taylor, Improved regularization of convolutional neural networks with cutout, arXiv preprint arXiv:1708.04552 (2017)

2017 arXiv
[60]

URL https://pip.pypa.io/en/stable/getting-started/

PyPI, pip documentation. URL https://pip.pypa.io/en/stable/getting-started/
[61]

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, H. Adam, MobileNets: E fficient Convolutional Neural Networks for Mobile Vision Applications, arXiv:1704.04861 [cs]ArXiv: 1704.04861 (Apr. 2017). URL http://arxiv.org/abs/1704.04861

2017 arXiv
[62]

Oquab, T

M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khali- dov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Bal- las, W. Galuba, R. Howes, P.-Y . Huang, S.-W. Li, I. Misra, M. Rabbat, V . Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut,...

2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Challenges and practices of deep learning model reengi- neering: A case study on computer vision,

W. Jiang, V . Banna, N. Vivek, A. Goel, N. Synovic, G. K. Thiruvathukal, and J. C. Davis, “Challenges and practices of deep learning model reengi- neering: A case study on computer vision,”Empirical Software Engineer- ing, vol. 29, no. 6, pp. 142, 2024. Springer

work page 2024

[2] [2]

A Partial Replication of MaskFormer in TensorFlow on TPUs for the TensorFlow Model Garden

V . Purohit, W. Jiang, A. R. Ravikiran, J. C. Davis, A partial replication of maskformer in tensorflow on tpus for the tensorflow model garden (2024). arXiv:2404.18801

work page Pith review arXiv 2024

[3] [3]

Hutson, Missing data hinder replication of artificial intelligence studies

M. Hutson, Missing data hinder replication of artificial intelligence studies. URL https : / / www.science.org / content / article / missing - data - hinder - replication - artificial - intelligence - studies

work page

[4] [4]

Lemay, K

A. Lemay, K. Hoebel, C. P. Bridge, B. Befano, S. De Sanjos ´e, D. Ege- men, A. C. Rodriguez, M. Schiffman, J. P. Campbell, J. Kalpathy-Cramer, Improving the repeatability of deep learning models with monte carlo dropout, NPJ Digit Med 5 (1) (2022) 174

work page 2022

[5] [5]

Artrith, K

N. Artrith, K. T. Butler, F.-X. Coudert, S. Han, O. Isayev, A. Jain, A. Walsh, Best practices in machine learning for chemistry (2021). URL https://www .nature.com/articles/s41557- 021- 00716- z#citeas

work page 2021

[6] [6]

B. Chen, M. Wen, Y . Shi, D. Lin, G. K. Rajbahadur, Z. M. Jiang, To- wards training reproducible deep learning models, CoRR abs/2202.02326 (2022). arXiv:2202.02326. URL https://arxiv.org/abs/2202.02326

work page Pith review arXiv 2022

[7] [7]

Haibe-Kains, G

B. Haibe-Kains, G. A. Adam, A. Hosny, F. Khodakarami, T. Shrad- dha, R. Kusko, S.-A. Sansone, W. Tong, R. D. Wolfinger, C. E. Mason, W. Jones, J. Dopazo, C. Furlanello, L. Waldron, B. Wang, C. McIn- tosh, A. Goldenberg, A. Kundaje, C. S. Greene, T. Broderick, M. M. Hoffman, J. T. Leek, K. Korthauer, W. Huber, A. Brazma, J. Pineau, R. Tibshirani, T. Hastie...

2020

[8] [8]

Isdahl, O

R. Isdahl, O. E. Gundersen, Out-of-the-box reproducibility: A sur- vey of machine learning platforms, in: 2019 15th International Con- ference on eScience (eScience), 2019, pp. 86–95. doi : 10.1109 / eScience.2019.00017

arXiv 2019