{"id":"86796d23-d8f3-4ea0-b64d-b766caca7dbf","arxiv_id":"2505.03165","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The authors reproduce the TRUNK neural network across three datasets, find that missing training details cause large accuracy gaps, and extend existing reproducibility guidelines with sensitivity analysis and minimal dependency manifests.","lead":"This paper studies why deep learning software is hard to reproduce by trying to re-run a specific neural network called TRUNK on three image datasets. It reports that even with code and data, missing details about training settings caused big accuracy gaps, and it proposes better documentation and sensitivity analysis as fixes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported accuracy gains rest on single runs and post-hoc recipe selection; without multi-seed variance or a pre-registered selection rule, the paper's central reproducibility claim is not yet empirically supported.","rationale":"The reader's conditional verdict already identified single-run reporting, post-hoc recipe selection, and the SVHN discrepancy, and my stress-test confirms that these are the load-bearing weaknesses. I focus on a slightly different aspect than the reader's stated weakest assumption: the reader emphasized representativeness and the non-independence of the ground truth, whereas the more immediately decisive issue is that the empirical support for the Section 5 claim is statistically unstable. The paper is otherwise transparent: it discloses the co-author relationship in Section 3.6, admits in Section 4.3 that the original tree-construction code was missing and full end-to-end reproduction was not achieved, and acknowledges external validity limits in Section 6. These explicit limitations should be weighed as the authors themselves flag them. The qualitative recommendations—minimal GPU-compatible manifests, pipeline disclosure, sensitivity analysis—remain reasonable and are usefully demonstrated as a worked example, so the paper is not unsalvageable. However, the headline quantitative claims should be treated as preliminary until multi-seed replication with a pre-registered selection rule is provided. Since the reader's verdict is already CONDITIONAL, my analysis does not move the verdict; it sharpens the condition that should be attached.","tokens_in":18824,"tokens_out":6318,"duration_ms":60739,"concrete_test":"Rerun the full pipeline for CIFAR-10 TR1 and TR6, EMNIST end-to-end, and SVHN with at least 10 independent seeds on identical A100 hardware, reporting per-condition means and standard deviations. Pre-register the recipe-selection rule (for example, choose the recipe with the highest validation mean) before evaluating on the held-out test set. If the TR6 mean does not exceed the TR1 mean by more than the pooled within-condition standard deviation, or if the EMNIST mean does not approach the original 85.77%, the reported accuracy gains should be attributed to noise and post-hoc selection rather than to the proposed guidelines. Record the SVHN result in the same rerun to determine whether 90.24% is reproducible or whether Table 6 contains a reporting error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 5 is that the proposed guideline extensions improve reproducibility, with quantitative support from the EMNIST gain (63.62% to 84.30%, Section 4.4) and the CIFAR-10 gain (67.61% under TR1 to 81.53% under TR6, Section 4.6.1, Table 7). Every key number is a single training run, and TR6 was selected after comparing multiple recipes that simultaneously vary learning rate, batch size, grouping volatility, augmentations, and normalization statistics. The paper itself demonstrates in Section 4.5 and Figure 8 that two runs with the same seed, code, and pipeline on different A100 GPUs produced 81.53% versus 79.72% accuracy and different tree structures. A roughly 1.8 percentage-point run-to-run spread is the same order as several differences between adjacent recipes in Table 7, so the best-of-many selection can inflate the reported 13.92-point TR1-to-TR6 improvement even if none of the guideline changes has a causal effect. The SVHN results add a further inconsistency: Table 5 reports 98.22% from the original training scripts, while Table 6 reports 90.24% after the 'improved' end-to-end pipeline, an unexplained 8-point drop that the paper never discusses. Section 4.3 also admits that full end-to-end reproduction was not achieved. Thus the quantitative evidence for the central claim is not yet stable enough to support the conclusions as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a case-study investigation of reproducibility practices for deep learning software, using the Tree-Based Unidirectional Neural Network (TRUNK) as the target system. The authors replicate the original environment, verify pre-trained weights, attempt to retrain TRUNK on EMNIST, CIFAR-10, and SVHN, and then introduce modifications: a minimal GPU-compatible dependency manifest, an end-to-end training implementation, a sensitivity analysis of the grouping-volatility hyperparameter, and explicit configuration files for the data-processing and training pipelines. They report accuracy improvements on EMNIST (63.62% to 84.30%) and CIFAR-10 (67.61% under TR1 to 81.53% under TR6) and conclude that the current reproducibility guidelines, extended with their three additions, improve the reproducibility of deep learning software. The paper also discloses that one co-author is a principal author of TRUNK and that the authors collaborated with that author during the investigation.","tokens_in":19093,"tokens_out":3098,"duration_ms":31187,"significance":"If the empirical claims were robust, the paper would be a useful contribution to the software-engineering reproducibility literature: it ships source code and configuration files, documents environment-setup failures and fixes, provides a sensitivity analysis for a non-standard hierarchical architecture, and explicitly discloses positionality. The suggested extensions—minimal GPU-compatible manifests, sensitivity analysis, and transparent training/data-processing pipelines—are reasonable and actionable. However, the quantitative evidence for the central claim is currently fragile because all reported accuracy numbers come from single runs without variance estimates, the CIFAR-10 improvement is based on post hoc selection among several recipes, and the SVHN results show an unexplained accuracy drop after the proposed changes. The significance of the paper therefore depends on whether the authors can provide multi-seed evidence and reconcile the inconsistent SVHN numbers.","major_comments":[{"comment":"The claim that the proposed pipeline changes improve accuracy is not supported by the reported measurements, because every accuracy number in Table 7 is a single run and the recipes were selected after observing results. Section 4.5 and Figure 8 show that two runs with the same seed, code, and pipeline on different A100 GPUs differed by 1.81 percentage points (81.53% versus 79.72%), which is the same order as several adjacent differences in Table 7. Without multiple seeds, confidence intervals, or a pre-registered recipe-selection rule, the 13.92-point TR1-to-TR6 improvement cannot be attributed to the guideline changes rather than to random variation and best-of-many selection. The authors should report per-recipe variance across seeds and either pre-specify the recipe search or frame the comparison as exploratory.","section":"§4.5, Figure 8, Table 7"},{"comment":"The SVHN results contradict the paper's narrative. Table 5 reports 98.22% accuracy after training with the original scripts, while Table 6 reports 90.24% after the supposedly improved end-to-end pipeline, and the original pre-trained weights give 96.75%. This is an unexplained 8-point drop relative to the original scripts and a 6.5-point drop relative to the pre-trained weights, yet Section 4.4 discusses only EMNIST and CIFAR-10 and never addresses SVHN. The authors need to explain why the enhanced, more transparent pipeline performs worse on SVHN, or the conclusion that these practices improve reproducibility is directly undermined.","section":"Table 6, §4.4, SVHN results"},{"comment":"The EMNIST comparison is not an apples-to-apples test of the proposed guidelines. Section 4.3 reports 63.62% using the original scripts with a pre-defined tree, while Section 4.4 reports 84.30% after integrating the tree-building code; however, Section 4.4 also states that the reproduced tree structure still differs from the original (Figure 5). Because the paper itself acknowledges that structural reproducibility was not achieved, reporting only the accuracy gain overstates what the end-to-end implementation contributes to reproducibility. The authors should report structural agreement explicitly and separate the effects of the new implementation from the effects of the guideline extensions.","section":"§4.3, §4.4, Table 6"},{"comment":"The ground truth used to judge reproducibility—the pre-trained weights and reported accuracies from TRUNK [15,16]—was produced by the same research group that includes a co-author of this paper, and Section 3.6 discloses active collaboration with the TRUNK principal author during the study. This is an independence risk for the verification claims in Section 4.2, because the 'original results' being reproduced are not independent of the investigating team. The authors should either obtain an independent external replication of the pre-trained-weight verification or explicitly bound the conclusions as a self-study, noting that the case-study findings may reflect insider knowledge rather than the sufficiency of the published guidelines.","section":"§3.6, §4.2"}],"minor_comments":[{"comment":"The YAML listings appear corrupted in the manuscript text (for example, 'tr an sf or m' and 'o pt im ize r'), which makes the configuration examples difficult to read; please ensure the final PDF renders them with proper spacing and indentation.","section":"Listing 3, Listing 4"},{"comment":"The entry '–.–' for CIFAR-10 in Table 5 should be explicitly explained in the caption, indicating that training was not completed because hyperparameters were missing, rather than leaving the notation undeclared.","section":"Table 5"},{"comment":"The sentence 'the same seed, training regime, and data processing pipeline were used from for Training Regime 6' contains a typo ('from for') and should be corrected to 'used for Training Regime 6'.","section":"§4.5"},{"comment":"The text refers to 'PyY AML' configuration files; this appears to be a typo for 'PyYAML', which should be fixed throughout the section.","section":"§4.6.2"},{"comment":"The 'RA.' abbreviation for Random Augmentation appears with an inconsistent period, and the column ordering in the table makes it hard to map the checkmarks to the abbreviation list; please reformat the header and legend for clarity.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely useful empirical component, but the current quantitative support for its central claim could be substantially strengthened by multi-seed runs and a pre-registered analysis plan. The positionality disclosure is commendable, yet the editor may wish to consider whether the close involvement of a TRUNK author requires the authors to frame the paper as a self-study rather than an independent verification; this does not by itself invalidate the work, but it should be made load-bearing in the claims. The unexplained SVHN drop should be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: this is a genuine case-study contribution—nobody has audited TRUNK's reproducibility before, and the observation that the same seed on the same GPU model can yield different tree structures and a ~1.8-point accuracy spread is a useful data point. The authors also disclose the conflict of interest plainly in Section 3.6 and admit in Section 4.3 that full end-to-end reproduction was not achieved. That honesty counts.\n\nWhat it does well: the empirical documentation of environment-setup failures (the manifest including unused packages), the inference verification of pre-trained weights, the grouping-volatility sensitivity sweeps, and the batch-size ablation are concrete and reproducible. The proposed guideline extensions—minimal GPU-compatible manifests, full pipeline disclosure, sensitivity analysis—are reasonable, though not especially novel; they largely restate existing recommendations with a case-study wrapper.\n\nWhere it softens: the central quantitative evidence for the extensions is thin. All key accuracy numbers are single runs. In Section 4.5/Figure 8 the paper itself shows a same-seed, same-GPU-type run-to-run difference of 81.53 vs 79.72 (about 1.8 points), and adjacent recipes in Table 7 differ by similar amounts. The 13.92-point TR1→TR6 gain comes from six recipes varied on multiple axes simultaneously, with TR6 selected post hoc. That is a selection result, not an independent confirmation. The SVHN tables also disagree: Table 5 reports 98.22 from original scripts, Table 6 reports 90.24 after the 'improved' pipeline, and the paper never explains the 8-point drop. Given the paper's own variance data, the stated conclusion that the extensions improve reproducibility is not yet empirically pinned down.\n\nNone of this invalidates the qualitative suggestions or the case-study narrative. But the paper should either run multi-seed experiments with variance, or pre-register the recipe-selection rule, or explicitly frame the recipes as exploratory. And it should address the SVHN discrepancy.\n\nMy sense: worth a serious referee. It's a well-structured empirical case study on a relevant topic, with new data and an honest limitations section. It needs major-to-moderate revision before the quantitative conclusions can stand. If I were the editor, I would send it to review, but I would push for multi-seed results and a revised conclusion.\n\nFor whom: people working on ML reproducibility and MLOps tooling; also as a teaching example in a software-engineering-for-ML course. I'd maybe bring it to reading group, but I wouldn't cite the accuracy gains.","headline":"A genuine case-study contribution with honest disclosure, but the headline accuracy gains are built on single runs and post-hoc recipe selection, so the causal claim needs more support.","tokens_in":19660,"tokens_out":2361,"would_cite":false,"duration_ms":21275,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A case study shows that three extra practices make deep learning models reproducible in practice.","keywords":["reproducibility","deep learning software","case study","sensitivity analysis","TRUNK","hierarchical neural network","training pipeline transparency","dependency manifest"],"falsifier":"Run the same three-step procedure on a diverse set of published deep learning repositories — for example, a transformer, a GAN, and a medical-imaging model — and measure how often the extended guidelines raise the reproduced accuracy closer to the published values relative to the original guidelines; if the gains are small or inconsistent outside hierarchical computer-vision models, the claimed general effectiveness of the extensions is falsified.","tokens_in":18578,"feed_emoji":"🔁","tokens_out":3598,"duration_ms":28352,"temperature":0.7,"pith_summary":"This paper tackles the reproducibility problem in deep learning: even when source code and data are shared, results often cannot be recreated. Using the hierarchical TRUNK classifier as a case study, it shows that the standard reproducibility guidelines are insufficient. It argues that adding three practices — a dependency manifest listing only primary GPU-compatible packages, full disclosure of the data-processing and training pipeline, and a hyperparameter sensitivity analysis — substantially improves reproducibility. When these were applied, reproduced accuracy on EMNIST rose from 63.62% to 84.30%, and on CIFAR-10 from 67.61% to 81.53%. A sympathetic reader would care because if the claim holds, researchers who follow this extended checklist can save the weeks of effort currently needed to interpret incomplete instructions and can trust that a published model can actually be re-trained.","feed_headline":"Three fixes close the deep learning reproducibility gap","feed_subtitle":"A case study lifts a model's reproduced accuracy from 63.62% to 84.30% and pinpoints the missing practices.","key_machinery":"The central object is TRUNK, the Tree-Based Unidirectional Neural Network, a hierarchical computer-vision model structured as a tree of small networks whose exact tree structure is built from the data using a 'grouping volatility' threshold. The key mechanism is the sensitivity analysis over this grouping-volatility hyperparameter: it reveals that small changes in the threshold drastically change both the tree geometry and the final accuracy, so an investigator can adjust for non-determinism and recover the published structure and performance. The manifest listing only primary GPU-compatible dependencies and the PyYAML configuration files disclosing the exact training and data-processing pipeline are the other two load-bearing mechanisms; together they turn an un-reproducible repository into one whose environment, pipeline, and hyperparameter choices are inspectable and adjustable.","core_discovery":"The central claim is that the current reproducibility guidelines for deep learning software, extended with (1) a manifest listing only primary GPU-compatible dependencies, (2) full disclosure of the data-processing and training pipeline, and (3) sensitivity analysis of hyperparameters, improve the reproducibility of deep learning software. The paper demonstrates this by taking the Tree-Based Unidirectional Neural Network (TRUNK), a hierarchical classifier whose structure adapts to the dataset, and attempting to reproduce its published results from source code alone. It found that simply following the provided scripts and official manifests failed: the EMNIST accuracy reproduced at 63.62% versus the published 85.77%, and CIFAR-10 could not be trained at all because hyperparameters and pipeline details were missing. The authors then re-implemented the tree-building component, fixed the manifest, documented the pipeline in configuration files, and conducted a sensitivity analysis over the grouping-volatility hyperparameter. With these changes, EMNIST accuracy rose to 84.30% and CIFAR-10 accuracy rose from 67.61% to 81.53% across a series of revised training recipes, coming much closer to the original 85.77% and 91.99% respectively. The paper interprets these numbers as evidence that the extended guidelines diagnose and mitigate the specific reproducibility failures exposed by this case study.","pith_inferences":["The same methodology could be tested prospectively on a randomly chosen set of published deep learning models to see whether the three extensions reliably predict which repositories reproduce; the paper only demonstrates them retrospectively on one model chosen for its complexity.","Because the sensitivity analysis targeted a single hyperparameter, grouping volatility, one might infer that other adaptive architectures harbor similar high-leverage parameters; identifying such parameters could be systematized into a reproducibility audit routine.","The finding that a 0.03 change in grouping volatility swings accuracy by up to 7% suggests that published accuracy claims for such models should be accompanied by a confidence interval over the sensitive parameter, not just a single number.","The co-authorship of a TRUNK author means the investigation had privileged access to the intended pipeline; a non-affiliated team attempting the same reproduction would test whether the documented recipe is sufficient on its own, which is the stronger version of the reproducibility claim."],"forward_implications":["If these extended guidelines are adopted, published deep learning repositories would carry a minimal GPU-compatible manifest, a configuration-file description of the training and data-processing pipeline, and a sensitivity analysis for sensitive hyperparameters, making re-training a deterministic expectation rather than a research project.","For hierarchical or adaptive neural networks like TRUNK, reproducibility requires disclosing not just the trained weights and training scripts but the structure-building algorithm itself, since the tree geometry is part of the model's behavior.","The roughly 14% accuracy gain from revising the CIFAR-10 training recipe shows that undisclosed pipeline choices cause errors of the same magnitude as algorithmic novelty, so documenting them is as important as sharing source code.","Sensitivity analysis of a single sensitive hyperparameter can serve as a practical substitute for full determinism: it shows which parameter to nudge when hardware- or software-induced randomness changes the result.","The paper's checklist, combining existing best practices with the three extensions, is concrete enough to be used as a template for other deep learning reproducibility studies."],"supporting_citations":[{"why":"Pineau et al. define reproducibility classification (same code and data) and supply the baseline reproducibility guidelines the paper extends.","marker":"[9]"},{"why":"Semmelrock et al. document environmental differences and dependency issues as reproducibility obstacles, motivating the minimal-manifest recommendation.","marker":"[10]"},{"why":"The original TRUNK journal paper provides the repository, manifest, documented hyperparameters, and the grouping-volatility concept that the case study reproduces.","marker":"[15]"},{"why":"The TRUNK IEEE Design & Test paper supplies the reported accuracies (85.77% EMNIST, 91.99% CIFAR-10, 96.75% SVHN) used as ground truth for comparison.","marker":"[16]"},{"why":"Haibe-Kains et al. document researcher reluctance to share code and list platforms for sharing code, dependencies, and models, framing the transparency problem.","marker":"[7]"},{"why":"Stojnic's ML code completeness checklist is the README and documentation standard applied to restructure TRUNK's repository.","marker":"[36]"},{"why":"Matsubara's TorchDistill configuration-driven framework inspires the PyYAML files the paper uses to disclose training and data-processing pipelines.","marker":"[37]"},{"why":"Chen et al. discuss randomness, PRNGs, and seeds in reproducibility, grounding the sensitivity-analysis recommendation.","marker":"[6]"}],"fun_headline_variants":["Deep learning reproduction: case study lifts accuracy to 84% from 63%","Three fixes boost reproduced deep learning accuracy from 63% to 84%","Case study: minimal guidelines close deep learning reproducibility gap","Don't trust deep learning results without these three fixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that TRUNK is a representative deep learning system, so lessons from reproducing it transfer to other models, pipelines, and domains; the paper itself notes this external threat, and the ground truth for \"reproduced\" is the original authors' own reported numbers and weights.","fun_headline_variants_meta":{"raw":{"variants":["Deep learning reproduction: case study lifts accuracy to 84% from 63%","Three fixes boost reproduced deep learning accuracy from 63% to 84%","Case study: minimal guidelines close deep learning reproducibility gap","Don't trust deep learning results without these three fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000745,"raw_usage":{"total_tokens":3396,"prompt_tokens":1092,"completion_tokens":2304,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":708,"completion_tokens_details":{"reasoning_tokens":2230}},"tokens_in":708,"tokens_out":2304,"duration_ms":16901,"temperature":1.0,"reasoning_tokens":2230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:57:59.135209+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three-step procedure on a diverse set of published deep learning repositories — for example, a transformer, a GAN, and a medical-imaging model — and measure how often the extended guidelines raise the reproduced accuracy closer to the published values relative to the original guidelines; if the gains are small or inconsistent outside hierarchical computer-vision models, the claimed general effectiveness of the extensions is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The original TRUNK journal paper provides the repository, manifest, documented hyperparameters, and the grouping-volatility concept that the case study reproduces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The TRUNK IEEE Design & Test paper supplies the reported accuracies (85.77% EMNIST, 91.99% CIFAR-10, 96.75% SVHN) used as ground truth for comparison."},{"cited_title":"Stojnic, Ml code completeness checklist (2020)","cited_arxiv_id":null,"evidence_quote":"Stojnic's ML code completeness checklist is the README and documentation standard applied to restructure TRUNK's repository."},{"cited_title":"torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation","cited_arxiv_id":"2011.12913","evidence_quote":"Matsubara's TorchDistill configuration-driven framework inspires the PyYAML files the paper uses to disclose training and data-processing pipelines."},{"cited_title":"Towards Training Reproducible Deep Learning Models","cited_arxiv_id":"2202.02326","evidence_quote":"Chen et al. discuss randomness, PRNGs, and seeds in reproducibility, grounding the sensitivity-analysis recommendation."}],"review_version":1}