Deep4ge: DNN Training Trajectories for Fault Detection and Diagnosis

Sigma Jahan

Deep4ge releases 14,227 labeled DNN training runs so models can detect and diagnose implementation faults from epoch-by-epoch behavior.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 02:44 UTC pith:6OLV62D6

load-bearing objection Useful SE-for-ML infrastructure paper: large controlled training-fault benchmark with clear reuse intent; main open question is real-world transfer. the 2 major comments →

arxiv 2607.12868 v1 pith:6OLV62D6 submitted 2026-07-14 cs.SE cs.LG

Deep4ge: DNN Training Trajectories for Fault Detection and Diagnosis

Sigma Jahan This is my paper

classification cs.SE cs.LG

keywords DNN fault detectiontraining trajectoriesfault diagnosisTensorFlow/Kerassoftware engineering benchmarkearly fault predictionsource-code fault injection

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Deep learning training can fail because of subtle coding mistakes that change how weights, gradients, and metrics evolve across epochs. Until now the software-engineering community has had no public, carefully controlled collection of such runs that records both the injected fault and the detailed training trajectory. Deep4ge supplies that resource: 14,227 complete training runs drawn from 59 real TensorFlow/Keras programs, half of them intentionally corrupted by 27 source-code transformations that cover seven fault categories. Every run stores four evaluation metrics and twenty-six per-epoch features describing weights, gradients, activations, loss and accuracy trends, learning-rate schedules, and hardware use. The resulting benchmark lets researchers train detectors for binary “faulty or not,” multi-class diagnosis of which fault occurred, and early prediction that can raise an alarm before training finishes. A sympathetic reader cares because the same signals that engineers already watch on dashboards can now be turned into automatic, reproducible fault-finding tools.

Core claim

A public, controlled benchmark of 14,227 TensorFlow/Keras training runs (9,845 faulty, 4,382 correct) generated from 59 Stack Overflow programs by 27 source-code fault transformations, each run annotated with four evaluation metrics and twenty-six per-epoch training-behavior features, enables binary fault detection, multi-class diagnosis, and early prediction from partial runs.

What carries the argument

The central object is the Deep4ge dataset itself: a matrix of 14,227 labeled training trajectories, each trajectory a sequence of 26 engineered features recorded at every epoch, produced by systematically applying 27 known fault-injection operators to 59 real DNN programs.

Load-bearing premise

The 27 synthetic source-code transformations applied to 59 Stack Overflow programs produce a representative and non-confounded sample of the implementation faults that actually appear in real DNN training code.

What would settle it

Train a detector on Deep4ge, then measure its precision and recall on a held-out corpus of naturally occurring, manually verified faults drawn from real open-source DNN projects that were never seen during dataset construction; a large drop in performance would falsify the claim of representativeness.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful SE-for-ML infrastructure paper: large controlled training-fault benchmark with clear reuse intent; main open question is real-world transfer.

read the letter

The one thing to know: this is a dataset/benchmark paper that ships a large controlled set of DNN training runs with known injected faults and per-epoch features, aimed at detection, diagnosis, and early prediction. That is real infrastructure for SE-for-ML, not a theory claim.

What is new is the combination: 14,227 runs from 59 Stack Overflow–adapted TensorFlow/Keras programs, 27 source-code transformations across seven fault categories, 9,845 faulty vs 4,382 correct runs, four evaluation metrics, and 26 per-epoch features (weights, gradients, activations, accuracy/loss trends, learning rate, hardware). Labels are by construction, so circularity is not the issue. They also claim a Zenodo release of the dataset and injection framework, which is the right way to make this reusable. If the artifact is as described, it can standardize evaluation of detectors and early-warning tools and cut wasted compute. That is earned credit for a resource paper.

The soft spot is representativeness, and it is the main one. Injected faults on SO-derived programs may not match the incidence or diversity of real implementation faults, architectures, or datasets. The abstract does not report external validation of transfer. That does not kill the contribution—controlled benchmarks are still useful—but anyone training models on Deep4ge should treat generalization as an open empirical question, not a given. Because we only have the abstract, we also cannot audit generation scripts, feature definitions, leakage, or epoch-window bias. Those are process risks, not proven flaws.

Who it is for: people building or evaluating training-time fault detectors, diagnosis models, or early-stop predictors in SE-for-ML. Not a general ML audience. It deserves a serious referee if the full paper and Zenodo artifact check out; desk-rejecting a clearly scoped public benchmark with stated scale and reuse support would be the wrong call. I would send it to review with a mandate to verify the artifact and to require an honest discussion of transfer limits.

Recommendation: engage. Accept for peer review conditional on artifact inspection; do not overclaim novelty beyond the shared resource.

Referee Report

2 major / 3 minor

Summary. The manuscript introduces Deep4ge, a controlled public benchmark of 14,227 DNN training runs (9,845 faulty, 4,382 correct) generated from 59 TensorFlow/Keras programs adapted from Stack Overflow. Faulty variants are produced by 27 source-code transformations spanning seven fault categories. For each run the authors record four evaluation metrics and 26 per-epoch training-behavior features (weights, gradients, activations, accuracy/loss trends, learning rate, hardware use). The resource is positioned to support binary fault detection, multi-class diagnosis, and early prediction from partial runs, and is released with a fault-injection framework under a Zenodo DOI.

Significance. If the artifact is complete, well-documented, and free of systematic confounds, Deep4ge would fill a genuine gap in the software-engineering literature on DNN training faults: a large, labeled, per-epoch trajectory corpus with known fault history. The scale (14k runs, 26 features) and the explicit support for early-prediction tasks are strengths that could enable reproducible comparisons of detection and diagnosis methods. Because labels are obtained by construction rather than by post-hoc fitting, circularity risk is low. The principal open question is external validity—whether the injected transformations and Stack Overflow–derived programs yield a distribution that transfers to naturally occurring implementation faults.

major comments (2)

The abstract asserts that the 27 transformations applied to 59 Stack Overflow programs produce a useful benchmark for real DNN implementation faults, yet supplies no evidence of external validation (e.g., comparison against real-world fault incidence, diversity of architectures/datasets, or transfer experiments). Because this representativeness claim is load-bearing for the resource’s intended reuse, the full manuscript must either (a) report such validation or (b) clearly scope the benchmark as a controlled synthetic corpus and discuss the resulting threats to external validity.
With only the abstract available it is impossible to verify that the 26 per-epoch features are free of leakage with respect to the four evaluation metrics, that transformation categories are non-overlapping, and that generation scripts are fully reproducible. These points are essential for a dataset paper; the camera-ready version must include precise feature definitions, category definitions, and a complete reproducibility package.

minor comments (3)

The abstract lists seven fault categories and 26 features without naming them; a concise table or enumeration in the full text would improve clarity and citability.
Clarify how the 59 programs were selected and adapted from Stack Overflow (inclusion criteria, filtering of duplicates, handling of incomplete snippets).
State the exact TensorFlow/Keras versions and hardware configurations used for the 14,227 runs so that future users can reproduce the recorded hardware-use features.

Circularity Check

0 steps flagged

No significant circularity: resource/benchmark paper with labels known by construction; no fitted-parameter-as-prediction or self-definitional derivation chain.

full rationale

This is an abstract-only resource contribution that constructs a controlled public benchmark (14,227 runs from 59 programs, 27 source-code fault transformations, 26 per-epoch features) for binary detection, multi-class diagnosis, and early prediction. Faulty vs. correct labels are known by design because faults are injected via the stated transformations; the paper does not claim a first-principles derivation, uniqueness theorem, or parameter-free prediction that reduces to its own fitted inputs. There is no self-definitional loop, no fitted quantity renamed as a prediction, no load-bearing uniqueness imported from the authors, and no ansatz smuggled via self-citation. Representativeness of the injected faults relative to real-world DNN faults is an external validity concern, not circularity. With only the abstract available and no equations or derivation chain present, the honest finding is score 0 and empty steps.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

As a dataset paper the main commitments are methodological: that the chosen programs, fault transformations, and feature extractors adequately represent the intended fault-detection problem. No free parameters are fitted to produce a scientific constant; no new physical entities are invented. Axioms are domain assumptions about representativeness and that injected faults are the ground-truth labels for detection tasks.

axioms (3)

domain assumption The 59 adapted Stack Overflow TensorFlow/Keras programs plus 27 source-code transformations across seven categories yield a useful and sufficiently representative sample of real DNN implementation faults for detection and diagnosis research.
Central to the claim that Deep4ge supports generalizable fault detection/diagnosis; abstract does not prove coverage of natural fault distributions.
domain assumption The 26 recorded per-epoch features (weights, gradients, activations, accuracy/loss trends, learning rate, hardware use) plus 4 evaluation metrics are sufficient signals for binary detection, multi-class diagnosis, and early prediction.
Implied by the supported-task claims; feature sufficiency is assumed rather than proven in the abstract.
domain assumption Faulty vs. correct labels are defined by the presence/absence of the injected source-code transformations.
Standard for mutation-style fault injection; ground truth is by construction.

pith-pipeline@v1.1.0-grok45 · 6100 in / 2536 out tokens · 19565 ms · 2026-07-15T02:44:11.954244+00:00 · methodology

0 comments

read the original abstract

Deep learning systems often fail due to subtle implementation faults that alter training behavior. Recent work has studied how to detect and diagnose such failures from changes observed across training epochs. However, the software engineering community still lacks a public dataset of per-epoch training runs with documented fault history, feature extraction details, and clear reuse support for fault detection and diagnosis tasks. We present Deep4ge, a controlled benchmark of 14,227 training runs generated from 59 adapted TensorFlow/Keras deep neural network (DNN) programs collected from Stack Overflow. We generated faulty variants using 27 source-code transformations that introduce known faults across seven categories. The dataset contains 9,845 faulty runs and 4,382 correct baseline runs. For each run, we record 4 evaluation metrics and 26 features that measure training behavior at every epoch. These features capture weights, gradients, activations, accuracy and loss trends, learning rate, and hardware use. Deep4ge supports binary fault detection, multi-class fault diagnosis, and early fault prediction from partial training runs. We release the dataset and fault-injection framework at https://doi.org/10.5281/zenodo.20337241.

Deep4ge: DNN Training Trajectories for Fault Detection and Diagnosis

Core claim

What carries the argument

Load-bearing premise

What would settle it

discussion (0)