Predicting Deep Neural Network Training Outcomes from Early Training Telemetry

Anh D. Nguyen; Pankaj Kumar Singh; Ranjita Naik

REVIEW 2 major objections 5 minor 12 references

Predicting Deep Neural Network Training Outcomes from Early Training Telemetry

T0 review · 2 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training run's own first five epochs reveal its final accuracy and failure risk, with no comparison to other runs.

desk verdict A careful, honest empirical study whose headline R^2/AUC numbers should be read as interpolation in a boundary-enriched sample, not as a test of generalization to distant configurations. read the letter →

arxiv 2608.03709 v1 pith:T56SZVAR submitted 2026-08-04 cs.CL

Ranjita Naik , Anh D. Nguyen , Pankaj Kumar Singh This is my paper

classification cs.CL

keywords earlytrainingtelemetrygradientsignal-to-noiseweight-normgrowthhyperparametersweeplearning-curvepredictionfailureboostingtransferlearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a deep network training run's eventual outcome is visible in its own first few epochs, without comparing it to any other run in a sweep. On 23,788 runs across six architecture/dataset combinations, gradient-boosted trees using five epochs of loss, accuracy, gradient-consistency and weight-norm telemetry plus the sampled hyperparameters predict final test accuracy with R² = 0.92–0.99 and predict upper-half-vs-lower-half standing with ROC-AUC 0.983–0.998 on a permanently frozen 20% of hyperparameter configurations. Most predictive power is present after a single epoch. A paired ablation shows gradient- and weight-level telemetry improves prediction over loss/accuracy curves alone in every domain and task tested, although the practical gain varies. If the claim holds, sweep schedulers could redirect compute away from doomed runs early, using only a single-run signal rather than a comparison cohort.

What carries the argument

The gradient signal-to-noise ratio (how consistently a fixed set of parameter-coordinate gradients points in the same direction across mini-batches within an epoch) and weight-norm growth (relative change in the total L2 norm of parameters since initialization), plus a one-time activation-saturation snapshot, are the novel signals. Ten engineered trajectory statistics per signal, combined with the four sampled hyperparameters, feed a gradient-boosted tree. A two-phase sampling procedure deliberately concentrates runs near the success/failure boundary, and a permanently frozen 20% configuration partition provides the one-shot generalization test.

What would settle it

Sample a fresh cohort of hyperparameter configurations uniformly across the full ranges, train them to completion, and evaluate the same five-epoch predictors: if final-accuracy R² drops well below 0.92 or relative-classification AUC below 0.98, the generalization claim fails. Alternatively, train multiple random-seed repeats of identical configurations: if between-seed accuracy variance approaches the predictor's error floor, single-run telemetry cannot separate luck from outcome.

Watch

Extended reading notes

Core claim

The central claim is that training-dynamics telemetry recorded inside a run—how consistently the gradient points in one direction, how fast the weight norm grows, and how saturated activations become—carries information about the run's final accuracy and failure risk that is not captured by the loss and accuracy curves practitioners already log. The paper demonstrates this with a controlled, hyperparameter-free ablation, showing that the relationship is strong enough after one epoch to support practical decision support. The authors are explicit that boundary-focused sampling makes some holdout configurations close to training configurations, and that results describe a boundary-heavy popula

Load-bearing premise

The strong numbers rest on treating the frozen 20 percent holdout as a test of generalization to new configurations; the paper concedes (Sections 5.5 and 8) that boundary-focused sampling makes many held-out configurations sit close to, though never identical to, training configurations, so the scores may overstate performance on genuinely distant configurations.

Editorial extensions

If this is right

Run-termination heuristics can flag or rank configurations from epoch 1 without a concurrent cohort, complementing comparison-based schedulers.
Gradient and weight-norm telemetry add information beyond loss/accuracy curves in every tested domain, so internal optimization signals are worth logging as routine telemetry.
Predictors transfer well across architecturally similar domains; across datasets the underlying rank relationship survives, so domain-relative calibration, not retraining, may handle scale mismatch.
Outright numerical divergence is predictable as a distinct outcome early, suggesting separate flags for usability-threshold failures versus NaN divergence.
Because the evaluation cohort is boundary-enriched, achieved error rates describe a hard near-boundary population, not a typical practitioner sweep.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension would sample new configurations uniformly across the full hyperparameter ranges and measure the drop in frozen-holdout performance; a large drop would show the headline numbers apply mainly to boundary-focused sweeps.
If single-epoch saturation holds at longer schedules, schedulers could use epoch-1 predictions as a near-final ranking, but seed-to-seed variance at fixed hyperparameters is untested in this data and could set the prediction floor.
The CIFAR-10 multilayer-perceptron asymmetry suggests telemetry-to-accuracy relationships are not architecture-universal; deploying a predictor on a new architecture family would likely require per-family calibration.
In language-model or large-scale vision settings, gradient consistency may be a cheap early proxy for later divergence, directly testable on existing training logs without new compute.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a competent, honest empirical paper whose headline numbers are probably true for the population it sampled, but that population is boundary-enriched and the 'frozen holdout' is a split of an adaptive sample, so the numbers overstate generalization to distant new configurations. The authors admit this in Section 8, but the abstract still leads with R²=0.92–0.99, and most readers will miss the caveat.

What's new and worth keeping: the paired, hyperparameter-free ablation showing gradient SNR and weight-norm growth improve prediction over loss/accuracy curves alone, in every domain/task with confidence intervals excluding zero. Even where the gains are small, the design is clean and the claim is modest. Treating numerical divergence as a directly predicted terminal state is a reasonable addition. The transfer analysis separating scale mismatch from loss of rank relationship is careful and useful.

Soft spots, in order: (1) The frozen holdout is not an independent generalization test. Because phase-two sampling concentrates draws near the estimated success/failure boundary, held-out configurations sit close to open ones. That makes near-duplicate prediction easy. The authors say the holdout 'should temper a literal reading' as a test for distant configurations, but that sentence is buried in the limitations; the abstract and intro present the frozen partition as if it settles generalization. I'd want a distance-based diagnostic or a validation set collected under independent sampling. (2) No code or data, and exact hyperparameter range endpoints are withheld. Fixable, but it blocks verification. (3) The practical telemetry gain is negligible in some domains. The authors are upfront about that, so it's a fairness issue, not an overclaim.

The paper is well above the median: leakage-aware protocol, Nadeau–Bengio-corrected intervals, paired ablations, and an unusually candid limitations section. The CIFAR-10 MLP transfer outlier is presented as a hypothesis, not a result. This deserves a serious referee. I'd send it out, then press the authors on the independence of the holdout and require code and data.

Referee Report

2 major / 5 minor

Summary. The paper asks whether a single training run's early telemetry (loss, training accuracy, gradient signal-to-noise, weight-norm growth, and a one-time activation-saturation snapshot) together with its four sampled hyperparameters can predict that run's eventual outcome, without comparing it to other runs. The authors generate 23,788 runs across six architecture/dataset domains, predict final test accuracy, relative cohort standing, and training-dynamics failure, and report gradient-boosted trees achieving R^2 = 0.92-0.99 and ROC-AUC = 0.983-0.998 on a permanently frozen 20% partition (Table 2). They also report a paired ablation claiming that gradient/weight telemetry adds statistically consistent signal beyond loss/accuracy curves in every domain and task (Table 3), and a transfer analysis separating scale mismatch from relationship transfer. The paper is framed as decision support rather than automatic termination and contains a detailed limitations section.

Significance. If the headline results hold, the paper offers a cheap, practical early-warning signal for compute allocation in hyperparameter sweeps. The study has substantial strengths: a large and carefully cleaned dataset, a leakage-safe multi-horizon protocol, a permanently frozen one-shot holdout, paired ablations with Nadeau-Bengio-corrected intervals, a complete transfer matrix, and unusually explicit limitations. The principal unresolved issue is whether the frozen holdout provides a valid estimate of generalization to genuinely independent hyperparameter configurations. Because the second sampling phase deliberately concentrates draws near the success/failure boundary, frozen configurations are likely close to open configurations, and the paper itself concedes this in Section 8. That concession is load-bearing for the paper's central empirical claim, and a distance-based diagnostic is needed before the headline numbers can be taken at face value.

major comments (2)

[Section 5.5, Section 8, Table 2] The frozen-holdout evaluation is the central evidence for the paper's headline generalization claim, but boundary-focused adaptive sampling compromises it. Section 4.2 concentrates roughly half of second-phase draws near the estimated success/failure boundary; the frozen partition is a random split of this adaptive sample. The model receives the four sampled hyperparameters, so any frozen configuration lying close to an open configuration in this low-dimensional space can be predicted almost by interpolation. Section 8 concedes that some frozen configurations 'likely lie close to, though never identical to, an open-partition configuration' and that this 'should temper a literal reading of the frozen holdout as a test of generalization to an arbitrarily distant new configuration.' This is not a conventional leakage failure, but it means the frozen holdout estimates interpolation performan
[Section 4.2, Section 6.2, Section 8] The strong one-epoch result is presented as a general finding, but its two pieces of evidence have different statuses. Figure 1 uses only CIFAR-10/ResNet-18 on the open 80% development partition; the all-domain frozen-holdout numbers in Table 4 show more variability at k=1 (R^2 ranges from 0.844 to 0.982). More importantly, Section 8 states that the proximity of frozen to open configurations is a 'partial, unconfirmed explanation' for the strong early-horizon result. The claim that 'useful prediction is already available after a single epoch' may well be true, but the current evaluation cannot separate the genuine early signal from the near-duplicate-configuration effect. A distance-stratified horizon analysis, or an evaluation of models trained only on first-phase broad samples and tested on second-phase boundary samples, would directly address this and should accompany any revised clai

minor comments (5)

[Abstract, Section 6.1] The abstract's 'R^2 = 0.92-0.99' range describes the five-epoch horizon on the non-diverged cohort. It would be helpful to state that explicitly in the abstract, since Table 4 shows lower values at earlier horizons and the eligible cohort excludes diverged runs.
[Table 3] The text says every one of the twelve classification-AUC deltas has an individual 95% CI that excludes zero, but those intervals are not shown. Since the table is the paper's central evidence for the telemetry ablation, the individual intervals should be included in an appendix or in the table itself.
[Section 5.2, Section 6.3] The term 'hyperparameter-free' is potentially confusing: the paired ablation excludes the four sampled hyperparameters from the feature set, but the gradient-boosting model itself has hyperparameters. A brief clarifying remark would avoid misreading.
[Section 4.5, Appendix G] Removing the entire finer-granularity export for Fashion-MNIST/ResNet-18 is a defensible conservative choice, but the 160 non-identical rows are concentrated near the boundary. Since boundary-region data are the most informative for this study, it would be useful to state whether the main Table 2/Table 6 metrics are robust to the alternative of retaining those rows or resolving them by an explicit policy.
[Section 6.2, Figure 1] The one-epoch claim is illustrated with a single domain on the open 80% partition. Given the variability across domains in Table 4, a small multi-panel version of Figure 1 for all six domains, or a rephrasing such as 'useful prediction is often available after one epoch,' would be more accurate.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: predictions are empirical mappings from early telemetry and hyperparameters to final outcomes, with a disclosed boundary-sampling generalization limitation that is not a definitional reduction.

full rationale

The paper's derivation chain is self-contained. Features are per-run telemetry (loss, accuracy, gradient SNR, weight-norm growth, activation saturation) plus four sampled hyperparameters, observed only through horizon k. Targets are final test accuracy, relative rank against a training-partition median, and failure (final accuracy below a pre-fixed threshold or NaN). No target is defined from the features, and no feature is constructed from the target. The frozen 20% holdout is a random split of the sampled configurations, never used in development; gradient boosting was fixed before the frozen evaluation. The ablation compares curves-only vs curves-plus-internal feature sets on matched folds, so the incremental telemetry claim is empirical, not forced. The paper explicitly acknowledges that boundary-focused sampling may place some frozen configurations near open ones, tempering generalization to distant configurations; this is a validity limitation, not circularity, because the model still must learn the telemetry-outcome relationship and the reported metrics are measured, not derived. There are no load-bearing self-citations: the references are prior work by others, and no uniqueness theorem or ansatz is imported from the authors' own previous papers. Accordingly, no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a few design choices that are free parameters in the problem definition: domain-specific usability thresholds, per-architecture sampling range endpoints, and the unstated gradient-boosting hyperparameters. The telemetry signals are measurements, not invented entities. The main domain assumption is that boundary-focused sampling plus a frozen holdout gives a meaningful generalization estimate, which the authors themselves partially question.

free parameters (3)

Usability threshold per domain = 45, 40, 30, 80, 78, 70 (percent final accuracy)
Manually chosen before data generation to define the failure label; different thresholds would shift class balance and failure-prediction metrics (Section 8, Appendix G).
Hyperparameter sampling range endpoints = Not disclosed (tuned per-architecture in pilot runs)
Appendix A states the exact endpoints were tuned per-architecture so both stable and unstable regions are populated; these endpoints shape the data distribution and are not reproducible from the paper.
Gradient-boosting model hyperparameters = Not reported
The primary model's hyperparameters (tree depth, learning rate, number of estimators) are not listed, which matters for exact replication.

assumptions (4)

domain assumption Configuration-level grouping prevents any single hyperparameter configuration from splitting across train and test
Section 5.4 relies on this grouping, though no repeated seeds actually exist in the data.
domain assumption The boundary-focused sampler fit on first-phase outcomes does not leak into the frozen evaluation beyond shaping the population
Section 5.5 argues the sampler is not an outcome predictor on the frozen partition, but the proximity caveat in Section 8 weakens the independence claim.
domain assumption The manually chosen usability thresholds are a valid definition of training-dynamics failure
Section 4.3 and Appendix G fix thresholds before data generation and note that different choices would shift the failure-prediction metrics.
standard math Nadeau-Bengio correction formula
Appendix F uses this correction for repeated overlapping splits; it is a standard statistical correction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Predicting Deep Neural Network Training Outcomes from Early Training Telemetry." pith.science (2026). https://pith.science/paper/T56SZVAR

@misc{pith2026260803709,
  author       = {Pith},
  title        = {Pith review of: Predicting Deep Neural Network Training Outcomes from Early Training Telemetry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T56SZVAR}},
  note         = {Machine review of arXiv:2608.03709}
}

read the original abstract

Large hyperparameter sweeps for deep neural networks spend substantial compute on configurations that are effectively doomed from the first few epochs. We study whether a single training run's own early telemetry - per-epoch loss, training accuracy, gradient signal-to-noise ratio, weight-norm growth, and an activation-saturation snapshot - together with its sampled hyperparameters, can predict that run's eventual outcome without reference to other runs. We evaluate three prediction tasks: final test accuracy, relative performance within a domain, and training-dynamics failure, including numerical divergence. Across 23,788 training runs spanning six architecture/dataset combinations, gradient-boosted trees using only the first five epochs of telemetry achieve R^2 = 0.92-0.99 for final-accuracy regression and ROC-AUC = 0.983-0.998 for relative classification on a permanently held-out set of hyperparameter configurations. Useful prediction is already available after a single epoch. A paired ablation shows that gradient- and weight-level telemetry provides a statistically consistent improvement over loss and accuracy curves alone, although the practical gain varies by domain. Transfer is strong between similar architectures, while cross-dataset transfer is limited mainly by differences in accuracy scale rather than loss of the underlying relationship. These results suggest that early-training telemetry can provide a practical decision-support signal for compute allocation while motivating human oversight for any automated intervention.

Figures

Figures reproduced from arXiv: 2608.03709 by the authors.

**Figure 1.** Final-accuracy regression R 2 by observation horizon (CIFAR-10/ResNet-18, open 80% development partition; shaded band is a Nadeau-Bengio-corrected 95% confidence interval over 10 resampled splits). Both tree ensembles are already close to their five-epoch accuracy after a single epoch, the main evidence behind our claim that useful prediction is available remarkably early [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗

**Figure 2.** Feature-set ablation for final-accuracy regression at the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 4.** Relative-classification ROC-AUC for a predictor trained [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Figures from the paper (6 more)

**Figure 5.** Figure 5: Three telemetry signals decomposed by outcome for CIFAR-10/ResNet-18: (a) training loss by final-accuracy quartile, (b) [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: Predicted vs. actual final accuracy, gradient boosting, [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

**Figure 7.** Figure 7: Complete 36-cell transfer matrix (18 one-factor pairs, 12 both-differ pairs, and 6 in-domain diagonal scores) as a heatmap; same [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

**Figure 8.** Figure 8: Regression baselines and ensembles compared at the five-epoch horizon (open 80% development partition, 10 resampled splits), [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

**Figure 9.** Figure 9: ROC curve, relative classification, five-epoch horizon, CIFAR-10/ResNet-18, frozen holdout. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

**Figure 10.** Figure 10: ROC curve, failure prediction, five-epoch horizon, CIFAR-10/ResNet-18, frozen holdout. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

[1]

Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas D

Mohamed S. Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas D. Lane. Zero-cost proxies for lightweight nas. InProceedings of the International Con- ference on Learning Representations (ICLR), 2021

work page 2021
[2]

Accelerating neural architecture search using per- formance prediction

Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using per- formance prediction. InProceedings of the International Conference on Learning Representations (ICLR) Workshop, 2018

work page 2018
[3]

Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves

Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. InPro- ceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2015

work page 2015
[4]

BOHB: Robust and efficient hyperparameter optimization at scale

Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. InProceedings of the International Conference on Machine Learning (ICML), 2018

work page 2018
[5]

Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: A ser- vice for black-box optimization. InProceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2017

work page 2017
[6]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

2016
[7]

Non-stochastic best arm identification and hyperparameter optimization

Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. InPro- ceedings of the International Conference on Artificial Intel- ligence and Statistics (AISTATS), 2016

work page 2016
[8]

Learning curve prediction with bayesian neural networks

Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. InProceedings of the International Conference on Learning Representations (ICLR), 2017

work page 2017

Show all 12 references

[9]

Hyperband: A novel bandit-based approach to hyperparameter optimization

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Ros- tamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research (JMLR), 18(185):1– 52, 2018

2018
[10]

An empirical model of large-batch training.arXiv preprint arXiv:1812.06162, 2018

Sam McCandlish, Jared Kaplan, and Dario Amodei. An empirical model of large-batch training.arXiv preprint arXiv:1812.06162, 2018

2018 arXiv
[11]

Predicting neural network accuracy from weights.arXiv preprint arXiv:2002.11448, 2020

Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights.arXiv preprint arXiv:2002.11448, 2020

2002 arXiv
[12]

obviously stable

Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. NAS-Bench-101: Towards reproducible neural architecture search. InProceedings of the International Conference on Machine Learning (ICML), 2019. A. Detailed Experimental Setup Hyperparamet...

2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas D

Mohamed S. Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas D. Lane. Zero-cost proxies for lightweight nas. InProceedings of the International Con- ference on Learning Representations (ICLR), 2021

work page 2021

[2] [2]

Accelerating neural architecture search using per- formance prediction

Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using per- formance prediction. InProceedings of the International Conference on Learning Representations (ICLR) Workshop, 2018

work page 2018

[3] [3]

Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves

Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. InPro- ceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2015

work page 2015

[4] [4]

BOHB: Robust and efficient hyperparameter optimization at scale

Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. InProceedings of the International Conference on Machine Learning (ICML), 2018

work page 2018

[5] [5]

Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: A ser- vice for black-box optimization. InProceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2017

work page 2017

[6] [6]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

2016

[7] [7]

Non-stochastic best arm identification and hyperparameter optimization

Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. InPro- ceedings of the International Conference on Artificial Intel- ligence and Statistics (AISTATS), 2016

work page 2016

[8] [8]

Learning curve prediction with bayesian neural networks

Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. InProceedings of the International Conference on Learning Representations (ICLR), 2017

work page 2017