{"id":"ff17198d-10e6-4c60-8fd7-2cada37cea11","arxiv_id":"2608.03709","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":0.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"After only a few epochs, a run's own loss, accuracy, gradient, and weight-norm telemetry predicts its final accuracy (R² = 0.92 to 0.99) and relative performance (AUC = 0.983 to 0.998) across six image-classification domains.","lead":"Early training telemetry from a single run, including gradient signal-to-noise and weight-norm growth, can predict that run's final accuracy and failure risk after only one to five epochs. The results suggest a cheap decision-support signal for allocating compute across hyperparameter sweeps, without comparing runs to each other.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frozen-holdout generalization may be inflated by boundary-focused sampling: held-out configurations likely sit near open ones, so the headline R²/AUC are not yet established for distant configurations.","rationale":"The reader identified the most load-bearing concern: the frozen holdout may not be a true test of generalization because boundary-focused sampling places held-out configurations near open ones. This is exactly the assumption on which the headline empirical claim rests. The paper itself acknowledges the issue in Section 8 but does not provide a quantitative assessment. A distance-stratified analysis of the existing frozen holdout would settle whether the proximity effect materially inflates the reported R²/AUC values. This is not an internal inconsistency or a matter of disagreement with consensus; it is an evaluation-design limitation that directly affects the strength of the central claim. The ablation claim about internal telemetry is also evaluated only on the development partition, but the proximity issue is more load-bearing because it affects the primary quantitative results. The reader's conditional verdict remains appropriate: the paper is honest and carefully executed, but the generalization claim should be accepted only conditionally on a distance-based robustness check or on explicitly limiting the claim to the boundary-focused population.","tokens_in":22711,"tokens_out":10414,"duration_ms":119612,"concrete_test":"Compute for each frozen-holdout configuration its minimum distance (e.g., Euclidean in log-scaled hyperparameter space, or Mahalanobis) to any open-partition configuration. Then recompute Table 2's R² and ROC-AUC separately for frozen configurations in the lower versus upper half of the distance distribution. If the far-half metrics drop by more than a pre-specified threshold (e.g., 0.05 in R² or 0.01 in AUC) relative to the full-holdout numbers, the proximity effect is confirmed; if the metrics are stable, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that gradient-boosted trees using five epochs of telemetry achieve R²=0.92–0.99 and ROC-AUC=0.983–0.998 on a permanently frozen 20% holdout of hyperparameter configurations (Table 2), presented as evidence of generalization to untouched configurations. However, the data-generation procedure (Section 4.2) adaptively concentrates second-phase draws near the success/failure boundary estimated from first-phase outcomes. The frozen partition is a random split of this adaptive sample, so frozen configurations are not independent of open ones: many frozen configurations likely lie close in the four-dimensional hyperparameter space to open configurations, and because the model includes the four sampled hyperparameters plus telemetry strongly correlated with outcome, near-duplicate configurations will be predicted almost perfectly. Section 8 explicitly concedes this 'should temper a literal reading of the frozen holdout as a test of generalization to an arbitrarily distant new configuration.' Without a distance-based diagnostic, the headline numbers may substantially overstate performance on truly novel configurations, and the claim that the frozen holdout tests generalization to untouched configurations is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether a single training run's early telemetry (loss, training accuracy, gradient signal-to-noise, weight-norm growth, and a one-time activation-saturation snapshot) together with its four sampled hyperparameters can predict that run's eventual outcome, without comparing it to other runs. The authors generate 23,788 runs across six architecture/dataset domains, predict final test accuracy, relative cohort standing, and training-dynamics failure, and report gradient-boosted trees achieving R^2 = 0.92-0.99 and ROC-AUC = 0.983-0.998 on a permanently frozen 20% partition (Table 2). They also report a paired ablation claiming that gradient/weight telemetry adds statistically consistent signal beyond loss/accuracy curves in every domain and task (Table 3), and a transfer analysis separating scale mismatch from relationship transfer. The paper is framed as decision support rather than automatic termination and contains a detailed limitations section.","tokens_in":22855,"tokens_out":6922,"duration_ms":83438,"significance":"If the headline results hold, the paper offers a cheap, practical early-warning signal for compute allocation in hyperparameter sweeps. The study has substantial strengths: a large and carefully cleaned dataset, a leakage-safe multi-horizon protocol, a permanently frozen one-shot holdout, paired ablations with Nadeau-Bengio-corrected intervals, a complete transfer matrix, and unusually explicit limitations. The principal unresolved issue is whether the frozen holdout provides a valid estimate of generalization to genuinely independent hyperparameter configurations. Because the second sampling phase deliberately concentrates draws near the success/failure boundary, frozen configurations are likely close to open configurations, and the paper itself concedes this in Section 8. That concession is load-bearing for the paper's central empirical claim, and a distance-based diagnostic is needed before the headline numbers can be taken at face value.","major_comments":[{"comment":"The frozen-holdout evaluation is the central evidence for the paper's headline generalization claim, but boundary-focused adaptive sampling compromises it. Section 4.2 concentrates roughly half of second-phase draws near the estimated success/failure boundary; the frozen partition is a random split of this adaptive sample. The model receives the four sampled hyperparameters, so any frozen configuration lying close to an open configuration in this low-dimensional space can be predicted almost by interpolation. Section 8 concedes that some frozen configurations 'likely lie close to, though never identical to, an open-partition configuration' and that this 'should temper a literal reading of the frozen holdout as a test of generalization to an arbitrarily distant new configuration.' This is not a conventional leakage failure, but it means the frozen holdout estimates interpolation performan","section":"Section 5.5, Section 8, Table 2"},{"comment":"The strong one-epoch result is presented as a general finding, but its two pieces of evidence have different statuses. Figure 1 uses only CIFAR-10/ResNet-18 on the open 80% development partition; the all-domain frozen-holdout numbers in Table 4 show more variability at k=1 (R^2 ranges from 0.844 to 0.982). More importantly, Section 8 states that the proximity of frozen to open configurations is a 'partial, unconfirmed explanation' for the strong early-horizon result. The claim that 'useful prediction is already available after a single epoch' may well be true, but the current evaluation cannot separate the genuine early signal from the near-duplicate-configuration effect. A distance-stratified horizon analysis, or an evaluation of models trained only on first-phase broad samples and tested on second-phase boundary samples, would directly address this and should accompany any revised clai","section":"Section 4.2, Section 6.2, Section 8"}],"minor_comments":[{"comment":"The abstract's 'R^2 = 0.92-0.99' range describes the five-epoch horizon on the non-diverged cohort. It would be helpful to state that explicitly in the abstract, since Table 4 shows lower values at earlier horizons and the eligible cohort excludes diverged runs.","section":"Abstract, Section 6.1"},{"comment":"The text says every one of the twelve classification-AUC deltas has an individual 95% CI that excludes zero, but those intervals are not shown. Since the table is the paper's central evidence for the telemetry ablation, the individual intervals should be included in an appendix or in the table itself.","section":"Table 3"},{"comment":"The term 'hyperparameter-free' is potentially confusing: the paired ablation excludes the four sampled hyperparameters from the feature set, but the gradient-boosting model itself has hyperparameters. A brief clarifying remark would avoid misreading.","section":"Section 5.2, Section 6.3"},{"comment":"Removing the entire finer-granularity export for Fashion-MNIST/ResNet-18 is a defensible conservative choice, but the 160 non-identical rows are concentrated near the boundary. Since boundary-region data are the most informative for this study, it would be useful to state whether the main Table 2/Table 6 metrics are robust to the alternative of retaining those rows or resolving them by an explicit policy.","section":"Section 4.5, Appendix G"},{"comment":"The one-epoch claim is illustrated with a single domain on the open 80% partition. Given the variability across domains in Table 4, a small multi-panel version of Figure 1 for all six domains, or a rephrasing such as 'useful prediction is often available after one epoch,' would be more accurate.","section":"Section 6.2, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is unusually honest and methodical, and the Section 8 limitation is the correct issue to probe. My recommendation of major revision is driven by one load-bearing point: the frozen holdout, which carries the headline generalization claim, is not currently a valid test for distant configurations because of boundary-focused adaptive sampling. The fix is concrete and within scope: add a distance-based diagnostic and, if it shows degradation, revise the claim accordingly. I would be comfortable with acceptance after that revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent, honest empirical paper whose headline numbers are probably true for the population it sampled, but that population is boundary-enriched and the 'frozen holdout' is a split of an adaptive sample, so the numbers overstate generalization to distant new configurations. The authors admit this in Section 8, but the abstract still leads with R²=0.92–0.99, and most readers will miss the caveat.\n\nWhat's new and worth keeping: the paired, hyperparameter-free ablation showing gradient SNR and weight-norm growth improve prediction over loss/accuracy curves alone, in every domain/task with confidence intervals excluding zero. Even where the gains are small, the design is clean and the claim is modest. Treating numerical divergence as a directly predicted terminal state is a reasonable addition. The transfer analysis separating scale mismatch from loss of rank relationship is careful and useful.\n\nSoft spots, in order: (1) The frozen holdout is not an independent generalization test. Because phase-two sampling concentrates draws near the estimated success/failure boundary, held-out configurations sit close to open ones. That makes near-duplicate prediction easy. The authors say the holdout 'should temper a literal reading' as a test for distant configurations, but that sentence is buried in the limitations; the abstract and intro present the frozen partition as if it settles generalization. I'd want a distance-based diagnostic or a validation set collected under independent sampling. (2) No code or data, and exact hyperparameter range endpoints are withheld. Fixable, but it blocks verification. (3) The practical telemetry gain is negligible in some domains. The authors are upfront about that, so it's a fairness issue, not an overclaim.\n\nThe paper is well above the median: leakage-aware protocol, Nadeau–Bengio-corrected intervals, paired ablations, and an unusually candid limitations section. The CIFAR-10 MLP transfer outlier is presented as a hypothesis, not a result. This deserves a serious referee. I'd send it out, then press the authors on the independence of the holdout and require code and data.","headline":"A careful, honest empirical study whose headline R^2/AUC numbers should be read as interpolation in a boundary-enriched sample, not as a test of generalization to distant configurations.","tokens_in":23204,"tokens_out":3673,"would_cite":true,"duration_ms":39668,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training run's own first five epochs reveal its final accuracy and failure risk, with no comparison to other runs.","keywords":["early training telemetry","gradient signal-to-noise","weight-norm growth","hyperparameter sweep","learning-curve prediction","training failure prediction","gradient boosting","transfer learning"],"falsifier":"Sample a fresh cohort of hyperparameter configurations uniformly across the full ranges, train them to completion, and evaluate the same five-epoch predictors: if final-accuracy R² drops well below 0.92 or relative-classification AUC below 0.98, the generalization claim fails. Alternatively, train multiple random-seed repeats of identical configurations: if between-seed accuracy variance approaches the predictor's error floor, single-run telemetry cannot separate luck from outcome.","tokens_in":22624,"feed_emoji":"📈","tokens_out":4803,"duration_ms":47031,"temperature":0.7,"pith_summary":"The paper sets out to show that a deep network training run's eventual outcome is visible in its own first few epochs, without comparing it to any other run in a sweep. On 23,788 runs across six architecture/dataset combinations, gradient-boosted trees using five epochs of loss, accuracy, gradient-consistency and weight-norm telemetry plus the sampled hyperparameters predict final test accuracy with R² = 0.92–0.99 and predict upper-half-vs-lower-half standing with ROC-AUC 0.983–0.998 on a permanently frozen 20% of hyperparameter configurations. Most predictive power is present after a single epoch. A paired ablation shows gradient- and weight-level telemetry improves prediction over loss/accuracy curves alone in every domain and task tested, although the practical gain varies. If the claim holds, sweep schedulers could redirect compute away from doomed runs early, using only a single-run signal rather than a comparison cohort.","feed_headline":"Five epochs of telemetry predict a network's final accuracy","feed_subtitle":"Gradient consistency and weight-norm growth in the first five epochs flag doomed runs before they waste compute.","key_machinery":"The gradient signal-to-noise ratio (how consistently a fixed set of parameter-coordinate gradients points in the same direction across mini-batches within an epoch) and weight-norm growth (relative change in the total L2 norm of parameters since initialization), plus a one-time activation-saturation snapshot, are the novel signals. Ten engineered trajectory statistics per signal, combined with the four sampled hyperparameters, feed a gradient-boosted tree. A two-phase sampling procedure deliberately concentrates runs near the success/failure boundary, and a permanently frozen 20% configuration partition provides the one-shot generalization test.","core_discovery":"The central claim is that training-dynamics telemetry recorded inside a run—how consistently the gradient points in one direction, how fast the weight norm grows, and how saturated activations become—carries information about the run's final accuracy and failure risk that is not captured by the loss and accuracy curves practitioners already log. The paper demonstrates this with a controlled, hyperparameter-free ablation, showing that the relationship is strong enough after one epoch to support practical decision support. The authors are explicit that boundary-focused sampling makes some holdout configurations close to training configurations, and that results describe a boundary-heavy popula","pith_inferences":["A natural extension would sample new configurations uniformly across the full hyperparameter ranges and measure the drop in frozen-holdout performance; a large drop would show the headline numbers apply mainly to boundary-focused sweeps.","If single-epoch saturation holds at longer schedules, schedulers could use epoch-1 predictions as a near-final ranking, but seed-to-seed variance at fixed hyperparameters is untested in this data and could set the prediction floor.","The CIFAR-10 multilayer-perceptron asymmetry suggests telemetry-to-accuracy relationships are not architecture-universal; deploying a predictor on a new architecture family would likely require per-family calibration.","In language-model or large-scale vision settings, gradient consistency may be a cheap early proxy for later divergence, directly testable on existing training logs without new compute."],"forward_implications":["Run-termination heuristics can flag or rank configurations from epoch 1 without a concurrent cohort, complementing comparison-based schedulers.","Gradient and weight-norm telemetry add information beyond loss/accuracy curves in every tested domain, so internal optimization signals are worth logging as routine telemetry.","Predictors transfer well across architecturally similar domains; across datasets the underlying rank relationship survives, so domain-relative calibration, not retraining, may handle scale mismatch.","Outright numerical divergence is predictable as a distinct outcome early, suggesting separate flags for usability-threshold failures versus NaN divergence.","Because the evaluation cohort is boundary-enriched, achieved error rates describe a hard near-boundary population, not a typical practitioner sweep."],"supporting_citations":[{"why":"Closest prior work: predicts a run's final validation accuracy from early curves plus hyperparameters; the paper extends it with internal telemetry and a controlled ablation.","marker":"[2]"},{"why":"Learning-curve extrapolation method that predicts a run's eventual value from its own partial curve; supplies the single-run, curve-only baseline the ablation contrasts against.","marker":"[3]"},{"why":"Successive halving, the canonical comparison-based sweep method; defines the relative-to-cohort scheduling alternative the paper distinguishes from its isolation setting.","marker":"[7]"},{"why":"Hyperband, the bandit-based generalization of successive halving; another comparison-based baseline the paper's single-run setting complements.","marker":"[9]"},{"why":"Gradient noise scale concept from large-batch training literature; conceptually motivates the gradient signal-to-noise statistic used as internal telemetry.","marker":"[10]"},{"why":"Median-stopping rule used in a production tuning service; illustrates the comparison-against-history approach the paper contrasts with its run-isolation predictions.","marker":"[5]"}],"fun_headline_variants":["First five epochs of telemetry forecast final accuracy","One epoch of telemetry flags doomed runs","Early gradient signals predict network success","Training dynamics from epoch one flag failures"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The strong numbers rest on treating the frozen 20 percent holdout as a test of generalization to new configurations; the paper concedes (Sections 5.5 and 8) that boundary-focused sampling makes many held-out configurations sit close to, though never identical to, training configurations, so the scores may overstate performance on genuinely distant configurations.","fun_headline_variants_meta":{"raw":{"variants":["First five epochs of telemetry forecast final accuracy","One epoch of telemetry flags doomed runs","Early gradient signals predict network success","Training dynamics from epoch one flag failures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1173,"prompt_tokens":753,"completion_tokens":420,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":497,"tokens_out":420,"duration_ms":4937,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:15:50.886167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample a fresh cohort of hyperparameter configurations uniformly across the full ranges, train them to completion, and evaluate the same five-epoch predictors: if final-accuracy R² drops well below 0.92 or relative-classification AUC below 0.98, the generalization claim fails. Alternatively, train multiple random-seed repeats of identical configurations: if between-seed accuracy variance approaches the predictor's error floor, single-run telemetry cannot separate luck from outcome.","supporting_citations":[{"cited_title":"Accelerating neural architecture search using per- formance prediction","cited_arxiv_id":null,"evidence_quote":"Closest prior work: predicts a run's final validation accuracy from early curves plus hyperparameters; the paper extends it with internal telemetry and a controlled ablation."},{"cited_title":"Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves","cited_arxiv_id":null,"evidence_quote":"Learning-curve extrapolation method that predicts a run's eventual value from its own partial curve; supplies the single-run, curve-only baseline the ablation contrasts against."},{"cited_title":"Non-stochastic best arm identification and hyperparameter optimization","cited_arxiv_id":null,"evidence_quote":"Successive halving, the canonical comparison-based sweep method; defines the relative-to-cohort scheduling alternative the paper distinguishes from its isolation setting."},{"cited_title":"Hyperband: A novel bandit-based approach to hyperparameter optimization","cited_arxiv_id":null,"evidence_quote":"Hyperband, the bandit-based generalization of successive halving; another comparison-based baseline the paper's single-run setting complements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Median-stopping rule used in a production tuning service; illustrates the comparison-against-history approach the paper contrasts with its run-isolation predictions."}],"review_version":1}