{"id":"4156f84f-1284-45c2-9c8c-d8db8a69d55e","arxiv_id":"2608.02705","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Arm-specific adjustment with a pre-chosen early-training log statistic narrows confidence intervals for model performance differences at sufficient run budgets, while broad automatic selection from the log pool adds noise.","lead":"This paper tests whether statistics recorded during neural network training, such as early validation loss, can tighten confidence intervals when comparing two models' test accuracies. It finds modest gains when the statistic is chosen before the comparison, and warns that searching many log statistics usually makes comparisons worse.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adjusted CI is centered at the raw mean but uses residual variance; the resampling diagnostic targets the finite pool mean, so it cannot validate coverage for the true performance difference.","rationale":"The reader identified the internal resampling diagnostics as the weakest point, and I agree that they are the critical evidence. My concern is sharper: even if the 50-run pool were perfectly representative, the reported adjusted interval is constructed by centering at the raw mean difference while using a residual variance that belongs to a different estimator. The raw mean difference has sampling variance determined by the raw outcome variance; subtracting a covariate and then re-centering to the raw mean does not reduce that variance. This is exactly the co-produced-covariate setting the paper flags in Section 3.3, where pre-treatment independence is missing and regression-adjusted standard errors are not automatically valid. The coverage diagnostics compound the problem by checking coverage of a finite pool mean rather than the super-population estimand. As n grows toward 50, the finite-population correction makes the resampled subset mean tightly concentrated around the pool mean, so a substantially too-narrow interval can still exhibit the reported 97%+ coverage; the n=50 row is degenerate by the paper's own admission. The synthetic-null test in Section 5.2 has the same finite-pool limitation: it randomizes within an already trained pool, so it cannot probe coverage of the repeated-training distribution. Because the central pairwise claim is precisely that training-log adjustment narrows confidence intervals without changing the raw mean difference, and because the interval width is only meaningful if the interval has valid coverage for that difference, the headline empirical claim is not supported by the evidence presented. This is not merely a missing-code issue or an artifact of one cell; it is a structural mismatch between the point estimator and the variance estimator. A corrected version could either report the adjusted mean as the point estimate and give it a valid cross-fitted variance, or validate coverage against truly independent re-trainings; but the paper as written draws an invalid inference. The single-arm variance-reduction and covariate-selection findings may still be useful, but they do not rescue the pairwise confidence-interval claim.","tokens_in":12320,"tokens_out":18130,"duration_ms":180809,"concrete_test":"Run a Monte Carlo simulation calibrated to a high-VR cell, e.g., ViT-Tiny/CIFAR-100 with raw mean 70.06 pp, raw SD 0.46 pp, and VR 17.8%. For 10,000 repetitions, draw two independent arms, each with n=50 bivariate normal (Y, X) pairs whose squared correlation matches the pre-specified validation-loss covariate; implement the paper's K=5 cross-fitting, recenter adjusted outcomes to the raw difference, and build the Welch interval centered at the raw difference. Record coverage of the true super-population difference Δ=0 and, in parallel, coverage of the realized 50-run pool difference as in Section 5.1. If pool-mean coverage is at least 97% while true-mean coverage is below 95% (or below 93% for cells with VR near 34%), the reported intervals are invalid for the stated estimand and the resampling diagnostic is too lenient.","verdict_should_be":"REJECT","load_bearing_attack":"Table 2 reports 95% Welch intervals for Δ = μ_A − μ_B whose half-width uses within-arm variances of cross-fitted adjusted outcomes, while the interval is centered at the raw mean difference (Section 3.4). The sampling variance of the raw mean difference is governed by the raw run variance divided by n, not by the residual variance after covariate adjustment; recentering adjusted outcomes to the raw mean changes only the location, not the estimator, so a smaller adjusted variance does not reduce the variance of the reported point estimate. A valid variance reduction requires the point estimate itself to be the adjusted estimator, for example with a fixed covariate mean or an influence-function construction. Because the covariates are co-produced (Section 3.3), the pre-treatment independence that justifies regression adjustment in A/B tests is absent, and the OLS residual variance does not estimate the variance of the raw mean. Freedman's warning, which the paper cites, applies directly here: regression adjustment can make standard errors misleading when the adjustment is not tied to the estimator. The paper's own coverage diagnostic cannot catch this. Section 5.1 resamples n runs from the fixed pool of 50 and checks coverage of the 50-run mean, not the super-population estimand μ_A − μ_B. For n close to 50, the subset mean is much less variable around the pool mean than around the true mean, with finite-population correction factor 1 − n/50, which is 0.4 at n=30; the n=50 row is degenerate. A too-narrow interval can therefore show coverage at or above 97% in Table 2 while under-covering the actual target. The synthetic-null test in Section 5.2 checks rejection under an exchangeable split of the same pool, again a finite-pool null rather than a repeated-training null.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether training-log covariates from the same stochastic runs used to evaluate a model can reduce the uncertainty of pairwise model-performance comparisons. It proposes arm-specific covariate adjustment: within each arm, training-log statistics are centered and a cross-fitted OLS coefficient is used to define adjusted outcomes, while the raw mean difference is retained as the point estimate. The adjusted within-arm variances are then combined into a Welch-style confidence interval. The paper reports a 3x3 factorial vision study (three architectures x three datasets, 50 runs per cell), including interval coverage under resampling from the trained runs, a synthetic-null calibration, pairwise interval-width comparisons, single-arm variance reductions, and an analysis of covariate-selection risk. The main empirical claims are that a pre-specified early validation-loss adjustment narrows eight of nine pairwise intervals at n=50 and that data-driven selection from a large candidate pool often backfires.","tokens_in":12569,"tokens_out":7406,"duration_ms":70590,"significance":"If the proposed adjustment were valid, the paper would make a practically useful contribution: training logs are cheap and already available, and a method to tighten comparisons without extra runs would be valuable to the benchmarking community. The experimental design is careful in several respects: 450 runs with seeded randomness, cross-fitted coefficient estimation, raw baselines kept in every table, pre-specified versus exploratory analyses kept separate, and a candid limitations section. The paper also makes a useful negative empirical point, that broad outcome-based search over many training-log statistics inflates variance even when useful signal exists. However, the central inferential claim is not supported: the reported confidence intervals are centered at the raw mean but their width is computed from residual variance, which is not the variance of the raw mean estimator. The calibration diagnostics check coverage of the finite pool mean rather than the repeated-training estimand, so they cannot remedy this. Because the main contribution rests on this interval construction, the paper in its current form does not establish its title claim.","major_comments":[{"comment":"The central claim is not supported because the reported interval uses a variance that does not correspond to the reported point estimate. The interval is centered at the raw mean difference \\bar Y_A - \\bar Y_B, but its width is computed from the within-arm variances of recentered cross-fitted adjusted outcomes. For a fixed estimator \\bar Y_m, Var(\\bar Y_m) = σ_m^2 / n_m; replacing σ_m^2 with the residual variance after regressing Y_m on a co-produced covariate does not reduce the variance of \\bar Y_m. After recentering, the adjusted outcomes are cross-correlated through the estimated covariate mean, so the iid variance formula used in Eq. (5) is not valid. A legitimate variance reduction requires changing the point estimator itself, e.g., to a regression estimator \\bar Y_m - \\hat θ_m(\\bar X_m - \\hat μ_{X,m}) or an influence-function-based estimator with the appropriate standard error. The paper’s own reference to Freedman (2008) is apt: regression adjustment can make standard errors misleading when the adjustment is not tied to the estimator.","section":"§3.1, §3.5, Eqs. (2)–(5)"},{"comment":"The calibration diagnostics target the wrong estimand. Section 5.1 repeatedly resamples n runs from the fixed pool of 50 and checks coverage of the 50-run mean, not μ_m = E[Y_{m,s}] over repeated training. For n < 50, the variance of the subset mean around the pool mean carries a finite-population correction 1 − n/50; at n = 30 this factor is 0.4, making the diagnostic target far less variable than the true repeated-training mean. The high coverage entries in Table 1 (e.g., 99.7–99.9% at n = 30) are therefore expected even for an invalid interval, and the n = 50 row is deterministic by construction. The synthetic-null split in Section 5.2 has the same finite-population issue: the two fake arms are drawn without replacement from a fixed pool, so the null distribution is not the repeated-training distribution of Eq. (1). These diagnostics cannot validate the coverage of intervals for Δ.","section":"§5.1, §5.2, Table 1"}],"minor_comments":[{"comment":"The caption defines ΔCI and HW but not the Cov column; the text says it is the minimum adjusted coverage over n, but the caption should state this directly and explain the row ordering.","section":"Table 2"},{"comment":"Calling the recenter-to-raw-mean interval a “conservative reporting choice” is misleading, because the interval is not necessarily conservative relative to the true coverage; a more neutral phrasing would be warranted.","section":"§3.4"},{"comment":"The variance-reduction percentages are reported without standard errors or confidence intervals, so it is not possible to tell whether differences between configurations reflect real effects or resampling noise; some measure of uncertainty would strengthen the table.","section":"§5.4, Table 3"},{"comment":"No code or data release is provided, and because the main evidence comes from resampling diagnostics, public release of the 450-run log and accuracy data would substantially improve reproducibility and allow independent checks of the central claim.","section":"General"}],"recommendation":"reject","confidential_remarks":"For the editor: the experimental effort and reporting transparency are commendable, but the core interval construction is not a valid confidence interval for the stated estimand, and the calibration diagnostics target a finite pool mean rather than the repeated-training distribution. Fixing this would require redefining the estimator and re-running the analysis, not just local edits, so I recommend rejection. The authors might be able to repurpose the study as a cautionary demonstration of why recentered residual variance should not be used for inference, but that would be a different paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chris,\n\nThe headline is not great: the paper's main claim—that arm-specific covariate adjustment tightens confidence intervals for model-performance differences—doesn't hold up on inspection. The reported interval is centered at the raw mean difference, but the standard error is computed from the cross-fitted adjusted outcomes. That's the variance of a different estimator. You can't shrink the uncertainty of the raw mean by subtracting a covariate and then keeping the raw mean as your point estimate. The adjusted variance is the residual variance, not the variance of the estimator you're actually using. The paper's own recentering (Section 3.4) makes this explicit: they preserve the raw difference, which means the adjusted variance is not tied to the point estimate.\n\nThe coverage diagnostics can't rescue it. Subsampling from the fixed pool of 50 runs and checking coverage of the 50-run mean is a finite-population check. The n=50 row is degenerate, and even at n=30 the subset mean is much less variable around the pool mean than around the true mean. So the high coverage numbers in Table 2 are expected even if the intervals badly under-cover the true performance difference. The synthetic null in Section 5.2 has the same problem—it's an exchangeable split of the same pool, not repeated training runs.\n\nWhat's genuinely good: the empirical study is careful and honest. The finding that outcome-guided selection from hundreds of log statistics usually backfires is real and useful. The single-best OLS selector producing negative VR in six of nine cells, while post hoc scans show signal exists, is a nice illustration of selection noise. The PCA comparison is also thoughtful. The fixed validation-loss adjustment is pre-specified and the arm-specific treatment of co-produced covariates is the right instinct. The paper is open about the co-produced nature of the covariates and cites Freedman and Lin, but doesn't fully apply that warning to its own estimator.\n\nThe lack of code and data is a problem for a results-driven paper. The encoded string near Section 5.5 is irrelevant but weird.\n\nWho is this for? Anyone running multi-seed benchmarks will benefit from the selection-risk warning. But they should not adopt the adjusted interval procedure as a way to get tighter CI bounds until the estimator is fixed—for example, by using the adjusted mean as the point estimate with a fixed covariate mean, or an influence-function construction.\n\nMy recommendation: send to a serious referee. The idea is worth discussing and the empirical result on selection noise is a legitimate contribution. But the pairwise CI claim needs a major rework, and the paper should not be accepted as is.\n\nBest.","headline":"The paper's hoped-for precision gain in pairwise model comparisons is not valid as written; the selection-noise warning is the real contribution.","tokens_in":13186,"tokens_out":4141,"would_cite":false,"duration_ms":39153,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Arm-specific adjustment of training-log covariates tightens confidence intervals for model-performance differences when the covariate is chosen before the comparison.","keywords":["model comparison","covariate adjustment","variance reduction","training logs","confidence intervals","cross-fitting","stochastic training"],"falsifier":"Run a fresh arm with many more repeats (say 500 per cell) on the same three-architecture, three-dataset grid, using the same fixed validation-loss covariate, and check whether the adjusted 95% intervals built from 50-run subsamples contain the 500-run mean difference at the advertised rate; material under-coverage would show the resampling diagnostic is too optimistic, while a null-calibration check that finds more than about 7% rejection in many cells would signal inflated Type I error.","tokens_in":12053,"feed_emoji":"📊","tokens_out":5623,"duration_ms":44729,"temperature":0.7,"pith_summary":"This paper asks whether the logs produced during a training run—early validation loss, gradient norms, batch-loss summaries—can reduce the uncertainty in comparing two stochastically trained models, without changing the reported accuracy difference. It proposes arm-specific covariate adjustment: each model is adjusted only with statistics from its own runs, and the raw mean difference stays the point estimate. In a 3×3 vision experiment (three architectures, three datasets, 50 runs per cell), a fixed early validation-loss covariate narrows eight of nine pairwise confidence intervals at the largest run budget, with half-width reductions of 1.9 to 8.8 percent. The gains are conditional: they require enough runs to estimate the adjustment stably, and searching a large log pool for the most correlated statistic usually adds more noise than it removes. A sympathetic reader would take the paper as establishing that training logs are a useful, cheap source of precision—provided the covariate is chosen in advance or avoids outcome-based selection.","feed_headline":"Pre-fixed training-log stats shrink model-comparison error bars","feed_subtitle":"A 450-run vision experiment: early validation loss narrows 8 of 9 comparison intervals, while log-pool searches backfire.","key_machinery":"The central mechanism is arm-specific covariate adjustment, defined by $Y^{\\text{adj}}_{m,s} = Y_{m,s} - \\theta_m^\\top (X_{m,s} - \\mu_{X,m})$, where $\\theta_m$ and $\\mu_{X,m}$ are estimated within each model from its own runs by five-fold cross-fitting. Because each arm is adjusted separately, the adjustment cannot absorb part of the model difference, and recentering keeps the point estimate equal to the raw mean difference. The workhorse identity is the variance reduction $\\text{VR}_m = 1 - \\text{Var}(Y^{\\text{adj}}_m)/\\text{Var}(Y_m) = \\rho^2$ for the population-optimal single covariate, with realized VR shrinking by roughly $1/n_{\\text{train}}$; this sets the threshold that a covariate needs $\\rho^2 > 1/n_{\\text{train}}$ to pay for itself. Cross-fitting, coverage diagnostics from resampling the 50 trained runs, and a synthetic-null calibration are the supporting apparatus that check whether the tighter intervals are trustworthy.","core_discovery":"The central claim is that arm-specific covariate adjustment with a training-log statistic fixed in advance reduces the standard error of a performance difference and narrows its confidence interval, while leaving the raw mean difference unchanged. The adjustment centers each run's outcome by subtracting the fitted coefficient times the centered covariate, with coefficients and covariate means estimated by five-fold cross-fitting to avoid direct overfitting. In the primary pairwise analysis, validation loss at the first-third epoch makes adjusted Welch intervals narrower in eight of nine model pairs at n=50 runs per arm, and the resampling diagnostic keeps adjusted coverage at or above 97.2 percent. Single-arm variance reduction is positive for ViT-Tiny on all datasets and for ConvNeXt-Tiny on two of three, but negative for ResNet-18; data-driven selection from hundreds of candidates produces negative variance reduction in six of nine cells, even where useful covariates exist in hindsight. The paper's conclusion is conditional: training logs can tighten model comparisons, but only when selection noise is controlled.","pith_inferences":["If the resampling and synthetic-null diagnostics faithfully represent repeated training, the method transfers directly to an always-valid confidence sequence, where covariate adjustment would shorten the time to a fixed-width decision; the paper mentions this direction but does not develop it.","The arm-specific design suggests a general recipe for model-comparison benchmarks: treat each model as its own experiment arm and use only within-arm covariates, which parallels pre-specified analysis plans and reduces the incentive to cherry-pick a covariate after seeing outcomes.","A testable extension would compare the fixed validation-loss covariate against a small pre-registered set of PCA summaries on larger models and datasets; the current evidence cannot say whether selection noise grows or shrinks as run budgets scale."],"forward_implications":["A pre-specified early training-log statistic can reduce the number of runs needed to reach a target precision; the largest single-arm gain (34.1% variance reduction) is equivalent to 75.9 effective runs from a 50-run budget.","With only 10 runs per arm, adjustment systematically widens intervals, so the method is useful only when the run budget is large enough for stable coefficient estimation.","Which training-log statistic helps depends on architecture and training recipe: validation-loss adjustment consistently helps ViT-Tiny and ConvNeXt-Tiny but not ResNet-18, so arm-specific reporting is necessary.","Searching a large candidate pool for the most correlated statistic usually backfires: the nested selector is negative in 39 of 45 family-cell combinations, so exploratory adjustments should be labeled as such."],"supporting_citations":[{"why":"Documents run-to-run variance in machine learning benchmarks, motivating the need to reduce standard errors without extra compute.","marker":"(Bouthillier et al., 2021)"},{"why":"Introduces regression adjustment (CUPED) using pre-experiment covariates, the template the paper adapts to training logs.","marker":"(Deng et al., 2013)"},{"why":"Shows that regression adjustment can worsen precision or mislead standard errors under randomization, motivating the paper's caution with co-produced covariates.","marker":"(Freedman, 2008)"},{"why":"Provides large-sample reassurance and an interaction-centered robust approach that informs the paper's arm-specific design choices.","marker":"(Lin, 2013)"},{"why":"Supplies the cross-fitting machinery used to estimate adjustment coefficients without direct overfitting.","marker":"(Chernozhukov et al., 2018)"},{"why":"Demonstrates that early training signals predict converged performance, the learning-curve structure the paper repurposes as a fixed covariate.","marker":"(Domhan et al., 2015)"},{"why":"Supplies the classical Monte Carlo variance-reduction principle that subtracting a correlated mean-zero signal reduces estimator variance.","marker":"(Owen, 2013)"}],"fun_headline_variants":["Preset training-log stats tighten model comparison error bars","Fix one training log and get sharper model comparisons","Early training logs slim model comparison uncertainty","Arm-specific log tweaks narrow comparison intervals","Preselected training stats beat blind log-pool searches"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The validity of the tighter intervals depends on whether the 50 trained runs used to build the resampling and synthetic-null diagnostics represent the true distribution of repeated training runs; because training-log covariates are co-produced with the outcome, there is no distribution-free finite-sample guarantee for adjusted intervals.","fun_headline_variants_meta":{"raw":{"variants":["Preset training-log stats tighten model comparison error bars","Fix one training log and get sharper model comparisons","Early training logs slim model comparison uncertainty","Arm-specific log tweaks narrow comparison intervals","Preselected training stats beat blind log-pool searches"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001351,"raw_usage":{"total_tokens":5453,"prompt_tokens":882,"completion_tokens":4571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":4500}},"tokens_in":498,"tokens_out":4571,"duration_ms":28690,"temperature":1.0,"reasoning_tokens":4500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:00:52.177813+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a fresh arm with many more repeats (say 500 per cell) on the same three-architecture, three-dataset grid, using the same fixed validation-loss covariate, and check whether the adjusted 95% intervals built from 50-run subsamples contain the 500-run mean difference at the advertised rate; material under-coverage would show the resampling diagnostic is too optimistic, while a null-calibration check that finds more than about 7% rejection in many cells would signal inflated Type I error.","supporting_citations":[{"cited_title":"Accounting for variance in machine learning benchmarks","cited_arxiv_id":null,"evidence_quote":"Documents run-to-run variance in machine learning benchmarks, motivating the need to reduce standard errors without extra compute."},{"cited_title":"Improving the sensitivity of online controlled experiments by utilizing pre-experiment data","cited_arxiv_id":null,"evidence_quote":"Introduces regression adjustment (CUPED) using pre-experiment covariates, the template the paper adapts to training logs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that regression adjustment can worsen precision or mislead standard errors under randomization, motivating the paper's caution with co-produced covariates."},{"cited_title":"Agnostic notes on regression adjustments to experimental data: R eexamining F reedman's critique","cited_arxiv_id":null,"evidence_quote":"Provides large-sample reassurance and an interaction-centered robust approach that informs the paper's arm-specific design choices."},{"cited_title":"T., and Hutter, F","cited_arxiv_id":null,"evidence_quote":"Demonstrates that early training signals predict converged performance, the learning-curve structure the paper repurposes as a fixed covariate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the classical Monte Carlo variance-reduction principle that subtracting a correlated mean-zero signal reduces estimator variance."}],"review_version":2}