Can Training Logs Make Model Comparisons More Precise?

Wei-Jung Huang

REVIEW 2 major objections 4 minor 22 references

Can Training Logs Make Model Comparisons More Precise?

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Arm-specific adjustment of training-log covariates tightens confidence intervals for model-performance differences when the covariate is chosen before the comparison.

desk verdict The paper's hoped-for precision gain in pairwise model comparisons is not valid as written; the selection-noise warning is the real contribution. read the letter →

arxiv 2608.02705 v1 pith:2R3ABMTD submitted 2026-08-03 cs.LG cs.AIstat.ML

Wei-Jung Huang This is my paper · ORCID

classification cs.LGcs.AIstat.ML

keywords modelcomparisoncovariateadjustmentvariancereductiontraininglogsconfidenceintervalscross-fittingstochastic

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the logs produced during a training run—early validation loss, gradient norms, batch-loss summaries—can reduce the uncertainty in comparing two stochastically trained models, without changing the reported accuracy difference. It proposes arm-specific covariate adjustment: each model is adjusted only with statistics from its own runs, and the raw mean difference stays the point estimate. In a 3×3 vision experiment (three architectures, three datasets, 50 runs per cell), a fixed early validation-loss covariate narrows eight of nine pairwise confidence intervals at the largest run budget, with half-width reductions of 1.9 to 8.8 percent. The gains are conditional: they require enough runs to estimate the adjustment stably, and searching a large log pool for the most correlated statistic usually adds more noise than it removes. A sympathetic reader would take the paper as establishing that training logs are a useful, cheap source of precision—provided the covariate is chosen in advance or avoids outcome-based selection.

What carries the argument

The central mechanism is arm-specific covariate adjustment, defined by $Y^{\text{adj}}_{m,s} = Y_{m,s} - \theta_m^\top (X_{m,s} - \mu_{X,m})$, where $\theta_m$ and $\mu_{X,m}$ are estimated within each model from its own runs by five-fold cross-fitting. Because each arm is adjusted separately, the adjustment cannot absorb part of the model difference, and recentering keeps the point estimate equal to the raw mean difference. The workhorse identity is the variance reduction $\text{VR}_m = 1 - \text{Var}(Y^{\text{adj}}_m)/\text{Var}(Y_m) = \rho^2$ for the population-optimal single covariate, with realized VR shrinking by roughly $1/n_{\text{train}}$; this sets the threshold that a covariate needs $\rho^2 > 1/n_{\text{train}}$ to pay for itself. Cross-fitting, coverage diagnostics from resampling the 50 trained runs, and a synthetic-null calibration are the supporting apparatus that check whether the tighter intervals are trustworthy.

What would settle it

Run a fresh arm with many more repeats (say 500 per cell) on the same three-architecture, three-dataset grid, using the same fixed validation-loss covariate, and check whether the adjusted 95% intervals built from 50-run subsamples contain the 500-run mean difference at the advertised rate; material under-coverage would show the resampling diagnostic is too optimistic, while a null-calibration check that finds more than about 7% rejection in many cells would signal inflated Type I error.

Watch

Extended reading notes

Core claim

The central claim is that arm-specific covariate adjustment with a training-log statistic fixed in advance reduces the standard error of a performance difference and narrows its confidence interval, while leaving the raw mean difference unchanged. The adjustment centers each run's outcome by subtracting the fitted coefficient times the centered covariate, with coefficients and covariate means estimated by five-fold cross-fitting to avoid direct overfitting. In the primary pairwise analysis, validation loss at the first-third epoch makes adjusted Welch intervals narrower in eight of nine model pairs at n=50 runs per arm, and the resampling diagnostic keeps adjusted coverage at or above 97.2 percent. Single-arm variance reduction is positive for ViT-Tiny on all datasets and for ConvNeXt-Tiny on two of three, but negative for ResNet-18; data-driven selection from hundreds of candidates produces negative variance reduction in six of nine cells, even where useful covariates exist in hindsight. The paper's conclusion is conditional: training logs can tighten model comparisons, but only when selection noise is controlled.

Load-bearing premise

The validity of the tighter intervals depends on whether the 50 trained runs used to build the resampling and synthetic-null diagnostics represent the true distribution of repeated training runs; because training-log covariates are co-produced with the outcome, there is no distribution-free finite-sample guarantee for adjusted intervals.

Editorial extensions

If this is right

A pre-specified early training-log statistic can reduce the number of runs needed to reach a target precision; the largest single-arm gain (34.1% variance reduction) is equivalent to 75.9 effective runs from a 50-run budget.
With only 10 runs per arm, adjustment systematically widens intervals, so the method is useful only when the run budget is large enough for stable coefficient estimation.
Which training-log statistic helps depends on architecture and training recipe: validation-loss adjustment consistently helps ViT-Tiny and ConvNeXt-Tiny but not ResNet-18, so arm-specific reporting is necessary.
Searching a large candidate pool for the most correlated statistic usually backfires: the nested selector is negative in 39 of 45 family-cell combinations, so exploratory adjustments should be labeled as such.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the resampling and synthetic-null diagnostics faithfully represent repeated training, the method transfers directly to an always-valid confidence sequence, where covariate adjustment would shorten the time to a fixed-width decision; the paper mentions this direction but does not develop it.
The arm-specific design suggests a general recipe for model-comparison benchmarks: treat each model as its own experiment arm and use only within-arm covariates, which parallels pre-specified analysis plans and reduces the incentive to cherry-pick a covariate after seeing outcomes.
A testable extension would compare the fixed validation-loss covariate against a small pre-registered set of PCA summaries on larger models and datasets; the current evidence cannot say whether selection noise grows or shrinks as run budgets scale.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Chris,

The headline is not great: the paper's main claim—that arm-specific covariate adjustment tightens confidence intervals for model-performance differences—doesn't hold up on inspection. The reported interval is centered at the raw mean difference, but the standard error is computed from the cross-fitted adjusted outcomes. That's the variance of a different estimator. You can't shrink the uncertainty of the raw mean by subtracting a covariate and then keeping the raw mean as your point estimate. The adjusted variance is the residual variance, not the variance of the estimator you're actually using. The paper's own recentering (Section 3.4) makes this explicit: they preserve the raw difference, which means the adjusted variance is not tied to the point estimate.

The coverage diagnostics can't rescue it. Subsampling from the fixed pool of 50 runs and checking coverage of the 50-run mean is a finite-population check. The n=50 row is degenerate, and even at n=30 the subset mean is much less variable around the pool mean than around the true mean. So the high coverage numbers in Table 2 are expected even if the intervals badly under-cover the true performance difference. The synthetic null in Section 5.2 has the same problem—it's an exchangeable split of the same pool, not repeated training runs.

What's genuinely good: the empirical study is careful and honest. The finding that outcome-guided selection from hundreds of log statistics usually backfires is real and useful. The single-best OLS selector producing negative VR in six of nine cells, while post hoc scans show signal exists, is a nice illustration of selection noise. The PCA comparison is also thoughtful. The fixed validation-loss adjustment is pre-specified and the arm-specific treatment of co-produced covariates is the right instinct. The paper is open about the co-produced nature of the covariates and cites Freedman and Lin, but doesn't fully apply that warning to its own estimator.

The lack of code and data is a problem for a results-driven paper. The encoded string near Section 5.5 is irrelevant but weird.

Who is this for? Anyone running multi-seed benchmarks will benefit from the selection-risk warning. But they should not adopt the adjusted interval procedure as a way to get tighter CI bounds until the estimator is fixed—for example, by using the adjusted mean as the point estimate with a fixed covariate mean, or an influence-function construction.

My recommendation: send to a serious referee. The idea is worth discussing and the empirical result on selection noise is a legitimate contribution. But the pairwise CI claim needs a major rework, and the paper should not be accepted as is.

Best.

Referee Report

2 major / 4 minor

Summary. The paper studies whether training-log covariates from the same stochastic runs used to evaluate a model can reduce the uncertainty of pairwise model-performance comparisons. It proposes arm-specific covariate adjustment: within each arm, training-log statistics are centered and a cross-fitted OLS coefficient is used to define adjusted outcomes, while the raw mean difference is retained as the point estimate. The adjusted within-arm variances are then combined into a Welch-style confidence interval. The paper reports a 3x3 factorial vision study (three architectures x three datasets, 50 runs per cell), including interval coverage under resampling from the trained runs, a synthetic-null calibration, pairwise interval-width comparisons, single-arm variance reductions, and an analysis of covariate-selection risk. The main empirical claims are that a pre-specified early validation-loss adjustment narrows eight of nine pairwise intervals at n=50 and that data-driven selection from a large candidate pool often backfires.

Significance. If the proposed adjustment were valid, the paper would make a practically useful contribution: training logs are cheap and already available, and a method to tighten comparisons without extra runs would be valuable to the benchmarking community. The experimental design is careful in several respects: 450 runs with seeded randomness, cross-fitted coefficient estimation, raw baselines kept in every table, pre-specified versus exploratory analyses kept separate, and a candid limitations section. The paper also makes a useful negative empirical point, that broad outcome-based search over many training-log statistics inflates variance even when useful signal exists. However, the central inferential claim is not supported: the reported confidence intervals are centered at the raw mean but their width is computed from residual variance, which is not the variance of the raw mean estimator. The calibration diagnostics check coverage of the finite pool mean rather than the repeated-training estimand, so they cannot remedy this. Because the main contribution rests on this interval construction, the paper in its current form does not establish its title claim.

major comments (2)

[§3.1, §3.5, Eqs. (2)–(5)] The central claim is not supported because the reported interval uses a variance that does not correspond to the reported point estimate. The interval is centered at the raw mean difference \bar Y_A - \bar Y_B, but its width is computed from the within-arm variances of recentered cross-fitted adjusted outcomes. For a fixed estimator \bar Y_m, Var(\bar Y_m) = σ_m^2 / n_m; replacing σ_m^2 with the residual variance after regressing Y_m on a co-produced covariate does not reduce the variance of \bar Y_m. After recentering, the adjusted outcomes are cross-correlated through the estimated covariate mean, so the iid variance formula used in Eq. (5) is not valid. A legitimate variance reduction requires changing the point estimator itself, e.g., to a regression estimator \bar Y_m - \hat θ_m(\bar X_m - \hat μ_{X,m}) or an influence-function-based estimator with the appropriate standard error. The paper’s own reference to Freedman (2008) is apt: regression adjustment can make standard errors misleading when the adjustment is not tied to the estimator.
[§5.1, §5.2, Table 1] The calibration diagnostics target the wrong estimand. Section 5.1 repeatedly resamples n runs from the fixed pool of 50 and checks coverage of the 50-run mean, not μ_m = E[Y_{m,s}] over repeated training. For n < 50, the variance of the subset mean around the pool mean carries a finite-population correction 1 − n/50; at n = 30 this factor is 0.4, making the diagnostic target far less variable than the true repeated-training mean. The high coverage entries in Table 1 (e.g., 99.7–99.9% at n = 30) are therefore expected even for an invalid interval, and the n = 50 row is deterministic by construction. The synthetic-null split in Section 5.2 has the same finite-population issue: the two fake arms are drawn without replacement from a fixed pool, so the null distribution is not the repeated-training distribution of Eq. (1). These diagnostics cannot validate the coverage of intervals for Δ.

minor comments (4)

[Table 2] The caption defines ΔCI and HW but not the Cov column; the text says it is the minimum adjusted coverage over n, but the caption should state this directly and explain the row ordering.
[§3.4] Calling the recenter-to-raw-mean interval a “conservative reporting choice” is misleading, because the interval is not necessarily conservative relative to the true coverage; a more neutral phrasing would be warranted.
[§5.4, Table 3] The variance-reduction percentages are reported without standard errors or confidence intervals, so it is not possible to tell whether differences between configurations reflect real effects or resampling noise; some measure of uncertainty would strengthen the table.
[General] No code or data release is provided, and because the main evidence comes from resampling diagnostics, public release of the 450-run log and accuracy data would substantially improve reproducibility and allow independent checks of the central claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the adjusted-variance and coverage diagnostics are used transparently, and the central comparison is pre-specified.

full rationale

The derivation chain is self-contained. The arm-specific adjustment coefficient is estimated by 5-fold cross-fitting (Eq. 4), so the variance reduction reported in Table 3 is an out-of-fold estimate rather than an in-sample fit renamed as prediction. The pairwise covariate (validation loss at the first-third epoch) is fixed before inspecting the candidate pool (Section 4.5), and the raw mean difference is preserved by recentering; the paper does not claim to predict unseen data from a fitted parameter. The coverage checks (Sections 5.1 and 5.2) resample from the 50 trained runs and are explicitly described as finite-pool diagnostics rather than distribution-free guarantees, including the n=50 row being 'by construction' — an acknowledged implementation check, not load-bearing evidence. The statistical concern that the adjusted interval is centered at the raw mean while using residual variance is a real validity question, but it is not a circularity: the interval width is not an input to the adjustment fit, and the empirical finding that cross-fitted adjusted variance is lower than raw variance is not guaranteed by construction. No self-citation chain or imported uniqueness theorem is load-bearing. Therefore no step of the claimed derivation reduces to its own input.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method depends on estimated per-arm OLS coefficients and a pre-chosen covariate definition, plus the exchangeability of repeated runs. The paper introduces no new physical or scientific entities. The main unstated burden is that coverage rests on internal resampling rather than an independent sample of training runs.

free parameters (4)

Per-arm OLS adjustment coefficient theta_m = Estimated from within-arm runs; individual values not reported
Used in Equation (2) to construct adjusted run values; its estimation cost is the reason realized variance reduction falls below rho squared, so the central comparisons depend on it.
Validation-loss covariate definition (first-third epoch) = Validation loss at approximately one third of training
Chosen before analyzing the candidate pool and motivated by learning-curve literature (Section 4.5); the reported pairwise gains in Table 2 are conditional on this specific pre-chosen statistic, not on a searched optimum.
Cross-fitting fold count K = 5
Chosen in Section 3.4; a different K would change the estimation penalty and the run-budget thresholds.
Principal components kept in PCA adjustment = 1 (first PC per covariate family)
Chosen in Section 4.5; the paper reports that k=2 or k=3 degrades performance, so the PCA result is tied to this hand-chosen value.

assumptions (4)

domain assumption Repeated runs for a fixed model and recipe are exchangeable draws from mu_m
Equation (1) defines the estimand and every variance and coverage calculation presupposes this repeated-sampling model.
domain assumption The 50 runs trained per cell are a sufficient proxy for the run distribution when evaluating coverage
Sections 5.1 and 5.2 resample from the same 50 runs; the conclusion that adjustment does not cause under-coverage depends on this proxy.
standard math Cross-fitted OLS residuals yield consistent variance estimates for the adjusted outcomes
Section 3.4 invokes cross-fitting (Chernozhukov et al., 2018); no formal proof is given in the paper, and calibration is checked empirically.
domain assumption Validation measurements do not leak into the outcome because the final-epoch test checkpoint is fixed before covariate analysis
Section 4.1; if validation-based early stopping or checkpoint selection were used, validation covariates would be post-treatment and the adjustment would be invalid.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Can Training Logs Make Model Comparisons More Precise?." pith.science (2026). https://pith.science/paper/2R3ABMTD

@misc{pith2026260802705,
  author       = {Pith},
  title        = {Pith review of: Can Training Logs Make Model Comparisons More Precise?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2R3ABMTD}},
  note         = {Machine review of arXiv:2608.02705}
}

read the original abstract

Comparing stochastically trained models requires estimating both a performance difference and its uncertainty from repeated runs. We study whether training logs from those same runs can make such comparisons more precise. Because training-log covariates are produced during training rather than measured before it, we use arm-specific covariate adjustment: each model is adjusted only with statistics from its own runs, and the raw mean difference remains the reported effect. In a vision study spanning three architectures and three datasets, simple adjustments based on early training logs often reduce uncertainty in model comparisons. The main limitation is covariate selection. Broadly searching the log pool for the most correlated statistic often adds more noise than it removes, even when useful statistics exist in hindsight. Training logs therefore appear useful for more precise model comparisons, but only when the adjustment avoids large selection noise.

Figures

Figures reproduced from arXiv: 2608.02705 by the authors.

**Figure 1.** Covariate-outcome scatter for four cells. Each point is one run. ViT-Tiny (bottom) shows stronger first-third relationships (ρ 2 = 23.0% and 30.3%); ResNet-18 (top) shows weaker relationships (ρ 2 = 13.8% and 14.6%). Data-driven selection. The single-best OLS selector produces negative VR in six of nine cells. The failures are not small: ViT-Tiny on CIFAR-10 has ρ 2 = 23% for its best early validation covariate, ye… view at source ↗

**Figure 3.** Median VR vs. run budget. Solid: fixed validation-loss adjustment; dashed: single-best selector. The fixed adjustment degrades more smoothly; automatic selection remains brittle at all tested budgets. 5.6. Variance Reduction vs. Run Budget At smaller run budgets, fixed validation-loss adjustment degrades more gracefully than data-driven selection, which is not reliably positive below n = 50 in these experiments (Fi… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 14 canonical work pages

[1]

Accounting for variance in machine learning benchmarks

Bouthillier, X., Delaunay, P., Bronzi, M., Trofimov, A., et al. Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems, volume 3, pp.\ 747--769, 2021

work page 2021
[2]

Double/debiased machine learning for treatment and structural parameters

Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21 0 (1): 0 C1--C68, 2018

2018
[3]

A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., and Vinyals, O

Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., and Vinyals, O. The Benchmark Lottery . arXiv preprint arXiv:2107.07002, 2021

arXiv 2021
[4]

Improving the sensitivity of online controlled experiments by utilizing pre-experiment data

Deng, A., Xu, Y., Kohavi, R., and Walker, T. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, pp.\ 123--132, 2013

work page 2013
[5]

Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping

Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., and Smith, N. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020

arXiv 2002
[6]

T., and Hutter, F

Domhan, T., Springenberg, J. T., and Hutter, F. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, pp.\ 3460--3468, 2015

work page 2015
[7]

Freedman, D. A. On regression adjustments to experimental data. Advances in Applied Mathematics, 40 0 (2): 0 180--193, 2008

work page 2008
[8]

B., Stern, H

Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. Bayesian Data Analysis. Chapman and Hall/CRC, 3rd edition, 2013

work page 2013

Show all 22 references

[9]

Deep residual learning for image recognition

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 770--778, 2016

2016
[10]

Deep reinforcement learning that matters

Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

2018
[11]

Always valid inference: Continuous monitoring of a/b tests

Johari, R., Koomen, P., Pekelis, L., and Walsh, D. Always valid inference: Continuous monitoring of a/b tests. Operations Research, 70 0 (3): 0 1806--1821, 2022

2022
[12]

Learning multiple layers of features from tiny images

Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

2009
[13]

and Yang, X

Le, Y. and Yang, X. Tiny imagenet visual recognition challenge. CS 231N, 7 0 (7): 0 3, 2015

2015
[14]

Agnostic notes on regression adjustments to experimental data: R eexamining F reedman's critique

Lin, W. Agnostic notes on regression adjustments to experimental data: R eexamining F reedman's critique. The Annals of Applied Statistics, 7 0 (1): 0 295--318, 2013

2013
[15]

A ConvNet for the 2020s

Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11976--11986, 2022

2022
[16]

Are GAN s created equal? A large-scale study

Lucic, M., Kurach, K., Michalski, M., Gelly, S., and Bousquet, O. Are GAN s created equal? A large-scale study. In Advances in Neural Information Processing Systems, volume 31, 2018

2018
[17]

Owen, A. B. Monte Carlo theory, methods and examples. Self-published, 2013

2013
[18]

Torch.manual\_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision

Picard, D. Torch.manual\_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision. arXiv preprint arXiv:2109.08203, 2021

2021 arXiv
[19]

Boosted decision tree regression adjustment for variance reduction in online controlled experiments

Poyarkov, A., Drutsa, A., Khalyavin, A., Gusev, G., and Serdyukov, P. Boosted decision tree regression adjustment for variance reduction in online controlled experiments. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp....

2016
[20]

Rosenbaum, P. R. The consequences of adjustment for a concomitant variable that has been affected by the treatment. Journal of the Royal Statistical Society: Series A, 147 0 (5): 0 656--666, 1984

1984
[21]

R., Eisenstein, J., Das, D., and Pavlick, E

Sellam, T., Yadlowsky, S., Tenney, I., Wei, J., Saphra, N., D'Amour, A., Linzen, T., Bastings, J., Turc, I. R., Eisenstein, J., Das, D., and Pavlick, E. MultiBERTs : BERT reproductions for robustness analysis. In International Conference on Learning Representations, 2022

2022
[22]

Training data-efficient image transformers & distillation through attention

Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J \'e gou, H. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp.\ 10347--10357. PMLR, 2021

2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Accounting for variance in machine learning benchmarks

Bouthillier, X., Delaunay, P., Bronzi, M., Trofimov, A., et al. Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems, volume 3, pp.\ 747--769, 2021

work page 2021

[2] [2]

Double/debiased machine learning for treatment and structural parameters

Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21 0 (1): 0 C1--C68, 2018

2018

[3] [3]

A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., and Vinyals, O

Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., and Vinyals, O. The Benchmark Lottery . arXiv preprint arXiv:2107.07002, 2021

arXiv 2021

[4] [4]

Improving the sensitivity of online controlled experiments by utilizing pre-experiment data

Deng, A., Xu, Y., Kohavi, R., and Walker, T. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, pp.\ 123--132, 2013

work page 2013

[5] [5]

Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping

Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., and Smith, N. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020

arXiv 2002

[6] [6]

T., and Hutter, F

Domhan, T., Springenberg, J. T., and Hutter, F. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, pp.\ 3460--3468, 2015

work page 2015

[7] [7]

Freedman, D. A. On regression adjustments to experimental data. Advances in Applied Mathematics, 40 0 (2): 0 180--193, 2008

work page 2008

[8] [8]

B., Stern, H

Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. Bayesian Data Analysis. Chapman and Hall/CRC, 3rd edition, 2013

work page 2013