Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Small Data, Big Decisions: Model Selection in the Small-Data Regime

T0 review · 3 major / 6 minor · reviewed 2026-08-27 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Overparameterized networks keep their generalization ranking at every training-set size, so model selection can be done on small data.

desk verdict A careful empirical study of how model rankings behave as training data shrinks; the ranking-preservation claim is plausible but rests on absence of crossings, and the MDL part is the weakest section. read the letter →

arxiv 2009.12583 v1 pith:HPX4RAAQ submitted 2020-09-26 cs.LG stat.ML

Jorg Bornschein , Francesco Visin , Simon Osindero This is my paper · ORCID
classification cs.LGstat.ML
keywords modelselectionsmall-dataregimerankinghypothesistemperaturecalibrationminimumdescriptionlengthoverparameterizedneuralnetworksgeneralizationarchitecturesearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that sufficiently overparameterized neural networks maintain their relative ranking in generalization performance across training-set sizes spanning several orders of magnitude, from the full dataset down to only a few examples per class. If true, model selection and architecture search can be performed on small data subsets, with decisions that remain valid for much larger datasets, while also cutting computational cost. The paper supports the claim with extensive experiments on MNIST, EMNIST, CIFAR10, and ImageNet across many architectures and optimizers, and uses temperature calibration to make cross-entropy a stable quantity in the overfitting regime.

What carries the argument

The key object is the performance curve of calibrated generalization cross-entropy as a function of training-set size for each architecture, together with the signal-to-noise ratio of the performance difference between model pairs. Temperature calibration, a single softmax temperature fitted on a held-out calibration set, makes cross-entropy well-behaved even when models are far larger than the training set, allowing fair comparisons across dataset sizes. The Minimum Description Length estimates are built by summing the area under these curves using a trapezoid integral under a prequential coding scheme.

What would settle it

Train two sufficiently overparameterized architectures on multiple dataset sizes from the same distribution, with well-tuned optimizers and all models beyond their interpolation threshold, and check whether their expected generalization ranking ever flips between two sizes; even one pair of curves crossing outside the uncertainty bands would refute the hypothesis.

Watch

Extended reading notes

Core claim

The paper's core discovery is an empirical ranking hypothesis: when two sufficiently large architectures are trained with well-tuned optimizers on datasets of size N, and one performs better in expectation, it also performs better for all other dataset sizes drawn from the same distribution, as long as the models remain well beyond the interpolation threshold. The authors report that across a wide range of architectures, model sizes, and optimizers, they could not find a pair of performance curves that crossed outside their uncertainty bands. Consequently, training on small or medium subsets not only preserves model rankings but often yields a more robust model-selection signal than training on the full dataset, while saving computation. The paper also shows that temperature calibration prevents the generalization cross-entropy from overfitting, enabling reliable Minimum Description Length estimates for deep learning models.

Load-bearing premise

The central claim rests on the assumption that every model tested is 'big enough'—meaning doubling its size no longer improves generalization—at every dataset size, and that this condition is correctly identified by the authors' heuristic.

Editorial extensions

If this is right

  • Model selection and neural architecture search can be run on small data subsets, and the chosen architecture will remain the best choice when trained on the full dataset.
  • Training on smaller subsets requires fewer gradient steps—the authors report 4-7x fewer steps for ImageNet ResNets with an automatic annealing schedule—so architecture exploration becomes cheaper.
  • Small training sets can give a higher signal-to-noise ratio for model selection than large ones, reducing the number of seeds needed to pick a winner.
  • Reliable calibrated cross-entropy enables Minimum Description Length estimates for fixed deep learning models, making Occam's-razor-based model comparison feasible.
  • The ranking stability implies that 'big enough' models do not need to be downsized for small datasets; their performance ordering is insensitive to dataset size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ranking hypothesis generalizes beyond image classification, small-data model selection could be applied to language, speech, or medical imaging, where full datasets are expensive to label or train on.
  • The hypothesis suggests a practical test for whether a model is 'big enough': if doubling its size does not improve generalization, its ranking should be stable across dataset sizes—this could become a standard check before running large-scale search.
  • The MDL estimates, because they are computed per architecture and dataset, could be aggregated into a dataset-difficulty metric that is independent of any single training run.
  • One could attempt to prove the ranking hypothesis in simplified settings, such as linear models or infinite-width limits, where generalization curves may be shown to be monotonic in dataset size under fixed overparameterization.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Request a human review

A listed scientist reviews the paper for a fee and the review publishes here regardless of verdict. See the reviewers or get listed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper empirically studies how the generalization performance of overparameterized neural networks varies as a function of training-set size, across MNIST, EMNIST, CIFAR10, and ImageNet. It introduces temperature calibration to obtain stable cross-entropy estimates, proposes a 'ranking hypothesis' that sufficiently large architectures preserve their relative performance ordering across dataset sizes, and uses calibrated cross-entropies to estimate Minimum Description Lengths via a prequential coding scheme. The main practical claim is that model selection can be performed on small data subsets and the decision transfers to larger datasets.

Significance. If the ranking hypothesis holds, the results have substantial practical value for architecture search and model evaluation, offering computational savings without sacrificing model-selection quality. The paper is also valuable as a large-scale empirical mapping of learning curves, and the calibration and SNR analyses are thoughtful, with uncertainty estimates and multiple seeds. The authors are transparent that the ranking claim is a hypothesis supported by absence of observed crossings rather than by a confirmed law.

major comments (3)
  1. [Section 4.3, Figures 5-7] The ranking hypothesis stated in Section 1.2 and tested in Section 4.3 is the load-bearing premise for the paper's practical claims, but the current evidence does not establish that expected performance curves do not cross. The paper reports that no pair of architectures crossed 'outside of their uncertainty bands', which is a statement about the observed curves, not about the underlying expectations. For ImageNet (Figure 6), only 5 seeds are used and the closest pairs (e.g., Inception vs ResNet-101, ResNet-101 vs ResNet-152) have overlapping uncertainty bands at several dataset sizes; the corresponding SNR curves in the bottom panel are near zero or below one at the smallest and largest sizes, indicating that the observed ranking differences are within estimation noise. Consequently, a true crossing of the expected curves could easily be hidden. I recommend either providing a statistical test or at least a power analysis that quantifies the probability of detecting a crossing of a given magnitude, and softening the conclusion accordingly.
  2. [Section 5] The derivation of the MDL estimates does not yet demonstrate that the reported numbers are valid prequential description lengths. A prequential code requires that the dataset used for training at size N is a superset of the data used at size M, and that the evaluation points are exactly the newly added examples; otherwise the area under the learning curve is not a code length. The text states 'We implement the suggested splitting procedure', but the manuscript does not describe how the nested subsets were generated, how the models were trained incrementally over these subsets, or how the trapezoid integration was performed (e.g., whether the integration variable is dataset size in linear or log scale). Without this information, the MDL numbers in Tables 1-3 appear to be computed from the same random-subset performance curves used elsewhere in the paper, and therefore do not correspond to a valid code. In addition, the inequality DL(D|H) ≤ sum_i log p(y_i|previous) is treated as an equality without addressing the slack; the approximation error should be discussed.
  3. [Table 2] Table 2, presented as 'MDL log-evidence estimates for selected model pairs on CIFAR10', contains exactly the same entries as Table 1, which reports MNIST results with models Logistic Regression, MLP, MLP+dropout, and ConvNet. This is either a copy-paste error or the CIFAR10 MDL estimates are missing. The table cannot be taken as evidence for any CIFAR10 model comparison, and the paper needs to provide the actual CIFAR10 numbers or correct the labeling.
minor comments (6)
  1. [Section 4.4] The definition SNR = sqrt(Delta^2 / Var[Delta]) equals |Delta|/std(Delta), which is effectively a t-statistic; the text should state this interpretation explicitly.
  2. [Figure 1] The caption does not explain the dashed versus solid curves; please state that dashed is uncalibrated cross-entropy and solid is calibrated cross-entropy.
  3. [Section 3.2] There is a typo: 'evaluate a range on non-optimal architectures' should be 'evaluate a range of non-optimal architectures'.
  4. [Section 4.3] Please define precisely what 'outside of their uncertainty bands' means; e.g., at any dataset size or at all dataset sizes, and how overlapping bands were treated.
  5. [Section 5] The uncertainty estimates in Tables 1-3 are based on only 3 seeds, whereas the performance curves in Figures 5 and 7 use 30 seeds and Figure 6 uses 5; please justify the smaller number of seeds for MDL and report whether the intervals are standard deviations or confidence intervals.
  6. [General] The manuscript does not mention whether code or data will be made available; given the empirical nature of the work, this would aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MDL estimates are valid prequential code-length computations, and the ranking hypothesis is an explicitly labeled empirical conjecture.

full rationale

The paper's central ranking hypothesis (Section 1.2) is presented as an empirical conjecture, not derived from a fitted parameter or from the MDL estimates. The evidence for it is a systematic absence of observed curve crossings across many architectures and datasets, which is a direct empirical observation rather than a reduction to its own inputs. The definition of 'big enough' models in Section 4.3 is an explicit scope condition, not a result smuggled in from the conclusion. The MDL estimates in Section 5 are computed via the standard prequential identity: the description length of a dataset equals the cumulative log-loss of sequentially trained and calibrated models, which the paper operationalizes as the area under the calibrated cross-entropy curves. This is a legitimate construction, not a fitted input being renamed as a prediction; the MDL log-evidence is by design the integral of the same curves, but it is not used to derive the ranking hypothesis, nor does the paper claim the ranking hypothesis follows from MDL. No load-bearing self-citation, imported uniqueness theorem, or ansatz-by-citation appears. The limitation that the ranking hypothesis is not theoretically proven is openly acknowledged ('Unfortunately this is only an hypothesis'). Therefore the derivation chain is self-contained and non-circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on two fitted parameters (calibration temperature and learning rate), the prequential coding theory from prior work, and several domain assumptions about the representativeness of the calibration split and the interpolation-threshold regime. No new theoretical entities are introduced.

free parameters (2)
  • Softmax temperature T = Optimized per model and dataset size on the 10% calibration split
    Used to calibrate cross-entropy; the temperature is a scalar fitted by gradient descent on the calibration loss. All calibrated curves and MDL estimates depend on this fitted value.
  • Learning rate = Selected per model, dataset size, and optimizer from fixed candidate sets
    Chosen by best calibration loss, which affects the learning curves and therefore the ranking and MDL estimates. Different effective training dynamics across sizes are possible.
assumptions (4)
  • standard math The prequential code construction of Blier & Ollivier (2018) yields a valid code over datasets.
    Used in Section 5 to relate expected description length to the area under the learning curve. This relies on the Kraft-McMillan theorem and the ability of both sender and receiver to reconstruct the same model from previously transmitted data.
  • domain assumption A 90/10 random split into training and calibration sets provides a calibration set representative of the underlying distribution at each dataset size.
    Section 3.2 states this split is used throughout. The MDL code construction assumes the split is derived from previously transmitted data without leakage, which is only approximately true in the experiments.
  • domain assumption The tested models are 'well beyond the interpolation threshold' across all dataset sizes.
    Section 4.3 states models are considered big enough if doubling their size does not improve generalization. The ranking hypothesis is only claimed in this regime, and the condition is validated only by this heuristic rather than a rigorous test.
  • domain assumption The trapezoid integral approximation of the area under the learning curve is accurate enough for MDL estimates.
    Section 5 uses a simple trapezoid rule over a sparse set of dataset sizes. No error analysis for the discretization is provided, and the uncertainty estimates do not account for integration error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Small Data, Big Decisions: Model Selection in the Small-Data Regime." pith.science (2026). https://pith.science/paper/HPX4RAAQ

@misc{pith2026200912583,
  author       = {Pith},
  title        = {Pith review of: Small Data, Big Decisions: Model Selection in the Small-Data Regime},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HPX4RAAQ}},
  note         = {Machine review of arXiv:2009.12583}
}
read the original abstract

Highly overparametrized neural networks can display curiously strong generalization performance - a phenomenon that has recently garnered a wealth of theoretical and empirical research in order to better understand it. In contrast to most previous work, which typically considers the performance as a function of the model size, in this paper we empirically study the generalization performance as the size of the training set varies over multiple orders of magnitude. These systematic experiments lead to some interesting and potentially very useful observations; perhaps most notably that training on smaller subsets of the data can lead to more reliable model selection decisions whilst simultaneously enjoying smaller computational costs. Our experiments furthermore allow us to estimate Minimum Description Lengths for common datasets given modern neural network architectures, thereby paving the way for principled model selection taking into account Occams-razor.

Figures

Figures reproduced from arXiv: 2009.12583 by the authors.

Figure 1
Figure 1. Learning curves for ResNet-101 on subsets of the ImageNet dataset using a RMSProp with a cosine learning-rate schedule. Left: the uncalibrated generalization cross-entropy shows a strong overfitting signature (dashed line), the calibrated cross-entropy does not. 1 2 3 4 xent 0.2 0.4 0.6 0.8 error EMNIST 1 2 xent 0.2 0.4 0.6 0.8 error CIFAR10 2 4 6 xent 0.2 0.4 0.6 0.8 1.0 error ImageNet Generalization error vs. cali… view at source ↗
Figure 2
Figure 2. Post-convergence generalization error vs. generalization cross-entropy without early-stopping for a range of model archi￾tectures and training set sizes. The colours represent the different model architectures. We observe that the calibrated cross entropy is strongly correlated with the generalization error rate. tectures seem to maintain their relative ranking in terms of generalization performance, when trained on… view at source ↗
Figure 3
Figure 3. Cross-entropy performance profiles for the ResNet-101 architecture on ImageNet when trained with RMSProp. We use 90% of the available data for training, 10% for calibration and report the generalization performance on the unseen validation set. Note that even when training with as little as ≈ 2.3 images per class, there is no harm in using a ResNet model with 4 × more channels (16× more parameters) than the standard… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance profiles for a fully connected MLP with 3 hidden layers on CIFAR10 as a function of the hidden layer size. Red points mark the smallest models that approach a close to zero training error-rate. number of strides, to keep the spatial resolution higher, and a…
Figure 5
Figure 5. Figure 5: Top row: Generalization performance for various model architectures trained with Adam on MNIST and EMNIST as a function of training set size. Uncertainty bands represent standard-deviation after training with 30 different seeds. Bottom row: SNR for the performance diff…
Figure 6
Figure 6. Figure 6: Top: Generalization performance for models trained on ImageNet; Uncertainty bands from training 5 different seeds. Bottom: SNR for the performance difference between models; estimated using 1000 bootstrap samples. training error tapers off into a slow decrease towards …
Figure 7
Figure 7. Figure 7: Left: Selected models trained on CIFAR10; uncertainty bands from 30 different seeds. The ConvNet has 4 hidden layers with 3x3 kernels and 256 channels; every second layer uses stride 2 width a single 2048 unit wide fully connected hidden layer on top. Right: Five model…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Semivalue-based data valuation is arbitrary and gameable

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Semivalue-based data valuations are shown to be highly sensitive to plausible utility-function choices and are gameable under the paper's weak definition of gameability.

  2. Small Data Explainer -- The impact of small data methods in everyday life

    cs.CY 2025-07 conditional novelty 3.0 of 10

    A review and explainer that frames small data methods through the recurring challenges of similarity, transfer, and uncertainty and maps them to application areas and technical approaches.

Reference graph

Works this paper leans on

31 extracted references · 26 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Advani, M. S. and Saxe, A. M. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXiv:1710.03667, 2017

    work page Pith review arXiv 2017

  3. [3]

    Learning and generalization in overparameterized neural networks, going beyond two layers

    Allen-Zhu, Z., Li, Y., and Liang, Y. Learning and generalization in overparameterized neural networks, going beyond two layers. In Advances in neural information processing systems, pp.\ 6155--6166, 2019

    work page 2019

  4. [4]

    Reconciling modern machine-learning practice and the classical bias--variance trade-off

    Belkin, M., Hsu, D., Ma, S., and Mandal, S. Reconciling modern machine-learning practice and the classical bias--variance trade-off. Proceedings of the National Academy of Sciences, 116 0 (32): 0 15849--15854, 2019

    work page 2019

  5. [5]

    and Ollivier, Y

    Blier, L. and Ollivier, Y. The description length of deep learning models. In Advances in Neural Information Processing Systems, pp.\ 2216--2226, 2018

    work page 2018

  6. [6]

    Emnist: Extending mnist to handwritten letters

    Cohen, G., Afshar, S., Tapson, J., and Van Schaik, A. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pp.\ 2921--2926. IEEE, 2017

    work page 2017

  7. [7]

    Dawid, A. P. Present position and potential developments: Some personal views statistical theory the prequential approach. Journal of the Royal Statistical Society: Series A (General), 147 0 (2): 0 278--290, 1984

    work page 1984

  8. [8]

    A unified bias-variance decomposition

    Domingos, P. A unified bias-variance decomposition. In Proceedings of 17th International Conference on Machine Learning, pp.\ 231--238, 2000

    work page 2000

Show all 31 references
  1. [9]

    Neural networks and the bias/variance dilemma

    Geman, S., Bienenstock, E., and Doursat, R. Neural networks and the bias/variance dilemma. Neural computation, 4 0 (1): 0 1--58, 1992

    1992

  2. [10]

    Gr\" u nwald, P. D. The Minimum Description Length Principle. The MIT Press, 2007. ISBN 0262072815

    2007

  3. [11]

    Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp.\ 1321--1330. JMLR. org, 2017

    2017

  4. [12]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  5. [13]

    Deep learning scaling is predictable, empirically

    Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M., Ali, M., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017

    2017 arXiv

  6. [14]

    Neural tangent kernel: Convergence and generalization in neural networks

    Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31, pp.\ 8571-...

    2018

  7. [15]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  8. [16]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

    2009

  9. [17]

    The mnist database of handwritten digits

    LeCun, Y. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998

    1998

  10. [18]

    Mac Kay, D. J. Information theory, inference and learning algorithms. Cambridge university press, 2003

    2003

  11. [19]

    D., and Blei, D

    Mandt, S., Hoffman, M. D., and Blei, D. M. Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research, 18 0 (1): 0 4873--4907, 2017

    2017

  12. [20]

    Deep double descent: Where bigger models and more data hurt

    Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. Deep double descent: Where bigger models and more data hurt. arXiv preprint arXiv:1912.02292, 2019

    1912 arXiv

  13. [21]

    Modeling by shortest data description

    Rissanen, J. Modeling by shortest data description. Automatica, 14 0 (5): 0 465--471, 1978

    1978

  14. [22]

    Stochastic complexity in statistical inquiry

    Rissanen, J. Stochastic complexity in statistical inquiry. World Scientific, 1989

    1989

  15. [23]

    C., and Fei-Fei, L

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision (IJCV), 115 0 (3): 0 211--252, 201...

    2015 doi

  16. [24]

    and Zisserman, A

    Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  17. [25]

    A jamming transition from under-to over-parametrization affects loss landscape and generalization

    Spigler, S., Geiger, M., d'Ascoli, S., Sagun, L., Biroli, G., and Wyart, M. A jamming transition from under-to over-parametrization affects loss landscape and generalization. arXiv preprint arXiv:1810.09665, 2018

    2018 arXiv

  18. [26]

    Rethinking the inception architecture for computer vision

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 2818--2826, 2016

    2016

  19. [27]

    and Hinton, G

    Tieleman, T. and Hinton, G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4 0 (2): 0 26--31, 2012

    2012

  20. [28]

    Nas-bench-101: Towards reproducible neural architecture search

    Ying, C., Klein, A., Real, E., Christiansen, E., Murphy, K., and Hutter, F. Nas-bench-101: Towards reproducible neural architecture search. arXiv preprint arXiv:1902.09635, 2019

    1902 arXiv

  21. [29]

    and Komodakis, N

    Zagoruyko, S. and Komodakis, N. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016

    2016 arXiv

  22. [30]

    Understanding deep learning requires rethinking generalization

    Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016

    2016 arXiv

  23. [31]

    Towards robust image classification using sequential attention models

    Zoran, D., Chrzanowski, M., Huang, P.-S., Gowal, S., Mott, A., and Kohl, P. Towards robust image classification using sequential attention models. arXiv preprint arXiv:1912.02184, 2019

    1912 arXiv

Pith tools

Reviewed August 27, 2026 · model on record in the stance chip above.