Pith. sign in

REVIEW 2 major objections 5 minor 20 references

Classical ML learning curves follow a power law, and one exponent per model family predicts them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:24 UTC pith:HH2OXD5X

load-bearing objection Useful benchmark resource and an honest empirical study, but the headline compression claim needs a LODO permutation null before it carries weight. the 2 major comments →

arxiv 2607.21866 v2 pith:HH2OXD5X submitted 2026-07-23 cs.LG stat.ML

Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study

Kaihua Ding This is my paper
classification cs.LG stat.ML
keywords learning curvesscaling lawstabular datapower lawclassical machine learningreproducibilitygradient boostingdata requirements
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the error of classical machine learning models on tabular data shrinks with training-set size according to a simple additive power law, error(N) = aN^{-b} + c, and that this law is more than a curve-fitting convenience. By running one fixed protocol across 127 independent implementations on 18 datasets and six model families, it finds that a single exponent per family predicts almost as well as per-dataset exponents for five of six families, enough to call family-level exponents approximately predictable though not universal. The paper also finds that even with the random seed fixed, compliant re-implementations shift the fitted exponent by about 14%, and that tree ensembles dominate at full data. If true, practitioners could estimate the data needed to reach a target error and know how much to trust a reported scaling exponent.

Core claim

For tabular data, the additive power law error(N)=aN^{-b}+c describes 77.7% of fitted learning curves at R^2>0.8, and the exponent b clusters by model family rather than by dataset: a shared family exponent leaves an R^2 gap below 0.011 on pooled data for boosting, random forests, SVM, linear models, and (fragilely) ridge, and transfers to held-out datasets with small error for the four robust families. Lasso fails as a negative control, and AIC shows real per-dataset heterogeneity remains, so the paper's claim is predictive compressibility, not dataset-independent universality. A companion result is that the fitted exponent varies across replicators with coefficient of variation 0.144 even

What carries the argument

The load-bearing object is the three-parameter power law error(N)=aN^{-b}+c, where a is a scale, b the scaling exponent, and c the irreducible error floor. The paper fits this law to per-replicator learning curves, then tests a shared-exponent variant that fixes b across datasets while leaving a and c free; leave-one-dataset-out transfer and curve-collapse analysis separate genuine compressibility from mere curve-fitting flexibility.

Load-bearing premise

The shared-exponent claim rests on treating the leave-one-dataset-out transfer test as discriminating: if the two free per-dataset parameters (scale a and floor c) can absorb a wrong exponent by adjusting their values, the test would pass even when the family exponent is not genuine.

What would settle it

Simulate datasets where each dataset truly has its own distinct exponent while keeping per-dataset a and c free, then run the paper's shared-exponent leave-one-dataset-out test; if the held-out R^2 gap stays as small as 0.007–0.035, the transfer test is too flexible to establish a shared exponent and the compression claim falls to a curve-fitting artifact.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A practitioner can invert the fitted law to estimate N*, the training rows needed to reach a target error, with concrete numbers such as roughly 200–500 rows for boosting on several easy classification datasets.
  • Tree ensembles win the full-data ranking on most datasets (boosting 50%, random forest 33%), and they occupy the favorable low-floor region of the (b, c) plane.
  • A reported scaling exponent from a single group should carry an uncertainty interval of roughly ±14% to account for implementation drift, not just seed variance.
  • The power-law description is not reliable for fixed-hyperparameter Lasso or Ridge; their failures are mechanistic, not noise.
  • A family-level exponent can serve as a compressed summary of a model family's data-efficiency across tabular datasets, at least for boosting, random forests, SVMs, and linear models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if family-level exponents transfer to datasets outside the 18 studied, they could be used as priors for learning-curve extrapolation on new tabular tasks, reducing the data needed to estimate b from scratch.
  • Beyond the paper: the 14% replicator CV suggests that scaling-law comparisons between models should require exponent differences larger than about 0.15 to be credible; this threshold could be tested by deliberately perturbing preprocessing choices.
  • Beyond the paper: the mechanistic failure of Lasso and Ridge hints that regularized linear models with fixed regularization may have no well-defined scaling exponent in the small-N regime; varying the regularization strength with N might bring power-law behavior back.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper reports a distributed benchmark study of classical ML scaling laws on tabular data. 127 graduate-student replicators each ran a fixed protocol on 3 of 18 datasets across 6 model families, yielding 11,536 training runs and 1,648 fitted power-law curves of the form error(N)=aN^{-b}+c. The authors report three findings: (1) the additive power law fits well for most cells (R^2>0.8 on 77.7%), with tree ensembles dominating at full data; (2) a single family-level exponent compresses cross-dataset curves for 5 of 6 families ('approximate predictive compressibility'), with Lasso as a negative control and Ridge fragile under leave-one-dataset-out; (3) fixed-seed replicator implementation variance leads to CV(b)=0.144. The paper also provides a data-requirement table and promises release of aggregated curves and code.

Significance. If the compression claim survives stricter testing, this is a useful empirical contribution: it would quantify the extent to which classical-ML learning curves share exponents, provide practical data-requirement estimates, and measure protocol-drift uncertainty. The paper is unusually transparent: it separates fit/compression/universality, reports AIC, LODO transfer, bound sensitivity, and a permutation null, and will release aggregated data and code. These strengths are real and should be credited. However, the central claim currently rests on an incompletely validated transfer test; the paper's own permutation null shows the in-sample R^2 gap is non-specific, and the LODO test lacks a null. The significance of the work depends on closing that gap.

major comments (2)
  1. [Section 4.3 / Appendix A] The held-out LODO transfer test is the load-bearing evidence for 'approximate predictive compressibility', because the in-sample R^2 gap is explicitly non-specific under the paper's own permutation null (one-sided p=0.55–0.97 for the four robust families). But the LODO test compares the frozen-b fit to the unconstrained fit without any null distribution under a misspecified b. Since the held-out fit still estimates two free per-dataset parameters (a_d, c_d), those parameters can absorb a wrong exponent over the observed N range (~10^1–10^4). The manuscript should run a LODO permutation null (e.g., freeze b(-d), then assign each held-out dataset a b drawn from another dataset's fit, or fit a grid of deliberately wrong b values) and report the distribution of held-out R^2 gaps. If the observed gaps (0.007–0.035) are not smaller than the null, the transfer evidence for predictive compressib
  2. [Abstract and Section 4.3 vs. Appendix A] The headline '5 of 6 families' is based on the in-sample R^2 gap (<0.011) and includes Ridge, yet Appendix A reports Ridge's LODO median gap of 0.088 and LODO CV of 0.96, and the paper itself concludes 'strong held-out support' only for Boosting, RandomForest, and LinearModel, with SVM moderate and Ridge fragile. The abstract and Section 4.3 should qualify the count to avoid overstating the evidence, e.g., 'in-sample compression for 5/6 families; held-out compression for 3/6, with SVM moderate and Ridge fragile.'
minor comments (5)
  1. [Appendix A] The abbreviation CV is used for both coefficient of variation and cross-validation (e.g., 'CV 0.04–0.09' in LODO results). Rename one to avoid ambiguity.
  2. [Section 3.5] The use of soft-L1 loss for Lasso but not for other families is not justified. If different loss functions are used across families, R^2 comparisons may not be apples-to-apples; please clarify.
  3. [Section 6(vi) and Appendix A] The language 'supported in-sample and held-out' for shared exponents is stronger than the evidence: the in-sample permutation p-values are 0.55–0.97, so only the LODO (if validated as recommended above) supports the claim. Please adjust the wording.
  4. [Figure 6] The label 'Universal b' is confusing; clarify that this is the shared-b estimate from the joint fit, not a universal exponent.
  5. [Checklist 8] The paper states compute resources were not aggregated because runs were on personal laptops. This is acceptable but should be stated in the main text (e.g., Section 3.2) rather than only in the checklist.

Circularity Check

0 steps flagged

No significant circularity: the shared-exponent claim rests on a genuine held-out LODO transfer, with the in-sample R^2 gap explicitly de-emphasized.

full rationale

The derivation chain is self-contained. The power-law fits (Eq. 1) are ordinary least-squares descriptions of observed (N,error) points; no fitted parameter is defined in terms of a claim it is supposed to prove. The shared-exponent analysis (Sec. 4.3, Table 4) is a nested model comparison: one b_model is fit jointly with per-dataset (a_d,c_d) and compared on the same pooled point cloud with the unconstrained per-dataset-b_d fit. A small R^2 gap is an empirical property, not a construction: the shared exponent minimizes that gap but does not force it to be small. Critically, the authors do not treat the in-sample gap as the proof: they state 'we therefore use the in-sample gap only as a compression metric, not as evidence for true dataset-independent exponents' and 'the held-out content lives in the LODO transfer and the tight shared-b CIs of Table 4, not the in-sample gap.' The LODO test is genuinely out-of-sample for b: b is refit on D-1 datasets, frozen, and only (a_d,c_d) are fit on the held-out dataset, then compared with the unconstrained 3-parameter fit. The concern that (a_d,c_d) offsets can absorb a misspecified b is a statistical specificity limitation, not a definitional equivalence; no equation reduces the LODO gap to the fitted input. The paper itself discloses the main weakness: Appendix A's permutation null gives p=0.55-0.97 and Sec. 6(vi) calls the shared-exponent stress tests 'descriptive.' That weakens the compression claim but does not make it circular. Self-citations (Ding 2025, 2026; Ding et al. 2016, 2024) are related-work or caveat citations, not load-bearing; the aN^{-b}+c ansatz is attributed to external work (Rosenfeld et al. 2020). Lasso's failure and Ridge's fragility are empirical negative results, not circular validations. Overall, the central claim is not equivalent to its inputs by construction.

Axiom & Free-Parameter Ledger

9 free parameters · 6 axioms · 0 invented entities

The paper is empirical: the central quantities (a, b, c, shared b_model) are all fitted to the data rather than derived. No new theoretical entities are introduced. The key assumptions are the power-law ansatz, representativeness of the 18 datasets, and the independence/compliance of the 127 replicators. The LODO transfer test's validity rests on the assumption that free per-dataset scale/floor parameters cannot mask a wrong exponent.

free parameters (9)
  • per-cell scale a_d = fitted per (dataset, model) cell
    Scale parameter in error(N)=a N^{-b}+c; fitted by NLLS; part of the power-law model.
  • per-cell exponent b_d = values in Table 3 and 4
    Central exponent; fitted rather than derived.
  • per-cell floor c_d = bounds [0,1] (classif) / [0,5] (reg)
    Irreducible-error floor; fitted.
  • shared family exponent b_model = Boosting 0.555, RandomForest 0.290, SVM 0.709, LinearModel 0.703, Ridge 0.593
    The shared-exponent compression parameter, fitted jointly across datasets.
  • fit-quality thresholds = R^2 > 0.7 and > 0.8
    Chosen by hand to filter fits; affects counts like 77.7%.
  • target error 0.15 = 0.15
    Chosen by hand for N* table.
  • seven training fractions = {0.01,0.05,0.10,0.25,0.50,0.75,1.00}
    Chosen by hand; the x-grid for learning curves.
  • bounds b∈[0,3], c∈[0,1]/[0,5]
    Chosen by hand; bound sensitivity tested.
  • SVM cap 10,000 rows = 10,000
    Chosen for compute; suppresses large-N SVM behavior.
axioms (6)
  • domain assumption Additive power-law form error(N)=aN^{-b}+c describes test error as a function of training set size
    Adopted from Rosenfeld et al. (2020) without derivation; the paper tests fit quality but the form is assumed before fitting (Section 3.3).
  • domain assumption The 18 UCI-style datasets are representative of tabular data for the purpose of shared-exponent generalization
    Used to draw cross-dataset conclusions (Section 3.1).
  • domain assumption Replicators' implementations are independent and compliant with the fixed protocol; residual variance is due to unconstrained choices, not shared conventions
    Needed for the CV(b)=0.144 reproducibility floor (Section 4.4); the paper itself flags in Section 6(vii) that agreement among estimators is not correctness.
  • domain assumption Leave-one-dataset-out transfer with free per-dataset (a_d,c_d) is a valid test of shared-exponent transferability
    The held-out fit still allows two free parameters per dataset that may absorb a misspecified exponent (Appendix A).
  • standard math Nonlinear least squares and AIC on the pooled point cloud are appropriate for comparing shared vs per-dataset exponent models
    Used in Section 3.5; AIC assumes i.i.d. Gaussian errors, which is not justified for pooled learning-curve points.
  • domain assumption The permutation null with Gaussian residuals at per-cell empirical std models the null distribution of the R^2 gap
    Used in Appendix A to test specificity of the in-sample gap.

pith-pipeline@v1.3.0-alltime-deepseek · 15725 in / 14654 out tokens · 135024 ms · 2026-08-01T06:24:47.421712+00:00 · methodology

0 comments
read the original abstract

Prior classical-ML learning-curve work fits power laws to tree, linear, and kernel models on tabular data, but at small scale: typically one curve, one team, a handful of cells. We present a distributed classroom-scale replication: 127 graduate students each ran a fixed protocol on 3 assigned datasets, drawn from 18 tabular classification and regression datasets and 6 model families (Boosting, Random Forest, SVM, Linear/Logistic, Ridge, Lasso), yielding 11,536 training runs and 1,648 fitted power-law curves of the form error(N) = a N^(-b) + c. Three findings. (1) Power laws fit: R^2 > 0.8 on 77.7% of cells, with tree ensembles dominating at full data (Boosting 50% of datasets, RandomForest 33%; linear models underperform on classification). (2) Approximate shared exponents within a model family: for 5 of 6 families, a single family-level exponent predicts each family's cross-dataset curves nearly as well as per-dataset exponents (R^2 gap < 0.011), though AIC favors the unconstrained fit and curve collapse is partial (32-58% of points within +/-0.5 dex). We frame this as approximate predictive compressibility, not dataset-independent universality; Lasso fails outright (negative control) and Ridge is fragile under leave-one-dataset-out. (3) Replicator-implementation variance: with random_state=42 fixed, independent re-implementations of the same protocol still differ by mean CV(b) = 0.144 on the fitted exponent -- not seed variance, but the spread induced by unconstrained parts of the protocol (preprocessing, encoding, missing-value handling). We release the aggregated curves, per-cell fits, and a practical data-requirement table for N* to reach target error 0.15.

Figures

Figures reproduced from arXiv: 2607.21866 by Kaihua Ding.

Figure 1
Figure 1. Figure 1: Boxplots of fitted scaling exponent b per model, split by task type. Tree ensembles cluster near b ≈ 0.3–0.5; linear models on classification show systematically larger but noisier exponents [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Coefficient of variation of test error across replicators at each train fraction, on representative [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pooled scaling curves on 4 representative classification (top row) and 4 representative [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Mean per-replicator R2 of power-law fits per (dataset, model) cell. Most cells exceed 0.9; diagnostic dark cells (Lasso, Ridge, and a notable Lasso/Ridge column on california_housing) flag cases where the power-law assumption is misleading. Companion to [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-(replicator, dataset, model) scatter of [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Forest plot of scaling exponents per model family. Square: the shared [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: log10 N⋆ to reach error 0.15 on each (classification dataset, model) cell. Empty cells indicate c ≥ 0.15 (unreachable). Companion to [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 4 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling Laws for Neural Language Models , author=. arXiv preprint arXiv:2001.08361 , year=

    Pith/arXiv arXiv 2001

  2. [2]

    arXiv preprint arXiv:2203.15556 , year=

    Training Compute-Optimal Large Language Models , author=. arXiv preprint arXiv:2203.15556 , year=

    Pith/arXiv arXiv

  3. [3]

    arXiv preprint arXiv:1712.00409 , year=

    Deep Learning Scaling is Predictable, Empirically , author=. arXiv preprint arXiv:1712.00409 , year=

    Pith/arXiv arXiv

  4. [4]

    International Conference on Learning Representations , year=

    A Constructive Prediction of the Generalization Error Across Scales , author=. International Conference on Learning Representations , year=

  5. [5]

    Advances in Neural Information Processing Systems Datasets and Benchmarks Track , year=

    Why do tree-based models still outperform deep learning on typical tabular data? , author=. Advances in Neural Information Processing Systems Datasets and Benchmarks Track , year=

  6. [6]

    Information Fusion , volume=

    Tabular Data: Deep Learning is Not All You Need , author=. Information Fusion , volume=

  7. [7]

    Journal of Machine Learning Research , volume=

    Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program) , author=. Journal of Machine Learning Research , volume=

    2019

  8. [8]

    International Conference on Machine Learning , year=

    Unreproducible Research is Reproducible , author=. International Conference on Machine Learning , year=

  9. [9]

    Chen, Tianqi and Guestrin, Carlos , journal=

  10. [10]

    Machine Learning , volume=

    Random Forests , author=. Machine Learning , volume=

  11. [11]

    Logistic Regression: A Learning-Curve Analysis , author=

    Tree Induction vs. Logistic Regression: A Learning-Curve Analysis , author=. Journal of Machine Learning Research , volume=

  12. [12]

    and Mesirov, Jill P

    Mukherjee, Sayan and Tamayo, Pablo and Rogers, Simon and Rifkin, Ryan and Engle, Anna and Campbell, Colin and Golub, Todd R. and Mesirov, Jill P. , journal=. Estimating Dataset Size Requirements for Classifying

  13. [13]

    International Joint Conference on Artificial Intelligence (IJCAI) , year=

    Speeding up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves , author=. International Joint Conference on Artificial Intelligence (IJCAI) , year=

  14. [14]

    BMC Medical Informatics and Decision Making , volume=

    Predicting Sample Size Required for Classification Performance , author=. BMC Medical Informatics and Decision Making , volume=

  15. [15]

    2019 , howpublished=

    Dua, Dheeru and Graff, Casey , title=. 2019 , howpublished=

    2019

  16. [16]

    Wickham, Hadley , title=

  17. [17]

    Ding, Kaihua , journal=. When

  18. [18]

    Variance-Bounded Evaluation of Entity-Centric

    Ding, Kaihua , journal=. Variance-Bounded Evaluation of Entity-Centric

  19. [19]

    arXiv preprint arXiv:2405.14743 , year=

    Iterative Causal Segmentation: Filling the Gap between Market Segmentation and Marketing Strategy , author=. arXiv preprint arXiv:2405.14743 , year=

    Pith/arXiv arXiv

  20. [20]

    54th AIAA Aerospace Sciences Meeting , year=

    Continuous Adjoint Based Error Estimation and r-Refinement for the Active-Flux Method , author=. 54th AIAA Aerospace Sciences Meeting , year=