Scaling Laws for Data-Efficient Visual Transfer Learning

Bo Yan; Chenxi Ma; Qingqu Wei; Weimin Tan; Wenxuan Yang

REVIEW 4 major objections 4 minor 1 cited by

Scaling Laws for Data-Efficient Visual Transfer Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Visual transfer error follows an additive power law with a crossover where distillation stops paying off.

desk verdict Real experimental effort, but the distillation-boundary proof contradicts the paper's own fitted exponents and the model-size law contradicts its own appendix. read the letter →

arxiv 2504.13219 v1 pith:UCDF7BQL submitted 2025-04-17 cs.LG cs.AI

Wenxuan Yang , Qingqu Wei , Chenxi Ma , Weimin Tan , Bo Yan This is my paper · ORCID

classification cs.LGcs.AI

keywords scalinglawstransferlearningknowledgedistillationdata-efficientvisiontransformerspowerlawdownstreamfine-tuningboundary

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual-transfer error, not just pretraining error, is claimed to follow an additive power law in three independent scales: pretraining data size $D_p$, model size $M$, and fine-tuning data size $D_f$. On top of that law, the paper argues that knowledge distillation from a small teacher to a larger student obeys its own power law with an extra teacher-size term, and that comparing the two laws forces a sharp crossover: distilled models win in data-scarce regimes, while ordinary pretrain-and-finetune wins once pretraining data exceeds a critical threshold $D^*_p$. If the claim holds, model-reuse strategy becomes a quantitative decision that can be computed from a handful of fitted exponents rather than a rule of thumb.

What carries the argument

The central object is the additive power-law error function (Eq. 1) and its distilled analogue (Eq. 4). The mechanism that produces the boundary is the difference $F(D_p)=E_1(D_p)-E_2(D_p)$: with the fitted parameter constraints $E<E'$, $\gamma>\gamma'$, $\beta<\beta'$, $\alpha-\alpha'\in(-1,0)$, $\lambda_m\approx\lambda'_m$, and $\lambda_f\approx\lambda'_f$, $F$ is positive for small $D_p$ and tends to a negative constant as $D_p\to\infty$, so continuity and the intermediate value theorem guarantee a root $D^*_p$ where the two strategies tie. That root is the claimed crossover point.

What would settle it

Fit Eq. (1) on a random subset of the data-size/model-size grid and evaluate on held-out sizes for any one of the four datasets; if held-out error swings upward as $M$ increases (as Appendix D shows for some settings) or the residuals do not decay as a power law in $D_p$, $D_f$, and $M$, then the additive law is not the right description and the derived threshold $D^*_p$ lacks predictive content.

Watch

Extended reading notes

Core claim

The paper's central claim is a two-part scaling description of transfer learning in vision. First, downstream error and cross-entropy loss follow $E(D_p,M,D_f) = E_\infty + D_p^{-\alpha}/\lambda_p + M^{-\beta}/\lambda_m + D_f^{-\gamma}/\lambda_f$ (and the analogous loss law), with fitted exponents showing pretraining data as the dominant factor. Second, distillation has a boundary: comparing this law with the distilled-model law $E = E_\infty + D_p^{-\alpha'}/\lambda'_p + M_s^{-\beta'}/\lambda'_s + D_f^{-\gamma'}/\lambda'_f + M_t^{-\eta'}/\delta'$ yields a critical pretraining-data threshold $D^*_p$ at which the error difference between the two strategies changes sign, so distillation is superior below the threshold and inferior above it. The paper reports the predicted sign flip, with error-difference curves crossing from positive to negative at a critical data size, across model sizes from 2.5M to 38M parameters.

Load-bearing premise

The load-bearing premise is that downstream error decreases monotonically as model size grows, so the term $M^{-\beta}/\lambda_m$ in the scaling law is a valid decreasing function; the paper's own Appendix D reports cases where error increases with model size, which would break both the fitted law and the crossover proof.

Editorial extensions

If this is right

In data-scarce downstream settings, a small teacher distilling into a larger model should beat the same larger model trained without distillation, so distillation is the recommended strategy below the threshold.
Beyond the critical pretraining-data threshold, the distilled model's inherited bias costs more than it saves, and standard pretrain-and-finetune should be preferred.
Because the distilled law contains explicit teacher-size and student-size terms, the threshold can be re-estimated for any small-to-large pair before running the full experiment.
The fitted exponents rank pretraining data as the strongest lever on downstream performance, suggesting that adding pretraining data is a more efficient use of resources than enlarging the model or adding fine-tuning examples in the tested range.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the additive law holds beyond the four benchmarks, the fitted equations could be read as a budget rule: spend on distillation only when the cost of more pretraining data is higher than the cost of a larger teacher, since the boundary shifts with both model sizes.
The non-monotonic model-size curves the paper itself reports in Appendix D suggest a natural extension: replacing the model-size term with a shape-aware term (depth versus width) would make both the fitted law and the crossover prediction safer outside the 2.5M to 38M parameter range.
The theory implies that 'data-efficient transfer' and 'knowledge distillation' are two ends of the same curve rather than separate techniques: inherited knowledge substitutes for data up to a saturation point, and that point is measurable in advance from a few cheap runs.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: the experimental study is real, but the central theoretical claim doesn't survive contact with the paper's own numbers. The observed crossover between distillation and direct fine-tuning might be genuine, but this paper doesn't establish it.

What's actually new and useful: the experimental sweep is substantial. Four downstream datasets, model sizes from 2.5M to 38M, pretraining data from 64K to 1.3M, fine-tuning data in the 1K-130K range, under a consistent small-to-large distillation protocol. Extending the additive power-law to vision with a model-size term is a reasonable move, and the observation that distillation's edge shrinks as pretraining data grows is practically interesting if it holds up.

The soft spots are load-bearing. Lemma 1's proof asserts F(D_p) is positive for small D_p and uses that plus lim F = Δ < 0 to get a guaranteed zero crossing. But the paper's own constraint (6) imposes α − α' in (−1, 0), i.e. α' > α, and its own fits give α' > α on both datasets. With α' > α, the −D_p^{−α'}/λ'_p term dominates as D_p → 0, so F(D_p) → −∞, not +∞. The claimed "distillation superiority in data-scarce conditions" is the opposite of what the fitted equations imply. The sign is backwards.

Second, the threshold is circular in the way that matters. Remark 1 says conditions (6) are "derived from extensive experimental evidence," and the proof of Theorem 1 then uses those same fitted values to derive the crossover. The theorem is algebra on the fit, not an independent prediction. The empirical crossover in Figures 5-6 may be real, but the theory section adds nothing beyond the fitted curves.

Third, the model-size term in Eq. (1) is contradicted by the paper's own Appendix D, which says error "do[es] not simply decrease; in some cases, they even increase with model size," matching Figures 4 and 13-15. Obs 3 claims the opposite. A monotone M^{−β} law with β > 0 is not supported by the data it was fitted to.

Minor: the Δ < 0 step waves a positive term out of the sign-determining position, and the CIFAR10 exponents (α = 10.129, β'' = 25.435) suggest fit instability.

Who this is for: a practitioner might get a rule of thumb from the empirical curves, and a referee could usefully sort out whether the crossover is robust. But the theory section as written shouldn't be trusted. Recommendation: send to peer review rather than desk-reject—the empirical question deserves referee time. Expect heavy revision: fix or drop the theorem, reconcile Appendix D with the main text, and recast the boundary as an empirical observation.

Referee Report

4 major / 4 minor

Summary. This paper studies scaling behavior of visual transfer learning in data-constrained settings. It proposes an additive power-law model E = E_inf + Dp^{-alpha}/lambda_p + M^{-beta}/lambda_m + Df^{-gamma}/lambda_f for downstream error (Eq. 1) and an analogous law for distilled models with teacher and student model sizes (Eq. 4). It fits these laws on ImageNet100, TinyImageNet, CIFAR100, and CIFAR10 over pretraining data sizes from roughly 64K to 1.3M, finetuning data from 2K to 130K, and model sizes from 2.5M to 38M. The central theoretical contribution is the 'distillation boundary theory' (Theorem 1), which asserts that a critical pretraining-data threshold D_p^* exists: distilled models are better for small Dp and worse for large Dp, under parametric constraints (6). The proof compares E1 and E2 through a differential function F(Dp) and applies the intermediate value theorem. The paper reports empirical error-difference plots (Figures 5 and 6) that it interprets as confirming the boundary.

Significance. If the claimed scaling laws and distillation boundary theorem were valid, the framework would offer practical guidance for choosing between standard fine-tuning and small-to-large distillation under data constraints, and it would extend the scaling-law literature to low-data transfer settings. The paper has genuine strengths: broad experiments across four downstream datasets, multiple model scales and data fractions, and fitted parameter tables. It is also transparent about limitations in Appendix F and about non-monotonic model-size behavior in Appendix D. However, the central theoretical result is not established: the sign analysis in Lemma 1 is internally inconsistent with the fitted exponents, the model-size term is not monotone as assumed, and the theorem's conditions are drawn from the same fits used to confirm the prediction. These are load-bearing concerns that affect the paper's main claim.

major comments (4)

[Section 4.1, Lemma 1 proof, Eq. (12), Tables 1 and 2] The proof asserts that F(Dp) is 'positive for small Dp' and then uses the intermediate value theorem to conclude a crossover. This assertion is not proven and is incompatible with the paper's own fitted exponents. Under Table 1 and Table 2, alpha' > alpha for both ImageNet100 (0.702 vs 0.620) and TinyImageNet (0.475 vs 0.412), and Eq. (6) imposes alpha - alpha' in (-1,0), i.e., alpha' > alpha. For fixed positive lambda_p and lambda_p', the term -Dp^{-alpha'}/lambda_p' dominates as Dp -> 0, so F(Dp) -> -infinity, not +infinity. Thus the claimed guarantee of 'distillation superiority in data-scarce conditions' does not follow from the fitted scaling laws; under those laws F could be negative for all Dp, or cross from negative to positive and back, and the simple one-crossing picture in Lemma 1 is not established.
[Section 4.1, Eqs. (9)-(11)] The proof of Delta < 0 is not sound. In Eq. (10), the first model-size term M^{-beta}/lambda_m - M^{-beta'}/lambda_m' is positive under the constraint beta < beta' and lambda_m ~ lambda_m', not negative. The argument that M^{-beta'} is 'vanishingly small' ignores the paper's own setup in which M is reported in millions with values 3, 10, 22, 38 (e.g., Figures 4 and 13-15); for M = 3 and beta' = 5.84, the term is not negligibly small relative to the other terms in Delta. Since the positive model-size contribution could offset the negative terms, the conclusion Delta < 0, which is needed to show that traditional training eventually wins for large Dp, is unsupported.
[Appendix D, Figures 4 and 13-15, Eq. (1), Obs. 3] The paper's own Appendix D states that 'as the downstream model size increases, the error rate and loss do not simply decrease; in some cases, they even increase with model size,' and Figures 13-15 show visibly non-monotonic curves. This directly contradicts the monotone decreasing M^{-beta} term in Eq. (1) and the claim in Obs. 3 and the Figure 4 caption that error and loss decrease as M increases. Because the model-size term is one of the three axes of the proposed scaling law and also enters Theorem 1 through E1 in Eq. (5), the empirical support for the model-size component of the central claim is missing.
[Theorem 1, Remark 1, Eq. (6), Tables 1, 2, and 4] The parametric constraints in Eq. (6) are described in Remark 1 as 'derived from extensive experimental evidence,' and the same fitted exponents and coefficients are then used to prove the existence of the crossover in Theorem 1. Since the inequalities E < E', gamma > gamma', beta < beta', alpha - alpha' in (-1,0), and the coefficient near-equalities are satisfied by construction of the fits (Tables 1, 2, and 4), the theorem states a consequence of the fitted parametric family rather than an independent, falsifiable prediction. Confirming the crossover on the same datasets used for fitting is therefore partly circular; a held-out dataset or a prediction made before fitting would be needed to validate the claimed boundary.

minor comments (4)

[Section 3.4, Eq. (4)] The sentence listing the fitted exponents says 'alpha, beta, gamma, alpha', beta', and gamma'' but Eq. (4) also contains the exponent eta' for teacher model size; please correct the list.
[Section 5.1, Pre-training dataset] ImageNet-100 is described as 'containing approximately 1.2 million training images distributed across 1000 categories'; since ImageNet100 is later defined as a 100-class subset of ImageNet1K, this appears to be a typo.
[Section 5.3, text after Table 1] The text says 'beta of 4.882 and -5.086 respectively' for ImageNet100 and TinyImageNet, but Table 1 lists 5.086 for TinyImageNet without a minus sign; the discrepancy should be fixed.
[Section 5.3 and Tables 1, 2, 4] Reporting goodness-of-fit measures (e.g., R^2, RMSE, or confidence intervals) for the fitted exponents would materially help readers assess whether the proposed power-law forms are distinguishable from simpler alternatives in the data range considered.

Circularity Check

2 steps flagged · score 7.0 of 10

Distillation boundary 'prediction' is encoded in fitted parametric constraints; Lemma 1 assumes the small-data superiority it claims to prove.

fitted input called prediction [Section 4, Theorem 1, Eq. (5)-(7) and Remark 1]
"Theorem 1 (Distillation Efficacy Boundary)... with the following parametric constraints: E < E′, γ > γ′, β < β′, α−α′∈(−1, 0), λ_m≈λ′_m, λ_f≈λ′_f (6) such that there exists a critical threshold D∗_p: ∃D∗_p, s.t. E1(D∗_p)≤E2(D∗_p). (7) Remark 1. The conditions (6), derived from extensive experimental evidence, reveal fundamental asymmetries in the learning dynamics between traditional and distillation-based approaches."

The theorem's existence claim is packaged inside the constraints ("such that there exists a critical threshold"), and Remark 1 says those constraints are derived from the same experiments later offered as empirical validation. The inequalities in (6) are exactly the signs read from the fitted Tables 1 and 2 (e.g., ImageNet100: α=0.620 < α′=0.702, γ=0.377 > γ′=0.338, β=4.882 < β′=5.840; TinyImageNet similarly). The paper never computes D*_p from Eq. (14) using fitted coefficients and compares it to an independently observed crossover; Figures 5-6 plot the same fitted curves. Thus the "distillation boundary" is a re-description of the fit, not an independent consequence of a first-principles law.
self definitional [Section 4.1, Lemma 1 proof]
"Considering the continuity of F(Dp), and the fact that it is positive for small Dp while lim_{Dp→∞} F(Dp) < 0, by the Intermediate Value Theorem, there must exist a point D∗∗p such that F(D∗∗p)=0, i.e.: E1(D∗∗p)=E2(D∗∗p)."

"Positive for small Dp" is exactly the claim that E1 > E2 in the data-scarce regime (distillation superiority), which is the phenomenon the theorem is supposed to establish. The premise is not derived from the fitted laws; the paper's own fitted exponents (Tables 1-2 and constraint (6) give α′ > α) imply F(Dp) → −∞ as Dp → 0 because the −D_p^{−α′}/λ′_p term dominates in Eq. (12). The proof therefore assumes the conclusion it claims to demonstrate, and the assumed premise is incompatible with the paper's own fitted numbers.

full rationale

The downstream scaling laws (Eqs. 1 and 4) are legitimate empirical fits; fitting a power law is not circular by itself. The circularity enters at Theorem 1: the constraints in Eq. (6) — E<E′, γ>γ′, β<β′, α−α′∈(−1,0), λ_m≈λ′_m, λ_f≈λ′_f — are read off from the same fitted Tables 1 and 2, and the theorem states them as 'such that there exists a critical threshold.' The existence of the boundary is therefore an input, not an output. Remark 1 explicitly says the conditions are 'derived from extensive experimental evidence,' so the subsequent 'theoretical demonstration' is a rearrangement of that evidence rather than an independent prediction. The paper never evaluates D*_p from Eq. (14) with the fitted coefficients and compares it to a held-out crossover; Figures 5-6 display the same data used for fitting. Additionally, Lemma 1's proof assumes 'F is positive for small Dp,' which is precisely the data-scarce distillation-superiority claim at issue, and this assumption is incompatible with the paper's fitted exponents: with α′>α, the −D_p^{−α′} term makes F(Dp)→−∞ as Dp→0 under Eq. (12). This is both an internal inconsistency and a begged question. Appendix D further concedes that model-size scaling is non-monotonic ('as the downstream model size increases, the error rate and loss do not simply decrease; in some cases, they even increase with model size'), which undermines the monotone M^{−β} term used in both E1 and the Δ<0 argument; that is a correctness risk rather than a circularity. No load-bearing self-citations or imported uniqueness theorems appear.

Assumptions & free parameters 16 free parameters · 3 assumptions · 0 invented entities

The central results rest on a large number of fitted exponents and coefficients. The theorem in Section 4 converts these fitted values into an existence claim, which is why the circularity burden is high. No new physical or architectural entities are introduced.

free parameters (16)

alpha (ImageNet100) = 0.620
Fitted exponent for pretraining data in Eq. 1; used in Eq. 6 and Lemma 1.
beta (ImageNet100) = 4.882
Fitted exponent for model size in Eq. 1; used in the condition beta < beta'.
gamma (ImageNet100) = 0.377
Fitted exponent for fine-tuning data in Eq. 1; used in gamma > gamma'.
alpha' (ImageNet100) = 0.702
Fitted pretraining exponent for distilled model law in Eq. 4; used in conditions (6).
beta' (ImageNet100) = 5.840
Fitted model-size exponent for distilled law in Eq. 4.
gamma' (ImageNet100) = 0.338
Fitted fine-tuning exponent for distilled law in Eq. 4.
eta' (ImageNet100) = 2.053
Fitted teacher model size exponent in Eq. 4.
alpha (TinyImageNet) = 0.412
Fitted exponent for pretraining data in Eq. 1.
beta (TinyImageNet) = 5.086
Fitted exponent for model size in Eq. 1.
gamma (TinyImageNet) = 0.359
Fitted exponent for fine-tuning data in Eq. 1.
alpha' (TinyImageNet) = 0.475
Fitted pretraining exponent for distilled law in Eq. 4.
beta' (TinyImageNet) = 5.496
Fitted model-size exponent for distilled law in Eq. 4.
gamma' (TinyImageNet) = 0.321
Fitted fine-tuning exponent for distilled law in Eq. 4.
eta' (TinyImageNet) = 1.982
Fitted teacher model size exponent in Eq. 4.
E_inf, lambda_p, lambda_m, lambda_f (ImageNet100) = 1.44e-14, 4.39e-3, 3.05e-2, 1.79e-1
Fitted coefficients for Eq. 1 from Table 4.
E_inf, lambda_p, lambda_m, lambda_f (TinyImageNet) = 1.76e-27, 2.58e-2, 4.55e-1, 1.44e-1
Fitted coefficients for Eq. 1 from Table 4.

assumptions (3)

domain assumption Downstream error follows the additive power-law Ansatz E = E_inf + D_p^{-alpha}/lambda_p + M^{-beta}/lambda_m + D_f^{-gamma}/lambda_f (Eq. 1).
This functional form is assumed without derivation and is used for both standard and distilled models; the paper's own data contradict monotonicity in M.
ad hoc to paper The parametric constraints E < E', gamma > gamma', beta < beta', alpha - alpha' in (-1, 0), lambda_m approximately lambda'_m, lambda_f approximately lambda'_f (Eq. 6) hold.
Remark 1 states these are derived from experimental evidence; no independent justification is given. The theorem's existence proof depends entirely on these inequalities.
domain assumption F(D_p) is positive for small D_p, allowing the Intermediate Value Theorem to be applied.
Lemma 1 asserts positivity without proof; with ImageNet100 fits (alpha' > alpha), F tends to -infinity as D_p approaches 0, so the assertion is doubtful.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Scaling Laws for Data-Efficient Visual Transfer Learning." pith.science (2026). https://pith.science/paper/UCDF7BQL

@misc{pith2026250413219,
  author       = {Pith},
  title        = {Pith review of: Scaling Laws for Data-Efficient Visual Transfer Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UCDF7BQL}},
  note         = {Machine review of arXiv:2504.13219}
}

read the original abstract

Current scaling laws for visual AI models focus predominantly on large-scale pretraining, leaving a critical gap in understanding how performance scales for data-constrained downstream tasks. To address this limitation, this paper establishes the first practical framework for data-efficient scaling laws in visual transfer learning, addressing two fundamental questions: 1) How do scaling behaviors shift when downstream tasks operate with limited data? 2) What governs the efficacy of knowledge distillation under such constraints? Through systematic analysis of vision tasks across data regimes (1K-1M samples), we propose the distillation boundary theory, revealing a critical turning point in distillation efficiency: 1) Distillation superiority: In data-scarce conditions, distilled models significantly outperform their non-distillation counterparts, efficiently leveraging inherited knowledge to compensate for limited training samples. 2) Pre-training dominance: As pre-training data increases beyond a critical threshold, non-distilled models gradually surpass distilled versions, suggesting diminishing returns from knowledge inheritance when sufficient task-specific data becomes available. Empirical validation across various model scales (2.5M to 38M parameters) and data volumes demonstrate these performance inflection points, with error difference curves transitioning from positive to negative values at critical data thresholds, confirming our theoretical predictions. This work redefines scaling laws for data-limited regimes, bridging the knowledge gap between large-scale pretraining and practical downstream adaptation, addressing a critical barrier to understanding vision model scaling behaviors and optimizing computational resource allocation.

Figures

Figures reproduced from arXiv: 2504.13219 by the authors.

**Figure 2.** Scaling laws for pre-training data size on ImageNet100. Illustration of the impact of pre-training data volume ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Scaling laws for fine-tuning data size on ImageNet100. Illustration of the impact of fine-tuning data volume ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (12 more)

**Figure 4.** Figure 4: Scaling laws for model size on ImageNet100. Illustration of the impact of model size ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Evidence for the distillation boundary theory. Error rates of original (blue) and distilled (purple) models are compared [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Evidence for the distillation boundary theory. As [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Impact of pre-training data volume (𝐷𝑝 ) on error rate and loss across different model sizes (𝑀) on CIFAR10. 0.0 0.2 0.4 0.6 0.8 1.0 1.2 10 20 30 40 50 60 70 E rro r Model Params = 3M Finetuning Data Size=2K Finetuning Data Size=5K Finetuning Data Size=12K Finetuning D…

**Figure 8.** Figure 8: Impact of pre-training data volume (𝐷𝑝 ) on error rate and loss across different model sizes (𝑀) on CIFAR100 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 9.** Figure 9: Impact of pre-training data volume (𝐷𝑝 ) on error rate and loss across different model sizes (𝑀) on TinyImageNet. 0 20 40 0 5 10 15 20 25 30 35 E rro r Model Params = 3M Pretraining Data Size=64K Pretraining Data Size=0.1M Pretraining Data Size=0.3M Pretraining Data Si…

**Figure 10.** Figure 10: Impact of finetuning data volume (𝐷𝑓 ) on error rate and loss across different model sizes (𝑀) on CIFAR10 [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

**Figure 11.** Figure 11: Impact of finetuning data volume (𝐷𝑓 ) on error rate and loss across different model sizes (𝑀) on CIFAR100. 0 20 40 60 80 100 20 30 40 50 60 70 80 E rro r Model Params = 3M Pretraining Data Size=64K Pretraining Data Size=0.1M Pretraining Data Size=0.3M Pretraining Dat…

**Figure 12.** Figure 12: Impact of finetuning data volume (𝐷𝑓 ) on error rate and loss across different model sizes (𝑀) on TinyImageNet [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

**Figure 13.** Figure 13: Impact of model size (𝑀) on downstream task error rate and cross-entropy loss across different pre-training data sizes (𝐷𝑝 ) on CIFAR10. 0 10 20 30 40 10 20 30 40 50 60 70 E rro r Pretraining Data Size = 64K Finetuning Data Size=2K Finetuning Data Size=5K Finetuning D…

**Figure 14.** Figure 14: Impact of model size (𝑀) on downstream task error rate and cross-entropy loss across different pre-training data sizes (𝐷𝑝 ) on CIFAR100. 0 10 20 30 40 20 30 40 50 60 70 80 E rro r Pretraining Data Size = 64K Finetuning Data Size=5K Finetuning Data Size=10K Finetuning…

**Figure 15.** Figure 15: Impact of model size (𝑀) on downstream task error rate and cross-entropy loss across different pre-training data sizes (𝐷𝑝 ) on TinyImageNet [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
cs.LG 2025-06 conditional novelty 6.0 of 10

A newly fitted scaling law with model-size-dependent data exponents predicts LLM loss more accurately than Chinchilla, including at a held-out 25.1B model.

Reference graph

Works this paper leans on

51 extracted references · 19 canonical work pages · cited by 1 Pith paper

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023
[2]

Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Ham- bardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. 2023. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning . PMLR, 265–279

work page 2023
[3]

Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. 2023. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Processing Systems 36 (2023), 16406–16425

work page 2023
[4]

Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. 2022. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10925–10934

2022
[5]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

2020
[6]

Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. 2025. Distillation Scaling Laws. arXiv preprint arXiv:2502.08606 (2025)

arXiv 2025
[7]

Jang Hyun Cho and Bharath Hariharan. 2019. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision. 4794–4802

work page 2019
[8]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Im- agenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition . Ieee, 248–255

2009

Show all 51 references

[9]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv prepri...

2020 arXiv
[10]

William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39

2022
[11]

Jonas Geiping and Tom Goldstein. 2023. Cramming: Training a Language Model on a single GPU in one day.. In International Conference on Machine Learning . PMLR, 11117–11143

2023
[12]

Anthony Gillioz, Jacky Casas, Elena Mugellini, and Omar Abou Khaled. 2020. Overview of the Transformer-based Models for NLP Tasks. In 2020 15th Confer- ence on computer science and information systems (FedCSIS) . IEEE, 179–183

2020
[13]

Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowl- edge distillation: A survey. International Journal of Computer Vision 129, 6 (2021), 1789–1819

2021
[14]

Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al . 2022. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence 45, 1 (2022), 87–110

2022
[15]

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Ja- cob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al
[16]

Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. Scaling laws for transfer. arXiv preprint arXiv:2102.01293 (2021)

2021 arXiv
[17]

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. 2017. Deep learning scaling is predictable. Empirically. arXiv 1712 (2017), 2

2017
[18]

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision- language representation learning with noisy text supervision. In International conference on machine learning . PMLR, 4904–4916

2021
[20]

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

2020 arXiv
[21]

Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fa- had Shahbaz Khan, and Mubarak Shah. 2022. Transformers in vision: A survey. ACM computing surveys (CSUR) 54, 10s (2022), 1–41

2022
[22]

Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009)

2009
[23]

Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020. The open images dataset v4: Unified image classification, object detection, and visual relationship detec...

2020
[24]

Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2022. A survey of transformers. AI open 3 (2022), 111–132

2022
[25]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proce...

2014
[26]

Yuzhen Lu and Sierra Young. 2020. A survey of public datasets for computer vision tasks in precision agriculture. Computers and Electronics in Agriculture 178 (2020), 105760

2020
[27]

Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. 2023. Scaling open- vocabulary object detection. Advances in Neural Information Processing Systems 36 (2023), 72983–73007

2023
[28]

Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2023. Scaling data-constrained language models. Advances in Neural Information Processing Systems 36 (2023), 50358–50376

2023
[29]

Narendra Patwardhan, Stefano Marrone, and Carlo Sansone. 2023. Transformers in the real world: A survey on nlp applications. Information 14, 4 (2023), 242

2023
[30]

Mary Phuong and Christoph Lampert. 2019. Towards understanding knowledge distillation. In International conference on machine learning . PMLR, 5142–5151

2019
[31]

Gabriele Prato, Simon Guiroy, Ethan Caballero, Irina Rish, and Sarath Chandar
[32]

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al . 2018. Improving language understanding by generative pre-training. (2018)

2018
[33]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9

2019
[34]

Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. 2019. A constructive prediction of the generalization error across scales. arXiv preprint arXiv:1909.12673 (2019)

2019 arXiv
[35]

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos
[36]

Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. 2017. Re- visiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision . 843–852

2017
[37]

Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning . PMLR, 6105–6114

2019
[38]

Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler
[39]

Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler
[40]

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In International conference on machine learning . PMLR, 10347–10357

2021
[41]

Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. 2023. Image captioners are scalable vision learners too.Advances in Neural Information Processing Systems 36 (2023), 46830–46855

2023
[42]

Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551 (2022)

2022 arXiv
[43]

I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. 2019. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546 (2019)

2019 arXiv
[44]

arXiv preprint arXiv:2109.10686 (2021)

Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109.10686 (2021)

2021 arXiv
[45]

Wenxuan Yang, Hanyu Zhang, Weimin Tan, Yuqi Sun, and Bo Yan. 2024. Opti- mizing V-information for Self-Supervised Pre-training Data-Effective Medical Foundation Models. arXiv preprint arXiv:2408.07107 (2024)

2024 arXiv
[46]

Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. 2017. A gift from knowl- edge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4133–4141

2017
[47]

Peihao Wang, Rameswar Panda, and Zhangyang Wang. 2023. Data efficient neural scaling law via model reusing. In International Conference on Machine Learning. PMLR, 36193–36204

2023
[49]

Wenxuan Yang, Weimin Tan, Yuqi Sun, and Bo Yan. 2024. A Medical Data- Effective Learning Benchmark for Highly Efficient Pre-training of Foundation Models. In Proceedings of the 32nd ACM International Conference on Multimedia . 3499–3508

2024
[52]

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. 2022. Scal- ing vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 12104–12113. Yang et al. Scaling Laws for Data-Efficient Visual Transfer Learning Tabl...

2022
[2020]

arXiv preprint arXiv:2010.14701 (2020)

Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701 (2020)

2020 arXiv
[2021]

arXiv preprint arXiv:2110.06990 (2021)

Scaling laws for the few-shot adaptation of pre-trained image classifiers. arXiv preprint arXiv:2110.06990 (2021)

2021 arXiv
[2022]

Advances in Neural Information Processing Systems 35 (2022), 19523–19536

Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems 35 (2022), 19523–19536

2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023

[2] [2]

Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Ham- bardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. 2023. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning . PMLR, 265–279

work page 2023

[3] [3]

Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. 2023. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Processing Systems 36 (2023), 16406–16425

work page 2023

[4] [4]

Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. 2022. Knowledge distillation: A good teacher is patient and consistent. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10925–10934

2022

[5] [5]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

2020

[6] [6]

Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. 2025. Distillation Scaling Laws. arXiv preprint arXiv:2502.08606 (2025)

arXiv 2025

[7] [7]

Jang Hyun Cho and Bharath Hariharan. 2019. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision. 4794–4802

work page 2019

[8] [8]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Im- agenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition . Ieee, 248–255

2009