REVIEW 3 major objections 3 minor 13 cited by
Distillation Scaling Laws
T0 review · 3 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims a single scaling-law formula, Equation 8, predicts the cross-entropy of a distilled student from student size, distillation tokens, and teacher cross-entropy, and derives compute-optimal recipes from it.
desk verdict A genuinely new empirical distillation scaling law with real finite-data predictive value, but the headline conclusion that supervised learning catches up at large compute is a hand-imposed assumption, not a measured result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is Equation 8, a broken power law in the teacher cross-entropy $L_T$ multiplied by a standard data-and-model power law in student size $N_S$ and distillation tokens $D_S$. The transition between power-law regimes happens when $L_T / e_{LS} = d_1$, so the relative learning capacities of teacher and student, not their raw sizes, carry the argument. The student supervised cross-entropy $e_{LS}$ is itself predicted from the supervised scaling law, which lets the paper express all teacher effects through a single scalar. Fitting is enabled by three experimental protocols—fixed-M teacher with IsoFLOP students, IsoFLOP teachers with fixed-M students, and fixed-M teacher with fixed-M student—and the applications use a compute-cost model that separates student training, teacher logit inference, and teacher pretraining costs.
What would settle it
A decisive check is to refit Equation 8 without enforcing the Appendix E.6 infinite-data equality, then distill a fixed student size (for example 1B parameters) at token budgets past the predicted supervised crossover and compare the measured student cross-entropy to a supervised model trained on the same budget; if the distilled student stays below the supervised curve, or if the crossover compute does not grow with student size as predicted, the law is falsified.
Extended reading notes
Core claim
The central claim is that the cross-entropy $L_S$ of a student with $N_S$ parameters distilled on $D_S$ tokens from a teacher with cross-entropy $L_T$ follows $L_S = L_T + L_T^{-c_0} \left(1 + \left(L_T / (e_{LS} d_1)\right)^{1/f_1}\right)^{-c_1 f_1} \left(A / N_S^{\alpha'} + B / D_S^{\beta'}\right)^{\gamma'}$, where $e_{LS}$ is the supervised cross-entropy the student would have achieved on the same budget. The formula is a broken power law in teacher quality: the inner ratio $L_T / e_{LS}$ controls the transition between the student being the stronger learner and being the weaker learner, which is how the capacity gap enters. The paper reports that this law fits observations at roughly 1% relative prediction error, including when extrapolated from weaker to stronger models, and that for the optimally chosen teacher the infinite-data limit is consistent with supervised learning.
Load-bearing premise
The paper assumes, by hand, that distillation with the optimally chosen teacher converges to the same cross-entropy as supervised learning at infinite data; the conclusion that supervised learning wins at large compute follows from that assumption, so the central practical recommendation would collapse if the equality is wrong.
Editorial extensions
If this is right
- Student cross-entropy becomes a predictable quantity: the paper reports fits and extrapolations at roughly 1% relative error, so planners can estimate the value of a candidate teacher before spending compute.
- Teacher size and teacher training tokens cease to be separate search dimensions, because both enter the law only through teacher cross-entropy $L_T$.
- With a teacher that already exists, distillation is the more compute-efficient choice only below a student-size-dependent threshold; beyond that threshold supervised learning on the same budget wins.
- If a teacher must be trained from scratch for a single student, supervised learning is generally preferable in the paper's controlled setting.
- The capacity gap is governed by the ratio $L_T / e_{LS}$, so the optimal teacher loss for a given student follows a power-law decrease with student size, generalizing earlier observations about teacher scale.
Reading between the lines
- If the law transfers across data distributions, it suggests a domain-shift test: a teacher with low cross-entropy on its own distribution but high cross-entropy on the student's target distribution would be predicted to be a poor teacher unless the ratio $L_T / e_{LS}$ stays below the transition $d_1$, and this could be checked by distilling on a shifted target set while measuring the teacher's cr
- An unstated corollary of the power-law decrease in optimal teacher loss is that the best teacher 'maturity'—the training-token count that minimizes student loss—could be precomputed from supervised scaling laws alone, before any distillation runs are performed.
- The paper fixes the data-mixing weight at pure distillation and the temperature at one; a natural extension is to mix in the supervised next-token loss, since the paper's own contours imply a supervised-then-distillation schedule may be more efficient than either alone.
- Because the large-compute crossover conclusion rests on the hand-imposed infinite-data equality, a relaxed model in which distillation's infinite-data limit could exceed the supervised one would change the recommendation to switch to supervised learning at large compute; this is directly testable by refitting Equation 8 without that constraint.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a distillation scaling law, Eq. (8), that predicts the cross-entropy of a student model distilled from a teacher, using student size NS, distillation tokens DS, and teacher cross-entropy LT as inputs, where teacher size and teacher training tokens enter only through LT. The law is fit to a controlled set of transformer distillation runs ranging from 143M to 12.6B parameters and up to 512B tokens on the C4 dataset, and the paper reports roughly 1% relative prediction error, including extrapolation from weaker to stronger students. The authors then use the law to study compute-optimal distillation: for a fixed student size and compute budget they compute optimal student tokens, teacher size, and teacher tokens in four compute-accounting scenarios, and conclude that distillation can outperform supervised learning at moderate compute but that supervised learning always matches or beats optimal distillation at sufficiently high compute. Appendices provide a kernel-regression analysis of the capacity gap, additional scaling-law fits, calibration studies, and practical hyperparameter checks.
Significance. If the distillation scaling law holds beyond the fitted range, this is the largest controlled empirical study of pretraining distillation for language models and would give practitioners a practical tool for allocating compute between teacher and student, a question of immediate industrial relevance. The empirical fit at 1% relative error, the explicit validation of weak-to-strong extrapolation in Figure 5b, the careful controlled setup (pure distillation, λ=1, temperature τ=1, µP), and the appended kernel-regression explanation of the capacity gap are genuine strengths. The paper is also unusually honest about its limitations, admitting in Appendix A that Eq. (8) is not derived from first principles and in Appendix E.6 that the infinite-data consistency was imposed by hand. The main reservation is that the headline high-compute conclusion in Section 5.3 is a direct consequence of that hand-imposed constraint rather than an empirical measurement, which limits the confidence in the paper's most sweeping practical recommendation.
major comments (3)
- [Section 5.3 / Appendix E.6] The claim that supervised learning always matches optimal distillation at sufficient compute is not established by the data. Appendix E.6 states that the infinite-data limit of the distillation law, Eq. (37), was 'imposed added by hand' so that LS(NS, D=∞, L*_T) equals the supervised limit L(NS, D=∞), and that this equality holds only for certain coefficient values. The experiments used to fit Eq. (8) cover DS up to 512B tokens, while the crossover to supervised learning is extrapolated to 10T+ tokens (Figures 17 and 40). Consequently, the Section 5.3 statement that 'supervised learning always matches optimal distillation at sufficient compute budget' is a consequence of the chosen ansatz, not a measured property of distillation. Please provide an independent test of the infinite-data equality, or explicitly reclassify this part of the analysis as a modeling assumption and qualify the corresponding practical advice.
- [Section 4.3 / Appendix A / Appendix F.3] The distillation law introduces seven additional fitted coefficients {c0, c1, d1, f1, α′, β′, γ′} on top of the six supervised coefficients, and the Limitations section concedes that Eq. (8) is not derived from first principles. The 1% relative error on interpolation and on weak-to-strong extrapolation is genuine evidence of a good in-range fit, but the asymptotic extrapolation to 100B+ parameter students and 10T+ tokens is far outside the fitted regime and depends entirely on a functional form selected for convenient limiting behavior. The paper should report the fitted coefficients with their uncertainties and should quantify how much the Section 5.3 crossover points move under reasonable perturbations of the hand-imposed infinite-data constraint; without such a sensitivity analysis, the extrapolated compute-optimal recipes should be presented as predictions of the model rather than as findings.
- [Section 4.2, finding 2 (teacher size/tokens enter only through LT)] The claim that teacher size NT and teacher tokens DT influence student cross-entropy only through LT is supported by Figure 3b for the specific family of transformer teachers trained on C4 in this study, but the paper extends this to 'removing teacher size and tokens as search dimensions' in the Impact Statement. Since all teachers in the main study share the same architecture, tokenizer, training distribution, and optimization setup, it is not yet established that LT alone summarizes teacher quality across architectures, training distributions, or distillation objectives. Please either narrow the scope of this recommendation or provide evidence from a more heterogeneous set of teachers.
minor comments (3)
- [Section 1] The text contains a grammatical typo: 'an comprehen-sive, controlled study' should read 'a comprehensive, controlled study.'
- [Appendix E.6] The phrase 'was imposed added by hand' is grammatically redundant; it should read 'was imposed by hand' or 'was added by hand,' and the sentence would be clearer if it explained whether the constraint is an equality constraint on the coefficients or a prior on the functional form.
- [Appendix D.4.2] The sentence 'The compute at which distillation and supervised learning produce similar models grows with student size' is correct, but the immediately following sentence begins 'Continuing the previous observation...' and the reader must infer that the growth is in student size, not in compute; please make the independent variable explicit.
Circularity Check
High-compute conclusion that supervised learning matches or beats distillation is enforced by a hand-imposed infinite-data constraint in Eq. 8, not by measured data; the finite-data distillation law itself remains an empirical fit.
-
self definitional
[Appendix E.6, Eq. 37]
"We stress that unlike our three motivations for the equation properties (Section 4.3), this infinite data limit was imposed added by hand, and is only true for certain values scaling coefficients."
The functional form of Eq. 8 / Eq. 37 was chosen so that LS(NS, D=infinity, L*_T) equals the supervised infinite-data limit L(NS, D=infinity). This equality is stated to be imposed by hand, not derived from measured distillation runs. All later conclusions that distillation cannot beat supervised learning at large compute or data are obtained by evaluating this same imposed limit, so the asymptotic 'prediction' is a restatement of the modeling constraint rather than an empirical finding.
-
self definitional
[Section 5.3, Figure 8]
"Supervised learning always matches optimal distillation at sufficient compute budget, with the intersection favoring supervised learning increasing as student size grows. In Figure 8 we see that supervised learning always matches the best case distillation setting at some total compute budget, as anticipated from the asymptotic analysis in Figure 40."
Figure 40 plots Eq. 37, which already contains the hand-imposed infinite-data equality. The compute-crossover conclusion is therefore anticipated by construction, not discovered from data. The measured distillation data extend only to DS roughly 512B tokens, while the crossover at 10T+ tokens is an extrapolation of the constrained ansatz. Without an independent test of the imposed limit, the Section 5.3 recommendation to switch to supervised learning at high compute is a consequence of the chosen equation form.
1 more flagged steps
-
self definitional
[Section 5.1]
"Supervised learning always outperforms distillation given enough student compute or tokens. This is expected; in the large data regime, supervised learning can find the best solution limited by model size N (Equation 1), whereas distillation only finds this solution for the optimal teacher L*_T (see Appendix E.6), and is otherwise limited by the distillation process."
The 'expected' claim is justified by citing Appendix E.6, which is precisely the limit the authors imposed by hand on the scaling law. The comparison between distillation and supervised learning in the asymptotic regime thus reduces to the constraint chosen when writing Eq. 8. The finite-data regime where distillation is measured to help remains an independent empirical result; only the asymptotic comparison is circular.
full rationale
The finite-data distillation scaling law is an empirical fit to a large controlled study, and the paper's interpolations and extrapolations to unseen teachers are genuine predictions evaluated against held-out data. The circularity is confined to the asymptotic comparison. Appendix E.6 explicitly states that the infinite-data equality was 'imposed added by hand,' and Sections 5.1 and 5.3 then present 'supervised learning always matches or outperforms distillation at sufficient compute' as a consequence of that same limit. Since no independent large-DS data are provided and the toy kernel-regression analysis in Appendix C does not establish the equality for overparameterized language models, the headline practical conclusion that distillation cannot beat supervised learning at high compute is a restatement of the ansatz rather than an empirical finding. The finite-data crossover and the compute-optimal recipes for modest budgets are still genuine fitted results. No load-bearing self-citation or uniqueness argument was found.
Assumptions & free parameters
free parameters (4)
- E (supervised irreducible error)
- A, B, alpha, beta, gamma (supervised law coefficients)
- c0, c1, d1, f1 (distillation law coefficients)
- alpha', beta', gamma' (student scaling exponents)
assumptions (5)
- domain assumption The supervised scaling law L(N, D) = E + (A/N^alpha + B/D^beta)^gamma holds and can be estimated from teacher runs.
- domain assumption Teacher size NT and teacher tokens DT affect the student only through teacher cross-entropy LT.
- domain assumption Pure distillation (lambda = 1) and temperature tau = 1 are representative of practical distillation.
- ad hoc to paper The infinite-data limit of the distillation law is imposed by hand to match the supervised infinite-data limit.
- domain assumption Repeating data up to four times on C4 has negligible impact on distillation loss, as in the supervised setting.
Cite this review
Pith. "Pith review of Distillation Scaling Laws." pith.science (2026). https://pith.science/paper/IJOWF3PN
@misc{pith2026250208606,
author = {Pith},
title = {Pith review of: Distillation Scaling Laws},
year = {2026},
howpublished = {\url{https://pith.science/paper/IJOWF3PN}},
note = {Machine review of arXiv:2502.08606}
}
read the original abstract
We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate the risks associated with large-scale distillation by enabling compute-optimal allocation for both the teacher and student to maximize student performance. We provide compute-optimal distillation recipes for two key scenarios: when a teacher already exists, and when a teacher needs training. In settings involving many students or an existing teacher, distillation outperforms supervised learning up to a compute level that scales predictably with student size. Conversely, if only one student is to be distilled and a teacher also requires training, supervised learning is generally preferable. Additionally, our large-scale study of distillation increases our understanding of the process and helps inform experimental design.
Figures
Figures from the paper (51 more)
Forward citations
Cited by 13 Pith papers
-
Bridging Compute- and Data-Optimal Pretraining
Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.
-
Weak-to-Strong Generalization via Direct On-Policy Distillation
Transferring the log-ratio of a small model's pre-RL and post-RL checkpoints provides a dense implicit reward that improves stronger student models at a fraction of the cost of direct RL.
-
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
Distilled pretraining improves test-time scaling via generation diversity but impairs induction-head-based in-context learning, with the trade-off explained by a bigram model analysis.
-
Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment
Distilling an 8B reasoning teacher into a 0.6B student recovers most summary quality at ~50× speed, but teacher type—not scale alone—determines which capabilities transfer.
-
Language Models Improve When Pretraining Data Matches Target Tasks
Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.
-
A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.
-
HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling
A compact hybrid GDN+attention model distilled from Nucleotide Transformer v2 outperforms similarly sized models and, on several tasks, its 500x larger teacher.
-
Scalable Strategies for Continual Learning with Replay
A replay-based continual learning toolkit that combines low-rank adaptation, a post-task consolidation phase, and sequential weight merging to cut replay sample usage by up to 65% at matched accuracy.
-
Skaling: Chinchilla's Exponents Meet Kaplan's Coupling
A one-parameter coupling of model size and data in the Chinchilla loss form reduces boundary prediction error and enables cheaper L-shaped profiling grids.
-
Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals
Relative policy-improvement signals from a weak proxy model, after simple calibration, can be transferred to improve stronger primary LLMs without re-exploring on the primary.
-
Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles
Soft-label distillation from same-architecture teacher ensembles improves respiratory sound classification and sets a new ICBHI score of 64.39, though gains are partly due to test-set-based selection of settings.
-
SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
SCOUT combines progressive distillation with a cross-attention module to make recursive latent reasoning work through fine-tuning, yielding up to 1.8% accuracy gains over standard fine-tuning.
-
Scaling Laws for Data-Efficient Visual Transfer Learning
The paper fits power-law scaling laws for downstream vision tasks and claims a pretraining-data threshold where distilled models stop outperforming non-distilled ones, but the theory's assumptions come from the fitted...
Reference graph
Works this paper leans on
-
[1]
Patient: Distillation works best when training for a large number of epochs, and
-
[2]
Our setting automatically satisfies consistency as there is no augmentation policy
Consistent: The teacher and the student see the same views of the data under an augmentation policy. Our setting automatically satisfies consistency as there is no augmentation policy. There is a remaining question about 30 Distillation Scaling Laws patience, which in our scenario corresponds to the large DS limit. We observe that for a given student size:
-
[5]
URL https: //doi.org/10.1145/3604930.3605705
doi: 10.1145/3604930.3605705. URL https: //doi.org/10.1145/3604930.3605705. Cho, J. H. and Hariharan, B. On the efficacy of knowledge distillation. In 2019 IEEE/CVF Interna- tional Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pp. 4793–4801. IEEE, 2019. doi: 10.1109/ICCV . 2019.00489. URL https://doi.org/1...
arXiv 2019
-
[6]
URL https://arxiv.org/abs/2110. 14168. Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenRe- view.net, 2024. URL https://openreview.net/ forum?id=mZn2Xyh9Ec. Dao, T., Fu, D. Y ., Ermon, S., Rudra, ...
-
[7]
URL https://arxiv.org/abs/2110. 12894. Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE Computer Society Con- ference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA , pp. 248–255. IEEE Computer Society, 2009. doi: 10.1109/CVPR.2...
arXiv 2009
-
[8]
URL https://epoch.ai/trends. Ac- cessed: 2025-02-11. Fan, W., Lu, S., Li, X., Zhan, D., and Gan, L. Re- visit the essence of distilling knowledge through cali- bration. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21- 27, 2024 . OpenReview.net, 2024. URL https:// openreview.net/forum?id=NZgbwzaOIx. Furlan...
work page 2025
-
[10]
URL https://openreview.net/forum? id=8jU7wy7N7mA. Havrilla, A. and Liao, W. Understanding scaling laws with statistical and approximation theory for transformer neural networks on intrinsically low-dimensional data. CoRR, abs/2411.06646, 2024. doi: 10.48550/ARXIV . 2411.06646. URL https://doi.org/10.48550/ arXiv.2411.06646. Hendrycks, D., Burns, C., Basar...
work page Pith review arXiv doi:10.48550/arxiv.2411.06646 2024
-
[11]
URL https://arxiv.org/abs/2102. 01293. Hestness, J., Narang, S., Ardalani, N., Diamos, G. F., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y ., and Zhou, Y . Deep learning scaling is predictable, empir- ically. CoRR, abs/1712.00409, 2017. URL http: //arxiv.org/abs/1712.00409. Hinton, G. E., Vinyals, O., and Dean, J. Distill- ing the knowledge in a ne...
arXiv 2017
Show all 39 references
-
[14]
URL https: //doi.org/10.1145/3458817.3476209
doi: 10.1145/3458817.3476209. URL https: //doi.org/10.1145/3458817.3476209. Nguyen, T. Q. and Salazar, J. Transformers without tears: Improving the normalization of self-attention. In Niehues, J., Cattoni, R., Stüker, S., Negri, M., Turchi, M., Ha, T., Salesky, E., Sanabria, R...
-
[16]
URL https://doi.org/10.48550/arXiv.2303
doi: 10.48550/ARXIV .2303.08774. URL https://doi.org/10.48550/arXiv.2303. 08774. OpenAI and Pilipiszyn, A. Gpt-3 powers the next gener- ation of apps, 2021. URL http://website-url. com. Accessed on Jan 19, 2025. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi...
-
[19]
Sakaguchi, K., Bras, R
URL https://openreview.net/forum? id=ryenvpEKDr. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y . Winogrande: an adversarial winograd schema chal- lenge at scale. Commun. ACM , 64(9):99–106, 2021. doi: 10.1145/3474381. URL https://doi.org/ 10.1145/3474381. Sardana, N...
2021 doi
-
[20]
Shazeer, N
URL https://openreview.net/forum? id=0bmXrtTDUu. Shazeer, N. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https://arxiv.org/ abs/2002.05202. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V ., Hinton, G. E., and Dean, J. Outrageously large neu...
2002 arXiv
-
[21]
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G
URL https://openreview.net/forum? id=SkgpBJrtvS. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage mo...
-
[22]
spread its capacity
URL http://proceedings.mlr.press/ v139/yang21c.html. Yang, G. and Littwin, E. Tensor programs ivb: Adap- tive optimization in the infinite-width limit. CoRR, abs/2308.01814, 2023. doi: 10.48550/ARXIV .2308. 01814. URL https://doi.org/10.48550/ arXiv.2308.01814. Yang, G., Hu, E...
-
[25]
If the teacher is optimally chosen for the student, distilling on a large number of tokens produces the same result as training the model in a supervised way on the same number of tokens (Appendix E.6)
-
[26]
The second statement implies that the student should not be trained for too long, appearing to contradict patient teachers
Otherwise supervised learning outperforms distillation (Section 5.3). The second statement implies that the student should not be trained for too long, appearing to contradict patient teachers. To resolve the contradiction, first we note that the modes in Beyer et al. (2022) a...
2022
-
[27]
(2022) do not see the teacher training distribution directly, whereas ours do
The students in Beyer et al. (2022) do not see the teacher training distribution directly, whereas ours do
2022
-
[28]
The absence of a supervised baseline means that Beyer et al
There is no supervised baseline where a supervised model has access to both ImageNet21k and the target dataset. The absence of a supervised baseline means that Beyer et al. (2022) were unable to observe the point at which supervised learning becomes preferred to distillation a...
2022
-
[29]
We can compare student outputs relative ground-truth data, as in Appendix E.8.1 for the teachers
-
[30]
Calibration against ground-truth
We can compare student outputs with the outputs of its teacher. Calibration against ground-truth. First, let’s consider comparison against ground truth data. In Figure 43 we show student calibration with respect to the dataset labels for both teacher distribution distillation ...
-
[31]
In Figure 43a, we observe that the student is well-calibrated against ground truth data
Distilled on the full teacher distribution. In Figure 43a, we observe that the student is well-calibrated against ground truth data. Similar to the teacher’s calibration plot in Figure 42, we see a small discrepancy at very low and very high confidence values, and the ECE value is low
-
[32]
In Figure 43b, we see that a student trained only on its teacher’s top- 1 prediction, is not calibrated against ground truth data
Distilled on teacher top-1. In Figure 43b, we see that a student trained only on its teacher’s top- 1 prediction, is not calibrated against ground truth data. The blue points below the dashed line indicate an overconfident student, i.e. , its predicted confidence is higher tha...
-
[33]
We see in Figure 44a that when distilled from the full teacher distribution, the student is not calibrated against the teacher top-1
Distilled on the full teacher distribution. We see in Figure 44a that when distilled from the full teacher distribution, the student is not calibrated against the teacher top-1. The blue points are above the dashed line, which means that the empirical accuracy is higher than t...
-
[34]
In Figure 44b we observe that a student is distilled from its teacher’s top- 1 is calibrated with respect to teacher’s top-1
Distilled on teacher top-1. In Figure 44b we observe that a student is distilled from its teacher’s top- 1 is calibrated with respect to teacher’s top-1. 0.0 0.5 1.0 NT =198M ECE=38.5% NT =546M ECE=31.3% NT =975M ECE=27.3% NT =1.82B ECE=24.5% 0.0 0.5 1.0 0.0 0.5 1.0 NT =2.72B ...
-
[35]
In Figure 45a, we see that when the student is confident, it matches the teacher confidence
Distilled on the full teacher distribution. In Figure 45a, we see that when the student is confident, it matches the teacher confidence. However, as the teacher model grows in size, when the student is less confident, it it systematically underestimates its confidence. This su...
-
[36]
In Figure 45b, for small teachers, we observe student overconfidence
Distilled on teacher top-1. In Figure 45b, for small teachers, we observe student overconfidence. As the teacher increases in size, the student’s overconfidence in low-confidence bins transitions to underconfidence. At the same time, 54 Distillation Scaling Laws the student’s ...
-
[37]
In Figure 45a we complete the picture from Figure 45a and see that the part of the distribution the student struggles to model is actually the place where teacher is most confident
Distilled on the full teacher distribution. In Figure 45a we complete the picture from Figure 45a and see that the part of the distribution the student struggles to model is actually the place where teacher is most confident
-
[38]
log LT , −c0 log LT − c1f1 log 1 + LT d1 eLS 1/f1 ! + γ log A′ N α S + B′ Dβ S !# (48) = LSE
Distilled on teacher top-1. In Figure 45b we see that the student is systematically overconfident for all values of teaacher confidence, except for the largest teachers, where the student is underconfident when those teachers are most confident. 0.0 0.5 1.0 NT =198M ECE=1.7% N...
2022
-
[39]
For all experiments, the English-only subset of the C4 dataset (Raffel et al., 2020) is used
of 10−4 is used for stability, slightly decreasing norm growth at the end of the training. For all experiments, the English-only subset of the C4 dataset (Raffel et al., 2020) is used. The C4 dataset was chosen because of its wide usage in the research community. While C4 is b...
2023
-
[507]
doi: 10.1007/978-3-030-58558- 7\_29
Springer, 2020. doi: 10.1007/978-3-030-58558- 7\_29. URL https://doi.org/10.1007/978- 3-030-58558-7_29 . Kraft, D. A Software Package for Sequential Quadratic Programming. Deutsche Forschungs- und Versuch- sanstalt für Luft- und Raumfahrt Köln: Forschungs- bericht. Wiss. Beric...
-
[2008]
URL https: //doi.org/10.1109/ICVGIP.2008.47
doi: 10.1109/ICVGIP.2008.47. URL https: //doi.org/10.1109/ICVGIP.2008.47. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774,
2008 arXiv
-
[2018]
Gadre, S
URL http://proceedings.mlr.press/ v80/furlanello18a.html. Gadre, S. Y ., Smyrnis, G., Shankar, V ., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., Xin, R., Nezhurina, M., Vasiljevic, I., Jit- sev, J., Dimakis, A. G., Ilharco, G., Song, S., Kol- ...
- [2020]
- [2021]
-
[2022]
Menon, A
URL https://openreview.net/forum? id=ph3AYXpwEb. Menon, A. K., Rawat, A. S., Reddi, S. J., Kim, S., and Kumar, S. Why distillation helps: a statistical per- spective. CoRR, abs/2005.10419, 2020. URL https: //arxiv.org/abs/2005.10419. Mesnard, T., Hardin, C., Dadashi, R., Bhupa...
2005 arXiv
-
[2023]
Carrell, A
URL https://openreview.net/forum? id=sckjveqlCZ. Carrell, A. M., Mallinar, N., Lucas, J., and Nakki- ran, P. The calibration generalization gap. CoRR, abs/2210.01964, 2022. doi: 10.48550/ARXIV .2210. 01964. URL https://doi.org/10.48550/ arXiv.2210.01964. CERN. Cern data centre...
-
[2024]
URL https://doi.org/10.48550/arXiv.2406
doi: 10.48550/ARXIV .2406.19146. URL https://doi.org/10.48550/arXiv.2406. 19146. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y ., Li, W., and Liu, P. J. Explor- ing the limits of transfer learning with a unified text-to- text transformer. J. Ma...
-
[6239]
v34i05.6239
URL https://doi.org/10.1609/aaai. v34i05.6239. Blasiok, J., Gopalan, P., Hu, L., and Nakkiran, P. When does optimizing a proper loss yield calibration? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processin...
- [7439]
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.