{"id":"8f476a4c-b20a-46a2-aa12-2fba708b1ab4","arxiv_id":"2502.08606","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A distillation scaling law predicts student cross-entropy from teacher loss, student size, and data, and gives compute-optimal teacher-student allocations.","lead":"The paper proposes a formula that predicts how well a small AI model trained by imitating a larger teacher model will perform, based on model sizes, training data, and compute budget. It shows when distillation is cheaper than ordinary training and how to divide compute between teacher and student.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The high-compute conclusion that distillation cannot beat supervised learning is enforced by a hand-imposed infinite-data constraint in Eq. 8, not by measured data; the claim is tautological unless the constraint is independently tested.","rationale":"The reader's weakest assumption identifies exactly the same concern: the hand-imposed infinite-data constraint in Appendix E.6 drives the Section 5.3 conclusion that distillation cannot beat supervised learning at large compute. I agree this is the single most load-bearing concern. The paper is otherwise strong: it presents a large controlled study, a functional form that fits observed data at about 1% relative error on interpolation and some extrapolation, and honest limitation statements in Appendix A. The machine-checked or reproducible artifacts are absent, and no code is released, which raises the burden on the empirical claims. But the correctness risk is concentrated in the extrapolation to the infinite-data regime, which is not just an extrapolation in D—it is an identity that the authors explicitly state was imposed rather than measured. Consequently, the headline practical conclusion ('supervised learning is generally preferable' when teacher training is counted) is not independently established. This does not invalidate the paper's more modest claims about finite-token distillation, and the reader's CONDITIONAL verdict already captures the need for additional justification. No verdict change is required, but the concern should be stated prominently in any revision request: the authors should either provide direct evidence at much larger DS, or refit with a free asymptotic parameter and show the crossover survives, or substantially soften the high-compute conclusion.","tokens_in":61717,"tokens_out":3077,"duration_ms":34629,"concrete_test":"Refit Equation 8 to the published data without imposing the infinite-data equality. Replace the asymptotic term so that LS(NS, D=∞, L*_T) is a free parameter δ per student size, e.g., LS(NS,∞) = δ · (E + (A NS^{-α})^γ), and re-estimate all coefficients. Then rerun the SLSQP compute-optimal analysis of Section 5.3 (Figures 17 and 18) with this unconstrained law. If the compute at which supervised loss becomes equal to or better than distillation still falls within an order of magnitude of the original crossover for all student sizes, the conclusion is robust; if the crossover shifts materially or disappears, the headline claim is an artifact of the imposed constraint.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix E.6 states that the infinite-data limit LS(NS, D=∞, L*_T) = L(NS, D=∞) 'was imposed added by hand' and is only true for certain coefficient values. This constraint is baked into the functional form of Equation 8, so the Section 5.3 claim that supervised learning always matches or beats distillation at sufficiently high compute (Figures 17 and 40) is not an empirical finding—it is a direct consequence of the chosen ansatz. The measured data only span DS up to 512B tokens, while the compute-optimal crossover is extrapolated to 10T+ tokens and beyond. For the central claim to hold, this imposed equality must be a genuine property of the distillation process, not merely a modeling convenience. The paper provides no independent evidence at large DS, and the toy kernel regression in Appendix C does not transfer to overparameterized language models. If the true infinite-data limit were lower or higher than the supervised limit, the practical recommendation to switch to supervised learning at high compute would be unsupported. This is the most load-bearing concern because it undermines the paper's headline practical conclusion, not just the quantitative accuracy of the law.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a distillation scaling law, Eq. (8), that predicts the cross-entropy of a student model distilled from a teacher, using student size NS, distillation tokens DS, and teacher cross-entropy LT as inputs, where teacher size and teacher training tokens enter only through LT. The law is fit to a controlled set of transformer distillation runs ranging from 143M to 12.6B parameters and up to 512B tokens on the C4 dataset, and the paper reports roughly 1% relative prediction error, including extrapolation from weaker to stronger students. The authors then use the law to study compute-optimal distillation: for a fixed student size and compute budget they compute optimal student tokens, teacher size, and teacher tokens in four compute-accounting scenarios, and conclude that distillation can outperform supervised learning at moderate compute but that supervised learning always matches or beats optimal distillation at sufficiently high compute. Appendices provide a kernel-regression analysis of the capacity gap, additional scaling-law fits, calibration studies, and practical hyperparameter checks.","tokens_in":61979,"tokens_out":5188,"duration_ms":60760,"significance":"If the distillation scaling law holds beyond the fitted range, this is the largest controlled empirical study of pretraining distillation for language models and would give practitioners a practical tool for allocating compute between teacher and student, a question of immediate industrial relevance. The empirical fit at 1% relative error, the explicit validation of weak-to-strong extrapolation in Figure 5b, the careful controlled setup (pure distillation, λ=1, temperature τ=1, µP), and the appended kernel-regression explanation of the capacity gap are genuine strengths. The paper is also unusually honest about its limitations, admitting in Appendix A that Eq. (8) is not derived from first principles and in Appendix E.6 that the infinite-data consistency was imposed by hand. The main reservation is that the headline high-compute conclusion in Section 5.3 is a direct consequence of that hand-imposed constraint rather than an empirical measurement, which limits the confidence in the paper's most sweeping practical recommendation.","major_comments":[{"comment":"The claim that supervised learning always matches optimal distillation at sufficient compute is not established by the data. Appendix E.6 states that the infinite-data limit of the distillation law, Eq. (37), was 'imposed added by hand' so that LS(NS, D=∞, L*_T) equals the supervised limit L(NS, D=∞), and that this equality holds only for certain coefficient values. The experiments used to fit Eq. (8) cover DS up to 512B tokens, while the crossover to supervised learning is extrapolated to 10T+ tokens (Figures 17 and 40). Consequently, the Section 5.3 statement that 'supervised learning always matches optimal distillation at sufficient compute budget' is a consequence of the chosen ansatz, not a measured property of distillation. Please provide an independent test of the infinite-data equality, or explicitly reclassify this part of the analysis as a modeling assumption and qualify the corresponding practical advice.","section":"Section 5.3 / Appendix E.6"},{"comment":"The distillation law introduces seven additional fitted coefficients {c0, c1, d1, f1, α′, β′, γ′} on top of the six supervised coefficients, and the Limitations section concedes that Eq. (8) is not derived from first principles. The 1% relative error on interpolation and on weak-to-strong extrapolation is genuine evidence of a good in-range fit, but the asymptotic extrapolation to 100B+ parameter students and 10T+ tokens is far outside the fitted regime and depends entirely on a functional form selected for convenient limiting behavior. The paper should report the fitted coefficients with their uncertainties and should quantify how much the Section 5.3 crossover points move under reasonable perturbations of the hand-imposed infinite-data constraint; without such a sensitivity analysis, the extrapolated compute-optimal recipes should be presented as predictions of the model rather than as findings.","section":"Section 4.3 / Appendix A / Appendix F.3"},{"comment":"The claim that teacher size NT and teacher tokens DT influence student cross-entropy only through LT is supported by Figure 3b for the specific family of transformer teachers trained on C4 in this study, but the paper extends this to 'removing teacher size and tokens as search dimensions' in the Impact Statement. Since all teachers in the main study share the same architecture, tokenizer, training distribution, and optimization setup, it is not yet established that LT alone summarizes teacher quality across architectures, training distributions, or distillation objectives. Please either narrow the scope of this recommendation or provide evidence from a more heterogeneous set of teachers.","section":"Section 4.2, finding 2 (teacher size/tokens enter only through LT)"}],"minor_comments":[{"comment":"The text contains a grammatical typo: 'an comprehen-sive, controlled study' should read 'a comprehensive, controlled study.'","section":"Section 1"},{"comment":"The phrase 'was imposed added by hand' is grammatically redundant; it should read 'was imposed by hand' or 'was added by hand,' and the sentence would be clearer if it explained whether the constraint is an equality constraint on the coefficients or a prior on the functional form.","section":"Appendix E.6"},{"comment":"The sentence 'The compute at which distillation and supervised learning produce similar models grows with student size' is correct, but the immediately following sentence begins 'Continuing the previous observation...' and the reader must infer that the growth is in student size, not in compute; please make the independent variable explicit.","section":"Appendix D.4.2"}],"recommendation":"major_revision","confidential_remarks":"This is a solid and unusually candid empirical scaling-law paper. The finite-data distillation law appears well supported internally, and the controlled experimental design is a strength. My concern is narrowly focused: the paper's headline asymptotic conclusion is enforced by an ansatz rather than measured, and the authors themselves disclose this in Appendix E.6. I would be willing to see the paper accepted after the high-compute claims are reframed as model predictions and the sensitivity of those predictions to the hand-imposed constraint is assessed. I do not see any novelty or attribution concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before reading. First, the distillation scaling law in Eq. 8 is a real empirical contribution: the claim that teacher quality enters only through teacher cross-entropy L_T, with a broken-power-law capacity-gap transition, is new and backed by a large controlled C4 study with roughly 1% relative fit errors and genuine extrapolation from weak to strong students. Second, the paper's most publicized conclusion—that supervised learning matches optimal distillation at sufficient compute—is not measured. It follows from an infinite-data equality that Appendix E.6 explicitly says was 'imposed added by hand' on the functional form. The stress-test concern lands; I checked, and the text is unambiguous.\n\nWhat is actually new: Eq. 8 predicts student loss from student size, student tokens, and teacher loss, generalizing Zhang et al.'s linear optimal-teacher rule. The finding that teacher size and pretraining tokens matter only through L_T is a strong, falsifiable empirical claim. The experimental design is careful: distinct splits for teacher and student, pure distillation (lambda=1) well motivated, and μP used for hyperparameter transfer. The kernel-regression demonstration of the capacity-gap U-shape is also a nice theoretical anchor.\n\nSoft spots, in proportion. The infinite-data constraint is the main one. Section 5.3's conclusion that supervised learning always catches up to optimal distillation is a direct consequence of that imposed limit, and the compute crossover is extrapolated to 10T+ tokens and 10^26 FLOPs, orders of magnitude beyond the 512B-token data. The law has 13 fitted coefficients and the functional form is chosen for desired limiting behavior, not derived; the authors say so themselves. There is no uncertainty quantification and no released artifacts. None of this kills the finite-data predictive value of Eq. 8, but it should cap how much practical weight the large-compute recipes carry.\n\nWho this is for: practitioners deciding teacher size and student token budgets under fixed compute, and scaling-law researchers. A serious referee should engage with it, but conditionally. The revisions that matter: state the infinite-data equality as an assumption and test it with at least one long-horizon run or a sensitivity analysis over coefficient values; add bootstrap or posterior intervals to the fitted law; and release run metadata even if not checkpoints. With those, I would be comfortable citing it for the empirical law and the L_T-mediation result.","headline":"A genuinely new empirical distillation scaling law with real finite-data predictive value, but the headline conclusion that supervised learning catches up at large compute is a hand-imposed assumption, not a measured result.","tokens_in":62546,"tokens_out":1726,"would_cite":true,"duration_ms":21634,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a single scaling-law formula, Equation 8, predicts the cross-entropy of a distilled student from student size, distillation tokens, and teacher cross-entropy, and derives compute-optimal recipes from it.","keywords":["knowledge distillation","scaling laws","language model pretraining","compute-optimal training","capacity gap","cross-entropy prediction","teacher-student distillation","power laws"],"falsifier":"A decisive check is to refit Equation 8 without enforcing the Appendix E.6 infinite-data equality, then distill a fixed student size (for example 1B parameters) at token budgets past the predicted supervised crossover and compare the measured student cross-entropy to a supervised model trained on the same budget; if the distilled student stays below the supervised curve, or if the crossover compute does not grow with student size as predicted, the law is falsified.","tokens_in":61511,"feed_emoji":"📉","tokens_out":7059,"duration_ms":74628,"temperature":0.7,"pith_summary":"This paper proposes a scaling law that predicts the cross-entropy a student language model will reach after being distilled from a teacher, given the student's parameter count, the number of distillation tokens, and the teacher's cross-entropy. The law states that teacher size and teacher training tokens matter only through the teacher's final cross-entropy, and that the capacity gap—the regime in which a stronger teacher hurts the student—is governed by the ratio of teacher loss to the student's own supervised loss. If correct, the law turns distillation planning into a small constrained-optimization problem: choose teacher loss and distillation tokens under a compute budget to minimize predicted student loss. The paper uses the law to derive compute-optimal recipes for four settings, and concludes that when a teacher already exists distillation wins up to a student-size-dependent compute threshold, whereas training a teacher for one student is generally worse than supervised learning.","feed_headline":"Distillation's payoff now fits one scaling equation","feed_subtitle":"Teacher size and tokens matter only through teacher loss; compute-optimal recipes follow.","key_machinery":"The central object is Equation 8, a broken power law in the teacher cross-entropy $L_T$ multiplied by a standard data-and-model power law in student size $N_S$ and distillation tokens $D_S$. The transition between power-law regimes happens when $L_T / e_{LS} = d_1$, so the relative learning capacities of teacher and student, not their raw sizes, carry the argument. The student supervised cross-entropy $e_{LS}$ is itself predicted from the supervised scaling law, which lets the paper express all teacher effects through a single scalar. Fitting is enabled by three experimental protocols—fixed-M teacher with IsoFLOP students, IsoFLOP teachers with fixed-M students, and fixed-M teacher with fixed-M student—and the applications use a compute-cost model that separates student training, teacher logit inference, and teacher pretraining costs.","core_discovery":"The central claim is that the cross-entropy $L_S$ of a student with $N_S$ parameters distilled on $D_S$ tokens from a teacher with cross-entropy $L_T$ follows $L_S = L_T + L_T^{-c_0} \\left(1 + \\left(L_T / (e_{LS} d_1)\\right)^{1/f_1}\\right)^{-c_1 f_1} \\left(A / N_S^{\\alpha'} + B / D_S^{\\beta'}\\right)^{\\gamma'}$, where $e_{LS}$ is the supervised cross-entropy the student would have achieved on the same budget. The formula is a broken power law in teacher quality: the inner ratio $L_T / e_{LS}$ controls the transition between the student being the stronger learner and being the weaker learner, which is how the capacity gap enters. The paper reports that this law fits observations at roughly 1% relative prediction error, including when extrapolated from weaker to stronger models, and that for the optimally chosen teacher the infinite-data limit is consistent with supervised learning.","pith_inferences":["If the law transfers across data distributions, it suggests a domain-shift test: a teacher with low cross-entropy on its own distribution but high cross-entropy on the student's target distribution would be predicted to be a poor teacher unless the ratio $L_T / e_{LS}$ stays below the transition $d_1$, and this could be checked by distilling on a shifted target set while measuring the teacher's cr","An unstated corollary of the power-law decrease in optimal teacher loss is that the best teacher 'maturity'—the training-token count that minimizes student loss—could be precomputed from supervised scaling laws alone, before any distillation runs are performed.","The paper fixes the data-mixing weight at pure distillation and the temperature at one; a natural extension is to mix in the supervised next-token loss, since the paper's own contours imply a supervised-then-distillation schedule may be more efficient than either alone.","Because the large-compute crossover conclusion rests on the hand-imposed infinite-data equality, a relaxed model in which distillation's infinite-data limit could exceed the supervised one would change the recommendation to switch to supervised learning at large compute; this is directly testable by refitting Equation 8 without that constraint."],"forward_implications":["Student cross-entropy becomes a predictable quantity: the paper reports fits and extrapolations at roughly 1% relative error, so planners can estimate the value of a candidate teacher before spending compute.","Teacher size and teacher training tokens cease to be separate search dimensions, because both enter the law only through teacher cross-entropy $L_T$.","With a teacher that already exists, distillation is the more compute-efficient choice only below a student-size-dependent threshold; beyond that threshold supervised learning on the same budget wins.","If a teacher must be trained from scratch for a single student, supervised learning is generally preferable in the paper's controlled setting.","The capacity gap is governed by the ratio $L_T / e_{LS}$, so the optimal teacher loss for a given student follows a power-law decrease with student size, generalizing earlier observations about teacher scale."],"supporting_citations":[{"why":"Supplies the supervised scaling law and the fixed-M / IsoFLOP protocol used to fit Equation 1 and to define the student supervised cross-entropy $e_{LS}$.","marker":"Hoffmann et al. (2022)"},{"why":"Provides the supervised power-law form and early scaling-law evidence that motivates the student ability term in the distillation law.","marker":"Kaplan et al. (2020)"},{"why":"Defines the KL-divergence distillation objective that the paper uses as the pure distillation setting.","marker":"Hinton et al. (2015)"},{"why":"Justifies measuring pure distillation with no data-loss mixing and supplies the student-fidelity perspective the paper builds on.","marker":"Stanton et al. (2021)"},{"why":"Contributes the broken power-law functional form used for the teacher-loss transition in Equation 8.","marker":"Caballero et al. (2023)"},{"why":"Is the patient-teacher baseline whose apparent contradiction with the paper's large-compute conclusion is resolved in Appendix D.1.","marker":"Beyer et al. (2022)"},{"why":"Is the capacity-gap law that the paper shows is a special case of its own equation when teachers are compute-optimal.","marker":"Zhang et al. (2023a)"},{"why":"Supplies the C4 dataset on which all supervised and distillation runs in the controlled study are performed.","marker":"Raffel et al. (2020)"}],"fun_headline_variants":["Distillation scaling law predicts student performance from compute","One equation now predicts how much a student learns from a teacher","Distillation's compute-optimal recipes now captured in one law","A single scaling law governs teacher-student distillation","Distillation scaling law fits to 1% error, predicts optimal compute split"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes, by hand, that distillation with the optimally chosen teacher converges to the same cross-entropy as supervised learning at infinite data; the conclusion that supervised learning wins at large compute follows from that assumption, so the central practical recommendation would collapse if the equality is wrong.","fun_headline_variants_meta":{"raw":{"variants":["Distillation scaling law predicts student performance from compute","One equation now predicts how much a student learns from a teacher","Distillation's compute-optimal recipes now captured in one law","A single scaling law governs teacher-student distillation","Distillation scaling law fits to 1% error, predicts optimal compute split"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000588,"raw_usage":{"total_tokens":2720,"prompt_tokens":866,"completion_tokens":1854,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":1771}},"tokens_in":482,"tokens_out":1854,"duration_ms":13138,"temperature":1.0,"reasoning_tokens":1771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:28:44.263205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to refit Equation 8 without enforcing the Appendix E.6 infinite-data equality, then distill a fixed student size (for example 1B parameters) at token budgets past the predicted supervised crossover and compare the measured student cross-entropy to a supervised model trained on the same budget; if the distilled student stays below the supervised curve, or if the crossover compute does not grow with student size as predicted, the law is falsified.","supporting_citations":[{"cited_title":"(2022) do not see the teacher training distribution directly, whereas ours do","cited_arxiv_id":null,"evidence_quote":"Is the patient-teacher baseline whose apparent contradiction with the paper's large-compute conclusion is resolved in Appendix D.1."}],"review_version":1}