Machine Learning · arXiv:2610.22236 v2 · submitted 15 Oct 2026
Training-Free On-Policy Distillation
The paper says
“On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher.”
more of the abstract
We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. One forward pass over a few hundred rollouts provides them. We call this training-free on-policy distillation. What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. On Olmo-3-7B, a few GPU-hours of forward computation per specialist yield a generalist that keeps most of each specialist's gain on each of seven mathematics and code benchmarks.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Useful, honest engineering result in model merging; the loop-filtering finding is real, but the headline average is not format-matched, so the central claim is not yet established. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Table 2; §3.5; Table 5
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The headline claim depends on the Table 2 means (TF-OPD 61.1, math specialist 59.8, code specialist 53.6), but the Table 2 footnote states that released checkpoints are evaluated in bfloat16 while the fused model is evaluated in float16. Section 3.5 and Table 5 show that changing only the format moves the mathematics specialist on AIME25 from 43.3 to 39.2, a 4.2-point shift, which is more than three times the 1.3-point mean gap that supports the claim.
more of objection 1
The paper is transparent about formats and provides a matched-precision comparison for AIME25, but there is no matched-format version of the all-seven-benchmark mean, so the abstract's "matched precision" language cannot be read as covering the headline result. Please report benchmark-by-benchmark and mean scores with all four models evaluated in a common precision, ideally float16 for all of them and also bfloat16 for the fused model, and state whether the mean advantage over both specialists survives.
The paper says, at Table 2; Appendix C (Table 6)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.76):
“The three competition benchmarks have thirty problems each, so differences of a few points are within what they resolve. 4.3 Case study The fused model is not only a compromise between its sources: on some problems it succeeds where every sample of that domain’s specialist fails.”
Pith says: Objection 2 of 3 · major
The seven-benchmark mean in Table 2 is reported without any uncertainty quantification, and the authors themselves note in Section 4.2 that the three competition benchmarks have only thirty problems each, so differences of a few points are within what they resolve.
more of objection 2
The paired bootstrap intervals in Table 6 are wide, and the only AIME25 difference whose interval excludes zero is the loss of the all-answers model relative to the math specialist. To make the "more accurate on average" claim secure, please provide paired, problem-level bootstrap intervals for the mean difference between TF-OPD and each specialist over the seven benchmarks, or explicitly restrict the claim to the differences that are resolvable at this benchmark size.
The paper says, at §4.5 / Table 5
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The claim that student-written rollouts do not improve the closed form is based on a student built from only 128 problems per domain with a 16k-token limit (Appendix C), while the main model uses 256 problems per domain at the evaluation length limit.
more of objection 3
This is a reasonable exploratory ablation, but it does not rule out that a stronger or longer student could provide useful on-policy statistics. Please either soften the corresponding statement in Section 4.5 or add a variant with a student trained from the full 256-problem configuration before drawing the general conclusion.
5 minor objections
The paper says, at Abstract and §4.2
Pith says: minor objection 1 of 5
The abstract says the fused model is "on average, more accurate than either specialist alone" without stating that this mean is computed across formats, while the matched-precision statement is made only for AIME25; please make the scope of the matched-precision language explicit in the abstract.
The paper says, at Figure 4b
Pith says: minor objection 2 of 5
The bar labeled "References" in Figure 4b is not explained in the main text; Appendix C clarifies that it refers to statistics from reference solutions, but this should be stated where the figure first appears.
show 3 more
The paper says, at §3.4
Pith says: minor objection 3 of 5
The DEFLATE-based loop detector uses a single threshold (last 20,000 characters compressing below 5%), but the paper reports no sensitivity of the downstream AIME25 result to these two hyperparameters; a short sensitivity check would strengthen the claim that the filter, rather than the exact threshold, carries the result.
The paper says, at Algorithm 1 / §3.3
Pith says: minor objection 4 of 5
The decision to combine embeddings, normalization gains, and biases by task arithmetic is stated but not separately ablated; a sentence noting the measured impact of this choice, or an ablation, would make the method's scope clearer.
The paper says, at Appendix B
Pith says: minor objection 5 of 5
The report mentions that the fused update rounded to bfloat16 has cosine about 0.96 with the float32 update, but it does not report the corresponding benchmark accuracy impact; including that number would support the claim in Section 3.5 that rounding measurably distorts the fused model.
What would settle it
Rerun Table 2 with all four models (base, math specialist, code specialist, TF-OPD) in float16, and again in bfloat16, on all seven benchmarks under the same sampling and grading protocol, then recompute the seven-benchmark means with bootstrap 95% intervals over problems and over the four AIME/HMMT samples. If TF-OPD's mean exceeds both specialists in both formats by the full 1.3-point margin, the concern is resolved; if the margin shrinks within bootstrap noise or either specialist overtakes it in either format, the "more accurate on average" claim must be qualified to a specific precision setting or dropped.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Two things to know. First, the real contribution is the loop-filtering result: a few repetition-loop rollouts carry a large share of the math specialist's tokens, and removing them lifts AIME25 from task-arithmetic level to near-specialist level. That is a concrete, reproducible failure-mode analysis people in this subfield will need to deal with. Second, the headline claim — fused model on average more accurate than either specialist — rests on a comparison that is not format-matched: Table 2 scores the released specialists in bfloat16 and the fused model in float16, and Table 5 shows the math specialist alone drops 4.2 points on AIME25 when moved to float16. The mean advantage is 1.3 points. So the abstract's strongest sentence is not supported on matched terms. The paper is fully transparent about formats; the problem is that the known format effect is larger than the effect being claimed.
What is good: the derivation in Equations (1)–(3) and Appendix A is clean and correctly credits RegMean. The ablations are informative, especially the ones separating loop removal from correctness filtering, student rollouts, and reweighting. The compression-based loop detector is concrete, the on-policy statistics story is well motivated, and the qualitative trajectory evidence is a nice touch. The authors also explicitly acknowledge that the 30-problem benchmarks resolve only a few points, so that limitation is not hidden.
The soft spots beyond the precision mismatch: no confidence intervals on the Table 2 means, no released code or checkpoints, and only one model family tested. There is also no RegMean-with-corpus-statistics baseline, so the specific benefit of on-policy rollouts over the original method is asserted but not isolated. None of these are fatal, but together they make the central empirical claim provisional.
This paper deserves a serious referee. The derivation is sound, the loop-filtering result is novel and credible, and the method is cheap enough that others will try it. A revision should evaluate all four models in one common precision, add intervals on the headline mean, release artifacts, and add a corpus-statistics RegMean baseline. For the right reader — anyone working on merging or distilling RL-tuned specialists — this is worth their time.
The strongest objection
The central claim — "on average, more accurate than either specialist alone" — rests on the Table 2 means: TF-OPD 61.1, math specialist 59.8, code specialist 53.6. But the Table 2 footnote says released checkpoints are evaluated in bfloat16 and the fused model in float16, and Section 3.5 explicitly says "a format change by itself shifts measured accuracy." Table 5 quantifies exactly this: the math specialist scores 43.3 on AIME25 in bfloat16 and 39.2 in float16, a 4.2-point shift. The claimed mean advantage over the math specialist is only 1.3 points. So the headline comparison is not format-matched, and the known format effect is larger than the effect being claimed. The abstract's "matched precision" language is used only for the AIME25 comparison; there is no matched-format version of the all-seven-benchmark mean. If the math specialist loses comparable points on the other math benchmarks when moved to float16, or the code specialist does, the mean gap could shrink, disappear, or reverse. The paper is transparent about formats, and this is not a dispute about the derivation in Equations (1)-(3); it is a dispute about whether the single number that supports the strongest claim is measured on comparable terms. As it stands, the headline claim is not secure until all four models are compared in a common precision.
I read the paper in good faith: the layer-wise least-squares derivation, the loop-filtering analysis, and the ablations are careful and honestly presented, and the authors acknowledge that the thirty-problem benchmarks resolve only a few points and that precision shifts measured accuracy. Those acknowledgements are real credit. My concern is narrower and more concrete than the reader's weakest assumption. The reader emphasized that per-layer fits need not compose into a globally close autoregressive model; that is a legitimate theoretical risk, but the paper already provides downstream autoregressive evaluations, including student-rollout variants, so the transfer question is at least empirically probed. The unresolved issue is that the headline number itself is not measured on matched terms: Table 2 compares bfloat16 specialists against a float16 fused model, and the paper's own Table 5 shows a 4.2-point format shift on AIME25, larger than the 1.3-point mean advantage being claimed. That makes the strongest claim fragile in a way that no amount of per-layer theory can fix. The concrete test is straightforward and cheap: evaluate all models in both precisions on all seven benchmarks and recompute the means. Because the reader's CONDITIONAL verdict already captures the need for stronger statistical support, I recommend no verdict change, but my reason partially differs from the reader's: the format asymmetry is the most load-bearing specific defect, alongside the absence of confidence intervals on the main mean.
The referee's report
If the headline empirical claim holds, the paper is a useful contribution: it gives a clean derivation of a layer-wise least-squares fusion, connects it explicitly to RegMean and task arithmetic, and provides an informative diagnostic analysis of which rollouts matter. The derivation in Equations (1)-(3) and Appendix A is sound, the propositions in Section 3.1 are correctly proved and clarify when the closed form reduces to averaging or reproduces a dominant teacher, and the authors are unusually transparent about evaluation formats, benchmark resolution, and compute cost. The finding that loop-truncated answers are few but token-heavy, and that removing them is what recovers AIME25 performance, is specific and falsifiable. The main unresolved issue is that the central quantitative claim, "on average more accurate than either specialist," is measured on non-matched precision terms, and the known format effect is larger than the claimed mean gap. The per-layer-to-global transfer concern identified by the stress-test is real but is honestly confronted by the teacher-forced agreement diagnostic in Table 8; I do not treat it as a fatal flaw.
Pith's reading of the paper (a machine's summary, not the authors')
Turning separately tuned specialist models — a mathematics champion and a coding champion grown from the same base checkpoint — into one generalist normally costs rollouts, teacher passes, and gradient updates per teacher. The paper's claim is that this training step can be skipped: for specialists sharing a base model, layer-wise distillation is an exact least-squares problem whose solution needs only the second moments of each specialist's layer inputs. Those moments are gathered by letting each specialist read a few hundred of its own filtered answers in one teacher-forced forward pass, and every layer of the fused model is then obtained by a closed-form solve, with no gradient computed. On Olmo 3 7B the fused generalist keeps most of each specialist's gain on each of seven mathematics and code benchmarks and posts a higher mean accuracy (61.1) than either specialist (59.8 and 53.6); the authors add that the three competition benchmarks have only thirty problems, so differences of a few points are within what they resolve. The finding that carries the method is that the content of the rollouts, not the solver, decides hard problems: a few repetition-loop answers dominate the token count, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's.
What carries the argument. The central object is the per-layer closed-form least-squares solve, Equation (2): the fused update to a linear layer is a second-moment-weighted combination of the teacher updates, $\Delta^\star = (\sum_d \Delta_d \tilde G_d)(\sum_d \tilde G_d)^{-1}$, where $\tilde G_d = w_d[\alpha G_d + (1-\alpha)\mathrm{diag}(G_d)]$ with $\alpha = 0.9$ adds per-coordinate shrinkage so the system is well conditioned. The Gram matrices $G_d$ come from forward hooks during one teacher-forced pass in which each specialist reads its own filtered answers, so the statistics describe the layer inputs the model actually sees while reasoning. Three properties of the solution (Proposition 3.1) locate where teachers can conflict: identical second moments reduce the rule to weight averaging, a dominant teacher is recovered as its weight grows, and orthogonal input ranges let each teacher be reproduced exactly on its own inputs, so interference is confined to shared input directions. A per-layer diagnostic $\varepsilon_d$ reports the share of each teacher's update the fused layers fail to reproduce, and the rollout filter (repetition loops detected by DEFLATE compressibility, plus test-based grading of code answers) supplies the statistics on which the solve depends.
Core claim. Stated as a fair reader would: when several teachers are post-trained from one base checkpoint, the distillation objective restricted to a single linear layer is quadratic, and its minimizer is the RegMean closed form written for weight updates, $$W^\star = W_b + \Big(\textstyle\sum_d \Delta_d \tilde G_d\Big)\Big(\textstyle\sum_d \tilde G_d\Big)^{-1},$$ with $\tilde G_d = w_d[\alpha G_d + (1-\alpha)\mathrm{diag}(G_d)]$ for conditioning. The paper's contribution is to feed that closed form statistics gathered the way on-policy distillation gathers data — from the specialists' own generated answers, read once under teacher forcing with forward hooks accumulating a Gram matrix per layer group — and to show that on Olmo 3 7B the result needs no gradient step yet keeps most of each specialist's gain on five mathematics and two code benchmarks and is on average more accurate than either source. Two further findings carry the argument. What the rollouts contain matters more than the solver: with unfiltered answers the closed form is no better than task arithmetic on AIME25, while removing the few looping answers alone brings it level with the mathematics specialist scored in the same format, and keeping only correct answers changes nothing further. A close token-level fit is not what protects hard problems: the unfiltered fused model already follows the specialist almost token for token under teacher forcing, so hard problems are decided by which inputs the statistics weigh, a limitation the paper itself documents.
The premise it rests on. Matching each specialist, layer by layer, on teacher-forced inputs with second moments from filtered rollouts transfers to the fused model's own autoregressive generation — the paper itself shows a model can follow a specialist almost token for token under teacher forcing while still losing hard problems, so the transfer is empirical, not guaranteed by Equation (2).
If the paper is right (Pith's inferences, not the authors')
Adding a further specialist costs only its rollouts and one more term in each layer's sums; nothing is retrained, so the cost of consolidation grows with forward passes, not gradient updates.
On hard competition problems the fused model can match the domain specialist in the same evaluation format once looping answers are filtered, and can solve problems that every sample of the domain specialist fails.
Because a near-token-perfect teacher-forced fit can coexist with lost hard problems, mean accuracy over easier benchmarks can hide a fusion failure; the content of the rollouts, not the solver, is the decisive choice.
Training-based on-policy distillation remains the tool when the teachers differ in architecture or size from the student, since the closed form assumes a shared base model and architecture.
Numerical format is part of the method: fused models are stored in 32-bit floats and evaluated in float16, because rounding the fused update to the bfloat16 release format measurably distorts it.
A testable boundary follows from the paper's own Proposition 3.1: the more similar two specialists' input distributions, the more the closed form degenerates toward weight averaging rather than task arithmetic, so the method should help most when specialists act on largely disjoint input directions — the paper's math/code pair fits that description but does not isolate it.
The centrality of rollout selection suggests that curation, not the merge rule, is the productive surface for weight-space fusion of RL-tuned reasoners; the same loop filter and answer-grading pass could plausibly dominate other data-aware merging recipes, a comparison the paper does not run.
Scaling the recipe to more than two teachers or to another base family is the natural unrun extension: the paper reports a single 7B pair, and nothing in the derivation bounds how the per-layer fit degrades as further teachers or longer trajectories crowd the Gram matrices.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. One forward pass over a few hundred rollouts provides them. We call this training-free on-policy distillation. What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. On Olmo-3-7B, a few GPU-hours of forward computation per specialist yield a generalist that keeps most of each specialist's gain on each of seven mathematics and code benchmarks.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-16
The authors' main claim, as the classifier picked it out 1.00
“We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.61 · human participants 0.02 · controlled comparison 0.72 · randomized assignment 0.02 · states quantitative result 0.75 · compares with prior results 0.90 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. | background 1.00 | 0.10 | 0.05 | asserted 1.00 |
| We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. | contribution 0.55 | 0.93 | 0.84 | asserted 1.00 |
| When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. | method 0.92 | 0.77 | 0.14 | asserted 1.00 |
| One forward pass over a few hundred rollouts provides them. | method 0.99 | 0.76 | 0.05 | asserted 0.98 |
| We call this training-free on-policy distillation (\method). | contribution 0.68 | 0.64 | 0.09 | asserted 0.67 |
| What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. | result 0.90 | 0.90 | 0.11 | asserted 1.00 |
| On Olmo~3 7B, a few GPU-hours of forward computation per specialist yield a generalist that keeps most of each specialist's gain on each of seven mathematics and code benchmarks. | result 1.00 | 0.93 | 0.23 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.77
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 5,687 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (6)
Figures are the authors', reproduced from the paper.
Cited by 6 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Keystone: Judging Previews at Key Steps for Test-Time Search in Diffusion Models cs.AI · Nov 2026
- Keel: A Frontier Model Teaches a Vision-Language Model by Rewriting Its Weights cs.CV · Nov 2026
- BaseRSI: Recursive Self-Improvement of a Base Vision-Language Model via Training-Free Harness Internalization cs.CV · Nov 2026
- Vanguard: Knowing When to Trust the Draft in Speculative Vision-Language Agents cs.AI · Nov 2026
- AudioRSI: Recursively Self-Improving Audio Reasoners cs.SD · Oct 2026
- Replay for LLM Continual Learning with Multi-Teacher On-Policy Distillation cs.LG · Nov 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment