{"id":"880e2852-88aa-4d4e-8af7-bcd5334c69b2","arxiv_id":"2505.17988","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Re-distilling a model's own RL-trained policy into 1K SFT samples reproduces RL accuracy at a fraction of the compute.","lead":"This paper shows that fine-tuning a small language model on about 1,000 solutions generated by its own reinforcement-learning-trained policy can match the full RL training result. The finding suggests a cheap way to compress expensive RL reasoning gains into a short supervised fine-tuning run.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unequal evaluation token budgets confound the flagship 're-distillation matches RL' claim on K&K: RD-rl-kk is evaluated at 8K tokens while the compared RL policy long-CoT-kk is capped at 4K.","rationale":"I read the paper as making two separable claims: (1) re-distillation empirically compresses an RL run into roughly 1K SFT samples, and (2) the linearized sample-effect theory explains why. The reader's weakest_assumption targets (2), but the theory being approximate would not, by itself, overturn the empirical result. A more immediate vulnerability sits inside claim (1) on the flagship K&K comparison: RD-rl-kk is evaluated with an 8K token budget while the RL policy long-CoT-kk, which it is said to match, is capped at 4K. This asymmetry can inflate RD-rl-kk on long responses and simultaneously suppress long-CoT-kk on truncated attempts, making the 'match' an artifact of evaluation protocol rather than learned capability. The comparison against DeepSeek-V3-0324 is internally fair because both are 8K, but the paper's central claim is about matching its own RL policy, and that comparison is not fair as reported. The numeric inconsistency between Figure 3 (78.8%) and Table 2 (0.820) for RD-rl-kk reinforces the need for a single, clearly specified evaluation configuration. I still credit the paper's broader evidence: the RD-rl-baseline control distinguishes re-distillation from simply changing the question set, the MATH and REASONING GYM results show the phenomenon is not confined to one dataset, and the cross-model RD-transfer experiment is a useful negative control. The concern is therefore not fatal, but it is a concrete condition that must be checked before the headline claim can be accepted. Since the reader already issued a CONDITIONAL verdict and my concern adds a specific condition rather than overturning the verdict, I leave the verdict unchanged.","tokens_in":21941,"tokens_out":6415,"duration_ms":52813,"concrete_test":"Run the K&K test evaluation of long-CoT-kk and RD-rl-kk under identical conditions: same greedy decoding, same max tokens (both 4096 and both 8192), same answer-extraction code, and report per-Nppl averaged accuracy with 95% CIs. If long-CoT-kk at 8192 remains at or below RD-rl-kk, the match survives; if it rises above by more than the current 3.2-point step-0 gap (or the Table 2 gap), the 'matched RL performance' claim must be requalified. Also reconcile Figure 3's 78.8% with Table 2's 0.820 for RD-rl-kk by stating which evaluation config each number uses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section E specifies max response tokens: RD-rl-kk gets 8K, while 'other trained models', including the RL policy long-CoT-kk it is claimed to match, get 4K. Table 2 lists RD-rl-kk at 0.820 and long-CoT-kk at 0.820; Figure 3 reports SFT-only RD-rl-kk at 78.8% vs 82.0% for the RL policy. If long-CoT-kk were evaluated at 8K, some truncated or zero-reward responses might become correct, raising its accuracy and erasing the apparent match. Conversely RD-rl-kk's 0.820 average may partly reflect a more permissive length budget rather than learned reasoning. The paper's own figure and table are also numerically inconsistent on RD-rl-kk (78.8% step-0 vs 0.820), and no confidence intervals are given for Table 2. This is the load-bearing defect for the central empirical claim on K&K: the '1K SFT samples surpass DeepSeek-V3-0324 and match RL' headline depends on a comparison in which the two sides are not evaluated under the same protocol. The theoretical linearization caveat in Appendix C is real but secondary: even if the theory is wrong, the empirical re-distillation result could still hold; the evaluation asymmetry, however, directly threatens the flagship empirical result.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies small-scale supervised fine-tuning (SFT, at most 1K examples) in R1-style reinforcement learning (RL). It first reports that SFT with about 1K examples substantially influences subsequent RL but that simple data-selection principles do not explain the observed patterns. The authors then propose a linearized analytical framework based on a per-sample 'sample effect' V(a,s,θ), defined so that the approximated reward-growth rate is an expectation of sample effects. They use this framework to argue that distillation policies should favor high-sample-effect responses and that RL training increases the dataset effect of RL-generated rollouts. Guided by this analysis, they introduce Re-distillation: sampling correct responses from an RL-trained policy and using them as SFT data for the base model. Experiments on K&K, MATH, and REASONING GYM with Qwen-2.5-1.5B and Llama-3.2-3B report that re-distilled models nearly match RL-trained policies using only SFT, that RD-rl-kk reaches 0.820 average accuracy on K&K and surpasses DeepSeek-V3-0324, and that re-distillation can merge two RL-trained policies. The paper also includes hyperparameter ablations, an interpolation study, and an analysis of position-wise token probabilities to support the proposed mechanism.","tokens_in":22308,"tokens_out":4781,"duration_ms":50319,"significance":"If the empirical result survives controlled evaluation, the claim that roughly 1K SFT samples can reproduce most of an RL-trained policy's performance is practically important and scientifically interesting. The paper has clear strengths: the RD-rl-baseline-kk control uses the same 1K questions with DeepSeek-R1 samples, providing a direct comparison that isolates the RL-trained policy as the data source; the method is tested on three datasets and two model families; the code is released; and the theoretical section is explicitly framed as hypothetical, with a limitations paragraph acknowledging the linearization assumption. At the same time, the theoretical contribution is largely definitional under the linearized SDE approximation, and the flagship K&K comparison is complicated by an evaluation-token asymmetry and by inconsistencies between the table and the text. The empirical claim therefore carries most of the weight, which is why experimental protocol issues are the central concern.","major_comments":[{"comment":"The headline K&K comparison is confounded by unequal evaluation token budgets. Section E states that RD-rl-kk is evaluated with Lmax=8192, while 'other trained models', including the RL policy long-CoT-kk it is claimed to match, are capped at 4096 tokens. Since the reward function in Section D.2 assigns zero reward to responses exceeding the length limit, a policy evaluated at 8K can receive credit for longer, slower reasoning that the same policy at 4K would lose. The apparent tie in Table 2 (0.820 vs 0.820) and the text's 78.8% vs 82.0% therefore do not establish that re-distillation matches RL performance. Please re-evaluate both models under identical max-token budgets, at minimum at 4K and 8K, and report the results with confidence intervals. This is load-bearing for the paper's central empirical claim.","section":"Section E and Table 2"},{"comment":"The theoretical explanation is partly definitional. The sample effect V(a,s,θ) is defined so that the approximated RL growth rate is exactly an expectation of sample effects (Eq. 3), and Theorem 5.2 then reduces, after linearization and noise-neglect, to an application of E[X^2] >= E[X]^2. The paper itself flags in Limitations that 'linearized sample effect may not be able to fully explain the non-linear effect in SFT and RL' and lists this as the core assumption in Appendix C. Because the theorem's conclusion follows from the definition under that assumption, the analytical framework does not by itself provide independent evidence for the mechanism; the empirical re-distillation results are the main support. I therefore recommend that the paper either soften the causal language around Theorem 5.2 or provide a direct empirical test that distinguishes sample effect from other correlates, such as response length or simple correctness.","section":"Section 5.3 and Eq. (43)-(50)"},{"comment":"The reported numbers for RD-rl-kk are inconsistent across the text and table. The text says 'RD-rl-kk is close to long-CoT-kk through SFT alone (78.8% vs 82.0%)', while Table 2 lists both RD-rl-kk and long-CoT-kk as 0.820. Similarly, Figure 3 appears to show step-0 SFT-only accuracy for RD-rl-kk, but the table appears to report a later or averaged value. Please clarify exactly which checkpoint or evaluation each number refers to and ensure the '1K SFT samples surpass DeepSeek-V3-0324' claim (0.820 vs 0.808) is based on the same protocol for both models. Without this clarification and without confidence intervals for the table averages, the headline result cannot be reliably assessed.","section":"Section 6.1, Figure 3, and Table 2"}],"minor_comments":[{"comment":"RD-transfer-gym is described as fine-tuning 'Llama-3.2-1.5B Instruct', but elsewhere the Llama experiments use Llama-3.2-3B; please correct this typo.","section":"Section A"},{"comment":"The sentence beginning 'For all Llama-3.2-3B based models...' is incomplete and grammatically garbled; it should specify the max response token setting for those models.","section":"Section E"},{"comment":"The interpolation study would be clearer if the direction of λ were stated consistently in the caption and text; currently the caption says 'interpolating from long-CoT-math initial policy to the 50-step checkpoint', while the text defines π(λθold + (1−λ)θ), which for λ=1 gives the initial policy and for λ=0 gives the checkpoint.","section":"Section F and Figure 6"},{"comment":"The footnote '†: We use Lmax = 8192 in these models' evaluation' does not specify whether it applies to RD-rl-baseline-kk as well as RD-rl-kk; please clarify the evaluation budget of every row.","section":"Table 2"},{"comment":"There is a typographical error in the corresponding author email ('wuji ee@tsinghua.edu.cn'); please fix the spacing.","section":"Title page"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the empirical core is real and worth engaging. This paper deserves a serious referee. The K&K headline, though, has a protocol problem that needs fixing.\n\nWhat's new: re-distillation—sampling SFT data from the same model's RL-trained policy—is a neat idea I haven't seen in the distillation literature. The sample-effect framework is a reasonable lens, even if more interpretation than proof. The control against DeepSeek-R1 distillation on the same questions is the right comparison, and it makes the effect look real. The result holds across three datasets and two base models, which is more than most papers of this type do.\n\nThe biggest soft spot is the K&K evaluation. Table 2 compares RD-rl-kk against long-CoT-kk, but RD-rl-kk is evaluated with 8K max tokens and long-CoT-kk with 4K. That is not the same protocol. The claim that RD-rl-kk 'matches' the RL policy at 0.820 needs to be re-run under identical budgets. It may survive—the RL policy was trained with a 4K cap, so extra budget may not help it—but the paper shouldn't ask the reader to take that on faith. There is also a numbers mismatch: Figure 3 says RD-rl-kk hits 78.8% at step 0 while Table 2 says 0.820. Those cannot both be right for the same model and test set.\n\nThe theory is honest but thin. Theorem 5.2's conclusion is baked into the definition of sample effect, and the paper's own caveat that linearization may not hold is accurate. I'd treat the theory as conceptual framing, not mechanism.\n\nMinor: Table 2 lacks confidence intervals, and the REASONING GYM task proportions are manually chosen, which introduces selection risk. Both are fixable.\n\nBottom line: the re-distillation phenomenon looks real and useful, especially on MATH and REASONING GYM where the evaluation is cleaner. The K&K confusion is a repair, not a rejection. This paper should go to review, with a request to equalize evaluation budgets and reconcile the numbers.","headline":"Re-distillation is a real and useful empirical finding, but the K&K 'matches RL' claim is undercut by unequal evaluation token budgets and a numeric inconsistency, and the theory is largely definitional.","tokens_in":22765,"tokens_out":5853,"would_cite":true,"duration_ms":44304,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the reasoning gains of R1-style RL are reproducible by fine-tuning on about one thousand responses sampled from the RL-trained policy, a procedure it calls re-distillation, and that this works because RL raises the…","keywords":["re-distillation","R1-style reinforcement learning","small-scale Supervised Fine-Tuning","sample effect","GRPO","reasoning generalization","policy distillation","LLM post-training"],"falsifier":"A direct test is to compute the linearized sample effect of rollout data from early and late checkpoints of a single RL run and compare with the actual SFT gains from distilling each checkpoint: the theory predicts late-checkpoint data loses effectiveness as dataset-effect growth decays, so if late-checkpoint re-distillation transfers undiminished, the explanation fails. A second observation, noted by the paper itself, is that the framework misorders long-CoT versus short-CoT SFT data on MATH, so any experiment where a predicted sample-effect ranking reverses observed SFT efficiency would also falsify the claim that sample effect carries the effect.","tokens_in":21758,"feed_emoji":"🧠","tokens_out":7924,"duration_ms":57349,"temperature":0.7,"pith_summary":"The paper tries to establish that the gains of R1-style reinforcement learning (RL) for LLM reasoning are not intrinsic to the RL loop: they can be transferred to a base model by supervised fine-tuning on about one thousand responses sampled from the converged RL policy. It calls this re-distillation. On the Knight & Knave puzzle, a 1.5B model fine-tuned on 1K re-distilled samples matches the RL policy's accuracy (reported as 78.8% versus 82.0% in one comparison, and 0.820 average in Table 2) and surpasses DeepSeek-V3-0324. The paper accounts for this with an analytic 'sample effect' that measures each training sample's contribution to test-accuracy growth, proving that the optimal distillation target over-weights high-effect samples and that RL raises the sample effect of its own outputs. If correct, this means expensive RL runs can be compressed into cheap, re-usable SFT datasets, and that SFT's poor sample efficiency is a property of the data, not a fixed limit of the method.","feed_headline":"RL training can be compressed into 1K supervised examples","feed_subtitle":"Fine-tuning on a model's own RL-trained responses matches full RL accuracy at a fraction of the compute.","key_machinery":"The carrying object is the sample effect, $V(a,s,\\theta)$—for a prompt-response pair $(a,s)$, the inner product of its policy-gradient direction with the test-set reward gradient. It lets the paper write the growth rate of test accuracy during SFT or RL as a weighted average of sample effects, ignoring gradient noise via an SDE/drift-term approximation. Theorem 5.1, whose proof follows the closed-form derivation pattern of DPO, gives the optimal distillation target as a policy whose log-odds are shifted proportionally to the sample-effect difference, so high-effect correct answers receive more probability mass. Theorem 5.2 shows the dataset effect of a policy trained by RL is bounded below by the square of its accuracy growth rate, which is why the RL policy's own outputs are unusually good SFT data. Re-distillation is the practical recipe that samples correct responses from the RL replay buffer and fine-tunes the base model on them.","core_discovery":"On the paper's own terms, the discovery is that a small supervised fine-tuning set drawn from an RL-trained policy can stand in for the entire RL run. On K&K, RD-rl-kk—1,000 correct responses sampled from the RL replay buffer at step 125—brings the base model to the same average accuracy as the RL policy (0.820, and 78.8% vs 82.0% in the step-0 comparison), beating DeepSeek-V3-0324's 0.808; on MATH, RD-sft-math with 496 samples matches the 50-step RL checkpoint (54.4% vs 56.0%) and the Instruct baseline (54.4% vs 55.0%); on REASONING GYM, re-distilled models keep most of the RL improvement while losing only about 1.5–1.7% accuracy on average. The explanation offered is that RL-trained policies have high output sample effect: Theorem 5.1 shows the optimal distillation target shifts probability mass toward correct responses with high sample effect, and Theorem 5.2 guarantees the dataset effect of an RL-trained policy grows whenever training accuracy grows, under the assumption that linearized sample effect explains the non-linear training dynamics.","pith_inferences":["If the linearized sample-effect story survives contact with larger models, sample effect could be computed cheaply and used to select SFT data before any RL run, turning re-distillation from a post-hoc recipe into a data-selection principle.","The success of re-distillation implies RL's role in R1-style training may be to generate a better target distribution rather than to provide a uniquely necessary optimization path; a testable extension is whether the same compression works for safety or instruction-following RL.","The paper's own failure to predict long-CoT vs short-CoT ordering on MATH suggests non-linear effects matter; a natural next experiment is measuring whether sample-effect rankings align better with observed SFT efficiency as model size grows.","Since cross-model transfer fails, a testable hypothesis for why is capacity: re-distilling Qwen samples into a larger Llama may recover the transfer that failed at 3B."],"forward_implications":["A single converged RL run can be compressed into roughly 1K supervised pairs, enabling transfer of the RL policy's reasoning ability to a base model by SFT alone.","Data quality for SFT should be judged by sample effect rather than human-perceived reasoning quality; on identical questions, re-distilled responses beat DeepSeek-R1-distilled ones (RD-rl-kk vs RD-rl-baseline-kk).","Re-distillation is model-specific; training Llama on Qwen-generated re-distilled data degrades markedly, so RL gains are tied to the source policy's own parameter trajectory.","RL policies trained for different goals can be merged by concatenating their re-distilled datasets, offering a cheap path to a Pareto frontier (RD-combined).","SFT's poor sample efficiency is not intrinsic; it reflects the sample effect of the data, so the analytic framework gives a principle for constructing more efficient SFT datasets."],"supporting_citations":[{"why":"Source of the R1-style RL recipe and the observation that curated SFT reasoning traces improve RL.","marker":"Guo et al. 2025"},{"why":"Evidence that 1K SFT samples substantially influence subsequent RL in Qwen and Llama models.","marker":"Gandhi et al. 2025"},{"why":"The influence-function approach that sample effect extends to non-converged, non-linear training settings.","marker":"Koh and Liang 2017"},{"why":"The neural-tangent-kernel linearization that justifies approximating RL dynamics by a drift term.","marker":"Jacot, Gabriel, and Hongler 2018"},{"why":"The DPO closed-form derivation pattern used to prove Theorem 5.1's optimal-policy form.","marker":"Rafailov et al. 2023"},{"why":"Provides the Knight and Knave logical reasoning dataset and its solver-generated solutions.","marker":"Xie et al. 2024"},{"why":"Provides the MATH dataset and its step-by-step solutions used in SFT and RL experiments.","marker":"Hendrycks et al. 2021"},{"why":"Provides REASONING GYM, the verifiable multi-task benchmark for the generalization experiment.","marker":"Stojanovski et al. 2025"},{"why":"DAPO, source of the GRPO variant (no KL, clipping) used in the RL implementation.","marker":"Yu et al. 2025"}],"fun_headline_variants":["RL training distilled into 1K fine-tuned examples","1K SFT samples replicate RL reasoning gains","Re-distillation matches RL with 1K samples","Small SFT from RL policy yields RL-level accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a linearized 'sample effect'—computed while ignoring gradient noise and nonlinear interactions—can explain how multi-step, non-linear LLM training improves accuracy; if nonlinear effects dominate in real SFT or RL, the theoretical prescriptions do not follow.","fun_headline_variants_meta":{"raw":{"variants":["RL training distilled into 1K fine-tuned examples","1K SFT samples replicate RL reasoning gains","Re-distillation matches RL with 1K samples","Small SFT from RL policy yields RL-level accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000347,"raw_usage":{"total_tokens":1944,"prompt_tokens":1031,"completion_tokens":913,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":850}},"tokens_in":647,"tokens_out":913,"duration_ms":9364,"temperature":1.0,"reasoning_tokens":850,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:36:37.997073+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to compute the linearized sample effect of rollout data from early and late checkpoints of a single RL run and compare with the actual SFT gains from distilling each checkpoint: the theory predicts late-checkpoint data loses effectiveness as dataset-effect growth decays, so if late-checkpoint re-distillation transfers undiminished, the explanation fails. A second observation, noted by the paper itself, is that the framework misorders long-CoT versus short-CoT SFT data on MATH, so any experiment where a predicted sample-effect ranking reverses observed SFT efficiency would also falsify the claim that sample effect carries the effect.","supporting_citations":[{"cited_title":"Neural tangent kernel: Convergence and generalization in neural networks","cited_arxiv_id":null,"evidence_quote":"The neural-tangent-kernel linearization that justifies approximating RL dynamics by a drift term."}],"review_version":1}