{"work":{"id":"b193541d-5853-4ea4-8e4b-8e4c08617eb6","openalex_id":"https://openalex.org/W7126034091","doi":"10.48550/arxiv.2601.20802","arxiv_id":"2601.20802","raw_key":null,"title":"Reinforcement Learning via Self-Distillation","authors":null,"authors_text":"Jonas H\\\"ubotter, Frederike L\\\"ubeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta","year":2026,"venue":"cs.LG","abstract":"Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO), which converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. In this way, SDPO leverages the model's ability to retrospectively identify its own mistakes in-context. Across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO improves sample efficiency and final accuracy over strong RLVR baselines. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. Finally, applying SDPO to individual questions at test time accelerates discovery on difficult binary-reward tasks, achieving the same discovery probability as best-of-k sampling or multi-turn conversations with 3x fewer attempts.","external_url":"https://arxiv.org/abs/2601.20802","cited_by_count":0,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2601.20802","created_at":"2026-05-09T05:45:22.361503+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"Reinforcement Learning via Self-Distillation","render_title":"Reinforcement Learning via Self-Distillation"},"hub":{"state":{"work_id":"b193541d-5853-4ea4-8e4b-8e4c08617eb6","tier":"super_hub","tier_reason":"100+ Pith inbound or 10,000+ external citations","pith_inbound_count":129,"external_cited_by_count":0,"distinct_field_count":7,"first_pith_cited_at":"2026-01-26T17:56:50+00:00","last_pith_cited_at":"2026-07-08T18:27:40+00:00","author_build_status":"needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-21T23:29:17.630961+00:00","tier_text":"super_hub"},"tier":"super_hub","role_counts":[{"context_role":"background","n":20},{"context_role":"method","n":3},{"context_role":"baseline","n":1}],"polarity_counts":[{"context_polarity":"background","n":19},{"context_polarity":"use_method","n":3},{"context_polarity":"baseline","n":1},{"context_polarity":"unclear","n":1}],"runs":{"ask_index":{"job_type":"ask_index","status":"succeeded","result":{"title":"Reinforcement Learning via Self-Distillation","claims":[{"claim_text":"Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO), which converts","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"vectors for steering generative large language model, 2024. URLhttps://arxiv.org/abs/2402. 01618. [29] A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. Solving quantitative reasoning problems with language models, 2022. URLhttps://arxiv.org/abs/ 2206.14858. [30] K. Li, J.-C. Pang, and Y. Yu. Rlvr training of llms does not improve thinking ability for general qa: Evaluation met","claim_type":"background","confidence":0.95,"evidence_strength":"citation_context"},{"claim_text":"GRPO[1] is a standard RLVR method that estimates token- level advantages from sparse scalar outcome rewards at the sequence level, serving as our primary RL baseline. OPSD[7] is an on-policy self-distillation method where a single model acts as both teacher and student, with the teacher conditioned on privileged information (e.g., verified reasoning traces) to provide dense per-token supervision.SDPO[8] extends self-distillation to the reinforcement learning with rich feedback setting, treating ","claim_type":"baseline","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"trains on output sequences sampled from the student,i.e., y∼p S(· |x tgt). Conversely, off-policy MSD trains on output sequences sampled from the teacher,i.e., y∼p T (· |x tgt, x∗,C) . To represent both the on-policy and off-policy settings, we introduce a generalized sampling distribution, q(·). In the on-policy setting,q(·)is directly equivalent to the student's sampling: q(·)≜p S(· |x tgt).(3) In the off-policy setting,q(·)is directly equivalent to the teacher's sampling: q(·)≜p T (· |x tgt, ","claim_type":"method","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"span locations, or span content are passed in. As in standard on-policy distillation it then evaluates token logits causally on the sampled prefixˆy