{"work":{"id":"3ac87f3c-6dc4-492a-bbfb-8cdc05a15706","openalex_id":null,"doi":null,"arxiv_id":"2409.12917","raw_key":null,"title":"Training Language Models to Self-Correct via Reinforcement Learning","authors":null,"authors_text":"Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh","year":2024,"venue":"cs.LG","abstract":"Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to be largely ineffective in modern LLMs. Current methods for training self-correction typically depend on either multiple models, a more advanced model, or additional forms of supervision. To address these shortcomings, we develop a multi-turn online reinforcement learning (RL) approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. To build SCoRe, we first show that variants of supervised fine-tuning (SFT) on offline model-generated correction traces are often insufficient for instilling self-correction behavior. In particular, we observe that training via SFT falls prey to either a distribution mismatch between mistakes made by the data-collection policy and the model's own responses, or to behavior collapse, where learning implicitly prefers only a certain mode of correction behavior that is often not effective at self-correction on test problems. SCoRe addresses these challenges by training under the model's own distribution of self-generated correction traces and using appropriate regularization to steer the learning process into learning a self-correction behavior that is effective at test time as opposed to fitting high-reward responses for a given prompt. This regularization process includes an initial phase of multi-turn RL on a base model to generate a policy initialization that is less susceptible to collapse, followed by using a reward bonus to amplify self-correction. With Gemini 1.0 Pro and 1.5 Flash models, we find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on MATH and HumanEval.","external_url":"https://arxiv.org/abs/2409.12917","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-05-21T00:49:18.801227+00:00","pith_arxiv_id":"2409.12917","created_at":"2026-05-09T06:55:40.707976+00:00","updated_at":"2026-05-21T00:49:18.801227+00:00","title_quality_ok":true,"display_title":"Training Language Models to Self-Correct via Reinforcement Learning","render_title":"Training Language Models to Self-Correct via Reinforcement Learning"},"hub":{"state":{"work_id":"3ac87f3c-6dc4-492a-bbfb-8cdc05a15706","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":22,"external_cited_by_count":null,"distinct_field_count":5,"first_pith_cited_at":"2024-12-25T15:12:34+00:00","last_pith_cited_at":"2026-05-19T06:04:23+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-06-05T15:19:13.241413+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":7}],"polarity_counts":[{"context_polarity":"background","n":5},{"context_polarity":"support","n":1},{"context_polarity":"unclear","n":1}],"runs":{},"summary":{},"graph":{},"authors":[]}}