{"paper":{"title":"Learning to Reason under Off-Policy Guidance","license":"http://creativecommons.org/licenses/by/4.0/","headline":"LUFFY mixes off-policy reasoning traces with on-policy rollouts to overcome the limits of standard RLVR in training reasoning models.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.LG","authors_text":"Ganqu Cui, Jianhao Yan, Xiaoye Qu, Yafu Li, Yu Cheng, Yue Zhang, Zhi Wang, Zican Hu","submitted_at":"2025-04-21T08:09:13Z","abstract_excerpt":"Recent advances in large reasoning models (LRMs) demonstrate that sophisticated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning with verifiable rewards~(\\textit{RLVR}). However, existing \\textit{RLVR} approaches are inherently ``on-policy'', limiting learning to a model's own outputs and failing to acquire reasoning abilities beyond its initial capabilities. To address this issue, we introduce \\textbf{LUFFY} (\\textbf{L}earning to reason \\textbf{U}nder o\\textbf{FF}-polic\\textbf{Y} guidance), a framework that augments \\textit{RLVR} with off-policy"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Compared with previous RLVR methods, LUFFY achieves an over +6.4 average gain across six math benchmarks and an advantage of over +6.2 points in out-of-distribution tasks. Most significantly, we show that LUFFY successfully trains weak models in scenarios where on-policy RLVR completely fails.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That off-policy reasoning traces can be mixed with on-policy rollouts via regularized importance sampling without introducing harmful distribution shift or superficial imitation that would degrade the learned policy.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"LUFFY mixes off-policy reasoning traces into RLVR training via Mixed-Policy GRPO and regularized importance sampling, delivering over 6-point gains on math benchmarks and enabling training of weak models where on-policy RLVR fails.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"LUFFY mixes off-policy reasoning traces with on-policy rollouts to overcome the limits of standard RLVR in training reasoning models.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"be06d94251a82b5fe6f308751045d5477423fe95369f697ebf5102bab74a1d6f"},"source":{"id":"2504.14945","kind":"arxiv","version":5},"verdict":{"id":"9925b16a-9ca3-47ed-b0c7-8abeac542e55","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T23:13:37.696664Z","strongest_claim":"Compared with previous RLVR methods, LUFFY achieves an over +6.4 average gain across six math benchmarks and an advantage of over +6.2 points in out-of-distribution tasks. Most significantly, we show that LUFFY successfully trains weak models in scenarios where on-policy RLVR completely fails.","one_line_summary":"LUFFY mixes off-policy reasoning traces into RLVR training via Mixed-Policy GRPO and regularized importance sampling, delivering over 6-point gains on math benchmarks and enabling training of weak models where on-policy RLVR fails.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That off-policy reasoning traces can be mixed with on-policy rollouts via regularized importance sampling without introducing harmful distribution shift or superficial imitation that would degrade the learned policy.","pith_extraction_headline":"LUFFY mixes off-policy reasoning traces with on-policy rollouts to overcome the limits of standard RLVR in training reasoning models."},"references":{"count":65,"sample":[{"doi":"","year":2024,"title":"OpenAI o1 System Card","work_id":"68d3c334-0fc9-49e3-b7b0-a69afae933e2","ref_index":1,"cited_arxiv_id":"2412.16720","is_internal_anchor":true},{"doi":"","year":2025,"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","ref_index":2,"cited_arxiv_id":"2501.12948","is_internal_anchor":true},{"doi":"","year":2025,"title":"Kimi k1.5: Scaling Reinforcement Learning with LLMs","work_id":"bff96ab1-bd6a-4585-be23-74fdb51969c7","ref_index":3,"cited_arxiv_id":"2501.12599","is_internal_anchor":true},{"doi":"","year":2022,"title":"Chi, Quoc V Le, and Denny Zhou","work_id":"15a94dfd-ad8c-4ad9-9b36-e591ddf4950d","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient","work_id":"18772931-c0f8-4b45-bbb2-eb3f626eb2ee","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":65,"snapshot_sha256":"401a7fe1d67250ae59b59008c1a85145e85e42eb5baff34273cbe813d200b517","internal_anchors":18},"formal_canon":{"evidence_count":2,"snapshot_sha256":"ad06fc1706691937f033154995f9c2b191515b50e75296522bf52595e1055527"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}