{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2025:MFIB4ZGCFJBF3UEPF3T2FITNBN","short_pith_number":"pith:MFIB4ZGC","schema_version":"1.0","canonical_sha256":"61501e64c22a425dd08f2ee7a2a26d0b79e76dd1293c56cebf08c0349ba48cd4","source":{"kind":"arxiv","id":"2504.14945","version":5},"attestation_state":"computed","paper":{"title":"Learning to Reason under Off-Policy Guidance","license":"http://creativecommons.org/licenses/by/4.0/","headline":"LUFFY mixes off-policy reasoning traces with on-policy rollouts to overcome the limits of standard RLVR in training reasoning models.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.LG","authors_text":"Ganqu Cui, Jianhao Yan, Xiaoye Qu, Yafu Li, Yu Cheng, Yue Zhang, Zhi Wang, Zican Hu","submitted_at":"2025-04-21T08:09:13Z","abstract_excerpt":"Recent advances in large reasoning models (LRMs) demonstrate that sophisticated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning with verifiable rewards~(\\textit{RLVR}). However, existing \\textit{RLVR} approaches are inherently ``on-policy'', limiting learning to a model's own outputs and failing to acquire reasoning abilities beyond its initial capabilities. To address this issue, we introduce \\textbf{LUFFY} (\\textbf{L}earning to reason \\textbf{U}nder o\\textbf{FF}-polic\\textbf{Y} guidance), a framework that augments \\textit{RLVR} with off-policy"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2504.14945","kind":"arxiv","version":5},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.LG","submitted_at":"2025-04-21T08:09:13Z","cross_cats_sorted":["cs.AI","cs.CL"],"title_canon_sha256":"a80d72f306e3f138b04b3428df5bee8762a2693c938769e3284fc45c99b69946","abstract_canon_sha256":"36476bf4f1c0c61dc9393470f17a1c08176e6eed4c404f139a466f0b6b250ab5"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:49.813136Z","signature_b64":"+XSUBs2nnRrFQaZXD454JsQghr4l2gZByQVSr10f5HIbNpolE5+nFE9ET8rf5FCti/W4YO0/lP/bD6izoPPoCg==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"61501e64c22a425dd08f2ee7a2a26d0b79e76dd1293c56cebf08c0349ba48cd4","last_reissued_at":"2026-05-17T23:38:49.812561Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:49.812561Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Learning to Reason under Off-Policy Guidance","license":"http://creativecommons.org/licenses/by/4.0/","headline":"LUFFY mixes off-policy reasoning traces with on-policy rollouts to overcome the limits of standard RLVR in training reasoning models.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.LG","authors_text":"Ganqu Cui, Jianhao Yan, Xiaoye Qu, Yafu Li, Yu Cheng, Yue Zhang, Zhi Wang, Zican Hu","submitted_at":"2025-04-21T08:09:13Z","abstract_excerpt":"Recent advances in large reasoning models (LRMs) demonstrate that sophisticated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning with verifiable rewards~(\\textit{RLVR}). However, existing \\textit{RLVR} approaches are inherently ``on-policy'', limiting learning to a model's own outputs and failing to acquire reasoning abilities beyond its initial capabilities. To address this issue, we introduce \\textbf{LUFFY} (\\textbf{L}earning to reason \\textbf{U}nder o\\textbf{FF}-polic\\textbf{Y} guidance), a framework that augments \\textit{RLVR} with off-policy"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Compared with previous RLVR methods, LUFFY achieves an over +6.4 average gain across six math benchmarks and an advantage of over +6.2 points in out-of-distribution tasks. Most significantly, we show that LUFFY successfully trains weak models in scenarios where on-policy RLVR completely fails.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That off-policy reasoning traces can be mixed with on-policy rollouts via regularized importance sampling without introducing harmful distribution shift or superficial imitation that would degrade the learned policy.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"LUFFY mixes off-policy reasoning traces into RLVR training via Mixed-Policy GRPO and regularized importance sampling, delivering over 6-point gains on math benchmarks and enabling training of weak models where on-policy RLVR fails.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"LUFFY mixes off-policy reasoning traces with on-policy rollouts to overcome the limits of standard RLVR in training reasoning models.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"be06d94251a82b5fe6f308751045d5477423fe95369f697ebf5102bab74a1d6f"},"source":{"id":"2504.14945","kind":"arxiv","version":5},"verdict":{"id":"9925b16a-9ca3-47ed-b0c7-8abeac542e55","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T23:13:37.696664Z","strongest_claim":"Compared with previous RLVR methods, LUFFY achieves an over +6.4 average gain across six math benchmarks and an advantage of over +6.2 points in out-of-distribution tasks. Most significantly, we show that LUFFY successfully trains weak models in scenarios where on-policy RLVR completely fails.","one_line_summary":"LUFFY mixes off-policy reasoning traces into RLVR training via Mixed-Policy GRPO and regularized importance sampling, delivering over 6-point gains on math benchmarks and enabling training of weak models where on-policy RLVR fails.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That off-policy reasoning traces can be mixed with on-policy rollouts via regularized importance sampling without introducing harmful distribution shift or superficial imitation that would degrade the learned policy.","pith_extraction_headline":"LUFFY mixes off-policy reasoning traces with on-policy rollouts to overcome the limits of standard RLVR in training reasoning models."},"references":{"count":65,"sample":[{"doi":"","year":2024,"title":"OpenAI o1 System Card","work_id":"68d3c334-0fc9-49e3-b7b0-a69afae933e2","ref_index":1,"cited_arxiv_id":"2412.16720","is_internal_anchor":true},{"doi":"","year":2025,"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","ref_index":2,"cited_arxiv_id":"2501.12948","is_internal_anchor":true},{"doi":"","year":2025,"title":"Kimi k1.5: Scaling Reinforcement Learning with LLMs","work_id":"bff96ab1-bd6a-4585-be23-74fdb51969c7","ref_index":3,"cited_arxiv_id":"2501.12599","is_internal_anchor":true},{"doi":"","year":2022,"title":"Chi, Quoc V Le, and Denny Zhou","work_id":"15a94dfd-ad8c-4ad9-9b36-e591ddf4950d","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient","work_id":"18772931-c0f8-4b45-bbb2-eb3f626eb2ee","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":65,"snapshot_sha256":"401a7fe1d67250ae59b59008c1a85145e85e42eb5baff34273cbe813d200b517","internal_anchors":18},"formal_canon":{"evidence_count":2,"snapshot_sha256":"ad06fc1706691937f033154995f9c2b191515b50e75296522bf52595e1055527"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2504.14945","created_at":"2026-05-17T23:38:49.812655+00:00"},{"alias_kind":"arxiv_version","alias_value":"2504.14945v5","created_at":"2026-05-17T23:38:49.812655+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2504.14945","created_at":"2026-05-17T23:38:49.812655+00:00"},{"alias_kind":"pith_short_12","alias_value":"MFIB4ZGCFJBF","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"MFIB4ZGCFJBF3UEP","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"MFIB4ZGC","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":28,"internal_anchor_count":28,"sample":[{"citing_arxiv_id":"2605.17734","citing_title":"Harnessing LLM Agents with Skill Programs","ref_index":38,"is_internal_anchor":true},{"citing_arxiv_id":"2605.19461","citing_title":"Beyond Mode Collapse: Distribution Matching for Diverse Reasoning","ref_index":38,"is_internal_anchor":true},{"citing_arxiv_id":"2508.00222","citing_title":"RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2508.07809","citing_title":"EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2508.06412","citing_title":"Sample-efficient LLM Optimization with Reset Replay","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2508.13755","citing_title":"Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2511.20814","citing_title":"SPHINX: A Synthetic Environment for Visual Perception and Reasoning","ref_index":63,"is_internal_anchor":true},{"citing_arxiv_id":"2512.11470","citing_title":"Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning","ref_index":43,"is_internal_anchor":true},{"citing_arxiv_id":"2601.06794","citing_title":"No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2603.11321","citing_title":"Hindsight-Anchored Policy Optimization: Turning Failure into Feedback in Sparse Reward Settings","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12652","citing_title":"Multi-Rollout On-Policy Distillation via Peer Successes and Failures","ref_index":55,"is_internal_anchor":true},{"citing_arxiv_id":"2605.13230","citing_title":"Teacher-Guided Policy Optimization for LLM Distillation","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2604.00626","citing_title":"A Survey of On-Policy Distillation for Large Language Models","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2604.03993","citing_title":"Can LLMs Learn to Reason Robustly under Noisy Supervision?","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12004","citing_title":"Learning Agentic Policy from Action Guidance","ref_index":68,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11636","citing_title":"Seir\\^enes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning","ref_index":33,"is_internal_anchor":true},{"citing_arxiv_id":"2604.23318","citing_title":"Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2604.18530","citing_title":"OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2604.14142","citing_title":"From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space","ref_index":65,"is_internal_anchor":true},{"citing_arxiv_id":"2604.13010","citing_title":"Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation","ref_index":36,"is_internal_anchor":true},{"citing_arxiv_id":"2604.11446","citing_title":"Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration","ref_index":37,"is_internal_anchor":true},{"citing_arxiv_id":"2604.09107","citing_title":"TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training","ref_index":43,"is_internal_anchor":true},{"citing_arxiv_id":"2604.08527","citing_title":"Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2604.13010","citing_title":"Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation","ref_index":36,"is_internal_anchor":true},{"citing_arxiv_id":"2605.07331","citing_title":"Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective","ref_index":7,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/MFIB4ZGCFJBF3UEPF3T2FITNBN","json":"https://pith.science/pith/MFIB4ZGCFJBF3UEPF3T2FITNBN.json","graph_json":"https://pith.science/api/pith-number/MFIB4ZGCFJBF3UEPF3T2FITNBN/graph.json","events_json":"https://pith.science/api/pith-number/MFIB4ZGCFJBF3UEPF3T2FITNBN/events.json","paper":"https://pith.science/paper/MFIB4ZGC"},"agent_actions":{"view_html":"https://pith.science/pith/MFIB4ZGCFJBF3UEPF3T2FITNBN","download_json":"https://pith.science/pith/MFIB4ZGCFJBF3UEPF3T2FITNBN.json","view_paper":"https://pith.science/paper/MFIB4ZGC","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2504.14945&json=true","fetch_graph":"https://pith.science/api/pith-number/MFIB4ZGCFJBF3UEPF3T2FITNBN/graph.json","fetch_events":"https://pith.science/api/pith-number/MFIB4ZGCFJBF3UEPF3T2FITNBN/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/MFIB4ZGCFJBF3UEPF3T2FITNBN/action/timestamp_anchor","attest_storage":"https://pith.science/pith/MFIB4ZGCFJBF3UEPF3T2FITNBN/action/storage_attestation","attest_author":"https://pith.science/pith/MFIB4ZGCFJBF3UEPF3T2FITNBN/action/author_attestation","sign_citation":"https://pith.science/pith/MFIB4ZGCFJBF3UEPF3T2FITNBN/action/citation_signature","submit_replication":"https://pith.science/pith/MFIB4ZGCFJBF3UEPF3T2FITNBN/action/replication_record"}},"created_at":"2026-05-17T23:38:49.812655+00:00","updated_at":"2026-05-17T23:38:49.812655+00:00"}