{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2025:QAEGPR6GQVUOWUPBXPUKW3PMDG","short_pith_number":"pith:QAEGPR6G","schema_version":"1.0","canonical_sha256":"800867c7c68568eb51e1bbe8ab6dec19ad280eec8e7f7d09d5ee95de3cb8e144","source":{"kind":"arxiv","id":"2511.20347","version":2},"attestation_state":"computed","paper":{"title":"Soft Adaptive Policy Optimization","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"A smooth temperature-controlled gate replaces hard clipping to stabilize reinforcement learning updates for language models.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.LG","authors_text":"An Yang, Bowen Yu, Chang Gao, Chujie Zheng, Jingren Zhou, Junyang Lin, Kai Dang, Shixuan Liu, Shuai Bai, Xiong-Hui Chen","submitted_at":"2025-11-25T14:25:19Z","abstract_excerpt":"Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of large language models (LLMs), yet stable and performant policy optimization remains challenging. Token-level importance ratios often exhibit high variance-a phenomenon exacerbated in Mixture-of-Experts models-leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2511.20347","kind":"arxiv","version":2},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.LG","submitted_at":"2025-11-25T14:25:19Z","cross_cats_sorted":["cs.AI","cs.CL"],"title_canon_sha256":"9dbd102b340aee7e9177a9024622d6d374cd500709201ceaa0298a3715c1ed8d","abstract_canon_sha256":"87ba5eef28f6f15dd14bb0c369fff2172ee7a06436f5bd193fe0f7ecba7898a4"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:53.165364Z","signature_b64":"mQC1FW5SvwD1Unm+ynwBV9eqK/APwXngGDYuQeLvMmoJhSxbMo185xl8X//I4rcY8r9v6tQ9vILXZs1R126KBQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"800867c7c68568eb51e1bbe8ab6dec19ad280eec8e7f7d09d5ee95de3cb8e144","last_reissued_at":"2026-05-17T23:38:53.164767Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:53.164767Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Soft Adaptive Policy Optimization","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"A smooth temperature-controlled gate replaces hard clipping to stabilize reinforcement learning updates for language models.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.LG","authors_text":"An Yang, Bowen Yu, Chang Gao, Chujie Zheng, Jingren Zhou, Junyang Lin, Kai Dang, Shixuan Liu, Shuai Bai, Xiong-Hui Chen","submitted_at":"2025-11-25T14:25:19Z","abstract_excerpt":"Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of large language models (LLMs), yet stable and performant policy optimization remains challenging. Token-level importance ratios often exhibit high variance-a phenomenon exacerbated in Mixture-of-Experts models-leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces "},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the smooth temperature-controlled gate selectively attenuates only harmful off-policy signals without suppressing useful learning gradients or introducing new instabilities that hard clipping avoided.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"SAPO introduces smooth adaptive gating to replace hard clipping in token- and sequence-level policy optimization for more stable LLM reinforcement learning.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"A smooth temperature-controlled gate replaces hard clipping to stabilize reinforcement learning updates for language models.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"fb07efa5fb8857581e125b44350b58b7001a59c3802c6c5dd0d2da4c6721c05a"},"source":{"id":"2511.20347","kind":"arxiv","version":2},"verdict":{"id":"0483695a-675c-4f01-bace-4c1b5f8b66dc","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T07:10:00.308877Z","strongest_claim":"Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes.","one_line_summary":"SAPO introduces smooth adaptive gating to replace hard clipping in token- and sequence-level policy optimization for more stable LLM reinforcement learning.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the smooth temperature-controlled gate selectively attenuates only harmful off-policy signals without suppressing useful learning gradients or introducing new instabilities that hard clipping avoided.","pith_extraction_headline":"A smooth temperature-controlled gate replaces hard clipping to stabilize reinforcement learning updates for language models."},"references":{"count":12,"sample":[{"doi":"","year":2025,"title":"Aime problems and solutions","work_id":"e9a59051-7b00-4ffc-902e-f0059e82fd2a","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2023,"title":"The sufficiency of off-policyness and soft clipping: Ppo is still insufficient according to an off-policy measure","work_id":"0993a20c-92b1-4188-991b-f800b41bc179","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","ref_index":3,"cited_arxiv_id":"2501.12948","is_internal_anchor":true},{"doi":"","year":2025,"title":"HMMT . Hmmt 2025. https://www.hmmt.org, 2025","work_id":"abbec9a8-4f45-42e0-bdfa-25286b96179f","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code","work_id":"ea9e51ce-1e75-4182-92d8-4d25f70d2ee4","ref_index":5,"cited_arxiv_id":"2403.07974","is_internal_anchor":true}],"resolved_work":12,"snapshot_sha256":"a08791fa11792a44147a60fbf2d95bf39d519569ca1e5fbfb604bfc4f91c6de6","internal_anchors":5},"formal_canon":{"evidence_count":2,"snapshot_sha256":"51bfdc33d4146af665e0611bc45d8f22658a7e91fff3d34f4945495e0036f036"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2511.20347","created_at":"2026-05-17T23:38:53.164846+00:00"},{"alias_kind":"arxiv_version","alias_value":"2511.20347v2","created_at":"2026-05-17T23:38:53.164846+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2511.20347","created_at":"2026-05-17T23:38:53.164846+00:00"},{"alias_kind":"pith_short_12","alias_value":"QAEGPR6GQVUO","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"QAEGPR6GQVUOWUPB","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"QAEGPR6G","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":40,"internal_anchor_count":40,"sample":[{"citing_arxiv_id":"2504.12501","citing_title":"Reinforcement Learning from Human Feedback","ref_index":141,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18141","citing_title":"A Brief Overview: On-Policy Self-Distillation In Large Language Models","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2605.22703","citing_title":"Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2602.04663","citing_title":"Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2605.20865","citing_title":"Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2605.21467","citing_title":"DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards","ref_index":52,"is_internal_anchor":true},{"citing_arxiv_id":"2605.21235","citing_title":"LamPO: A Lambda Style Policy Optimization for Reasoning Language Models","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08879","citing_title":"Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT","ref_index":28,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12969","citing_title":"Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16714","citing_title":"GRID: Graph Representation of Intelligence Data for Security Text Knowledge Graph Construction","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18141","citing_title":"A Brief Overview: On-Policy Self-Distillation In Large Language Models","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2604.27859","citing_title":"Rethinking Agentic Reinforcement Learning In Large Language Models","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2601.02954","citing_title":"The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2602.09782","citing_title":"Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2602.15620","citing_title":"STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2603.15432","citing_title":"Gym-V: A Unified Vision Environment System for Agentic Vision Research","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11775","citing_title":"Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control","ref_index":62,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11739","citing_title":"Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation","ref_index":65,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12969","citing_title":"Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11775","citing_title":"Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2604.10500","citing_title":"Visual Enhanced Depth Scaling for Multimodal Latent Reasoning","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11739","citing_title":"Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation","ref_index":65,"is_internal_anchor":true},{"citing_arxiv_id":"2604.28020","citing_title":"Cost-Aware Learning","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2604.27859","citing_title":"Rethinking Agentic Reinforcement Learning In Large Language Models","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08862","citing_title":"BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning","ref_index":6,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/QAEGPR6GQVUOWUPBXPUKW3PMDG","json":"https://pith.science/pith/QAEGPR6GQVUOWUPBXPUKW3PMDG.json","graph_json":"https://pith.science/api/pith-number/QAEGPR6GQVUOWUPBXPUKW3PMDG/graph.json","events_json":"https://pith.science/api/pith-number/QAEGPR6GQVUOWUPBXPUKW3PMDG/events.json","paper":"https://pith.science/paper/QAEGPR6G"},"agent_actions":{"view_html":"https://pith.science/pith/QAEGPR6GQVUOWUPBXPUKW3PMDG","download_json":"https://pith.science/pith/QAEGPR6GQVUOWUPBXPUKW3PMDG.json","view_paper":"https://pith.science/paper/QAEGPR6G","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2511.20347&json=true","fetch_graph":"https://pith.science/api/pith-number/QAEGPR6GQVUOWUPBXPUKW3PMDG/graph.json","fetch_events":"https://pith.science/api/pith-number/QAEGPR6GQVUOWUPBXPUKW3PMDG/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/QAEGPR6GQVUOWUPBXPUKW3PMDG/action/timestamp_anchor","attest_storage":"https://pith.science/pith/QAEGPR6GQVUOWUPBXPUKW3PMDG/action/storage_attestation","attest_author":"https://pith.science/pith/QAEGPR6GQVUOWUPBXPUKW3PMDG/action/author_attestation","sign_citation":"https://pith.science/pith/QAEGPR6GQVUOWUPBXPUKW3PMDG/action/citation_signature","submit_replication":"https://pith.science/pith/QAEGPR6GQVUOWUPBXPUKW3PMDG/action/replication_record"}},"created_at":"2026-05-17T23:38:53.164846+00:00","updated_at":"2026-05-17T23:38:53.164846+00:00"}