{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2025:DAATHSCBPPTSQC6F3DVN7HDE2A","short_pith_number":"pith:DAATHSCB","schema_version":"1.0","canonical_sha256":"180133c8417be7280bc5d8eadf9c64d018a7830496441949e808ed3313acc502","source":{"kind":"arxiv","id":"2506.14758","version":4},"attestation_state":"computed","paper":{"title":"Reasoning with Exploration: An Entropy Perspective","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Augmenting the RL advantage function with an entropy term improves LLM reasoning on Pass@K by encouraging longer exploratory chains.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Bo Dai, Daixuan Cheng, Furu Wei, Shaohan Huang, Wayne Xin Zhao, Xuekai Zhu, Zhenliang Zhang","submitted_at":"2025-06-17T17:54:03Z","abstract_excerpt":"Balancing exploration and exploitation is a central goal in reinforcement learning (RL). Despite recent advances in enhancing large language model (LLM) reasoning, most methods lean toward exploitation, and increasingly encounter performance plateaus. In this work, we revisit entropy -- a signal of exploration in RL -- and examine its relationship to exploratory reasoning in LLMs. Through empirical analysis, we uncover positive correlations between high-entropy regions and three types of exploratory reasoning actions: (1) pivotal tokens that determine or connect logical steps, (2) reflective a"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":true},"canonical_record":{"source":{"id":"2506.14758","kind":"arxiv","version":4},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.CL","submitted_at":"2025-06-17T17:54:03Z","cross_cats_sorted":[],"title_canon_sha256":"034df4168332dcc08d8cea9f107cc98a5b3c3e9ff5e87576f78fdb2d99b5faf0","abstract_canon_sha256":"5bec794faeb56b17b0c7c956ea9d005937306fb3a32dab0f0ba478a155f70bf3"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:48.850010Z","signature_b64":"AlVoGmJk0Rp2SqTaSaYQpVUYzM1JcEDqhMrBW2WhI3up9vuYolMeS5j8nV9aMUiT2CZsQFQdFTUNP9+bs9qBAA==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"180133c8417be7280bc5d8eadf9c64d018a7830496441949e808ed3313acc502","last_reissued_at":"2026-05-17T23:38:48.849568Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:48.849568Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Reasoning with Exploration: An Entropy Perspective","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Augmenting the RL advantage function with an entropy term improves LLM reasoning on Pass@K by encouraging longer exploratory chains.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Bo Dai, Daixuan Cheng, Furu Wei, Shaohan Huang, Wayne Xin Zhao, Xuekai Zhu, Zhenliang Zhang","submitted_at":"2025-06-17T17:54:03Z","abstract_excerpt":"Balancing exploration and exploitation is a central goal in reinforcement learning (RL). Despite recent advances in enhancing large language model (LLM) reasoning, most methods lean toward exploitation, and increasingly encounter performance plateaus. In this work, we revisit entropy -- a signal of exploration in RL -- and examine its relationship to exploratory reasoning in LLMs. Through empirical analysis, we uncover positive correlations between high-entropy regions and three types of exploratory reasoning actions: (1) pivotal tokens that determine or connect logical steps, (2) reflective a"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"our method achieves significant gains on the Pass@K metric -- an upper-bound estimator of LLM reasoning capabilities -- even when evaluated with extremely large K values, pushing the boundaries of LLM reasoning.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The observed positive correlations between high-entropy regions and beneficial exploratory actions (pivotal tokens, reflection, rare behaviors) will translate into improved downstream reasoning performance when the entropy term is added to the advantage function.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Augmenting the RL advantage with an entropy term promotes deeper LLM reasoning chains and raises Pass@K scores.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Augmenting the RL advantage function with an entropy term improves LLM reasoning on Pass@K by encouraging longer exploratory chains.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"a3afc3401fde5f9d05b268a7e41f9da27fefaeaff8b0a447c0741d38b7f98b00"},"source":{"id":"2506.14758","kind":"arxiv","version":4},"verdict":{"id":"18c9b05a-a8e2-40c4-b50c-66b778b5563c","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-16T06:16:18.830942Z","strongest_claim":"our method achieves significant gains on the Pass@K metric -- an upper-bound estimator of LLM reasoning capabilities -- even when evaluated with extremely large K values, pushing the boundaries of LLM reasoning.","one_line_summary":"Augmenting the RL advantage with an entropy term promotes deeper LLM reasoning chains and raises Pass@K scores.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The observed positive correlations between high-entropy regions and beneficial exploratory actions (pivotal tokens, reflection, rare behaviors) will translate into improved downstream reasoning performance when the entropy term is added to the advantage function.","pith_extraction_headline":"Augmenting the RL advantage function with an entropy term improves LLM reasoning on Pass@K by encouraging longer exploratory chains."},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":2,"snapshot_sha256":"7f64489350aa56258060107325552a54b65d97d7d4b28cdbdcd013d6d13f10be"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2506.14758","created_at":"2026-05-17T23:38:48.849633+00:00"},{"alias_kind":"arxiv_version","alias_value":"2506.14758v4","created_at":"2026-05-17T23:38:48.849633+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2506.14758","created_at":"2026-05-17T23:38:48.849633+00:00"},{"alias_kind":"pith_short_12","alias_value":"DAATHSCBPPTS","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"DAATHSCBPPTSQC6F","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"DAATHSCB","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":37,"internal_anchor_count":37,"sample":[{"citing_arxiv_id":"2507.15778","citing_title":"Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2510.08141","citing_title":"SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18864","citing_title":"SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18191","citing_title":"Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.19358","citing_title":"Taming the Thinker: Conditional Entropy Shaping for Adaptive LLM Reasoning","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2605.19461","citing_title":"Beyond Mode Collapse: Distribution Matching for Diverse Reasoning","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08401","citing_title":"AIPO: Learning to Reason from Active Interaction","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2509.20712","citing_title":"CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2509.23629","citing_title":"Emergent Slow Thinking in LLMs as Inverse Tree Freezing","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2509.26522","citing_title":"Entropy After </Think> for reasoning model early exiting","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2510.10150","citing_title":"Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2510.10649","citing_title":"Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2509.08827","citing_title":"A Survey of Reinforcement Learning for Large Reasoning Models","ref_index":84,"is_internal_anchor":true},{"citing_arxiv_id":"2511.05993","citing_title":"Revisiting Entropy in Reinforcement Learning for Large Reasoning Models","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2512.05591","citing_title":"Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2512.21815","citing_title":"High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2601.04731","citing_title":"Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Models","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2602.00513","citing_title":"Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2602.09782","citing_title":"Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2604.00860","citing_title":"Policy Improvement Reinforcement Learning","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2604.04894","citing_title":"Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12112","citing_title":"When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11491","citing_title":"Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2503.09567","citing_title":"Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models","ref_index":119,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08401","citing_title":"AIPO: Learning to Reason from Active Interaction","ref_index":9,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/DAATHSCBPPTSQC6F3DVN7HDE2A","json":"https://pith.science/pith/DAATHSCBPPTSQC6F3DVN7HDE2A.json","graph_json":"https://pith.science/api/pith-number/DAATHSCBPPTSQC6F3DVN7HDE2A/graph.json","events_json":"https://pith.science/api/pith-number/DAATHSCBPPTSQC6F3DVN7HDE2A/events.json","paper":"https://pith.science/paper/DAATHSCB"},"agent_actions":{"view_html":"https://pith.science/pith/DAATHSCBPPTSQC6F3DVN7HDE2A","download_json":"https://pith.science/pith/DAATHSCBPPTSQC6F3DVN7HDE2A.json","view_paper":"https://pith.science/paper/DAATHSCB","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2506.14758&json=true","fetch_graph":"https://pith.science/api/pith-number/DAATHSCBPPTSQC6F3DVN7HDE2A/graph.json","fetch_events":"https://pith.science/api/pith-number/DAATHSCBPPTSQC6F3DVN7HDE2A/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/DAATHSCBPPTSQC6F3DVN7HDE2A/action/timestamp_anchor","attest_storage":"https://pith.science/pith/DAATHSCBPPTSQC6F3DVN7HDE2A/action/storage_attestation","attest_author":"https://pith.science/pith/DAATHSCBPPTSQC6F3DVN7HDE2A/action/author_attestation","sign_citation":"https://pith.science/pith/DAATHSCBPPTSQC6F3DVN7HDE2A/action/citation_signature","submit_replication":"https://pith.science/pith/DAATHSCBPPTSQC6F3DVN7HDE2A/action/replication_record"}},"created_at":"2026-05-17T23:38:48.849633+00:00","updated_at":"2026-05-17T23:38:48.849633+00:00"}