Pith. sign in

REVIEW 3 major objections 4 minor 71 references

BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reinforcement learning with verifiable rewards prunes semantically distinct reasoning branches in LLMs, not just stylistic variation, and this trade-off drives sample efficiency at the cost of diversity.

desk verdict A solid, honest measurement study of RLVR entropy collapse—new enough to referee, but the semantic-vs-syntactic conclusion is not yet nailed down because the comparison lacks surface-form controls. read the letter →

arxiv 2608.02867 v1 pith:JDS6EKHC submitted 2026-08-03 cs.CL cs.AI

Soumadeep Saha , Krish Sharma , Akshay Chaturvedi , Nicholas Asher This is my paper · ORCID
classification cs.CLcs.AI
keywords reinforcementlearningwithverifiablerewardsRLVRpolicyentropycollapsesemanticbranchingBODHI-TREEcandidatepreferencetest-timeexplorationmathematicalreasoningdiversity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether reinforcement learning with verifiable rewards (RLVR) expands an LLM's reasoning ability or merely makes it sample more efficiently. The authors build a tree of semantically equivalent reasoning steps (BODHI-TREE) from thousands of correct AIME solutions, separating stylistic variation from genuine inferential branching. Using this tree plus maze-solving probes, they show that RLVR-trained models have significantly lower branching entropy: they prefer fewer continuations, and the collapse is stronger when the alternatives are semantically distinct rather than merely syntactic variants. The paper concludes that RLVR works by constricting the space of continuations, improving constraint adherence and backtracking at the cost of genuine trajectory diversity.

What carries the argument

The BODHI-TREE, a prefix tree whose nodes are sets of semantically equivalent mathematical statements (merged by an LLM-as-a-judge matching function), and the candidate preference entropy (CPE) computed at branch nodes. The tree lets the authors separate syntactic variation (two continuations that say the same thing differently) from semantic variation (continuations that use different mathematical approaches), and CPE quantifies how strongly a model prefers one continuation over another at a given branch.

What would settle it

If human mathematicians re-label a sample of BODHI-TREE branch points and the CPE drop between distilled and RLVR models disappears or reverses, the semantic-collapse claim falls. A more mechanical version: rerun the experiment on a domain where semantic equivalence is decidable (e.g., rewritten symbolic algebra expressions) and test whether RLVR still reduces branching at provably distinct nodes.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the policy entropy collapse observed in RLVR-trained LLMs is not merely syntactic. By segmenting correct math traces into reasoning nodes and merging semantically equivalent statements via an LLM judge, the authors construct BODHI-TREEs in which each branching node offers distinct continuations. They define candidate preference entropy (CPE) at these branch points and find that RLVR-trained models show a statistically significant drop in CPE relative to distilled models across four model families, with the drop larger for semantically distinct continuations than for syntactic variants. Maze experiments confirm reduced test-time exploration, while also showing that RLVR improves the model's ability to avoid wall collisions and to backtrack out of dead-ends. The authors interpret this as evidence that RLVR's sample-efficiency gains come from pruning verifier-invalid continuations and ossifying trajectory preferences, at the cost of semantic rollout diversity.

Load-bearing premise

The BODHI-TREE construction assumes that the LLM judge used for semantic equivalence correctly decides when two reasoning steps are mathematically equivalent; the authors validate it only against other LLMs (Cohen's kappa ~0.82-0.84) and note they could not perform rigorous human evaluation, so the 'semantic' branches may partly reflect judge artifacts.

Editorial extensions

If this is right

  • If RLVR inherently prunes semantically distinct reasoning paths, then techniques that sample many rollouts at test time, such as self-consistency or tree-of-thought, will see diminishing returns on RLVR-trained models unless the training objective includes a semantic-diversity term.
  • The paper's maze results imply that RLVR's pass@1 gains come largely from learning environmental constraints (avoiding invalid moves) and better backtracking, not from discovering novel solution strategies.
  • Monitoring CPE on BODHI-TREEs during RL training could serve as an early-warning signal for reasoning-diversity collapse, before it shows up in pass@k metrics.
  • Long-CoT distillation preserves more semantic branching than RLVR on the same base model, suggesting that distillation and RLVR occupy different points on a diversity-versus-validity trade-off.
  • If the effect generalizes beyond AIME-style math, RLVR-trained coding or agentic models may also suffer from reduced exploration, limiting their ability to handle novel or adversarial states.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The stronger collapse for semantically distinct continuations suggests that the reward signal itself, which checks only final answers, implicitly penalizes intermediate steps that are verifier-equivalent but less probable; an explicit diversity reward computed at the semantic level (e.g., over BODHI-TREE nodes) could counteract this without hurting validity.
  • A direct testable extension: build BODHI-TREEs using a human-annotated equivalence function on a small subset of AIME questions and recompute the CPE drop; if the drop persists, the semantic collapse is robust to judge error.
  • The maze 'Legal Only' intervention shows that masking invalid moves recovers most of RLVR's gain on a distilled model, which suggests that a lightweight decoding-time constraint could substitute for some RLVR training while preserving diversity.
  • The paper's distinction between syntactic and semantic branching entropy could be applied to other post-training schemes (e.g., DPO variants) to see whether all preference optimization methods calcify semantic choices or only reward-based ones.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies whether RLVR training collapses test-time exploration in LLMs in a way that is merely syntactic/stylistic or genuinely semantic. It contributes a maze-solving probe with three model families and the BODHI-TREE dataset for AIME mathematics, where reasoning traces are segmented and merged into a tree by an LLM-based semantic-equivalence judge. Defining candidate preference entropy (CPE) from token-level log-probabilities at branch nodes, the authors report that RLVR models have lower branching entropy than distillation-only models, that the drop is stronger for semantically distinct continuations than for syntactic variants, and that RLVR improves legality and backtracking at the cost of rollout diversity. The paper closes with an intervention analysis suggesting RLVR's sample-efficiency gains come from constricting the accessible continuation space.

Significance. If the central claim holds, the paper makes a valuable contribution to the debate on whether RLVR expands reasoning capabilities or merely sharpens sampling: it provides evidence that entropy collapse is not reducible to surface style. The strengths are the controlled matched training setup, the open-sourced artifacts and checkpoints, and the careful use of paired bootstrap confidence intervals and KS tests across four model families. The significance is conditional, however, because the semantic-versus-syntactic comparison rests on an unvalidated LLM judge and on an uncontrolled surface-form distance between the compared continuation pairs, as detailed below.

major comments (3)
  1. [§4.4, Eqs. (3)–(4)] The comparison that supports the paper's central claim is not controlled for surface-form distance. The semantic pair (a,g) is drawn from different BODHI children, so it will typically have less token overlap and different lengths than the syntactic pair (a,b), while CPE is computed from per-token average log-probabilities of the raw continuations. An RLVR policy that simply sharpens preferences for a surface form—shorter traces, particular notation, or formula layout—would produce a larger ΔCPE for the more surface-distant pair without any specifically semantic calcification. Please add a matched analysis (e.g., pairs balanced for length and token overlap, or regression with edit distance as a covariate) or otherwise show that the μ_sem − μ_synt difference survives when surface distance is equated.
  2. [Appendix A and Limitations] The semantic equivalence function M(x,y) is validated only against two other LLMs (Cohen's κ = 0.815 and 0.837), and the paper explicitly states that no rigorous human evaluation was performed. If M merges or splits reasoning steps incorrectly, the BODHI branch structure itself is mismeasured, and the semantic CPE values in §4.2 and §4.4 inherit those errors. A human-annotated sample (even a few hundred pairs) with disagreement analysis would materially strengthen the claim that the entropy collapse is semantic rather than an artifact of tree construction.
  3. [Appendix B.2] The CPE computation truncates candidate continuations to 20 tokens. For mathematical reasoning steps, the point at which two strategies diverge semantically can occur after the first 20 tokens, so the reported CPE may measure only a shared surface prefix. Please report sensitivity to the truncation length (e.g., 20/50/100 tokens) or verify on a subset that branch identity is already evident within 20 tokens.
minor comments (4)
  1. [Figure 4] The caption and text contain placeholder glyphs (□) and the typo 'dashedlines'; the rendered means and confidence intervals should be checked.
  2. [Table 3] Table 3 is difficult to read: the 'Method' and 'Original(%)/Distracted(%)' columns are not cleanly separated in the text, and the 'Acc. [Pass@3]' header appears twice; reformatting is needed.
  3. [Figure 8] The distractor example includes an unrelated quadratic step ('Solving gives c=...') that may confuse readers; a cleaner example or a clearer annotation would help.
  4. [§5.1] The 'Avoid Short Path' intervention states that another verifier-equivalent continuation is guaranteed to exist, but the guarantee is not justified; please specify the maze property that ensures this.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CPE values come from the evaluated models' own token log-probabilities, and the BODHI-TREE semantic labels are not fitted inputs to those probabilities.

full rationale

The paper's central comparison—RLVR versus distilled CPE at BODHI-TREE branch nodes—is not circular. The tree is constructed from frontier-model traces using an LLM-as-a-judge equivalence function (Algorithm 1), whereas the CPE in Eq. 3 is a function only of the evaluated model's token log-probabilities over two fixed continuations; the tree structure does not enter the probability computation. The distinction between 'syntactic' pairs (a,b, from the same child) and 'semantic' pairs (a,g, from different children) is a measurement design, not an equation whose output is preset by its inputs. The only self-citation (AIME questions taken from Saha et al. 2026, Appendix A) supplies a benchmark question set rather than the conclusion that RLVR collapses semantic branching entropy, so it is not load-bearing. The LLM-judge validation is against other frontier LLMs rather than humans, and the §4.4 comparison may carry a surface-form/token-distance confound; these are empirical validity concerns, not instances in which a claimed prediction reduces to a fitted parameter or to a definition. No specific circular step can be exhibited, so the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim relies on the BODHI-TREE construction and the semantic/syntax distinction. No parameters are fitted to data; the main load-bearing assumptions are the reliability of the LLM-based semantic equivalence judge and the segmentation/merging procedure.

free parameters (1)
  • CPE continuation truncation length = 20 tokens
    Section B.2 states candidate continuations are truncated to 20 tokens for CPE calculation; this was chosen by the authors and could bias CPE if branches differ in typical length.
assumptions (4)
  • domain assumption Semantic equivalence of reasoning steps is a well-defined equivalence relation that can be reliably judged by an LLM (GPT-oss-120b).
    Algorithm 1 and Figure 6 use M(x,y) to merge reasoning segments into nodes; the entire BODHI-TREE and the syntax/semantics distinction depend on this. Validated only against other LLMs in Appendix A.
  • domain assumption Reasoning traces can be segmented into discrete, meaning-preserving steps such that a prefix tree of merged steps represents the space of solutions.
    Section 3.2 and Algorithm 1 assume segmentation with o4-mini and that merging by equivalence yields a tree whose paths are alternate solutions.
  • standard math Standard definitions of conditional entropy and average log-probability apply to token-level model distributions.
    Equations 1-3 in Section 3.3.
  • domain assumption The base model's probability distribution is the appropriate reference for measuring post-training preference shifts.
    Equation 4 defines Delta CPE relative to the base model; if the base model is a poor reference, the differences could be misattributed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?." pith.science (2026). https://pith.science/paper/JDS6EKHC

@misc{pith2026260802867,
  author       = {Pith},
  title        = {Pith review of: BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JDS6EKHC}},
  note         = {Machine review of arXiv:2608.02867}
}
read the original abstract

Although reinforcement learning with verifiable rewards (RLVR) has improved the performance of large language models (LLMs) across a variety of reasoning tasks, there is significant debate as to whether RLVR expands the reasoning capability boundary, or just improves sampling efficiency. In this paper, we investigate the nature of test-time exploration in RLVR-trained LLMs by employing controlled maze-solving experiments and extracting a tree structure from mathematical reasoning traces (BODHI-Trees) based on semantic equivalence. This helps us delineate between entropy arising from stylistic variations and genuine inferential branching. Our findings demonstrate that the policy entropy collapse observed in RLVR models is not merely syntactic, and is accompanied by a significant reduction in semantic branching entropy. While RLVR improves adherence to environmental constraints and backtracking capabilities, it constricts the space of continuations; we provide evidence suggesting that this might be responsible for the sample efficiency gains of RLVR, albeit at the cost of genuine rollout diversity.

Figures

Figures reproduced from arXiv: 2608.02867 by the authors.

Figure 1
Figure 1. Construction of mathematical reasoning trees. We first segment reasoning traces into a chain of reasoning nodes, followed by merging said nodes based on semantic similarity. Further details are provided in Appendix A. ber of alternate traces corresponding to a problem into a tree structure (BODHI-TREE, see [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. RL-trained models are less exploratory. We plot how often a node is visited by the distilled and RLVR-trained Qwen3-8B. ture of step-by-step problem-solving—the model must sequentially execute valid, discrete actions to reach a goal state—while stripping away the lin￾guistic confounders that arise from NL reasoning tasks. Because the action space is strictly con￾strained, alternate trace realizations reflect a shift… view at source ↗
Figure 3
Figure 3. RL-trained models successfully learn the semantics of mazes. We plot the entropy of the mod￾els at hall nodes (degree 2) to see if different training modalities produce models which correctly model the lack of ambiguity at halls. A part of the observed entropy collapse could be explained by the model assigning less probability 4Bootstrapped 95% confidence intervals. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Is the entropy collapse just syntactic? We observe that RLVR-trained models usually have lower entropy (CPE) both when choosing between two syntactic variants of a mathematical statement and two semantically distinct continuations. The entropy collapse in choosing betw…
Figure 5
Figure 5. Figure 5: Environmental constraint-adherence im￾proves during RL-training. We plot the baseline per￾formance (dashed line) of different RL-training check￾points of Qwen3-8B, and the ratio of performance with the Legal Only decoding strategy to baseline. Legal Only intervention h…
Figure 6
Figure 6. Figure 6: System prompt to instantiate the M(x, y) function in Algorithm 1. AdamW optimizer. RL-training was performed with the TRL library (von Werra et al., 2020) using the DAPO-Math-17k dataset (Yu et al., 2026) for 1K steps with lr = 1e−6, group size of 8, batch size of 48, …
Figure 7
Figure 7. Figure 7: Examples of extracted BODHI-TREEs. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Example distractor (red) for experiment in §5.2. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Effect of post-training on LLM branching. (top) We plot the distribution of ∆CPE as described in §4.2 for distilled and RLVR-trained models. There is a significant leftward shift in the distribution of ∆CPE suggesting that RL-trained models have enhanced preference at …
Figure 10
Figure 10. Figure 10: RL-trained models are more likely to back￾track. We plot the distribution of probability of back￾tracking (pb) as described in §5.2. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 27 canonical work pages

  1. [2]

    Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, and 4 others. 2025. SmolLM3: smol, ...

    2025

  2. [3]

    Pengxiang Cai, Tianchen Fang, Xiaohan Li, Qingyuan Zeng, Guocong Li, and Jintai Chen. 2026. https://arxiv.org/abs/2606.22317 Curriculum reinforcement learning can incentivize reasoning capacity in llms beyond the base model . Preprint, arXiv:2606.22317

    arXiv 2026

  3. [5]

    Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Xin Zhao, and Guang Shi. 2026 b . https://openreview.net/forum?id=eslxxopXTF Pass@k training for adaptively balancing exploration and exploitation of large reasoning models

    work page 2026

  4. [7]

    DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyu Hou, Chenhao Xu, Chenze Shao, Chong Ruan, Conner Sun, and 300 others. 2026. https://arxiv.org/abs/2606.19348 Deepseek-v4: Towards highly efficient million-token context i...

    arXiv 2026

  5. [9]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. https://doi.org/10.1038/s41586-025-09422-z DeepSeek-R1 incentivizes reasoning in LLM s through reinforcement lea...

    doi:10.1038/s41586-025-09422-z 2025

  6. [12]

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. https://openreview.net/forum?id=NFM8F5cV0V Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model . In The Thirty-ninth Annual Conference on Neural Information Processing Systems

    work page 2025

  7. [14]

    Zican Hu, Shilin Zhang, Yafu Li, Jianhao Yan, Xuyang Hu, Leyang Cui, Xiaoye Qu, Chunlin Chen, Yu Cheng, and Zhi Wang. 2026 b . https://openreview.net/forum?id=9G7AbBrd27 Diversity-incentivized exploration for versatile reasoning . In The Fourteenth International Conference on Learning Representations

    work page 2026

  8. [16]

    Spies, Tilman Räuker, Dan Valentine, Can Rager, Lucia Quirke, Chris Mathwin, Guillaume Corlouer, Cecilia Diniz Behn, and Samy Wu Fung

    Michael Igorevich Ivanitskiy, Rusheb Shah, Alex F. Spies, Tilman Räuker, Dan Valentine, Can Rager, Lucia Quirke, Chris Mathwin, Guillaume Corlouer, Cecilia Diniz Behn, and Samy Wu Fung. 2023. https://arxiv.org/abs/2309.10498 A configurable library for generating and manipulating maze datasets . Preprint, arXiv:2309.10498

    arXiv 2023

Show all 71 references
  1. [21]

    Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing, Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, Zheng Zhang, Wei Shen, Qian Liu, Chenghua Lin, Jian Yang, Ge Zhang, and Wenhao Huang. 2025. https://arxiv.org/abs/2508.17445 Treepo: Bridging the gap...

    2025 arXiv

  2. [22]

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. https://arxiv.org/abs/2503.20783 Understanding r1-zero-like training: A critical perspective . Preprint, arXiv:2503.20783

    2025 arXiv

  3. [24]

    Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others

    OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. https://a...

    2025 arXiv

  4. [25]

    OpenAI. 2025. https://openai.com/index/o3-o4-mini-system-card/ OpenAI o3 and o4-mini system card

    2025

  5. [26]

    OpenAI. 2026. https://openai.com/index/gpt-5-6/ GPT -5.6: Frontier intelligence that scales with your ambition

    2026

  6. [27]

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2024. https://arxiv.org/abs/2412...

    2024 arXiv

  7. [29]

    Soumya Rani Samineni, Durgesh Kalwar, Vardaan Gangal, Siddhant Bhambri, and Subbarao Kambhampati. 2025. https://arxiv.org/abs/2510.18176 Local coherence or global validity? investigating rlvr traces in math domains . Preprint, arXiv:2510.18176

    2025

  8. [30]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. https://arxiv.org/abs/2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . Preprint, arXiv:2402.03300

    2024 arXiv

  9. [31]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, ...

    2025 arXiv

  10. [32]

    Qwen Team. 2026. https://qwen.ai/blog?id=qwen3.5 Qwen3.5: Accelerating productivity with native multimodal agents

    2026

  11. [33]

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. https://github.com/huggingface/trl TRL: Transformers Reinforcement Learning

    2020

  12. [34]

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. 2025. https://openreview.net/forum?id=yfcpdY4gMP Beyond the 8...

    2025

  13. [35]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Conferen...

    2023

  14. [36]

    Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. 2026. https://openreview.net/forum?id=jGbRWwIidy Reinforcement learning with verifiable rewards implicitly incentivizes correct reaso...

    2026

  15. [37]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025 a . https://arxiv.org/abs/2505.09388...

    2025 arXiv

  16. [38]

    Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, and Jing Tang. 2025 b . https://arxiv.org/abs/2506.05183 Treerpo: Tree relative policy optimization . Preprint, arXiv:2506.05183

    2025

  17. [39]

    Griffiths, Yuan Cao, and Karthik R Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. https://openreview.net/forum?id=5Xc1ecxO1h Tree of thoughts: Deliberate problem solving with large language models . In Thirty-seventh Conference on Neural Informat...

    2023

  18. [40]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, and 17 others. 2026. https://openreview.net/foru...

    2026

  19. [41]

    Suqin Yuan, Jinkun Chen, Jiyang Zheng, Muyang Li, Lei Feng, Dadong Wang, Tao Xiang, Tongliang Liu, and Bo An. 2026. https://arxiv.org/abs/2606.15455 Understanding diversity collapse in rlvr via the lens of overtraining . Preprint, arXiv:2606.15455

    2026

  20. [42]

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025. https://openreview.net/forum?id=upehLVgq1b Does reinforcement learning really incentivize reasoning capacity in LLM s beyond the base model? In 2nd AI for Math Workshop @ ICML 2025

    2025

  21. [44]

    Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, and Zejun Ma. 2025. https://arxiv.org/abs/2507.07017 First return, entropy-eliciting explore . Preprint, arXiv:2507.07017

    2025 arXiv

  22. [45]

    Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for

    Shenzhi Wang and Le Yu and Chang Gao and Chujie Zheng and Shixuan Liu and Rui Lu and Kai Dang and Xiong-Hui Chen and Jianxin Yang and Zhenru Zhang and Yuqiong Liu and An Yang and Andrew Zhao and Yang Yue and Shiji Song and Bowen Yu and Gao Huang and Junyang Lin , booktitle=. B...

    2025

  23. [46]

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in

    Yang Yue and Zhiqi Chen and Rui Lu and Andrew Zhao and Zhaokai Wang and Yang Yue and Shiji Song and Gao Huang , booktitle=. Does Reinforcement Learning Really Incentivize Reasoning Capacity in. 2025 , url=

    2025

  24. [47]

    Transactions of the Association for Computational Linguistics , volume =

    Saha, Soumadeep and Chaturvedi, Akshay and Saha, Saptarshi and Garain, Utpal and Asher, Nicholas , title =. Transactions of the Association for Computational Linguistics , volume =. 2026 , month =. doi:10.1162/TACL.a.729 , url =

    2026 doi

  25. [48]

    2026 , url=

    Qiying Yu and Zheng Zhang and Ruofei Zhu and Yufeng Yuan and Xiaochen Zuo and YuYue and Weinan Dai and Tiantian Fan and Gaohong Liu and Juncai Liu and LingJun Liu and Xin Liu and Haibin Lin and Zhiqi Lin and Bole Ma and Guangming Sheng and Yuxuan Tong and Chi Zhang and Mofan Z...

    2026

  26. [49]

    2025 , eprint=

    Understanding R1-Zero-Like Training: A Critical Perspective , author=. 2025 , eprint=

    2025

  27. [50]

    2024 , eprint=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

    2024

  28. [51]

    Le and Ed H

    Xuezhi Wang and Jason Wei and Dale Schuurmans and Quoc V. Le and Ed H. Chi and Sharan Narang and Aakanksha Chowdhery and Denny Zhou , title =. The Eleventh International Conference on Learning Representations,. 2023 , url =

    2023

  29. [52]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  30. [53]

    2025 , eprint=

    TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling , author=. 2025 , eprint=

    2025

  31. [54]

    2025 , eprint=

    TreeRPO: Tree Relative Policy Optimization , author=. 2025 , eprint=

    2025

  32. [55]

    2025 , eprint=

    First Return, Entropy-Eliciting Explore , author=. 2025 , eprint=

    2025

  33. [56]

    Findings of the

    Chen, Xiwen and Zhu, Wenhui and Qiu, Peijie and Dong, Xuanzhao and Wang, Hao and Wu, Haiyu and Li, Huayu and Sotiras, Aris and Wang, Yalin and Razi, Abolfazl , editor =. Findings of the. 2026 , address =. doi:10.18653/v1/2026.findings-acl.685 , pages =

    2026 doi

  34. [57]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , month = nov, year =

    Group-Aware Reinforcement Learning for Output Diversity in Large Language Models , author =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , month = nov, year =. doi:10.18653/v1/2025.emnlp-main.1649 , pages =

    2025 doi

  35. [58]

    Findings of the

    Mishra, Kshitij and Lukas, Nils and Lahlou, Salem , editor =. Findings of the. 2026 , address =. doi:10.18653/v1/2026.findings-eacl.323 , pages =

    2026 doi

  36. [59]

    Proceedings of the 64th Annual Meeting of the

    Reinforced Efficient Reasoning via Semantically Diverse Exploration , author =. Proceedings of the 64th Annual Meeting of the. 2026 , address =. doi:10.18653/v1/2026.acl-long.2216 , pages =

    2026 doi

  37. [60]

    Rewarding the Rare: Uniqueness-Aware

    Hu, Zhiyuan and Wang, Yucheng and He, Yufei and Wu, Jiaying and Zhao, Yilun and Ng, See-Kiong and Breazeal, Cynthia and Luu, Anh Tuan and Park, Hae Won and Hooi, Bryan , editor =. Rewarding the Rare: Uniqueness-Aware. Findings of the. 2026 , address =. doi:10.18653/v1/2026.fin...

    2026 doi

  38. [61]

    Findings of the

    Li, Wendi and Li, Sharon , editor =. Findings of the. 2026 , address =. doi:10.18653/v1/2026.findings-acl.702 , pages =

    2026 doi

  39. [62]

    Findings of the

    He, Xu and Guo, Jialiang and Xiong, Fucheng and Zhao, Haodong and Li, Xingyang and Zeng, Ke and Cai, Xunliang , editor =. Findings of the. 2026 , address =. doi:10.18653/v1/2026.findings-acl.1434 , pages =

    2026 doi

  40. [63]

    2026 , eprint=

    Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model , author=. 2026 , eprint=

    2026

  41. [64]

    The Fourteenth International Conference on Learning Representations , year=

    Diversity-Incentivized Exploration for Versatile Reasoning , author=. The Fourteenth International Conference on Learning Representations , year=

  42. [65]

    A Few Bad Apples Spoil the Bunch: Preventing Global Entropy Collapse Driven by a Small Set of Tokens in

    Jang, Jaeeun and Lee, Hansle and Kim, Sangmin , editor =. A Few Bad Apples Spoil the Bunch: Preventing Global Entropy Collapse Driven by a Small Set of Tokens in. Findings of the. 2026 , address =. doi:10.18653/v1/2026.findings-acl.641 , pages =

    2026 doi

  43. [66]

    Findings of the

    Revisiting Entropy in Reinforcement Learning for Large Reasoning Models , author =. Findings of the. 2026 , address =. doi:10.18653/v1/2026.findings-acl.1266 , pages =

    2026 doi

  44. [67]

    2026 , eprint=

    Understanding Diversity Collapse in RLVR via the Lens of Overtraining , author=. 2026 , eprint=

    2026

  45. [68]

    Rethinking Entropy Interventions in

    Hao, Zhezheng and Wang, Hong and Liu, Haoyang and Luo, Jian and Yu, Jiarui and Dong, Hande and Lin, Qiang and Wang, Can and Chen, Jiawei , editor =. Rethinking Entropy Interventions in. Proceedings of the 64th Annual Meeting of the. 2026 , address =. doi:10.18653/v1/2026.acl-l...

    2026 doi

  46. [69]

    Semantic-Space Exploration and Exploitation in

    Huang, Fanding and Huang, Guanbo and Fan, Xiao and He, Yi and Liang, Xiao and Chen, Xiao and Jiang, Qinting and Khan, Faisal Nadeem and Jiang, Jingyan and Wang, Zhi , editor =. Semantic-Space Exploration and Exploitation in. Findings of the. 2026 , address =. doi:10.18653/v1/2...

    2026 doi

  47. [70]

    Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base

    Xumeng Wen and Zihan Liu and Shun Zheng and Shengyu Ye and Zhirong Wu and Yang Wang and Zhijian Xu and Xiao Liang and Junjie Li and Ziming Miao and Jiang Bian and Mao Yang , booktitle=. Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in...

    2026

  48. [71]

    Proceedings of the AAAI Conference on Artificial Intelligence , year =

    Reasoning with Exploration: An Entropy Perspective , author =. Proceedings of the AAAI Conference on Artificial Intelligence , year =. doi:https://doi.org/10.1609/aaai.v40i36.40290 , volume =

    doi

  49. [72]

    2025 , eprint=

    Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains , author=. 2025 , eprint=

    2025

  50. [73]

    2026 , url=

    Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models , author=. 2026 , url=

    2026

  51. [74]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , month = nov, year =

    What Makes a Good Reasoning Chain? Uncovering Structural Patterns in Long Chain-of-Thought Reasoning , author =. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , month = nov, year =. doi:10.18653/v1/2025.emnlp-main.329 , pages =

    2025 doi

  52. [75]

    QLoRA: Efficient Finetuning of Quantized LLMs , url =

    Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke , booktitle =. QLoRA: Efficient Finetuning of Quantized LLMs , url =. doi:10.52202/075280-0441 , editor =

    doi

  53. [76]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  54. [77]

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and Zhang, Xiaokang and Yu, Xingkai and Wu, Yu and Wu, Z. F. and Gou, Zhibin and Shao, Zhihong and Li, Zhuoshu and Gao, Zi...

  55. [78]

    Qwen2.5 Technical Report , url =

    Qwen and An Yang and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Chengyuan Li and Dayiheng Liu and Fei Huang and Haoran Wei and Huan Lin and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Yang and Jiaxi Yang and Jingren Zhou and Junyan...

  56. [79]

    2023 , eprint=

    A Configurable Library for Generating and Manipulating Maze Datasets , author=. 2023 , eprint=

    2023

  57. [80]

    von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin , license =

  58. [81]

    2025 , eprint=

    Gemma 3 Technical Report , author=. 2025 , eprint=

    2025

  59. [82]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  60. [83]

    Qwen3.5: Accelerating Productivity with Native Multimodal Agents , url =

    Qwen Team , month =. Qwen3.5: Accelerating Productivity with Native Multimodal Agents , url =

  61. [84]

    Bakouch, Elie and Ben Allal, Loubna and Lozhkov, Anton and Tazi, Nouamane and Tunstall, Lewis and Patiño, Carlos Miguel and Beeching, Edward and Roucher, Aymeric and Reedi, Aksel Joonas and Gallouédec, Quentin and Rasul, Kashif and Habib, Nathan and Fourrier, Clémentine and Ky...

  62. [85]

    2025 , eprint=

    gpt-oss-120b & gpt-oss-20b Model Card , author=. 2025 , eprint=

    2025

  63. [86]

    2026 , eprint=

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author=. 2026 , eprint=

    2026

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.