Pith. sign in

REVIEW 3 major objections 5 minor 17 cited by

TreeRL: LLM Reinforcement Learning with On-Policy Tree Search

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Entropy-guided tree search with tree-derived process rewards improves LLM reasoning RL over independent chain sampling at the same token budget.

desk verdict A promising entropy-guided tree search for LLM RL, but the headline RL gain is confounded by a 30-vs-16 response mismatch and the paper's own equal-response ablation shows no TreeRL advantage. read the letter →

arxiv 2506.11902 v1 pith:DQZWRNYP submitted 2025-06-13 cs.LG cs.CL

Zhenyu Hou , Ziniu Hu , Yujiang Li , Rui Lu , Jie Tang , Yuxiao Dong This is my paper
classification cs.LGcs.CL
keywords treesearchLLMreinforcementlearningprocesssupervisionentropy-guidedbranchingmathematicalreasoningcodecreditassignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that reinforcement learning for LLM reasoning is better done with on-policy tree search than with independent chain sampling at the same inference budget. It proposes TreeRL, whose sampling stage expands a search tree by forking new answer branches from the tokens where the policy is most uncertain, and whose learning stage turns the tree's own structure into per-step process rewards. The central comparison is against ChainRL, RL with i.i.d. multi-response sampling and outcome-only reward. If the claim holds, RL can get denser credit for intermediate reasoning steps without training a separate process reward model, avoiding distribution shift and reward hacking.

What carries the argument

The load-bearing object is EPTree, an entropy-guided tree search that (a) selects the top-N tokens with highest cross-entropy across existing branches, (b) forks T new completions from each selected token, and (c) repeats for L iterations, yielding $M\times(N\times T\times L+1)$ leaves per prompt. The second half is the tree-derived process reward: for each node $s_n$, $V(s_n)$ is the fraction of correct leaf descendants; the global advantage is $V(s_n)-V(\text{root})$, the local advantage is $V(s_n)-V(p(s_n))$, and the optimization reward is $(GA+LA)/\sqrt{|L(s_n)|}$, with the reweighting preventing shared non-leaf prefixes from being over-trained.

What would settle it

Run TreeRL on the same EPTree rollouts but replace the per-step $(GA+LA)/\sqrt{|L|}$ reward with the final-answer outcome reward (or with a random per-step reward); if final benchmark accuracy does not drop, the tree-derived process supervision is not the source of the gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that on-policy tree search can replace independent chain sampling as the rollout generator for LLM reinforcement learning and improve final reasoning performance. TreeRL builds a small tree per prompt by continuing generation from the top-N highest-entropy tokens, then assigns every step a process reward from the tree: the global advantage of a node over the root plus its local advantage over its parent, where node value is the Monte Carlo fraction of correct descendant leaves. These reward signals are on-policy by construction and require no trained reward model. On six math and code reasoning benchmarks, TreeRL trained this way outperforms ChainRL trained with i.i.d. multi-chain sampling under comparable generation token budgets.

Load-bearing premise

The argument depends on trusting the tree's own leaf statistics: the fraction of correct answers under an intermediate step is assumed to measure that step's quality, even though it comes from a small sample and no learned value model.

Editorial extensions

If this is right

  • Under a fixed generation-token budget, branching at high-entropy tokens yields more distinct responses and a higher PassRate than i.i.d. multi-chain sampling or MCTS.
  • TreeRL supplies dense, on-policy step-level credit without a separate process reward model, sidestepping distribution mismatch and reward hacking.
  • Process supervision and the extra training traces from tree structure both contribute; ablations show that removing global or local advantage, or using only a subset of leaves, degrades results.
  • The approach maintains comparable performance on general benchmarks such as MMLU-Pro, Arena-Hard, and IFEval while gaining on reasoning benchmarks.
  • Because the same generation budget produces more training sequences, TreeRL achieves better prompt efficiency during RL training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the entropy-forking rule makes a testable claim that the model's own uncertainty marks where exploration is most valuable; one could compare EPTree's forking distribution against an oracle that branches where Monte Carlo value has the highest variance.
  • Inference: since tree-derived Monte Carlo values replace a learned value model, a bootstrap or value head trained on the same trees could extend TreeRL to larger branching factors without extra rollouts.
  • Inference: the roughly uniform relative position of forking tokens suggests the method is not secretly exploiting errors clustered at one location, so a transfer test on multi-step tool-use or agentic tasks would show whether the benefit generalizes beyond math and code.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes TreeRL, a reinforcement learning framework for LLMs that replaces independent chain sampling with an entropy-guided tree search (EPTree) and derives dense, on-policy process rewards from the tree structure. The approach avoids training a separate process reward model. The paper claims superior performance over ChainRL on math and code reasoning benchmarks under comparable inference token budgets, with gains attributed to both better exploration from tree search and process supervision.

Significance. If the claimed improvements are real, on-policy tree search with process supervision could be a useful alternative to outcome-only chain RL for LLM reasoning, especially because it avoids a separate reward model. The paper is open-sourced, which is a concrete strength, and the EPTree sampling method is simple to describe. However, the current experimental evidence does not isolate the claimed effect: the main comparison changes both the sampling structure and the number of training responses, and the ablation does not test the process-supervision component in isolation. The central claim is therefore not yet established.

major comments (3)
  1. [§4.1, Table 1, Table 3] The headline comparison confounds tree search with the number of training responses. TreeRL uses (M,N,L,T)=(6,2,1,2), producing 30 leaves and an RL batch of 480 sequences, while ChainRL uses 16 responses and a batch of 256. The paper justifies this by approximate generation-token parity, but the training update therefore consumes 480 vs 256 sequences per gradient step, and the evaluation changes two variables at once. The paper's own Table 3 row '(GA+LA)/sqrt(n), n=16' trains TreeRL on 16 leaves and gives an average of about 41.25 on Qwen-2.5-14B, below ChainRL's 41.6 average from Table 1. Thus, when the number of training responses is matched, the tree-based method does not outperform the chain baseline in this comparison. The reported advantage appears to come largely from the extra 14 responses per prompt obtained within the token budget, rather than from the tree structure itself.
  2. [§4.1, Table 4, Figure 6] The EPTree hyperparameters were selected on Omni-MATH-500, which is also one of the headline evaluation datasets in Table 1 and Figure 6. The text says the authors 'investigate various combinations of hyperparameters using the trained Qwen-2.5-14B-SFT model on the Omni-MATH-500 dataset with PassRate as the target metric.' Since Omni-MATH-500 appears in the final evaluation, the reported Omni-MATH-500 results are selected on the evaluation set. The effect may be modest, but it is a direct evaluation-set contamination and should be fixed by tuning on a separate development set or reporting results for a fixed configuration chosen before evaluation.
  3. [§4.5, Table 3] The ablation does not isolate the contribution of process supervision from the contribution of more training responses. All TreeRL rows in Table 3 use process rewards, and there is no outcome-only TreeRL baseline with 30 responses or with 16 responses. Consequently, the paper's statement that 'the gain benefits from both EPTree with promising PassRate performance and process supervision' is not directly supported. The n=16 row shows that using process rewards with the same number of responses as ChainRL yields no average gain, which further raises the possibility that process supervision is not the driver of the reported improvement. Adding an outcome-supervision TreeRL condition with the same leaf count would directly test this.
minor comments (5)
  1. [§3.2.2] The heading contains a typo: 'Process Superivison' should be 'Process Supervision'.
  2. [Table 1] In the GLM4-9B row, the LiveCodeBench and Avg entries appear as '15.829.3', missing a space or separator between the two numbers.
  3. [§3.2.1] Equation (4) defines R(s_n) as GA + LA, but Algorithm 1 and Section 3.2.2 introduce the additional |L(s_n)|^{-1/2} reweighting. This is not inconsistent, but it would be clearer to include the reweight factor in the main equation or to explicitly state that Eq. (4) is before reweighting.
  4. [Appendix B] The analysis in Theorem 1 relies on assumptions of uniformly distributed forking positions and fixed generation lengths, and the l=2 case uses Monte Carlo simulation to bound phi. The statement that the ratio is in [4/3, 12/5] should be labeled as conditional on those assumptions, which are not justified from the actual EPTree behavior beyond the position distribution plot in Figure 8.
  5. [References] The reference to 'Team et al., 2023' should be 'Gemini Team, 2023' to match the reference list entry, and several entries appear both as arXiv preprints and in proceedings (e.g., Lightman et al. and Rafailov et al.); consider using a consistent citation style.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: process rewards are Monte Carlo estimates from leaf correctness, and the headline comparison rests on external benchmark results rather than on a definitional reduction.

full rationale

TreeRL's derivation chain is self-contained rather than circular. The process-supervision signal is defined in Section 3.2.1 by Equations (2)-(4), where V(s_n) is the Monte Carlo ratio of correct descendant leaves; GA and LA are differences of these values. These are definitions computed from the on-policy tree and the final-answer correctness labels, not parameters fitted to the later benchmark outcomes and then reported as predictions. The EPTree efficiency claim is supported by the empirical PassRate measurements in Table 4 and by Theorem 1, which bounds the leaf-count ratio under explicitly stated assumptions about uniform forking positions and fixed lengths; the theorem does not assume the empirical PassRate conclusion it is used to motivate. The central claim that TreeRL outperforms ChainRL is an external benchmark observation, not a consequence of the definitions. The 30-response versus 16-response comparison and the larger RL batch size are genuine threats to the experimental conclusion, and the paper's own Table 3 row with n=16 shows no TreeRL gain over ChainRL when the response count is matched; that is a correctness and experimental-design concern, not circularity. Self-citations (e.g., Hou et al. 2025 for the public SFT dataset and Zhang et al. 2024a as related work) are data sources and background references, not load-bearing uniqueness claims or imported constraints that force the paper's conclusion. No step in the claimed derivation is equivalent to its own input by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method introduces no invented entities and no new physical content. Its load-bearing assumptions are domain assumptions: token-level entropy locates useful forking points, MC value estimates from descendant leaves are reliable process supervision, and the Appendix B leaf-count theorem assumes uniform forking positions and fixed lengths. The free parameters are the EPTree configuration, the sqrt reweight factor, and the hand-set GA and LA weights. The theoretical claims in Appendix B are heuristic and do not validate the RL results.

free parameters (3)
  • EPTree configuration (M,N,L,T) = (6,2,1,2) for the main RL runs, plus (8,4,2,2) and others in Table 4
    Selected by PassRate search on Omni-MATH-500, which is also a reported evaluation set, in Section 4.1 and Table 4.
  • Process reward reweight exponent = -1/2, i.e. divide by sqrt of descendant leaf count
    Chosen empirically; the text in Section 3.2.2 says 'This adjustment leads to improved performance in our experiments.'
  • GA and LA combination weights = 1.0 for each, so R(s_n) = GA(s_n) + LA(s_n)
    Set by hand as a special case of GAE in Section 3.2.1, Eq. 4; no derivation or fitting fixes these weights.
assumptions (4)
  • domain assumption Token-level cross-entropy of the sampled token is a reliable proxy for model uncertainty and a good place to fork for exploration.
    Used to select forking tokens in EPTree in Section 3.1; the paper provides only a frequency case study in Figure 7, not an isolated validation.
  • domain assumption Monte Carlo value V(s_n) = fraction of correct descendant leaves is an unbiased and low-variance estimate of step quality for process supervision.
    Value estimates define GA and LA in Section 3.2.1, Eq. 2-3; only leaves carry correctness and no value model or bootstrap is used.
  • domain assumption Forking positions follow U(0,1), generation lengths are fixed, and L is at most 2 for the leaf-count theorem.
    These are the explicit assumptions of Theorem 1 in Appendix B; Figure 8 provides only a rough empirical distribution and the theorem bounds leaf count, not correctness.
  • standard math The policy-gradient objective in Eq. 1 with advantage computed from tree process rewards is a valid RL update.
    Standard policy gradient with baselines; the paper does not prove convergence or unbiasedness for the tree-structured reward.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TreeRL: LLM Reinforcement Learning with On-Policy Tree Search." pith.science (2026). https://pith.science/paper/DQZWRNYP

@misc{pith2026250611902,
  author       = {Pith},
  title        = {Pith review of: TreeRL: LLM Reinforcement Learning with On-Policy Tree Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DQZWRNYP}},
  note         = {Machine review of arXiv:2506.11902}
}
read the original abstract

Reinforcement learning (RL) with tree search has demonstrated superior performance in traditional reasoning tasks. Compared to conventional independent chain sampling strategies with outcome supervision, tree search enables better exploration of the reasoning space and provides dense, on-policy process rewards during RL training but remains under-explored in On-Policy LLM RL. We propose TreeRL, a reinforcement learning framework that directly incorporates on-policy tree search for RL training. Our approach includes intermediate supervision and eliminates the need for a separate reward model training. Existing approaches typically train a separate process reward model, which can suffer from distribution mismatch and reward hacking. We also introduce a cost-effective tree search approach that achieves higher search efficiency under the same generation token budget by strategically branching from high-uncertainty intermediate steps rather than using random branching. Experiments on challenging math and code reasoning benchmarks demonstrate that TreeRL achieves superior performance compared to traditional ChainRL, highlighting the potential of tree search for LLM. TreeRL is open-sourced at https://github.com/THUDM/TreeRL.

Figures

Figures reproduced from arXiv: 2506.11902 by the authors.

Figure 1
Figure 1. Left: Performance comparison of sampling strategies. EPTree consistently outperforms i.i.d multi￾chain sampling and MCTS under different inference budgets. Right: TreeRL powered with EPTree demon￾strates better performance than ChainRL with i.i.d multi￾chain sampling. feedback (OpenAI, 2024; Guo et al., 2025; Hou et al., 2025; Shao et al., 2024b). Current RL methods for LLM training generally independently sample mu… view at source ↗
Figure 2
Figure 2. Illustration of offline training with tree search (Left), traditional ChainRL with online i.i.d multi-response [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Illustration of TreeRL. In each iteration, TreeRL first performs a tree search using EPTree, progressively [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Generation diversity comparison of EPTree, [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Search performance of EPTree, MCTS, and multi-chain sampling on Omni-MATH-500 using Qwen￾2.5-14B-SFT. EPTree consistently outperforms all base￾lines under the different inference costs. quences and will be repeatedly computed in op￾timization, we downweight the reward …
Figure 6
Figure 6. Figure 6: Performance comparison between TreeRL and ChainRL during training. We report the average perfor [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Frequent words of sampled forking tokens in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 1
Figure 1. Figure 1: Branching process illustration when Figure 9: Forking token illustration when n = 3, t = [PITH_FULL_IMAGE:figures/full_fig_p012_1.png]
Figure 10
Figure 10. Figure 10: Performance comparison between TreeRL and ChainRL on MATH500 (Upper Left), AMC (Upper [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison between TreeRL and ChainRL on MATH500 (Upper Left), AMC (Upper [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization

    cs.LG 2026-06 conditional novelty 7.0 of 10

    KernelPro combines LLM code generation, roofline-guided tool orchestration, and domain-adapted MCTS to produce GPU kernels that outperform prior automated and some hand-tuned baselines on KernelBench and VeOmni workloads.

  2. From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning

    cs.LG 2026-06 conditional novelty 6.5 of 10

    SFT supplies entangled compositional traces of atomic skills and routing modules; RL identifies those modules and enables recombination on novel compositions outside the SFT support.

  3. From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Introduces a hierarchical latent selection model showing SFT supplies raw module materials in compound traces while RL decomposes them to identify atomic modules and enable recombination for new reasoning configurations.

  4. Learn from Your Mistakes: Tree-like Self-Play for Secure Code LLMs

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    TSP reframes secure code generation as a tree-structured self-play process that supplies dense on-policy signals at vulnerability-prone nodes, yielding higher security pass rates and cross-language generalization than...

  5. Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modeling

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    GDCR assigns step-level rewards via distance to the answer node in a training-time ER graph and SAPO combines these with trajectory advantages for credit assignment in agentic search.

  6. Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Reflective Prompted Policy Optimization uses a Critic-LLM to inspect full trajectories and propose grounded revisions, yielding higher mean best rewards, faster near-optimal performance, and greater stability than sca...

  7. A$^2$TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    A²TGPO improves RL policy optimization for multi-turn agentic LLMs by normalizing information gain within same-depth turn groups, rescaling cumulative advantages by sqrt of term count, and modulating clipping ranges p...

  8. Tree Training: Accelerating Agentic LLMs Training via Shared Prefix Reuse

    cs.LG 2025-11 unverdicted novelty 6.0 of 10

    Tree Training serializes tree trajectories via DFS and uses redundancy-free partitioning to compute weighted per-token losses exactly once per token, achieving up to 6.2x training speedup on dense and MoE models.

  9. CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

    cs.LG 2026-07 conditional novelty 5.0 of 10

    CompactionRL trains LLM agents to generate context summaries during RL rollouts, enabling long-horizon task completion under fixed context budgets with consistent gains on SWE-bench Verified and Terminal-Bench 2.0.

  10. Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.

  11. Trust Region On-Policy Distillation

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.

  12. Mind DeepResearch Technical Report

    cs.AI 2026-04 unverdicted novelty 5.0 of 10

    MindDR combines a Planning Agent, DeepSearch Agent, and Report Agent with SFT cold-start, Search-RL, Report-RL, and preference alignment to reach competitive scores on research benchmarks using 30B-scale models.

  13. Your Model Diversity, Not Method, Determines Reasoning Strategy

    cs.AI 2026-04 unverdicted novelty 5.0 of 10

    The optimal reasoning strategy for LLMs depends on the model's diversity profile rather than the exploration method itself.

  14. ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learning

    cs.LG 2026-02 conditional novelty 5.0 of 10

    ECHO's entropy-confidence hybrid tree search plus confidence-adaptive clipping improves test-time RL accuracy by 1-5 points on several math and visual reasoning benchmarks.

  15. XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation

    cs.LG 2025-10 conditional novelty 5.0 of 10

    XRPO extends GRPO with adaptive rollout allocation, in-context example seeding for unsolved prompts, and novelty-weighted advantages, reporting roughly 1-4% higher accuracy and faster convergence.

  16. ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanism

    cs.LG 2025-08 conditional novelty 5.0 of 10

    ETTRL improves test-time RL for LLMs by forking rollouts at high-entropy tokens and reshaping advantages with a relative entropy bonus, reporting large AIME 2024 gains at lower token cost.

  17. A Survey of Reinforcement Learning for Large Reasoning Models

    cs.CL 2025-09 accept novelty 3.0 of 10

    A survey compiling RL methods, challenges, data resources, and applications for enhancing reasoning in large language models and large reasoning models since DeepSeek-R1.

Reference graph

Works this paper leans on

49 extracted references · 11 canonical work pages · cited by 16 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Ahmet \"U st \"u n, and Sara Hooker. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. In ACL

    work page 2024

  3. [3]

    Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1--43

    2012

  4. [5]

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024 b . https://arxiv.org/abs/2405.03553 Alphamath almost zero: Process supervision without process . Preprint, arXiv:2405.03553

    arXiv 2024

  5. [6]

    Wesley Chung, Valentin Thomas, Marlos C Machado, and Nicolas Le Roux. 2021. Beyond variance reduction: Understanding the true impact of baselines on policy optimization. In International Conference on Machine Learning, pages 1999--2009. PMLR

    work page 2021

  6. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  7. [8]

    Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2024. https://arxiv.org/abs/2309.17179 Alphazero-like tree-search can guide large language model decoding and training . Preprint, arXiv:2309.17179

    arXiv 2024

  8. [9]

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. 2024. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985

    arXiv 2024

Show all 49 references
  1. [10]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    2024 arXiv

  2. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  3. [12]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv...

    2024 arXiv

  4. [13]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  5. [14]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  6. [15]

    Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. 2025. Advancing language model reasoning through reinforcement learning and inference scaling. arXiv preprint arXiv:2501.11651

    2025 arXiv

  7. [16]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  8. [17]

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. 2024. https://arxiv.org/abs/2410.01679 Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment . Preprint, arXiv:2410.01679

    2024 arXiv

  9. [18]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  10. [19]

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. 2024 a . Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face...

    2024

  11. [20]

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024 b . From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939

    2024 arXiv

  12. [21]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. https://arxiv.org/abs/2305.20050 Let's verify step by step . Preprint, arXiv:2305.20050

    2023 arXiv

  13. [22]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations

  14. [23]

    Jieyi Long. 2023. https://arxiv.org/abs/2305.08291 Large language model guided tree-of-thought . Preprint, arXiv:2305.08291

    2023 arXiv

  15. [24]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173

    2024 arXiv

  16. [25]

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. 2024. https://arxiv.org/abs/2406.06592 Improve mathematical reasoning in language models by automated process supervision . Pr...

    2024 arXiv

  17. [26]

    Jincheng Mei, Wesley Chung, Valentin Thomas, Bo Dai, Csaba Szepesvari, and Dale Schuurmans. 2022. The role of baselines in policy gradient optimization. Advances in Neural Information Processing Systems, 35:17818--17830

    2022

  18. [27]

    OpenAI. 2024. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms

    2024

  19. [28]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference o...

    2022

  20. [29]

    Qwen. 2024. https://qwenlm.github.io/blog/qwen2.5 Qwen2.5: A party of foundation models

    2024

  21. [30]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728--53741

    2023

  22. [31]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems

    2024

  23. [32]

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438

    2015 arXiv

  24. [33]

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. 2024. https://arxiv.org/abs/2410.08146 Rewarding progress: Scaling automated process verifiers for llm reasoning . Preprint, arXiv:2410.08146

    2024 arXiv

  25. [35]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. 2024 b . Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  26. [36]

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2017. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint ar...

    2017 arXiv

  27. [37]

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward hacking. Advances in Neural Information Processing Systems, 35:9460--9471

    2022

  28. [38]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. https://arxiv.org/abs/2408.03314 Scaling llm test-time compute optimally can be more effective than scaling model parameters . Preprint, arXiv:2408.03314

    2024 arXiv

  29. [39]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  30. [40]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  31. [41]

    Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. 2024 a . https://arxiv.org/abs/2312.08935 Math-shepherd: Verify and reinforce llms step-by-step without human annotations . Preprint, arXiv:2312.08935

    2024 arXiv

  32. [42]

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024 b . Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    2024

  33. [43]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024 c . Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574

    2024 arXiv

  34. [44]

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451

    2024 arXiv

  35. [45]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. https://arxiv.org/abs/2305.10601 Tree of thoughts: Deliberate problem solving with large language models . Preprint, arXiv:2305.10601

    2023 arXiv

  36. [47]

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024 b . https://arxiv.org/abs/2406.03816 Rest-mcts*: Llm self-training via process reward guided tree search . Preprint, arXiv:2406.03816

    2024 arXiv

  37. [48]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

  38. [49]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2024. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations

    2024

  39. [50]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931

    2024 arXiv

  40. [51]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  41. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.