Pith. sign in

REVIEW 4 major objections 6 minor 7 cited by

TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TreePO reduces RL rollout GPU hours by up to 43 percent by sampling reasoning paths as a shared-prefix tree.

desk verdict A practical compute-saving rollout scheme with a real efficiency result, undermined by an overclaimed 'free lunch' narrative and missing error bars. read the letter →

arxiv 2508.17445 v1 pith:ZU2Y5F6B submitted 2025-08-24 cs.LG cs.CL

Yizhi Li , Qingshui Gu , Zhoufutu Wen , Ziniu Li , Tianshun Xing , Shuyue Guo , Tianyu Zheng , Xin Zhou
show 9 more authors
Xingwei Qu Wangchunshu Zhou Zheng Zhang Wei Shen Qian Liu Chenghua Lin Jian Yang Ge Zhang Wenhao Huang
This is my paper · ORCID
classification cs.LGcs.CL
keywords tree-basedsamplingreinforcementlearningcreditassignmentKVcachereusereasoningbenchmarkspolicyoptimizationefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TreePO tries to solve two bottlenecks in reinforcement-learning post-training of large language models: each rollout recomputes the same early reasoning tokens independently, and sparse final rewards are hard to attribute to specific tokens. The paper claims that treating generation as a segment-wise tree search—spawning branches where the model is locally uncertain and pruning flawed paths early—cuts the compute per policy update while keeping or improving reasoning accuracy. It reports a 22–43% reduction in GPU hours for trained models, up to 40% trajectory-level and 35% token-level sampling compute savings, and stable training directly from a base model without supervised fine-tuning. If this holds, RL-based post-training can be scaled with substantially fewer samples and less GPU time than current grouped sampling.

What carries the argument

The carrying mechanism is a segment-level tree-search rollout paired with a subgroup-relative advantage estimator. Generation proceeds in fixed-length segments (e.g., 512 tokens); at each depth the active prompts are forked with a branching budget, and early stopping prunes repetitive or answered branches, while fallback tops up missing trajectories. The advantage of a token is the average, across all tree depths, of the trajectory reward minus the mean reward of the trajectories sharing the same ancestor segment at that depth, normalized by the global standard deviation. That estimator is what converts the tree structure into denser, hierarchy-aware credit assignment, and the shared-prefix tree is what lets one forward pass serve many rollouts.

What would settle it

Take a fixed prompt set and run TreePO's tree sampling with a fallback that pads or truncates every fallback segment to a fixed length so all prefixes align, versus a fallback that leaves segments at natural lengths. If the method's advantage estimation is insensitive to alignment, both runs should show the same accuracy and response lengths; the paper's account predicts the misaligned run to be worse, isolating alignment as the load-bearing component. Alternatively, force all trajectories to diverge at their first token and compare throughput with independent rollouts; if the speedup persists, shared-prefix amortization is not the explanation.

Watch

Extended reading notes

Core claim

The central claim is that the standard practice of rolling out independent trajectories for the same prompt is sub-optimal, and that a heuristic tree-structured rollout is both cheaper and no worse for RL training. TreePO generates fixed-length segments, forks active prompts according to a dynamic branching budget, stops branches that repeat patterns or produce answers, and falls back only when a query lacks enough complete trajectories. Advantages are then computed not from whole-sequence rewards but by averaging, over the shared-prefix subgroups of the tree, the difference between a trajectory's reward and the mean reward of its subgroup. The paper shows this estimator, combined with the tree sampling, raises overall majority-vote accuracy from 46.63% to 54.61% over a GRPO baseline and, in the best configuration, matches a strong sequential baseline at 58.21% while cutting GPU hours by up to 43%. It also reports that deliberately breaking token alignment of segments degrades accuracy and inflates response length, which the authors take as evidence that the method's benefits rely on aligned shared prefixes.

Load-bearing premise

The whole efficiency and credit-assignment argument depends on generated segments lining up token-for-token across trajectories at each tree depth; when fallback produces a segment of a different length, the grouped advantages are computed over misaligned groups and accuracy drops, as the paper's own ablation shows.

Editorial extensions

If this is right

  • RL post-training from a base model can proceed without supervised fine-tuning, so the pipeline becomes shorter and cheaper for new domains.
  • A family of test-time compute-scaling curves emerges: small branching budgets win at low compute, wider trees win at high compute, so the tree shape itself becomes a compute-optimality knob.
  • Because shared prefixes are decoded once, the same batch of GPUs supports more queries or more rollouts per query, directly lowering the GPU-hour cost per policy update.
  • The subgroup-relative advantage estimator stabilizes training curves compared with sequence-level GRPO, which should make training runs more reproducible and less sensitive to reward spikes.
  • Exploration budget can be steered per segment using log-probabilities; the paper finds that aggressive low-probability branching hurts accuracy, so budget control is a usable but risky lever.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to align fallback segments algorithmically—for example by forcing a canonical segment length or using a prefix-alignment loss—which would remove the paper's identified failure mode and may let deeper trees close the gap to the 14×512 sweet spot.
  • The reported disconnect between entropy and accuracy suggests that diversity metrics should be weighted by downstream correctness; a reward-conditioned branching policy might outperform both low- and high-probability encouragement.
  • The same tree structure could be reused at inference time: a trained TreePO policy likely benefits from tree-based decoding without further training, since the sampling scheme is already built around KV-cache reuse.
  • If the compute savings transfer to longer-horizon tasks such as tool use or multi-turn dialogue, the segment-tree abstraction gives a natural way to insert external feedback at branch points rather than only at final answers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. TreePO proposes a tree-based rollout sampling scheme for RL training of LLMs, replacing independent sequential rollouts with segment-level tree search that shares prefixes, prunes low-value paths, and uses a hierarchical advantage estimator built on GRPO/DAPO-style objectives. The paper reports that TreePO reduces training GPU hours by 22–43% while maintaining or improving benchmark accuracy on math reasoning, and claims up to 40% trajectory-level and 35% token-level sampling compute reductions in offline efficiency experiments. Training is done from the Qwen2.5-7B base model without SFT, and ablations study segment length, branching, fallback, and advantage aggregation choices.

Significance. If the efficiency and accuracy claims hold, TreePO would be a practically useful contribution: it directly targets the expensive rollout phase of RLHF/RLVR, exploits KV-cache reuse in a principled way, and provides a finer-grained credit assignment signal. The offline throughput measurements in Section 4.1 are direct, reproducible in principle, and show a plausible efficiency benefit. The paper also ships a relatively complete set of ablations (segment budgets, advantage variants, branching heuristics) and identifies an important failure mode (token-misaligned fallback) that future work can build on. The main value is therefore in the sampling framework and its engineering insights, rather than in a new theoretical principle. However, the central 'free lunch' claim—compute savings without accuracy loss—is not statistically established as presented, and the method's reliance on token alignment under its own dynamic heuristics is underexplored.

major comments (4)
  1. [Table 2 and Abstract] The abstract claims GPU-hour savings 'from 22% up to 43%' while 'maintaining or improving benchmark accuracy,' but Table 2 does not support this: the 43% saving (More Init Divergence, b=2) comes with a 3.54-point overall accuracy drop (58.21% to 54.67%), and the 29% saving (b=4) also loses accuracy (57.26%), while only b=8 is near parity (58.06% vs 58.21%) at 22% savings. In the Fixed Init Divergence block, b=2 and b=8 lose accuracy. No error bars, seeds, or repeated runs are reported, so 'near parity' and 'improvement' are not statistically supported. Additionally, the abstract's lower bound of 22% conflicts with the 12% GPU-hour saving shown for Fixed Init Divergence b=8. The paper should either present accuracy-normalized comparisons (e.g., matching accuracy by tuning compute) or report variance across seeds, or the claim must be weakened.
  2. [Section 4.2 and Figure 6] The advantage estimator in Eq. (5) requires that trajectories sharing a tree prefix also share token-level segment boundaries. The paper itself shows in Figure 6 that a 512-token fallback on 7x1024 segments produces token-misaligned trajectories, degrading AIME accuracy and inflating response length. Since the proposed dynamic branching and fallback heuristics (Section 2.2) can produce such misalignment in normal operation—for example when a stopped path falls back with a different segment length—the manuscript must either bound the frequency of misalignment under the recommended configurations, or modify the fallback to preserve alignment. As written, the practical validity of the advantage estimator under the very heuristics that define TreePO is not established.
  3. [Section 4.1] The efficiency numbers that anchor the paper's central claim are single-run measurements with no variance or repetition count. Figure 4 and the text report +40% TrajPS and +30% TokenPS as geometric means over configurations and models, but no error bars are given and the setup fixes GPU utilization at 60%, which may not reflect production conditions. Since the GPU-hour savings in Table 2 are the key differentiator of TreePO, at least two or three independent runs per configuration with standard deviations should be reported to support the claimed magnitudes.
  4. [Sections 1 and 5] The introduction and related work claim that TreePO's advantage estimation is 'distinct' from TreeRL and SPO and is uniquely suited for training from a base model, but no direct empirical comparison to these methods is provided under matched conditions (same base model, data, compute budget). Without such a comparison, the reader cannot tell whether the reported gains come from the tree sampling, the advantage estimator, or simply from the GRPO/DAPO backbone. Adding at least one baseline (e.g., TreeRL or SPO with the same sampling tree but their advantage, or the same advantage with sequential sampling) would make the contribution of the estimator concrete.
minor comments (6)
  1. [Figure 1 caption] The caption contains an incomplete and ungrammatical sentence: 'When cooperate the health.' This should be fixed or removed.
  2. [Section 3.1] The sentence 'we set the rollout N as 16 and use the majority voting accuracy via 1000 times of sampling as the main metric' is ambiguous: it is unclear whether N=16 is the training rollout group size, whether the 1000 samples are used for evaluation only, and how majority voting is computed. Please clarify the evaluation protocol.
  3. [Section 2.2] In the branching description, 'see the details in the following literature' should read 'see the details in the following text' or similar; 'literature' is the wrong word here.
  4. [Equation (5)] The normalization term std({A_i,t,j}_{J-1}) is confusing: the subscript J-1 suggests the set excludes the root group, but the summation in the numerator includes j=1..J. Define the exact set over which the standard deviation is computed.
  5. [Section 4.2] The text says the misalignment experiment uses '7x1024 rollout but still a 512-token fallback,' but Figure 6 is referenced for the results while the setup paragraph cites Figure 7; check the figure numbering throughout Section 4.2 and 4.3.
  6. [Section 2.3] The condition 'std({Ri}G) != 0' is stated for the full group, but the per-subgroup estimator in Eq. (5) requires nonzero variance within each subgroup. The paper does not state how subgroups with zero variance (e.g., all trajectories in a subgroup having the same reward) are handled; please specify the fallback rule.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TreePO's efficiency numbers are direct measurements and its advantage estimator is a proposed training objective, not a prediction derived from fitted inputs.

full rationale

I walked the paper's derivation chain and found no circular step that reduces a claimed result to its inputs by construction. The central efficiency claims (GPU-hour reductions of 12–43%, trajectory-level and token-level savings) are presented as measured outcomes of implemented runs in Table 2 and Section 4.1, not as predictions obtained from fitted constants or from the estimator's definition. The TreePO advantage estimator (Eqs. 1 and 5) is a proposed RL objective built on GRPO/DAPO-style clipping plus subgroup-mean baselines; it is not derived from the efficiency results, and the efficiency results are not used to fit any parameter of the estimator. The paper's own ablation in Section 4.2 shows that the method degrades when segment alignment is broken, which is evidence that the reported gains are not automatic consequences of the formalism. There are self-citations: DAPO [3] shares an author with this paper and is explicitly adopted as a starting point, and FR3E [32] overlaps with the author list but appears only in related-work discussion. Neither is load-bearing: no claim hinges on an unverified self-cited theorem, and the comparisons are run against external baselines and measured directly. The skeptic's concern about Table 2 — that larger GPU-hour cuts coincide with accuracy drops and no error bars are reported — is a correctness/statistical-support issue, not a circularity issue, and therefore does not raise the circularity score. The non-finding is appropriate: the derivation is self-contained and empirically grounded.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method's gains hinge on several hand-chosen hyperparameters (segment length, tree depth, branching factor, initial divergence) and two empirical assumptions about trajectory structure and segment alignment. No new physical entities are introduced.

free parameters (4)
  • segment token length L_seg = 512 tokens (tried 128/256/512/1024)
    Chosen by hand as the best among configurations; the central efficiency and accuracy results depend on this choice (Sections 3.1, 4.3).
  • tree depth d = 14 for best balance (7, 28, 56 also tested)
    Chosen by hand; efficiency peaks at intermediate depth, so the reported 22-43% savings depend on this selection.
  • branching budget b = 2 (binary tree) default; 2, 4, 8 tested
    The per-segment branching factor is a hand-set hyperparameter; results in Table 2 vary by b.
  • initial divergence count = 2 to 8 random branches
    Introduced in the 'More Init Divergence' setting; without it performance differs (Section 3.1).
assumptions (4)
  • domain assumption Reasoning trajectories share long common prefixes that can be reused without biasing the sampled distribution.
    Motivates the whole tree sampling design; verified only with a small case study on an aligned model (Section 2.1).
  • domain assumption Log-probability of the last segment is a useful signal for allocating branching budget.
    Used in probability-based branching (Section 4.4); experiments show static heuristics underperform, so the assumption is not robust.
  • domain assumption Token-aligned segments across trajectories preserve the validity of the advantage estimator.
    Required for Eq. 5; Section 4.2 shows misaligned fallback degrades accuracy, confirming the assumption is load-bearing.
  • domain assumption The GRPO/DAPO policy objective with clip-higher, dynamic sampling and token-level loss is a valid starting point for policy optimization.
    Taken from cited prior works; TreePO builds on this objective without re-deriving its validity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling." pith.science (2026). https://pith.science/paper/ZU2Y5F6B

@misc{pith2026250817445,
  author       = {Pith},
  title        = {Pith review of: TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZU2Y5F6B}},
  note         = {Machine review of arXiv:2508.17445}
}
read the original abstract

Recent advancements in aligning large language models via reinforcement learning have achieved remarkable gains in solving complex reasoning problems, but at the cost of expensive on-policy rollouts and limited exploration of diverse reasoning paths. In this work, we introduce TreePO, involving a self-guided rollout algorithm that views sequence generation as a tree-structured searching process. Composed of dynamic tree sampling policy and fixed-length segment decoding, TreePO leverages local uncertainty to warrant additional branches. By amortizing computation across common prefixes and pruning low-value paths early, TreePO essentially reduces the per-update compute burden while preserving or enhancing exploration diversity. Key contributions include: (1) a segment-wise sampling algorithm that alleviates the KV cache burden through contiguous segments and spawns new branches along with an early-stop mechanism; (2) a tree-based segment-level advantage estimation that considers both global and local proximal policy optimization. and (3) analysis on the effectiveness of probability and quality-driven dynamic divergence and fallback strategy. We empirically validate the performance gain of TreePO on a set reasoning benchmarks and the efficiency saving of GPU hours from 22\% up to 43\% of the sampling design for the trained models, meanwhile showing up to 40\% reduction at trajectory-level and 35\% at token-level sampling compute for the existing models. While offering a free lunch of inference efficiency, TreePO reveals a practical path toward scaling RL-based post-training with fewer samples and less compute. Home page locates at https://m-a-p.ai/TreePO.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Branch Policy Optimization for Multimodal Large Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    MBPO uses branch-level, sibling-relative advantages in tree-structured rollouts to improve credit assignment when training multimodal large language models with reinforcement learning.

  2. BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?

    cs.CL 2026-08 conditional novelty 6.0 of 10

    RLVR-trained LLMs show a stronger collapse in semantic branching entropy than in syntactic style entropy, suggesting the policy prunes genuinely distinct reasoning continuations.

  3. Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning

    cs.AI 2026-08 conditional novelty 6.0 of 10

    MMPO jointly minimizes the first T moments of the per-problem failure-probability distribution — equivalent to minimizing expected truncated time to first success — and reports average gains over single-moment RL obje...

  4. Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

    cs.LG 2026-07 conditional novelty 6.0 of 10

    OC-GRPO reweights GRPO gradients with an importance ratio so that hints used during rollout generation still optimize the original unguided objective, delivering a 13.8% relative Pass@1 gain over vanilla GRPO.

  5. Distilled Reinforcement Learning for LLM Post-training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Using teacher preferences to reweight RL gradients only on correct responses improves LLM post-training on math and knowledge benchmarks in both within- and cross-family settings.

  6. Process Reward Informed Tree Rollout for Effective Multi-Turn RL

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Using process feedback to guide which partial trajectories to branch on, PATR replaces uniform independent rollouts with adaptive tree rollouts for multi-turn agent RL, reporting +5.0 points on SWE-Bench and +9.3 on F...

  7. Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning

    cs.AI 2025-10 conditional novelty 5.0 of 10

    A plan-then-reason SFT plus a plan-quality reward in GRPO improves math-reasoning accuracy by small but consistent margins over GRPO and DAPO.

Reference graph

Works this paper leans on

32 extracted references · 14 canonical work pages · cited by 7 Pith papers

  1. [1]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    arXiv 2024

  2. [2]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    arXiv 2024

  3. [3]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    arXiv 2025

  4. [4]

    Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models

    Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In International Conference on Machine Learning, pages 29128–29163. PMLR, 2024

    work page 2024

  5. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  6. [6]

    Bandit based monte-carlo planning

    Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. InEuropean conference on machine learning, pages 282–293. Springer, 2006

    2006

  7. [7]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm.arXiv preprint arXiv:1712.01815, 2017

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm.arXiv preprint arXiv:1712.01815, 2017

    arXiv 2017

  8. [8]

    Monte carlo tree search: A review of recent modifications and applications.Artificial Intelligence Review, 56(3):2497–2562, 2023

    Maciej Świechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Mańdziuk. Monte carlo tree search: A review of recent modifications and applications.Artificial Intelligence Review, 56(3):2497–2562, 2023

    work page 2023

Show all 32 references
  1. [9]

    Truncated proximal policy optimization, 2025

    Tiantian Fan, Lingjun Liu, Yu Yue, Jiaze Chen, Chengyi Wang, Qiying Yu, Chi Zhang, Zhiqi Lin, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Bole Ma, Mofan Zhang, Gaohong Liu, Ru Zhang, Haotian Zhou, Cong Xie, Ruidong Zhu, Zhi Zhang, Xin Liu, Mingxuan Wang, Lin Yan, and Yonghui Wu. Tr...

    2025

  2. [10]

    Infinite sampling: Efficient and stable grouped rl training for large language models, 2025

    Liangyu Wang, Huanyi Xie, Xinhai Wang, Tianjin Huang, Mengdi Li, and Di Wang. Infinite sampling: Efficient and stable grouped rl training for large language models, 2025

    2025

  3. [11]

    Treerl: Llm reinforcement learning with on-policy tree search.arXiv preprint arXiv:2506.11902, 2025

    Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. Treerl: Llm reinforcement learning with on-policy tree search.arXiv preprint arXiv:2506.11902, 2025

    2025 arXiv

  4. [12]

    Segment policy optimization: Effective segment-level credit assignment in rl for large language models, 2025

    Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu. Segment policy optimization: Effective segment-level credit assignment in rl for large language models, 2025

    2025

  5. [13]

    Chain-of-thought prompting elicits reasoning in large language models.Advancesin neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advancesin neural information processing systems, 35:24824–24837, 2022

    2022

  6. [14]

    Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models.arXiv preprint arXiv:2501.03262, 2025

    Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models.arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  7. [15]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025

  8. [16]

    Measuring mathematical problem solving with the math dataset.NeurIPS, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.NeurIPS, 2021

    2021

  9. [17]

    7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient.https://hkust-nlp

    Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient.https://hkust-nlp. notion.site/simplerl-reason, 2025. Notion Blog. 15

    2025

  10. [18]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Sur- passing o1-preview with a 1.5b model by scaling rl. https://pretty-radio-b75.notion.site/ DeepScaleR-Surpa...

  11. [19]

    Aime 2024 problems, 2024

    MAA. Aime 2024 problems, 2024. Accessed: 2025-05-11

    2024

  12. [20]

    Amc 2023 problems, 2023

    MAA. Amc 2023 problems, 2023. Accessed: 2025-05-11

    2023

  13. [21]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35...

    2022

  14. [22]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.1...

    2024 arXiv

  15. [23]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  16. [24]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  17. [25]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2025

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2025

    2025

  18. [26]

    Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Suhong Moon, Kerem Dilmen, Monishwaran Maheswaran, Nicholas Lee, Michael W. Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Ets: Efficient tree search for inference-time scaling, 2025

    2025

  19. [27]

    Batchllm: Optimizing large batched llm inference with global prefix sharing and throughput-oriented token batching, 2025

    Zhen Zheng, Xin Ji, Taosong Fang, Fanghao Zhou, Chuanjie Liu, and Gang Peng. Batchllm: Optimizing large batched llm inference with global prefix sharing and throughput-oriented token batching, 2025

    2025

  20. [28]

    Multi-bin batching for increasing llm inference throughput, 2024

    Ozgur Guldogan, Jackson Kunde, Kangwook Lee, and Ramtin Pedarsani. Multi-bin batching for increasing llm inference throughput, 2024

    2024

  21. [29]

    Value-guided search for efficient chain-of-thought reasoning, 2025

    Kaiwen Wang, Jin Peng Zhou, Jonathan Chang, Zhaolin Gao, Nathan Kallus, Kianté Brantley, and Wen Sun. Value-guided search for efficient chain-of-thought reasoning, 2025

    2025

  22. [30]

    Spo: Multi-dimensional preference sequential alignment with implicit reward modeling, 2024

    Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, and Kaiqi Huang. Spo: Multi-dimensional preference sequential alignment with implicit reward modeling, 2024

    2024

  23. [31]

    Agentic reinforced policy optimization, 2025

    Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. Agentic reinforced policy optimization, 2025

    2025

  24. [32]

    First return, entropy-eliciting explore, 2025

    Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, and Zejun Ma. First return, entropy-eliciting explore, 2025. 16

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.