Pith. sign in

REVIEW 5 major objections 5 minor 41 references

Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A small external reward model reduces LLM tree-search cost to about one tenth while keeping or improving accuracy.

desk verdict Useful trick for pruning LLM tree search with a small reward model, but the headline 10x cost claim doesn't survive contact with the paper's own tables. read the letter →

arxiv 2506.00396 v1 pith:DXHY7QOI submitted 2025-05-31 cs.CL

Jiawei Gu , Shangsong Liang This is my paper
classification cs.CL
keywords speculativerewardmodelLLMdecision-makingtreesearchcostefficiencyprocesssupervisionconsistencyverificationextensibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the most expensive part of LLM decision-making—calling the generator to score every candidate step during tree search—can be offloaded to a much smaller, separately trained reward model. Its Speculative Reward Model (SRM) wraps existing DFS, BFS, and MCTS search strategies, assigns each candidate action a speculative reward, and uses a rejection-sampling step to prune weak branches before they are expanded. To frame the problem, the authors first introduce the 3E Criteria (Effectiveness, Efficiency, Extensibility) and report that conventional tree-search methods spend large multiples of compute for only a few points of accuracy. The paper reports that SRM and its fine-tuned variant SRM+ reduce time and token costs to roughly one-tenth of the original search framework on average across GSM8K, BlocksWorld, and FinQA, while keeping accuracy flat or raising it. If correct, the result makes test-time scaling through search substantially cheaper and suggests that small verifier models can steer much larger generators without retraining them.

What carries the argument

The load-bearing machinery is the Speculative Reward Model (SRM), a roughly 500M-parameter reward assigner trained with a pairwise ranking loss on weak step-level labels built from PRM800K (positive, negative, neutral), and optionally fine-tuned with strong search-derived rewards to become SRM+. The SRM is used in two interacting mechanisms. The Speculative Reward (SR) is the normalized score $R_\theta^{\mathrm{SRM}}(s_n,a_i^n)$; an action is accepted with probability $\min(1, L(P_{\mathrm{LLM}}(a_i^n|s_n))/L(R_\theta^{\mathrm{SRM}}(s_n,a_i^n)))$, so the generator's own probability mass serves as a prior and weak branches are rejected. Reward Consistency (RC), $\mathrm{RC}=1/(1+|SR-1|)\in[0,1]$, then ranks accepted branches by how closely external and internal signals agree, and the combined accumulated reward $\mathrm{SR}^\alpha\cdot\mathrm{RC}^{(1-\alpha)}$ selects the next state. Rejection sampling prunes suboptimal choices before expansion, and the ranking guides expansion toward promising nodes, which is the mechanism behind the reported cost reduction.

What would settle it

Profile an end-to-end MCTS+SRM run on GSM8K with instrumentation that meters the DeBERTa-v3-large reward model's forward passes separately. If the reported time and token numbers exclude those calls, the 1/10 cost claim is not a system-level comparison; if wall-clock time including SRM is not near one-tenth of unmodified MCTS time, the central efficiency claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a small external reward model can substitute for the LLM's internal self-evaluation during search. For each state $s_n$ and candidate action $a_i^n$, SRM produces a speculative reward $R_\theta^{\mathrm{SRM}}(s_n,a_i^n)=P_\theta(a_i^n|s_n,\mathrm{prompt}_1)$, and the action is accepted with probability $\min(1, L(P_{\mathrm{LLM}}(a_i^n|s_n))/L(R_\theta^{\mathrm{SRM}}(s_n,a_i^n)))$, where $L$ normalizes over candidates. Surviving actions are ranked by Reward Consistency, $\mathrm{RC}=1/(1+|SR-1|)$, which measures how well the external score agrees with the generator's own probability, and the cumulative reward $R_{\mathrm{accumulated}}=\mathrm{SR}^\alpha \cdot \mathrm{RC}^{(1-\alpha)}$ guides the next expansion. In the reported experiments, this prunes enough bad branches that MCTS+SRM on GSM8K with GPT-3.5-turbo reaches 80.5% accuracy at about a fifth of the token cost of MCTS alone (20.6K/0.9K versus 105.2K/2.5K), and similar reductions hold for DFS, BFS, BlocksWorld, and FinQA. The authors interpret this as evidence that external process supervision can stabilize search, prevent error propagation, and do so far more cheaply than LLM self-evaluation.

Load-bearing premise

The reported cost reductions assume that the time and token figures include the external reward model's own inference, but the paper lists only LLM prompt and completion tokens and does not state how SRM's compute is counted.

Editorial extensions

If this is right

  • Wrapping an existing tree search with SRM cuts its average time and token cost to roughly a tenth while holding accuracy flat or slightly improving it across GSM8K, BlocksWorld, and FinQA.
  • MCTS+SRM on GSM8K with GPT-3.5-turbo raises accuracy from 74.7% to 80.5% while using far fewer tokens, and MCTS+SRM+ on LLaMA-2-70B similarly nudges accuracy upward at a fraction of the cost.
  • SRM+ fine-tuned on MCTS-derived rewards transfers to a new domain with only 10% of the task reward data, improving BlocksWorld Hard accuracy from 0.51 to 0.54 and FinQA accuracy from 0.60 to 0.68 with GPT-3.5.
  • The generator LLM does not need retraining; SRM is a plug-and-play wrapper, so existing DFS, BFS, and MCTS pipelines can be accelerated by replacing the reward source.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported savings are truly end-to-end, the freed compute budget could be reinvested in larger candidate sets ($K$) or deeper search ($N$), which the paper does not test, and this could push accuracy further on the same budget.
  • The framework suggests a general pattern for test-time scaling: use a cheap small model to propose and rank candidate steps and reserve the large generator for the most promising branches, an approach that may extend beyond math, planning, and financial reasoning to coding or tool-use agents.
  • Because SRM's training relies on PRM800K-style weak labels and MCTS-style strong labels, its benefit in a new domain depends on whether such process-level supervision can be obtained; a stress test would apply SRM to a domain with no search-derived rewards and measure how much of the gain survives.
  • A useful follow-up experiment would compare SRM against a non-speculative external scorer with no rejection sampling, to isolate whether the acceptance ratio or merely the external scoring produces the cost reduction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Speculative Reward Model (SRM), a plug-and-play framework that attaches a small external reward model (DeBERTa-v3-large) to LLM tree-search paradigms (DFS, BFS, MCTS). SRM assigns a speculative reward to each candidate action and uses a rejection-sampling rule, combined with a Reward Consistency term, to prune unpromising branches and prioritize promising ones. A fine-tuned version, SRM+, is trained on strong rewards collected from MCTS search. Experiments on GSM8K, BlocksWorld, and FinQA compare time and token costs and accuracy against search baselines, and the abstract claims that SRM reduces costs to 1/10 of the original search framework on average while maintaining effectiveness.

Significance. If the central claims are established, the paper would make a useful practical contribution: an external reward model that reduces the number of LLM calls in tree search, applied across multiple search algorithms and tasks, with code released. The base SRM is trained on PRM800K, which is independent of the downstream search evaluation, and the idea of reusing speculative-sampling-style acceptance rules for search pruning is interesting. However, the headline quantitative claim is not currently supported by the reported tables, the cost accounting omits the SRM's own inference cost, and the SRM+ evaluation has a circularity concern because SRM+ is trained on MCTS-generated rewards and then compared against MCTS. These issues are load-bearing for the paper's main message, so the claims need substantial revision and additional experiments before publication.

major comments (5)
  1. [Abstract and §4.2, Table 1] The claim that 'SRM reduces costs to 1/10 of the original search framework on average' is not supported by the numbers in Table 1. For GPT-3.5-turbo on GSM8K, the time reductions are 150→34.7s (4.3x), 180→44s (4.1x), 66.4→44s (1.5x), and 122.6→45.2s (2.7x); the corresponding token reductions are roughly 3.8x, 3.9x, 7.3x, and 5.1x in prompt tokens. No weighting of these rows, or of the LLaMA rows in Table 2, yields a transparent average of 10x; the geometric mean of the Table 1 time ratios is about 3x. The paper should either correct the abstract and contributions to state the observed reductions (e.g., roughly 3-5x for GPT-3.5, with larger reductions for some LLaMA configurations) or provide a precise, reproducible aggregation rule that justifies the '1/10' statement.
  2. [§4.2, Tables 1-4; §A.1] The efficiency comparison does not state whether the reported time and token costs include the inference cost of the SRM model itself. Table 1 reports token cost as 'LLM prompt/completion tokens' and says nothing about the DeBERTa-v3-large forward passes that SRM must run for every candidate action at every node. Since SRM's inference adds both wall-clock time and compute, the reported cost reductions are not end-to-end unless this overhead is included. The paper should state explicitly how SRM's compute is accounted for, or re-report all cost numbers as end-to-end totals; without this, the '1/10 cost' claim is not an apples-to-apples system comparison.
  3. [§3, Eq. (after Eq. 4)] The cumulative reward is defined as Raccumulated = SR^α · RC^(1−α), with α called a hyperparameter, but no value or sensitivity analysis is given. This quantity is central to the selection mechanism, and Table 3 reports results for 'SR^α·RC^(1−α)' without specifying α. The paper should report the α used in each experiment, describe how it was chosen, and ideally include a small sensitivity study; otherwise the main ablation in Table 3 is not reproducible.
  4. [§4.3, §4.2, and Appendix A.2.2] The evaluation of SRM+ has a partial circularity. Appendix A.2.2 and §4.2 state that the RewardTuning dataset is built from strong rewards obtained by MCTS search, and SRM+ is fine-tuned on this data. Tables 2 and 4 then compare MCTS+SRM+ against plain MCTS and attribute the improvement to SRM+. If the RewardTuning data was collected from the same tasks (or the same test questions) that are later used for evaluation, the comparison is not independent. The paper should specify whether the reward-tuning data and the evaluation data are disjoint, and should also compare SRM+ against a reward model trained on a different source of strong rewards, to show that the reported gains are not simply the reward model copying MCTS's own value estimates.
  5. [§4.2, Table 2] Table 2 reports that results were averaged over 10 runs, but no variance, confidence intervals, or significance tests are given. Many of the claimed effectiveness improvements are very small, e.g., LLaMA-2-70B on DFS from 0.52 to 0.54, or LLaMA-2-13B on MCTS from 0.30 to 0.29 for SRM and 0.29 for SRM+. These differences are within normal sampling noise for 10 runs, especially at temperature 0.8, so the claim that SRM 'maintains effectiveness' is not statistically supported. The paper should add error bars or significance tests, at least for the main effectiveness comparisons.
minor comments (5)
  1. [§3, Eq. (3)] The normalization operator is defined with the symbol M(f(x)) in Eq. (3), but the preceding text and Eq. (2) use L(·); the notation should be made consistent.
  2. [§3] The expression for Raccumulated is typeset incompletely: 'Raccumulated = SR^α · RC^(1−α)' is followed by 'where α is a hyperparameter' without a complete sentence or clear exponent formatting. Please rewrite the formula and define all terms.
  3. [Limitations] The Limitations section contains an incomplete sentence: 'SRM needs to be fine-tuned with task reward data to improve the corresponding performance on the specific task. relies on external reward models...' The first sentence appears to be missing a subject or should be merged with the following clause.
  4. [Table 2] The column header 'LLaMA-33B' is inconsistent with the naming of the other columns ('LLaMA-2-70B', 'LLaMA-2-13B'); please clarify whether this is LLaMA-2-33B or another model.
  5. [§1 and Table 1] The 'Ext.' column in Table 1 is not clearly defined: all rows except Toolchain* and QwQ are marked '✓', but the criterion is only described informally in the text. Please state the exact meaning of the check mark and how it was determined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SRM is trained on an external process-reward dataset and evaluated empirically; the main concerns are cost-accounting gaps and SRM+'s dependence on MCTS-generated labels, not definitional reductions.

full rationale

The derivation chain is not circular: SRM's base training uses the external PRM800K process-reward dataset, and the speculative acceptance rule (Eq. 2) is an empirical design adapted from external speculative-sampling work, not a quantity fitted to the reported outcomes. The few self-citations (Liang et al., 2017; Liang, 2019; Xian et al., 2025) are parenthetical or forward-looking and carry no argumentative weight. The strongest circularity-adjacent concern is that SRM+ is fine-tuned on strong rewards harvested from tree-based search, explicitly MCTS (Appendix A.2.2 and Section 4.2), before being evaluated with MCTS in Table 2; this makes the SRM+ effectiveness comparison dependent on the baseline's own reward estimates, but the paper does not derive the accuracy gain from that training, so it is a data-independence weakness rather than a definitional reduction. The abstract's 1/10-cost claim is also not uniformly supported by Table 1 (GPT-3.5 reductions are roughly 3-5x), and the token metric counts only LLM prompt/completion tokens while omitting the DeBERTa-v3-large SRM's inference cost; the Limitations section itself concedes dependence on external reward models. These are accounting and measurement problems, not circularity: no reported result is made equal to its input by construction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the framework relies on a trained reward model and existing search algorithms. The main free parameters are the alpha weighting, the search settings K and N, and the fitted weights of SRM/SRM+.

free parameters (3)
  • alpha (SR/RC trade-off) = not reported
    Raccumulated = SR^alpha * RC^(1-alpha), defined in Section 3; no value or sensitivity study is given, and the main results in Table 3 depend on this weighting.
  • K and N (candidate actions and depth) = K=4, N=5
    Set by hand in Section 4.1 for all tasks; results may depend on these choices.
  • SRM model weights = trained on PRM800K, fine-tuned on MCTS rewards
    The reward model parameters are fitted to process reward labels and search rewards, so the central claim relies on these fitted values.
assumptions (3)
  • domain assumption Speculative acceptance rule (Eq. 2) is a valid pruning mechanism that preserves decision quality.
    Borrowed from speculative decoding and applied to action pruning; the paper provides no theoretical guarantee, only empirical results.
  • domain assumption A 500M-parameter reward model can predict the quality of LLM-generated reasoning steps.
    The whole framework assumes that the small model's scores are informative enough to guide search; success depends on this empirical premise.
  • domain assumption PRM800K step-level labels are reliable process supervision.
    SRM is trained on weak labels derived from PRM800K (Appendix A.2.1); if these labels are noisy or biased, the reward model inherits those errors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively." pith.science (2026). https://pith.science/paper/DXHY7QOI

@misc{pith2026250600396,
  author       = {Pith},
  title        = {Pith review of: Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DXHY7QOI}},
  note         = {Machine review of arXiv:2506.00396}
}
read the original abstract

Effective decision-making in Large Language Models (LLMs) is essential for handling intricate tasks. However, existing approaches prioritize performance but often overlook the balance between effectiveness and computational cost. To address this, we first introduce the 3E Criteria to systematically assess the cost-effectiveness of search strategies, revealing that existing methods often trade significant efficiency for marginal performance gains. To improve LLM decision-making while maintaining efficiency, we propose the Speculative Reward Model (SRM), a plug-and-play framework that seamlessly integrates with existing search strategies. Specifically, SRM employs an external reward assigner to predict optimal actions, reducing reliance on LLMs' internal self-evaluation. And a speculative verification mechanism is used to prune suboptimal choices and guide the search toward more promising steps. We evaluate SRM on several complex decision-making tasks including mathematical reasoning, planning and numerical reasoning in specialized domains. Experimental results show that SRM reduces costs to 1/10 of the original search framework on average while maintaining effectiveness.

Figures

Figures reproduced from arXiv: 2506.00396 by the authors.

Figure 1
Figure 1. An example in GSM8K (K = 4, N = 5), where our SRM uniquely solves the case correctly across all baselines in 10 tests while achieving the lowest time and token costs. The decision-making process showcases SRM’s pruning via Speculative Reward (SR), with green actions for acceptance and red for rejection. By SR, searching bypasses bad nodes and expands promising ones first. The selection strategy is determined by Rewa… view at source ↗
Figure 2
Figure 2. SRM was trained using the PRM800K dataset, in conjunction with LLMs, to provide weak Speculative Rewards (SR) for each action. Subsequently, SRM+ underwent fine-tuning with the RewardTuning dataset, enabling it to generate strong SR for task-specific ac￾tions. Various actions are denoted by dots, with the intensity of their green hue indicating the magnitude of the Reward Consistency (RC) on each accepted node. A de… view at source ↗
Figure 3
Figure 3. Comparison of the effectiveness and effi [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Example of an efficient selection process [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 6
Figure 6. Figure 6: Bad Case of MCTS Decision-making {“state”: “In convex quadrilateral $ABCD$, $\\angle A = \\angle C$, $AB=CD=180$, and $AD \\ne BC$. The perimeter of $ABCD$ is 640. Find $\\cos A$.”, “action”: [“Let $AD=a$ and $BC=b$. So, we know that $a+b+360=640$. So, $a+b=280$.”, “Le…
Figure 7
Figure 7. Figure 7: The process of generating weak reward data pairs. As the example showed, we process the data from [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: The process of generating strong reward data [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 11 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. 2023. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687

    arXiv 2023

  4. [4]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318

    arXiv 2023

  5. [5]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. 2021. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122

    arXiv 2021

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  7. [7]

    DeepSeek. 2024. Deepseek-r1-lite-preview: Unleashing supercharged reasoning power. https://api-docs.deepseek.com/news/news1120. Accessed: 2024-12-29

    work page 2024

  8. [8]

    Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. 2023. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint arXiv:2311.04254

    arXiv 2023

Show all 41 references
  1. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992

    2023 arXiv

  3. [11]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798

    2023 arXiv

  4. [12]

    Ali Jahan, Kevin L Edwards, and Marjan Bahraminasab. 2016. Multi-criteria decision analysis for supporting the selection of engineering materials in product design. Butterworth-Heinemann

    2016

  5. [13]

    Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. 2023. Reward design with language models. arXiv preprint arXiv:2303.00001

    2023 arXiv

  6. [14]

    Shangsong Liang. 2019. Collaborative, dynamic and diversified user profiling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4269--4276

    2019

  7. [15]

    Shangsong Liang, Emine Yilmaz, Hong Shen, Maarten De Rijke, and W Bruce Croft. 2017. Search result diversification in short text streams. ACM Transactions on Information Systems (TOIS), 36(1):1--35

    2017

  8. [16]

    OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Bel...

    2023 arXiv

  9. [17]

    OpenAI. 2024. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/. [Accessed 19-09-2024]

    2024

  10. [18]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 3...

    2022

  11. [19]

    Panupong Pasupat and Percy Liang. 2015. https://doi.org/10.3115/v1/P15-1142 Compositional semantic parsing on semi-structured tables . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natur...

    2015 doi

  12. [20]

    Martin L Puterman. 1990. Markov decision processes. Handbooks in operations research and management science, 2:331--434

    1990

  13. [21]

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. 2024. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199

    2024 arXiv

  14. [22]

    Qwen. 2024. https://qwenlm.github.io/blog/qwq-32b-preview/ Qwq: Reflect deeply on the boundaries of the unknown

    2024

  15. [23]

    QwenTeam. 2024. Qwq-32b: Embracing the power of reinforcement learning. https://qwenlm.github.io/blog/qwq-32b/

    2024

  16. [24]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. http://arxiv.org/abs/2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    2024 arXiv

  17. [25]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  18. [26]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  19. [27]

    Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. 2025. Prmbench: A fine-grained and challenging benchmark for process-level reward models. arXiv preprint arXiv:2501.03124

    2025 arXiv

  20. [28]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  21. [29]

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Thirty-seventh Conference on Neural Information Processing Sy...

    2023

  22. [30]

    Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2024. Alphazero-like tree-search can guide large language model decoding and training. In Forty-first International Conference on Machine Learning

    2024

  23. [31]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  24. [32]

    Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, et al. 2024. Chain-of-table: Evolving tables in the reasoning chain for table understanding. arXiv preprint arXiv:2401.04398

    2024 arXiv

  25. [33]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824--24837

    2022

  26. [34]

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.257 Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation . In Findings of the Association for Computational Ling...

    2023 doi

  27. [35]

    Ziting Xian, Jiawei Gu, Lingbo Li, and Shangsong Liang. 2025. Molrag: unlocking the power of large language models for molecular property prediction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics

    2025

  28. [36]

    Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. 2025. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09686

    2025 arXiv

  29. [37]

    Han Xu, Jingyang Ye, Yutong Li, and Haipeng Chen. 2024. Can speculative sampling accelerate react without compromising reasoning quality? In The Second Tiny Papers Track at ICLR 2024

    2024

  30. [38]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601

    2023 arXiv

  31. [39]

    Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. 2024. Quiet-star: Language models can teach themselves to think before speaking. In First Conference on Language Modeling

    2024

  32. [40]

    Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. 2023. Toolchain*: Efficient action space navigation in large language models with a* search. arXiv preprint arXiv:2310.13227

    2023 arXiv

  33. [41]

    Mingchen Zhuge, Changsheng Zhao, Dylan R Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. 2025. Agent-as-a-judge: Evaluating agents with agents

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.