Pith. sign in

REVIEW 4 major objections 7 minor 49 references

Reward Model Generalization for Compute-Aware Test-Time Reasoning

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A process reward model's generalization error dictates the test-time compute needed for reliable reasoning.

desk verdict The core theory doesn't hold up—Theorem 4.5's independence assumption is invalid—but CATS is a sensible method worth a serious look. read the letter →

arxiv 2505.18065 v1 pith:S6UOY474 submitted 2025-05-23 cs.LG

Zeen Song , Wenwen Qiang , Siyu Zhao , Changwen Zheng , Gang Hua This is my paper
classification cs.LG
keywords test-timecomputeoptimalityprocessrewardmodelPAC-Bayesgeneralizationboundgapcompute-awaretreesearchactor-criticcontrolmathematicalreasoningLLMinferencescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the generalization error of a process reward model (PRM) is a primary driver of test-time compute optimality: under a fixed inference budget, the number of candidate reasoning paths needed to reach a target accuracy shrinks as the PRM's error bound shrinks. The proof path is a lower bound on final-answer accuracy in which the chance of selecting the correct answer is at least the coverage probability of sampling a good path times a mis-ranking penalty that depends on the reward gap divided by the PRM's error. From that bound the paper derives a compute-budget corollary: to guarantee accuracy $\alpha$, the sampling coverage must satisfy $p_{N,\tau}(q) \ge \alpha / [1 - \delta - (N-1)\exp(-\gamma(q)^2/(8\varepsilon^2))]$. Because the true error $\varepsilon$ is unobservable, the paper proposes parameter sparsity as a proxy and builds CATS, an actor-critic search controller that widens the reward gap by adaptively choosing how many paths to sample and keep. Experiments on MATH-500 and AIME24 across several policy models and PRMs show CATS outperforming Best-of-N, beam search, and majority voting, consistent with the predicted dependence on $\varepsilon$.

What carries the argument

The load-bearing object is the reward gap $\gamma(q) = R^*(q,h^*) - \max_{h \ne h^*} R^*(q,h)$, the difference in true reward between the best candidate path and the runner-up; the argument shows that a mis-ranking happens only when the PRM's bounded errors let a competitor overcome this gap. That mis-ranking tail is carried by the exponential term $\exp(-\gamma(q)^2/(8\varepsilon^2))$ in Theorem 4.5, where $\varepsilon$ is the uniform bound on the PRM's absolute deviation. The generalization bound that supplies $\varepsilon$ is the PAC-Bayes inequality of Theorem 4.2, which upper-bounds the expected absolute reward error by $\sqrt{(\mathrm{KL}(Q\|P) + \log(n/\delta))/(2(n-1))}$; the paper then proposes parameter sparsity, through the bound $\varepsilon \le \sqrt{(c\|\hat\phi\|_0 \log d + \log(n/\delta))/(2(n-1))}$, as an observable proxy for that error. CATS is the mechanism that operationalizes the bound: an actor-critic controller whose reward function penalizes compute, rewards reward-gap widening, and rewards high PRM scores.

What would settle it

Sample candidate sets from a fixed policy model, score them with a PRM, and compute the empirical uniform deviation $\hat\varepsilon = \sup_{h\in H}|R_\varphi(q,h) - R^*(q,h)|$ on held-out questions with known per-step labels; also measure the correlation of deviations across paths that share a prefix. If $\hat\varepsilon$ routinely exceeds the value used in Theorem 4.5, or if the deviations are strongly correlated, the stated accuracy floor cannot be expected to hold.

Watch

Extended reading notes

Core claim

The paper's central claim, stated as Theorem 4.5, is that for a fixed question $q$ with $N$ independently sampled reasoning paths, the probability that the PRM-selected path gives the correct answer is at least $p_{N,\tau}(q)[1 - \delta - (N-1)\exp(-\gamma(q)^2/(8\varepsilon^2))]$, where $p_{N,\tau}(q)$ is the probability that at least one sampled path has true reward above threshold $\tau$, $\gamma(q)$ is the gap between the true-best and the second-best path rewards, and $\varepsilon$ is a uniform bound on $|R_\varphi(q,h) - R^*(q,h)|$ over the candidate set. The argument bounds the mis-ranking event: with deviations bounded by $\varepsilon$, a competitor must overcome a gap of roughly $\gamma(q) - 2\varepsilon$, and a standard tail bound for bounded independent deviations produces the exponential term. Corollary 4.6 turns this into a budget statement: guaranteeing accuracy $\alpha$ forces the sampling coverage $p_{N,\tau}(q)$ to be at least $\alpha/[1 - \delta - (N-1)\exp(-\gamma(q)^2/(8\varepsilon^2))]$. The paper's conclusion is that smaller PRM generalization error and larger reward gap are the two levers that cut required test-time compute, and it builds CATS to widen the reward gap while using sparsity as a proxy for the error.

Load-bearing premise

The argument relies on treating the theoretical bound on the reward model's average error as a bound on the worst-case error for every candidate reasoning path, and on assuming those path-wise errors are independent and centered; that equivalence is asserted, not proven.

Editorial extensions

If this is right

  • Tightening the PRM generalization bound $\varepsilon$ raises the accuracy floor exponentially in $1/\varepsilon^2$ for a fixed number of sampled paths $N$.
  • To reach a target accuracy $\alpha$, the required sampling coverage grows like $\alpha/[1 - \delta - (N-1)\exp(-\gamma(q)^2/(8\varepsilon^2))]$, so better-generalizing PRMs translate directly into a smaller compute budget.
  • The reward gap $\gamma(q)$ is the controllable lever at inference: adjusting sampling hyperparameters such as top-$p$, top-$k$, and temperature to separate good from bad paths improves accuracy without changing the PRM.
  • Parameter sparsity can serve as an observable proxy for the unobservable generalization error, enabling adaptation at test time.
  • CATS, as an actor-critic policy over sampling and retention actions, is reported to outperform Best-of-N, beam search, and majority voting on MATH-500 and AIME24 across the tested policy and PRM combinations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the authors could test the theory directly by artificially degrading a strong PRM (for example, adding calibrated label noise) and checking whether the required number of paths grows roughly as predicted by the $\exp(-\gamma(q)^2/(8\varepsilon^2))$ term.
  • My inference: because the theorem assumes deviations for different candidate paths are independent, it likely understates mis-ranking risk when candidates share reasoning prefixes; an experiment on prefix-sharing candidate sets would show whether practical margins need to be wider than the formula suggests.
  • My inference: the reward-gap argument should transfer beyond math word problems to other verifier-guided planning settings, since the proof uses only sampled paths, true rewards, and gaps, not any language-model-specific structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper studies how the generalization error of a process reward model (PRM) affects test-time compute optimality in external test-time reasoning for large language models. The authors first derive a PAC-Bayes generalization bound for PRMs (Theorem 4.2), then use it to state an answer-accuracy lower bound in terms of the reward gap, coverage probability, and a uniform error bound ε (Theorem 4.5), and from this derive a target-accuracy constraint on the sampling budget (Corollary 4.6). They then propose Compute-Aware Tree Search (CATS), an actor-critic method that uses PRM parameter sparsity as a proxy for generalization error and adaptively controls search hyperparameters. Experiments on MATH-500 and AIME24 with several policies and PRMs report that CATS outperforms Best-of-N, beam search, and majority voting across budgets.

Significance. The question addressed is timely and important: understanding how verifier quality controls the compute-accuracy trade-off in inference-time scaling. If the theoretical results were valid, the connection between PAC-Bayes generalization bounds, reward margin, and the number of sampled paths would be a useful contribution. The empirical component is substantial: CATS is a well-motivated controller, the experiments cover multiple policy models and PRMs, and the appendices include ablations for the sparsity proxy, reward coefficients, and discount factor, as well as pseudocode. However, the load-bearing steps of the theory are not sound: the proof of Theorem 4.2 contains an incorrect optimization, the assumptions of Theorem 4.5 are incompatible with the actual random structure of a fixed PRM, and the bridge from the PAC-Bayes expected-loss bound to the per-candidate uniform error ε is not justified. These problems undermine the central claim that a smaller PRM generalization error provably reduces the number of candidate paths required for a target accuracy.

major comments (4)
  1. [Appendix B, proof of Theorem 4.2] The λ optimization in the proof is incorrect. Equation (21) gives λ E_Q[L_D − L_S] ≤ λ²/(8n) + KL(Q∥P) + log(1/δ). For this inequality the optimal λ is λ* = 4n E_Q[L_D − L_S], not the claimed λ* = 4√((n−1)/2). Substituting the claimed value leaves an uncancelled term √((n−1)/2)/(2n), and the final bound with denominator 2(n−1) and log(n/δ) does not follow from the preceding algebra. The theorem may be repairable by citing a standard PAC-Bayes bound, but the derivation as written does not prove Eq. (4).
  2. [Section 4.2, Theorem 4.5 assumptions] The assumption that, conditioned on H, the deviations Δ_h = R_φ(q,h) − R*(q,h) are independent and mean-zero is not justified for the setting of the paper. For a fixed trained PRM, φ is a single parameter vector, so conditioned on H the deviations are deterministic; unconditionally, the Δ_{h_i} are independent only because the h_i are independent, but they need not be mean-zero since E_{h∼π_θ}[R_φ(q,h) − R*(q,h)] is not zero in general. If φ is instead drawn from the PAC-Bayes posterior Q, all paths share the same draw of φ, so the deviations are coupled. Consequently, the Hoeffding bound in Appendix C has no valid target, and the exponential term (N−1)exp(−γ(q)²/(8ε²)) in Eq. (6) is unsupported.
  3. [Section 4.2, Eq. (5) to Theorem 4.5] The sentence 'Define this upper-bound as ε' is not a valid step. Equation (5) bounds the expected absolute loss E_{φ∼Q}[L_D(φ) − L_S(φ)] averaged over the data distribution D, whereas Theorem 4.5 requires a high-probability uniform error bound ε on sup_{h∈H}|R_φ(q,h) − R*(q,h)| over a finite candidate set for a single question q. No argument is given that the PAC-Bayes bound controls this sup-norm quantity or that the event G has probability at least 1−δ. This missing bridge is load-bearing: it is precisely the link that would support the claim that smaller PRM generalization error reduces the required number of samples, and it is not established.
  4. [Appendix C, proof of Theorem 4.5] The final algebra of the proof does not match the statement. The derivation gives Pr(E1 ∪ E2) ≤ (1 − p) + p(N−1)exp(−γ²/8ε²) + δ, which yields success probability at least p − p(N−1)exp(−γ²/8ε²) − δ, i.e., p[1 − (N−1)exp(−γ²/8ε²)] − δ, not p[1 − δ − (N−1)exp(−γ²/8ε²)] as stated in Eq. (6). In addition, the intermediate statement that a competitor must overcome a gap of at least γ(q) − 2ε is not the quantity that appears in the Hoeffding exponent. The stated bound therefore does not follow from the proof as written.
minor comments (7)
  1. [Section 5.1, Eq. (8)] The dimension d in the term log d is never defined, and the constant c is introduced without explanation or calibration; the formal step from a sparsity-based prior to this specific bound is omitted.
  2. [Section 4.2, Assumption 4.4] The notation is inconsistent: Assumption 4.4 states lim_{N→∞} p_N(q) = 1, but the quantity used throughout the rest of the section is p_{N,τ}(q).
  3. [Section 5.2 and Algorithm 1] The PRM parameters and the critic parameters are both denoted φ (R_φ and V_φ), which is confusing; the critic should use a different symbol, e.g., V_ξ as in Eq. (10).
  4. [Section 6.2] The comparison with Best-of-N and Beam Search uses the number of candidate paths as the shared budget, but CATS's adaptive tree search may consume different amounts of compute per path; the paper should state how the budget is matched and ideally report token counts or wall-clock time.
  5. [Appendix H.2] The text says 'using Qwen2.5-Math-PRM-7B as the policy model,' which appears to be a typo; the policy in the corresponding experiments is Qwen2.5-Instruct-7B, and the intended model should be clarified.
  6. [Figures 1–2 and Tables 3–6] The empirical results are reported without error bars or significance tests; given that several ablations show differences of about one percentage point, the claim of consistent improvement would be stronger with variance estimates or multiple seeds.
  7. [Appendix A] The Limitations section asserts that the assumptions 'do not undermine the validity of the analysis,' but in light of Major Comments 2 and 3 this assertion is not supported; the appendix should be expanded to address the independence and uniform-error assumptions rather than dismissing them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the theory is conditional and CATS is a design motivated by it; the central weaknesses are unsupported bridges and proof gaps, not equivalence to inputs.

full rationale

The paper's derivation chain is conditional: Theorem 4.2 is a standard PAC-Bayes bound proved in Appendix B from Hoeffding and Donsker–Varadhan; Theorem 4.5 is stated under explicit assumptions (sup-norm error event G, independent mean-zero deviations, coverage and threshold assumptions) and its conclusion is a function of those assumed quantities. ε is an assumption parameter, not a value fitted to accuracy data, so the bound is not an output that has been fed back as an input. The informal claim that Eq. (5) 'Define this upper-bound as ε' is an unsupported identification (the PAC-Bayes bound controls an expectation, not a sup-norm over H), and the independence of Δ_h for a fixed φ is questionable; these are correctness gaps, not circularity. The sparsity proxy of Section 5.1 is introduced as a heuristic surrogate with empirical support in Appendix G, not as a fitted prediction. CATS's use of a margin term in its reward is design motivation from Theorem 4.5, and the method is evaluated against external benchmarks (MATH-500, AIME24) with no fitted parameter renamed as a prediction. No load-bearing self-citation or imported uniqueness theorem appears. The Limitations appendix asserts assumptions may not hold but are not undermining; that is an unsupported limitation claim, but it does not make the derivation circular.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

The central claim depends on a chain: PAC-Bayes bound on expected loss, an unjustified identification of that bound with a uniform per-candidate error ε, strong independence and zero-mean assumptions on PRM deviations, and a sparsity-based proxy for ε. The fitted reward coefficients and discount factor are method-level, not theory-level, parameters.

free parameters (4)
  • λ_c, λ_m, λ_r = 0.2, 0.5, 0.3
    Coefficients in the CATS reward function (Eq. 9); selected by ablation on MATH-500 (Appendix H.2).
  • discount factor γ = 0.9
    MDP discount factor in CATS; selected by ablation (Appendix H.2).
  • constant c in sparsity bound = not specified
    Unspecified constant in Eq. 8 that converts parameter count to a PAC-Bayes KL term; no derivation.
  • sparsity threshold = 1e-4
    Parameters below 1e-4 are counted as sparse in Appendix G; arbitrary choice that affects the proxy.
assumptions (6)
  • domain assumption Data samples (q, h, y) are drawn i.i.d. from distribution D.
    Assumption 4.1, Section 4.1.
  • domain assumption There exists threshold τ such that R*(q,h) ≥ τ implies a(h,q) = a*(q).
    Assumption 4.3, Section 4.2.
  • domain assumption As N grows, probability of sampling at least one high-reward path tends to 1.
    Assumption 4.4, Section 4.2.
  • domain assumption Conditioned on H, deviations Δ_h = R_φ - R* are independent, mean-zero, and |Δ_h| ≤ ε a.s.
    Theorem 4.5, second bullet.
  • standard math PAC-Bayes machinery: Donsker-Varadhan inequality and Hoeffding's inequality.
    Appendix B proof of Theorem 4.2.
  • ad hoc to paper log(1/P(φ)) can be approximated by c·||φ||_0·log d.
    Section 5.1, Eq. 8; the approximation is asserted without derivation.
invented entities (1)
  • R* (ground-truth reward function)
    purpose: Defines reward gap γ(q) and threshold τ for the correctness bound.
    R* is never observed or estimated; it is a theoretical construct introduced in Section 4.2 to separate PRM error from path quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reward Model Generalization for Compute-Aware Test-Time Reasoning." pith.science (2026). https://pith.science/paper/S6UOY474

@misc{pith2026250518065,
  author       = {Pith},
  title        = {Pith review of: Reward Model Generalization for Compute-Aware Test-Time Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S6UOY474}},
  note         = {Machine review of arXiv:2505.18065}
}
read the original abstract

External test-time reasoning enhances large language models (LLMs) by decoupling generation and selection. At inference time, the model generates multiple reasoning paths, and an auxiliary process reward model (PRM) is used to score and select the best one. A central challenge in this setting is test-time compute optimality (TCO), i.e., how to maximize answer accuracy under a fixed inference budget. In this work, we establish a theoretical framework to analyze how the generalization error of the PRM affects compute efficiency and reasoning performance. Leveraging PAC-Bayes theory, we derive generalization bounds and show that a lower generalization error of PRM leads to fewer samples required to find correct answers. Motivated by this analysis, we propose Compute-Aware Tree Search (CATS), an actor-critic framework that dynamically controls search behavior. The actor outputs sampling hyperparameters based on reward distributions and sparsity statistics, while the critic estimates their utility to guide budget allocation. Experiments on the MATH and AIME benchmarks with various LLMs and PRMs demonstrate that CATS consistently outperforms other external TTS methods, validating our theoretical predictions.

Figures

Figures reproduced from arXiv: 2505.18065 by the authors.

Figure 1
Figure 1. The comparison results on the MATH-500 dataset for different policy models and PRMs. [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. The comparison results on the AIME24 dataset for different policy models and PRMs. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Full results on the MATH500 dataset. G Analysis of Sparsity in PRM In this section, we evaluate the validity of using sparsity as a proxy for reward model generalization error. Specifically, we consider two sparsity-based indicators: the overall parameter sparsity of the model and the sparsity of its output layer. The sparsity is calculated by counting the ratio of parameters with values smaller than 1 × 10−4 . To a… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Full results on the AIME24 dataset. H Ablation Study H.1 The Role of Parameter Sparsity in CATS To investigate the impact of parameter sparsity in the CATS framework, we conduct an ablation study comparing model performance with and without the parameter sparsity inclu…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 27 canonical work pages

  1. [1]

    Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    work page 2022

  2. [2]

    Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    2022

  3. [3]

    Griffiths, Yuan Cao, and Karthik R

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R. Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. InThirty-Seventh Conference on Neural Information Processing Systems, November 2023

    work page 2023

  4. [4]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/ learning-to-reason-with-llms/, 2024. Accessed: 2025-04-29

    work page 2024

  5. [5]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 2025

    DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 2025

    2025

  6. [6]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haoz...

    work page 2025

  7. [7]

    Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning. InThe Thirteenth International Conference on Learning Representations, October 2024

    work page 2024

  8. [8]

    Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling, February 2025

    Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling, February 2025

    work page 2025

Show all 49 references
  1. [9]

    Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving. In The Thirteenth International Conference on Learning Representations, October 2024. 10

    2024

  2. [10]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, and Piotr Nyczyk. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI Conferenc...

    2024

  3. [11]

    Yuxiao Qu, Matthew Y . R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing Test-Time Compute via Meta Reinforce- ment Fine-Tuning, March 2025

    2025

  4. [12]

    Let’s Verify Step by Step, May 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step, May 2023

    2023

  5. [13]

    Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y . Wu, and Zhifang Sui. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations, February 2024

    2024

  6. [14]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  7. [15]

    AIME 2024

    AI-MO. AIME 2024. https://huggingface.co/datasets/AI-MO/ aimo-validation-aime, July 2024. Accessed 2024-07

    2024

  8. [16]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  9. [17]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Llama 3 to connect 2024: Vision, edge, and mobile devices.https://ai.meta.com/ blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ , 2024

    Meta AI. Llama 3 to connect 2024: Vision, edge, and mobile devices.https://ai.meta.com/ blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ , 2024. Accessed: 2025-05-12

    2024

  11. [19]

    The Lessons of Developing Process Reward Models in Mathematical Reasoning, January 2025

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The Lessons of Developing Process Reward Models in Mathematical Reasoning, January 2025

    2025

  12. [20]

    Rlhf workflow: From reward modeling to online rlhf

    Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024

    2024 arXiv

  13. [21]

    Skywork-o1 open series

    Skywork o1 Team. Skywork-o1 open series. https://huggingface.co/Skywork, Novem- ber 2024. Accessed: 2025-05-12

    2024

  14. [22]

    Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models, April 2025

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models, April 2025

    2025

  15. [23]

    Self- Refine: Iterative Refinement with Self-Feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- Refine: Iterati...

    2023

  16. [24]

    Self-critiquing models for assisting human evaluators.arXiv preprint arXiv:2206.05802, 2022

    William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators.arXiv preprint arXiv:2206.05802, 2022

    2022 arXiv

  17. [25]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, April 2024. 11

    2024

  18. [26]

    Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models

    Bilgehan Sel, Ahmad Tawaha, Vanshaj Khattar, Ruoxi Jia, and Ming Jin. Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models. InForty-First International Conference on Machine Learning, June 2024

    2024

  19. [27]

    Automatic Chain of Thought Prompting in Large Language Models, October 2022

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic Chain of Thought Prompting in Large Language Models, October 2022

    2022

  20. [28]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V . Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, December 2024

    2024

  21. [29]

    Solving math word problems with process-and outcome-based feedback.arXiv preprint arXiv:2211.14275, 2022

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback.arXiv preprint arXiv:2211.14275, 2022

    2022 arXiv

  22. [30]

    Self-Evaluation Guided Beam Search for Reasoning, October 2023

    Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. Self-Evaluation Guided Beam Search for Reasoning, October 2023

    2023

  23. [31]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. InThe Eleventh International Conference on Learning Representations, September 2022

    2022

  24. [32]

    Sparsity-aware generalization theory for deep neural networks

    Ramchandran Muthukumar and Jeremias Sulam. Sparsity-aware generalization theory for deep neural networks. InThe Thirty Sixth Annual Conference on Learning Theory, pages 5311–5342. PMLR, 2023

    2023

  25. [33]

    Pac-bayes compression bounds so tight that they can explain generalization

    Sanae Lotfi, Marc Finzi, Sanyam Kapoor, Andres Potapczynski, Micah Goldblum, and An- drew G Wilson. Pac-bayes compression bounds so tight that they can explain generalization. Advances in Neural Information Processing Systems, 35:31459–31473, 2022

    2022

  26. [34]

    Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021

    Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers.Transactions of the Association for Computational Linguistics, 9:53–68, 2021

    2021

  27. [35]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.arXiv preprint arXiv:2407.02490, 2024

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.arXiv preprint arXiv:2407.02490, 2024

    2024 arXiv

  28. [36]

    Policy gradient meth- ods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation.Advances in neural information processing systems, 12, 1999

    1999

  29. [37]

    Pac-bayesian model averaging

    David A McAllester. Pac-bayesian model averaging. InProceedings of the twelfth annual conference on Computational learning theory, pages 164–170, 1999

    1999

  30. [38]

    Pac-bayesian generalisation error bounds for gaussian process classification

    Matthias Seeger. Pac-bayesian generalisation error bounds for gaussian process classification. Journal of machine learning research, 3(Oct):233–269, 2002

    2002

  31. [39]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  32. [40]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scienti...

    2024

  33. [41]

    Technical report: Enhancing llm reasoning with reward-guided tree search.arXiv preprint arXiv:2411.11694, 2024

    Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, et al. Technical report: Enhancing llm reasoning with reward-guided tree search.arXiv preprint arXiv:2411.11694, 2024. 12

    2024 arXiv

  34. [42]

    Litesearch: Efficacious tree search for llm.arXiv preprint arXiv:2407.00320, 2024

    Ante Wang, Linfeng Song, Ye Tian, Baolin Peng, Dian Yu, Haitao Mi, Jinsong Su, and Dong Yu. Litesearch: Efficacious tree search for llm.arXiv preprint arXiv:2407.00320, 2024

    2024 arXiv

  35. [43]

    Alphamath almost zero: process supervision without process.arXiv preprint arXiv:2405.03553, 2024

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process.arXiv preprint arXiv:2405.03553, 2024

    2024 arXiv

  36. [44]

    Monte carlo tree search boosts reasoning via iterative preference learning

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451, 2024

    2024 arXiv

  37. [45]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repository, 13:9, 2024

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repositor...

    2024

  38. [46]

    Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning.arXiv preprint arXiv:2410.02884, 2024

    Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning.arXiv preprint arXiv:2410.02884, 2024

    arXiv

  39. [47]

    Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b.arXiv preprint arXiv:2406.07394, 2024

    Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b.arXiv preprint arXiv:2406.07394, 2024

    2024 arXiv

  40. [48]

    Booststep: Boosting mathematical capability of large language models via improved single-step reasoning.arXiv preprint arXiv:2501.03226, 2025

    Beichen Zhang, Yuhong Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Haodong Duan, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Booststep: Boosting mathematical capability of large language models via improved single-step reasoning.arXiv preprint arXiv:2501.03226, 2025. 13 Appendix A Lim...

    2025 arXiv

  41. [49]

    Subtracting from 1 yields the bound Equation 6

    + Pr(G c) ≤(1−p N,τ (q)) +p N,τ (q) (N−1)e −γ(q) 2/8ε2 +δ. Subtracting from 1 yields the bound Equation 6. The asymptotic form for ε/γ(q)→0 follows by observing thatexp(−γ(q) 2/8ε2)→0. 15 D Proof of Corollary 4.6 Corollary D.1(Target Accuracy Constraint on Sampling and Margin)...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.