Pith. sign in

REVIEW 4 major objections 6 minor 44 references

Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage self-correction loop is claimed to improve LLM arithmetic accuracy by up to 4.94 percentage points on GSM8K and MATH.

desk verdict A plausible incremental extension of MCTS-DPO that overclaims its 'intrinsic' self-correction, with fixable but load-bearing gaps in the mechanism. read the letter →

arxiv 2412.17397 v1 pith:JKV5SC77 submitted 2024-12-23 cs.LG cs.CV

Huchen Jiang , Yangyang Ma , Chaofan Ding , Kexin Luan , Xinhan Di This is my paper · ORCID
classification cs.LGcs.CV
keywords largelanguagemodelsself-correctionreinforcementlearningMonteCarloTreeSearchdirectpreferenceoptimizationmathematicalreasoningGSM8KMATH
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that an LLM's arithmetic reasoning improves reliably when self-correction is learned before step-level preference learning, rather than only at the end. The proposed two-stage procedure first uses REINFORCE to train a policy that, given its own earlier attempts at a problem, produces a corrected answer; this is the 'intrinsic self-correction' stage. The second stage plugs that self-correcting policy into Monte Carlo Tree Search boosted Direct Preference Optimization, using it as a step-level verifier that flags and repairs misleading intermediate reasoning. The reported result is that this combination beats both the base models and the MCTS-DPO-only baseline on GSM8K and MATH, with accuracy gains between 2.00 and 4.94 percentage points across four base models. The authors' intended takeaway is that the two abilities are synergistic: a better self-corrector makes the tree-search preference learner a better judge, and the gains would not be available from either stage alone.

What carries the argument

The load-bearing machinery is a two-loop training setup. Stage I trains a policy $\pi_{\theta_1}$ under a REINFORCE objective with a KL penalty, maximizing the summed oracle reward over multiple attempts (Eq. 1), where the policy receives the problem $x$, its earlier attempts $\hat{y}_{1:l}$, and an instruction to improve. Stage II is the step-level iterative preference learning: an MCTS tree grows by selection (PUCT), expansion, an 'enhanced-self-verify' step, and backup; the reward at a state is $R(s_t) = O(s_t) + C(s_t)$ with outcome correctness $O$ and self-evaluation $C$, and the Stage-I policy is inserted as an additional correction signal $\hat{C}(s_t) = \pi_{\theta_1}(A \mid \text{prompt}_{\text{eval}}, x, s_t)$, so the corrected state reward becomes $R(s_t^{\text{scorrect}}) = R(s_t) + \hat{C}(s_t)$. That insertion—the Stage-I policy acting as an extra verifier inside MCTS—is the specific mechanism the paper claims is responsible for the improvement.

What would settle it

Retrain Stage I with the reward function changed from the oracle answer-checker $\hat{r}(y, y^*)$ to a purely internal signal such as self-consistency across sampled rollouts, and keep Stage II fixed; if the reported GSM8K and MATH gains disappear, the mechanism named 'intrinsic self-correction' is not the source of the improvement.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that an 'intrinsic self-correction' stage—training the model to improve its own responses using only its own outputs as data—can be composed with the existing step-level MCTS-DPO method to yield consistent accuracy gains on two arithmetic benchmarks. The new ingredient is not a new search algorithm but a new ordering: the self-correcting policy is trained first (Stage I), then reused in Stage II as the verification model in the MCTS expansion and roll-out reward. The paper reports concrete numbers: on GSM8K, accuracy rises to 86.76% from an 84.76% baseline for Llama-3.1-8B-Instruct and to 38.06% from 35.78% for Mistral-7B-Instruct-v0.1; on MATH, accuracy rises to 71.34% from 67.16% for OpenMath2-Llama3.1-8B and to 48.06% from 43.42% for dart-math-mistral-7b-uniform. The ablation on MATH reports that the best configuration uses the Stage II policy (SPL-Model) as both policy and reward model, indicating the two stages work together rather than independently.

Load-bearing premise

The argument collapses if Stage I is assumed to be a purely self-supervised 'intrinsic' correction loop while Eq. (1) still rewards success against ground-truth answers; if those rewards are doing the work, the data is not self-generated in the sense claimed, and the improved verifier in Stage II may simply be a better answer-matching model rather than a genuine self-corrector.

Editorial extensions

If this is right

  • If the two-stage claim holds, any step-level MCTS-DPO training pipeline can benefit by first running an intrinsic self-correction loop on self-generated rollouts, at no extra label cost beyond the final answer checker.
  • The gains are consistent across two different base-model families and two dataset difficulties, so the procedure is not tailored to a single architecture.
  • Stage-I-only self-correction already improves over the base model in all four configurations, which suggests the self-correction policy transfers some verifier ability before any tree search is used.
  • Ablations on MATH indicate the full combination dominates either stage alone, so the method's value is in the composition rather than in a single component.
  • Because the method only assumes an oracle correctness check at the end of a rollout, it should extend to other tasks where final answers can be checked objectively.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Eq. (1) uses an oracle reward $\hat{r}(y,y^*)$ that compares against ground truth $y^*$, despite the abstract claiming no external feedback; this leaves open whether the 'intrinsic' stage would work without ground-truth answers, and testing self-consistency alone as the reward would separate true self-correction from supervised answer-matching.
  • A natural extension not tested in the paper is to use the Stage-I policy's correction signal as a reward in a policy-gradient pass rather than only inside MCTS, which would show whether the benefit is specific to tree-search integration.
  • If the corrected-step reward is what helps, one expects the gap between Ours and Base Model + MCTS-DPO to widen as problems get harder; checking accuracy per MATH difficulty band would make the mechanism testable.
  • The method should transfer to code generation or tool-use benchmarks where a unit test or interpreter supplies the oracle correctness signal, without changing the two-stage structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a two-stage procedure for improving step-level reasoning in LLMs on arithmetic benchmarks. In Stage I, a policy pi_theta1 is trained with a REINFORCE-style objective plus a KL penalty to produce corrected responses across multiple attempts, using an oracle answer checker r_hat(y,y*) as reward. In Stage II, the authors run step-level MCTS-DPO in which the reward at a state combines outcome correctness with a self-evaluation term C(s_t) = pi_theta2(A | prompt_eval, x, s_t), and an 'Enhanced-Self-Verify' term C_hat(s_t) = pi_theta1(A | prompt_eval, x, s_t). Results are reported on GSM8K and MATH for four base models, with the full method claimed to outperform both the base model and the Base + MCTS-DPO baseline; an ablation on MATH compares configurations of policy/reference/reward models.

Significance. If the claimed mechanism were validated, the paper would address a real bottleneck in MCTS-boosted reasoning, namely the reliance on accurate step-level verifiers, by reusing a self-trained policy as an intrinsic verifier. The reported numerical gains are consistent in direction across four models and two datasets, and the MATH ablation is a useful first attempt at isolating the contribution of each stage. However, the central 'intrinsic self-correction' claim is currently entangled with oracle answer-level supervision, the Enhanced-Self-Verify term is not operationalized, and the experimental section omits code, hyperparameters, and error bars. The significance is therefore conditional on resolving these issues.

major comments (4)
  1. [Stage I (Eq. (1))] Eq. (1) maximizes E[sum_{i=1}^{l+1} r_hat(y_i, y*)] using an oracle answer checker that compares outputs with the ground-truth y*. The abstract and the Stage I text say the model is trained 'relying entirely on self-generated data' and 'without any external feedback.' This is internally inconsistent: y* is external, answer-level supervision. To support the 'intrinsic' claim, the paper must distinguish between self-generated rollouts (the inputs are self-generated) and the oracle-graded reward, and it must add an ablation in which Stage I is trained with a self-evaluation-only reward or with y* removed. As written, the Table 2 gains could be produced entirely by standard answer-checker RL, with the 'intrinsic' label doing no work.
  2. [Enhanced-Self-Verify (Eqs. (5)-(6))] The reward R(s_correct_t) is defined as R(s_t) + C_hat(s_t), where C_hat(s_t) = pi_theta1(A | prompt_eval, x, s_t), but prompt_eval is never specified and no training objective is given that makes pi_theta1 output a token-level confidence A for a partial reasoning state s_t. Stage I optimizes whole-response correctness over l+1 attempts, not step-level verifier scores. The authors need to specify the exact prompt, the mapping from pi_theta1 logits to A, and any validation of this confidence against step-level correctness. Without this, the mechanism behind the Table 3 gains cannot be distinguished from Stage I answer-checker RL alone.
  3. [Eq. (4) and Stage II reward] The self-evaluation term C(s_t) = pi_theta2(A | prompt_eval, x, s_t) uses the same policy being trained by MCTS-DPO. This self-referential reward can reward high confidence rather than true correctness. Since final accuracy is measured on held-out test sets, the evaluation itself is not circular, but the training signal may be. Please report whether pi_theta2's confidence score A correlates with step-level correctness on a validation set; otherwise the self-evaluation component could be a confidence-seeking objective rather than a verifier.
  4. [Experiments and Ablation (Table 3)] Table 3 does not contain the conditions needed to attribute the improvement to the proposed mechanism. The design varies policy/reference/reward model identity among base, ISC, and SPL, but there is no row that suppresses the oracle reward in Stage I and no row that isolates the Enhanced-Self-Verify term from the plain self-evaluation term. Moreover, the ISC/ISC/SPL row underperforms the base/base/base row (65.34 vs. 68.06 on OpenMath2-MATH and 47.22 vs. 45.56 on dart-math-MATH), a result that is not discussed and is hard to reconcile with a monotone benefit of Stage I. The absence of hyperparameters (c_puct, lambda, KL coefficient, number of rollouts), random seeds, and code makes the reported gains (e.g., 71.34 vs. 68.06 and 48.06 vs. 45.56) uncheckable and prevents assessment of statistical significance.
minor comments (6)
  1. [Abstract and Main Results] The relative gain for dart-math-mistral-7b-uniform on MATH is reported as +4.94% in the abstract but +4.64% in the main text (48.06 vs. 43.42); please harmonize.
  2. [Eq. (2)] Eq. (2) writes s*_{t+1} = argmax_{st} [...], but the maximization should be over child nodes/actions, not over st; please correct the notation.
  3. [Figures 1 and 2] The captions of Figures 1 and 2 introduce terms such as 'self-correctness-policy' and 'outer-loop-policy' without connecting them explicitly to pi_theta1 and pi_theta2; please define all acronyms and box labels.
  4. [Table 3] The column labeled 'Reference model' is not defined in the text; clarify whether it is the KL reference model in REINFORCE/DPO or a different quantity.
  5. [Discussion] The Discussion refers to 'self-supervised learning' for Stage I, while Stage I is described methodologically as a REINFORCE RL objective; use consistent terminology.
  6. [Methodology] Numerical values for lambda in Eq. (2), the KL penalty coefficient in Stage I, and the number of MCTS rollouts are not reported; please provide these or state that they were tuned on a validation split.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracy gains are measured against external benchmarks, and although the Stage II reward is self-referential, it is not used as the final evaluation metric.

full rationale

The paper's derivation chain is empirical rather than definitional. Stage I (Eq. 1) trains the policy with REINFORCE to maximize the sum of oracle rewards r-hat(ŷ_i, y*), where r-hat is an answer checker comparing against ground-truth y*; this is external supervision, not a self-derived quantity. Stage II constructs MCTS rewards as R(st)=O(st)+C(st) with C(st)=πθ2(A|prompt_eval,x,st) (Eqs. 3-4) and adds an enhanced-verify term from πθ1 (Eqs. 5-6); these are explicit modeling choices, and the final accuracies in Tables 1-2 are independent benchmark measurements, not algebraic consequences of the training objectives. The paper contains no self-citations, no imported uniqueness theorems, and no renaming of known results. The central concern that the Stage II reward uses the same policy πθ2 being trained is a real reward-hacking risk, but it is not circularity in the sense of a prediction reducing to its input by construction, because the outcome metric is external. Two non-circular weaknesses should be weighed separately: the claim that Stage I uses 'self-generated data without any external feedback' is contradicted by the oracle reward r-hat(y,y*) in Eq. (1), and Table 3 does not include an ablation that removes y* from the Stage I reward, so the 'intrinsic' mechanism is not cleanly isolated. These affect the validity of the mechanism claim, but they do not make the reported results equivalent to the inputs. Therefore no circular step is exhibited.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The method rests on unreported hyperparameters and on the untested reliability of using the model's own confidence as a reward signal.

free parameters (4)
  • c_puct
    PUCT exploration constant in Eq. (2); value not reported.
  • lambda
    Length penalty exponent in the policy probability p(a|s_t) = πθ2(a|x,s_t)/|a|^λ; value not reported.
  • KL penalty coefficient
    KL-divergence penalty coefficient in the REINFORCE objective in Stage I; value not reported.
  • MCTS rollouts
    Number of Monte Carlo Tree Search simulations or rollouts; value not reported.
assumptions (3)
  • domain assumption The oracle answer checker r̂(y, y*) provides correct rewards for self-correction training.
    Used in Eq. (1) to score responses; if the checker is noisy, the trained self-correction policy will be mis-calibrated.
  • domain assumption The self-evaluation confidence score C(s_t) = πθ2(A | prompt_eval, x, s_t) is a reliable proxy for step correctness.
    Used as the reward in MCTS (Eq. 4); no calibration or correlation evidence is provided.
  • ad hoc to paper The Stage I self-correct policy improves Stage II verification beyond the base policy.
    This is the premise of the two-stage design; Table 3 shows that using the self-correct model as reward sometimes reduces accuracy (e.g., OpenMath2 with ISC-Model reward gives 67.06 vs. 68.06 baseline), so the transfer is not guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning." pith.science (2026). https://pith.science/paper/JKV5SC77

@misc{pith2026241217397,
  author       = {Pith},
  title        = {Pith review of: Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JKV5SC77}},
  note         = {Machine review of arXiv:2412.17397}
}
read the original abstract

With current state-of-the-art approaches aimed at enhancing the reasoning capabilities of Large Language Models(LLMs) through iterative preference learning inspired by AlphaZero, we propose to further enhance the step-wise reasoning capabilities through intrinsic self-correction to some extent. Our work leverages step-wise preference learning to enhance self-verification via reinforcement learning. We initially conduct our work through a two-stage training procedure. At the first stage, the self-correction reasoning ability of an LLM is enhanced through its own predictions, relying entirely on self-generated data within the intrinsic self-correction to some extent. At the second stage, the baseline step-wise preference learning is leveraged via the application of the enhanced self-correct policy achieved at the first stage. In the evaluation of arithmetic reasoning tasks, our approach outperforms OpenMath2-Llama3.1-8B, dart-math-mistral-7b-uniform on MATH with increases in accuracy to 71.34%(+4.18%) and 48.06%(+4.94%) and LLama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.1 on GSM8K with increases in accuracy to 86.76%(+2.00%) and 38.06%(+2.28%).

Figures

Figures reproduced from arXiv: 2412.17397 by the authors.

Figure 1
Figure 1. Overview of Towards Intrinsic Self-Correction En [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Towards intrinsic Self-Correct LLM in the Inner [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Step-wise Iterative Preference Learning in the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 7 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ahmadian, A.; Cremer, C.; Gall \'e , M.; Fadaee, M.; Kreutzer, J.; Pietquin, O.; \"U st \"u n, A.; and Hooker, S. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740

    arXiv 2024

  4. [4]

    u rek, A. F.; Aky \

    Aky \"u rek, A. F.; Aky \"u rek, E.; Madaan, A.; Kalyan, A.; Clark, P.; Wijaya, D.; and Tandon, N. 2023. Rl4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arXiv preprint arXiv:2305.08844

    arXiv 2023

  5. [5]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  6. [6]

    Coulom, R. 2006. Efficient selectivity and backup operators in Monte-Carlo tree search. In International conference on computers and games, 72--83. Springer

    2006

  7. [7]

    Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    A.; Chebotar, Y.; Xiao, T.; Irpan, A.; Levine, S.; Castro, P

    Farebrother, J.; Orbay, J.; Vuong, Q.; Ta \" ga, A. A.; Chebotar, Y.; Xiao, T.; Irpan, A.; Levine, S.; Castro, P. S.; Faust, A.; et al. 2024. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.03950

    arXiv 2024

Show all 44 references
  1. [9]

    Grill, J.-B.; Altch \'e , F.; Tang, Y.; Hubert, T.; Valko, M.; Antonoglou, I.; and Munos, R. 2020. Monte-Carlo tree search as regularized policy optimization. In International Conference on Machine Learning, 3769--3778. PMLR

    2020

  2. [10]

    J.; Wang, Z.; Wang, D

    Hao, S.; Gu, Y.; Ma, H.; Hong, J. J.; Wang, Z.; Wang, D. Z.; and Hu, Z. 2023. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992

    2023 arXiv

  3. [11]

    Havrilla, A.; Raparthy, S.; Nalmpantis, C.; Dwivedi-Yu, J.; Zhuravinskyi, M.; Hambro, E.; and Raileanu, R. 2024. Glore: When, where, and how to improve llm reasoning via global and local refinements. arXiv preprint arXiv:2402.10963

    2024 arXiv

  4. [12]

    Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  5. [13]

    Hong, J.; Lee, N.; and Thorne, J. 2024. Reference-free monolithic preference optimization with odds ratio. arXiv e-prints, arXiv--2403

    2024

  6. [14]

    Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D

    Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825

    2023 arXiv

  7. [15]

    Kim, G.; Baldi, P.; and McAleer, S. 2024. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36

    2024

  8. [16]

    Kocsis, L.; and Szepesv \'a ri, C. 2006. Bandit based monte-carlo planning. In European conference on machine learning, 282--293. Springer

    2006

  9. [17]

    D.; Singh, A.; Baumli, K.; Iqbal, S.; Bishop, C.; Roelofs, R.; et al

    Kumar, A.; Zhuang, V.; Agarwal, R.; Su, Y.; Co-Reyes, J. D.; Singh, A.; Baumli, K.; Iqbal, S.; Bishop, C.; Roelofs, R.; et al. 2024. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917

    2024 arXiv

  10. [18]

    Li, Y.; Lin, Z.; Zhang, S.; Fu, Q.; Chen, B.; Lou, J.-G.; and Chen, W. 2023. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5315--5333

    2023

  11. [19]

    Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050

    2023 arXiv

  12. [20]

    Liu, J.; Cohen, A.; Pasunuru, R.; Choi, Y.; Hajishirzi, H.; and Celikyilmaz, A. 2023. Making ppo even better: Value-guided monte-carlo tree search decoding. arXiv preprint arXiv:2309.15028

    2023 arXiv

  13. [21]

    Llama Team, A. . M. 2024. The Llama 3 Herd of Models. Journal of Mathematical Physics, 36(92)

    2024

  14. [22]

    Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Paul, D.; Ismayilzada, M.; Peyrard, M.; Borges, B.; Bosselut, A.; West, R.; and Faltings, B. 2023. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904

    2023 arXiv

  16. [24]

    Qu, Y.; Zhang, T.; Garg, N.; and Kumar, A. 2024. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint arXiv:2407.18219

    2024 arXiv

  17. [25]

    D.; Ermon, S.; and Finn, C

    Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  18. [26]

    Rosin, C. D. 2011. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence, 61(3): 203--230

    2011

  19. [27]

    Shani, L.; Rosenberg, A.; Cassel, A.; Lang, O.; Calandriello, D.; Zipori, A.; Noga, H.; Keller, O.; Piot, B.; Szpektor, I.; et al. 2024. Multi-turn Reinforcement Learning from Preference Human Feedback. arXiv preprint arXiv:2405.14655

    2024 arXiv

  20. [28]

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  21. [29]

    Shinn, N.; Labash, B.; and Gopinath, A. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2(5): 9

    2023 arXiv

  22. [30]

    Silver, D.; Hubert, T.; Schrittwieser, J.; Antonoglou, I.; Lai, M.; Guez, A.; Lanctot, M.; Sifre, L.; Kumaran, D.; Graepel, T.; et al. 2017. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815

    2017 arXiv

  23. [31]

    Snell, C.; Kostrikov, I.; Su, Y.; Yang, M.; and Levine, S. 2022. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871

    2022 arXiv

  24. [32]

    Tong, Y.; Zhang, X.; Wang, R.; Wu, R.; and He, J. 2024. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. arXiv preprint arXiv:2407.13690

    2024 arXiv

  25. [34]

    Toshniwal, S.; Du, W.; Moshkov, I.; Kisacanin, B.; Ayrapetyan, A.; and Gitman, I. 2024 b . OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data. arXiv preprint arXiv:2410.01560

    2024 arXiv

  26. [35]

    Uesato, J.; Kushman, N.; Kumar, R.; Song, F.; Siegel, N.; Wang, L.; Creswell, A.; Irving, G.; and Higgins, I. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275

    2022 arXiv

  27. [36]

    Welleck, S.; Lu, X.; West, P.; Brahman, F.; Shen, T.; Khashabi, D.; and Choi, Y. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053

    2022 arXiv

  28. [37]

    A.; Ostendorf, M.; and Hajishirzi, H

    Wu, Z.; Hu, Y.; Shi, W.; Dziri, N.; Suhr, A.; Ammanabrolu, P.; Smith, N. A.; Ostendorf, M.; and Hajishirzi, H. 2023. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36: 59008--59033

    2023

  29. [38]

    P.; Kawaguchi, K.; and Shieh, M

    Xie, Y.; Goyal, A.; Zheng, W.; Kan, M.-Y.; Lillicrap, T. P.; Kawaguchi, K.; and Shieh, M. 2024. Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning. arXiv preprint arXiv:2405.00451

    2024 arXiv

  30. [39]

    Xie, Y.; Kawaguchi, K.; Zhao, Y.; Zhao, X.; Kan, M.-Y.; He, J.; and Xie, Q. 2023. Decomposition enhances reasoning via self-evaluation guided decoding. arXiv preprint arXiv:2305.00633, 2

    2023 arXiv

  31. [40]

    Xiong, W.; Shi, C.; Shen, J.; Rosenberg, A.; Qin, Z.; Calandriello, D.; Khalman, M.; Joshi, R.; Piot, B.; Saleh, M.; et al. 2024. Building math agents with multi-turn iterative preference learning. arXiv preprint arXiv:2409.02392

    2024 arXiv

  32. [41]

    Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  33. [42]

    T.; Li, Z.; Weller, A.; and Liu, W

    Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y.; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284

    2023 arXiv

  34. [43]

    Zhang, Y.; Khalifa, M.; Logeswaran, L.; Kim, J.; Lee, M.; Lee, H.; and Wang, L. 2024. Small Language Models Need Strong Verifiers to Self-Correct Reasoning. arXiv preprint arXiv:2404.17140

    2024 arXiv

  35. [44]

    Zhou, Y.; Zanette, A.; Pan, J.; Levine, S.; and Kumar, A. 2024. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446

    2024 arXiv

  36. [45]

    Zhu, X.; Wang, J.; Zhang, L.; Zhang, Y.; Gan, R.; Zhang, J.; and Yang, Y. 2022. Solving math word problems via cooperative reasoning induced language models. arXiv preprint arXiv:2210.16257

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.