BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning

Boyi Liu; Han Zhong; Hongyi Guo; Liwei Wang; Mingyi Hong; Shenao Zhang; Sirui Zheng; Xiaojun Xu; Yifei Zuo; Yuanxin Liu

REVIEW 4 major objections 5 minor 75 references

BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a two-step EM-style algorithm, BRiTE, can learn high-quality reasoning processes by reinforcement learning, and that these RL-generated rationales improve math and coding performance more than rejection-sampling…

desk verdict A genuinely novel RL-approximated E-step for bootstrapping rationales, with an honest but idealized theory; the empirical overclaims and unfair RS baseline are the real problems. read the letter →

arxiv 2501.18858 v2 pith:HSUSPBZO submitted 2025-01-31 cs.LG cs.AIcs.CL

Han Zhong , Yutong Yin , Shenao Zhang , Xiaojun Xu , Yuanxin Liu , Yifei Zuo , Zhihan Liu , Boyi Liu

show 5 more authors

Sirui Zheng Hongyi Guo Liwei Wang Mingyi Hong Zhaoran Wang

This is my paper · ORCID

classification cs.LGcs.AIcs.CL

keywords LLMreasoningchain-of-thoughtreinforcementlearningexpectation-maximizationrejectionsamplingrewardshapinglatentvariablemodelmathematical

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BRiTE treats an LLM's hidden reasoning process as a latent variable and the correctness of its answer as an observable evaluation signal, then maximizes the joint probability of the two via an EM-style loop. In the first step a policy is trained by reinforcement learning to generate the reasoning process most likely to yield a verified-correct answer; in the second the base model is fine-tuned on those generated rationales. The paper claims this RL-based bootstrapping produces better rationales than rejection sampling, and that the resulting models match or exceed supervised fine-tuning on human-annotated chains of thought across math and code benchmarks. The authors also prove a 1/T convergence rate for the loop under an exponential-family and concavity assumption. If the claims hold, reasoning improvements could be obtained without human-annotated reasoning data.

What carries the argument

The argument is carried by three pieces of machinery. Lemma 3.1 rewrites the log-marginal objective as a variational maximum over a distribution Q, turning the problem into an EM loop. Proposition 3.7 supplies the reward-shaping identity: in a deterministic entropy-regularized MDP, the optimal policy is proportional to exp of the sum of token rewards divided by β, so choosing β=1 and the total reward equal to log P(z,y,o|x,θ_t) makes the RL-trained policy the exact target posterior of the E-step. The M-step then maximizes the joint probability of the generated rationale and answer under the base model. Theorem 3.3 ties the loop to a 1/T convergence rate via a mirror-descent telescoping argument, under an exponential-family logit assumption and concavity.

What would settle it

Train two models on the same number of rationales per problem, one selected by the RL-trained policy and one by rejection sampling with matched answer correctness; if the benchmark accuracy gap disappears, the claim that RL bootstrapping beats rejection sampling is not supported. On a small finite-state reasoning task, the exact posterior can be enumerated, so the KL divergence between it and the RL-trained policy would directly test whether the 1/T convergence theorem applies to the implementation.

Watch

Extended reading notes

Core claim

The central discovery is that rationale bootstrapping can be cast as maximum-likelihood estimation in a graphical model where the thought process is unobserved, and that the intractable posterior over thoughts can be approached with a reward-shaped reinforcement learning step instead of rejection sampling. The paper's key theoretical statement is that this alternating procedure converges at rate 1/T, and its key empirical statement is that on GSM8K, MATH, HumanEval, BigCodeBench, and harder competition benchmarks, BRiTE consistently beats rejection-sampling EM and iterative DPO, and matches or exceeds SFT with human-annotated rationales.

Load-bearing premise

The convergence proof assumes the reasoning-process distribution is updated exactly and the training objective is concave, while the implemented algorithm learns that distribution with PPO/GRPO and transformer log-likelihoods are not concave.

Editorial extensions

If this is right

Rejection-sampling self-training pipelines can be upgraded by replacing the sampling filter with an RL-trained rationale generator, yielding consistent accuracy gains across base models.
Because no human-annotated rationales are needed, the same procedure could scale reasoning fine-tuning to domains where expert annotations are scarce or expensive.
If the framework's unification claim holds, its convergence analysis transfers to PPO-style RLHF, latent DPO, and ReST-style EM objectives under the stated assumptions.
On code generation, BRiTE removes the requirement that training data come with unit-test-verified samples, since the RL-trained policy generates rationales directly.
Iterating the procedure (BRiTE-iter-2) yields additional small gains on hard math benchmarks, suggesting the bootstrapping loop can be continued beyond one round.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension would apply the same reward-shaping identity to other verification signals, such as unit-test feedback or human preference judgments, without changing the algorithm's structure.
The paper evaluates instruction-tuned open-weight models; applying BRiTE to base (non-instruction-tuned) models or as a pretraining step might show larger margins over rejection sampling, since the evaluated instruct models already had post-training.
The theoretical gap between the exact E-step assumed in Theorem 3.3 and the RL-approximated E-step used in practice suggests a concrete research question: bounding the approximation error of the RL policy, which would bring the 1/T result closer to the implemented algorithm.
Because BRiTE-DPO constructs its preference pairs from RL-generated rationales, the same construction could be dropped into any online preference learning method, not only DPO.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new piece is the use of PPO/GRPO as an approximate E-step to bootstrap rationales from the intractable posterior P(z,y|x,θ), with a reward-shaping construction (β=1, total reward log P) that makes the optimal policy match that posterior. That construction checks out; it is not circular. The accompanying unification of SFT, PPO, DPO, and rejection sampling as instances of one latent-variable objective is clean and useful for orienting the field.

The proof is not the problem. Theorem 3.3 is a legitimate 1/T mirror-descent/EM bound for exact updates under the stated concavity assumption. The paper also honestly notes that a weaker stationary-point guarantee holds without concavity. The soft spot is that the concavity assumption is unrealistic for transformer log-linear models, and the implemented BRiTE approximates both the E-step (PPO/GRPO training of Q) and the M-step (finite LoRA steps). The paper never bounds that approximation error, so the theorem covers an idealized algorithm, not what was run. That gap is common in LLM theory, but here the abstract states convergence for BRiTE without the caveat.

The experiments are where I'd push hardest. Table 1 gives consistent small gains over rejection sampling on all eight model-benchmark cells, and that is real evidence in favor of the mechanism. But the headline comparisons to SFT are overstated: BRiTE is below SFT on Mistral on both GSM8K and MATH, and below SFT on BigCodeBench Full; it is also below RS on HumanEval Plus. The abstract's "match or exceed" is doing too much. The larger-scale results in Table 2 use an RS baseline with N=2 rollouts while BRiTE uses 8 GRPO rollouts, so the 15-plus-point gains do not isolate RL-quality rationales from compute budget. There are no error bars or multiple seeds anywhere, which makes all the empirical claims hard to evaluate.

The self-citations are not the issue. The reward shaping is a legitimate construction, the EM framing is standard, and the proof does not assume its conclusion. I agree with the reader's overall verdict: reject in its current form. But this is not a desk-reject paper. The core mechanism is salvageable with matched compute baselines, error bars, tempered claims, and an explicit statement that the theory covers exact EM only. I would send it to reviewers.

Referee Report

4 major / 5 minor

Summary. The paper proposes BRiTE, a two-stage EM-style procedure for improving LLM reasoning. In the E-step, an RL-trained policy (via PPO or GRPO with a designed token-level reward) approximates the posterior distribution over latent rationales conditioned on the prompt, a correct answer, and an evaluation signal; in the M-step, the base LLM is fine-tuned to maximize the joint probability of the sampled rationale and answer. The authors present a graphical model (Figure 1), derive a 1/T convergence rate for their generic EM formulation in Theorem 3.3, and report experiments on GSM8K, MATH, and code-generation benchmarks, including a larger-scale experiment on Qwen2.5-7B with additional math/science benchmarks.

Significance. If fully established, the paper would offer a valuable practical direction: automated generation of high-quality rationales without human annotations, with a unified view connecting rejection-sampling EM, SFT, PPO/DPO, and latent-variable reasoning. The reward-shaping construction in Proposition 3.7 is a legitimate and useful contribution: it correctly identifies an entropy-regularized token-level MDP whose optimal policy is the desired posterior, and this is not circular. The framework's unification of existing algorithms is conceptually appealing. However, the manuscript's central theoretical claim is established only for an idealized exact-EM procedure, and the main large-scale empirical comparison is confounded by unequal sampling budgets. These issues are load-bearing for the abstract's promises of provable convergence and of superior performance over rejection sampling.

major comments (4)

[Theorem 3.3 and Section 3.4, Eqs. (3.4)-(3.5)] Theorem 3.3 is proved for the exact EM updates: the E-step sets Q exactly to the posterior in (3.4), and the M-step solves the argmax over theta exactly in (3.5). The implemented BRiTE, by contrast, approximates the E-step by training a policy Q_psi with PPO/GRPO (Section 3.4) and approximates the M-step by a finite number of LoRA gradient steps (Appendix D). No epsilon-bound on the E-step or M-step approximation error appears in Theorem 3.3 or in its proof. Consequently, the 1/T convergence guarantee applies to an idealized algorithm, not to the BRiTE procedure evaluated in Section 4; the abstract's statement that BRiTE has a provable convergence rate is not supported.
[Assumption 3.2 and the concavity hypothesis of Theorem 3.3] The theorem assumes P(z,y|x,theta) = exp(f_theta(x,z,y) - A(x,theta)) with f_theta in an RKHS and L(theta) concave. This globally normalized energy-model parameterization is not the autoregressive token-level factorization used by the transformer LMs in Section 4. Moreover, L(theta) is a difference of log-partition functions and is not generally concave in f_theta; no argument is given that the actual network parameterization satisfies this condition. The proof invokes only a local concave inequality at (C.11), so the stated global assumption is both unverified for the deployed model class and stronger than what the proof uses. The weaker stationary-point result in Theorem C.3 does not provide the claimed 1/T optimality gap. Thus the main theoretical result does not apply to the models actually trained.
[Table 2 and Appendix D.3] The scaling experiments in Table 2 compare BRiTE against an RS baseline that samples N=2 candidate rationales per prompt, while BRiTE uses 8 GRPO rollouts per prompt. The large reported gains on MATH500, Minerva Math, and AMC23 are therefore confounded by sampling budget and cannot be attributed solely to the RL bootstrapping mechanism. A matched-compute or matched-sample comparison is needed before the superiority claim over rejection sampling can be accepted.
[Tables 1 and 2] The experimental tables report single-run accuracies without error bars, standard deviations, or repeated seeds. Given the modest differences in Table 1 (often 1-3 points) and the additional sampling-budget confound in Table 2, the claimed consistent improvements are not established with statistical reliability.

minor comments (5)

[Section 1.1] The sentence 'we provide a more general and rigorous mathematical framework fro LLM reason and unified theoretical guarantees' should be corrected to 'for LLM reasoning and unified theoretical guarantees'.
[Section 4.2, item 4 vs Figure 2] The code-generation results are referenced in the text as 'Table 2' but appear as Figure 2; the caption and the referring text should be harmonized.
[Section 4.3] The sentence 'BRiTE with an external verifier can improve reject sampling (RS) significantly' should read 'rejection sampling'.
[Appendix D.3] The phrase 'and bath size to be 8' should read 'batch size'.
[Eq. (3.2) and Section 3.1] The notation overloads Z, Y, and O for both the full spaces and the restricted subsets in the objective; the distinction should be made explicit to avoid confusion when comparing (3.1) and (3.2).

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the EM/mirror-descent proof is self-contained and the reward shaping is a standard softmax construction; the central empirical claim is evaluated out-of-sample.

full rationale

The paper's core derivation chain is an EM-type bound: Lemma 3.1 is the variational KL identity; (3.4) defines the E-step posterior Q proportional to P(z,y,o|x,theta_t); (3.5) is the exact M-step; and Theorem 3.3 (Appendix C.1) proves a 1/T rate by telescoping KL divergences with reproducing-kernel gradient identities. None of these steps assumes the theorem's conclusion, and the proof is contained in the paper rather than imported from the authors' prior work. The reward shaping in Proposition 3.7 is a legitimate construction rather than a fitted parameter called a prediction: for a deterministic entropy-regularized MDP, the optimal policy is proportional to exp(r/beta), so choosing beta=1 and total reward r = log P(z,y,o|x,theta_t) makes the optimal policy equal the normalized posterior Q by the standard softmax identity. This is a reduction of posterior sampling to RL, not a prediction that secretly uses benchmark outcomes. Self-citations (Zhong et al. 2024 for MDP notation, and Xiong et al. 2024 for the iterative-DPO baseline) are contextual and not load-bearing for the convergence or empirical claims. The empirical claims are out-of-sample benchmark comparisons against rejection sampling and SFT; the only mild self-referential element is that the RL reward is the base model's own log-likelihood, which is a form of self-training bootstrap rather than a circular derivation. Approximation gaps between the idealized EM proof and the PPO/GRPO/LoRA implementation, and the unequal rollouts in the scaling-up comparison (Appendix D.3), are correctness and fairness concerns, not circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or computational entities. The latent variable Z (thinking process) and observation O (evaluation signal) are standard latent variable constructs. The main theoretical axioms are the RKHS log-linear assumption and concavity, both unrealistic for actual LLMs, plus the standard assumption of an external evaluation signal independent of θ.

free parameters (1)

β (entropy regularization coefficient) = 1
In Section 3.4, β is set to 1 so that the optimal policy in Proposition 3.7 matches the desired posterior Q ∝ P. This is a hand-chosen value, not derived from data.

assumptions (4)

ad hoc to paper Assumption 3.2: P(z,y|x,θ) ∝ exp(f_θ(x,z,y)) with f_θ in an RKHS.
This assumes a joint softmax over all (z,y) sequences, which does not match the autoregressive per-token normalization of transformer LMs. Used throughout the proof of Theorem 3.3.
ad hoc to paper Concavity of L(θ) = log P(z∈Z,y∈Y,o∈O|x,θ).
Stated before Theorem 3.3 as 'Given that L is concave'. Not guaranteed for neural network parameterizations, and crucial for the global 1/T convergence rate.
domain assumption P(o|x,z,y) is independent of θ (Equation 3.1).
Assumes a fixed ground-truth evaluation signal (e.g., verifier or correct answer) independent of the model. Reasonable for math/code with verified answers.
domain assumption The MDP transition is deterministic (Proposition 3.7).
Text generation in an LLM deterministically concatenates tokens, so this modeling assumption holds for token-level MDPs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning." pith.science (2026). https://pith.science/paper/HSUSPBZO

@misc{pith2026250118858,
  author       = {Pith},
  title        = {Pith review of: BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HSUSPBZO}},
  note         = {Machine review of arXiv:2501.18858}
}

abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, yet generating reliable reasoning processes remains a significant challenge. We present a unified probabilistic framework that formalizes LLM reasoning through a novel graphical model incorporating latent thinking processes and evaluation signals. Within this framework, we introduce the Bootstrapping Reinforced Thinking Process (BRiTE) algorithm, which works in two steps. First, it generates high-quality rationales by approximating the optimal thinking process through reinforcement learning, using a novel reward shaping mechanism. Second, it enhances the base LLM by maximizing the joint probability of rationale generation with respect to the model's parameters. Theoretically, we demonstrate BRiTE's convergence at a rate of $1/T$ with $T$ representing the number of iterations. Empirical evaluations on math and coding benchmarks demonstrate that our approach consistently improves performance across different base models without requiring human-annotated thinking processes. In addition, BRiTE demonstrates superior performance compared to existing algorithms that bootstrap thinking processes use alternative methods such as rejection sampling, and can even match or exceed the results achieved through supervised fine-tuning with human-annotated data.

Figures

Figures reproduced from arXiv: 2501.18858 by the authors.

**Figure 1.** LLM as a probabilistic graphical model. X and Y represent prompt and response, respectively. The latent variable Z indicates the intrinsic thinking process behind generation. Evaluation signal O is influenced by X, Z, and Y . focus but our method is based on reinforcement learning and thus is different from previous works. Previous methods can be roughly regarded as an EM-type algorithm, and detailed comparisons are… view at source ↗

**Figure 3.** Comparison between BRiTE and iterative DPO in the RLHF stage. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

**Figure 2.** Results of BRiTE on coding generation task using the [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Left: Training dynamics of BRiTE with an external verifier and rejection sampling. Right: Mean accuracy of benchmark scores of models trained by BRiTE with an external verifier and reject sampling during the training process. pass@1 accuracy (64 sampling times for AIME…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 33 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := ...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

, Kakade, S

Agarwal, A. , Kakade, S. M. , Lee, J. D. and Mahajan, G. (2021). On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22 1--76

work page 2021
[4]

Introducing claude

Anthropic (2023). Introducing claude. ://www.anthropic.com/index/introducing-claude

work page 2023
[5]

Azar, M. G. , Guo, Z. D. , Piot, B. , Munos, R. , Rowland, M. , Valko, M. and Calandriello, D. (2024). A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics. PMLR

work page 2024
[6]

Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 324--345

work page 1952
[7]

Bubeck, S. et al. (2015). Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning , 8 231--357

work page 2015
[8]

, Yang, Z

Cai, Q. , Yang, Z. , Jin, C. and Wang, Z. (2020). Provably efficient exploration in policy optimization. In International Conference on Machine Learning. PMLR

work page 2020

Show all 75 references

[9]

, Mei, J

Cen, S. , Mei, J. , Goshvadi, K. , Dai, H. , Yang, T. , Yang, S. , Schuurmans, D. , Chi, Y. and Dai, B. (2024). Value-incentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405.19320

2024 arXiv
[10]

, Feng, Y

Chen, H. , Feng, Y. , Liu, Z. , Yao, W. , Prabhakar, A. , Heinecke, S. , Ho, R. , Mui, P. , Savarese, S. , Xiong, C. et al. (2024). Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding. arXiv preprint arXiv:2411.04282

2024 arXiv
[11]

, Zhong, H

Chen, X. , Zhong, H. , Yang, Z. , Wang, Z. and Wang, L. (2022). Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In International Conference on Machine Learning. PMLR

2022
[12]

Christiano, P. F. , Leike, J. , Brown, T. , Martic, M. , Legg, S. and Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

2017
[13]

, Kosaraju, V

Cobbe, K. , Kosaraju, V. , Bavarian, M. , Chen, M. , Jun, H. , Kaiser, L. , Plappert, M. , Tworek, J. , Hilton, J. , Nakano, R. et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

2021 arXiv
[14]

Dempster, A. P. , Laird, N. M. and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39 1--22

1977
[15]

, Xiong, W

Dong, H. , Xiong, W. , Goyal, D. , Zhang, Y. , Chow, W. , Pan, R. , Diao, S. , Zhang, J. , Shum, K. and Zhang, T. (2023 a ). Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767

2023 arXiv
[16]

, Wang, Z

Dong, Y. , Wang, Z. , Sreedhar, M. N. , Wu, X. and Kuchaiev, O. (2023 b ). Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf. arXiv preprint arXiv:2310.05344

2023 arXiv
[17]

, Paine, T

Gulcehre, C. , Paine, T. L. , Srinivasan, S. , Konyushkova, K. , Weerts, L. , Sharma, A. , Siddhant, A. , Ahern, A. , Wang, M. , Gu, C. et al. (2023). Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998

2023 arXiv
[18]

, Tan, B

Guo, H. , Tan, B. , Liu, Z. , Xing, E. P. and Hu, Z. (2021). Efficient (soft) q-learning for text generation with limited good data. arXiv preprint arXiv:2106.07704

2021 arXiv
[19]

, Luo, R

He, C. , Luo, R. , Bai, Y. , Hu, S. , Thai, Z. L. , Shen, J. , Hu, J. , Han, X. , Huang, Y. , Zhang, Y. , Liu, J. , Qi, L. , Liu, Z. and Sun, M. (2024). Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

2024
[20]

, Burns, C

Hendrycks, D. , Burns, C. , Kadavath, S. , Arora, A. , Basart, S. , Tang, E. , Song, D. and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

2021 arXiv
[21]

Hoffman, M. D. , Phan, D. , Dohan, D. , Douglas, S. , Le, T. A. , Parisi, A. , Sountsov, P. , Sutton, C. , Vikram, S. and A Saurous, R. (2024). Training chain-of-thought via latent-variable inference. Advances in Neural Information Processing Systems, 36

2024
[22]

Hu, E. J. , Jain, M. , Elmoznino, E. , Kaddar, Y. , Lajoie, G. , Bengio, Y. and Malkin, N. (2023). Amortizing intractable inference in large language models. arXiv preprint arXiv:2310.04363

2023 arXiv
[23]

Hu, E. J. , Shen, Y. , Wallis, P. , Allen-Zhu, Z. , Li, Y. , Wang, S. , Wang, L. and Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

2021 arXiv
[24]

, Cheng, T

Huang, S. , Cheng, T. , Liu, J. K. , Hao, J. , Song, L. , Xu, Y. , Yang, J. , Liu, J. , Zhang, C. , Chai, L. et al. (2024). Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905

2024 arXiv
[25]

Jiang, A. Q. , Sablayrolles, A. , Mensch, A. , Bamford, C. , Chaplot, D. S. , Casas, D. d. l. , Bressand, F. , Lengyel, G. , Lample, G. , Saulnier, L. et al. (2023). Mistral 7b. arXiv preprint arXiv:2310.06825

2023 arXiv
[26]

Kingma, D. P. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

2013 arXiv
[27]

, Andreassen, A

Lewkowycz, A. , Andreassen, A. , Dohan, D. , Dyer, E. , Michalewski, H. , Ramasesh, V. , Slone, A. , Anil, C. , Schlag, I. , Gutman-Solo, T. et al. (2022). Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35 3843--3857

2022
[28]

Liu, G. , Ji, K. , Zheng, R. , Wu, Z. , Dun, C. , Gu, Q. and Yan, L. (2024 a ). Enhancing multi-step reasoning abilities of language models through direct q-function optimization. arXiv preprint arXiv:2410.09302

2024 arXiv
[29]

, Xia, C

Liu, J. , Xia, C. S. , Wang, Y. and Zhang, L. (2023). Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems

2023
[30]

Liu, Z. , Lu, M. , Zhang, S. , Liu, B. , Guo, H. , Yang, Y. , Blanchet, J. and Wang, Z. (2024 b ). Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436

2024 arXiv
[31]

, Welleck, S

Lu, X. , Welleck, S. , Hessel, J. , Jiang, L. , Qin, L. , West, P. , Ammanabrolu, P. and Choi, Y. (2022). Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35 27591--27609

2022
[32]

American mathematics competitions amc 12, 2023

Mathematical Association of America (2023). American mathematics competitions amc 12, 2023. ://maa.org/math-competitions/amc-12

2023
[33]

American invitational mathematics examination aime i & ii, 2024

Mathematical Association of America (2024). American invitational mathematics examination aime i & ii, 2024. ://maa.org/maa-invitational-competitions

2024
[34]

, Xia, M

Meng, Y. , Xia, M. and Chen, D. (2024). Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734

2024 arXiv
[35]

Neal, R. M. and Hinton, G. E. (1998). A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models. Springer, 355--368

1998
[36]

Nemirovskij, A. S. and Yudin, D. B. (1983). Problem complexity and method efficiency in optimization

1983
[37]

Gpt-4 technical report

OpenAI (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774

2023 arXiv
[38]

Introducing openai o1

OpenAI (2024). Introducing openai o1. https://openai.com/o1/

2024
[39]

Ouyang, L. , Wu, J. , Jiang, X. , Almeida, D. , Wainwright, C. , Mishkin, P. , Zhang, C. , Agarwal, S. , Slama, K. , Ray, A. et al. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35 27730--27744

2022
[40]

, Saha, A

Pacchiano, A. , Saha, A. and Lee, J. (2021). Dueling rl: reinforcement learning with trajectory preferences. arXiv preprint arXiv:2111.04850

2021 arXiv
[41]

Pang, R. Y. , Yuan, W. , Cho, K. , He, H. , Sukhbaatar, S. and Weston, J. (2024). Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733

2024 arXiv
[42]

, Sharma, A

Rafailov, R. , Sharma, A. , Mitchell, E. , Manning, C. D. , Ermon, S. and Finn, C. (2024). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

2024
[43]

, Hou, B

Rein, D. , Hou, B. L. , Stickland, A. C. , Petty, J. , Pang, R. Y. , Dirani, J. , Michael, J. and Bowman, S. R. (2024). Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

2024
[44]

and Ritter, D

Rush, S. and Ritter, D. (2024). Speculations on test-time scaling. https://srush.github.io/awesome-o1/o1-tutorial.pdf

2024
[45]

, Wolski, F

Schulman, J. , Wolski, F. , Dhariwal, P. , Radford, A. and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

2017 arXiv
[46]

, Wang, P

Shao, Z. , Wang, P. , Zhu, Q. , Xu, R. , Song, J. , Bi, X. , Zhang, H. , Zhang, M. , Li, Y. , Wu, Y. et al. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

2024 arXiv
[47]

, Zhang, C

Sheng, G. , Zhang, C. , Ye, Z. , Wu, X. , Zhang, W. , Zhang, R. , Peng, Y. , Lin, H. and Wu, C. (2024). Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256

2024 arXiv
[48]

, Co-Reyes, J

Singh, A. , Co-Reyes, J. D. , Agarwal, R. , Anand, A. , Patil, P. , Garcia, X. , Liu, P. J. , Harrison, J. , Lee, J. , Xu, K. et al. (2023). Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585

2023 arXiv
[49]

, Lee, J

Snell, C. , Lee, J. , Xu, K. and Kumar, A. (2024). Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

2024 arXiv
[50]

, Guo, Z

Tang, Y. , Guo, Z. D. , Zheng, Z. , Calandriello, D. , Munos, R. , Rowland, M. , Richemond, P. H. , Valko, M. , Pires, B. \'A . and Piot, B. (2024). Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749

2024 arXiv
[51]

, Mesnard, T

Team, G. , Mesnard, T. , Hardin, C. , Dadashi, R. , Bhupatiraju, S. , Pathak, S. , Sifre, L. , Rivi \`e re, M. , Kale, M. S. , Love, J. et al. (2024 a ). Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

2024 arXiv
[52]

, Riviere, M

Team, G. , Riviere, M. , Pathak, S. , Sessa, P. G. , Hardin, C. , Bhupatiraju, S. , Hussenot, L. , Mesnard, T. , Shahriari, B. , Ram \'e , A. et al. (2024 b ). Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

2024 arXiv
[53]

Team, Q. (2024). Qwen2.5: A party of foundation models. ://qwenlm.github.io/blog/qwen2.5/

2024
[54]

, Lavril, T

Touvron, H. , Lavril, T. , Izacard, G. , Martinet, X. , Lachaux, M.-A. , Lacroix, T. , Rozi \`e re, B. , Goyal, N. , Hambro, E. , Azhar, F. et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

2023 arXiv
[55]

, Hao, S

Wang, H. , Hao, S. , Dong, H. , Zhang, S. , Bao, Y. , Yang, Z. and Wu, Y. (2024). Offline reinforcement learning for llm multi-step reasoning. arXiv preprint arXiv:2412.16145

2024 arXiv
[56]

, Wei, J

Wang, X. , Wei, J. , Schuurmans, D. , Le, Q. , Chi, E. , Narang, S. , Chowdhery, A. and Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

2022 arXiv
[57]

, Wang, X

Wei, J. , Wang, X. , Schuurmans, D. , Bosma, M. , Xia, F. , Chi, E. , Le, Q. V. , Zhou, D. et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35 24824--24837

2022
[58]

, Akrour, R

Wirth, C. , Akrour, R. , Neumann, G. and F \"u rnkranz, J. (2017). A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18 1--46

2017
[59]

, Lan, J

Wu, T. , Lan, J. , Yuan, W. , Jiao, J. , Weston, J. and Sukhbaatar, S. (2024). Thinking llms: General instruction following with thought generation. arXiv preprint arXiv:2410.10630

2024 arXiv
[60]

, Foster, D

Xie, T. , Foster, D. J. , Krishnamurthy, A. , Rosset, C. , Awadallah, A. and Rakhlin, A. (2024). Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf. arXiv preprint arXiv:2405.21046

2024 arXiv
[61]

, Dong, H

Xiong, W. , Dong, H. , Ye, C. , Wang, Z. , Zhong, H. , Ji, H. , Jiang, N. and Zhang, T. (2024). Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning

2024
[62]

, Pan, X

Yang, R. , Pan, X. , Luo, F. , Qiu, S. , Zhong, H. , Yu, D. and Chen, J. (2024). Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402.10207

2024 arXiv
[63]

Yao, S. , Yu, D. , Zhao, J. , Shafran, I. , Griffiths, T. , Cao, Y. and Narasimhan, K. (2024). Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

2024
[64]

, Yuan, Z

Yuan, H. , Yuan, Z. , Tan, C. , Wang, W. , Huang, S. and Huang, F. (2024). Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36

2024
[65]

, Yuan, H

Yuan, Z. , Yuan, H. , Li, C. , Dong, G. , Lu, K. , Tan, C. , Zhou, C. and Zhou, J. (2023). Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825

2023 arXiv
[66]

, Broder, J

Yue, Y. , Broder, J. , Kleinberg, R. and Joachims, T. (2012). The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78 1538--1556

2012
[67]

, Harik, G

Zelikman, E. , Harik, G. , Shao, Y. , Jayasiri, V. , Haber, N. and Goodman, N. D. (2024). Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629

2024 arXiv
[68]

Zelikman, E. , Wu, Y. , Mu, J. and Goodman, N. (2022). Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35 15476--15488

2022
[69]

Zhang, S. , Yu, D. , Sharma, H. , Zhong, H. , Liu, Z. , Yang, Z. , Wang, S. , Hassan, H. and Wang, Z. (2024). Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405.19332

2024 arXiv
[70]

, Joshi, R

Zhao, Y. , Joshi, R. , Liu, T. , Khalman, M. , Saleh, M. and Liu, P. J. (2023). Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425

2023 arXiv
[71]

, Feng, G

Zhong, H. , Feng, G. , Xiong, W. , Zhao, L. , He, D. , Bian, J. and Wang, L. (2024). Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922

2024 arXiv
[72]

and Zhang, T

Zhong, H. and Zhang, T. (2024). A theoretical analysis of optimistic proximal policy optimization in linear markov decision processes. Advances in Neural Information Processing Systems, 36

2024
[73]

, Sch \"a rli, N

Zhou, D. , Sch \"a rli, N. , Hou, L. , Wei, J. , Scales, N. , Wang, X. , Schuurmans, D. , Cui, C. , Bousquet, O. , Le, Q. et al. (2022). Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625

2022 arXiv
[74]

Zhuo, T. Y. , Vu, M. C. , Chim, J. , Hu, H. , Yu, W. , Widyasari, R. , Yusuf, I. N. B. , Zhan, H. , He, J. , Paul, I. et al. (2024). Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877

2024 arXiv
[75]

Ziegler, D. M. , Stiennon, N. , Wu, J. , Brown, T. B. , Radford, A. , Amodei, D. , Christiano, P. and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

2019 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := ...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

, Kakade, S

Agarwal, A. , Kakade, S. M. , Lee, J. D. and Mahajan, G. (2021). On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22 1--76

work page 2021

[4] [4]

Introducing claude

Anthropic (2023). Introducing claude. ://www.anthropic.com/index/introducing-claude

work page 2023

[5] [5]

Azar, M. G. , Guo, Z. D. , Piot, B. , Munos, R. , Rowland, M. , Valko, M. and Calandriello, D. (2024). A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics. PMLR

work page 2024

[6] [6]

Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 324--345

work page 1952

[7] [7]

Bubeck, S. et al. (2015). Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning , 8 231--357

work page 2015

[8] [8]

, Yang, Z

Cai, Q. , Yang, Z. , Jin, C. and Wang, Z. (2020). Provably efficient exploration in policy optimization. In International Conference on Machine Learning. PMLR

work page 2020