Pith. sign in

REVIEW 3 major objections 5 minor 52 references

Probability-Consistent Preference Optimization for Enhanced LLM Reasoning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PCPO claims that adding token-level probability consistency to final-answer correctness when selecting preference pairs improves LLM math reasoning beyond outcome-only baselines.

desk verdict A plausible new preference-pair selection heuristic for math reasoning, with broad but statistically weak experiments and an untested mechanism; worth refereeing, not worth believing at face value. read the letter →

arxiv 2505.23540 v1 pith:HBWF4XMT submitted 2025-05-29 cs.CL

Yunqiao Yang , Houxing Ren , Zimu Lu , Ke Wang , Weikang Shi , Aojun Zhou , Junting Pan , Mingjie Zhan
show 1 more author
Hongsheng Li
This is my paper
classification cs.CL
keywords preferenceoptimizationmathematicalreasoninglargelanguagemodelstoken-levelconsistencydirectself-consistencychain-of-thoughtPCPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that preference data for math reasoning can be improved without new annotations by looking inside responses, not just at their final answers. Its method, Probability-Consistent Preference Optimization (PCPO), pairs each wrong answer with the correct answer whose token-by-token generation probabilities are most similar, on the theory that such pairs highlight a localized logical divergence. It trains the next model with a direct preference optimization (DPO) loss plus a language-modeling term, both weighted by that similarity score. Across four open LLMs and four math benchmarks, the paper reports that PCPO consistently beats two outcome-only baselines—IRPO, which pairs by gold-answer correctness, and ScPO, which pairs by answer self-consistency—with the largest gap on the hardest test set (AMC23). If this holds, the internal probability structure of generated tokens is a reusable signal for curating training pairs.

What carries the argument

The load-bearing object is the pair-weighted score $s_w(y_w|y_l)$, the mean over tokens aligned by longest-common-subsequence matching of $\exp(-|\log P_w - \log P_l|)$, where the two log-probabilities come from the current model's next-token predictions for the chosen and rejected responses. A high score means the correct and incorrect responses assign similar probabilities to the same tokens, which the paper interprets as a sign that the pair differs in a localized, learnable way rather than in overall style. The score does two jobs: it selects, for each rejected response, the chosen response with the maximum $s_w$, and it reweights both the DPO and the NLL terms in the training loss. Candidate pairs are pre-filtered by Levenshtein distance to keep computation feasible.

What would settle it

Train a control version of PCPO that keeps the same loss and the same Levenshtein-filtered correct-wrong candidate pairs but selects the pair at random instead of by the highest $s_w$; if random selection matches PCPO's Pass@1 on GSM8K and MATH-500, the token-consistency score is not what drives the reported gains.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that response-internal coherence carries information that final-answer correctness misses, and that this information can be read off the model's own token probabilities. For each problem, PCPO generates several responses, splits them by whether the final answer matches the gold answer, keeps only correct-wrong pairs that are close under Levenshtein distance, and scores each pair by the average exponential negative absolute log-probability difference over matched tokens. The correct response with the highest such score is paired with each wrong response, and the pair is used to train the next iteration with a weighted DPO+NLL loss. The paper reports that this selection rule outperforms outcome-only rules for Llama-3-8B-Instruct, Mathstral-7B-v0.1, Qwen2.5-7B-Instruct, and Qwen2.5-Math-7B-Instruct on GSM8K, MATH-500, OlympiadBench, and AMC23, and that the curated data also improve other DPO variants including RPO, IPO, ORPO, and TDPO.

Load-bearing premise

The load-bearing premise is that the pair-weighted score $s_w$, the mean token-probability similarity between a correct and an incorrect response, tracks the internal logical coherence that makes a preference pair instructive; the paper selects pairs by it but does not independently validate that link.

Editorial extensions

If this is right

  • Preference data for math reasoning can be enriched without new human labels; the consistency score is computed from the current model's own token probabilities.
  • PCPO-curated pairs improve plain DPO training as well as the PCPO loss, separating data-quality gains from loss-design gains.
  • The framework generalizes to other preference losses, improving RPO, IPO, ORPO, and TDPO on most reported benchmarks.
  • The largest reported gains are on AMC23, where PCPO's Pass@1 on Llama-3-8B-Instruct leads ScPO by 7.5 points and IRPO by 15 points.
  • Token-probability scoring adds roughly 15% more GPU hours per training iteration, which the paper counts as acceptable for the accuracy increase.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the consistency score really localizes reasoning errors, the same correct-wrong pairing rule could transfer to code generation or proof checking, where a verifier supplies labels without human annotation; the paper does not test those domains.
  • Because the score is computed with the same model that produced the responses, an overconfident model could assign high similarity to a confidently wrong answer; a calibration-aware variant might change which pairs are selected.
  • The appendix's Levenshtein analysis shows the top five edit-distance ranks contain over 95% of final pairs, so a smaller candidate pool may preserve most of the benefit at lower compute cost.
  • The reported metrics are exact-match final-answer accuracies, so the paper does not show whether paired training changes step-level reasoning quality; that remains a testable question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Probability-Consistent Preference Optimization (PCPO), an iterative preference-optimization pipeline for mathematical reasoning. In each iteration, the current model generates N responses, which are partitioned by final-answer correctness; for each rejected response, the top-k closest chosen responses by Levenshtein distance form candidate pairs. A pair-weighted score s_w, defined in Eq. (3) as the mean over matched tokens of exp(-|log P_w - log P_l|) normalized by the rejected response length, is then used to select the chosen response with the highest token-level consistency. The resulting pairs are trained with a DPO+NLL loss weighted by s_w. Experiments cover four 7-8B instruction-tuned models on GSM8K, MATH-500, Olympiadbench, and AMC23, with Pass@1 and Maj@8 metrics, plus ablations on data curation, loss, and DPO variants.

Significance. If the reported gains are real and attributable to the proposed token-level consistency criterion, PCPO would be a simple and modest-cost enhancement to preference-based reasoning training, and the release of code and the use of a fixed training set are assets. The paper also includes several useful control comparisons, such as training different curation methods under a shared DPO loss (Table 2) and applying the curation to multiple DPO variants (Table 4). However, the evidence as presented does not yet isolate the proposed mechanism, and the 'consistently outperforms' claim is stronger than the table supports. This is best viewed as a promising empirical study requiring additional control experiments and statistical rigor.

major comments (3)
  1. [Section 4.1, Table 1] The abstract's and Section 4.1's claim that PCPO 'consistently outperforms' IRPO and ScPO is not supported by the table. Qwen2.5-7B-Instruct on AMC23 Pass@1 at M1 is 45.0 for PCPO versus 50.0 and 55.0 for IRPO and ScPO. For Qwen2.5-Math-7B-Instruct, assuming the two unlabeled PCPO rows are M1 and M2 in order, the M2 Olympiadbench Pass@1 is 44.3, below IRPO (44.6) and ScPO (44.7). Many other comparisons differ by 1 point or less on test sets of 40 to 500 items, and no error bars, multiple seeds, or significance tests are reported. Because the central conclusion rests on these comparisons, this is a load-bearing issue.
  2. [Section 2.1/2.2 and Table 2] The experiments do not isolate the contribution of the token-level consistency score s_w from the Levenshtein pre-filter and the correct/wrong split. Candidate pairs are first restricted by Levenshtein distance (top k=8); s_w is then used only to choose among those candidates. In Table 2, the IRPO+DPO and ScPO+DPO controls do not apply the same Levenshtein pre-filter, so the observed gains could come from edit-distance filtering or the correctness split rather than from s_w. A control that trains on a random, or Levenshtein-minimal, chosen response from the same candidate set is needed to support the paper's attribution of the gains to token-level consistency.
  3. [Section 2.2, Eq. (3); Section 5.4, Table 5] The paper's load-bearing premise is that higher s_w indicates greater 'internal logical coherence' and yields better preference pairs, but no experiment links s_w to answer correctness, reasoning validity, or downstream generalization. The case study in Figure 3 and Table 5 only demonstrates that s_w separates two surface-level token patterns; it does not show that pairing by s_w, rather than by Levenshtein distance, improves training outcomes. A direct test, such as comparing high-s_w pairs against low-s_w pairs matched for Levenshtein distance, would substantiate the mechanism.
minor comments (5)
  1. [Section 2.2, Eq. (3)] The text says the score is normalized by 'the total length of the preference pair,' but the equation divides by l_yl, the token length of the rejected response; please reconcile this inconsistency.
  2. [Table 1] The two PCPO rows for Qwen2.5-Math-7B-Instruct are not labeled M1/M2, and the column header appears as 'MA TH-500' with an unwanted space.
  3. [Section 4.1] The sentence 'We provide a theoretical analysis based on some literature' is not a theoretical analysis of PCPO; either supply an actual argument or remove the sentence.
  4. [Section 3 and Section 4] There are typos: 'leaning rate' should be 'learning rate', and the Section 4 heading 'Main Rresults' should be 'Main Results'.
  5. [Appendix A] The experiments use k=8 for candidate filtering, but Figure 4 shows that ranks 1-5 already cover 95.4% of selected pairs; the text suggests threshold 5 but does not use it in the experiments, which should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: PCPO's selection score is a defined heuristic, not a fitted predictor of the reported accuracies.

full rationale

The paper's derivation chain is self-contained and non-circular. The token consistency score in Eq. (2) and the pair-weighted score in Eq. (3) are explicit definitions; the selection rule in Eqs. (4)-(5) uses these definitions to choose preference pairs, and the PCPO loss in Section 2.3 incorporates sw as a weight. The reported accuracies on GSM8K, MATH-500, Olympiadbench, and AMC23 are measured externally after training; no equation reduces these accuracies to sw, nor is any parameter fitted to the evaluation benchmarks. The use of the model's own token probabilities to construct training data is an iterative self-training scheme, not a logical reduction of the output to the input. There are no load-bearing self-citations: the cited references to Qwen-2.5 and other models are external technical reports, not prior work by this paper's authors, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The case study in Section 5.4 merely illustrates that the argmax rule selects pairs with high sw by construction, which is a definitional property rather than a separate empirical prediction; it does not by itself establish the paper's central claim of improved reasoning. The paper's limitation statements about dependence on gold answers and computational cost are acknowledged constraints, not circular steps. Concerns that the sw contribution is not isolated from the Levenshtein filter, and that Tables 1 lacks error bars, are empirical support and reporting issues, not circularity. Therefore no significant circularity is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The PCPO pipeline is a heuristic composition of known techniques. Its central metric is an in-paper construction and is loaded with the assumption that probability agreement equals logical coherence. Hyperparameters such as N, k, alpha, and beta are carried from prior work or chosen by convenience; only modest ablations of the loss and data are provided, and none test the selection metric directly.

free parameters (5)
  • Number of generated responses per prompt N = 16
    Controls candidate pool size; no ablation reported; higher N would increase compute and potentially change selected pairs.
  • Levenshtein candidate threshold k = 8 (Appendix A later recommends 5)
    Filters candidate pairs; the paper used 8, then uses the observed distribution to argue 5 would suffice, which looks post hoc.
  • DPO coefficient beta = 0.5
    Copied from ScPO; scales the reward margin in the weighted DPO term.
  • NLL coefficient alpha = 1.0
    Copied from ScPO; balances the weighted NLL term against preference loss.
  • Training iterations T = 2 (M1, M2)
    Experiments stop at two iterations; no analysis of more iterations.
assumptions (5)
  • domain assumption Token-level probability differences on matched tokens proxy the internal logical coherence of responses.
    Section 2.2 defines ct from |log Pw - log Pl| and uses sw to select pairs, but no evidence links this quantity to reasoning validity.
  • domain assumption Levenshtein-distance filtering keeps informative preference pairs without discarding useful correct answers.
    Section 2.1 and Appendix A; threshold is heuristic and later revised from 8 to 5 based on selected-pair statistics.
  • domain assumption SequenceMatcher longest-common-subsequence alignment maps semantically corresponding tokens between chosen and rejected responses.
    Appendix B; token alignment for paraphrased or restructured steps may be arbitrary.
  • domain assumption The fixed 30k training mixture is sufficient for preference optimization on the target benchmarks.
    Section 3; exact 7.5k subsets are not specified, and no data scaling analysis is given.
  • domain assumption Standard DPO plus NLL remains a valid training objective when weighted by the pair score.
    Section 2.3; the weighted loss is inspired by IRPO and ScPO but not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Probability-Consistent Preference Optimization for Enhanced LLM Reasoning." pith.science (2026). https://pith.science/paper/HBWF4XMT

@misc{pith2026250523540,
  author       = {Pith},
  title        = {Pith review of: Probability-Consistent Preference Optimization for Enhanced LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HBWF4XMT}},
  note         = {Machine review of arXiv:2505.23540}
}
read the original abstract

Recent advances in preference optimization have demonstrated significant potential for improving mathematical reasoning capabilities in large language models (LLMs). While current approaches leverage high-quality pairwise preference data through outcome-based criteria like answer correctness or consistency, they fundamentally neglect the internal logical coherence of responses. To overcome this, we propose Probability-Consistent Preference Optimization (PCPO), a novel framework that establishes dual quantitative metrics for preference selection: (1) surface-level answer correctness and (2) intrinsic token-level probability consistency across responses. Extensive experiments show that our PCPO consistently outperforms existing outcome-only criterion approaches across a diverse range of LLMs and benchmarks. Our code is publicly available at https://github.com/YunqiaoYang/PCPO.

Figures

Figures reproduced from arXiv: 2505.23540 by the authors.

Figure 1
Figure 1. Overview of the PCPO method. The pipeline mainly consists of three steps. (1) Given a prompt set, utilize [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Rewards of PCPO and DPO. The chosen and rejected responses reward comparison of PCPO and DPO training on the same preference pairs. DPO loss. Here, "IRPO+DPO" denotes training data curated by IRPO with the DPO loss, and sim￾ilarly for other methods. The results demonstrate that models trained with PCPO’s preference pairs consistently outperform those trained with IRPO or ScPO pairs. Specifically, PCPO M2 achieves 1.… view at source ↗
Figure 3
Figure 3. A few right and wrong responses from the same prompt. The four responses can be divided into two groups, where each has a similar response pattern. Pairs a & c b & c a & d b & d sweighted 0.791 0.525 0.559 0.793 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Frequency Distribution and Cumulative Per [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: The Match Function pipeline. For a given pair of chosen and rejected responses, we first utilize the current iteration model Mt to tokenize them and then use the algorithm 1 to get the longest common token subsequences, as highlighted in different colors. (e.g., "Pleas…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 7 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157

    arXiv 2024

  4. [4]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier - Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing...

    arXiv doi:10.48550/arxiv.2305.10403 2023

  5. [5]

    Anthropic. 2024. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The claude 3 model family: Opus, sonnet, haiku

    2024

  6. [6]

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447--4455. PMLR

    2024

  7. [7]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv doi:10.48550/arxiv.2309.16609 2023

  8. [8]

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073

    arXiv 2022

Show all 52 references
  1. [9]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

    2017

  2. [10]

    Fenia Christopoulou, Ronald Cardenas, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. 2024. Sparsepo: Controlling preference alignment of llms via sparse token masks. arXiv preprint arXiv:2410.05102

    2024

  3. [11]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    u rnkranz and Eyke H \

    Johannes F \"u rnkranz and Eyke H \"u llermeier. 2010. Preference learning and ranking by pairwise comparison. In Preference learning, pages 65--82. Springer

    2010

  6. [14]

    Zeyu Gan and Yong Liu. 2024. Towards a theoretical understanding of synthetic data in llm post-training: A reverse-bottleneck perspective. arXiv preprint arXiv:2410.01720

    2024 arXiv

  7. [15]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv...

    2024 arXiv

  8. [16]

    Wilbert Jan Heeringa. 2004. Measuring dialect pronunciation differences using levenshtein distance

    2004

  9. [17]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  10. [18]

    Edwin Hewitt and Leonard J Savage. 1955. Symmetric measures on cartesian products. Transactions of the American Mathematical Society, 80(2):470--501

    1955

  11. [19]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751

    2019 arXiv

  12. [20]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170--11189

    2024

  13. [21]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 a . Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  14. [22]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \' e lio Renard Lavaud, Marie - Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, ...

    2023 doi arXiv

  15. [23]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L \' e lio Renard Lavaud, Lucile Saulnier, Marie...

    2024 doi arXiv

  16. [24]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  17. [25]

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. 2024. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face re...

    2024

  18. [26]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050

    2023 arXiv

  19. [27]

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. 2024 a . Rho-1: Not all tokens are what you need. arXiv preprint arXiv:2404.07965

    2024 arXiv

  20. [28]

    Zicheng Lin, Tian Liang, Jiahao Xu, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. 2024 b . Critical tokens matter: Token-level contrastive estimation enhence llm's reasoning capability. arXiv preprint arXiv:2411.19943

    2024 arXiv

  21. [29]

    Mathematical Association of America . 2023. American Mathematics Competitions (AMC) . https://www.maa.org/math-competitions

    2023

  22. [30]

    Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al. 2023. Inverse scaling: When bigger isn't better. arXiv preprint arXiv:2306.09479

    2023 arXiv

  23. [31]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196

    2024 arXiv

  24. [32]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  25. [33]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  26. [34]

    Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733

    2024 arXiv

  27. [35]

    Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. 2024. Self-consistency preference optimization. arXiv preprint arXiv:2411.04109

    2024 arXiv

  28. [36]

    Alec Radford. 2018. Improving language understanding by generative pre-training

    2018

  29. [37]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  30. [38]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008--3021

    2020

  31. [39]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . https://doi.org/10....

    2023 doi arXiv

  32. [40]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 doi arXiv

  33. [41]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  34. [42]

    Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Qiaozhi He, Murun Yang, Tong Xiao, Chunliang Zhang, Tongran Liu, and Jingbo Zhu. 2024. Lrhp: Learning representations for human preferences via preference pairs. arXiv preprint arXiv:2410.04503

    2024 arXiv

  35. [43]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  36. [44]

    Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. 2020. Consistency of a recurrent language model with respect to incomplete decoding. arXiv preprint arXiv:2002.02492

    2020 arXiv

  37. [45]

    Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes F \"u rnkranz. 2017. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1--46

    2017

  38. [46]

    Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2023. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682

    2023 arXiv

  39. [47]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 doi arXiv

  40. [48]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024 b . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  41. [49]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024 c . Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122

    2024 arXiv

  42. [50]

    Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. 2023. Rlcd: Reinforcement learning from contrast distillation for language model alignment. arXiv preprint arXiv:2307.12950

    2023 arXiv

  43. [51]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020

    2024 arXiv

  44. [52]

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. 2024. Token-level direct preference optimization. arXiv preprint arXiv:2404.11999

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.