Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An adversarial self-play game between a 'sneaky' generator that corrupts reasoning steps and a critic that must catch the corruption can train a step-level critic without human step annotations, and using that critic to guide search…

desk verdict A genuinely interesting self-play scheme for training step-level critics, held back by an uncalibrated solver-success proxy and an over-strong 'no annotation' claim; send it to reviewers. read the letter →

arxiv 2504.19162 v2 pith:KFQSQRHQ submitted 2025-04-27 cs.CL cs.AIcs.LG

Jiaqi Chen , Bang Zhang , Ruotian Ma , Peisong Wang , Xiaodan Liang , Zhaopeng Tu , Xiaolong Li , Kwan-Yee K. Wong This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords self-playcriticstep-levelerrordetectionprocessrewardmodelLLMreasoningchain-of-thoughtverificationadversarialtrainingtest-timesearchmathematical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a step-level critic for checking LLM mathematical reasoning can be trained without step-by-step human annotations. It sets up an adversarial game in which one model rewrites a correct reasoning step into a subtly wrong one and a critic must catch the error; both are updated by reinforcement learning from win/loss rewards. Across three human-annotated process benchmarks, the critic's accuracy rises over self-play rounds and surpasses same-size baselines, including a distilled R1 model. The same critic can be used during inference to flag and regenerate bad steps, improving math accuracy on MATH500 and AIME2024. If the claim holds, step verification no longer needs to wait for expensive new human annotations each time LLMs change.

What carries the argument

The central mechanism is the adversarial self-play game between a sneaky generator and a step critic. The sneaky generator converts correct reasoning steps into subtle wrong steps meant to both reduce solver success and fool the critic; the critic must classify each step and produce a brief critique. The load-bearing machinery is the automated validation rule replacing human annotation: a step counts as wrong only if the original step yields at least 75% solver success while the sneaky step yields 0%. This rule, combined with paired positive-negative samples and an asymmetric opponent-selection strategy, keeps the reinforcement learning signal informative and self-renewing across rounds.

What would settle it

Take the step pairs where the automatic 75%-to-0% rule and human annotators disagree, and check whether SPC's judgments follow the human labels or the automatic rule; if the critic tracks the automatic rule on those disagreements, the reported benchmark gains would reflect the proxy rather than genuine detection of human-annotated errors.

Watch

Extended reading notes

Core claim

SPC claims that a step-level critic's ability to detect errors in mathematical reasoning can evolve through self-play rather than through static annotation. A sneaky generator, initialized by supervised fine-tuning on correct-incorrect step pairs, edits correct steps into difficult wrong steps using predefined error types, while a critic, initialized on standardized critiques distilled from long chain-of-thought reasoning, judges each step's correctness. The paper defines a valid sneaky step by its effect on an LLM solver's success rate: the original step must let the solver succeed at least 75% of the time, and the sneaky step must bring that rate to 0%. With this automated label, the game rewards the winner with +1 and the loser with -1, and offline reinforcement learning updates both roles. A second round of evolution uses an asymmetric matchup, the round-1 sneaky generator against the round-0 critic, to keep the game balanced. The paper reports that this continuous evolution raises average accuracy from 70.8% to 77.7% on ProcessBench and from 71.0% to 75.8% on PRM800K, and that the resulting critic improves test-time search for multiple LLM solvers on MATH500 and AIME2024.

Load-bearing premise

The load-bearing premise is that the automatic rule, where a step counts as wrong only when the original step yields at least 75% solver success and the sneaky step yields 0%, is a faithful proxy for human-annotated step correctness.

Editorial extensions

If this is right

  • Step-level error detection for LLM reasoning can keep pace with new LLMs because the game generates fresh training steps from current solvers without new human annotations.
  • The trained critic can be used at test time to prune or regenerate erroneous steps, improving math accuracy on MATH500 and AIME2024, and it combines with self-consistency for further gains.
  • A critic trained only on short chain-of-thought data can generalize to long chain-of-thought reasoning models like R1 and QwQ on DeltaBench, where prior process reward models show large performance drops.
  • Balanced or asymmetric self-play is necessary for continued improvement, since an overly unbalanced game leads to performance degradation in later rounds.
  • The sneaky generator itself becomes more effective over rounds, meaning the self-play loop generates increasingly hard training examples rather than a static dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to use the automatic 75%-to-0% rule as a data engine for training reasoners directly, not just critics; if that works, the same game could supply process supervision at scale for other tasks.
  • The adversarial game could be extended beyond mathematics by defining analogous automated success-rate proxies for code generation or multi-step tool use, where step correctness also affects downstream outcomes.
  • The paper's paired-sample trick suggests a testable design for other self-play setups: collect both correct and incorrect model outputs on identical inputs so the reinforcement learning signal explicitly contrasts the two.
  • Because the paper limits its experiments to mathematical reasoning, the strongest extension would be to measure whether the same self-play critic improves step verification in scientific or formal domains where step-level labels are even scarcer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes SPC, an adversarial self-play framework for training a step-level critic of LLM mathematical reasoning. Two Qwen2.5-7B-Instruct copies are initialized by SFT: a 'sneaky generator' that rewrites correct reasoning steps into subtle erroneous steps, and a 'critic' that classifies the current step and writes a critique. The generator is trained on PRM800K-derived step transformations; the critic is trained on 21.8K PRM800K-derived critiques. In self-play, the generator proposes transformed steps, a solver-success criterion (original step success ≥75% and transformed step success 0%) labels them incorrect, and the critic attempts to detect them; rewards from Eq. (2) and (3) drive offline RL updates via Eq. (4). Two rounds of self-play improve average recall on ProcessBench from 70.8% to 77.7%, on PRM800K from 71.0% to 75.8%, and on DeltaBench from 54.9% to 60.5%. The critic is also used to verify steps during test-time search, improving MATH500 and AIME2024 accuracy for Llama-3.1-8B, Qwen2.5-32B, and DeepSeek-R1-Distill-Qwen-7B.

Significance. If the results are reproducible and the proxy-validation concern is addressed, the framework is a useful contribution: it demonstrates that an adversarial game anchored to solver outcomes can produce iterative gains on human-labeled step benchmarks, with a coherent ablation showing the importance of balanced games and solver-derived rewards. The paper provides transparent experimental details, uses three human-annotated benchmarks for evaluation, and includes ablations on game-balancing strategies. However, the current evidence does not fully establish that the critic is learning human-relevant step correctness, because the self-play labels are generated by a solver-success proxy, and the evaluation protocol in Tables 1 and 2 is modified from the official benchmarks. The claim of eliminating manual annotation is also stronger than the method supports, since both initializations use PRM800K human labels.

major comments (4)
  1. [§3.2, §3.5, §C.2] The automated validity criterion (original step success ≥75% and transformed step success 0%) is the sole source of RL labels for positive and negative sneaky steps, and yet the paper reports no measurement of agreement between this proxy and human step-incorrectness judgments on the generated sneaky steps. Because the evaluation benchmarks (ProcessBench, PRM800K, DeltaBench) are human-annotated, the headline improvements depend on the proxy being a faithful surrogate; if it instead rewards steps that merely mislead the particular solver, the critic's evolution could be an artifact of the proxy. I request an explicit human-agreement study on a random sample of the 6.4K and 6.8K self-play examples (or on PRM800K validation pairs), reporting confusion rates, and a discussion of how label noise from the proxy would affect the RL training in Eq. (4).
  2. [§4.1, Tables 1–2] The evaluation protocol in §4.1 changes the ProcessBench and DeltaBench tasks from locating the first (or all) errors in complete solutions to predicting the correctness of a single current step on a constructed 1:1 balanced subset. This makes the reported 'accuracy' and 'recall' numbers not comparable with published ProcessBench and DeltaBench leaderboard scores, because the official metrics are positional error-detection scores on the original class distribution. Since the abstract claims to 'surpass strong baselines' and 'state-of-the-art process reward models', the authors should either report results under the original official metrics or explicitly constrain all comparative claims to the modified current-step protocol, and should explain how the rebalancing affects each baseline.
  3. [Abstract, §3.2–3.3, §C.2] The claim that SPC 'eliminates the need for manual step-level annotation' is inaccurate as stated. The critic initialization uses 21.8K critiques filtered by PRM800K human step labels (§3.3, §C.2), and the sneaky-generator initialization uses correct–incorrect step pairs from PRM800K (§3.2). The contribution is better described as removing the need for new human annotation during the self-play phase, while still relying on existing human-labeled resources. The abstract and conclusion should be rephrased accordingly.
  4. [Abstract, Table 3] The abstract's claim that SPC-based search 'surpasses' state-of-the-art PRMs is not uniformly supported by Table 3. On AIME2024 with DeepSeek-R1-Distill-Qwen-7B, Self-Consistency + SPC and Self-Consistency + Qwen2.5-Math-7B-PRM800K both achieve 73.3%, and on Llama-3.1-8B with AIME2024 both achieve 6.67%. Either report statistically meaningful margins or soften the claim to 'matches or improves' on the evaluated settings.
minor comments (5)
  1. [§4.1] The sentence 'We therefore extract a 1:1 ratio of correct and erroneous steps' should specify whether the 1:1 sampling is stratified per source dataset and per solver, since PRM800K and ProcessBench have different step distributions.
  2. [Appendix B] There are typos in the prompt texts: 'converse the last step' should be 'convert the last step', 'ImcompleteCondition' should be 'IncompleteCondition', and 'input acritique' should be 'input a critique'.
  3. [Table 2] The abbreviation 'HarMean' is not defined in the table caption; it should be defined as the harmonic mean of the correct-step and error-step recalls.
  4. [§4.2, Table 3] For the MATH500 and AIME2024 runs without self-consistency, the paper says results are averaged over at least three runs, but no standard deviations are reported, making it impossible to assess whether the small differences in Table 3 (e.g., 4.27 vs 5.63 for Llama) are significant.
  5. [§5, Appendix A] The limitation statement mentions extending to more general domains, but it does not mention the proxy-validation caveat, which is the most important limitation; add a sentence acknowledging that the self-play labels depend on the solver-success criterion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the self-play critic's reward is anchored to an external solver-success criterion, and the human-labeled benchmarks used for evaluation are independent of the training labels.

full rationale

The paper's derivation chain is not self-referential. The sneaky generator is initialized from PRM800K correct/incorrect pairs and then rewarded only for steps that satisfy an external automatic validation rule: the original step must yield at least 75% solver success while the transformed step yields 0% (Section 3.2). The critic is initialized from PRM800K-derived critiques filtered by human annotations and then rewarded for detecting those externally validated sneaky steps (Sections 3.3 and 3.5). Evaluation on ProcessBench, PRM800K, and DeltaBench uses human-annotated steps that are not produced by the paper's own validity criterion, and the paper explicitly states that self-play data are generated from the PRM800K training set (Section C.2) while PRM800K evaluation uses a held-out test split (Section C.1). The abstract's claim of 'eliminating the need for manual step-level annotation' is overstated because the initialization stages use PRM800K human labels, and the solver-success proxy may not coincide with human step correctness; but that is a construct-validity or correctness concern, not a circular reduction of the claimed result to its inputs. No load-bearing self-citation or imported uniqueness theorem is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The method relies on several hand-chosen thresholds and ratios that shape the training labels, on an unvalidated proxy for step correctness, and on the correctness of PRM800K annotations. The 'sneaky generator' is not a physically invented entity, but it is a new software component with an external evaluation handle.

free parameters (4)
  • Solver success threshold pair = >=75% original, 0% sneaky
    Hand-chosen threshold for defining 'correct' vs 'incorrect' steps in automated validation; central to all self-play data generation.
  • Negative sample ratio = 1:1:1
    The sneaky generator training data is balanced among three categories; this ratio is chosen without sensitivity analysis.
  • Balancing strategy = Round 2 uses Sneaky-1 vs Critic-0
    Asymmetric evolution is selected based on win rates (32.5% vs 13.2%) rather than a principled rule; a designer choice that affects the critic's progress.
  • KL coefficient = 0.1
    Regularization strength in RL; chosen heuristically.
assumptions (4)
  • domain assumption Solver success rate is a valid proxy for step correctness
    The entire self-play labeling depends on the assumption that a step which drops solver accuracy from >=75% to 0% is genuinely incorrect and representative of human-annotated errors.
  • domain assumption PRM800K human annotations are correct and sufficient for initialization
    Both SFT initialization and critique filtering rely on PRM800K labels; errors in those labels propagate.
  • domain assumption Five predefined error types cover the space of step-level reasoning errors
    The sneaky generator only produces errors from these five types, limiting the critic's exposure to other error classes.
  • domain assumption Base model Qwen2.5-7B-Instruct is a reasonable foundation for both roles
    All results are tied to this base model; generality to other bases is not shown.
invented entities (2)
  • Sneaky generator independent evidence
    purpose: Produces subtle erroneous steps to challenge the critic
    It is a trained model, not a hypothesized entity, but it is a new component introduced by this paper. Its outputs are validated against solver success rates, providing an external handle.
  • Self-play critic
    purpose: Assesses correctness of reasoning steps
    The critic is a fine-tuned LLM; it is the central method component, but it is not an independent observable outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning." pith.science (2026). https://pith.science/paper/KFQSQRHQ

@misc{pith2026250419162,
  author       = {Pith},
  title        = {Pith review of: SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KFQSQRHQ}},
  note         = {Machine review of arXiv:2504.19162}
}
read the original abstract

Evaluating the step-by-step reliability of large language model (LLM) reasoning, such as Chain-of-Thought, remains challenging due to the difficulty and cost of obtaining high-quality step-level supervision. In this paper, we introduce Self-Play Critic (SPC), a novel approach where a critic model evolves its ability to assess reasoning steps through adversarial self-play games, eliminating the need for manual step-level annotation. SPC involves fine-tuning two copies of a base model to play two roles, namely a "sneaky generator" that deliberately produces erroneous steps designed to be difficult to detect, and a "critic" that analyzes the correctness of reasoning steps. These two models engage in an adversarial game in which the generator aims to fool the critic, while the critic model seeks to identify the generator's errors. Using reinforcement learning based on the game outcomes, the models iteratively improve; the winner of each confrontation receives a positive reward and the loser receives a negative reward, driving continuous self-evolution. Experiments on three reasoning process benchmarks (ProcessBench, PRM800K, DeltaBench) demonstrate that our SPC progressively enhances its error detection capabilities (e.g., accuracy increases from 70.8% to 77.7% on ProcessBench) and surpasses strong baselines, including distilled R1 model. Furthermore, SPC can guide the test-time search of diverse LLMs and significantly improve their mathematical reasoning performance on MATH500 and AIME2024, surpassing those guided by state-of-the-art process reward models.

Figures

Figures reproduced from arXiv: 2504.19162 by the authors.

Figure 1
Figure 1. We continuously generate reinforcement training samples for the critic through adversarial [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The framework of our proposed SPC. We randomly select a correct step along with the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Ablation study of our critic and sneaky generator. Left: The impact of different strategies [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Prompt for querying GPT-4 to collect raw data of sneaky transformation CoT. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Prompt for querying DeepSeek-R1-Distill-Qwen-7B to collect raw critiques with long CoT. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Prompt for querying GPT-4o to rewrite a long critique into a brief and standardized critique. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Prompt for training the sneaky generator. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Prompt for training the critic model. ### Analysis of the General Thought of the Partial Solution The partial solution begins by analyzing the given conditions: \( k_1 + k_2 + \cdots + k_n = 5n - 4 \) and \( \frac{1}{k_1} + \frac{1}{k_2} + \cdots + \frac{1}{k_n} = 1 \)…
Figure 9
Figure 9. Figure 9: SPC critiques on ProcessBench before and after self-play training. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A role-playing LLM that reasons about the scene and its own state before responding, trained with two semantic rewards, beats stronger baselines on role-play benchmarks.

  2. RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback

    cs.CL 2025-07 conditional novelty 6.0 of 10

    RefCritic uses reinforcement learning with two rule-based rewards, judgment accuracy and refinement accuracy, to train long-chain-of-thought critic models that improve LLM math reasoning and detect erroneous steps.

  3. Lifelong Safety Alignment for Language Models

    cs.CR 2025-05 conditional novelty 6.0 of 10

    A co-evolutionary attacker-defender loop, warmed up by strategies extracted from jailbreak papers, reduces jailbreak success rate on a robust model from 73% to 7% in two iterations.

  4. Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers

    cs.AI 2025-02 conditional novelty 5.0 of 10

    A taxonomy-based survey of bidirectional game theory and LLM research, spanning evaluation, alignment, economic competition, and LLM-driven game solving.

  5. Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models

    cs.CL 2025-06 reject novelty 2.0 of 10

    RLSC, a method that uses a language model's self-confidence as reward, is shown to improve math benchmark accuracy, but the results are undermined by training on the AIME test set and the method reduces to known self-...

Reference graph

Works this paper leans on

74 extracted references · 29 canonical work pages · cited by 5 Pith papers

  1. [1]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022

    2022

  2. [2]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    arXiv 2022

  3. [3]

    Thoughts are all over the place: On the underthinking of o1-like llms

    Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, et al. Thoughts are all over the place: On the underthinking of o1-like llms. arXiv preprint arXiv:2501.18585, 2025

    arXiv 2025

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott G...

    work page 2020

  5. [5]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  6. [6]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  7. [7]

    Gemini: A family of highly capable multimodal models, 2024

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: A family of highly capable multimodal models, 2024

    work page 2024

  8. [8]

    Introducing the next generation of claude, 2024

    Anthropic. Introducing the next generation of claude, 2024. URL https://www.anthropic. com/news/claude-3-family

    2024

Show all 74 references
  1. [9]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  2. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Qwen2 technical report

    Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  4. [12]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// qwenlm.github.io/blog/qwen2.5/

    2024

  5. [13]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 10

    2024 arXiv

  6. [14]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:277...

    2022

  7. [15]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022

    arXiv

  8. [16]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  9. [17]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  10. [18]

    Self-play fine-tuning converts weak language models to strong language models

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024

    2024 arXiv

  11. [19]

    Prover-verifier games improve legibility of llm outputs

    Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of llm outputs. arXiv preprint arXiv:2407.13692, 2024

    2024 arXiv

  12. [20]

    Openai o1 system card

    OpenAI. Openai o1 system card. preprint, 2024

    2024

  13. [21]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  14. [22]

    Qwq: Reflect deeply on the boundaries of the unknown, November 2024

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https://qwenlm.github.io/blog/qwq-32b-preview/

    2024

  15. [23]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/...

    2024

  16. [24]

    Skywork-o1 open series

    Skywork o1 Team. Skywork-o1 open series. https://huggingface.co/Skywork, Novem- ber 2024. URL https://huggingface.co/Skywork

    2024

  17. [25]

    Solving math word problems with process- and outcome-based feedback, 2022

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022

    2022

  18. [26]

    Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y . Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024

    2024

  19. [27]

    Processbench: Identifying process errors in mathematical reasoning

    Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559, 2024

    2024 arXiv

  20. [28]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24 , 2024. URL https://openreview.net/ forum?id=CxHRoTLmPX

    2024

  21. [29]

    Self-generated critiques boost reward modeling for language models

    Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, et al. Self-generated critiques boost reward modeling for language models. arXiv preprint arXiv:2411.16646, 2024. 11

    2024 arXiv

  22. [30]

    Critic-cot: Boosting the reasoning abilities of large language model via chain-of-thoughts critic

    Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji, Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, and Le Sun. Critic-cot: Boosting the reasoning abilities of large language model via chain-of-thoughts critic. arXiv preprint arXiv:2408.16326, 2024

    2024 arXiv

  23. [31]

    S 2r: Teaching llms to self-verify and self-correct via reinforcement learning

    Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S 2r: Teaching llms to self-verify and self-correct via reinforcement learning. arXiv preprint arXiv:2502.12853, 2025

    2025 arXiv

  24. [32]

    Mutual reasoning makes smaller llms stronger problem-solvers

    Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller llms stronger problem-solvers. arXiv preprint arXiv:2408.06195 , 2024

    2024 arXiv

  25. [33]

    To- ward self-improvement of llms via imagination, searching, and criticizing

    Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. To- ward self-improvement of llms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253, 2024

    2024 arXiv

  26. [34]

    O1 replication journey – part 2: Surpassing o1-preview through simple distillation big progress or bitter lesson? Github, 2024

    Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey – part 2: Surpassing o1-preview through simple distillation big progress or bitter lesson? Github, 2024. URL https://github.c...

    2024

  27. [35]

    Can large language models detect errors in long chain-of-thought reasoning?, 2025

    Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, and Bo Zheng. Can large language models detect errors in long chain-of-thought reasoning?, 2025

    2025

  28. [36]

    Measuring faithfulness in chain-of-thought reasoning

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023

    2023 arXiv

  29. [37]

    Aime 2024, 2024

    AI-MO. Aime 2024, 2024. URL https://huggingface.co/datasets/Maxwell-Jia/ AIME_2024

    2024

  30. [38]

    Introducing meta llama3: The most capable openly available llm to date, 2024

    Meta. Introducing meta llama3: The most capable openly available llm to date, 2024. URL https://ai.meta.com/blog/meta-llama-3/

    2024

  31. [39]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1

    2025

  32. [40]

    7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient

    Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nlp.notion.site/simplerl-reason , 2025. Notion Blog

    2025

  33. [41]

    Tinyzero

    Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 2025. Accessed: 2025-01-24

    2025

  34. [42]

    The lighthouse of language: Enhancing llm agents via critique-guided improvement

    Ruihan Yang, Fanghua Ye, Jian Li, Siyu Yuan, Yikai Zhang, Zhaopeng Tu, Xiaolong Li, and Deqing Yang. The lighthouse of language: Enhancing llm agents via critique-guided improvement. arXiv preprint arXiv:2503.16024, 2025

    2025

  35. [43]

    Some studies in machine learning using the game of checkers

    Arthur L Samuel. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3):210–229, 1959

    1959

  36. [44]

    Temporal difference learning and td-gammon

    Gerald Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58–68, 1995

    1995

  37. [45]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:17...

    2017 arXiv

  38. [46]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017. 12

    2017

  39. [47]

    Self-playing adversarial language game enhances llm reasoning

    Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, Xiaolong Li, et al. Self-playing adversarial language game enhances llm reasoning. Advances in Neural Information Processing Systems, 37:126515–126543, 2025

    2025

  40. [48]

    Evolving alignment via asymmetric self-play

    Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V Le, Qijun Tan, and Yuan Liu. Evolving alignment via asymmetric self-play. arXiv preprint arXiv:2411.00062, 2024

    2024 arXiv

  41. [49]

    Large language models are super- positions of all characters: Attaining arbitrary role-play via self-alignment

    Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. Large language models are super- positions of all characters: Attaining arbitrary role-play via self-alignment. arXiv preprint arXiv:2401.12474, 2024

    2024 arXiv

  42. [50]

    Self-play preference optimization for language model alignment

    Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024

    2024 arXiv

  43. [51]

    Justice or prejudice? quantifying biases in llm-as-a-judge

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024

    arXiv

  44. [52]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024

    2024 arXiv

  45. [53]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  46. [54]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  47. [55]

    Qwen2.5-math-7b, 2024

    Qwen. Qwen2.5-math-7b, 2024. URL https://huggingface.co/Qwen/Qwen2. 5-Math-7B

    2024

  48. [56]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  49. [57]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402....

    2024 arXiv

  50. [58]

    Omni-math: A universal olympiad level mathematic benchmark for large language models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024. Appendices A Limitations and So...

    arXiv

  51. [62]

    Note: For each question, you will be given a reference incorrect last step

    Wrap the final incorrect step with <Answer> </Answer>. Note: For each question, you will be given a reference incorrect last step. Y ou need to convert to this reference incorrect laststep, but **you must not reveal that you know this reference step in advance during the conve...

  52. [65]

    At the end of the response, output \\boxed{{Correct}} or \\boxed{{Incorrect}} to represent the correctness of theLast Step

    Conclusion. At the end of the response, output \\boxed{{Correct}} or \\boxed{{Incorrect}} to represent the correctness of theLast Step. ## Problem {problem} ## Partial Solution {partial_solution} ## Last Step {last_step} User Prompt Figure 5: Prompt for querying DeepSeek-R1-Di...

  53. [66]

    If the Draft Critique includes this analysis, you can directly summarize from it

    Analyzing the general thought of the Partial Solution. If the Draft Critique includes this analysis, you can directly summarize from it

  54. [67]

    You should write a new version of brief critique here

    Critique. You should write a new version of brief critique here. The current Draft Critique is accurate but may contain redundant information, such as extensive consideration and attempts at derivation and analysis of the problem. Y ou only need to select the useful analysis f...

  55. [68]

    Please draw a conclusion about the correctness of the Last Step

    Conclusion. Please draw a conclusion about the correctness of the Last Step. Based on the analysis of the provided Draft Critique, determine whether the Last Step is correct or incorrect. If it is incorrect, you should also summarize a specific type of error, such as calculati...

  56. [69]

    the critique

    Y ou need to refer to the draft critique, but pretend you didn't know this information beforehand, avoiding phrases like "the critique". Just write a new version of the critique for the Problem, Partial Solution, and Last Step

  57. [70]

    In your revised version of Critique, you only need to focus on the Last Step, and it is not necessary to solve the problemtoobtain the final answer

  58. [71]

    Instead, in the Critique, you start with an analysis of the Last Step

    DO NOT write the conclusion first and then the explanations for it. Instead, in the Critique, you start with an analysis of the Last Step. Then, in the Conclusion, drawing a conclusion about whether the Last Step is correct or incorrect

  59. [72]

    Y ou only need to focus on the correctness of the Last Step

    The current partial solution may be incomplete, which is not an error type. Y ou only need to focus on the correctness of the Last Step. ## Problem {problem} ## Partial Solution {partial_solution} ## Last Step {last_step} ## Draft Critique {draft_critique} System Prompt User P...

  60. [73]

    Clearly explain the solving process in the last step

    Analyze the given partial solution and the last step. Clearly explain the solving process in the last step

  61. [74]

    Predefined Error Types

    Choose an appropriate error type from the "Predefined Error Types" to complete the error generation. Specify your error generation method based on the current case, making the error less noticeable

  62. [75]

    Step-by-step, write out the detailed error generation process for converting the Correct Last Step into the Incorrect Step

  63. [76]

    Wrap the final incorrect step with <Answer> </Answer>. # Predefined Error Types Error Type 1: Logical Error Reference cases: -incorrect orientation of geometric figures -systematic counting error -incomplete and inaccurate listing of factors -Incorrect interpretation and conne...

  64. [77]

    Analyzing the general thought of the Partial Solution

  65. [78]

    You should write a brief critique here

    Critique. You should write a brief critique here. This part should also maintain logical coherence with the summary of thegeneral thought of the Partial Solution

  66. [79]

    At the end of the response, output <Answer>Correct</Answer> or <Answer>Incorrect</Answer> to represent the correctness of the Last Step

    Conclusion. At the end of the response, output <Answer>Correct</Answer> or <Answer>Incorrect</Answer> to represent the correctness of the Last Step. ## Problem {problem} ## Partial Solution {partial_solution} ## Last Step {last_step} System Prompt User Prompt Figure 8: Prompt ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.