Pith. sign in

REVIEW 4 major objections 6 minor 49 references

The paper claims that cooperative Coach–Player training can improve mathematical reasoning in LLMs with no external training data during the loop, because the Coach is rewarded with the Player's measured learning progress.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:08 UTC pith:7RRY4RV6

load-bearing objection The coach–player reward recipe is new and plausible, but the headline gains over R-Zero come from a compute-unmatched comparison; the paper's own matched-step table cuts the margin to near zero. the 4 major comments →

arxiv 2602.02979 v3 pith:7RRY4RV6 submitted 2026-02-03 cs.CL cs.LG

CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning

Ran Li , Zeyuan Liu , Yinghao Chen , Bingxiang He , Jiarui Yuan , Zixuan Fu , Weize Chen , Jinyi Hu
show 3 more authors
Chen Qian Zhiyuan Liu Maosong Sun
This is my paper
classification cs.CL cs.LG
keywords data-free reinforcement learningcoach-player collaborationcurriculum generationmathematical reasoningpseudo-labelingself-playGRPOlearning-progress reward
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CPMobius claims that a large-language-model reasoner can keep improving with no human-written training tasks once the loop begins, as long as two models play cooperative roles: a Coach that proposes math problems and a Player that solves them. The Coach is rewarded with a single number—the change in the Player's accuracy on a fixed validation set after one update—so it is pushed to propose tasks that produce real learning, not tasks that stump the Player. The Player learns by solving Coach-generated problems, using majority-voted pseudo-labels and GRPO, so no ground-truth answers are needed during the loop. On four base models spanning pretraining, SFT, and RL-trained checkpoints, the paper reports gains of roughly 2–5 points overall and out of distribution, beating prior unsupervised methods. If the result holds, it matters because reasoning post-training would no longer need to scale with the cost of human-curated tasks and labels.

Core claim

The paper's central claim is that a fully self-generated curriculum can improve LLM mathematical reasoning: a Coach model proposes a batch of instructions, a Player model attempts each instruction multiple times, majority voting turns the attempts into pseudo-labels, and GRPO updates the Player against those pseudo-labels. The Coach is then updated by REINFORCE with an instruction-level reward equal to the Player's average accuracy on that instruction multiplied by a single progress scalar, the change in the Player's validation-set accuracy after the update. On four base models spanning pretraining, SFT, and RL stages, the loop reports overall average gains of roughly 2–5 points and out-of-d

What carries the argument

The load-bearing object is the Coach's multiplicative instruction reward R^Coach_i = R^Player_i · Δt: R^Player_i is the fraction of the Player's n rollouts that agree with the majority-voted answer on instruction i, and Δt is the difference in validation-set accuracy before and after one GRPO update. This product reinforces a proposed question only when the Player solved it in training and the whole update produced a measurable accuracy gain, so it is the mechanism that turns a language model into a curriculum designer. A difficulty filter that accepts only instructions with rollout accuracy between 0.2 and 0.8 keeps tasks at the Player's capability frontier. The Player side is standard GRPO

Load-bearing premise

The entire Coach signal reduces to one number—the change in validation accuracy after a single 16-sample GRPO update—and the method assumes this number is a trustworthy, transferable measure of whether the Coach's questions caused real learning, applied uniformly to every instruction in the batch.

What would settle it

Keep the loop identical but replace Δt in R^Coach_i with noise drawn to match the observed mean and standard deviation of Δt; if the Player still gains the reported 3–5 points, the learning-progress signal is not doing the causal work. Alternatively, run the same two-agent loop with a Coach rewarded by Player accuracy only (Δt set to 1) and compare final benchmark scores.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Post-training of a reasoner can in principle run with no human-written tasks and no external curriculum: after one-time initialization, the only signals are majority-vote pseudo-labels and a fixed validation set.
  • Because gains appear on models already tuned by SFT or RL (for example, +2.6 on OpenMath-Nemotron-1.5B and +4.9 on Qwen2.5-Math-7B-Instruct), the approach is claimed to push beyond apparent performance ceilings rather than only rescuing weak models.
  • The curriculum is adaptive and emergent: Coach-proposed problems lengthen and become more complex over training while Player responses shorten, which the paper reads as automatic difficulty calibration.
  • The improvements transfer outside the validation domain: scores rise on AIME, MATH, Minerva, and Olympiad-Bench even though the progress signal is computed on AMC, so the learned ability is not just AMC memorization.
  • All three design choices are load-bearing: removing Coach updates, Coach warm-up, or the 0.2–0.8 filter reduces overall average accuracy by roughly 3–5 points on Qwen2.5-Math-1.5B.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The single scalar Δt applied to every instruction is the most fragile link: if Δt is noisy or measures only AMC-specific drift, the Coach reward becomes random, and the reported gains would reduce to Player self-training with pseudo-labels. An easy check is to compare task distributions from a Coach trained with real Δt versus one trained with a shuffled or randomly rewired Δt.
  • Nothing in the objective rewards task diversity, so the Coach could collapse to a narrow family of templates even while Δt stays positive. A testable extension is to add a diversity term or measure the Coach's output entropy over training; the current paper tracks difficulty and length but not topical diversity.
  • The loop should transfer to any domain with verifiable outcomes—code correctness, formal proofs, or puzzle solving—because the math-specific pieces are only the verifier and the choice of validation set.
  • The 20%-AMC experiment hints that the validation set can be small, but it also raises the question of how far Δt can be shrunk before noise dominates; a useful stress test is to vary D_val size and measure when Coach updates stop helping.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces CPMobius, a cooperative two-agent RL loop for improving LLM mathematical reasoning without an external training set during co-evolution. A Coach LLM proposes math questions and is trained with REINFORCE using a reward that multiplies the Player's average self-consistency reward on those questions by the Player's validation-accuracy improvement Δt on the AMC set. The Player LLM is trained with GRPO to agree with the majority-vote pseudo-label over its own n responses. Experiments on four base models (Qwen2.5-Math-1.5B, OpenMath-Nemotron-1.5B, OctoThinker-3B-Hybrid-Zero, Qwen2.5-Math-7B-Instruct) report gains on six benchmarks, with headline improvements of +4.9 overall and +5.4 OOD on Qwen2.5-Math-7B-Instruct relative to RENT and R-Zero. The appendix also contains ablations, a 20%-AMC leakage check, and a compute-matched comparison against R-Zero.

Significance. If the claimed results hold, CPMobius is a useful alternative to adversarial self-play and a concrete instantiation of self-generated curriculum RL. Strengths include released code, ablations isolating Coach update/warm-up/filter, a leakage check with 20% of AMC, and, importantly, an explicit compute-matched comparison in Appendix A.3. However, that compute-matched experiment substantially weakens the central empirical claim rather than supporting it: at the same number of training steps, CPMobius is roughly on par with, or worse than, R-Zero, so the advertised advantage is not yet established. The conceptual novelty is real, but the paper's headline quantification is not supported by its own numbers.

major comments (4)
  1. [Appendix A.3, Table 3 vs. Abstract and Table 1] The compute-matched comparison directly contradicts the central claim. Table 1 compares CPMobius after 1000 steps (batch 16, rollouts 16, i.e. about 1000*16*16 = 256,000 Player rollouts) with R-Zero at 'Iter 3' (60 total steps, solver batch 128, rollouts 5, i.e. at most 60*128*5 = 38,400 rollouts). Appendix A.3's own Table 3 shows that at CPMobius's 60th step it is worse than R-Zero on Qwen2.5-Math-1.5B (average 26.9 vs. 27.1; OOD 24.2 vs. 24.7) and only +0.5 OOD on Qwen2.5-Math-7B-Instruct (34.7 vs. 34.2), not the +4.2 claimed in the abstract. The statement in A.3 that the advantage stems from 'raising the performance upper bound' is not supported by these numbers. The authors must either provide a properly compute-matched comparison with comparable total rollouts, or explicitly reframe the contribution as a more-compute method.
  2. [Section 5.1, Eq. (5): 'Data-free' claim is qualified in a way that weakens the paper's framing] The abstract says CPMobius works 'without relying on any external training data,' but Section 5.1 states that the Coach is warmed up on 4K PRIME Eurus-2-RL-Data, and the Coach's reward in Eq. (5) uses Δt computed on the AMC validation set with ground-truth labels. This is external supervision used during co-evolution. The ablation (Table 2) shows that removing Coach warm-up drops average accuracy from 28.8 to 23.7 and removing Coach update drops it to 25.3, so both the external warm-up and the AMC-based reward are essential. The phrase 'data-free Player training with an optionally warmed-up Coach' should replace 'data-free' throughout the abstract and introduction, and the role of AMC labels as a reward signal should be clearly disclosed in the contribution statement.
  3. [Section 4.2, Eq. (8): Player reward is self-referential and unverified] The Player's reward ri,j = I[yi,j = y*_i] compares each answer against the majority-vote pseudo-label of the Player's own n samples. This measures self-consistency, not correctness, and the same self-consistency is used in the difficulty filter (Eq. 4). If the model collapses toward a single deterministic answer pattern, RPlayer can increase without any improvement in true accuracy. The paper should report pseudo-label accuracy against ground truth on a held-out set over the course of training, or at least show that self-consistency correlates with oracle accuracy on representative benchmarks. Without such evidence, the reported benchmark gains could partly reflect increased determinism rather than improved reasoning.
  4. [Section 5.2, Table 1: No error bars, seeds, or statistical significance] All main results are single runs. Several headline differences are small (e.g., CPMobius 28.8 vs. RENT 27.1 on Qwen2.5-Math-1.5B; OOD 26.8 vs. 24.7), and the compute-matched gaps in Table 3 are even smaller. Without multiple seeds or bootstrap confidence intervals, the claimed improvements, especially the OOD gains of 0.5–2 points, are not statistically distinguishable. At minimum, the authors should report 2–3 seeds with variance for the main comparisons, and ideally for the matched-step experiment as well.
minor comments (6)
  1. [Title and abstract] The title contains 'CPM¨obius' with a stray diacritic; use 'CPMobius' consistently.
  2. [Figure 3 caption] The caption says 'OpenMath-Memotron-1.5B' — typo for 'OpenMath-Nemotron-1.5B'. Also, 'Answer Consistency' is not formally defined; specify how it is computed.
  3. [Section 4.1] 'guaranties' should be 'guarantees'.
  4. [Section 5.1, Baselines] The sentence 'because RENT was trained on AMC' is vague. State whether RENT uses AMC prompts only, or also ground-truth labels, and clarify whether R-Zero uses the same validation set.
  5. [Appendix A.5] Coach Training lists 'Number of Rollout: 1', but the main text says the Coach generates 16 questions. Clarify how m=16 instructions are obtained when Coach rollout is 1, and whether the difficulty filter resamples per slot.
  6. [Appendix A.4, Table 4] The 20%-AMC leakage check is reported only for Qwen2.5-Math-1.5B. If the claim is general, report it for at least one other model.

Circularity Check

0 steps flagged

No significant circularity: the Coach–Player loop is externally grounded by validation-set feedback and benchmark evaluation, and no prediction reduces to a fitted input.

full rationale

The central claim of CPMobius is that a cooperative Coach–Player loop improves mathematical reasoning without external training data during co-evolution. The derivation chain is not circular: the Player's reward (Eq. 8) uses a pseudo-label y*_i that is the majority vote of the Player's own n samples, making the reward a self-consistency signal rather than an external correctness label. However, the paper explicitly frames this as an unsupervised verifiable reward ('In the absence of labels, verification can be performed in an unsupervised manner using self-consistency...'), and it does not claim that this self-consistency score is an external prediction. The Coach's reward (Eq. 5) multiplies this self-consistency by an external, ground-truth-based progress signal Δt computed on the fixed AMC validation set, so the Coach's learning is grounded in real accuracy changes rather than being defined solely by the model's own outputs. The final claim of accuracy improvement is evaluated on held-out benchmarks, including OOD benchmarks excluding AMC, so the method is externally falsifiable. Appendix A.4 additionally tests with only 20% AMC data to address leakage concerns. No load-bearing self-citation appears: references to the authors' prior work (e.g., Internet of Agents) are inspirational only, and no uniqueness theorem or fitted parameter is invoked to force the result. The matched-step comparison in Table 3 is weaker than the headline comparison, but that is a fairness/correctness concern, not a circularity one. Therefore the paper does not exhibit the specific equation-to-equation reduction or fit-renamed-as-prediction pattern that would constitute circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central method rests on self-generated pseudo-labels and a validation-accuracy progress signal; neither is an independently verified ground-truth benchmark. The Coach warm-up and AMC validation set are external supervision, so the 'data-free' framing is narrower than the title suggests.

free parameters (4)
  • difficulty filter thresholds = 0.2–0.8
    Tasks are accepted only when the Player's self-consistency accuracy is between 0.2 and 0.8; thresholds are chosen by hand with no sensitivity analysis.
  • Coach warm-up data size = 4,000 PRIME Eurus-2-RL-Data samples
    The Coach is warmed up on 4K human-curated problem-solution pairs (Section 5.1), which is external supervision used to make the Coach capable of posing questions.
  • validation set D_val = AMC
    The environment feedback Δt is computed on AMC; this choice defines the reward signal and can bias the Coach toward AMC-like problems.
  • training steps = 1000
    All runs use 1000 Coach/Player update steps; the compute-matched comparison uses the 60-step checkpoint.
axioms (4)
  • domain assumption Majority-vote pseudo-labels approximate ground-truth answers well enough for GRPO to improve reasoning.
    Section 4.2 Eq. 8: The Player reward is based on y*_i = majority vote over the Player's own samples; if the majority is wrong, the reward is wrong. No external verification is used.
  • ad hoc to paper A single scalar Δt on AMC validation measures genuine learning progress caused by the Coach's instructions.
    Section 4.1 Eq. 5: Each round's Coach reward multiplies per-instruction R^Player_i by Δt; the same Δt is applied to all 16 instructions, so credit assignment is global.
  • domain assumption The Coach, warmed up on 4K PRIME samples, can generate math problems in the 0.2–0.8 self-consistency band from scratch.
    Section 5.1 Coach Model Selection; ablation shows removing warm-up degrades results, so this initialization is load-bearing.
  • standard math GRPO and REINFORCE update rules behave as specified with the listed hyperparameters, and the KL penalties prevent distribution collapse.
    Section 3.1 and Appendix A.5; no formal convergence guarantee is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 19830 in / 12552 out tokens · 125069 ms · 2026-08-03T05:08:33.973209+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) have demonstrated strong potential in complex reasoning, yet their progress remains fundamentally constrained by reliance on massive high-quality human-curated tasks and labels, either through supervised fine-tuning (SFT) or reinforcement learning (RL) on reasoning-specific data. This dependence renders supervision-heavy training paradigms increasingly unsustainable, with signs of diminishing scalability already evident in practice. To overcome this limitation, we introduce CPM\"obius (CPMobius), a collaborative Coach-Player paradigm for data-free reinforcement learning of reasoning models. Unlike traditional adversarial self-play, CPM\"obius, inspired by real world human sports collaboration and multi-agent collaboration, treats the Coach and Player as independent but cooperative roles. The Coach proposes instructions targeted at the Player's capability and receives rewards based on changes in the Player's performance, while the Player is rewarded for solving the increasingly instructive tasks generated by the Coach. This cooperative optimization loop is designed to directly enhance the Player's mathematical reasoning ability. Remarkably, CPM\"obius achieves substantial improvement without relying on any external training data, outperforming existing unsupervised approaches. For example, on Qwen2.5-Math-7B-Instruct, our method improves accuracy by an overall average of +4.9 and an out-of-distribution average of +5.4, exceeding RENT by +1.5 on overall accuracy and R-zero by +4.2 on OOD accuracy. Our codebase has been released at https://github.com/thunlp/CPMobius.

Figures

Figures reproduced from arXiv: 2602.02979 by Bingxiang He, Chen Qian, Jiarui Yuan, Jinyi Hu, Maosong Sun, Ran Li, Weize Chen, Yinghao Chen, Zeyuan Liu, Zhiyuan Liu, Zixuan Fu.

Figure 1
Figure 1. Figure 1: CPMobius ¨ starts with the coach proposing tasks of suitable difficulty. The player learns by solving these tasks, then reviews on a predefined environment. Finally, the coach adjusts the next training plan based on the player’s performance. 1. Introduction Large Language Models (LLMs) (OpenAI, 2025a; Yang et al., 2024a; Touvron et al., 2023) have demonstrated remarkable capabilities in complex reasoning t… view at source ↗
Figure 2
Figure 2. Figure 2: The illustration on the conceptual layered architecture on the design of CPMobius ¨ . The iterative process includes four stages. Coach designs plan: The coach gives instructions of suitable difficulty based on the player’s current ability. Player executes training: The player executes each instruction multiple times, uses majority voting to get pseudo-labels, and updates with GRPO. Player evaluates result… view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of the training dynamics of CPMobius ¨ using validation results on AMC dataset. The curves are smoothed with Time Weighted EMA, where CPMobius ¨ shows consistent performance improvement for different base models. ment (23.6% relative gain), suggesting that models with domain-specific pre-training provide strong foundations for CPMobius ¨ ’s optimization approach. (2) SFT-enhanced models show … view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of the training dynamics on CPMobius ¨ and different ablation experiments using validation results on AMC dataset. ually generates more complex tasks to adapt to the Player’s growing capabilities. Meanwhile, the Player’s response length is decreasing, suggesting that the Player is generating increasingly efficient answers. Details can be found in Appendix A.6 and Appendix A.7. Together, these… view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of the growing trend of output length of different models. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 37 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    K., Bai, Y., Baker, B., Bao, H., et al

    Agarwal, S., Ahmad, L., Ai, J., Altman, S., Applebaum, A., Arbus, E., Arora, R. K., Bai, Y., Baker, B., Bao, H., et al. gpt-oss-120b & gpt-oss-20b model card. arxiv preprint arXiv: 2508.10925, 2025 a

    Pith/arXiv arXiv 2025

  3. [3]

    The unreasonable effectiveness of entropy minimization in llm reasoning

    Agarwal, S., Zhang, Z., Yuan, L., Han, J., and Peng, H. The unreasonable effectiveness of entropy minimization in llm reasoning. arXiv preprint arXiv:2505.15134, 2025 b

    Pith/arXiv arXiv 2025

  4. [4]

    H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al

    Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Self-questioning language models

    Chen, L., Prabhudesai, M., Fragkiadaki, K., Liu, H., and Pathak, D. Self-questioning language models. arXiv preprint arXiv:2508.03682, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  7. [7]

    Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence

    Chen, W., You, Z., Li, R., Guan, Y., Qian, C., Zhao, C., Yang, C., Xie, R., Liu, Z., and Sun, M. Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence. arXiv preprint arXiv:2407.07061, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  9. [9]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arxiv preprint arXiv: 2507.06261, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Process reinforcement through implicit rewards

    Cui, G., Yuan, L., Wang, Z., Wang, H., Li, W., He, B., Fan, Y., Yu, T., Xu, Q., Chen, W., et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Reinforcement pre-training

    Dong, Q., Dong, L., Tang, Y., Ye, T., Sun, Y., Sui, Z., and Wei, F. Reinforcement pre-training. arXiv preprint arXiv:2506.08007, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    M., Radenovic , F., Guzm \'a n , F., Zhang , F., Synnaeve , G., Lee , G., Anderson , G

    Grattafiori , A., Dubey , A., Jauhri , A., Pandey , A., Kadian , A., Al-Dahle , A., Letman , A., Mathur , A., Schelten , A., Vaughan , A., Yang , A., Fan , A., Goyal , A., Hartshorn , A., Yang , A., Mitra , A., Sravankumar , A., Korenev , A., Hinsvark , A., Rao , A., Zhang , A., Rodriguez , A., Gregerson , A., Spataru , A., Roziere , B., Biron , B., Tang ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  13. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    O lympiad B ench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems

    He, C., Luo, R., Bai, Y., Hu, S., Thai, Z., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., Liu, J., Qi, L., Liu, Z., and Sun, M. O lympiad B ench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Associ...

    2024

  15. [15]

    Measuring mathematical problem solving with the math dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    Pith/arXiv arXiv 2021

  16. [16]

    Cooper: Co-optimizing policy and reward models in reinforcement learning for large language models

    Hong, H., Yan, Y., Wu, X., Hou, G., Zhang, W., Lu, W., Shen, Y., and Xiao, J. Cooper: Co-optimizing policy and reward models in reinforcement learning for large language models. arXiv preprint arXiv:2508.05613, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    R-zero: Self-evolving reasoning llm from zero data, 2025

    Huang, C., Yu, W., Wang, X., Zhang, H., Li, Z., Li, R., Huang, J., Mi, H., and Yu, D. R-zero: Self-evolving reasoning llm from zero data, 2025. URL https://arxiv.org/abs/2508.05004

    Pith/arXiv arXiv 2025

  18. [18]

    Openai o1 system card

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Pag: Multi-turn reinforced llm self-correction with policy as generative verifier

    Jiang, Y., Xiong, Y., Yuan, Y., Xin, C., Xu, W., Yue, Y., Zhao, Q., and Yan, L. Pag: Multi-turn reinforced llm self-correction with policy as generative verifier. arXiv preprint arXiv:2506.10406, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V

    Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V. V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., and Misra, V. Solving quantitative reasoning problems with language models. ArXiv, abs/2206.14858, 2022. URL https://api.semanticscholar.org/CorpusID:250144408

    Pith/arXiv arXiv 2022

  22. [22]

    Confidence is all you need: Few-shot rl fine-tuning of language models

    Li, P., Skripkin, M., Zubrey, A., Kuznetsov, A., and Oseledets, I. Confidence is all you need: Few-shot rl fine-tuning of language models. arXiv preprint arXiv:2506.06395, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Urpo: A unified reward & policy optimization framework for large language models

    Lu, S., Wang, H., Chen, Z., and Tang, Y. Urpo: A unified reward & policy optimization framework for large language models. arXiv preprint arXiv:2507.17515, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset

    Moshkov, I., Hanley, D., Sorokin, I., Toshniwal, S., Henkel, C., Schifferer, B., Du, W., and Gitman, I. Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.16891, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Gpt-5 system card

    OpenAI. Gpt-5 system card. Blog, 2025 a

    2025

  26. [26]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. Blog, 2025 b

    2025

  27. [27]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  28. [28]

    Maximizing confidence alone improves reasoning

    Prabhudesai, M., Chen, L., Ippoliti, A., Fragkiadaki, K., Liu, H., and Pathak, D. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22660, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    and Cong, X

    Qian, C. and Cong, X. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 6 0 (3): 0 1, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  31. [31]

    Seed, B., Chen, J., Fan, T., Liu, X., Liu, L., Lin, Z., Wang, M., Wang, C., Wei, X., Xu, W., et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914, 2025

    arXiv 2025

  32. [32]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Hybridflow: A flexible and efficient rlhf framework

    Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., and Wu, C. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp.\ 1279--1297, 2025

    2025

  34. [34]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm

    Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017

    Pith/arXiv arXiv 2017

  35. [35]

    Kimi k2: Open agentic intelligence, 2025

    Team, K. Kimi k2: Open agentic intelligence, 2025. URL https://arxiv.org/abs/2507.20534

    Pith/arXiv arXiv 2025

  36. [36]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Zephyr: Direct distillation of lm alignment

    Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., Von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    V., Chi, E

    Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw

    2023

  39. [39]

    Octothinker: Mid-training incentivizes reinforcement learning scaling

    Wang, Z., Zhou, F., Li, X., and Liu, P. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512, 2025. URL https://arxiv.org/abs/2506.20512

    Pith/arXiv arXiv 2025

  40. [40]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  41. [41]

    Self-rewarding correction for mathematical reasoning

    Xiong, W., Zhang, H., Ye, C., Chen, L., Jiang, N., and Zhang, T. Self-rewarding correction for mathematical reasoning. arXiv preprint arXiv:2502.19613, 2025

    Pith/arXiv arXiv 2025

  42. [43]

    Qwen2 technical report

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024 b

    Pith/arXiv arXiv 2024

  43. [44]

    Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J

    Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 3, 2024

    Pith/arXiv arXiv 2024

  44. [45]

    S., and Katabi, D

    Zha, K., Gao, Z., Shen, M., Hong, Z.-W., Boning, D. S., and Katabi, D. Rl tango: Reinforcing generator and verifier together for language reasoning. arXiv preprint arXiv:2505.15034, 2025

    arXiv 2025

  45. [46]

    Right question is already half the answer: Fully unsupervised llm reasoning incentivization, 2025 a

    Zhang, Q., Wu, H., Zhang, C., Zhao, P., and Bian, Y. Right question is already half the answer: Fully unsupervised llm reasoning incentivization, 2025 a . URL https://arxiv.org/abs/2504.05812

    Pith/arXiv arXiv 2025

  46. [47]

    Critique-grpo: Advancing llm reasoning with natural language and numerical feedback

    Zhang, X., Sun, H., Zhang, Y., Feng, K., Lu, C., Yang, C., and Meng, H. Critique-grpo: Advancing llm reasoning with natural language and numerical feedback. arXiv preprint arXiv:2506.03106, 2025 b

    Pith/arXiv arXiv 2025

  47. [48]

    Absolute zero: Reinforced self-play reasoning with zero data

    Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Lin, M., Wang, S., Wu, Q., Zheng, Z., and Huang, G. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03335, 2025

    Pith/arXiv arXiv 2025

  48. [49]

    Ttrl: Test-time reinforcement learning

    Zuo, Y., Zhang, K., Sheng, L., Qu, S., Cui, G., Zhu, X., Li, H., Zhang, Y., Long, X., Hua, E., et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084, 2025

    Pith/arXiv arXiv 2025

  49. [50]

    Self-adapting language models

    Zweiger, A., Pari, J., Guo, H., Aky \"u rek, E., Kim, Y., and Agrawal, P. Self-adapting language models. arXiv preprint arXiv:2506.10943, 2025

    arXiv 2025