Pith. sign in

REVIEW 3 major objections 4 minor 4 cited by

MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An open training pipeline reproduces near-top math reasoning in 7B and 32B models, with all data and settings released.

desk verdict A genuinely useful open-source RLVR recipe with unusually detailed pipeline documentation, but the SOTA and CAMPO-attribution claims rest on benchmarks that are both noisy and repeatedly used for design decisions. read the letter →

arxiv 2507.14683 v1 pith:QTSJT6B4 submitted 2025-07-19 cs.CL

Xingxuan Li , Yao Xiao , Dianwen Ng , Hai Ye , Yue Deng , Xiang Lin , Bin Wang , Zhanfeng Mo
show 10 more authors
Chong Zhang Yueyi Zhang Zonglin Yang Ruilin Li Lei Lei Shihao Xu Han Zhao Weiling Chen Feng Ji Lidong Bing
This is my paper
classification cs.CL
keywords mathematicalreasoningreinforcementlearningwithverifiablerewardscontext-awaremulti-stagepolicyoptimizationrepetitionpenaltyopen-sourcemodeltokenefficiencyQwen2.5AIMEbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Mathematical reasoning models have been dominated by closed systems whose training data and settings are secret. This paper tries to show that a fully open recipe can get close to the same results: it curates 719K verified reasoning traces for supervised fine-tuning, then 62K verifiable contest problems for reinforcement learning, and releases models, data, and configuration. The introduced algorithm, CAMPO, gradually lengthens the allowed response budget across training stages, penalizes repetitive loops, and uses a more accurate answer verifier to keep reward signals clean. On AIME24, AIME25, and MATH500, the resulting 7B and 32B models match or beat other open Qwen-2.5-based reasoning models while using fewer tokens. A sympathetic reader would care because the entire stack is public, so the recipe can be reused and tested rather than guessed at.

What carries the argument

The load-bearing object is CAMPO, Context-Aware Multi-Stage Policy Optimization, an RL objective built on DAPO's decoupled clipping. It does three things: it trains in stages with progressively larger maximum response lengths, so early rollouts are cheap and later stages allow deeper reasoning; it subtracts a repetition score $f(o_i)$, computed as the fraction of tokens inside a detected repeating loop and weighted toward early repetition, from the group-normalized advantage; and it replaces the stock verifier with a cascade verifier that handles units, constants, percentages, and numerical precision, so correct answers are less often penalized. The objective also masks out batches where all rollouts are correct or all incorrect, keeping gradient updates informative.

What would settle it

Run the released MiroMind-M1-SFT-7B and MiroMind-M1-RL-7B on a decontaminated, not-yet-public contest set such as a future AIME or a freshly written set of equally hard problems, and compare against the same benchmarks used during development. If the gap between SFT and RL shrinks or reverses on the fresh set, the reported gains are partly an artifact of selection on the evaluation benchmarks. A second check is to retrain with the same 62K problems but with all design choices frozen before any AIME scores are observed; if the final results differ materially, the development loop itself was responsible.

Watch

Extended reading notes

Core claim

The paper's central claim is that a two-stage SFT-then-RLVR pipeline, with all data and settings disclosed, produces near-state-of-the-art mathematical reasoning in 7B and 32B models built on Qwen-2.5. The SFT stage alone reaches 60.4 on AIME24, 45.0 on AIME25, and 94.6 on MATH500 starting from Qwen-2.5-Math-7B. Adding CAMPO raises the 7B model to 73.4, 57.8, and 96.7, and the 32B model to 77.5, 65.6, and 96.4, with shorter average responses than the Skywork-OR1-32B-Preview comparison at matched accuracy. The paper attributes these gains to three mechanisms: staged context expansion, a repetition penalty that down-weights early repeated loops, and a cascade verifier with human-curated edge-case fixes. The contribution is framed as transparency as much as performance, because every component needed to reproduce the models is released.

Load-bearing premise

The load-bearing premise is that tuning the data selection, repetition penalty, verifier, and stage schedule on AIME24, AIME25, and MATH500 does not inflate the scores later reported on those same benchmarks, so a held-out evaluation would show similar gains.

Editorial extensions

If this is right

  • Other teams can run the released SFT and RLVR stages as-is, so near-SOTA 7B math performance no longer requires proprietary data or closed training details.
  • Models trained with CAMPO should keep most of their accuracy when output budgets are capped well below 32K tokens, since the multi-stage schedule already forces concise reasoning.
  • The cascade verifier's edge-case fixes should transfer to any RLVR pipeline that uses rule-based answer checking, reducing false-negative rewards on answers with units, constants, percentages, or precision differences.
  • Because the paper reports per-token efficiency as a first-class result, future reasoning-model papers will likely need to report both accuracy and output length to be comparable.
  • The same SFT-plus-RLVR recipe can be applied to newer base models than Qwen-2.5, since CAMPO is not tied to a particular backbone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The released 719K SFT corpus makes it possible for others to measure how much of the final score comes from imitating an existing strong model's traces versus from the RL stage, because the SFT-only checkpoint is also released.
  • If the repetition penalty is the main driver of token efficiency, the same penalty should improve pass rate per token in other verifiable domains such as code generation; the paper does not test this.
  • The paper's single-stage comparison suggests the two-stage schedule may be primarily a compute-saving device, meaning a practitioner with ample GPU budget could skip the curriculum and still get similar accuracy.
  • The evaluation-stability discussion implies that differences of a few percent between models on 30-question AIME sets may be within noise, so readers should compare 64-run averages and confidence intervals rather than single runs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces the MiroMind-M1 series, a fully open-source two-stage pipeline (SFT on 719K curated math traces followed by RLVR on 62K verifiable problems) built on Qwen2.5, together with a new RL algorithm, CAMPO, that combines multi-stage context-length expansion, an adaptive repetition penalty, and an improved verifier. The authors report state-of-the-art or competitive results among Qwen-2.5-based 7B and 32B open models on AIME24, AIME25, and MATH500, along with token-efficiency advantages over Skywork-OR1. They release models, datasets, and configurations. The central claims are (1) that the released pipeline reproduces or exceeds the performance of comparable open RLMs and (2) that CAMPO is the cause of the observed gains.

Significance. If the claims hold, this is a valuable contribution to reproducibility in reasoning-model research: the full stack is released, the training pipeline is described in unusual detail, decontamination against AIME24/AIME25/MATH500 is attempted, and the stability analysis in §4.5.2 is honest about the noise inherent in AIME-scale evaluation. The potential significance is real, both as a reproducible recipe and as an algorithmic proposal for token-efficient RLVR. However, the empirical evidence as presented does not yet establish the headline claims. The evaluation is confined to the three benchmarks that were used throughout development for design decisions, and the paper's own noise analysis shows that most of the reported margins are within one standard deviation. The attribution to CAMPO is also underdetermined because no standard GRPO or DAPO control is trained from the same checkpoints. These are fixable with additional experiments, so the contribution remains potentially valuable.

major comments (3)
  1. [§3.2.2, §4.3.2, §4.3.3, §4.5.2, Table 5, Figure 13] The central empirical claim of Table 5 is not robust because the evaluation protocol both tunes and reports on the same small benchmarks, and the paper's own stability analysis shows that the reported margins are within noise.
  2. [§4.3, Eq. (5)-(6), §4.5.2] The attribution of gains to CAMPO is underdetermined without a DAPO/GRPO control run from the same SFT checkpoint and data.
  3. [§4.3.1, Figure 9, Figure 5] The token-efficiency claim lacks a joint accuracy-cost comparison with uncertainty, and is based on the same benchmarks used for tuning.
minor comments (4)
  1. [§4.5.2 (title and text)] These typos are distracting but do not affect the technical content.
  2. [§3.2.1] Reporting the mean without variance is inconsistent with the paper's transparency about evaluation noise.
  3. [Table 5] The table's scope and missing values should be clarified.
  4. [§4.3.2, Figure 6] This is a reproducibility issue that is presentation-level in the current version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the reported scores are empirical, decontaminated evaluations rather than quantities built from the method's own definitions or fitted parameters.

full rationale

MiroMind-M1 is an empirical systems paper, not a derivation paper. The AIME24/AIME25/MATH500 scores in Table 5 are measurements of released checkpoints, not quantities that are equal to any training objective or fitted parameter by construction. The SFT and RLVR data are explicitly decontaminated against those benchmarks (Sections 3.1.2 and 4.1), so the results are not generated by benchmark leakage. Development decisions such as long-trajectory selection (Table 4), the repetition penalty (Figure 6), the verifier upgrade (Figure 8), and the multi-stage schedule (Figure 12) were indeed ablated on the same three benchmarks that are later reported, which creates a genuine selection-bias and overfitting risk; the paper's own Section 4.5.2 concedes that two-sided standard deviation on AIME24 can exceed 8% across 64 runs, and several reported margins over Skywork-OR1 are smaller than that noise. That is a threat to external validity and to the robustness of the SOTA claim, but it is not circularity: no equation in the paper reduces a reported result to an input, and no fitted parameter is renamed as a prediction. The only self-citations (e.g., Zhang et al. 2025) appear in related-work context as survey support for curriculum RLVR and are not load-bearing for any of the paper's quantitative claims. The CAMPO attribution is underdetermined because no DAPO/GRPO control is trained from the same checkpoints, but that is a missing-control issue rather than a circular one. Overall, no load-bearing argument collapses into its own assumptions.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the quality of the filtered RL training set and the reliability of rule-based verification, neither of which is independently quantified. Several ad hoc thresholds (20-character answer cutoff, 10-gram dedup, pass-rate difficulty bands) shape the dataset and therefore the models, but no sensitivity analysis is given. No new entities are introduced.

free parameters (6)
  • max response length schedule = 16K/32K/49K tokens for 32B; 16K/32K for 7B
    Chosen by hand to balance training cost and capability; the stage transition criterion ('once response length saturates') is not precisely specified.
  • repetition penalty f(oi) = proportion of tokens inside detected repeating loop
    Hand-designed scoring function; the loop-detection algorithm and penalty magnitude are not described.
  • difficulty filter pass-rate thresholds = exclude fully correct and fully incorrect rollouts; keep pass rate near 0.8 for anchors
    Ad hoc thresholds chosen from prior findings, with no sensitivity analysis; they determine which 62K problems enter RL training.
  • answer length cutoff = answers longer than 20 characters excluded
    Ad hoc filter to ensure verifier reliability; biases the training distribution toward short-answer problems.
  • 10-gram deduplication threshold = 10-gram overlap
    Near-duplicate removal threshold chosen without stated justification or sensitivity analysis.
  • clip ratio distributions phi_low and phi_high = not disclosed
    The CAMPO objective samples epsilon_low and epsilon_high from stage-dependent distributions that are never specified; this is a hidden tuning choice.
assumptions (5)
  • domain assumption A rule-based verifier can reliably judge correctness for the filtered 62K problems
    Section 4.1 asserts verifier accuracy is critical; the entire RL reward depends on it, yet verifier accuracy is not quantified.
  • domain assumption N-gram decontamination removes all benchmark overlap
    Section 3.1.2 applies n-gram overlap filtering; reformulated benchmark questions could evade this filter.
  • domain assumption AIME24, AIME25, and MATH500 scores are stable indicators of reasoning ability despite high variance
    Section 4.5.2 admits two-sided standard deviation exceeds 8% over 64 runs on AIME24, yet headline comparisons treat small differences as meaningful.
  • domain assumption Training on longer CoT traces is generally better, motivating CAMPO
    Section 3.2.2 shows 'long' beats 'random' at 30K and 50K scales, but the mechanism (complexity versus verbosity) is assumed, not proven.
  • domain assumption Existing Qwen-2.5 base models and DeepSeek-R1-distill checkpoints are suitable initializations
    The pipeline inherits the capabilities and biases of these pretrained backbones and claims no control over them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization." pith.science (2026). https://pith.science/paper/QTSJT6B4

@misc{pith2026250714683,
  author       = {Pith},
  title        = {Pith review of: MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QTSJT6B4}},
  note         = {Machine review of arXiv:2507.14683}
}
read the original abstract

Large language models have recently evolved from fluent text generation to advanced reasoning across diverse domains, giving rise to reasoning language models. Among these domains, mathematical reasoning serves as a representative benchmark as it requires precise multi-step logic and abstract reasoning, which can be generalized to other tasks. While closed-source RLMs such as GPT-o3 demonstrate impressive reasoning capabilities, their proprietary nature limits transparency and reproducibility. Although many open-source projects aim to close this gap, most of them lack sufficient openness by omitting critical resources such as datasets and detailed training configurations, which hinders reproducibility. To contribute toward greater transparency in RLM development, we introduce the MiroMind-M1 series, a set of fully open-source RLMs built on the Qwen-2.5 backbone that match or exceed the performance of existing open-source RLMs. Specifically, our models are trained in two stages: SFT on a carefully curated corpus of 719K math-reasoning problems with verified CoT trajectories, followed by RLVR on 62K challenging and verifiable problems. To enhance the robustness and efficiency of the RLVR process, we introduce Context-Aware Multi-Stage Policy Optimization, an algorithm that integrates length-progressive training with an adaptive repetition penalty to encourage context-aware RL training. Our model achieves state-of-the-art or competitive performance and superior token efficiency among Qwen-2.5-based open-source 7B and 32B models on the AIME24, AIME25, and MATH benchmarks. To facilitate reproducibility, we release the complete stack: models (MiroMind-M1-SFT-7B, MiroMind-M1-RL-7B, MiroMind-M1-RL-32B); datasets (MiroMind-M1-SFT-719K, MiroMind-M1-RL-62K); and all training and evaluation configurations. We hope these resources will support further research and foster community advancement.

Figures

Figures reproduced from arXiv: 2507.14683 by the authors.

Figure 1
Figure 1. Length distribution for of datasets. Data Decontamination To avoid data contami￾nation and ensure fair evaluation, we perform de￾contamination for training data with respect to our target evaluation sets. Specifically, we apply an N￾gram overlap filter to remove any training samples whose questions match those in Math500, AIME24, or AIME25. This filtering effectively prevents data leakage. Data Statistics of the Fin… view at source ↗
Figure 2
Figure 2. Initial composition distribution. Big-Math comprises HARP and reformu￾lated machine outputs (Albalak et al., 2025); Skywork-OR1-RL-Data (He et al., 2025a) contains only maths [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 4
Figure 4. Inclusion-Exclusion Criteria. Overview of the filtering strategy used to construct the final training [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Average token length of model responses computed across all rollout attempts, evaluated over 64 independent runs on the corresponding test sets. This analysis includes both correct and incorrect answers, highlighting overall response efficiency regardless of correctnes…
Figure 6
Figure 6. Figure 6: The training process is more stable with [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Average token count of model responses con [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 9
Figure 9. Figure 9: Comparison of MiroMind-M1-RL-32B and Skywork-OR1-32B-Preview on AIME24 and AIME25 across different maximum response lengths. Nevertheless, our 32B model still lags behind certain SOTA models that also employ RL for training. For instance, Skywork-OR1-32B-Preview surpas…
Figure 10
Figure 10. Figure 10: The model’s performance steadily improves throughout the training process. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Response length trend during two-stage training. Under the 16K generation cap, the model [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Performance trend of the model trained using a single-stage 32K max context length schema. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Evaluation stability assessment: 64 repeated evaluations of [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs

    cs.AI 2026-04 unverdicted novelty 7.0 of 10

    A multi-agent framework reconstructs the evolutionary graph of post-training LLM datasets, revealing domain patterns like vertical refinement in math data and systemic issues like redundancy and benchmark contaminatio...

  2. AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    AnE combines Truth Anchor Expansion and Scaffold-Stripping to deliver 10.3% gains on eight multimodal reasoning benchmarks for MLLMs.

  3. Trust Region On-Policy Distillation

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    TrOPD stabilizes on-policy distillation for LLMs with trust-region learning, outlier estimation, and off-policy guidance, outperforming prior OPD methods on reasoning and code benchmarks.

  4. A Survey of Reinforcement Learning for Large Reasoning Models

    cs.CL 2025-09 accept novelty 3.0 of 10

    A survey compiling RL methods, challenges, data resources, and applications for enhancing reasoning in large language models and large reasoning models since DeepSeek-R1.

Reference graph

Works this paper leans on

27 extracted references · 27 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models

    Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502.17387,

    arXiv

  2. [4]

    Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, ...

    arXiv

  3. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    21 Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  4. [6]

    Skywork open reaonser series

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reaonser series. Notion Blog, 2025a. Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen...

    arXiv

  5. [7]

    Am-thinking-v1: Advancing the frontier of reasoning at 32b scale

    Yunjie Ji, Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Han Zhao, and Xiangang Li. Am-thinking-v1: Advancing the frontier of reasoning at 32b scale. arXiv preprint arXiv:2505.08311, 2025a. Yunjie Ji, Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Han Zhao, and Xiangang Li. Am-thinking-v1: Advancing the frontie...

    arXiv 2001

  6. [8]

    Kimi-Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haoz...

    arXiv

  7. [9]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050,

    arXiv

  8. [10]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783,

    arXiv

Show all 27 references
  1. [11]

    Mistral-AI, :, Abhinav Rastogi, Albert Q. Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, Léonard Blier, Lucile Saulnier, Matthieu Dinot, Maxime Darrin, Neha Gupta, Roman Soletskyi, Saga...

    arXiv

  2. [12]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan...

    arXiv

  3. [14]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  4. [15]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295,

    arXiv

  5. [16]

    Open Thoughts, 2025a

    OpenThoughts Team. Open Thoughts, 2025a. Qwen Team. Qwen3: Think deeper, act faster, 2025b. Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, 2025c. Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Yunjie Ji, Han Zhao, and Xiangang Li. Deep...

    arXiv

  6. [18]

    Dai, and Quoc V Le

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Proceedings of ICLR, 2022a. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia,...

    arXiv

  7. [19]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768,

    arXiv

  8. [20]

    Towards large reasoning models: A survey of reinforced reasoning with large language models

    Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning m...

    arXiv

  9. [22]

    What’s behind ppo’s collapse in long-cot? value optimization holds the secret

    Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What’s behind ppo’s collapse in long-cot? value optimization holds the secret. arXiv preprint arXiv:2503.01491,

    arXiv

  10. [23]

    Harp: A challenging human-annotated math reasoning benchmark

    Albert S Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, and Aaditya K Singh. Harp: A challenging human-annotated math reasoning benchmark. arXiv preprint arXiv:2412.08819,

    arXiv

  11. [24]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Ming...

    arXiv

  12. [25]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892,

    arXiv

  13. [26]

    100 days after deepseek-r1: A survey on replication studies and more directions for reasoning language models

    Chong Zhang, Yue Deng, Xiang Lin, Bin Wang, Dianwen Ng, Hai Ye, Xingxuan Li, Yao Xiao, Zhanfeng Mo, Qi Zhang, and Lidong Bing. 100 days after deepseek-r1: A survey on replication studies and more directions for reasoning language models. arXiv preprint arXiv:2505.00551,

    arXiv

  14. [27]

    1.4 million open-source distilled reasoning dataset to empower large language model training

    Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li. 1.4 million open-source distilled reasoning dataset to empower large language model training. arXiv preprint arXiv:2503.19633,

    arXiv

  15. [2017]

    Plangenllms: A modern survey of llm planning capabilities

    Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan, and Fei Liu. Plangenllms: A modern survey of llm planning capabilities. arXiv preprint arXiv:2502.11221,

    arXiv

  16. [2020]

    Proximal policy optimiza- tion algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  17. [2023]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476,

    arXiv

  18. [2024]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  19. [2025]

    Reasoning does not necessarily improve role-playing ability

    Xiachong Feng, Longxu Dou, and Lingpeng Kong. Reasoning does not necessarily improve role-playing ability. arXiv preprint arXiv:2502.16940,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.