Pith. sign in

REVIEW 4 major objections 6 minor 83 references

Zero RL applied directly to a trillion-parameter base model yields frontier-level math reasoning, and the paper argues the best behaviors emerge from scale, not from engineered reasoning heuristics.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 06:31 UTC pith:SWDCDYLD

load-bearing objection Useful 1T RL engineering recipe buried under an overclaimed 'bitter lesson' narrative that the paper's own reward design contradicts. the 4 major comments →

arxiv 2607.12395 v2 pith:SWDCDYLD submitted 2026-07-14 cs.CL

Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

Xinyu Tang , Qianggang Cao , Yurou Liu , Yuliang Zhan , Xiaochong Lan , Yifan Li , Yuchen Yan , Han Peng
show 8 more authors
Zican Dong Zhenduo Zhang Tianshu Wang Xinyu Kong Zujie Wen Wayne Xin Zhao Zhiqiang Zhang Jun Zhou
This is my paper
classification cs.CL
keywords zero reinforcement learningtrillion-parameter modelschain-of-thought reasoningRLVRemergent behaviorsself-distillationmathematical reasoningbitter lesson
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that reinforcement learning with verifiable rewards — no human-annotated reasoning examples — is not just viable but much stronger at the trillion-parameter scale. Its main evidence is a four-stage training pipeline that bootstraps chain-of-thought reasoning from a pretrained 1T base model and reaches 92.5% on AIME 2026 after the second RL stage, far ahead of the same recipe on a 104B model. The authors frame this as validation of the 'bitter lesson': computation and scale beat hand-crafted heuristics, since the 1T model spontaneously adopts structured formatting, self-verification, parallel exploration, anthropomorphic asides, and even 'context anxiety'. A sympathetic reader would care because it suggests that frontier reasoning may not require curated demonstration data or explicit reasoning templates — only a stable way to let a sufficiently large model explore.

Core claim

At its core, this paper claims that zero RL scales: starting from a 1-trillion-parameter base model with no supervised chain-of-thought data, a deliberately minimal pipeline of clipped importance-sampling policy gradients, a self-distillation reset, and tiered second/third RL stages produces a model whose math reasoning outperforms an identically trained 104B model and reaches competitive levels on seven benchmarks. The paper further claims that the scaling process is a two-phase phenomenon — an early discovery phase that expands the model's reasoning boundary, followed by a sharpening phase that refines the policy within it — and that advanced cognitive behaviors such as step-structured rea

What carries the argument

The load-bearing mechanism is the multi-stage training pipeline. First-stage RL uses a clipped importance-ratio policy gradient in which the clipped ratio carries a stop-gradient, and replaces the numerator of the importance ratio with actual training-engine logits to cancel floating-point mismatch between training and rollout engines — a correction the ablations show prevents collapse. A self-distillation phase compresses correct traces and resets the engine gap before resuming RL; second-stage RL switches to sample-level loss normalization to stop unbounded length growth; third-stage RL partitions questions into difficulty tiers with different token budgets and system prompts. The reward i

Load-bearing premise

The paper's 'spontaneous emergence' claim assumes that the observed structured formatting and context anxiety are caused by scale rather than by its own format reward, system prompts, and truncation windows — and this assumption is not directly tested at the 1T scale.

What would settle it

Run the same trillion-parameter pipeline with the format reward removed and with no tier-based truncation prompts; if structured formatting and context anxiety disappear or change character, the 'spontaneous emergence from scale' claim fails. A cheaper signal: check whether the 104B model trained with the identical reward design exhibits the same five behaviors — if it does, the behaviors are reward-driven, not scale-driven.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Scaling zero RL from 104B to 1T parameters raises both the performance ceiling (84.2% vs 65.3% on AIME 2026 after first-stage RL) and the rate of improvement.
  • Each pipeline stage contributes measurable gains: self-distillation and second-stage RL lift AIME 2026 to 92.5%, and the tiered third stage gives users a controllable accuracy/efficiency trade-off.
  • Reasoning quality is separable from final accuracy: the proposed comprehensibility/reproducibility/efficiency framework shows the 1T model's traces win pairwise readability judgments, distill into weaker models using 100K samples (vs 800K for a prior approach), and use about half the tokens on shared-correct problems.
  • The pass@1024 plateau followed by continued pass@1 growth supports a discovery-then-sharpening account of RL for reasoning, rather than a single mechanism.
  • If the emergent behaviors are genuinely unsupervised, explicit supervision of formatting, verification, and parallel search may become unnecessary for sufficiently large models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because a format reward and tier-based system prompts with truncation windows appear in every training stage, the paper's 'spontaneous' emergence reading is not yet isolated from these design choices; ablating them at 1T scale would settle the causal story.
  • The 'context anxiety' behavior is a plausible direct artifact of truncation windows rather than a general cognitive trait; a simple test is to run inference without truncation and see whether the panic-guessing behavior disappears.
  • The discovery/sharpening conclusion is entangled with the deliberate mid-training switch to harder data; a same-data control would determine whether the plateau in pass@1024 is a phase transition or a data-curriculum effect.
  • The CoT-quality framework could extend beyond math: the same reproducibility-via-distillation metric could rank reasoning traces for coding or agent planning tasks, where token efficiency and transferability matter as much as correctness.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports a multi-stage 'zero RL' pipeline applied to a 1-trillion-parameter MoE base model, with a 104B model as the scale comparison. Stage 1 uses a clipped-importance policy gradient with token-level loss and KL regularization; this is followed by self-distillation, a second RL stage with sample-level loss, and a third tier-based RL stage with tier-specific system prompts and truncation windows. The authors report strong benchmark results (e.g., Ring-2.5-1T-Zero reaches 92.5% on AIME 2026 after the second RL stage) and claim three headline findings: scale improves sample efficiency and ceilings; training proceeds through distinct 'discovery' and 'sharpening' phases; and the model 'spontaneously develops' advanced cognitive behaviors, making hand-crafted heuristics redundant. The paper also proposes a CoT-quality evaluation framework with comprehensibility, reproducibility, and efficiency dimensions. The central 'spontaneous emergence' interpretation is, however, contradicted by the paper's own reward design: a format reward is active in every RL stage, and the tier/truncation setup is hand-designed.

Significance. If the technical results are reproducible, the engineering contribution is substantial: this appears to be one of the first demonstrations of stable RLVR directly on a 1T-parameter base model, with a clear set of stability mechanisms (training-inference ratio correction, KL penalty, sample-level loss normalization) and supporting ablations on the 104B model. The CoT-quality evaluation framework is a useful addition to the field, and the distillation-based reproducibility results are interesting. However, the abstract and Section 6 make a much stronger scientific claim about scale-driven spontaneous emergence that is not supported by the experiments as reported. The paper should be revised to separate the solid scaling/engineering results from the unsupported emergence narrative; the current framing overstates what the evidence shows.

major comments (4)
  1. [Abstract; §6; Eq. (10); §4.4.3] The claim that the model 'spontaneously discovers these advanced cognitive strategies from scratch, without any explicit supervision or auxiliary rewards' is contradicted by the paper's own methods. The reward in Eq. (10) is r_i = r_acc,i + r_format,i, and §4.4.3 states that the format reward is adopted 'across all training stages' and that Format B requires <think>...</think><answer>...</answer> plus EOS. Thus the structured formatting and the 'failing to format correctly yields zero reward' behavior in §6.4 are explicit consequences of the reward signal, not spontaneous discoveries. Figure 7 compares two format reward specifications, not a no-format condition, and no 1T run without r_format is reported. The claim must be weakened to acknowledge the presence of this minimal hand-designed reward, or supported by a no-format ablation.
  2. [§5.2; Fig. 10b; §4.2] The 'discovery vs. sharpening' conclusion is not cleanly identified. The pass@1024 plateau in Fig. 10b could be caused by the deliberate curriculum switch to new training data described in §4.2 and Fig. 2, and by the response-window curriculum (4k to 64k, doubling every 800 steps), rather than by an intrinsic exhaustion of the model's reasoning boundary. No control run is reported that keeps data and window size fixed while measuring pass@1024 and pass@1. Because this two-phase interpretation is one of the three headline findings, it needs either an appropriate control or a carefully qualified discussion of the confound.
  3. [§6.4; §3.1.4] 'Context anxiety' is presented as a spontaneously learned awareness of the game's rules, but the rules themselves are hand-designed: the tier-based system prompts and the truncation windows in §3.1.4 define the token budget, and the format reward in Eq. (10) defines the zero-credit condition for malformed outputs. The exhibited behavior ('I will proceed to make an educated guess' when approaching the limit) is exactly what this reward structure incentivizes. The paper provides no evidence that this behavior would emerge without these explicit engineering choices, so it should not be listed among scale-driven 'spontaneous' capabilities.
  4. [§5.6; §5.2] The paper contains an internal tension about the reasoning boundary. Section 5.6 states that 'RL can sharpen and optimize within this boundary, but it cannot fundamentally transcend it' and that the pretrained model's knowledge is a 'rigid upper bound,' while §5.2 claims RL 'expands the model's reasoning boundary' during the discovery phase. These statements need to be reconciled with clear definitions of 'boundary' and 'discovery'; as written, the two-phase discovery claim and the 'bitter lesson' framing are mutually undermining.
minor comments (6)
  1. [§3.1.1; Eq. (3)] The text says 'We do not apply a lower bound on the IS weight,' but Eq. (3) includes an epsilon_low value in the clip. Please state explicitly that epsilon_low is effectively 0 or is not enforced.
  2. [§4.1] The text mentions a 'K3 divergence' penalty; this appears to be a typo for KL divergence. Please correct and ensure the notation matches Eq. (4).
  3. [§4.1] Inference uses 'top-k=0.95'; this is unusual because top-k is normally an integer. Please clarify whether this is top-p (nucleus) sampling or a different scheme.
  4. [§3.1.2; §1] The self-distillation phase is an SFT stage on the model's own rollouts. This is not 'zero RL' in the strict sense of pure RL. The paper should define the scope of 'zero' and avoid implying that the pipeline contains no supervised learning at all.
  5. [§5.2; §5.6] The distinction between 'discovery' and 'sharpening' should be defined more carefully, especially in light of the acknowledged pretrained-prior limitation. Consider using a consistent terminology throughout.
  6. [General] No code, training data, or model weights are released or announced. For a paper whose stated goal is to provide a 'reproducible path,' a data/code availability statement should be added.

Circularity Check

3 steps flagged

The 'spontaneous emergence' claim reduces to the paper's own format reward, and the discovery/sharpening and context-anxiety findings are confounded by the paper's curriculum and truncation design.

specific steps
  1. fitted input called prediction [Section 6 (Bitter Lesson); Eq. 10; Section 4.4.3]
    "Under pure zero RL, the 1T model spontaneously discovers these advanced cognitive strategies from scratch, without any explicit supervision or auxiliary rewards. (Sec. 6) Throughout training, the reward consists of an accuracy component and a format component: ri = racc,i + rformat,i. (Eq. 10) The format reward rformat,i forces the model to comply with structural rules ... We adopt Format B across all training stages. (Sec. 4.4.3)"

    The paper's headline 'spontaneous emergence' of structured formatting is the very behavior that r_format explicitly rewards. Eq. 10 includes the format reward in every training stage, and §4.4.3 states that r_format 'forces' structural compliance and that Format B is adopted 'across all training stages.' Thus the model is directly optimized to produce the tags and termination that Section 6 presents as spontaneously discovered from scratch. No 1T run with r_format removed is reported, so the 'without any explicit supervision' claim is not a prediction from scaling alone but a restatement of the reward design.

  2. fitted input called prediction [Section 6.4; Section 3.1.4; Section 4.1]
    "As the model approaches what it perceives as its maximum token limit, it experiences a strategic panic ... demonstrating a deep, learned awareness of the game's rules: failing to format correctly yields zero reward, but guessing at least offers a non-zero probability of success. (Sec. 6.4) The rollout policy π generates responses conditioned on both the system prompt and the question, truncating the output at the given window size. (Sec. 3.1.4)"

    'Context anxiety' is presented as an emergent cognitive strategy, but the 'maximum token limit' and 'game's rules' are the paper's own hand-designed truncation windows and tier-based system prompts (§3.1.4, §4.1). The behavior is a direct consequence of the hard truncation and the format reward, not a spontaneously learned awareness arising from scale. The paper reports no condition without these hand-designed constraints, so the emergent interpretation is effectively an artifact of the authors' own input design.

  3. other [Section 4.2; Section 5.2; Figure 2]
    "By dividing the training into two distinct phases with progressively difficult data (i.e., starting with relatively simpler data followed by harder data to push the upper bounds of reasoning), we ensure a stable and continuous optimization process. (Sec. 4.2) our results reveal that zero RL is actually a distinct two-phase phenomenon. (Sec. 5.2)"

    The claimed 'discovery vs. sharpening' two-phase phenomenon is read off from training curves, but the paper's own protocol deliberately splits training into two phases with a data switch at roughly step 2800 (Figure 2). The observed pass@1024 plateau and the shift to pass@1 sharpening are therefore entangled with the authors' curriculum switch rather than being an unforced, emergent property of RL at scale. The paper presents the two-phase outcome as a discovery about RL dynamics without controlling for its own two-phase data schedule, so the 'phenomenon' is at least partly a reflection of the input design.

full rationale

The raw performance and sample-efficiency claims are not circular: the AIME/HMMT/IMO benchmark numbers are external and independent of the training objective, so the scaling comparisons rest on real evidence. However, the paper's strongest interpretive claims—the 'bitter lesson' narrative of spontaneous emergence—are partially circular. Structured formatting is literally the quantity optimized by r_format in Eq. 10, and the paper itself says this reward 'forces' structural compliance across all training stages, making Section 6's 'without any explicit supervision' a fitted input called a spontaneous prediction. Context anxiety is also a direct byproduct of the authors' truncation windows and tier prompts, and the 'discovery vs. sharpening' conclusion is confounded by the paper's own two-phase curriculum. These issues undercut the central emergence interpretation but do not invalidate the benchmark results, so the score is 6 rather than higher.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical or architectural entities are introduced; the named 'emergent behaviors' (context anxiety, anthropomorphism, etc.) are interpretive labels for observed text patterns, not new entities with independent falsifiable handles.

free parameters (4)
  • Importance ratio clip upper bound ε_high = 5.0
    Hand-set in Section 4.1; the paper reports removing the lower bound and only clipping the upper bound to keep updates stable; the central stability claim depends on this choice.
  • KL penalty coefficient β = 1e-4
    Hand-set in Section 4.1; used only in first-stage RL; ablation in Section 4.4.2 shows training collapses without it, so the stability recipe depends on this value.
  • Response-window curriculum schedule (incl. tier windows) = 4k→64k doubling every 800 steps; tiers at 4k/16k/64k
    Hand-designed schedule (Section 4.1, Section 3.1.4); the 'context anxiety' and length-inertia findings are direct consequences of these truncation windows.
  • Self-distillation SFT learning rate / steps = 7e-5, 3 epochs, seq len 64k
    Chosen by hand for the compression phase (Section 4.1); the reproducibility of the pipeline depends on it.
axioms (4)
  • domain assumption Verifiable reward assumption: deterministic rule-based matching and an LLM judge (Qwen3-Next-80B-A3B-Instruct) provide correct binary rewards for math answers.
    Adopted in Section 4.1 reward design; the entire RL training signal depends on reward correctness, and LLM judges are not verifiable.
  • domain assumption pass@1024 on AIME 2024 is a valid proxy for the model's reasoning boundary.
    Used in Section 5.2 to distinguish 'discovery' from 'sharpening' phases; no error bars and no other benchmarks support the boundary claim.
  • ad hoc to paper The 'emergent' behaviors would also appear without the format reward, truncation windows, and tiered prompts.
    The bitter-lesson claim in Section 6 requires this counterfactual, but Section 4.4.3 reports the format reward is used across all stages, so the assumption is untested and arguably contradicted.
  • domain assumption LLM-as-a-judge pairwise comparisons without human agreement measure comprehensibility.
    Section 2.1/4.3.1; the judge may share training-data biases with the evaluated model (Qwen-based judge), and no human validation is reported.

pith-pipeline@v1.3.0-alltime-deepseek · 29117 in / 16398 out tokens · 158230 ms · 2026-08-02T06:31:34.612628+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards without human-annotated data, often referred to as zero RL, has emerged as a powerful paradigm for eliciting chain-of-thought reasoning. However, due to computational constraints, existing studies are largely restricted to small models, leaving the training dynamics and emergent capabilities at a large scale unexplored. To meaningfully explore this frontier, we aim to elicit high-quality reasoning behaviors from the model. However, we find that naive scaling often suffers from poor readability, token redundancy, and a lack of adaptive reasoning depth. To address these challenges, we present a stable and efficient training pipeline, incorporating algorithmic and system optimizations such as clipped importance sampling, training-inference ratio correction, and mixed-precision control. Our experiments offer three key findings that validate the "bitter lesson" of scaling: (1) scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; (2) the training process progresses sequentially through an initial discovery phase followed by a sharpening phase; and (3) the model spontaneously develops advanced cognitive behaviors, including anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety, rendering hand-crafted heuristics redundant. Evaluated on seven mathematical benchmarks, Ring-2.5-1T-Zero achieves competitive performance. Additionally, to assess CoT quality beyond final-answer correctness, we propose a structured evaluation framework across three dimensions: comprehensibility, reproducibility, and efficiency, where our model demonstrates clear advantages in producing structured and concise reasoning traces. By sharing our observed emergent phenomena, we hope to provide the community with deeper insights into scaling behaviors, particularly at the 1-trillion scale.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

83 extracted references · 22 linked inside Pith

  1. [1]

    Chi and Quoc V

    Jason Wei and Xuezhi Wang and Dale Schuurmans and Maarten Bosma and Brian Ichter and Fei Xia and Ed H. Chi and Quoc V. Le and Denny Zhou , title =. NeurIPS , year =

  2. [2]

    NeurIPS , year =

    Takeshi Kojima and Shixiang Shane Gu and Machel Reid and Yutaka Matsuo and Yusuke Iwasawa , title =. NeurIPS , year =

  3. [3]

    Hunter Lightman and Vineet Kosaraju and Yuri Burda and Harrison Edwards and Bowen Baker and Teddy Lee and Jan Leike and John Schulman and Ilya Sutskever and Karl Cobbe , title =

  4. [4]

    A Survey of Large Language Models , journal =

    Wayne Xin Zhao and Kun Zhou and Junyi Li and Tianyi Tang and Zican Dong and Yupeng Hou and Beichen Zhang and Yingqian Min and Junjie Zhang and Peiyu Liu and Xiaolei Wang and Yifan Du and Chen Yang and Yushuo Chen and Zhipeng Chen and Jinhao Jiang and Ruiyang Ren and Yifan Li and Xinyu Tang and Zikang Liu and Yiwen Hu and Jian. A Survey of Large Language M...

  5. [5]

    Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert. Language Models are Few-Shot Learners , booktitle =

  6. [6]

    Brown and Benjamin Chess and Rewon Child and Scott Gray and Alec Radford and Jeffrey Wu and Dario Amodei , title =

    Jared Kaplan and Sam McCandlish and Tom Henighan and Tom B. Brown and Benjamin Chess and Rewon Child and Scott Gray and Alec Radford and Jeffrey Wu and Dario Amodei , title =. CoRR , volume =

  7. [7]

    Daya Guo and Dejian Yang and Haowei Zhang and Junxiao Song and Peiyi Wang and Qihao Zhu and Runxin Xu and Ruoyu Zhang and Shirong Ma and Xiao Bi and Xiaokang Zhang and Xingkai Yu and Yu Wu and Z. F. Wu and Zhibin Gou and Zhihong Shao and Zhuoshu Li and Ziyi Gao and Aixin Liu and Bing Xue and Bingxuan Wang and Bochao Wu and Bei Feng and Chengda Lu and Chen...

  8. [8]

    CoRR , volume =

    Konstantin Dobler and Simon Lehnerer and Federico Scozzafava and Jonathan Janke and Mohamed Ali , title =. CoRR , volume =

  9. [9]

    CoRR , volume =

    Hsiu. CoRR , volume =

  10. [10]

    CoRR , volume =

    Weihao Zeng and Yuzhen Huang and Qian Liu and Wei Liu and Keqing He and Zejun Ma and Junxian He , title =. CoRR , volume =

  11. [11]

    CoRR , volume =

    Qiying Yu and Zheng Zhang and Ruofei Zhu and Yufeng Yuan and Xiaochen Zuo and Yu Yue and Tiantian Fan and Gaohong Liu and Lingjun Liu and Xin Liu and Haibin Lin and Zhiqi Lin and Bole Ma and Guangming Sheng and Yuxuan Tong and Chi Zhang and Mofan Zhang and Wang Zhang and Hang Zhu and Jinhua Zhu and Jiaze Chen and Jiangjie Chen and Chengyi Wang and Hongli ...

  12. [12]

    CoRR , volume =

    MiniMax , title =. CoRR , volume =

  13. [13]

    Group Sequence Policy Optimization , journal =

    Chujie Zheng and Shixuan Liu and Mingze Li and Xiong. Group Sequence Policy Optimization , journal =

  14. [14]

    Your Efficient RL Framework Secretly Brings You Off-Policy RL Training , url =

    Yao, Feng and Liu, Liyuan and Zhang, Dinghuai and Dong, Chengyu and Shang, Jingbo and Gao, Jianfeng , journal =. Your Efficient RL Framework Secretly Brings You Off-Policy RL Training , url =

  15. [15]

    The Fourteenth International Conference on Learning Representations , year=

    Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward , author=. The Fourteenth International Conference on Learning Representations , year=

  16. [16]

    CoRR , volume =

    Zhipeng Chen and Xiaobo Qin and Youbin Wu and Yue Ling and Qinghao Ye and Wayne Xin Zhao and Guang Shi , title =. CoRR , volume =

  17. [17]

    CoRR , volume =

    Xinyu Tang and Yuliang Zhan and Zhixun Li and Wayne Xin Zhao and Zhenduo Zhang and Zujie Wen and Zhiqiang Zhang and Jun Zhou , title =. CoRR , volume =

  18. [18]

    CoRR , volume =

    Jian Hu and Xibin Wu and Weixun Wang and Xianyu and Dehao Zhang and Yu Cao , title =. CoRR , volume =

  19. [19]

    2025 , howpublished =

    Zilin Zhu and Chengxing Xie and Xin Lv and slime Contributors , title =. 2025 , howpublished =

    2025

  20. [20]

    EuroSys , pages =

    Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu , title =. EuroSys , pages =

  21. [21]

    CoRR , volume =

    Yang Yue and Zhiqi Chen and Rui Lu and Andrew Zhao and Zhaokai Wang and Yang Yue and Shiji Song and Gao Huang , title =. CoRR , volume =

  22. [22]

    CoRR , volume =

    Mingjie Liu and Shizhe Diao and Ximing Lu and Jian Hu and Xin Dong and Yejin Choi and Jan Kautz and Yi Dong , title =. CoRR , volume =

  23. [23]

    CoRR , volume =

    Kimi , title =. CoRR , volume =

  24. [24]

    Qwen3.5: Accelerating Productivity with Native Multimodal Agents , url =

    Qwen , month =. Qwen3.5: Accelerating Productivity with Native Multimodal Agents , url =

  25. [25]

    CoRR , volume =

    GLM , title =. CoRR , volume =

  26. [26]

    CoRR , volume =

    Hao Liu and Matei Zaharia and Pieter Abbeel , title =. CoRR , volume =

  27. [27]

    DeepSeek-V2:

    DeepSeek. DeepSeek-V2:. CoRR , volume =

  28. [28]

    CoRR , volume =

    Zhen Qin and Weigao Sun and Dong Li and Xuyang Shen and Weixuan Sun and Yiran Zhong , title =. CoRR , volume =

  29. [29]

    CoRR , volume =

    Ling Team and Inclusion AI , title =. CoRR , volume =

  30. [30]

    CoRR , volume =

    Wei Fu and Jiaxuan Gao and Xujie Shen and Chen Zhu and Zhiyu Mei and Chuyi He and Shusheng Xu and Guo Wei and Jun Mei and Jiashu Wang and Tongkai Yang and Binhang Yuan and Yi Wu , title =. CoRR , volume =

  31. [31]

    CoRR , volume =

    Ziqi Wang and Boye Niu and Zipeng Gao and Zhi Zheng and Tong Xu and Linghui Meng and Zhongli Li and Jing Liu and Yilong Chen and Chen Zhu and Hua Wu and Haifeng Wang and Enhong Chen , title =. CoRR , volume =

  32. [32]

    CoRR , volume =

    Tong Zheng and Hongming Zhang and Wenhao Yu and Xiaoyang Wang and Runpeng Dai and Rui Liu and Huiwen Bao and Chengsong Huang and Heng Huang and Dong Yu , title =. CoRR , volume =

  33. [33]

    Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo , title =. CoRR , volume =

  34. [34]

    Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model , booktitle =

    Jingcheng Hu and Yinmin Zhang and Qi Han and Daxin Jiang and Xiangyu Zhang and Heung. Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model , booktitle =

  35. [35]

    CoRR , volume =

    Yang Chen and Zhuolin Yang and Zihan Liu and Chankyu Lee and Peng Xu and Mohammad Shoeybi and Bryan Catanzaro and Wei Ping , title =. CoRR , volume =

  36. [36]

    CoRR , volume =

    Yu Yue and Yufeng Yuan and Qiying Yu and Xiaochen Zuo and Ruofei Zhu and Wenyuan Xu and Jiaze Chen and Cheng. CoRR , volume =

  37. [37]

    CoRR , volume =

    Mohammad Shoeybi and Mostofa Patwary and Raul Puri and Patrick LeGresley and Jared Casper and Bryan Catanzaro , title =. CoRR , volume =

  38. [38]

    Deepak Narayanan and Mohammad Shoeybi and Jared Casper and Patrick LeGresley and Mostofa Patwary and Vijay Korthikanti and Dmitri Vainbrand and Prethvi Kashinkunti and Julie Bernauer and Bryan Catanzaro and Amar Phanishayee and Matei Zaharia , title =

  39. [39]

    Samyam Rajbhandari and Jeff Rasley and Olatunji Ruwase and Yuxiong He , title =

  40. [40]

    Ziheng Jiang and Haibin Lin and Yinmin Zhong and Qi Huang and Yangrui Chen and Zhi Zhang and Yanghua Peng and Xiang Li and Cong Xie and Shibiao Nong and Yulu Jia and Sun He and Hongmin Chen and Zhihao Bai and Qi Hou and Shipeng Yan and Ding Zhou and Yiyao Sheng and Zhuo Jiang and Haohan Xu and Haoran Wei and Zhang Zhang and Pengfei Nie and Leqi Zou and Si...

  41. [41]

    Gonzalez and Clark W

    Lianmin Zheng and Liangsheng Yin and Zhiqiang Xie and Chuyue Sun and Jeff Huang and Cody Hao Yu and Shiyi Cao and Christos Kozyrakis and Ion Stoica and Joseph E. Gonzalez and Clark W. Barrett and Ying Sheng , title =. NeurIPS , year =

  42. [42]

    CoRR , volume =

    Zichen Liu and Changyu Chen and Wenjun Li and Penghui Qi and Tianyu Pang and Chao Du and Wee Sun Lee and Min Lin , title =. CoRR , volume =

  43. [43]

    NeurIPS , year =

    Xiaoxue Cheng and Junyi Li and Zhenduo Zhang and Xinyu Tang and Xin Zhao and Xinyu Kong and Zhiqiang Zhang , title =. NeurIPS , year =

  44. [44]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    2020

  45. [45]

    Acereason-nemotron: Advancing math and code reasoning through reinforcement learning

    Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotron: Advancing math and code reasoning through reinforcement learning. CoRR, abs/2505.16400, 2025 a

    Pith/arXiv arXiv 2025

  46. [46]

    Pass@k training for adaptively balancing exploration and exploitation of large reasoning models

    Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@k training for adaptively balancing exploration and exploitation of large reasoning models. CoRR, abs/2508.10751, 2025 b

    Pith/arXiv arXiv 2025

  47. [47]

    Incentivizing dual process thinking for efficient large language model reasoning

    Xiaoxue Cheng, Junyi Li, Zhenduo Zhang, Xinyu Tang, Xin Zhao, Xinyu Kong, and Zhiqiang Zhang. Incentivizing dual process thinking for efficient large language model reasoning. In NeurIPS, 2025

    2025

  48. [48]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    DeepSeek - AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, abs/2405.04434, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    macereason-math: A dataset of high-quality multilingual math problems ready for RLVR

    Konstantin Dobler, Simon Lehnerer, Federico Scozzafava, Jonathan Janke, and Mohamed Ali. macereason-math: A dataset of high-quality multilingual math problems ready for RLVR . CoRR, abs/2603.10767, 2026

    arXiv 2026

  50. [50]

    Areal: A large-scale asynchronous reinforcement learning system for language reasoning

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. CoRR, abs/2505.24298, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    GLM-5: from vibe coding to agentic engineering

    GLM. GLM-5: from vibe coding to agentic engineering. CoRR, abs/2602.15763, 2026

    Pith/arXiv arXiv 2026

  52. [52]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, ...

    2025

  53. [53]

    Openrlhf: An easy-to-use, scalable and high-performance RLHF framework

    Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance RLHF framework. CoRR, abs/2405.11143, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung - Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. In NeurIPS, 2025

    2025

  55. [55]

    RLVR datasets and where to find them: Tracing data lineage for better training data

    Hsiu - Yuan Huang, Weijie Liu, Chenming Tang, Sanwoo Lee, Kai Yang, Yangkun Chen, Saiyong Yang, and Yunfang Wu. RLVR datasets and where to find them: Tracing data lineage for better training data. CoRR, abs/2605.26971, 2026

    Pith/arXiv arXiv 2026

  56. [56]

    Megascale: Scaling large language model training to more than 10, 000 gpus

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin J...

    2024

  57. [57]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020

    Pith/arXiv arXiv 2001

  58. [58]

    Kimi K2.5: visual agentic intelligence

    Kimi. Kimi K2.5: visual agentic intelligence. CoRR, abs/2602.02276, 2026

    Pith/arXiv arXiv 2026

  59. [59]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022

    2022

  60. [60]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In ICLR . OpenReview.net, 2024

    2024

  61. [61]

    Ring attention with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. CoRR, abs/2310.01889, 2023

    Pith/arXiv arXiv 2023

  62. [62]

    Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models

    Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. CoRR, abs/2505.24864, 2025 a

    Pith/arXiv arXiv 2025

  63. [63]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. CoRR, abs/2503.20783, 2025 b

    Pith/arXiv arXiv 2025

  64. [64]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention

    MiniMax. Minimax-m1: Scaling test-time compute efficiently with lightning attention. CoRR, abs/2506.13585, 2025

    Pith/arXiv arXiv 2025

  65. [65]

    Efficient large-scale language model training on GPU clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on GPU clusters using megatron-lm. In SC , page 58. ACM , 2021

    2021

  66. [66]

    Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models

    Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models. CoRR, abs/2401.04658, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    Zero: memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In SC , page 20. IEEE/ACM , 2020

    2020

  68. [68]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024

    Pith/arXiv arXiv 2024

  69. [69]

    Hybridflow: A flexible and efficient RLHF framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. In EuroSys, pages 1279--1297. ACM , 2025

    2025

  70. [70]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019

    Pith/arXiv arXiv 1909

  71. [71]

    Rethinking sample polarity in reinforcement learning with verifiable rewards

    Xinyu Tang, Yuliang Zhan, Zhixun Li, Wayne Xin Zhao, Zhenduo Zhang, Zujie Wen, Zhiqiang Zhang, and Jun Zhou. Rethinking sample polarity in reinforcement learning with verifiable rewards. CoRR, abs/2512.21625, 2025

    arXiv 2025

  72. [72]

    Every step evolves: Scaling reinforcement learning for trillion-scale thinking model

    Ling Team and Inclusion AI. Every step evolves: Scaling reinforcement learning for trillion-scale thinking model. CoRR, abs/2510.18855, 2025

    arXiv 2025

  73. [73]

    A survey on parallel reasoning

    Ziqi Wang, Boye Niu, Zipeng Gao, Zhi Zheng, Tong Xu, Linghui Meng, Zhongli Li, Jing Liu, Yilong Chen, Chen Zhu, Hua Wu, Haifeng Wang, and Enhong Chen. A survey on parallel reasoning. CoRR, abs/2510.12164, 2025

    arXiv 2025

  74. [74]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022

    2022

  75. [75]

    Your efficient rl framework secretly brings you off-policy rl training, August 2025

    Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, August 2025. https://fengyao.notion.site/off-policy-rl

    2025

  76. [76]

    DAPO: an open-source LLM reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

    Pith/arXiv arXiv 2025

  77. [77]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? CoRR, abs/2504.13837, 2025 a

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? CoRR, abs/2504.13837, 2025 a

    Pith/arXiv arXiv 2025

  78. [78]

    VAPO: efficient and reliable reinforcement learning for advanced reasoning tasks

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Cheng - Xiang Wang, Tiantian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan. VAPO: efficient and reliab...

    Pith/arXiv arXiv 2025

  79. [80]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. CoRR, abs/2503.18892, 2025 b

    Pith/arXiv arXiv 2025

  80. [81]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Zican Dong, Yupeng Hou, Beichen Zhang, Yingqian Min, Junjie Zhang, Peiyu Liu, Xiaolei Wang, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Yiwen Hu, Jian - Yun Nie, and Ji - Rong Wen. A survey of large language models. Frontiers Comput. Sci., 20 ...

    2026

Showing first 80 references.