Pith. sign in

REVIEW 4 major objections 5 minor 19 cited by

SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A reinforcement-learning recipe that trains a 32B model on math first and code second, skipping samples the model already solves, beats a published R1-Zero baseline on AIME24 and LiveCodeBench while using roughly one-tenth of the training…

desk verdict A useful empirical recipe for cheaper cross-domain LLM RL, but the headline comparison to DeepSeek-R1-Zero is not yet solid: single-run numbers, no decontamination check, and a gap on AIME24 that is roughly one question. read the letter →

arxiv 2504.14286 v2 pith:3DTCYULK submitted 2025-04-19 cs.LG

Xiaojiang Zhang , Jinghui Wang , Zifei Cheng , Wenhao Zhuang , Zheng Lin , Minglei Zhang , Shaojie Wang , Yinghan Cui
show 9 more authors
Chao Wang Junyi Peng Shimiao Jiang Shiqi Kuang Shouyu Yin Chaohang Wen Haotian Zhang Bin Chen Bing Yu
This is my paper · ORCID
classification cs.LG
keywords reinforcementlearningLLMreasoningGRPOtwo-stagetraininghistoryresamplingmathematicalcodegenerationchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that large-scale RL for reasoning does not need to mix all domains into one training run. By training a 32B base model first on hard mathematical problems and then introducing code, SRPO lets the model develop long chain-of-thought behavior that transfers to programming. An epoch-level history resampling step removes problems the current policy already solves perfectly, keeping reward variance non-zero so gradient updates stay informative. The reported result is 50.0 pass@1 on AIME24 and 41.6 on LiveCodeBench, surpassing the reference R1-Zero baseline of 47.0 and 40.2 with the same base model in about one-tenth of the steps. If the comparison holds, a carefully staged curriculum and data filtering can substitute for much larger RL compute budgets.

What carries the argument

The load-bearing object is SRPO — two-Staged history-Resampling Policy Optimization — built on GRPO (Group Relative Policy Optimization), where each prompt's G rollouts produce rewards normalized within the group into advantages. The two-stage schedule makes math-only training the first stage, so long chain-of-thought behaviors such as rechecking and backtracking develop before code is introduced; the paper reports that mixed training stagnates in response length while staged training grows. History Resampling is an epoch-level filter that drops samples with all-correct rollouts and keeps mixed or all-wrong samples, maintaining nonzero group variance and aligning with curriculum learning. A rule-based reward (format score, math accuracy score, code test-case score) and removal of the KL term encourage exploration.

What would settle it

Run the exact SRPO data pipeline and training recipe, then measure on a freshly released math and code benchmark that cannot have appeared in training data; if the 50.0 AIME24 / 41.6 LiveCodeBench scores fall to or below the published baseline, the reported lead is explained by training-data overlap rather than by two-stage training and history resampling.

Watch

Extended reading notes

Core claim

The central claim is that the two difficulties that block vanilla GRPO training on mixed math and code data—conflicting response-length incentives and vanishing advantages—are both addressable by training design rather than by more compute. SRPO stages the curriculum: Stage 1 trains only on math to elicit long reflective chains, and Stage 2 adds code; the paper reports that this staged order clearly beats math-only, code-only, and naive mixed training. History Resampling filters each epoch's dataset by removing questions whose every rollout is correct, so each sampled group retains positive reward variance and a meaningful advantage signal. With these mechanisms, plus a curated difficulty-tiered dataset and a rule-based reward, the final model reaches 50.0 pass@1 on AIME24 and 41.6 on LiveCodeBench, surpassing the published R1-Zero baseline of 47.0 and 40.2 on the same base model after about one-tenth of the baseline's training steps.

Load-bearing premise

The claim rests on the assumption that the published baseline scores were measured under comparable conditions and that SRPO's training data does not overlap the AIME24 or LiveCodeBench test sets; no decontamination check or variance estimate is reported, so leakage or an evaluation mismatch would erase the efficiency and superiority claims.

Editorial extensions

If this is right

  • A math-first, code-second curriculum order can transfer reasoning skills across domains within a single policy, making training order a direct lever for cross-domain RL.
  • History Resampling turns too-easy data into a recoverable compute saving: removing it keeps gradients informative without dynamically adjusting rollout counts.
  • Models trained this way spontaneously emit verification behavior, such as writing code to check a math answer, suggesting self-correction can be elicited through rewards rather than hand-crafted prompts.
  • The same recipe reaches higher benchmark scores on both math and code with roughly one-tenth of the baseline's training steps, making strong reasoning RL more accessible at smaller compute budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: If decontamination checks confirm no leakage, the implication generalizes beyond math and code: any two domains with different response-length incentives might be staged similarly.
  • Editorial: History Resampling is orthogonal to dynamic rollout-count methods; combining both could further reduce compute while preserving gradient quality.
  • Editorial: The spontaneous code-to-verify-math behavior suggests RL can induce simple tool use from pure outcome rewards, potentially extending to search or calculators without explicit tool-use training.
  • Editorial: The report gives no seed variance or error bars, so the true size of the 2.0–3.0 point lead over the baseline is unknown; multi-seed reruns would reveal whether staging and resampling are robust.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SRPO, a GRPO-based reinforcement learning recipe for training LLM reasoning across mathematics and coding. The method combines two innovations: a two-stage curriculum that trains on math first and then introduces code, and an epoch-level History Resampling mechanism that removes prompts for which all rollouts are correct while retaining hard all-incorrect prompts. Using Qwen2.5-32B-Base as the initial checkpoint, the authors report 50.0 pass@1 on AIME24 and 41.6 pass@1 on LiveCodeBench (2024-08 to 2025-01), compared with 47.0 and 40.2 for DeepSeek-R1-Zero-Qwen-32B, using roughly 1/10 of the training steps. The paper also describes a data curation pipeline, a rule-based reward design, training dynamics, and qualitative observations of self-reflection and code-assisted verification behaviors.

Significance. If the headline results hold, SRPO is a valuable engineering contribution: it demonstrates that a carefully staged curriculum, data filtering, and history resampling can match or exceed an open R1-Zero result on both math and code benchmarks at a fraction of the training steps, and the model weights are publicly released. The qualitative analysis of emergent reflection patterns and code-assisted verification is also useful for the community. However, the empirical claims currently rest on a single run without variance estimates, on a comparison against externally published baseline numbers, and on training data whose provenance and overlap with the evaluation sets are not reported. These gaps directly affect the strength of the stated superiority and efficiency conclusions, making the current evidence promising but not yet conclusive.

major comments (4)
  1. [Section 4.2, Table 2] The central claim that SRPO surpasses DeepSeek-R1-Zero-Qwen-32B rests entirely on externally published numbers (47.0 AIME24, 40.2 LiveCodeBench) compared with a single SRPO run. Since AIME24 contains only 30 problems, the 3.0-point gap corresponds to roughly one question, and no seed variance, confidence interval, or statistical test is reported. Please report results over multiple seeds or provide bootstrap/permutation intervals, and state the exact evaluation harness, decoding settings, and LiveCodeBench version and date range used for both models; otherwise the 'surpasses' wording is not statistically supported.
  2. [Section 3.4 and Section 4.2] The paper never names the HuggingFace datasets used for training and reports no decontamination check. Because the data is collected from public sources and the History Resampling rule in Section 3.3 deliberately retains hard all-incorrect samples over multiple epochs, any overlap between the training pool and AIME24 or LiveCodeBench (2024-08 to 2025-01) would inflate pass@1 through memorization rather than learned reasoning. Please name the source datasets, quantify overlap with the evaluation sets (e.g., exact-match and n-gram-match statistics), and release the training data or a decontaminated version; this is load-bearing for both the superiority and the 1/10-step efficiency claims.
  3. [Section 3.3 and Table 1] The effectiveness of History Resampling on the final benchmark scores is not demonstrated. Table 1 compares Naive Mixed Training with Staged Training only under the 'w/o HR' condition, while the headline scores in Table 2 come from the full SRPO that includes HR. Add an ablation with and without HR under the same staged-training protocol, reporting AIME24, LiveCodeBench, and total training steps, so the contribution of HR to the final result and to the efficiency claim can be quantified.
  4. [Section 4.1] The efficiency claim of 'about 1/10 of the training steps' is not substantiated in comparable units. The paper reports 840 Stage-1 steps followed by an unspecified number of Stage-2 steps, and compares to DeepSeek-R1-Zero's step count without reporting batch size, tokens processed, or hardware for either run. Please report the total number of optimizer steps, tokens processed, and wall-clock time (or a normalized compute measure) for both SRPO and the DeepSeek baseline, and state clearly whether the 1/10 factor refers to optimizer steps or to overall compute.
minor comments (5)
  1. [Figure 1 and Section 4.2] State whether the '32 samples per question' evaluation is applied identically to the DeepSeek baseline and SRPO, and whether the reported DeepSeek numbers use the same decoding temperature and maximum length.
  2. [Section 3.5] The reward components R_format and Penalty_mix are described, but the training section does not give the exact numerical combination formula or any weighting; please provide the complete reward expression used in the experiments.
  3. [Section 4.1] The paper says the KL term is removed from the GRPO loss, but Eq. (1) includes a KL penalty with coefficient beta; clarify whether beta was set to 0 or the term was otherwise disabled, and whether Eq. (2) still refers to the reference policy.
  4. [Figure 16 and Abstract] There are minor presentation issues: 'optimazition' in the Figure 16 caption is a typo, the phrase 'two-Staged' is capitalized inconsistently between the abstract and Section 3.2, and some equations in the preprint render with unusual spacing (e.g., Eq. (3)); a proofreading pass is needed.
  5. [Abstract and Contributions] The wording claims to 'fully reproduce DeepSeek-R1's performance,' but the experimental comparison is only to DeepSeek-R1-Zero-Qwen-32B, not to the full DeepSeek-R1 model that includes cold-start SFT; please adjust the claim to match the actual baseline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SRPO's benchmark results are external evaluations, and its two innovations are training-data selection methods rather than fitted predictions.

full rationale

SRPO is an empirical reinforcement-learning recipe, not a mathematical derivation, so there is no equation-level tautology. The two-stage training paradigm and History Resampling are training-data selection and gradient-signal mechanisms; they filter samples based on the model's own rollout outcomes, but the headline claims are pass@1 scores on the external AIME24 and LiveCodeBench benchmarks. No parameter is fit to those benchmarks, and no result is defined in terms of itself. The comparison to DeepSeek-R1-Zero-Qwen-32B uses externally published numbers and is not justified by self-citation; the authors' own contributions are presented as engineering choices. The absence of a decontamination check is a legitimate validity risk, but it is a data-leakage concern about benchmark integrity, not a circularity in the derivation. Under the hard rule requiring a quoted reduction, no circular step can be exhibited, so the correct finding is no significant circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central empirical results depend on standard assumptions about GRPO and rule-based rewards, plus unverified assumptions about benchmark comparability and test-train separation. The only hand-tuned quantities are training hyperparameters and reward weights; there are no invented entities or fitted theory parameters.

free parameters (7)
  • learning_rate = 1e-6
    Constant learning rate; standard AdamW choice, not derived from theory.
  • group_size (rollouts per prompt) = 32
    Used for GRPO advantage normalization; chosen for variance estimation.
  • minibatch_size = 8192
    Effective batch for on-policy training; a hyperparameter.
  • max_response_length = 10000 tokens
    Responses exceeding this have advantage zeroed; chosen to avoid length bias.
  • reward_weights = format 0.2, mixing penalty -0.1, partial math 0.2
    Hand-chosen reward shaping coefficients.
  • stage1_steps = 840
    Stage 1 math-only training length chosen when reward plateaued; stage 2 length unspecified.
  • kl_weight = 0 (removed)
    KL term removed from GRPO loss to encourage exploration; a design choice that departs from standard GRPO.
assumptions (5)
  • domain assumption GRPO with rule-based rewards is an effective RL method for eliciting LLM reasoning
    Adopted from DeepSeek-R1 and DAPO; the paper does not re-derive this.
  • domain assumption Qwen2.5-32B-Base is a comparable base model to that used by DeepSeek-R1-Zero-Qwen-32B
    The comparison inherits this from the selection of base model; differences in tokenizer/data are assumed negligible.
  • domain assumption Benchmark pass@1 computed from 32 samples is stable and comparable to reported DeepSeek numbers
    Evaluation noise is not quantified; single-run numbers are treated as decisive.
  • domain assumption Training data and evaluation benchmarks are disjoint
    No decontamination is reported; if AIME24 or LiveCodeBench problems appear in training, the claimed generalization is invalid.
  • ad hoc to paper Math-first training builds transferable reasoning that benefits code
    This is the central hypothesis of the two-stage paradigm, supported only by the paper's own ablations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM." pith.science (2026). https://pith.science/paper/3DTCYULK

@misc{pith2026250414286,
  author       = {Pith},
  title        = {Pith review of: SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3DTCYULK}},
  note         = {Machine review of arXiv:2504.14286}
}
read the original abstract

Recent advances of reasoning models, exemplified by OpenAI's o1 and DeepSeek's R1, highlight the significant potential of Reinforcement Learning (RL) to enhance the reasoning capabilities of Large Language Models (LLMs). However, replicating these advancements across diverse domains remains challenging due to limited methodological transparency. In this work, we present two-Staged history-Resampling Policy Optimization (SRPO), which surpasses the performance of DeepSeek-R1-Zero-32B on the AIME24 and LiveCodeBench benchmarks. SRPO achieves this using the same base model as DeepSeek (i.e. Qwen2.5-32B), using only about 1/10 of the training steps required by DeepSeek-R1-Zero-32B, demonstrating superior efficiency. Building upon Group Relative Policy Optimization (GRPO), we introduce two key methodological innovations: (1) a two-stage cross-domain training paradigm designed to balance the development of mathematical reasoning and coding proficiency, and (2) History Resampling (HR), a technique to address ineffective samples. Our comprehensive experiments validate the effectiveness of our approach, offering valuable insights into scaling LLM reasoning capabilities across diverse tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evidence-Grounded Forensic Reasoning for Detecting and Grounding Multi-Modal Media Manipulation

    cs.CV 2026-08 conditional novelty 7.0 of 10

    EFR trains a vision-language model to bind each forensic conclusion to specific image and text locations, improving detection and grounding on DGM4 while making explanations spatially verifiable.

  2. ReCo: Reweighting GRPO Against Distributional Concentration

    cs.LG 2026-07 conditional novelty 6.0 of 10

    ReCo fixes GRPO’s collapse onto high-probability base-model answers by normalizing response weights by expected occurrence and scaling token updates by Bernoulli variance, improving large-k Pass@k on math reasoning.

  3. Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SARA uses a Bayesian stopping rule on a prompt's first few responses to abandon saturated response groups early and reallocate rollouts, matching accuracy with up to 67% fewer rollouts than dynamic sampling.

  4. CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention

    cs.LG 2025-08 conditional novelty 6.0 of 10

    CURE re-generates responses from prefixes ending at high-entropy tokens to maintain exploration, then continues DAPO training to consolidate gains, achieving 54.3% average accuracy on six math benchmarks.

  5. CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learning

    cs.SE 2025-07 conditional novelty 6.0 of 10

    CodeReasoner combines a concise execution-focused dataset, instruction tuning, and GRPO RL to make 7B/14B models match or beat GPT-4o on code reasoning benchmarks.

  6. The Challenge of Teaching Reasoning to LLMs Without RL or Distillation

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Twenty high-quality chain-of-thought examples from a reasoning model are enough to activate strong math reasoning in a 32B base model with lightweight fine-tuning.

  7. AdapThink: Adaptive Thinking Preferences for Reasoning Language Model

    cs.LG 2025-06 conditional novelty 6.0 of 10

    AdapThink is an RL post-training framework that adaptively reduces overthinking and underthinking in reasoning language models by rewarding confidence-appropriate reasoning depth and diverse training samples.

  8. Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts

    cs.AI 2025-06 conditional novelty 6.0 of 10

    An online pre-rollout filter that predicts and skips zero-variance prompts from reward dynamics reduces GRPO rollout time by up to 2.4x with comparable math reasoning accuracy.

  9. rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A large verified dataset of 418K competition code problems built by synthesis, test-input utility functions, and majority-vote output labeling improves small model code reasoning to frontier level.

  10. Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning

    cs.AI 2025-10 conditional novelty 5.0 of 10

    A plan-then-reason SFT plus a plan-quality reward in GRPO improves math-reasoning accuracy by small but consistent margins over GRPO and DAPO.

  11. KAT-V1: Kwai-AutoThink Technical Report

    cs.CL 2025-07 conditional novelty 5.0 of 10

    KAT-V1-40B is a 40B language model that switches between deep reasoning and direct answering per query, reporting matching-or-better benchmark scores with lower token use.

  12. From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization

    cs.LG 2025-07 conditional novelty 5.0 of 10

    A sample-centric RLVR method (LPPO), combining hint prefixes and learning-progress weighting, improves math reasoning pass@1 by 2 to 4 points over a GRPO baseline.

  13. Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A 2.75B-active-parameter MoE model, trained with a token-budget-stabilized RL method (C3PO), matches or exceeds several 7-8B dense reasoning models on AIME, LiveCodeBench, and GPQA benchmarks.

  14. FinLMM-R1: Enhancing Financial Reasoning in LMM through Scalable Data and Reward Design

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A two-stage RL framework with length, image-selection, and adversarial rewards, trained on 89,378 ASP-built financial image-question pairs, improves multimodal reasoning over LMM-R1.

  15. SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization

    cs.AI 2025-05 conditional novelty 5.0 of 10

    SEED-GRPO scales GRPO advantages by normalized semantic entropy so high-uncertainty prompts receive smaller policy updates, and reports average accuracy improvements on five math reasoning benchmarks.

  16. Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math

    cs.CL 2025-04 conditional novelty 5.0 of 10

    A four-stage training recipe, combining large-scale distillation, preference learning, and reinforcement learning, produces Phi-4-Mini-Reasoning, a 3.8B model that outperforms 7-8B open reasoning models on math benchmarks.

  17. First Return, Entropy-Eliciting Explore

    cs.AI 2025-07 conditional novelty 4.0 of 10

    FR3E improves LLM math reasoning by finding high-entropy tokens in a trajectory and doing extra rollouts from those points to shape rewards.

  18. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

  19. 100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models

    cs.CL 2025-05 conditional novelty 3.0 of 10

    A survey of DeepSeek-R1 replication studies covering SFT, RLVR, algorithm variants, reward design, and future directions.

Reference graph

Works this paper leans on

25 extracted references · 5 canonical work pages · cited by 19 Pith papers

  1. [1]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024

    2024

  2. [2]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    arXiv 2024

  3. [3]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  4. [4]

    Gpt4 technical report

    OpenAI. Gpt4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  5. [5]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024

    2024

  6. [6]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  7. [7]

    Grok 3 beta — the age of reasoning agents, 2024

    XAI. Grok 3 beta — the age of reasoning agents, 2024

    2024

  8. [8]

    Gemini 2.0 flash thinking, 2024

    Google DeepMind. Gemini 2.0 flash thinking, 2024

    2024

Show all 25 references
  1. [9]

    Qwq-32b: Embracing the power of reinforcement learning, 2024

    Qwen. Qwq-32b: Embracing the power of reinforcement learning, 2024

    2024

  2. [10]

    Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  3. [11]

    An empirical study on eliciting and improving r1-like reasoning models

    Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al. An empirical study on eliciting and improving r1-like reasoning models. arXiv preprint arXiv:2503.04548, 2025

    2025 arXiv

  4. [12]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  5. [13]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  6. [14]

    Livecodebench: Holistic and con- tamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and con- tamination free evaluation of large language models for code. CoRR, 2024

    2024

  7. [15]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  8. [16]

    Curriculum learning for reinforcement learning domains: A framework and survey

    Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1–50, 2020

    2020

  9. [17]

    Concrete problems in AI safety

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. 14

    2016 arXiv

  10. [18]

    Re- inforcement learning with a corrupted reward channel

    Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Re- inforcement learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417, 2017

    2017 arXiv

  11. [19]

    Specification gaming: the flip side of AI ingenuity

    Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of AI ingenuity. arXiv preprint arXiv:2002.04871, 2020

    2002 arXiv

  12. [20]

    Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective

    Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. arXiv preprint arXiv:2108.08901, 2021

    2021 arXiv

  13. [21]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.01753, 2022

    2022 arXiv

  14. [22]

    Reward hacking in reinforcement learning

    Lillian Weng. Reward hacking in reinforcement learning. lilianweng.github.io, 2023. Accessed: 2025-04-18

    2023

  15. [23]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  16. [24]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023

    2023

  17. [25]

    Demystifying long chain-of-thought reasoning in llms

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373, 2025. 15 A. More Show Cases of the Thinking Behavior Figure 14| Response with self-correction Figure 15| Response with numer...

    2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.