Pith. sign in

REVIEW 4 major objections 6 minor 64 references

Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Offline preference training alone lifts LLM reasoning by 3.3 points on average.

desk verdict Plausible direction, shaky evidence: the single-run results with alpha tuned in preliminary experiments do not support the headline 3.3%/10.1% claims yet. read the letter →

arxiv 2505.02142 v1 pith:LIBEC3GO submitted 2025-05-04 cs.CL

Xiaoyu Tian , Sitong Zhao , Haotian Wang , Shuaiting Chen , Yiping Peng , Yunjie Ji , Han Zhao , Xiangang Li This is my paper · ORCID
classification cs.CL
keywords offlinereinforcementlearningdirectpreferenceoptimizationLD-DPOlengthbiasLLMreasoningArena-Hardinstructionfollowing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that cheap offline preference optimization—Direct Preference Optimization and its length-desensitized variant LD-DPO—can improve long-context reasoning in a strong 32-billion-parameter language model without the costly rollout loops of online reinforcement learning. On five reasoning benchmarks, a single epoch of LD-DPO training on verified preference pairs raised the model's average score from 71.6% to 74.9%, a gain of 3.3 percentage points, with the largest jump on Arena-Hard (78.9% to 89.0%, +10.1 points). The authors attribute the gains to LD-DPO's handling of output length: plain DPO biases models toward longer, often redundant answers and can hurt precision-oriented tasks, while the length-desensitized variant keeps outputs concise and meaningful. The paper also supplies a detailed recipe for building the preference data, including a verification step that rejects failed responses, and argues this recipe makes offline RL a practical alternative to online RL for reasoning.

What carries the argument

The load-bearing object is the length-desensitized likelihood of LD-DPO, defined as $\hat{\pi}_\theta(y|x) = \prod_{i=1}^{l_p} p(y_i|x,y_{<i}) \prod_{i=l_p+1}^{l} p^{\alpha}(y_i|x,y_{<i})$, where $l_p$ is the common length of the shorter response in a preference pair and $\alpha \in [0,1]$ controls how much the model cares about tokens beyond it. Plugging this likelihood into the DPO implicit-reward loss removes the incentive to pad responses, which is the paper's central explanation for stable gains. The second piece of machinery is the data pipeline: queries sampled from math, code, science, instruction-following, and general reasoning categories, with chosen and rejected answers drawn from a stronger teacher's distilled outputs, rejected answers required to fail verification, and a further cleaning step that corrects ground truth when the teacher's mode disagrees with it.

What would settle it

Rerun the exact one-epoch LD-DPO recipe from the same base model with several random seeds and neighboring $\alpha$ values, and evaluate Arena-Hard with an independent judge. If the 89.0% result does not consistently clear the 78.9% baseline, or if a different judge shrinks the gap, the paper's headline gain is not reproducible.

Watch

Extended reading notes

Core claim

The core claim is that a one-epoch, offline preference-optimization pass can materially improve an already-strong reasoning model across math, code, instruction following, and general reasoning, averaging +3.3 percentage points and reaching +10.1 points on Arena-Hard. The mechanism the authors identify is length desensitization: DPO's implicit reward favors longer outputs, and the extra tokens are often redundant, which explains why their plain-DPO run lost 13.1 points on IFEval (72.8 to 59.7) even as output length rose by about 25%. LD-DPO reparameterizes the policy likelihood $\hat{\pi}_\theta(y|x)$ so that tokens beyond the shorter response's length are discounted by an exponent $\alpha \in [0,1]$; with $\alpha = 0.3$, gains appear on four of five benchmarks while generations stay shorter than plain DPO's. The paper's thesis is that offline RL can improve long-context reasoning when response-length growth is aligned with semantic richness rather than pursued blindly.

Load-bearing premise

The entire reported advantage rests on a single training run with $\alpha=0.3$ picked in preliminary experiments; if that run's 10.1-point Arena-Hard jump is evaluation noise rather than a stable method effect, the central claim softens substantially.

Editorial extensions

If this is right

  • A one-epoch offline LD-DPO run on verified preference pairs can lift a strong base model above its SFT baseline on math, code, instruction following, and general reasoning.
  • Plain DPO's length bias can make instruction-following performance drop sharply (IFEval falls from 72.8 to 59.7 in this paper) even while generation length grows, so controlling length is a practical necessity.
  • The hyperparameter $\alpha$ is a direct knob practitioners can use to balance conciseness and reasoning depth, with 0.3 documented as a working default.
  • Because the recipe needs no reward model and no online interaction, it can be reproduced with modest compute, which is the paper's stated advantage for open-source and academic settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: if the Arena-Hard gain is stable, the same offline recipe could be applied iteratively, using the improved model's own verified outputs to build the next round of preference pairs, potentially narrowing the gap to online RL without rollout infrastructure.
  • Our inference: the paper's 'length should track semantic richness' principle yields a testable prediction—under a fixed token budget, LD-DPO should outperform plain DPO by an even wider margin, because constrained decoding removes verbosity as a confound.
  • Our inference: since $\alpha=0.3$ was chosen once from preliminary experiments, per-domain tuning of $\alpha$ is the natural next experiment; the IFEval failure of plain DPO suggests instruction-following may need a different $\alpha$ than code or math.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates whether simple offline RL methods, specifically Direct Preference Optimization (DPO) and its length-desensitized variant LD-DPO, can improve reasoning in large language models. Using DeepDistill-32B as the base model, the authors construct a preference dataset from DeepSeek-R1 distilled outputs and train with DPO and LD-DPO, evaluating on AIME2024, GPQA-Diamond, LiveCodeBench, IFEval, and Arena-Hard. The central claim is an average improvement of 3.3 percentage points, with a 10.1-point gain on Arena-Hard, attributed primarily to LD-DPO. The paper also analyzes generation lengths and argues that increasing response length is only beneficial when accompanied by semantic richness. The analysis is framed as a preliminary study of offline RL for reasoning.

Significance. If the empirical claims are reliable, the paper would provide useful evidence that inexpensive offline preference optimization, particularly LD-DPO, can yield substantial reasoning gains over a strong base model, and it offers a practical datapoint on the length-bias problem in DPO. The paper's strengths include a detailed data-processing description, the use of external public benchmarks, and comparisons against several published models. However, the quantitative headline rests on a single training run per configuration, an ad hoc choice of the LD-DPO hyperparameter alpha, and an uncharacterized LLM-judged benchmark (Arena-Hard). These issues make the current evidence insufficient to support the strength of the claims. The paper is a reasonable pilot study, but the central quantitative conclusions need substantially stronger evaluation methodology before they can be accepted.

major comments (4)
  1. [Section 3.2] The hyperparameter alpha in LD-DPO is reported as 'empirically set to 0.3 based on preliminary experiments.' If those preliminary experiments used the same five evaluation benchmarks, then the reported gains (especially the 10.1-point Arena-Hard improvement) are a selected maximum, not an unbiased estimate of the method's effect. Please state explicitly whether alpha was chosen on a held-out validation split, report results across a range of alpha values (e.g., 0.0, 0.1, 0.3, 0.5, 1.0), and run multiple training seeds to demonstrate that the chosen value is not an artifact of selection.
  2. [Tables 2, 3, and 8] Every benchmark score in the paper comes from a single training run and a single evaluation pass, with no standard errors, confidence intervals, or significance tests. This is a load-bearing issue for the central 3.3% average and 10.1% Arena-Hard claims, because training and evaluation variance cannot be separated from the reported effect. In particular, Arena-Hard is an LLM-judged benchmark, yet the paper does not report the judge model, prompt, temperature, number of judge samples, or any measure of judge variability. Please provide these details and, ideally, multiple evaluation runs or bootstrap confidence intervals.
  3. [Table 8 vs. Tables 2 and 3] There is an internal inconsistency in model labeling: Table 8 labels the LD-DPO results as 'Ours-32B-DPO,' while Table 3 labels the same configuration as 'DeepDistill-32B-lddpo' and uses 'DeepDistill-32B-dpo' for vanilla DPO. This makes it impossible for the reader to track which model produced the 74.9 average. Moreover, Table 3 shows that vanilla DPO's average (70.9) is below the baseline (71.6), so the abstract's claim that 'these simpler Offline RL methods substantially improve model performance' is not supported for DPO itself; the claim effectively rests on LD-DPO alone. Please correct the labeling and qualify the claim accordingly.
  4. [Figure 1 caption and Section 5] The paper repeatedly states that the method yields 'consistent enhancement' or 'consistent improvements' across benchmarks, but Table 2 shows GPQA-Diamond is unchanged (66.3 to 66.3), and Table 3 shows vanilla DPO's IFEval score drops from 72.8 to 59.7. The 'consistent improvement' claim is therefore not accurate for all benchmarks or for both methods. Please temper the wording to reflect the actual pattern, e.g., 'improvements on four of five benchmarks for LD-DPO, with no change on GPQA-Diamond.'
minor comments (6)
  1. [Section 2.1] There are typos in the subsection headings: 'acquring quries' should be 'Acquiring Queries'.
  2. [Tables 2, 3, 4] The column header 'A VG' should be 'AVG' (average). This appears in Tables 2, 3, and 4.
  3. [Section 3.1] The sentence listing benchmarks has missing punctuation: 'GPQA-Diamond[41], covering challenging problems in biology, physics, and chemistry LiveCodeBench[12] (2024-08–2025-01) for code generation tasks' needs a period or semicolon after 'chemistry'.
  4. [Section 3.3.2] Cross-references are inconsistent: 'Table 8 illustrates the performance differences between DPO and LD-DPO' should refer to Table 3, and 'Further analysis of response generation lengths in Table 8' should refer to Table 4. Please correct these references.
  5. [Tables 2 and 8] The base model is called 'DeepDistill-32B' in Tables 2 and 3, but 'DS-Distill-32B' and 'DS-Distill-32BB' appear in Tables 2 and 8. Please unify the naming to avoid confusion.
  6. [Section 3.1] The evaluation protocol is underspecified: please report the number of test samples per benchmark, decoding parameters (temperature, top-p, max tokens), the exact pass@1 implementation, and the version of Arena-Hard used. Without these, the reported scores are not reproducible.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: gains are external benchmark measurements; self-citations and alpha tuning are context, not by-construction reduction.

full rationale

The paper's central quantitative claims are measured scores on external public benchmarks (AIME2024, GPQA-Diamond, LiveCodeBench, IFEval, Arena-Hard) compared against the DeepDistill-32B baseline. The baseline and data-processing recipe come from the authors' prior work [48], so there is a self-citation, but the improvement claim does not depend on any theorem or fitted constant from [48]; the benchmark results are externally measured. The LD-DPO objective and its length-decoupling equation (Eq. 2) are imported from an external reference [26], not invented here or defined in terms of the reported outcome. The hyperparameter alpha=0.3 was "empirically set ... based on preliminary experiments" (Section 3.2); without evidence that those preliminary experiments were the same benchmark evaluations, this is a model-selection or reproducibility concern, not a fitted input being renamed as a prediction. The abstract's phrase "these simpler Offline RL methods" is not supported for vanilla DPO because Table 3 shows DeepDistill-32B-dpo averaging 70.9, below the baseline's 71.6; that is an overclaim or inconsistency, not circularity. No uniqueness theorem, no ansatz smuggled through self-citation, and no renaming of a known result are present. The derivation chain is therefore not circular; the score of 2 reflects only the presence of a minor, non-load-bearing self-citation and a tuned hyperparameter, not a by-construction reduction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The main tunable is alpha; beta is a standard DPO constant. The paper's conclusions rest on domain assumptions about benchmark validity and the reliability of distillation-based preference labels, rather than on new mathematical postulates.

free parameters (2)
  • alpha (LD-DPO length sensitivity) = 0.3
    Hyperparameter controlling sensitivity to tokens beyond the common length in Eq. (2); set empirically based on preliminary experiments (Section 3.2), not a held-out validation.
  • beta (DPO temperature) = 0.1
    Standard DPO regularization coefficient used in both DPO and LD-DPO runs (Section 3.2); chosen by hand, not fitted.
assumptions (3)
  • domain assumption Arena-Hard scores from an undisclosed GPT-4-based judge reliably measure general reasoning.
    Used as the main evidence for the 10.1 point improvement; no judge version, prompts, or sampling details are provided (Section 3.1, Table 2).
  • domain assumption DeepSeek-R1 distilled outputs with verify_score in (0,1) and pass_rate not in {0,1} yield valid and unbiased preference pairs.
    The entire training set is built from these filtering rules (Section 2.1); the paper does not analyze selection bias.
  • domain assumption The five benchmarks are uncontaminated and their evaluation protocols are stable enough for single-run comparison.
    Used to interpret the reported deltas as method effects; no decontamination or multiple-run variance is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study." pith.science (2026). https://pith.science/paper/LIBEC3GO

@misc{pith2026250502142,
  author       = {Pith},
  title        = {Pith review of: Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LIBEC3GO}},
  note         = {Machine review of arXiv:2505.02142}
}
read the original abstract

Despite significant advances in long-context reasoning by large language models (LLMs), primarily through Online Reinforcement Learning (RL) methods, these approaches incur substantial computational costs and complexity. In contrast, simpler and more economical Offline RL methods remain underexplored. To address this gap, we investigate the effectiveness of Offline RL methods, specifically Direct Preference Optimization (DPO) and its length-desensitized variant LD-DPO, in enhancing the reasoning capabilities of LLMs. Extensive experiments across multiple reasoning benchmarks demonstrate that these simpler Offline RL methods substantially improve model performance, achieving an average enhancement of 3.3\%, with a particularly notable increase of 10.1\% on the challenging Arena-Hard benchmark. Furthermore, we analyze DPO's sensitivity to output length, emphasizing that increasing reasoning length should align with semantic richness, as indiscriminate lengthening may adversely affect model performance. We provide comprehensive descriptions of our data processing and training methodologies, offering empirical evidence and practical insights for developing more cost-effective Offline RL approaches.

Figures

Figures reproduced from arXiv: 2505.02142 by the authors.

Figure 1
Figure 1. Benchmark performance of DeepDistill-32B and DeepDistill-32B-lddpo on AIME 2024, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Category distribution of queries during RL training. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of character length in chosen versus rejected responses. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Left: training loss. Right: training rewards accuracy. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 8 canonical work pages

  1. [1]

    Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025

    Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025. URL https://arxiv.org/abs/2502.17387

    arXiv 2025

  2. [2]

    2023 AMC 8 Problems/Problem 10

    Art of Problem Solving. 2023 AMC 8 Problems/Problem 10. https:// artofproblemsolving.com/wiki/index.php/2023_AMC_8_Problems/Problem_10,

    2023

  3. [3]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  4. [4]

    Aime_1983_2024 (revision 6283828), 2025

    Di Zhang. Aime_1983_2024 (revision 6283828), 2025. URL https://huggingface.co/ datasets/di-zhang-fdu/AIME_1983_2024

    work page 2025

  5. [5]

    Kto: Model alignment as prospect theoretic optimization, 2024

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024. URL https://arxiv.org/abs/ 2402.01306

    arXiv 2024

  6. [6]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1

    2025

  7. [7]

    Ipo: Your language model is secretly a preference classifier, 2025

    Shivank Garg, Ayush Singh, Shweta Singh, and Paras Chopra. Ipo: Your language model is secretly a preference classifier, 2025. URL https://arxiv.org/abs/2502.16182

    arXiv 2025

  8. [8]

    Fine flan: Seqio to parquet so you don’t have to

    Bleys Goodson. Fine flan: Seqio to parquet so you don’t have to. https://huggingface. co/datasets/Open-Orca/FLAN, 2023

    2023

Show all 64 references
  1. [9]

    Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical datas...

    2025

  2. [10]

    Logic-701: A benchmark dataset for logical reasoning in english and russian

    hivaze. Logic-701: A benchmark dataset for logical reasoning in english and russian. https: //huggingface.co/datasets/hivaze/LOGIC-701, 2023. Hugging Face Dataset

    2023

  3. [11]

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tie...

    2024 arXiv

  4. [12]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  5. [13]

    Ncert biology 11th dataset

    Parth Kadam. Ncert biology 11th dataset. https://huggingface.co/datasets/ KadamParth/NCERT_Biology_11th, 2023. Accessed: 2024-04-23

    2023

  6. [14]

    Ncert biology 12th dataset

    Parth Kadam. Ncert biology 12th dataset. https://huggingface.co/datasets/ KadamParth/NCERT_Biology_12th, 2023. Accessed: 2024-04-23

    2023

  7. [15]

    Ncert chemistry 11th dataset

    Parth Kadam. Ncert chemistry 11th dataset. https://huggingface.co/datasets/ KadamParth/NCERT_chemistry_11th, 2023. Accessed: 2024-04-23

    2023

  8. [16]

    Ncert chemistry 12th dataset

    Parth Kadam. Ncert chemistry 12th dataset. https://huggingface.co/datasets/ KadamParth/NCERT_chemistry_12th, 2023. Accessed: 2024-04-23

    2023

  9. [17]

    Ncert physics 11th dataset

    Parth Kadam. Ncert physics 11th dataset. https://huggingface.co/datasets/ KadamParth/NCERT_Physics_11th, 2023. Accessed: 2024-04-23

    2023

  10. [18]

    Ncert physics 12th dataset

    Parth Kadam. Ncert physics 12th dataset. https://huggingface.co/datasets/ KadamParth/NCERT_Physics_12th, 2023. Accessed: 2024-04-23. 9

    2023

  11. [19]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Lu...

    2024

  12. [20]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Lu...

    2025 arXiv

  13. [21]

    Numinamath

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. https://huggingface.co/ AI-MO/NuminaMath...

    2024

  14. [22]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024

    2024 arXiv

  15. [23]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The Arena-Hard pipeline, apr 2024. URL https://lmsys.org/blog/2024-04-19-arena-hard/

    2024

  16. [24]

    Openorca: An open dataset of gpt augmented flan reasoning traces

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet V ong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https:// huggingface.co/datasets/Open-Orca/OpenOrca, 2023

    2023

  17. [25]

    Kw-r1: A simple implementation of the grpo algorithm

    Jiaqing Liang, Jinyi Han, Xinyi Wang, Zishang Jiang, Chengyuan Xiong, Boyu Zhu, Jie Shi, Weijia Li, Tingyun Li, and Yanghua Xiao. Kw-r1: A simple implementation of the grpo algorithm. https://github.com/lsdefine/simple_GRPO, 2025

    2025

  18. [26]

    Length desensitization in direct preference optimization, 2024

    Wei Liu, Yang Bai, Chengcheng Han, Rongxiang Weng, Jun Xu, Xuezhi Cao, Jingang Wang, and Xunliang Cai. Length desensitization in direct preference optimization, 2024. URL https://arxiv.org/abs/2409.06411

    2024 arXiv

  19. [27]

    Understanding r1-zero-like training: A critical perspective, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025. URL https: //arxiv.org/abs/2503.20783

    2025 arXiv

  20. [28]

    longface. logiclm. https://huggingface.co/datasets/longface/logicLM, 2025. Ac- cessed: 2025-04-22

    2025

  21. [29]

    Le, Barret Zoph, Jason Wei, and Adam Roberts

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V . Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023

    2023

  22. [30]

    Deepcoder: A fully open-source 14b coder at o3-mini level

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at o3-mini level. https://pretty-radio-b75. notion.s...

    2025

  23. [31]

    Chemistry-qa

    Microsoft. Chemistry-qa. https://github.com/microsoft/chemistry-qa, 2021. [GitHub repository]

    2021

  24. [32]

    s1: Simple test-time scaling, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URL https://arxiv.org/abs/2501.19393

    2025 arXiv

  25. [33]

    Llama-nemotron-post-training-dataset

    NVIDIA. Llama-nemotron-post-training-dataset. https://huggingface.co/datasets/ nvidia/Llama-Nemotron-Post-Training-Dataset , 2025. Version 1.1, released on April 8, 2025. 10

    2025

  26. [34]

    Llama-3_1-nemotron-ultra-253b-v1

    NVIDIA. Llama-3_1-nemotron-ultra-253b-v1. https://huggingface.co/nvidia/ Llama-3_1-Nemotron-Ultra-253B-v1 , 2025. Released on 2025-04-07 under the NVIDIA Open Model License

    2025

  27. [35]

    Infinity instruct

    Beijing Academy of Artificial Intelligence (BAAI). Infinity instruct. arXiv preprint arXiv:2406.XXXX, 2024

    2024

  28. [36]

    Verifiable coding problems (python)

    Open R1. Verifiable coding problems (python). https://huggingface.co/datasets/ open-r1/verifiable-coding-problems-python , 2025. Hugging Face Dataset

    2025

  29. [37]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/ learning-to-reason-with-llms/

    2024

  30. [38]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  31. [39]

    Codeforces cots

    Guilherme Penedo, Anton Lozhkov, Hynek Kydlí ˇcek, Loubna Ben Allal, Edward Beech- ing, Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces cots. https://huggingface.co/datasets/open-r1/ codeforces-cots, 2025

    2025

  32. [40]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728–53741, 2023

    2023

  33. [41]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  34. [42]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Ni...

    2022

  35. [43]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  36. [44]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300

    2024 arXiv

  37. [45]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// qwenlm.github.io/blog/qwen2.5/

    2024

  38. [46]

    Wizardlm evol-instruct 70k dataset

    WizardLM Team. Wizardlm evol-instruct 70k dataset. https://huggingface.co/ datasets/WizardLMTeam/WizardLM_evol_instruct_70k, 2023. Accessed: 2025-04-23

    2023

  39. [47]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL https://huggingface.co/datasets/teknium/OpenHermes-2.5

    2023

  40. [48]

    Deepdistill: Enhancing llm reasoning capabilities via large-scale difficulty- graded data training, 2025

    Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Yunjie Ji, Han Zhao, and Xiangang Li. Deepdistill: Enhancing llm reasoning capabilities via large-scale difficulty- graded data training, 2025. URL https://arxiv.org/abs/2504.17565. 11

    2025 arXiv

  41. [49]

    Smith, Hannaneh Hajishirzi, and Daniel Khashabi

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby ...

    2022 arXiv

  42. [50]

    Smith, Hannaneh Hajishirzi, and Daniel Khashabi

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby ...

    2022

  43. [51]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y . Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V . Le. Finetuned language models are zero-shot learners, 2022

    2022

  44. [52]

    Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding

    Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. 2025. URL https://arxiv. org/abs/2503.02951

    2025 arXiv

  45. [53]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  46. [54]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  47. [55]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wa...

    2025 arXiv

  48. [56]

    Free process rewards without process labels

    Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.01981, 2024

    2024 arXiv

  49. [57]

    Naturalreasoning: Rea- soning in the wild with 2.8m challenging questions, 2025

    Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, and Xian Li. Naturalreasoning: Rea- soning in the wild with 2.8m challenging questions, 2025. URL https://arxiv.org/abs/ 2502.13124

    2025

  50. [58]

    Vapo: 12 Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, X...

    2025 arXiv

  51. [59]

    Acecoder: Acing coder rl via automated test-case synthesis

    Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. ArXiv, abs/2207.01780, 2025

    2025 arXiv

  52. [60]

    Infinitymath: A scalable instruction tuning dataset in programmatic mathematical reasoning, 2024

    Bo-Wen Zhang, Yan Yan, Lin Li, and Guang Liu. Infinitymath: A scalable instruction tuning dataset in programmatic mathematical reasoning, 2024. URL https://arxiv.org/abs/ 2408.07089

    2024 arXiv

  53. [61]

    Beyond iid: Optimizing instruction learning from the perspective of instruction interaction and dependency

    Hanyu Zhao, Li Du, Yiming Ju, Chengwei Wu, and Tengfei Pan. Beyond iid: Optimizing instruction learning from the perspective of instruction interaction and dependency. 2024. URL https://arxiv.org/abs/2409.07045

    2024 arXiv

  54. [62]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911. 13 A Some Examples of RL training data Table 5: A training example...

    2023 arXiv

  55. [64]

    Hey Siri

    Hence, ∠ADB = 60 ◦ .</answer> Rejected <think>In triangle ABC, which is right-angled at A, with ∠B = 30◦, a square is constructed outwardly on hypotenuse BC, and its center is point D. Coordinates are assigned as A = (0, 0), B = ( √ 3, 0), and C = (0, 1), forming a 30-60-90 tr...

  56. [2023]

    Accessed: 2025-04-23

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.