Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper maps how reinforcement learning in math, code, and puzzle domains transfers across domains: puzzle and math reinforce each other, code helps only the instruction-tuned model, and three-domain training maximizes overall accuracy.

desk verdict A broad, carefully documented RLVR mapping study whose cross-domain transfer claims need a format-control run before the headline takeaways are trusted. read the letter →

arxiv 2507.17512 v1 pith:KJZSTJM6 submitted 2025-07-23 cs.AI cs.LG

Yu Li , Zhuoshi Pan , Honglin Lin , Mengyuan Sun , Conghui He , Lijun Wu This is my paper
classification cs.AIcs.LG
keywords multi-domainreasoningreinforcementlearningwithverifiablerewardsGRPOcross-domaingeneralizationmathematicalcodegenerationlogicalpuzzlescurriculum
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether reinforcement-learning training in one reasoning domain helps or hurts other domains, using a 7-billion-parameter model family and the GRPO algorithm. It trains on math, code, and logical-puzzle data separately, then in pairs, then on all three together, and measures in-domain and out-of-domain benchmarks. The paper claims that puzzle and math training reinforce each other; that code training transfers positively for the instruction-tuned model but constrains the base model; and that combining all three domains gives the highest overall accuracy, 56.57, while puzzle accuracy drops to 49.73 from 61.98 with puzzle-only training. It also claims that template consistency, curriculum design, reward-shape choices, and training language all materially change results. If these patterns hold, they give concrete guidance for assembling multi-domain reasoning data and rewards.

What carries the argument

The machinery is a controlled three-domain RLVR pipeline built on the GRPO update, in which each sampled response is scored by a task-specific reward and its advantage is computed relative to the mean and spread of scores in its rollout group. The reward set is load-bearing: binary 0-1 correctness for math, code, and the easier puzzle set, and a proportional cell-fraction reward for the harder puzzle set whose answers are treated as pseudo ground truth. The evaluation matrix organizes every training-data combination against the same math, code, and puzzle benchmarks, and the R1-style template with explicit thinking and answer tags is used in both training and testing, with template mismatch treated as a decisive confound.

What would settle it

Have human experts solve a random sample of the 2,400 pseudo-labeled puzzles and compare agreement; then retrain the puzzle and cross-domain runs on a corrected subset. If the puzzle gains or the puzzle-to-math transfer shrink or disappear once the labels are corrected or generated by a different teacher model, the central interaction claims rest on the annotation rather than on genuine cross-domain transfer.

Watch

Extended reading notes

Core claim

The central discovery is an empirical map of cross-domain transfer and conflict under reinforcement learning with verifiable rewards (RLVR). Logical-puzzle training and math training are mutually supportive: puzzle runs lifted the base model's math scores substantially and math runs raised puzzle scores, while the joint math-plus-puzzle setup beat math-only performance on math. Code reasoning has split effects: the instruction-tuned model generally improved on out-of-domain benchmarks after code training, whereas the base model often declined, a pattern the paper attributes to the rigid structure of code-format outputs constraining flexible answering elsewhere. Combined training across math, code, and puzzle produced the best overall average, 56.57, with better balance across tasks than any dual-domain setup, but every configuration containing an extra domain scored worse on puzzles than puzzle-only training, showing a specialization trade-off. The paper also reports that a supervised fine-tuning stage before RL substantially improves results; that mismatched training and evaluation templates can collapse performance; that curriculum learning with periodic refresh of the reference model and optimizer state raises a puzzle benchmark from 94.29 to 99.71; that reward design must track task sparsity, with binary rewards collapsing on the harder puzzle set; and that English training consistently beats Chinese training under otherwise identical settings.

Load-bearing premise

The load-bearing assumption is that the pseudo ground-truth answers generated for the 2,400 puzzle training items, needed because that dataset has no official answers, are accurate enough to serve as reward labels; if those labels are systematically wrong, the puzzle-domain gains and the puzzle-to-math transfer results built on them are biased.

Editorial extensions

If this is right

  • If the mapping is correct, logical-puzzle data is a viable math booster: the base model's MATH500 score rose from 56.40 to 68.40 and AIME24 from 10.00 to 20.00 after puzzle training.
  • Code training should not be assumed to transfer: the base model's MATH500 fell from 56.40 to 50.80 and its Knights-and-Knaves score from 17.86 to 13.85, while the instruction-tuned model gained on both.
  • Multi-domain data is a balancing strategy: math plus code plus puzzle training reached the best overall average of 56.57, surpassing the best dual-domain setup at 50.89, while accepting lower puzzle scores than puzzle-only training.
  • Template matching is a precondition for meaningful RLVR evaluation: mismatched templates dropped several benchmarks to near zero.
  • Reward design and curriculum can dominate data choice: binary rewards collapsed on the hard puzzle set, while proportional, format, and rescaled rewards succeeded; curriculum with policy refresh raised puzzle accuracy from 94.29 to 99.71.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the puzzle-to-math transfer it observes may come from shared search structure rather than shared content; a targeted test would train on synthetic puzzles with no arithmetic and measure math gains.
  • An untested extension is to use the measured interaction matrix as a curriculum scheduler, for example starting with puzzle data to establish structured search and then injecting math, which the mutual-support finding suggests.
  • Because the hard puzzle set's answers are teacher-generated pseudo-labels, the puzzle-specific numbers in this study carry annotation noise; a human-validated subset would bound that noise and show whether the transfer results survive it.
  • The strong template-sensitivity result implies that some RLVR gains reported elsewhere may be template-fitting rather than reasoning gains, so evaluations should report and match template details.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents a systematic empirical study of multi-domain reasoning under reinforcement learning with verifiable rewards (RLVR). Using GRPO and Qwen2.5-7B base and instruct models, the authors train on single-domain datasets from math (DeepScaleR, CountDown), code (CodeR1), and logical puzzles (Knights-and-Knaves, Logic Puzzle Baron), then evaluate on in-domain and out-of-domain benchmarks. They report that single-domain training improves in-domain performance and induces cross-domain effects, that math and puzzle data mutually support each other, that code training has mixed effects between base and instruct models, and that multi-domain combinations improve overall performance at some cost to specialization. The paper further studies template sensitivity, curriculum learning with a policy-refresh strategy, reward design variants, and training-language effects. The main contribution is a large set of controlled ablation experiments with documented prompts and hyperparameters, plus a data-centric analysis of domain interactions.

Significance. If the findings are taken at face value, this is a useful empirical map of how RLVR behaves when training data spans multiple reasoning domains. The paper's strengths include extensive documentation of templates and hyperparameters, public code release, and a large number of ablation conditions (single-domain, dual/triple combinations, curriculum, reward, language). The observation that template mismatch can swing performance by tens of points is an important negative result for the RLVR community. However, the central claims about cross-domain mutual support and multi-domain benefits currently rest on analyses that do not control for total data volume, output-format learning, or pseudo-label quality, and all quantitative results come from single training runs. These issues are addressable within the scope of the paper, but until they are resolved the more ambitious conclusions should be regarded as provisional.

major comments (4)
  1. [§4, Table 1 and Table 9/12] The comparison across domain combinations in §4 is confounded by total training data volume. Table 1 lists training sizes of 10k each for DSR and CD, 12k for CodeR1, and 5.4k+2.4k=7.8k for Puzzle; consequently, the single-domain baselines and the dual/triple combinations in Table 9 use very different numbers of samples (e.g., Math=20k, Puzzle=7.8k, Math+Puzzle=27.8k, Math+Code=32k, triple=39.8k). The claim that 'expanding domain coverage... achieves the highest overall performance' (§4.2, Figure 4) is therefore not separable from a simple data-quantity effect. The authors should either match the total sample count across configurations (e.g., by subsampling the fused data) or control for data volume in the analysis (e.g., by also training on repeated samples of a single domain with the same token budget). Without such a control, the central multi-domain advantage claim is not uniquely identified.
  2. [§5, Table 11 and §1/§3] The cross-domain transfer claims rest on improvements that may be largely attributable to learning the R1 output format rather than to domain reasoning. The base model starts with near-zero scores on structured-output tasks (KK 17.86, Zebra 0.27, CountDown 1.05 in Table 4), and §5 shows that template choice alone changes scores by tens of points (e.g., the Base-KK checkpoint scores 31.29 on KK with the Base template versus 94.29 with R1; Instruct-KK drops from 73.20 to 1.80 on MATH500 under the Qwen template). Single-domain RL with the R1 template therefore plausibly improves OOD benchmarks by teaching the model to emit parseable <think>/<answer> blocks and follow instructions, independent of any transfer of domain-specific reasoning. The paper does not include a format-control run (e.g., RL on a non-reasoning dataset with the same R1 template) and does not report OOD gains under a mismatched template. Consequently, the takeaway that 'Puzzle and math data provide mutual support' (§1) is not uniquely identified. Please add a control experiment or an analysis that isolates format learning (e.g., evaluating the trained models with multiple templates and showing that cross-domain gains either survive template variation or disappear, which would directly test the format-transfer hypothesis).
  3. [§2.1] The Logic Puzzle Baron pseudo-labels are used as ground truth for RL training without any validation. Section 2.1 states that DeepSeek-R1 annotations are 'treating these annotations as pseudo ground truth answer for our RL training.' There is no mention of human verification, agreement analysis, or filtering of low-confidence annotations. If the pseudo-labels contain systematic errors (which is plausible for hard logic puzzles), the reward signal for LPB is corrupted, and this would bias the reported puzzle improvements (e.g., Zebra scores in Table 8) and the cross-domain conclusions that depend on puzzle training (e.g., puzzle-to-math transfer). The authors should validate a sample of annotations (e.g., against a ground-truth subset constructed by hand, or by measuring self-consistency of an ensemble of annotators) and report the estimated label accuracy. At minimum, they should discuss how label noise could affect the reward and the downstream conclusions.
  4. [§3-§4 (throughout)] All reported results are from single training runs without seed variation or confidence intervals. Because many of the paper's conclusions are based on small differences (e.g., Table 9: Math+Puzzle improves Math Avg by only 2.24 points over Math-only; Figure 6: Standard vs Policy Refresh differ by a few points at several difficulty levels), the reader cannot distinguish systematic effects from training noise. Particularly for the RL setting, which is known to be seed-sensitive, at least 2-3 independent runs for the key comparisons (single-domain, dual/triple combination, curriculum, and reward schemes) are needed, or the authors should provide an explicit justification for why variance is negligible (e.g., using a large evaluation set). Without this, the quantitative claims are not reproducible in a statistically meaningful sense.
minor comments (5)
  1. [§1] In the paragraph following the Overall Takeaways, 'throughful' should be 'thoughtful'.
  2. [§2.2] The benchmark name 'AIME241' appears to be a typo for 'AIME24'.
  3. [Appendix C.6] The few-shot examples for MBPP appear to contain invalid Python syntax as printed, e.g., 'def similar elements(test tup1, test tup2):', 'range{ 2,int(math.sqrt(n)) + 1 }', and 'assert heap queue largest( [25, 35, 22, 85, 14, 65, 75, 22, 58],3} == (85, 75, 65)'. If these strings are used verbatim in the evaluation prompts, the MBPP scores in Tables 4 and 8 could be affected; please verify that the displayed code is correct (likely a LaTeX escaping issue) and, if so, clarify that the actual prompts contain valid Python identifiers.
  4. [Figure 6] The caption is ambiguous: it says the x-axis represents KK difficulty levels and the y-axis shows the training data sequence, but the displayed matrix appears to have difficulty levels on both axes; please clarify the axes and the meaning of the entries (accuracy at a given stage, or averaged performance).
  5. [Figure 11] The bar chart for training-language effects has no axis labels or legend; the text does not explain what the four groups of bars (En/Zh pairs) correspond to. Please add a legend and label the y-axis as accuracy (%).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: an empirical RLVR study whose conclusions are read off evaluation tables rather than derived from fitted inputs.

full rationale

This is a benchmark study, not a derivation. The central claims—puzzle-math mutual support, code's mixed cross-domain effects, multi-domain balance, SFT benefits, template sensitivity, and curriculum/reward/language effects—are empirical findings reported from held-out evaluation results (Tables 4, 6, 8–12 and Figures 1–11). None of the quantities claimed as findings is defined in terms of another claimed output: reward schemes are design choices selected before training, not fitted parameters renamed as predictions; template matching is demonstrated by evaluating the same checkpoint under three templates rather than assumed; curriculum and policy-refresh improvements are measured against standard-curriculum and mixed-training baselines. Self-citations in the reference list (e.g., [16], [23], [24], [26]) support background claims only and carry no load-bearing weight. The potential confounds noted in the paper itself—LPB pseudo-labels from DeepSeek-R1 and the large template/format sensitivity quantified in Section 5—are validity threats to the domain-transfer interpretation, but they are not circular: the reported scores do not reduce to the model's inputs by construction. Therefore no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

There are no fitted numerical parameters in the mathematical sense. The study uses standard hyperparameters (LR, rollout, epochs) and task-specific reward schemes chosen based on observed training behavior, but none of these are fitted to the test benchmarks, and they do not enter the final claims as fitted constants.

assumptions (3)
  • domain assumption Verifiable reward functions correctly determine answer correctness for math and code datasets
    Training relies on exact match or test-case execution; errors in these checkers would propagate to the measured improvements.
  • domain assumption DeepSeek-R1 pseudo-annotations for the LPB dataset are accurate enough to serve as ground truth
    Section 2.1 states LPB lacks ground truth and uses DeepSeek-R1 annotations as pseudo ground truth; no validation of annotation accuracy is reported.
  • domain assumption Single-run evaluation results are representative of model performance
    All reported accuracies are point estimates from a single training run, so the study implicitly assumes that the RL training is deterministic enough or that differences are large enough that variance would not change the conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning." pith.science (2026). https://pith.science/paper/KJZSTJM6

@misc{pith2026250717512,
  author       = {Pith},
  title        = {Pith review of: Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJZSTJM6}},
  note         = {Machine review of arXiv:2507.17512}
}
read the original abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of LLMs. Existing research has predominantly concentrated on isolated reasoning domains such as mathematical problem-solving, coding tasks, or logical reasoning. However, real world reasoning scenarios inherently demand an integrated application of multiple cognitive skills. Despite this, the interplay among these reasoning skills under reinforcement learning remains poorly understood. To bridge this gap, we present a systematic investigation of multi-domain reasoning within the RLVR framework, explicitly focusing on three primary domains: mathematical reasoning, code generation, and logical puzzle solving. We conduct a comprehensive study comprising four key components: (1) Leveraging the GRPO algorithm and the Qwen-2.5-7B model family, our study thoroughly evaluates the models' in-domain improvements and cross-domain generalization capabilities when trained on single-domain datasets. (2) Additionally, we examine the intricate interactions including mutual enhancements and conflicts that emerge during combined cross-domain training. (3) To further understand the influence of SFT on RL, we also analyze and compare performance differences between base and instruct models under identical RL configurations. (4) Furthermore, we delve into critical RL training details, systematically exploring the impacts of curriculum learning strategies, variations in reward design, and language-specific factors. Through extensive experiments, our results offer significant insights into the dynamics governing domain interactions, revealing key factors influencing both specialized and generalizable reasoning performance. These findings provide valuable guidance for optimizing RL methodologies to foster comprehensive, multi-domain reasoning capabilities in LLMs.

Figures

Figures reproduced from arXiv: 2507.17512 by the authors.

Figure 1
Figure 1. Performance on HumanEval. Initial 50 100 150 200 240 Training Steps 62 64 66 68 70 Accuracy (%) Base-CodeR1 Instruct-CodeR1 [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 3
Figure 3. Performance comparison of triple￾domain and optimal dual-domain data. Base Code Math Puzzle M+P M+C P+C M+P+C Domain Combination 30 35 40 45 50 55 60 Overall Accuracy 31.50 35.78 45.11 50.72 48.36 48.92 50.89 56.57 Base Single Domain Dual Domain Triple Domain Trend Line [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 5
Figure 5. The average test performance of base and instruct models on different templates. As shown in [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figures from the paper (18 more)
Figure 6
Figure 6. Figure 6: Model performance on the KK dataset with different curriculum settings. The x-axis [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Performance on KK. 100 200 300 400 500 570 Training Steps 10 15 20 25 30 35 40 ZebraLogicBench Performance (%) Reward 1 Reward 2 Reward 3 Reward 4 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 9
Figure 9. Figure 9: KK-impact of reward configurations (base model shown with dashed lines). [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: LPB-impact of reward configurations (base model shown with dashed lines). [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: The effect of different training language. To ensure the model uses Chinese for reasoning during training, we employ the “langid”4 package to detect the language of each rollout trajectory. A reward of 1 is given only when the language is Chinese and the final answer …
Figure 12
Figure 12. Figure 12: Base model’s detailed performance on DeepscaleR. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Instruct model’s detailed performance on DeepscaleR. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Base model’s detailed performance on CountDown. [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Instruct model’s detailed performance on CountDown. [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Base model’s detailed performance on CodeR1. [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Instruct model’s detailed performance on CodeR1. [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Base model’s detailed performance on KK. [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: Instruct model’s detailed performance on KK. [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: Base model’s detailed performance on LPB. [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]
Figure 21
Figure 21. Figure 21: Instruct model’s detailed performance on LPB. [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]
Figure 22
Figure 22. Figure 22: Base model’s detailed performance on Puzzle + Math domain data. [PITH_FULL_IMAGE:figures/full_fig_p024_22.png]
Figure 23
Figure 23. Figure 23: Base model’s detailed performance on Math + Code domain data. [PITH_FULL_IMAGE:figures/full_fig_p024_23.png]
Figure 24
Figure 24. Figure 24: Base model’s detailed performance on Puzzle + Code domain data. [PITH_FULL_IMAGE:figures/full_fig_p024_24.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning

    cs.LG 2026-08 conditional novelty 7.0 of 10

    CoKL regularizes only the correctness-conditioned response distribution, decoupling total correctness from mode preservation and improving retention-adaptation balance in LLM RL.

  2. Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    TAC is a new online curriculum for multi-domain RLVR that estimates cross-domain transferability from gradient geometry to achieve up to 2.8 point macro-averaged accuracy gains over learnability-only baselines on six-...

  3. Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    TAC is a bandit curriculum for multi-domain RLVR that prioritizes domains whose gradient updates align with and benefit other domains, yielding up to 2.8-point macro accuracy gains over learnability-only baselines on ...

  4. Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs

    cs.AI 2026-04 unverdicted novelty 7.0 of 10

    A multi-agent framework reconstructs the evolutionary graph of post-training LLM datasets, revealing domain patterns like vertical refinement in math data and systemic issues like redundancy and benchmark contaminatio...

  5. From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    The LLM-as-Environment-Engineer framework lets the policy model redesign its own RL environments on the new MAPF-FrozenLake testbed, outperforming larger models and fixed baselines with Qwen3-4B.

  6. Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix It

    cs.CL 2026-06 conditional novelty 6.0 of 10

    CoT SFT disrupts long-range routing in hybrid models via changes to W_Q and W_K; QK-Restore restores pre-SFT projections to recover NIAH performance.

  7. From Pixels to Digital Agents: An Empirical Study on the Taxonomy and Technological Trends of Reinforcement Learning Environments

    cs.AI 2026-03 unverdicted novelty 5.0 of 10

    An empirical literature analysis reveals a bifurcation in RL environments into Semantic Prior (LLM-dominated) and Domain-Specific Generalization ecosystems with distinct cognitive fingerprints.

Reference graph

Works this paper leans on

49 extracted references · 10 canonical work pages · cited by 6 Pith papers

  1. [1]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021

    2021

  2. [2]

    The logic puzzle baron dataset

    Oleg Bask. The logic puzzle baron dataset. https://huggingface.co/datasets/olegbask/LogicPuzzleBaron, 2024

    work page 2024

  3. [3]

    Sft or rl? an early investigation into training r1-like reasoning large vision-language models

    Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1-like reasoning large vision-language models. arXiv preprint arXiv:2504.11468, 2025

    arXiv 2025

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  5. [5]

    Self-evolving curriculum for llm reasoning

    Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Pich´e, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for llm reasoning. arXiv preprint arXiv:2505.14970, 2025

    arXiv 2025

  6. [6]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025

    arXiv 2025

  7. [7]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023

    2023

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 49 references
  1. [9]

    Does prompt formatting have any impact on llm performance?, 2024

    Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. Does prompt formatting have any impact on llm performance?, 2024. URL https://arxiv.org/abs/2411.10541

    2024 arXiv

  2. [10]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  3. [11]

    Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025

    2025 arXiv

  4. [12]

    Curricularface: adaptive curriculum learning loss for deep face recognition

    Yuge Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 590...

    2020

  5. [13]

    Chatbug: A common vulnerability of aligned llms induced by chat templates, 2025

    Fengqing Jiang, Zhangchen Xu, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Chatbug: A common vulnerability of aligned llms induced by chat templates, 2025. URL https://arxiv.org/abs/2406.12935

    2025 arXiv

  6. [14]

    Adaptive curriculum learning

    Yajing Kong, Liu Liu, Jun Wang, and Dacheng Tao. Adaptive curriculum learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5067–5076, 2021

    2021

  7. [15]

    Taco: Topics in algorithmic code generation dataset

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852, 2023. 18 A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning

    2023 arXiv

  8. [16]

    Cipherbank: Exploring the boundary of llm reasoning capabilities through cryptography challenges

    Yu Li, Qizhi Pei, Mengyuan Sun, Honglin Lin, Chenlin Ming, Xin Gao, Jiang Wu, Conghui He, and Lijun Wu. Cipherbank: Exploring the boundary of llm reasoning capabilities through cryptography challenges. arXiv preprint arXiv:2504.19093, 2025

    2025 arXiv

  9. [17]

    Zebralogic: On the scaling limits of llms for logical reasoning, 2025

    Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of llms for logical reasoning, 2025. URL https://arxiv.org/ abs/2502.01100

    2025 arXiv

  10. [18]

    Code-r1: Reproducing r1 for code with reliable rewards

    Jiawei Liu and Lingming Zhang. Code-r1: Reproducing r1 for code with reliable rewards. 2025

    2025

  11. [19]

    Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models

    Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864, 2025

    2025 arXiv

  12. [20]

    Understanding r1-zero-like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  13. [21]

    Deepscaler: Sur- passing o1-preview with a 1.5b model by scaling rl

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Sur- passing o1-preview with a 1.5b model by scaling rl. https://pretty-radio-b75.notion.site/ DeepSca...

  14. [22]

    Tinyzero

    Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 2025. Accessed: 2025-01-24

    2025

  15. [23]

    Lemma: Learning from errors for mathematical advancement in llms

    Zhuoshi Pan, Yu Li, Honglin Lin, Qizhi Pei, Zinan Tang, Wei Wu, Chenlin Ming, H Vicky Zhao, Conghui He, and Lijun Wu. Lemma: Learning from errors for mathematical advancement in llms. arXiv preprint arXiv:2503.17439, 2025

    2025 arXiv

  16. [24]

    Vicky Zhao, Conghui He, and Lijun Wu

    Zhuoshi Pan, Qizhi Pei, Yu Li, Qiyao Sun, Zinan Tang, H. Vicky Zhao, Conghui He, and Lijun Wu. Rest: Stress testing large reasoning models by asking multiple problems at once, 2025. URL https://arxiv.org/ abs/2507.10541

    2025 arXiv

  17. [25]

    Curriculum reinforcement learning from easy to hard tasks improves llm reasoning

    Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, et al. Curriculum reinforcement learning from easy to hard tasks improves llm reasoning. arXiv preprint arXiv:2506.06632, 2025

    2025

  18. [26]

    Mathfusion: Enhancing mathematical problem-solving of llm through instruction fusion

    Qizhi Pei, Lijun Wu, Zhuoshi Pan, Yu Li, Honglin Lin, Chenlin Ming, Xin Gao, Conghui He, and Rui Yan. Mathfusion: Enhancing mathematical problem-solving of llm through instruction fusion. arXiv preprint arXiv:2503.16212, 2025

    2025 arXiv

  19. [27]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  20. [28]

    Magistral

    Abhinav Rastogi, Albert Q Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Bar- mentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, et al. Magistral. arXiv preprint arXiv:2506.10910, 2025

    2025 arXiv

  21. [29]

    Proximal policy optimiza- tion algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  22. [30]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  23. [31]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256 , 2024. 19 A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning

    2024 arXiv

  24. [32]

    Template matters: Understanding the role of instruction templates in multimodal language model evaluation and training

    Shijian Wang, Linxin Song, Jieyu Zhang, Ryotaro Shimizu, Ao Luo, Li Yao, Cunjian Chen, Julian McAuley, and Hanqian Wu. Template matters: Understanding the role of instruction templates in multimodal language model evaluation and training. In ICLR 2025 Workshop on Navigating an...

    2025

  25. [33]

    Dump: Automated distribution-level curriculum learning for rl-based llm post-training

    Zhenting Wang, Guofeng Cui, Yu-Jhe Li, Kun Wan, and Wentian Zhao. Dump: Automated distribution-level curriculum learning for rl-based llm post-training. arXiv preprint arXiv:2504.09710, 2025

    2025

  26. [34]

    Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond

    Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503.10460, 2025

    2025 arXiv

  27. [35]

    Rlvr-world: Training world models with reinforcement learning

    Jialong Wu, Shaofeng Yin, Ningya Feng, and Mingsheng Long. Rlvr-world: Training world models with reinforcement learning. arXiv preprint arXiv:2505.13934, 2025

    2025

  28. [36]

    Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms, 2025

    Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms, 2025. URL https://arxiv.org/abs/2504.14655

    2025 arXiv

  29. [37]

    On memorization of large language models in logical reasoning

    Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. 2024. URL https: //arxiv.org/abs/2410.23123

    2024 arXiv

  30. [38]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768, 2025

    2025 arXiv

  31. [39]

    Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284, 2025

    Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, et al. Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284, 2025

    2025 arXiv

  32. [40]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  33. [41]

    Demystifying long chain-of-thought reasoning in llms

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373, 2025

    2025 arXiv

  34. [42]

    Rlpr: Extrapolating rlvr to general domains without verifiers

    Tianyu Yu, Bo Ji, Shouli Wang, Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, et al. Rlpr: Extrapolating rlvr to general domains without verifiers. arXiv preprint arXiv:2506.18254, 2025

    2025 arXiv

  35. [43]

    Large language model as attributed training data generator: A tale of diversity and bias, 2023

    Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias, 2023. URL https://arxiv.org/abs/2306.15895

    2023 arXiv

  36. [44]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  37. [45]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892, 2025

    2025 arXiv

  38. [46]

    R1-zero’s” aha moment” in visual reasoning on a 2b non-sft model

    Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero’s” aha moment” in visual reasoning on a 2b non-sft model. arXiv preprint arXiv:2503.05132, 2025. 20 A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning Appendix ...

    2025 arXiv

  39. [47]

    Peter is in the second house

  40. [48]

    Arnold is directly left of the one who only drinks water

  41. [49]

    ## Answer to the Example Puzzle { ”reasoning”: ”Given Clue 1, we know Peter is in House 2

    The one who only drinks water is directly left of the person who likes milk. ## Answer to the Example Puzzle { ”reasoning”: ”Given Clue 1, we know Peter is in House 2. According to Clue 2, Arnold is directly left of the one who only drinks water. The person in House 3 cannot b...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.