Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reinforcement-trained math models improve by up to 10 points when they synthesize practice problems from their own persistent failures.

desk verdict Useful weakness-signal idea, well-documented pipeline, but the main experiments never isolate weakness-driven synthesis from generic data augmentation, so the headline claim is conditional pending the missing random-synthetic control. read the letter →

arxiv 2506.08989 v1 pith:QUJ4RH3B submitted 2025-06-10 cs.LG cs.CL

Xiao Liang , Zhong-Zhi Li , Yeyun Gong , Yang Wang , Hengyuan Zhang , Yelong Shen , Ying Nian Wu , Weizhu Chen This is my paper · ORCID
classification cs.LGcs.CL
keywords reinforcementlearningwithverifiablerewardsproblemsynthesisweaknessidentificationmathematicalreasoninglargelanguagemodelsdataaugmentationself-improvementGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SwS asks a simple question: can a math model's own learning failures during reinforcement training be turned into better training data? The paper proposes a loop where the model first trains briefly on a standard problem set while its per-question accuracy is tracked, then flags the questions it consistently fails (never reaching 50% accuracy with a downward trend), extracts the mathematical concepts behind those failures, and generates new questions that recombine those concepts. The new questions are filtered for quality, answer consistency, and difficulty before the model continues training on the augmented set. Across eight reasoning benchmarks this yields average absolute gains of 10.0 points for the 7B model and 7.7 points for the 32B model over continued training on the original problems alone, and it also lets the 7B model solve up to 20 percentage points more of its previously intractable problems in its weakest domains. The claim matters because it addresses a practical bottleneck for verifiable-reward RL: well-annotated human math problems are scarce, and indiscriminate synthetic expansion wastes training signal.

What carries the argument

The central machinery is the failure-case definition of Eq. (4) — a problem is a weakness only if its per-epoch accuracy never reaches 50% and the accuracy trend has negative slope — feeding a generation-and-filtering pipeline of concept extraction and recombination (with co-occurrence and embedding similarity sampling), failure-rate-proportional budget allocation (Eq. 5), instruction-model quality verification, self-consistency answer labeling with a 50% consistency threshold, and difficulty filtering to the $[25\%,75\%]$ accuracy window. This pipeline converts the model's own training log into targeted practice problems that stay learnable and keep producing nonzero gradient signal under GRPO-style group-relative advantages.

What would settle it

Audit a random sample of the retained synthetic problems from the 32B run (about 41.7 thousand after difficulty filtering) by computing exact reference answers independently of the labeling model; if more than 5% of the sampled retained answers are wrong, the RL training signal is contaminated and the reported gains would need to be reinterpreted as partially fitting to label noise.

Watch

Extended reading notes

Core claim

The paper's central claim is that the model's own RL training dynamics reveal which reasoning skills it lacks, and that those deficits can be converted directly into targeted synthetic practice. A problem counts as a weakness if the model never exceeds 50% per-epoch accuracy on it and its accuracy trend slopes downward (Eq. 4). The framework extracts core concepts from such failure cases, recombines them into new questions with category budgets proportional to failure rates (Eq. 5), and retains only questions that pass model-based quality checks, a 50% self-consistency answer label from a strong reasoning model, and a difficulty band of roughly $[25\%,75\%]$ accuracy on the model being trained. Continuing RL on this augmented set beats continued training on the original data, surpasses models trained on curated human-labeled and distillation-oriented sets in most comparisons, and shows the largest recovery precisely in the domains flagged as weakest.

Load-bearing premise

The pipeline assumes that the self-consistency-labeled answers on the retained synthetic problems are correct often enough to serve as clean RL rewards, even though those labels come from a single reasoning model and are never checked against known ground truth.

Editorial extensions

If this is right

  • Continued RL on the weakness-driven augmented set recovers up to 20 percentage points more of the 7B model's consistently failed problems in its weakest domains (Intermediate Algebra, Geometry, Precalculus) than continued training on the original set, while each original problem is sampled four times less often.
  • The pipeline transfers across scales from 3B to 32B and across both initial problem sets (MATH-12k for smaller models and the DAPO/LightR1 combined set for 14B and 32B), with mean absolute gains of 10.0% (7B) and 7.7% (32B) on eight benchmarks.
  • SwS-trained models surpass several recent RL-trained baselines that rely on carefully curated or human-labeled problem sets, not just the base and instruct models.
  • Difficulty filtering is an active ingredient: simple synthetic problems are mastered quickly and then stop producing learning signal, while medium and hard retained problems give slower convergence but more sustained gains on evaluation benchmarks.
  • Weakness-driven selection of problems from a large external dataset (Big-Math) outperforms random selection on both overall and competition-level accuracy, and targeted problems remain challenging longer during GRPO training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, the same weakness-identification signal should transfer beyond exact-answer math to settings where a reward model supplies feedback instead of a verifiable answer; the paper names this as an open direction, and the natural test is whether failure-rate trends under a reward model identify the same kinds of persistent deficits.
  • The paper ablates difficulty but not the failure-rate-proportional budget allocation of Eq. 5, so a controlled comparison against uniform per-category budgets would tell whether weakness-relative allocation adds anything beyond the synthesis and filtering pipeline itself.
  • The Self-evolving extension (+1.2% on a 14B model) is consistent with the interpretation that the main experimental gains come partly from the strength of the external generation and labeling models (LLaMA-3.3-70B-Instruct and QwQ-32B), not only from the weakness-targeting idea; a decisive test would use weak teachers for generation while keeping labeling fixed, or vice versa.
  • Because the baseline continues training on the original set for the same number of steps, the headline comparison already controls for compute; a matched-difficulty random-synthetic set would isolate how much of the gain comes from difficulty curation versus weakness targeting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SwS, a self-aware weakness-driven problem synthesis framework for RLVR training of LLM reasoners. After a preliminary RL phase, problems the model consistently fails (max epoch accuracy below 50% with a negative accuracy trend) are treated as weaknesses. Concepts are extracted from these failure cases, recombined within categories, and used to prompt an instruction model to generate new problems. Generated problems pass quality filtering, reference-answer labeling via a strong reasoning model with a 50% self-consistency threshold, and difficulty filtering to a 25%–75% initial-model accuracy range. The retained synthetic problems (40k per model) are added to the initial training set for a second RL stage. Experiments on Qwen2.5 3B/7B/32B across eight benchmarks report consistent gains over a BaseRL baseline trained on the initial set, with average improvements of 10.0% and 7.7% on 7B and 32B. The paper also explores weak-to-strong generalization, self-evolving synthesis, and weakness-driven selection from Big-Math.

Significance. If the central mechanism is validated, the contribution is valuable: it offers a data-augmentation strategy for RLVR that does not require human-labeled synthetic answers and that targets the model's own failure regions, with results across model scales. The paper's strengths include a thorough evaluation across eight benchmarks, a transparent data-workflow diagram (Figure 8), a concrete quality-filtering case study (Table 4), difficulty ablations (Section 4.4), and public code/links. The main claim, however, requires a matched random-synthetic control of the generation pipeline; currently the evidence supports 'adding synthetic problems helps' more strongly than 'adding weakness-driven synthetic problems helps.' The reference-answer reliability issue is real and only partially mitigated by the difficulty filter. For these reasons the result is plausible but not yet established at the level of the paper's headline claim.

major comments (4)
  1. [Section 3.1, Table 1; Section 4.3, Figure 5] The headline comparison conflates two variables: dataset size and weakness targeting. SwS trains on the initial set plus 40k synthetic problems, while BaseRL trains on the initial set alone, so the reported gains (e.g., +10.0% for 7B) may be attributable to additional training data volume rather than to the weakness-driven selection mechanism. The only weakness-vs-random comparison in the paper, Section 4.3 and Figure 5, uses retrieval from Big-Math via KNN, not the concept-extraction/recombination generation pipeline of Section 2.4. A control that augments the initial set with 40k problems generated from randomly sampled concepts, and processed through identical quality, answer-labeling, and difficulty filtering, is needed to isolate the weakness-driven component. Without it, the paper's title claim is not directly tested.
  2. [Section 2.4, 'Reference Answer Generation'] The RL training signal depends on reference answers produced by a single external reasoning model with a 50% self-consistency threshold (Skywork-OR1-Math-7B for small models, QwQ-32B for 32B), and these answers are never checked against known ground truth. If a non-trivial fraction of retained problems has incorrect labels, the RL objective is corrupted and the reported gains may partly reflect fitting to labeler bias. The weak-to-strong analysis in Table 5 is informative, but it validates labels only on MATH-500 where the true answers are known, not on the generated synthetic problems. Please report a label-accuracy estimate on a human- or independently verified sample of the synthetic set, and show sensitivity to the 50% consistency threshold.
  3. [Table 1] All main results appear to be single runs without variance estimates. Several benchmark deltas are driven by very small numbers of problems: AIME24 and AIME25 each contain 30 problems, so a 16.7-point difference corresponds to five correct responses, and the Avg@1 numbers fluctuate accordingly. Since the headline average gains are the central empirical claim, at least the aggregate numbers and the competition-level subset should be accompanied by multiple seeds or confidence intervals to establish that the differences are not sampling noise.
  4. [Section 4.3 and Appendix H] The random augmentation baseline in the weakness-driven selection experiment is under-specified. The text says the baseline uses 'randomly selected problems from Big-Math,' but it does not state whether the random subset is matched to the weakness-driven subset in size, category distribution, and quality filtering. If the random set differs in any of these dimensions, Figure 5 cannot cleanly attribute the observed gap to weakness-driven selection. Please specify the exact construction of the random baseline, including category budgets and filtering steps.
minor comments (5)
  1. [Equation (4)] The 'slope' of the accuracy sequence is not defined; please specify whether it is the slope of a linear fit over epochs or a finite difference between the first and last epochs, and state the number of epochs used for the trend test.
  2. [Abstract and Section 3.1] The abstract states that gains are achieved 'without relying on external knowledge distillation,' but the pipeline uses LLaMA-3.3-70B-Instruct for generation and quality assessment and QwQ-32B/Skywork-OR1 for answer labeling. This is not necessarily distillation, but the phrasing should be clarified to distinguish 'no distillation of teacher solutions' from 'no external models at all.'
  3. [Appendix H, Algorithm 1] Line 3 of Algorithm 1 refers to 'Eq. 2' for the selection budget, but the budget allocation is given by Eq. (5) in the main text. Please correct the cross-reference.
  4. [Table 1 and Section 4.1] The table header 'Avg.' is not defined; clarify whether it is an unweighted average over the eight benchmark columns, and how the Avg@1 and Avg@32 AIME values are combined. The text uses 'average absolute improvement' without specifying the exact aggregation rule.
  5. [Section 4.2, Table 3] The self-evolving result shows negative deltas on MATH-500 and AIME; the attribution of this decline to the initial DAPO/LightR1 data being 'already specifically tailored' to those benchmarks would be stronger with a quantitative comparison of the initial data's overlap with the evaluation sets.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SwS is a self-referential curriculum/synthesis pipeline, but the benchmark results are measured against external held-out data and do not reduce to the pipeline's own fit.

full rationale

The paper's derivation chain is an empirical pipeline, not a formal derivation that assumes its conclusion. Weaknesses are defined operationally in Eq. 4 (max accuracy < 0.5 and negative slope over RL epochs); synthetic problems are generated from concepts extracted from those failures and then filtered by the initial model's accuracy (Section 2.4, difficulty filtering). This is a curriculum-style, model-adaptive data-selection procedure, but it is not circular: the reported gains in Table 1 are evaluated on held-out external benchmarks (GSM8K, MATH-500, AIME, etc.) whose answers are known independently of the synthetic-problem labeling pipeline. The difficulty filter selects training problems the model can partially solve, which may explain part of the gain, but that is a confounding experimental factor, not an equation that equates the output with the input. The paper cites prior work for concept extraction and recombination, including [15] with overlapping Microsoft authors, but the citation is used as a design antecedent ('Inspired by [15, 73]'), not as a load-bearing uniqueness or correctness theorem. The missing random-synthetic control for the full Section 2.4 generation pipeline is a real experimental gap, since Table 1 only compares SwS (initial set + weakness-derived synthetic problems) against BaseRL (initial set alone), and the target-vs-random comparison in Section 4.3 uses KNN retrieval from Big-Math rather than the concept-recombination generator; however, an absent control is not circularity. No step in the paper reduces by construction to its own inputs, so the appropriate circularity score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the hand-set weakness definition (two thresholds), the difficulty filter range, the self-consistency threshold for labels, and the budgets. None of these are derived from theory; they are chosen values. The main domain assumptions are about the reliability of LLM-based generation and self-consistency labeling for synthetic problems, which are not validated against known ground truth in the main experiments.

free parameters (6)
  • max_accuracy_threshold = 0.5
    Equation 4 defines a failed problem as one where max accuracy across epochs never reaches 50%. This threshold is hand-set and directly determines which problems are labeled weaknesses.
  • slope_threshold = <0
    Equation 4 requires a negative accuracy trend to call a problem a failure. This hand-set criterion is intended to capture problems showing no learning improvement.
  • difficulty_filter_range = [25%, 75%]
    Section 2.4 retains synthetic problems where the initially trained model's accuracy falls in [acclow, acchigh], e.g., [25%, 75%]. This range is chosen by hand and controls the entire augmented training set composition.
  • self_consistency_threshold = 50%
    Reference answers are kept only if the most common answer appears in at least 50% of QwQ-32B's generations. This threshold is hand-set and is the only label-sanity check for synthetic problems.
  • synthetic_budget = 40,000
    Section 3.1 sets the final synthetic problem budget to 40k per base model. This budget is arbitrary and could affect the magnitude of the reported gains.
  • initial_synthetic_budget = 1,000,000
    Appendix D.1 sets an initial generation budget of 1M synthetic problems before filtering. The choice of a larger initial pool affects diversity and quality after filtering.
assumptions (4)
  • domain assumption GRPO advantages are informative only when group rewards are mixed, so filtering to moderate difficulty improves training efficiency.
    Sections 2.1 and 2.4 argue that all-correct or all-incorrect rollout groups give zero advantages. This is a known property of the algorithm but still an assumption about why the difficulty filter helps.
  • domain assumption Self-consistency of QwQ-32B is a reliable proxy for correctness on novel, unverified synthetic problems.
    Section 2.4 uses a 50% agreement threshold to label answers, with no check against ground truth. The validity of this assumption is load-bearing for the entire training signal.
  • domain assumption Concept extraction and recombination from failed problems produce mathematically coherent and solvable new problems.
    Section 2.4 relies on LLaMA-3.3-70B-Instruct to extract concepts and generate problems, with quality filtering by another LLM. There is no independent verification that the generated problems are well-posed.
  • standard math Math-Verify correctly determines answer equivalence across different formats (fractions, decimals, etc.).
    Section 2.4 uses Math-Verify to check self-consistency. This is an external tooling assumption that the answer-matching is accurate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning." pith.science (2026). https://pith.science/paper/QUJ4RH3B

@misc{pith2026250608989,
  author       = {Pith},
  title        = {Pith review of: SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QUJ4RH3B}},
  note         = {Machine review of arXiv:2506.08989}
}
read the original abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for training large language models (LLMs) on complex reasoning tasks, such as mathematical problem solving. A prerequisite for the scalability of RLVR is a high-quality problem set with precise and verifiable answers. However, the scarcity of well-crafted human-labeled math problems and limited-verification answers in existing distillation-oriented synthetic datasets limit their effectiveness in RL. Additionally, most problem synthesis strategies indiscriminately expand the problem set without considering the model's capabilities, leading to low efficiency in generating useful questions. To mitigate this issue, we introduce a Self-aware Weakness-driven problem Synthesis framework (SwS) that systematically identifies model deficiencies and leverages them for problem augmentation. Specifically, we define weaknesses as questions that the model consistently fails to learn through its iterative sampling during RL training. We then extract the core concepts from these failure cases and synthesize new problems to strengthen the model's weak areas in subsequent augmented training, enabling it to focus on and gradually overcome its weaknesses. Without relying on external knowledge distillation, our framework enables robust generalization byempowering the model to self-identify and address its weaknesses in RL, yielding average performance gains of 10.0% and 7.7% on 7B and 32B models across eight mainstream reasoning benchmarks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Solvers to Research: Large Language Model-Driven Formal Mathematics at the Research Frontier

    cs.CL 2026-07 accept novelty 6.0 of 10

    LLM formal provers must shift from competition solvers to research agents that handle open-ended, under-specified frontier mathematics under machine-checked rigor.

  2. Libra: Large Chinese-based Safeguard for AI Content

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Libra-Guard, a Chinese-language AI content safety model, reaches 86.8% accuracy on the new 5,720-sample Libra-Test benchmark, beating open-source baselines and approaching GPT-4o and Claude-3.5-Sonnet.

  3. Spectral Origins of the Self-Correction Blind Spot in Autoregressive Generation

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Self-correction blind spots in residual-stream autoregressive models arise iff the product of attention Jacobians has spectral radius ≥1, with a sharp marker threshold and RL coupling condition derived from that radius.

  4. Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematics

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A self-evolving curriculum that retrains a language model on variants of problems it can mostly get right lifts AIME pass@1 from 5.6% to 16.5%, beating static augmentation under the same data budget.

  5. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

93 extracted references · 23 canonical work pages · cited by 5 Pith papers

  1. [1]

    Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models.arXiv preprint arXiv:2502.17387, 2025

    Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, et al. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models.arXiv preprint arXiv:2502.17387, 2025

    arXiv 2025

  2. [2]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390, 2023

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390, 2023

    arXiv 2023

  3. [3]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training.arXiv preprint arXiv:2501.17161, 2025

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training.arXiv preprint arXiv:2501.17161, 2025

    arXiv 2025

  4. [4]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    KarlCobbe,VineetKosaraju,MohammadBavarian,MarkChen,HeewooJun,LukaszKaiser,Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  5. [5]

    Nearest neighbor pattern classification.IEEE transactions on information theory, 13(1):21–27, 1967

    Thomas Cover and Peter Hart. Nearest neighbor pattern classification.IEEE transactions on information theory, 13(1):21–27, 1967

    1967

  6. [6]

    Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    arXiv 2025

  7. [7]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URLhttps: //github.com/huggingface/open-r1

    2025

  8. [8]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 93 references
  1. [9]

    rstar-math: Small llms can master math reasoning with self-evolved deep thinking.arXiv preprint arXiv:2501.04519, 2025

    Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking.arXiv preprint arXiv:2501.04519, 2025

    2025 arXiv

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  3. [11]

    Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scien...

    2024

  4. [12]

    Skyworkopenreasonerseries

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, andYahuiZhou. Skyworkopenreasonerseries. https://capricious-hydrogen-41c.notion. site...

    2025

  5. [13]

    Measuring mathematical problem solving with the math dataset.Sort, 2(4): 0–6, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.Sort, 2(4): 0–6, 2021

    2021

  6. [14]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    2025 arXiv

  7. [15]

    Key-point-driven data synthesis with its enhancement on mathematical reasoning.arXiv preprint arXiv:2403.02333, 2024

    Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning.arXiv preprint arXiv:2403.02333, 2024

    2024 arXiv

  8. [16]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  9. [17]

    Knowledge- augmented reasoning distillation for small language models in knowledge-intensive tasks.Advances in Neural Information Processing Systems, 36:48573–48602, 2023

    Minki Kang, Seanie Lee, Jinheon Baek, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge- augmented reasoning distillation for small language models in knowledge-intensive tasks.Advances in Neural Information Processing Systems, 36:48573–48602, 2023

    2023

  10. [18]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems ...

    2023

  11. [19]

    Solving quantitative reasoning problems with language models.Advances in Neural Information Processing Systems, 35: 3843–3857, 2022

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models.Advances in Neural Information Processing Systems, 35:...

    2022

  12. [20]

    Common 7b language models already possess strong math capabilities.arXiv preprint arXiv:2403.04706, 2024

    Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities.arXiv preprint arXiv:2403.04706, 2024

    2024 arXiv

  13. [21]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge.arXiv preprint arXiv:2411.16594, 2024

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge.arXiv preprint arXiv:2411.16594, 2024

    2024

  14. [22]

    Limr: Less is more for rl scaling.arXiv preprint arXiv:2502.11886, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling.arXiv preprint arXiv:2502.11886, 2025

    2025 arXiv

  15. [23]

    Tl; dr: Too long, do re-weighting for effcient llm reasoning compression.arXiv preprint arXiv:2506.02678, 2025

    Zhong-Zhi Li, Xiao Liang, Zihao Tang, Lei Ji, Peijie Wang, Haotian Xu, Haizhen Huang, Weiwei Deng, Ying Nian Wu, Yeyun Gong, et al. Tl; dr: Too long, do re-weighting for effcient llm reasoning compression.arXiv preprint arXiv:2506.02678, 2025

    2025 arXiv

  16. [24]

    From system 1 to system 2: A survey of reasoning large language models.arXiv preprint arXiv:2502.17419, 2025

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system 1 to system 2: A survey of reasoning large language models.arXiv preprint arXiv:2502.17419, 2025

    2025 arXiv

  17. [25]

    Task oriented in-domain data augmentation.arXiv preprint arXiv:2406.16694, 2024

    Xiao Liang, Xinyu Hu, Simiao Zuo, Yeyun Gong, Qiang Lou, Yi Liu, Shao-Lun Huang, and Jian Jiao. Task oriented in-domain data augmentation.arXiv preprint arXiv:2406.16694, 2024. 15 SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning

    2024 arXiv

  18. [26]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  19. [27]

    Augmenting math word problems via iterative question composing

    Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew C Yao. Augmenting math word problems via iterative question composing. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24605–24613, 2025

    2025

  20. [28]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    2025 arXiv

  21. [29]

    Scp- 116k: A high-quality problem-solution dataset and a generalized pipeline for automated extraction in the higher education science domain, 2025

    Dakuan Lu, Xiaoyu Tan, Rui Xu, Tianchu Yao, Chao Qu, Wei Chu, Yinghui Xu, and Yuan Qi. Scp- 116k: A high-quality problem-solution dataset and a generalized pipeline for automated extraction in the higher education science domain, 2025. URLhttps://arxiv.org/abs/2501.15587

    2025 arXiv

  22. [30]

    Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.arXiv preprint arXiv:2308.09583, 2023

    HaipengLuo,QingfengSun,CanXu,PuZhao,JianguangLou,ChongyangTao,XiuboGeng,Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.arXiv preprint arXiv:2308.09583, 2023

    2023 arXiv

  23. [31]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. DeepScaleR Notion Page, 2025. Notion Blog

    2025

  24. [32]

    Reft: Reasoning with reinforced fine-tuning.arXiv preprint arXiv:2401.08967, 3, 2024

    Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning.arXiv preprint arXiv:2401.08967, 3, 2024

    2024 arXiv

  25. [33]

    American mathematics competitions (AMC 10/12)

    MAA. American mathematics competitions (AMC 10/12). Mathematics Competition Series, 2023. URLhttps://maa.org/math-competitions/amc

    2023

  26. [34]

    American invitational mathematics examination (AIME)

    MAA. American invitational mathematics examination (AIME). Mathematics Competition Series,

  27. [35]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  28. [36]

    Smoothing out hallucinations: Mitigating llm hallucination with smoothed knowledge distillation.arXiv preprint arXiv:2502.11306, 2025

    Hieu Nguyen, Zihao He, Shoumik Atul Gandre, Ujjwal Pasupulety, Sharanya Kumari Shivakumar, and Kristina Lerman. Smoothing out hallucinations: Mitigating llm hallucination with smoothed knowledge distillation.arXiv preprint arXiv:2502.11306, 2025

    2025 arXiv

  29. [37]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730– 27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  30. [38]

    Mathfusion: Enhancing mathematic problem-solving of llm through instruction fusion.arXiv preprint arXiv:2503.16212, 2025

    Qizhi Pei, Lijun Wu, Zhuoshi Pan, Yu Li, Honglin Lin, Chenlin Ming, Xin Gao, Conghui He, and Rui Yan. Mathfusion: Enhancing mathematic problem-solving of llm through instruction fusion.arXiv preprint arXiv:2503.16212, 2025. 16 SwS: Self-aware Weakness-driven Problem Synthesis ...

    2025 arXiv

  31. [39]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  32. [40]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    ZhihongShao,PeiyiWang,QihaoZhu,RunxinXu,JunxiaoSong,XiaoBi,HaoweiZhang,Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  33. [41]

    Exploring data scaling trends and effects in reinforcement learning from human feedback.arXiv preprint arXiv:2503.22230, 2025

    Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue, and Lin Yan. Exploring data scaling trends and effects in reinforcement learning from human feedback.arXiv preprint arXiv:2503.22230, 2025

    2025 arXiv

  34. [42]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  35. [43]

    Efficient reinforcement finetuning via adaptive curriculum learning.arXiv preprint arXiv:2504.05520, 2025

    Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning.arXiv preprint arXiv:2504.05520, 2025

    2025 arXiv

  36. [44]

    Large language models for data annotation and synthesis: A survey

    Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey. InProceedings of the 2024 Conference on Empirical Methods in Natural ...

    2024

  37. [45]

    Mathscale: Scaling instruction tuning for mathematical reasoning

    Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. InInternational Conference on Machine Learning, pages 47885– 47900. PMLR, 2024

    2024

  38. [46]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  39. [47]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/

    2025

  40. [48]

    Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving.Advances in Neural Information Processing Systems, 37:7821–7846, 2024

    Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving.Advances in Neural Information Processing Systems, 37:7821–7846, 2024

    2024

  41. [49]

    Openmathinstruct-1: A 1.8 million math instruction tuning dataset.Advances in Neural Information Processing Systems, 37:34737–34774, 2024

    Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-1: A 1.8 million math instruction tuning dataset.Advances in Neural Information Processing Systems, 37:34737–34774, 2024

    2024

  42. [50]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations.arXiv preprint arXiv:2312.08935, 2023

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations.arXiv preprint arXiv:2312.08935, 2023

    2023 arXiv

  43. [51]

    Explore the reasoning capability of llms in the chess testbed.arXiv preprint arXiv:2411.06655, 2024

    Shu Wang, Lei Ji, Renxi Wang, Wenxiao Zhao, Haokun Liu, Yifan Hou, and Ying Nian Wu. Explore the reasoning capability of llms in the chess testbed.arXiv preprint arXiv:2411.06655, 2024

    2024 arXiv

  44. [52]

    Examining false positives under inference scaling for mathematical reasoning.arXiv preprint arXiv:2502.06217, 2025

    Yu Wang, Nan Yang, Liang Wang, and Furu Wei. Examining false positives under inference scaling for mathematical reasoning.arXiv preprint arXiv:2502.06217, 2025. 17 SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning

    2025

  45. [53]

    Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.arXiv preprint arXiv:2503.10460, 2025

    Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.arXiv preprint arXiv:2503.10460, 2025

    2025 arXiv

  46. [54]

    Fine-grained human feedback gives better rewards for language model training.Advances in Neural Information Processing Systems, 36:59008–59033, 2023

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training.Advances in Neural Information Processing Systems, 36:59008–59033, 2023

    2023

  47. [55]

    A minimalist approach to llm reasoning: from rejection sampling to reinforce.arXiv preprint arXiv:2504.11343, 2025

    Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to llm reasoning: from rejection sampling to reinforce.arXiv preprint arXiv:2504.11343, 2025

    2025 arXiv

  48. [56]

    Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  49. [57]

    AnYang,BaosongYang,BeichenZhang,BinyuanHui,BoZheng,BowenYu,ChengyuanLi,Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  50. [58]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024

    2024 arXiv

  51. [59]

    Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387, 2025

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387, 2025

    2025 arXiv

  52. [60]

    Demystifying long chain- of-thought reasoning in llms.arXiv preprint arXiv:2502.03373, 2025

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain- of-thought reasoning in llms.arXiv preprint arXiv:2502.03373, 2025

    2025 arXiv

  53. [61]

    Long-short chain-of- thought mixture supervised fine-tuning eliciting efficient reasoning in large language models.arXiv preprint arXiv:2505.03469, 2025

    Bin Yu, Hang Yuan, Yuliang Wei, Bailing Wang, Weizhen Qi, and Kai Chen. Long-short chain-of- thought mixture supervised fine-tuning eliciting efficient reasoning in large language models.arXiv preprint arXiv:2505.03469, 2025

    2025 arXiv

  54. [62]

    Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  55. [63]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  56. [64]

    Chain-of-reasoning: Towards unified mathematical reasoning in large language models via a multi-paradigm perspective.arXiv preprint arXiv:2501.11110, 2025

    Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Ziyi Yang, Mahmoud Khademi, Hany Awadalla, Junjie Wang, et al. Chain-of-reasoning: Towards unified mathematical reasoning in large language models via a multi-paradigm perspective.arXiv prepri...

    2025 arXiv

  57. [65]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025

    Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118, 2025. 18 SwS: Self-aware W...

    2025 arXiv

  58. [66]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  59. [67]

    Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild.arXiv preprint arXiv:2503.18892, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild.arXiv preprint arXiv:2503.18892, 2025

    2025 arXiv

  60. [68]

    Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    2024

  61. [69]

    Balancing speciality and versatility: a coarse to fine framework for supervised fine-tuning large language model

    Hengyuan Zhang, Yanru Wu, Dawei Li, Sak Yang, Rui Zhao, Yong Jiang, and Fei Tan. Balancing speciality and versatility: a coarse to fine framework for supervised fine-tuning large language model. InFindings of the Association for Computational Linguistics ACL 2024, pages 7467–7...

    2024

  62. [70]

    Process-based self-rewarding language models.arXiv preprint arXiv:2503.03746, 2025

    Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, and Yeyun Gong. Process-based self-rewarding language models.arXiv preprint arXiv:2503.03746, 2025

    2025 arXiv

  63. [71]

    Evaluating the performance of large language models on gaokao benchmark.arXiv preprint arXiv:2305.12474, 2023

    Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark.arXiv preprint arXiv:2305.12474, 2023

    2023 arXiv

  64. [72]

    1.4 million open-source distilled reasoning dataset to empower large language model training.arXiv preprint arXiv:2503.19633, 2025

    Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li. 1.4 million open-source distilled reasoning dataset to empower large language model training.arXiv preprint arXiv:2503.19633, 2025

    2025 arXiv

  65. [73]

    Promptcot: Synthesizing olympiad-level problems for mathematical reasoning in large language models.arXiv preprint arXiv:2503.02324, 2025

    Xueliang Zhao, Wei Wu, Jian Guan, and Lingpeng Kong. Promptcot: Synthesizing olympiad-level problems for mathematical reasoning in large language models.arXiv preprint arXiv:2503.02324, 2025

    2025

  66. [74]

    Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  67. [75]

    Ttrl: Test-time reinforcement learning, 2025

    Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning, 2025. URLhttps: //arxiv.org/abs/2504.16084. 19 SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Lear...

    2025 arXiv

  68. [77]

    Let’s think step by step and output the final answer within “∖boxed{}

    successfully incorporated a process reward model (PRM), trained on process-level labels generated via Monte Carlo sampling at each step, into RL training and demonstrated its effectiveness. Beyond RL training, PRM can also be used to guide inference [4] and provide value estim...

    1905

  69. [78]

    Begin by ou tl in in g which concepts you will combine and how you plan to s tr uc tu re the question

  70. [79]

    Ensure that the question is coherent , relevant , and a p p r o p r i a t e l y c h a l l e n g i n g for the s pe ci fi ed level

  71. [80]

    The question must be a single s t a n d a l o n e problem , not split into multiple sub - q ue st io n s

  72. [81]

    Do not generate proof - based , multiple - choice , or true / false qu es ti on s

  73. [82]

    The answer to the question should be e x p r e s s i b l e using numbers and m a t h e m a t i c a l symbols

  74. [83]

    ### Output Format : - First , provide your brief outline and planning for the question design

    Provide a final version of the question that is polished and ready for use . ### Output Format : - First , provide your brief outline and planning for the question design . 37 SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning - Then ...

  75. [84]

    Problem & Design Ra ti on al e : {rationale_and_problem} ( The r at io na le d es cr ib es the author ’ s thinking process and j u s t i f i c a t i o n in d es ig ni ng this problem )

  76. [85]

    F o u n d a t i o n a l Concepts : { concepts }

  77. [86]

    Target D i f f i c u l t y Level : { level } === E V A L U A T I O N CRITERIA === Rate each cr it er io n as : [ Perfect | A c c e p t a b l e | Bad ]

  78. [87]

    FORMAT - Verify correct i m p l e m e n t a t i o n of markup tags : <! - BEGIN RA TI ON AL E - > [ design thinking process ] <! - END RA TI ON AL E - > <! - BEGIN PROBLEM - > [ problem ] <! - END PROBLEM - >

  79. [88]

    FACTUAL ACCURACY - Check for any in co rr ec t or m i s l e a d i n g i n f o r m a t i o n in both problem and r at io na le - Verify mathematical , scientific , or logical c o n s i s t e n c y

  80. [89]

    D I F F I C U L T Y AL IG NM EN T - Assess if problem c o m p l e x i t y matches the sp ec if ie d d i f f i c u l t y level - Evaluate if co gn it iv e demands align with target level

  81. [90]

    CONCEPT COVERAGE - Evaluate how well the problem i n c o r p o r a t e s the given f o u n d a t i o n a l concepts - Check for missing concept a p p l i c a t i o n s

  82. [91]

    S O L V A B I L I T Y 38 SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning - Verify if the problem has at least one valid solution - Check if all ne ce ss ar y i n f o r m a t i o n for solving is provided === RESPONSE FORMAT === For...

  83. [92]

    Rating : [ Perfect | A c c e p t a b l e | Bad ]

  84. [93]

    J u s t i f i c a t i o n : Clear e x p l a n a t i o n for the rating === FINAL VERDICT === After p ro vi di ng all c ri te ri on evaluations , conclude your response with : ‘ Final J ud g em en t : [ verdict ] ’ where verdict must be one of : - ‘ perfect ’ ( if both FACTUAL ...

  85. [2024]

    URLhttps://maa.org/math-competitions/aime

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.