Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

A single language model switching between teacher and student roles, trained only on its own self-generated problems from one trivial seed, improves across seven math benchmarks and beats models trained on over 7,000 labeled examples.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 00:31 UTC pith:EASDRUWR

load-bearing objection OpenSIR is a genuine, well-ablated extension of verifier-free self-play, but its reward loop never validates majority-vote labels against ground truth and the abstract oversells what the body shows. the 3 major comments →

arxiv 2511.00602 v3 pith:EASDRUWR submitted 2025-11-01 cs.CL

OpenSIR: Open-Ended Self-Improving Reasoner

Wai-Chung Kwan , Aryo Pradipta Gema , Joshua Ong Jun Leang , Pasquale Minervini This is my paper
classification cs.CL
keywords LLM reasoningself-playreinforcement learningmathematical problem generationopen-ended learningself-consistencydiversity rewardcurriculum learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

OpenSIR sets out to show that a language model can improve its own mathematical reasoning without any human-written training problems and without an external answer verifier. The same policy alternates between teacher and student: it invents problems conditioned on a growing pool of its own earlier problems, then solves each new problem several times, using majority voting among its answers as the reference and the solve rate as a reliability signal. With a novelty reward made of solvability, solution length, embedding-based diversity, and format, plus a correctness reward for matching the majority answer, training starts from the single seed "What is 1+1?" and reports average gains of +3.6 points on instruction models and +3.1 on reasoning models. The claim matters because if it holds, the most expensive parts of reasoning RL—annotated datasets and verifiable rewards—are not necessary for continued self-improvement, and the same loop could be pointed at other domains.

Core claim

OpenSIR's central claim is that verifier-free, annotation-free self-improvement is possible for LLM math reasoning when the same policy is trained in both roles. The teacher samples reference problems from a pool initialised to a single trivial problem and generates new problems, each required to name its mathematical concepts. The student produces G solution attempts per problem; the most frequent parsed answer becomes the reference answer, and the solve rate is its frequency. The teacher's novelty score is a weighted sum of a triangular solvability reward (peaking at an intermediate solve rate and zero outside the accepted band), a normalised solution-length reward, a diversity reward meas

What carries the argument

The load-bearing mechanism is the closed teacher-student loop inside one policy, driven by two reward functions. The teacher reward is a novelty score: a triangular solvability term that rewards problems whose self-solve rate is near 0.7 and assigns zero outside the [0.5, 0.9] band; a solution-length term that rewards longer multi-step solutions; a diversity term that measures cosine distance from all problems already in the pool; and a format term. The student reward is self-consistency: a solution earns credit if its parsed answer equals the majority-vote answer across G attempts, plus a format bonus. Both rewards are standardised into advantages and combined in a policy-gradient update wi

Load-bearing premise

The load-bearing assumption is that the majority-vote answer produced by the model's own solutions is correct for the problems that pass the [0.5, 0.9] solve-rate filter; the paper's single external check of that filter agrees with the reference answer on only 70.82% of sampled problems.

What would settle it

Audit the training labels directly: take OpenSIR-generated problems that pass the [0.5, 0.9] solve-rate filter, obtain independent ground-truth solutions from human annotators or a much stronger solver, and compare them with the majority-vote references used in training. If agreement is at or below the 70.82% reported in the paper's external check, the correctness signal is too noisy to carry the claimed gains. A complementary test: run the same OpenSIR training but replace self-consistency labels with independent ground-truth labels and see whether the benchmark advantage persists.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the results hold, neither a human-annotated dataset nor an external verifier is required for reinforcement-learning-based math reasoning; training on self-generated problems is enough.
  • The gains are not an artifact of the starting point: switching from a trivial arithmetic seed to a geometry or a competition-level problem changes final accuracy by less than 0.5 points.
  • The additional compute for self-generated problems is modest: one extra forward pass per problem on top of G solution attempts, an overhead of about 12.5% at G=8.
  • The improvement appears to be genuine reasoning capacity rather than memorisation: pass@k curves improve at all sampling budgets and on benchmarks far from the training distribution.
  • The method can also be combined with annotated data: mixing self-generated problems with GSM8K raises average accuracy from 38.42 to 41.29, above either source alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the correctness reward trusts the model's own majority answer, the framework can only be as sound as that label. The paper's own external check of the 0.5-threshold variant found agreement with the reference answer on 70.82% of sampled problems; if that rate is representative, roughly 29% of retained training labels are wrong, and part of the reported gain could be for
  • Editorial inference: the difficulty-calibration idea—keep self-solve rate near a fixed target while maximising diversity—is domain-agnostic. In code, formal proof, or puzzle domains where self-consistency or executable checks can stand in for the majority vote, the same two-term novelty reward could yield open-ended self-improvement.
  • Editorial inference: the diversity reward as implemented is a global repulsion from the pool; the paper's own extended training shows a plateau after step 350, suggesting this mechanism eventually saturates. A richer diversity signal operating on discovered concepts or skills, rather than raw embeddings, is the natural next lever for unbounded exploration.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. OpenSIR proposes a self-play reinforcement-learning framework in which a single instruction-tuned LLM alternates between a teacher role (generating math problems) and a student role (solving them). Problem generation is seeded from a growing pool of self-generated problems, and rewards are computed from solve-rate-based difficulty, solution length, embedding diversity, and format. Reference answers are obtained by majority voting over G sampled solutions, and the student reward is the agreement of each solution with that reference. Training uses a GRPO-style on-policy update. Starting from a single trivial seed, the paper reports consistent average gains on five math benchmarks across four model families, with ablations attributing the gains to difficulty calibration, diversity rewards, and joint teacher-student training. The central claim is that verifier-free, annotation-free self-improvement for LLM math reasoning is possible and can match or exceed GRPO baselines trained on thousands of human-annotated examples.

Significance. If the central claim holds, the result is significant: it would remove the reward-verification bottleneck that currently ties RLVR to curated, labeled datasets. The paper has notable strengths: per-seed results are reported in Tables 5-6, ablations isolate the contribution of each reward component, the seed-robustness study addresses a natural concern, and the pass@k analysis in A.6 speaks against a purely format-only gain. The method is also comparatively lightweight, with code and datasets released. However, the core claim rests on the reliability of self-generated majority-vote reference answers, and the manuscript does not establish that reliability; its own external check (Table 2) shows only 70.82% agreement with GPT-5 for the main configuration. The contribution is therefore promising but not yet proven to the standard required for the advertised verifier-free claim.

major comments (3)
  1. [§2.2 Eq. (1), §2.3.2 Eq. (6), Table 2] The entire student training signal is defined with respect to the reference answer a* = arg max_a count(a_j) from the policy's own G samples, and Eq. (6) rewards every solution that matches a*. The paper asserts that high solve rates indicate reliable answers, but no ground-truth validation of a* is provided. In Table 2, the column labeled 'vality' is actually measured by comparing GPT-5's majority answer to the reference answer, and for the main OpenSIR_0.5 configuration this agreement is 70.82%. If this estimate is even approximately correct, roughly 29% of training labels are wrong: correct solutions are penalized and the policy is rewarded for consistently reproducing majority errors. This is a load-bearing issue for the central claim of verifier-free, annotation-free self-improvement. The paper also conflates 'validity' (logical well-posedness, as defined in §4.1) with answer agreem
  2. [§3.4, Tables 5 and 6] The headline 'OpenSIR outperforms all GRPO baselines' is supported almost entirely by average differences, but with three seeds the evidence is often weak. For Llama-3.2-3B-Instruct, OpenSIR vs. GRPO_gsm8k is 38.42±0.27 vs. 37.83±0.37 (difference 0.59, overlapping standard deviations); for Qwen-2.5-3B-Instruct, 50.85±0.38 vs. 50.56±0.45 (difference 0.29). These are not statistically reliable differences. The claim should be restricted to configurations where the difference is robust (e.g., Gemma-2-2B-Instruct, where the gap is about 1.9 points) or should be backed by paired significance testing and effect sizes. As written, the abstract and §3.4 overstate the evidence for the comparison that is central to the paper's positioning.
  3. [A.7 and Abstract/Conclusion] The paper's own extended-training analysis states that performance plateaus after step 350 and that generated problems become increasingly similar and repetitive, attributing this to a less effective diversity reward over longer horizons. This is an admitted bound on the open-endedness that the title, abstract ('sustains open-ended exploration'), and conclusion ('continuously discover and master increasingly challenging mathematical concepts') claim. The claims should be qualified to the observed 200-350 step horizon, or the diversity mechanism should be shown to be extendable. As written, the evidence directly supports self-improvement over a bounded curriculum, not sustained open-ended learning.
minor comments (6)
  1. [Eq. (2)] The text says the solvability score 'decreases to 1/n at the boundaries,' but n is not defined and the formula gives 1/G at the boundaries. Replace '1/n' with '1/G'.
  2. [Abstract vs. §3.2] The abstract (as provided) claims 'seven math benchmarks' and reports gains on 'reasoning models' (+3.1 and +4.4), but the body evaluates five math benchmarks and does not report reasoning-model training results. Align the abstract with the experiments actually presented.
  3. [Table 2] The column labeled 'Validity' is not problem validity as defined in §4.1 — it is agreement between GPT-5's majority answer and the reference answer. Rename it to 'Answer agreement with GPT-5' to avoid misleading readers. Similarly, 'Solve Rate' in this table is GPT-5's solve rate, not the model's solve rate in Eq. (1); clarify the column.
  4. [Algorithm 1, line 7] The symbol q_i is overloaded. In line 4, q_{i,1:G} denotes a group of G problems generated from reference problem p_i; in line 7, the loop should iterate over individual generated problems q_{i,j} (or a renamed variable). The current notation makes the subsequent selection steps hard to follow and should be corrected for reproducibility.
  5. [Figure 2 / Appendix B] The human evaluation uses only 20 problems per source and the annotators are the paper's authors. Report inter-annotator agreement and state the small sample as a limitation, or the difficulty-ranking conclusions in §4.1 should be treated as illustrative.
  6. [Section 2.2] In the sentence 'we denote s qi = SolveRate(q_i)', the subscript formatting is broken; it should be s_{q_i} or a similar consistent notation.

Circularity Check

2 steps flagged

Training rewards reduce to self-agreement by construction, but the headline benchmark result is externally measured, so the core claim is not fully circular.

specific steps
  1. self definitional [§2.2 Eq. (1); §2.3.2 Eq. (6)]
    "SolveRate(qi) = count(a∗)/G where a∗ = arg max a∈a1:G count(a), (1) ... scorecorrect(oj, aj) = 1[aj = a∗] + δscoreS_fom(oj), (6)"

    The reference answer a* is the majority answer over G solutions sampled from the same policy πθ. The student correctness reward is then defined as agreement with a*. Therefore 'correctness' is, by construction, self-consistency rather than agreement with any external answer. The paper's own Table 2 check (GPT-5 majority vs reference answer) reports only 70.82% agreement on the main configuration, so a substantial fraction of training labels can be wrong while still being reinforced. This makes the training loop closed, although the reported benchmark gains are measured on held-out external benchmarks and are therefore not forced by this definition alone.

  2. other [§2.1–§2.3.1, Eq. (1)–(2), Eq. (5)]
    "We use solve rate as a proxy for solvability—problems with s qi > smax are likely too easy, while those with s qi < smin are either too difficult or malformed. ... score_sol(qi) = ..."

    The difficulty signal that drives the teacher reward is computed from SolveRate in Eq. (1), i.e., from the same policy's majority-vote consistency. The teacher is therefore rewarded for generating problems for which the student policy itself reaches moderate self-consistency; 'appropriate challenge' is defined relative to the policy's own current outputs rather than an independent notion of mathematical difficulty. This is a deliberate self-play curriculum, but it means the difficulty-calibration component also reduces to self-agreement.

full rationale

OpenSIR's central advertised result—that a model trained only on self-generated problems with majority-vote labels improves on external math benchmarks—is not itself a tautology: the numbers in Table 1 come from solving held-out GSM8K, MATH-500, Minerva, OlympiadBench, and College Math problems, so the main empirical claim has independent content. The circularity is internal to the training signal: Eq. (1) defines the reference answer a* as the majority of G solutions drawn from the current policy πθ, Eq. (6) rewards a student solution iff it matches a*, and Eq. (2)/Eq. (5) reward the teacher for problems whose self-consistency lands in a target interval. Thus 'correctness' and 'appropriate difficulty' are, by construction, self-agreement objectives rather than externally verified quantities. The paper's own GPT-5 check (Table 2) reports only 70.82% agreement with the reference answer on the main configuration, corroborating that the labels can diverge from an independent judge. Since the loop is transparent and the final evaluation is external, this is partial, not total, circularity: the method's internal rewards reduce to self-consistency, but the benchmark gains are not derived from that definition alone. No load-bearing self-citation or imported uniqueness theorem is present.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central loop rests on one big assumption: that a model's majority answer on its own generated problems is correct whenever the model can solve the problem 50-90% of the time. That assumption is load-bearing and weakly checked (GPT-5 agrees only 70.82% of the time). Beyond that, the method rides on the latent math knowledge of instruction-tuned base models, the choice of embedding space for diversity, and a half-dozen hand-set hyperparameters. No genuinely new entities are introduced; the teacher/student split is a procedural framing of one policy.

free parameters (6)
  • Solve-rate window (s_min, s_max) = (0.5, 0.9)
    Defines 'appropriately challenging' in Eq. 2; the lower bound was chosen via ablation on the same evaluation metric (Table 2 shows 0.5 > 0.3 > 0.1).
  • Novelty reward weights (alpha, lambda, gamma, delta) = (1.0, 1.0, 1.0, 0.1)
    Weights of solvability, length, diversity, format in Eq. 5 (Table 11); set by hand, no sensitivity analysis reported for these weights.
  • Solution attempts G = 8
    Number of rollouts for majority voting and solve-rate estimation (Eq. 1); sets the reward scale and the training compute.
  • Length normalization (l_base, l_cap) = l_base = 1000 tokens; l_cap value not stated
    Scales the solution-length reward in Eq. 3; l_base is hand-set and l_cap is left unspecified.
  • GRPO hyperparameters (KL, LR, batch, steps) = 1e-4, 3e-7, 256, 200
    Standard training settings (Table 11); OpenSIR runs 200 steps vs GRPO's 100, giving roughly 2.25x rollouts (Appendix A.9).
  • Embedding encoder for diversity = Linq-Embed-Mistral (7B, frozen)
    Defines the semantic space for Eq. 4; Table 10 shows robustness to two diversity metrics, but the encoder itself is chosen by hand.
axioms (5)
  • domain assumption For problems with self-reported solve rate in [0.5, 0.9], the majority-vote answer over G=8 solutions is correct (or a usable training label).
    Invoked in §2.2 (Eq. 1) and §2.3.2 (Eq. 6) to define correctness without external verification; never validated against ground truth — GPT-5 agrees with reference answers on only 70.82% of problems (Table 2).
  • domain assumption Base instruction models already contain enough mathematical knowledge to bootstrap open-ended exploration from the trivial seed '1+1'.
    §2.1 initializes the pool with 'What is 1+1?'; §4.1 shows expansion from arithmetic to calculus/optimization but also that step-0 problems are <50% valid, so bootstrapping depends on latent pretraining knowledge.
  • domain assumption Cosine distance in embedding space tracks mathematical-concept novelty.
    Eq. 4 defines diversity as min cosine distance to the problem pool; indirectly validated by Fig. 13 (low ROUGE-L/concept overlap) and Table 10 (metric robustness).
  • domain assumption Self-consistency (solve rate) is a stable estimate of problem difficulty for the evolving policy.
    Eq. 1 is used to calibrate difficulty; Table 4 shows high variance (sigma=17.37) when the teacher is frozen, which the paper interprets as unstable difficulty — the estimator is accepted without ground-truth difficulty labels.
  • standard math GRPO on-policy updates with KL penalty provide stable optimization.
    Eqs. 7-8 are standard RL machinery adopted without modification; no novelty is claimed here.

pith-pipeline@v1.3.0-alltime-deepseek · 23068 in / 21421 out tokens · 218852 ms · 2026-08-04T00:31:31.812501+00:00 · methodology

0 comments
read the original abstract

Recent advances in large language model (LLM) reasoning through reinforcement learning rely on annotated datasets for verifiable rewards, which may limit models' ability to surpass human-level performance. While self-play offers a promising alternative, prior methods yield only marginal or even negative gains on post-trained models because they generate problems that cluster around familiar concepts rather than discovering novel ones. We introduce Open-Ended Self-Improving Reasoner (OpenSIR), a self-play framework in which a single LLM alternates teacher and student roles to generate and solve novel problems without external verifiers or annotated data. Starting from a single seed problem, OpenSIR sustains open-ended exploration through diversity rewards that push the model toward unfamiliar concepts and difficulty calibration that keeps problems learnable. Across seven math benchmarks, OpenSIR consistently improves all models, averaging +3.6 points on instruction models and +3.1 on reasoning models, while recent self-play baselines yield marginal or even negative gains; starting from a single trivial seed, it also surpasses GRPO baselines trained on over 7K annotated examples. Despite training only on self-generated math, OpenSIR is the only self-play method that transfers to general reasoning, improving by at least +4.4 points on reasoning models.

Figures

Figures reproduced from arXiv: 2511.00602 by Aryo Pradipta Gema, Joshua Ong Jun Leang, Pasquale Minervini, Wai-Chung Kwan.

Figure 1
Figure 1. Figure 1: Overview of the OpenSIR framework. A single policy [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Evolution of problem difficulty, validity, and topic diversity during OpenSIR training. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: t-SNE visualization of problem embed￾dings showing the effect of diversity reward on problem distribution. With diversity reward, prob￾lems explore broader regions of the embedding space compared to the clustered distribution with￾out diversity reward. Model Acc # Concepts w diversity 38.42 5914 w/o diversity 36.45 3328 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: An invalid arithmetic question generated in step 0 with solve rate of 0.25. This question [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: An invalid arithmetic question generated in step 0 with solve rate of 0.125. This question [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: An invalid optimisation question generated in step 0 with solve rate of 0.125. This ques [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: An invalid optimisation question generated in step 0 with solve rate of 0.125. This ques [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: A valid arithmetic problem involving geometric series with a solve rate of 0.125. The [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: A valid optimisation problem with a solve rate of 0.375 generated at step 124. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: A valid calculus problem with a solve rate of 0.375 generated at step 156. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: A valid physics problem that involves trigonometry with a solve rate of 0.5 generated at [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: A valid probability problem with a solve rate of 0.25 generated at step 188. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Heatmap visualisation of n-gram similarity (ROUGE-L scores) and concept overlap be [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Progression of solve rates of OpenSIR and the initial instruction model as training goes. [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: A geometry problem from the MATH dataset, representing a different mathematical [PITH_FULL_IMAGE:figures/full_fig_p019_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: A competition-level problem from AIME 2024, significantly more challenging than the [PITH_FULL_IMAGE:figures/full_fig_p020_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: t-SNE visualisation of problem embeddings generated by OpenSIR from three different [PITH_FULL_IMAGE:figures/full_fig_p020_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Pass@k curves comparing base instruction models and OpenSIR across five mathemat [PITH_FULL_IMAGE:figures/full_fig_p023_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Performance of OpenSIR extended training using Llama-3.2-3B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p023_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: The instruction provided to the annotators to annotate problems. [PITH_FULL_IMAGE:figures/full_fig_p025_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Prompt for generating math problems. {Problem} is a placeholder for the reference [PITH_FULL_IMAGE:figures/full_fig_p027_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Prompt for generating solutions to math problems. {Problem} is a placeholder for the [PITH_FULL_IMAGE:figures/full_fig_p027_22.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play

    cs.AI 2026-05 unverdicted novelty 6.0

    PopuLoRA shows that co-evolving populations of LoRA adapters through cross-evaluated self-play can outperform compute-matched single-agent baselines on multiple code and math reasoning benchmarks.

Reference graph

Works this paper leans on

17 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [2]

    Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping

    doi: 10.1126/science.aay2400. Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. AceReason-Nemotron: Advancing Math and Code Reasoning through Rein- forcement Learning, May

    doi:10.1126/science.aay2400

  2. [6]

    Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, and Natasha Jaques

    URLhttps://proceedings.neurips.cc/paper_files/paper/2022/ hash/18abbeef8cfe9203fdf9053c9c4fe191-Abstract-Conference.html. Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, and Natasha Jaques. SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-T...

    2022

  3. [7]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y

    URLhttps://proceedings.neurips.cc/paper_files/paper/2024/ hash/7d0c6ff18f16797b92e77d7cc95b3c53-Abstract-Conference.html. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, April

    2024

  4. [8]

    Every morning Aya goes for a9-kilometer-long walk and stops at a coffee shop afterwards

    How many units are in the length of AD? Figure 15: A geometry problem from the MATH dataset, representing a different mathematical domain from the trivial arithmetic seed. Every morning Aya goes for a9-kilometer-long walk and stops at a coffee shop afterwards. When she walks at a constant speed ofskilometers per hour, the walk takes her 4 hours, including...

    2024

  5. [10]

    mlr.press/v235/tang24k.html

    URLhttps://proceedings. mlr.press/v235/tang24k.html. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    Pith/arXiv arXiv

  6. [11]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671,

    Qwen Team. Qwen2 technical report.arXiv preprint arXiv:2407.10671,

    Pith/arXiv arXiv

  7. [12]

    doi: 10.1038/s41586-019-1724-z

    ISSN 1476-4687. doi: 10.1038/s41586-019-1724-z. 13 Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning.https://github.com/huggingface/trl,

    doi:10.1038/s41586-019-1724-z

  8. [20]

    set_id":

    You will be presented with multiple sets of 5 math problems to evaluate. For each set, please complete the following three-step annotation process. # Step 1: Identify Topics Foreach problem, identify ALL relevant mathematical topics from the following list: 24 - Algebra - Geometry - Calculus - Probability - Statistics - Number Theory - Combinatorics - Opt...

    2023

  9. [44]

    Model Acc OpenSIR 38.42 OpenSIRMATH 38.67 OpenSIRAIME 38.81 Table 7: Performance of OpenSIR with different initial seed problem

    and their average with standard deviation as super- script. Model Acc OpenSIR 38.42 OpenSIRMATH 38.67 OpenSIRAIME 38.81 Table 7: Performance of OpenSIR with different initial seed problem. 22 A.6 OPENSIR INCENTIVISESREASONINGCAPACITY To verify whether OpenSIR elicits genuine reasoning improvements rather than memorisation, we evaluate pass@k performance o...

    2025

  10. [80]

    What is 1+1?

    then declines (0.58→0.32 at step 200), confirming the V-shaped difficulty pattern trend from Section 4.1: problems initially become easier as OpenSIR learns appropriate calibration, then progressively harder as it increases challenge. Crucially, this pattern shows that OpenSIR’s reasoning ability improved over training. A.5 SENSITIVITY TO THE INITIAL SEED...

    2024

  11. [2016]

    doi: 10.1038/nature16961

    ISSN 0028-0836, 1476-4687. doi: 10.1038/nature16961. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without hu...

    doi:10.1038/nature16961

  12. [2017]

    doi: 10.1038/nature24270

    ISSN 1476-4687. doi: 10.1038/nature24270. Yifan Sun, Jingyan Shen, Yibin Wang, Tianyu Chen, Zhendong Wang, Mingyuan Zhou, and Huan Zhang. Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty- targeted Online Data Selection and Rollout Replay, June

    doi:10.1038/nature24270

  13. [2018]

    Table 11: The training configurations for the experiments

    Warmup steps 20 Training steps 100/200 KL loss coefficient1×10 −4 Gradient norm clipping 0.5 Seeds 42/43/44 GPUs 3 H100 Rollout Batch size† 256 Max prompt length 1024 Max solution length 2048 Number of rollouts per prompt 8 Temperature 1.0 Teacher Rewards Solvability weight (α) 1.0 Solution length weight (λ) 1.0 Diversity weight (γ) 1.0 Format weight (δ) ...

    2048

  14. [2019]

    URLhttps://openreview.net/forum? id=SkxpxJBKwS. Jakob Bauer, Kate Baumli, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gre- gor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker- Holder, Shreya Pathak, Nicolas Perez-Nie...

    1935

  15. [2021]

    Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu

    URLhttps: //datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html. Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-Zero: Self-Evolving Reasoning LLM from Zero Data, August

    2021

  16. [2022]

    Alex Havrilla, Edward Hughes, Mikayel Samvelyan, and Jacob Abernethy

    doi: 10.1126/science.ade9097. Alex Havrilla, Edward Hughes, Mikayel Samvelyan, and Jacob Abernethy. Synthetic Problem Gen- eration for Reasoning via Quality-Diversity Algorithms, June

    doi:10.1126/science.ade9097

  17. [2024]

    doi: 10.18653/v1/2024.acl-long.211

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.211. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solv- ing With the MATH Dataset.Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1, December

    doi:10.18653/v1/2024.acl-long.211 2024