{"id":"5f58fb52-60ce-41e0-b624-a2808940472c","arxiv_id":"2511.00602","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A language model trained to both generate and solve its own math problems, rewarded for difficulty, diversity, and self-consistency, improves math benchmark scores without annotated data.","lead":"OpenSIR is a training method where a single language model writes its own math problems and then solves them, using its own agreement as the only feedback — no human-annotated data. On five math benchmarks it improved four instruction-tuned models, in several cases matching or beating models trained on more than 7,000 human-written problems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Majority-vote reference labels are never validated against ground truth; the paper's own GPT-5 check puts reference-answer agreement at only 70.82% (Table 2), so ~29% of student rewards may be rewarding wrong answers.","rationale":"The reader's weakest assumption — that majority-vote reference answers are reliable training labels — is exactly the most load-bearing point. The paper's own Table 2 quantifies the risk: at the chosen solve-rate threshold, GPT-5 agrees with the reference answer on only 70.82% of problems, which is far from a clean signal. Since the student reward (Eq. 6) is defined against this reference, a substantial fraction of training may reinforce wrong answers. This does not by itself falsify the paper's empirical gains, but it directly threatens the central claim of verifier-free, annotation-free self-improvement. The proposed expert-validation and filtered-retraining test would settle whether the gains are genuine reasoning improvements or an artifact of unreliable self-generated labels. The reader's CONDITIONAL verdict is appropriate; my concern reinforces the need for that condition without moving the verdict to ACCEPT or REJECT.","tokens_in":23552,"tokens_out":3299,"duration_ms":39959,"concrete_test":"Take 300–500 OpenSIR training problems whose solve rate fell in [0.5, 0.9] at the iteration they were used, and have two independent expert annotators produce final answers (with disagreement resolved by a third). Compute the majority-vote reference answer's accuracy against those ground-truth answers, stratified by solve rate. Then run OpenSIR to convergence with a filter that only admits problem–solution pairs whose majority answer matches the expert-verified answer, keeping all other hyperparameters and compute identical. If the verified-label run's gains over base disappear or shrink to within noise, the majority-vote assumption is load-bearing; if it retains the +3.6/+3.1 gains, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"At §2.2 Eq. 1, the reference answer a* is defined by majority voting over G solutions, and §2.3.2 Eq. 6 rewards every solution attempt that matches a*. The entire student training signal therefore inherits the correctness of a*. The paper never validates a* against ground truth for training problems. Its only independent check, Table 2, reports that GPT-5's majority answer agrees with the reference answer on 70.82% of problems sampled from the OpenSIR_0.5 run (the main configuration). If that estimate is even approximately right, roughly 30% of training labels are wrong, meaning correct solutions are actively penalized and the model is rewarded for consistently reproducing the majority error. This is load-bearing because the advertised result — annotation-free, verifier-free self-improvement — depends on majority-vote labels being reliable enough to train on. It also creates a feedback loop: as the policy drifts, future problem generation and majority labels can become systematically wrong while still achieving low loss. The comparison to GRPO does not rescue this, since GRPO baselines use human-verified labels while OpenSIR does not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"OpenSIR proposes a self-play reinforcement-learning framework in which a single instruction-tuned LLM alternates between a teacher role (generating math problems) and a student role (solving them). Problem generation is seeded from a growing pool of self-generated problems, and rewards are computed from solve-rate-based difficulty, solution length, embedding diversity, and format. Reference answers are obtained by majority voting over G sampled solutions, and the student reward is the agreement of each solution with that reference. Training uses a GRPO-style on-policy update. Starting from a single trivial seed, the paper reports consistent average gains on five math benchmarks across four model families, with ablations attributing the gains to difficulty calibration, diversity rewards, and joint teacher-student training. The central claim is that verifier-free, annotation-free self-improvement for LLM math reasoning is possible and can match or exceed GRPO baselines trained on thousands of human-annotated examples.","tokens_in":23736,"tokens_out":10860,"duration_ms":114805,"significance":"If the central claim holds, the result is significant: it would remove the reward-verification bottleneck that currently ties RLVR to curated, labeled datasets. The paper has notable strengths: per-seed results are reported in Tables 5-6, ablations isolate the contribution of each reward component, the seed-robustness study addresses a natural concern, and the pass@k analysis in A.6 speaks against a purely format-only gain. The method is also comparatively lightweight, with code and datasets released. However, the core claim rests on the reliability of self-generated majority-vote reference answers, and the manuscript does not establish that reliability; its own external check (Table 2) shows only 70.82% agreement with GPT-5 for the main configuration. The contribution is therefore promising but not yet proven to the standard required for the advertised verifier-free claim.","major_comments":[{"comment":"The entire student training signal is defined with respect to the reference answer a* = arg max_a count(a_j) from the policy's own G samples, and Eq. (6) rewards every solution that matches a*. The paper asserts that high solve rates indicate reliable answers, but no ground-truth validation of a* is provided. In Table 2, the column labeled 'vality' is actually measured by comparing GPT-5's majority answer to the reference answer, and for the main OpenSIR_0.5 configuration this agreement is 70.82%. If this estimate is even approximately correct, roughly 29% of training labels are wrong: correct solutions are penalized and the policy is rewarded for consistently reproducing majority errors. This is a load-bearing issue for the central claim of verifier-free, annotation-free self-improvement. The paper also conflates 'validity' (logical well-posedness, as defined in §4.1) with answer agreem","section":"§2.2 Eq. (1), §2.3.2 Eq. (6), Table 2"},{"comment":"The headline 'OpenSIR outperforms all GRPO baselines' is supported almost entirely by average differences, but with three seeds the evidence is often weak. For Llama-3.2-3B-Instruct, OpenSIR vs. GRPO_gsm8k is 38.42±0.27 vs. 37.83±0.37 (difference 0.59, overlapping standard deviations); for Qwen-2.5-3B-Instruct, 50.85±0.38 vs. 50.56±0.45 (difference 0.29). These are not statistically reliable differences. The claim should be restricted to configurations where the difference is robust (e.g., Gemma-2-2B-Instruct, where the gap is about 1.9 points) or should be backed by paired significance testing and effect sizes. As written, the abstract and §3.4 overstate the evidence for the comparison that is central to the paper's positioning.","section":"§3.4, Tables 5 and 6"},{"comment":"The paper's own extended-training analysis states that performance plateaus after step 350 and that generated problems become increasingly similar and repetitive, attributing this to a less effective diversity reward over longer horizons. This is an admitted bound on the open-endedness that the title, abstract ('sustains open-ended exploration'), and conclusion ('continuously discover and master increasingly challenging mathematical concepts') claim. The claims should be qualified to the observed 200-350 step horizon, or the diversity mechanism should be shown to be extendable. As written, the evidence directly supports self-improvement over a bounded curriculum, not sustained open-ended learning.","section":"A.7 and Abstract/Conclusion"}],"minor_comments":[{"comment":"The text says the solvability score 'decreases to 1/n at the boundaries,' but n is not defined and the formula gives 1/G at the boundaries. Replace '1/n' with '1/G'.","section":"Eq. (2)"},{"comment":"The abstract (as provided) claims 'seven math benchmarks' and reports gains on 'reasoning models' (+3.1 and +4.4), but the body evaluates five math benchmarks and does not report reasoning-model training results. Align the abstract with the experiments actually presented.","section":"Abstract vs. §3.2"},{"comment":"The column labeled 'Validity' is not problem validity as defined in §4.1 — it is agreement between GPT-5's majority answer and the reference answer. Rename it to 'Answer agreement with GPT-5' to avoid misleading readers. Similarly, 'Solve Rate' in this table is GPT-5's solve rate, not the model's solve rate in Eq. (1); clarify the column.","section":"Table 2"},{"comment":"The symbol q_i is overloaded. In line 4, q_{i,1:G} denotes a group of G problems generated from reference problem p_i; in line 7, the loop should iterate over individual generated problems q_{i,j} (or a renamed variable). The current notation makes the subsequent selection steps hard to follow and should be corrected for reproducibility.","section":"Algorithm 1, line 7"},{"comment":"The human evaluation uses only 20 problems per source and the annotators are the paper's authors. Report inter-annotator agreement and state the small sample as a limitation, or the difficulty-ranking conclusions in §4.1 should be treated as illustrative.","section":"Figure 2 / Appendix B"},{"comment":"In the sentence 'we denote s qi = SolveRate(q_i)', the subscript formatting is broken; it should be s_{q_i} or a similar consistent notation.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The central concern is evidential rather than logical: self-consistency is a legitimate inductive bias, but the paper does not show that the majority-vote labels are reliable enough to train on, and its own GPT-5 check suggests a substantial mismatch. The statistical strength of the headline comparison is also weaker than the text suggests. These issues are fixable within the manuscript's scope by additional validation and more cautious claims. I do not see grounds for rejection, but the advertised verifier-free, annotation-free result is not yet established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on self-play or data-free RL for reasoning. The core idea—one policy alternating teacher/student, with diversity and difficulty rewards—is a real extension of R-Zero and Absolute Zero, and the ablations give it teeth: the diversity reward is worth about 2 points, the length reward about 0.6, and the teacher-student co-training about 2.5. Per-seed results are included, the seed-sensitivity analysis is a nice touch, and the pass@k curves support genuine reasoning gains rather than just format overfitting. I believe the authors believe the results, and the experiments are honestly reported.\n\nThe soft spots are real, though. The biggest one is the reward label. Equation 6 rewards a solution for matching the majority vote, and Equation 1 defines that vote from the same policy. Nowhere is the reference answer checked against ground truth for training problems. The only independent check, Table 2, says GPT-5 agrees with the reference answer on 70.82% of problems from the main configuration. If that number is even close, roughly 30% of the training rewards are tied to wrong answers. That doesn't kill the direction—self-consistency can still be a useful shaping signal—but it does mean the headline claim \"verifier-free self-improvement\" is not actually established. The authors need to validate a sample of reference answers against gold labels and show the reward isn't just amplifying systematic errors.\n\nSecond, the abstract overclaims relative to the body: seven benchmarks become five in Table 1, and the \"transfer to general reasoning, improving by at least +4.4 points on reasoning models\" appears nowhere in the experiments. That's a framing problem, not a result problem, but it should be fixed before this is cited as evidence for general-reasoning transfer.\n\nThird, the comparison to GRPO is not compute-matched. OpenSIR runs 200 steps at 1.125x the per-step cost of the 100-step GRPO baselines, so it gets roughly 2.25x the training budget. The claim \"outperforms GRPO without human-annotated data\" is technically true from Table 1, but the compute difference weakens the comparison.\n\nNone of this is fatal. The math is sound, the ablations are thorough, and the limitations section even acknowledges saturation after step 350, which is honest. The paper deserves a serious referee. My recommendation: send it to peer review, but request ground-truth validation of the reference answers, a compute-matched GRPO baseline, and an abstract that matches the actual content.\n\nFor the reading group, I'd bring it—the circularity discussion is exactly what we should chew on.","headline":"OpenSIR is a genuine, well-ablated extension of verifier-free self-play, but its reward loop never validates majority-vote labels against ground truth and the abstract oversells what the body shows.","tokens_in":24413,"tokens_out":1858,"would_cite":true,"duration_ms":22725,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single language model switching between teacher and student roles, trained only on its own self-generated problems from one trivial seed, improves across seven math benchmarks and beats models trained on over 7,000 labeled examples.","keywords":["LLM reasoning","self-play","reinforcement learning","mathematical problem generation","open-ended learning","self-consistency","diversity reward","curriculum learning"],"falsifier":"Audit the training labels directly: take OpenSIR-generated problems that pass the [0.5, 0.9] solve-rate filter, obtain independent ground-truth solutions from human annotators or a much stronger solver, and compare them with the majority-vote references used in training. If agreement is at or below the 70.82% reported in the paper's external check, the correctness signal is too noisy to carry the claimed gains. A complementary test: run the same OpenSIR training but replace self-consistency labels with independent ground-truth labels and see whether the benchmark advantage persists.","tokens_in":23296,"feed_emoji":"🧮","tokens_out":9453,"duration_ms":98051,"temperature":0.7,"pith_summary":"OpenSIR sets out to show that a language model can improve its own mathematical reasoning without any human-written training problems and without an external answer verifier. The same policy alternates between teacher and student: it invents problems conditioned on a growing pool of its own earlier problems, then solves each new problem several times, using majority voting among its answers as the reference and the solve rate as a reliability signal. With a novelty reward made of solvability, solution length, embedding-based diversity, and format, plus a correctness reward for matching the majority answer, training starts from the single seed \"What is 1+1?\" and reports average gains of +3.6 points on instruction models and +3.1 on reasoning models. The claim matters because if it holds, the most expensive parts of reasoning RL—annotated datasets and verifiable rewards—are not necessary for continued self-improvement, and the same loop could be pointed at other domains.","feed_headline":"One seed problem, zero labels: AI teaches itself math","feed_subtitle":"A single model writes and solves its own problems, beating models trained on thousands of human-written examples.","key_machinery":"The load-bearing mechanism is the closed teacher-student loop inside one policy, driven by two reward functions. The teacher reward is a novelty score: a triangular solvability term that rewards problems whose self-solve rate is near 0.7 and assigns zero outside the [0.5, 0.9] band; a solution-length term that rewards longer multi-step solutions; a diversity term that measures cosine distance from all problems already in the pool; and a format term. The student reward is self-consistency: a solution earns credit if its parsed answer equals the majority-vote answer across G attempts, plus a format bonus. Both rewards are standardised into advantages and combined in a policy-gradient update wi","core_discovery":"OpenSIR's central claim is that verifier-free, annotation-free self-improvement is possible for LLM math reasoning when the same policy is trained in both roles. The teacher samples reference problems from a pool initialised to a single trivial problem and generates new problems, each required to name its mathematical concepts. The student produces G solution attempts per problem; the most frequent parsed answer becomes the reference answer, and the solve rate is its frequency. The teacher's novelty score is a weighted sum of a triangular solvability reward (peaking at an intermediate solve rate and zero outside the accepted band), a normalised solution-length reward, a diversity reward meas","pith_inferences":["Editorial inference: because the correctness reward trusts the model's own majority answer, the framework can only be as sound as that label. The paper's own external check of the 0.5-threshold variant found agreement with the reference answer on 70.82% of sampled problems; if that rate is representative, roughly 29% of retained training labels are wrong, and part of the reported gain could be for","Editorial inference: the difficulty-calibration idea—keep self-solve rate near a fixed target while maximising diversity—is domain-agnostic. In code, formal proof, or puzzle domains where self-consistency or executable checks can stand in for the majority vote, the same two-term novelty reward could yield open-ended self-improvement.","Editorial inference: the diversity reward as implemented is a global repulsion from the pool; the paper's own extended training shows a plateau after step 350, suggesting this mechanism eventually saturates. A richer diversity signal operating on discovered concepts or skills, rather than raw embeddings, is the natural next lever for unbounded exploration."],"forward_implications":["If the results hold, neither a human-annotated dataset nor an external verifier is required for reinforcement-learning-based math reasoning; training on self-generated problems is enough.","The gains are not an artifact of the starting point: switching from a trivial arithmetic seed to a geometry or a competition-level problem changes final accuracy by less than 0.5 points.","The additional compute for self-generated problems is modest: one extra forward pass per problem on top of G solution attempts, an overhead of about 12.5% at G=8.","The improvement appears to be genuine reasoning capacity rather than memorisation: pass@k curves improve at all sampling budgets and on benchmarks far from the training distribution.","The method can also be combined with annotated data: mixing self-generated problems with GSM8K raises average accuracy from 38.42 to 41.29, above either source alone."],"fun_headline_variants":["Self-play, zero labels: AI builds its own math curriculum","One seed, no data: AI outperforms 7K annotated examples","OpenSIR: AI teaches itself math, beating human-written sets","From trivial seed to general reasoning: AI self-improves without labels"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that the majority-vote answer produced by the model's own solutions is correct for the problems that pass the [0.5, 0.9] solve-rate filter; the paper's single external check of that filter agrees with the reference answer on only 70.82% of sampled problems.","fun_headline_variants_meta":{"raw":{"variants":["Self-play, zero labels: AI builds its own math curriculum","One seed, no data: AI outperforms 7K annotated examples","OpenSIR: AI teaches itself math, beating human-written sets","From trivial seed to general reasoning: AI self-improves without labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1754,"prompt_tokens":741,"completion_tokens":1013,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":938}},"tokens_in":485,"tokens_out":1013,"duration_ms":8999,"temperature":1.0,"reasoning_tokens":938,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T00:31:31.812501+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the training labels directly: take OpenSIR-generated problems that pass the [0.5, 0.9] solve-rate filter, obtain independent ground-truth solutions from human annotators or a much stronger solver, and compare them with the majority-vote references used in training. If agreement is at or below the 70.82% reported in the paper's external check, the correctness signal is too noisy to carry the claimed gains. A complementary test: run the same OpenSIR training but replace self-consistency labels with independent ground-truth labels and see whether the benchmark advantage persists.","supporting_citations":[],"review_version":1}