{"id":"915211d1-8358-40c0-bfba-bf89068cdeb6","arxiv_id":"2506.08989","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SwS uses failures during RL training to synthesize targeted math problems, improving reasoning accuracy on eight benchmarks.","lead":"This paper introduces a training pipeline that lets an LLM identify the math problems it consistently fails during reinforcement learning, then generates new practice problems targeting those weaknesses. In tests on eight reasoning benchmarks, the approach improved average accuracy by 10.0% for a 7B model and 7.7% for a 32B model over a same-dataset baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing random-synthetic control: the weakness-driven component of the synthesis pipeline is never isolated, so the reported gains may reflect generic synthetic augmentation.","rationale":"The reader's verdict is CONDITIONAL, and I agree it should be. My load-bearing concern differs from the reader's explicit weakest assumption. The reader highlights reference-answer correctness; that is a genuine risk, but the paper offers indirect evidence (Table 5, weak-to-strong labeling accuracy rising from 80.6% to 97.5% after filtering, plus use of QwQ-32B for the 32B experiments) that mislabeling is at least partially controlled. The more central gap is experimental: the design never isolates the 'weakness-driven' part of the synthesis pipeline. The strongest claim says the framework enables robust generalization by empowering the model to self-identify and address its weaknesses. To credit self-identification, one must compare against random or non-failure-derived synthesis. Section 4.3 does this for retrieval-based selection, not for generation. Because the paper's contribution is specifically problem synthesis (Section 2.4), the missing control is load-bearing. If the concern lands, the correct conclusion is not that synthetic augmentation is useless, but that the evidence does not support the weakness-driven mechanism as the cause of the gains; the paper would then need to claim only that concept-based synthetic augmentation improves RLVR. This remains consistent with the reader's CONDITIONAL verdict, so I keep the verdict unchanged. Agreement with the reader is partial: they noted the data-quantity confound in their rationale, but their explicit weakest assumption was answer-label correctness.","tokens_in":29970,"tokens_out":5375,"duration_ms":56000,"concrete_test":"Re-run the full SwS pipeline for Qwen2.5-7B with a 40k budget, replacing only the concept-sampling source: sample concept combinations uniformly from all MATH-12k problems (or from the non-failed subset) instead of from the failure set, keeping generation prompts, quality verification, answer labeling, and difficulty filtering identical. Compare final average accuracy on the eight benchmarks against SwS-7B and BaseRL-7B. If the random-concept synthetic set matches SwS-7B within run-to-run noise, the weakness-driven synthesis claim is unsupported; if it is clearly worse, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result (Table 1) compares SwS (initial set plus 40k synthetic problems generated from failure-case concepts) with BaseRL (initial set only, same training steps). This establishes that adding synthetic problems helps, but it does not establish that generating them from self-identified weaknesses is what helps. The only head-to-head weakness-vs-random comparison (Section 4.3, Figure 5) retrieves problems from Big-Math via KNN; it does not use the concept-extraction/recombination problem-generation pipeline of Section 2.4. Therefore the core mechanism advertised in the title and abstract—weakness-driven problem synthesis—is not tested against the natural alternative of synthesizing the same number of problems from randomly chosen (or non-failed) concepts. Since all synthetic problems already pass quality filtering, difficulty filtering, and 50% self-consistency answer labeling, a random-concept set could plausibly yield similar or even larger gains, making the 'self-aware weakness' component unsubstantiated. The reference-answer concern raised by the reader is real but partly mitigated by the difficulty-filtering analysis in Section 4.1/Table 5 and by using a strong labeler; the missing random-generation control is unaddressed anywhere.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SwS, a self-aware weakness-driven problem synthesis framework for RLVR training of LLM reasoners. After a preliminary RL phase, problems the model consistently fails (max epoch accuracy below 50% with a negative accuracy trend) are treated as weaknesses. Concepts are extracted from these failure cases, recombined within categories, and used to prompt an instruction model to generate new problems. Generated problems pass quality filtering, reference-answer labeling via a strong reasoning model with a 50% self-consistency threshold, and difficulty filtering to a 25%–75% initial-model accuracy range. The retained synthetic problems (40k per model) are added to the initial training set for a second RL stage. Experiments on Qwen2.5 3B/7B/32B across eight benchmarks report consistent gains over a BaseRL baseline trained on the initial set, with average improvements of 10.0% and 7.7% on 7B and 32B. The paper also explores weak-to-strong generalization, self-evolving synthesis, and weakness-driven selection from Big-Math.","tokens_in":30187,"tokens_out":5077,"duration_ms":53062,"significance":"If the central mechanism is validated, the contribution is valuable: it offers a data-augmentation strategy for RLVR that does not require human-labeled synthetic answers and that targets the model's own failure regions, with results across model scales. The paper's strengths include a thorough evaluation across eight benchmarks, a transparent data-workflow diagram (Figure 8), a concrete quality-filtering case study (Table 4), difficulty ablations (Section 4.4), and public code/links. The main claim, however, requires a matched random-synthetic control of the generation pipeline; currently the evidence supports 'adding synthetic problems helps' more strongly than 'adding weakness-driven synthetic problems helps.' The reference-answer reliability issue is real and only partially mitigated by the difficulty filter. For these reasons the result is plausible but not yet established at the level of the paper's headline claim.","major_comments":[{"comment":"The headline comparison conflates two variables: dataset size and weakness targeting. SwS trains on the initial set plus 40k synthetic problems, while BaseRL trains on the initial set alone, so the reported gains (e.g., +10.0% for 7B) may be attributable to additional training data volume rather than to the weakness-driven selection mechanism. The only weakness-vs-random comparison in the paper, Section 4.3 and Figure 5, uses retrieval from Big-Math via KNN, not the concept-extraction/recombination generation pipeline of Section 2.4. A control that augments the initial set with 40k problems generated from randomly sampled concepts, and processed through identical quality, answer-labeling, and difficulty filtering, is needed to isolate the weakness-driven component. Without it, the paper's title claim is not directly tested.","section":"Section 3.1, Table 1; Section 4.3, Figure 5"},{"comment":"The RL training signal depends on reference answers produced by a single external reasoning model with a 50% self-consistency threshold (Skywork-OR1-Math-7B for small models, QwQ-32B for 32B), and these answers are never checked against known ground truth. If a non-trivial fraction of retained problems has incorrect labels, the RL objective is corrupted and the reported gains may partly reflect fitting to labeler bias. The weak-to-strong analysis in Table 5 is informative, but it validates labels only on MATH-500 where the true answers are known, not on the generated synthetic problems. Please report a label-accuracy estimate on a human- or independently verified sample of the synthetic set, and show sensitivity to the 50% consistency threshold.","section":"Section 2.4, 'Reference Answer Generation'"},{"comment":"All main results appear to be single runs without variance estimates. Several benchmark deltas are driven by very small numbers of problems: AIME24 and AIME25 each contain 30 problems, so a 16.7-point difference corresponds to five correct responses, and the Avg@1 numbers fluctuate accordingly. Since the headline average gains are the central empirical claim, at least the aggregate numbers and the competition-level subset should be accompanied by multiple seeds or confidence intervals to establish that the differences are not sampling noise.","section":"Table 1"},{"comment":"The random augmentation baseline in the weakness-driven selection experiment is under-specified. The text says the baseline uses 'randomly selected problems from Big-Math,' but it does not state whether the random subset is matched to the weakness-driven subset in size, category distribution, and quality filtering. If the random set differs in any of these dimensions, Figure 5 cannot cleanly attribute the observed gap to weakness-driven selection. Please specify the exact construction of the random baseline, including category budgets and filtering steps.","section":"Section 4.3 and Appendix H"}],"minor_comments":[{"comment":"The 'slope' of the accuracy sequence is not defined; please specify whether it is the slope of a linear fit over epochs or a finite difference between the first and last epochs, and state the number of epochs used for the trend test.","section":"Equation (4)"},{"comment":"The abstract states that gains are achieved 'without relying on external knowledge distillation,' but the pipeline uses LLaMA-3.3-70B-Instruct for generation and quality assessment and QwQ-32B/Skywork-OR1 for answer labeling. This is not necessarily distillation, but the phrasing should be clarified to distinguish 'no distillation of teacher solutions' from 'no external models at all.'","section":"Abstract and Section 3.1"},{"comment":"Line 3 of Algorithm 1 refers to 'Eq. 2' for the selection budget, but the budget allocation is given by Eq. (5) in the main text. Please correct the cross-reference.","section":"Appendix H, Algorithm 1"},{"comment":"The table header 'Avg.' is not defined; clarify whether it is an unweighted average over the eight benchmark columns, and how the Avg@1 and Avg@32 AIME values are combined. The text uses 'average absolute improvement' without specifying the exact aggregation rule.","section":"Table 1 and Section 4.1"},{"comment":"The self-evolving result shows negative deltas on MATH-500 and AIME; the attribution of this decline to the initial DAPO/LightR1 data being 'already specifically tailored' to those benchmarks would be stronger with a quantitative comparison of the initial data's overlap with the evaluation sets.","section":"Section 4.2, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and is likely to be of interest to the RL/LLM community. The decisive missing experiment is a random-concept synthetic-generation control through the full Section 2.4 pipeline; given that the pipeline already exists, this is an expensive but feasible addition. There is no indication of a fundamental flaw that would force rejection, but the headline claim as currently worded is not supported by the main-table comparison alone. No concerns about citation practices or novelty disclosure beyond the standard need to position against MathScale, KP-Math, and PromptCoT, which are already cited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a genuinely new idea and a genuine missing control, and the title claim outruns the experiments. The new piece is the weakness signal—problems where the model's rolling accuracy never reaches 50% and trends down during preliminary RL. That is a cheap, sensible way to find what a model can't learn, and I don't know prior work using the learning trajectory itself this way. The rest (concept extraction, recombination, self-consistency answers, difficulty filtering) is assembled from KP-Math, MathScale, and PromptCoT, and the authors mostly say so.\n\nWhat I like: the documentation. The data-workflow figure shows the funnel from 1M generated candidates to 41k retained, the difficulty ablation in 4.4 is clean, and the Weak-to-Strong analysis in Table 5 is the best evidence in the paper: combining ≥50% self-consistency with student-accuracy filtering raises labeler accuracy from 80.6% to 97.5% where truth is known. The limitations section is honest, including the admission that self-evolved problems come out too easy.\n\nSoft spots, in proportion. The headline comparison (Table 1) pits SwS (original plus 40k synthetic) against BaseRL (original only, same steps). That shows extra data helps training; it does not show weakness-driven data is what helps. The stress-test is right that the only weakness-vs-random head-to-head is the Big-Math retrieval experiment (Section 4.3), which tests selection, not the concept-recombination synthesis pipeline. A control synthesizing the same 40k from randomly drawn concepts, through the same quality and difficulty filters, is the experiment the title needs, and it is missing. Also: single runs, no variance. The +33.3 on AIME25 for 32B is four questions out of thirty. And 'without external knowledge distillation' is doing work: generation, verification, and answer labeling all use external models; the pipeline just doesn't distill solutions.\n\nThe label-quality worry the reader raised is real but mostly defused by the difficulty filtering analysis; I'd rank it below the missing control. Net: the weakness-identification signal is worth keeping, the mechanism is plausible, and the retrieval experiment gives it indirect support, but the synthesis-side advantage is unestablished as submitted. It still deserves a serious referee; the fix is concrete—add the random-concept control, run seeds, audit a sample of synthetic answers against ground truth. People building RLVR data pipelines should read it now; just don't cite the 10%/7.7% as settled numbers.","headline":"Useful weakness-signal idea, well-documented pipeline, but the main experiments never isolate weakness-driven synthesis from generic data augmentation, so the headline claim is conditional pending the missing random-synthetic control.","tokens_in":30749,"tokens_out":4929,"would_cite":true,"duration_ms":47374,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement-trained math models improve by up to 10 points when they synthesize practice problems from their own persistent failures.","keywords":["reinforcement learning with verifiable rewards","problem synthesis","weakness identification","mathematical reasoning","large language models","data augmentation","self-improvement","GRPO"],"falsifier":"Audit a random sample of the retained synthetic problems from the 32B run (about 41.7 thousand after difficulty filtering) by computing exact reference answers independently of the labeling model; if more than 5% of the sampled retained answers are wrong, the RL training signal is contaminated and the reported gains would need to be reinterpreted as partially fitting to label noise.","tokens_in":29760,"feed_emoji":"🎯","tokens_out":9009,"duration_ms":79419,"temperature":0.7,"pith_summary":"SwS asks a simple question: can a math model's own learning failures during reinforcement training be turned into better training data? The paper proposes a loop where the model first trains briefly on a standard problem set while its per-question accuracy is tracked, then flags the questions it consistently fails (never reaching 50% accuracy with a downward trend), extracts the mathematical concepts behind those failures, and generates new questions that recombine those concepts. The new questions are filtered for quality, answer consistency, and difficulty before the model continues training on the augmented set. Across eight reasoning benchmarks this yields average absolute gains of 10.0 points for the 7B model and 7.7 points for the 32B model over continued training on the original problems alone, and it also lets the 7B model solve up to 20 percentage points more of its previously intractable problems in its weakest domains. The claim matters because it addresses a practical bottleneck for verifiable-reward RL: well-annotated human math problems are scarce, and indiscriminate synthetic expansion wastes training signal.","feed_headline":"Weakness-driven problem synthesis lifts RL math scores by 10%.","feed_subtitle":"The model mines its own math failures to build targeted practice sets, beating training on curated data.","key_machinery":"The central machinery is the failure-case definition of Eq. (4) — a problem is a weakness only if its per-epoch accuracy never reaches 50% and the accuracy trend has negative slope — feeding a generation-and-filtering pipeline of concept extraction and recombination (with co-occurrence and embedding similarity sampling), failure-rate-proportional budget allocation (Eq. 5), instruction-model quality verification, self-consistency answer labeling with a 50% consistency threshold, and difficulty filtering to the $[25\\%,75\\%]$ accuracy window. This pipeline converts the model's own training log into targeted practice problems that stay learnable and keep producing nonzero gradient signal under GRPO-style group-relative advantages.","core_discovery":"The paper's central claim is that the model's own RL training dynamics reveal which reasoning skills it lacks, and that those deficits can be converted directly into targeted synthetic practice. A problem counts as a weakness if the model never exceeds 50% per-epoch accuracy on it and its accuracy trend slopes downward (Eq. 4). The framework extracts core concepts from such failure cases, recombines them into new questions with category budgets proportional to failure rates (Eq. 5), and retains only questions that pass model-based quality checks, a 50% self-consistency answer label from a strong reasoning model, and a difficulty band of roughly $[25\\%,75\\%]$ accuracy on the model being trained. Continuing RL on this augmented set beats continued training on the original data, surpasses models trained on curated human-labeled and distillation-oriented sets in most comparisons, and shows the largest recovery precisely in the domains flagged as weakest.","pith_inferences":["If the central claim holds, the same weakness-identification signal should transfer beyond exact-answer math to settings where a reward model supplies feedback instead of a verifiable answer; the paper names this as an open direction, and the natural test is whether failure-rate trends under a reward model identify the same kinds of persistent deficits.","The paper ablates difficulty but not the failure-rate-proportional budget allocation of Eq. 5, so a controlled comparison against uniform per-category budgets would tell whether weakness-relative allocation adds anything beyond the synthesis and filtering pipeline itself.","The Self-evolving extension (+1.2% on a 14B model) is consistent with the interpretation that the main experimental gains come partly from the strength of the external generation and labeling models (LLaMA-3.3-70B-Instruct and QwQ-32B), not only from the weakness-targeting idea; a decisive test would use weak teachers for generation while keeping labeling fixed, or vice versa.","Because the baseline continues training on the original set for the same number of steps, the headline comparison already controls for compute; a matched-difficulty random-synthetic set would isolate how much of the gain comes from difficulty curation versus weakness targeting."],"forward_implications":["Continued RL on the weakness-driven augmented set recovers up to 20 percentage points more of the 7B model's consistently failed problems in its weakest domains (Intermediate Algebra, Geometry, Precalculus) than continued training on the original set, while each original problem is sampled four times less often.","The pipeline transfers across scales from 3B to 32B and across both initial problem sets (MATH-12k for smaller models and the DAPO/LightR1 combined set for 14B and 32B), with mean absolute gains of 10.0% (7B) and 7.7% (32B) on eight benchmarks.","SwS-trained models surpass several recent RL-trained baselines that rely on carefully curated or human-labeled problem sets, not just the base and instruct models.","Difficulty filtering is an active ingredient: simple synthetic problems are mastered quickly and then stop producing learning signal, while medium and hard retained problems give slower convergence but more sustained gains on evaluation benchmarks.","Weakness-driven selection of problems from a large external dataset (Big-Math) outperforms random selection on both overall and competition-level accuracy, and targeted problems remain challenging longer during GRPO training."],"supporting_citations":[{"why":"GRPO is the group-relative RL algorithm whose token advantages collapse when all responses are correct or all incorrect; SwS uses its objective as the base for Eq. 3 and difficulty filtering.","marker":"[40]"},{"why":"DAPO supplies the clip-higher token-level loss, dynamic sampling, and the accuracy thresholds (10% to 90%) used in the RLVR training recipe.","marker":"[63]"},{"why":"KP-Math provides the key-point (concept) extraction and recombination approach that SwS adapts for concept-based problem synthesis.","marker":"[15]"},{"why":"PromptCoT supplies the concept-based competition-level generation protocol and the quality-evaluation criteria (format, factual accuracy, difficulty, concept coverage, solvability) that SwS reuses.","marker":"[73]"},{"why":"QwQ-32B is the strong reasoning model that labels reference answers for the 32B experiments through the 50% self-consistency paradigm.","marker":"[47]"},{"why":"Skywork-OR1-Math-7B labels reference answers for the models up to 7B in size.","marker":"[12]"},{"why":"LLaMA-3.3-70B-Instruct performs concept extraction, problem generation, and quality verification; LLaMA-3.1-8B-base provides concept embeddings.","marker":"[8]"},{"why":"MATH-12k is the initial problem set on which weaknesses are identified for models up to 7B.","marker":"[13]"},{"why":"Big-Math is the external dataset used in the weakness-driven selection extension that competes with random augmentation.","marker":"[1]"},{"why":"PRIME serves as a baseline model and contributes one of the two verifiers combined into the hybrid rule-based answer checker.","marker":"[6]"}],"fun_headline_variants":["Self-aware RL: model mines its own failures to boost reasoning","LLMs train on their own weak spots, lifting reasoning scores by 10%","Self-identified gaps become targeted practice: RL math gains 10%","Mining own errors boosts LLM reasoning by 10% on benchmarks","Model turns weaknesses into training data, lifting reasoning scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the self-consistency-labeled answers on the retained synthetic problems are correct often enough to serve as clean RL rewards, even though those labels come from a single reasoning model and are never checked against known ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Self-aware RL: model mines its own failures to boost reasoning","LLMs train on their own weak spots, lifting reasoning scores by 10%","Self-identified gaps become targeted practice: RL math gains 10%","Mining own errors boosts LLM reasoning by 10% on benchmarks","Model turns weaknesses into training data, lifting reasoning scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000732,"raw_usage":{"total_tokens":3285,"prompt_tokens":966,"completion_tokens":2319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":2227}},"tokens_in":582,"tokens_out":2319,"duration_ms":18542,"temperature":1.0,"reasoning_tokens":2227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:56:30.357895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a random sample of the retained synthetic problems from the 32B run (about 41.7 thousand after difficulty filtering) by computing exact reference answers independently of the labeling model; if more than 5% of the sampled retained answers are wrong, the RL training signal is contaminated and the reported gains would need to be reinterpreted as partially fitting to label noise.","supporting_citations":[],"review_version":1}