{"id":"4e460aa5-c6f0-4c38-a135-57976c18d226","arxiv_id":"2508.01245","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"A defect-aware, multi-expert data synthesis and progressive training framework improves LLM performance on six mathematical benchmarks by 12.57% on average, according to the abstract.","lead":"WarriorMath uses several large language models to write, critique, and rewrite math problems that weaker models get wrong, then trains a model on these progressively harder problems. The authors report an average 12.57% accuracy gain over strong baselines on six math benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 12.57% average improvement is not interpretable unless the synthesized training data is audited for overlap with the six evaluation benchmarks.","rationale":"The paper is abstract-only, so the reader's UNVERDICTED status is appropriate. The single most load-bearing assumption is that the generated defect-aware data does not overlap with the six benchmarks. This is not merely a reproducibility detail; it is the foundation of the causal claim that the framework improves mathematical ability. The proposed concrete test directly targets this assumption by quantifying overlap and measuring the sensitivity of the headline result to removing contaminated examples. I agree with the reader that this premise is unstated and unverifiable from the abstract. No other concern is as central: baselines, error bars, and compute budgets are also missing, but they would not invalidate the core novelty as directly as leakage does. Therefore the verdict should remain unchanged: the evidence is insufficient to assess correctness, and the contamination audit is the decisive next step.","tokens_in":710,"tokens_out":2629,"duration_ms":31325,"concrete_test":"Perform a systematic contamination audit: for each of the six evaluation benchmarks, compute exact-match and semantic-similarity (e.g., ROUGE-L, embedding cosine) between every benchmark question and every question in the WarriorMath training set. Any pair with ROUGE-L > 0.8 or cosine > 0.95 should be flagged as a near-duplicate. Then rerun the final model evaluation after removing all flagged training examples and retraining from scratch; if the average gain drops materially (e.g., by more than the 12.57% margin), the original claim is invalid. Additionally, evaluate the trained model on a held-out set of newly authored math problems that were not present in any pretraining or generation corpus.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on an unstated and untested premise: the defect-aware questions generated by WarriorMath are sufficiently disjoint from the six benchmark test sets that performance gains reflect improved mathematical reasoning rather than memorization. Because the synthesis stage uses LLMs to generate, critique, and refine problems, and these LLMs are trained on web-scale corpora that include public math benchmarks, the generation process can easily produce near-duplicates of evaluation items either directly or through paraphrase. The progressive training stage then selects the hardest examples for the base model, which could further concentrate training on benchmark-like content. If even a small fraction of the training data leaks from or closely resembles the test sets, the 12.57% average improvement and the claim of a new state-of-the-art are largely explained by data contamination. This concern is load-bearing because the paper's novelty is the defect-aware, multi-expert synthesis pipeline; without a contamination check, the framework's contribution to genuine mathematical ability is not established. No full text, code, or data is available in this abstract-only review, so the premise cannot be verified from the submitted material.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes WarriorMath, a defect-aware framework for improving LLM mathematical problem solving. The synthesis stage uses multiple expert LLMs to generate, critique, and refine problems that base LLMs fail, while the training stage progressively fine-tunes the model on increasingly challenging data. The abstract reports that on six mathematical benchmarks, WarriorMath outperforms strong baselines by 12.57% on average and claims a new state-of-the-art. Only the abstract was available for review, so the assessment is based on the claims and descriptions in that abstract.","tokens_in":894,"tokens_out":2487,"duration_ms":30557,"significance":"If the reported results are reproducible and free of contamination, WarriorMath would offer a useful contribution by directly targeting LLM failure modes rather than merely increasing data volume. The multi-expert critique-and-refine loop is a plausible and well-motivated mechanism for generating high-quality training data. However, the abstract alone provides no experimental details, no error bars, no benchmark or baseline specifications, no ablation results, and no contamination analysis. The central empirical claim of a 12.57% average improvement is therefore currently unverifiable. The paper would be substantially strengthened by releasing the synthetic data and code, and by including an overlap audit between generated training data and evaluation benchmarks.","major_comments":[{"comment":"The headline result, 'outperforms strong baselines by 12.57% on average,' is presented without error bars, number of runs, or any statistical significance measure. Since the evaluation spans six benchmarks with likely different scales, the aggregation method (e.g., simple average versus weighted) is undefined. Without this information, the reported improvement is not interpretable and cannot be independently verified.","section":"Abstract"},{"comment":"The defect-aware generation pipeline uses expert LLMs trained on web-scale corpora, which very likely include public math benchmarks that overlap with the six evaluation benchmarks. The abstract does not report any n-gram, embedding, or human-level overlap audit between generated training questions and benchmark test sets. Because the progressive training stage selects questions that the base model fails—which are often benchmark-like hard problems—any test-set leakage would be amplified. Without a contamination check, the 12.57% gain cannot be attributed to improved mathematical reasoning rather than memorization.","section":"Abstract, synthesis stage"},{"comment":"The progressive learning framework is described only qualitatively as 'iteratively fine-tunes the model using increasingly challenging data tailored to its weaknesses.' The abstract omits the number of iterations, the precise difficulty metric, the fine-tuning protocol (e.g., learning rates, batch sizes, number of epochs), and the base model architecture. These details are necessary to evaluate the framework's novelty, reproducibility, and incremental contribution over standard curriculum learning.","section":"Abstract, training stage"},{"comment":"The six mathematical benchmarks are not named, and the 'strong baselines' are not specified. The evaluation metric and prompting protocol (e.g., chain-of-thought, tool use, answer extraction) are also absent. As a result, the claim of a 'new state-of-the-art' is unfalsifiable from the submitted material.","section":"Abstract, benchmarks and baselines"}],"minor_comments":[{"comment":"The term 'defect-aware' is used informally; please provide a formal definition, such as the exact criterion for identifying a defect or failure mode.","section":"Abstract"},{"comment":"The phrase 'questions that base LLMs fail to solve' lacks a failure criterion; specify, for example, whether failure means an incorrect final answer, a flawed reasoning chain, or a low confidence score.","section":"Abstract, synthesis stage"},{"comment":"Please include a reproducibility statement with data and code availability, since the contribution is an empirical pipeline whose value depends on the synthetic data and training code being accessible.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"Given that only the abstract was provided, no full-text review was possible. The central empirical claim is unsupported by the abstract alone, and the contamination concern is load-bearing. I recommend major revision because the issues are fixable in a full submission with proper experimental reporting and a data-leakage audit, rather than being inherently fatal to the approach."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read, based only on the abstract. The idea is actually decent: instead of blindly rephrasing or cranking up difficulty, generate problems that the base model currently fails, use multiple experts to critique and refine them, then train progressively on the harder failures. That is a plausible way to spend synthetic data budget, and the multi-expert critique step is a nice addition over the usual single-generator pipeline. The writing is clear about what the framework does, which counts for something.\n\nThe soft spot is exactly the one the stress test flags. The headline 12.57% average gain over strong baselines is an assertion, not evidence. There are no error bars, no baseline names, no benchmark details, and no code or data. More importantly, the contamination risk is load-bearing. The synthesis stage uses LLMs that were trained on web-scale corpora, and public math benchmarks are all over those corpora. If the generate-critique-refine loop produces near-duplicates of evaluation items, the progressive training stage could be concentrating on exactly the content that later shows up on the test sets. Without an overlap analysis between the synthetic training data and the six evaluation benchmarks, the 12.57% number is uninterpretable. That is not a minor omission; it is central to the claim.\n\nI want to be fair though. Abstract-only reviews are inherently limited. The lack of detail is normal for an abstract, and the leakage concern is a concern, not a demonstrated flaw. The real question is whether the full paper ships with the necessary checks. If the authors provide code, data, and a contamination audit, the framework could be a useful contribution to synthetic data generation for math LLMs. If they don't, the result is just another unverifiable SOTA claim.\n\nMy verdict: the paper deserves a serious referee, because the method is coherent, the combination is new enough, and the empirical claim is strong enough that checking it is worthwhile. But I would want the referee to push hard on the leakage issue and on error bars. I would not cite this yet, since the evidence is too thin. I might bring it to reading group mainly to discuss the contamination methodology, which is a useful cautionary example.","headline":"The framework is coherent and worth a look, but the 12.57% claim is a single unverified number that needs a contamination audit before it can be taken seriously.","tokens_in":1446,"tokens_out":1309,"would_cite":false,"duration_ms":19076,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A defect-aware framework has LLMs generate problems they fail to solve, then trains on those weaknesses, claiming a 12.57% average gain over strong baselines on six math benchmarks.","keywords":["defect-aware","data synthesis","progressive training","mathematical reasoning","large language models","multi-expert collaboration","fine-tuning"],"falsifier":"A concrete check would be to train WarriorMath on problems that are deliberately disjoint from the evaluation benchmarks in style and topic, then measure whether the reported 12.57% average improvement persists; if the gain largely disappears on such held-out benchmarks, the central claim of improved mathematical ability would be undermined.","tokens_in":514,"feed_emoji":"🧮","tokens_out":1270,"duration_ms":16847,"temperature":0.7,"pith_summary":"WarriorMath tries to show that large language models can become substantially better at mathematical problem solving if the training data is built around the model's own failure modes rather than around rephrased or progressively harder problems. The authors propose a defect-aware synthesis stage where multiple expert LLMs generate, critique, and refine problems, focusing on questions the base model cannot solve. They then add a progressive training stage that fine-tunes the model on increasingly challenging data matched to its weaknesses. The paper claims this approach outperforms strong baselines by 12.57% on average across six mathematical benchmarks, setting a new state-of-the-art. A sympathetic reader would take the core claim to be that targeting defects directly, instead of generic augmentation, yields measurable gains in mathematical ability.","feed_headline":"LLMs learn math from problems they fail","feed_subtitle":"WarriorMath generates, critiques, and refines questions that expose a model's weaknesses, then trains on them, claiming a 12.57% average…","key_machinery":"The central mechanism is the defect-aware data synthesis loop: expert LLMs generate candidate problems, another expert critiques them, and refinement produces questions that expose and target the base model's specific mistakes. These questions are then used in a progressive training curriculum that escalates difficulty according to the model's weaknesses. The machinery works by converting the model's failures into training signal, so each fine-tuning round is aimed at the gap between current ability and required competence.","core_discovery":"The paper's central claim is that LLM mathematical ability improves most when training data is generated adversarially to the model's current deficiencies. WarriorMath's synthesis stage uses multiple expert LLMs in a collaborative generate-critique-refine loop, and only questions that the base LLM fails to solve are kept and iteratively improved through expert feedback. The training stage then uses progressive fine-tuning, feeding the model increasingly challenging defect-aware data. The authors report that this defect-aware, multi-expert framework outperforms strong baselines by 12.57% on average over six mathematical benchmarks, which they describe as a new state-of-the-art.","pith_inferences":["A testable extension would be to evaluate WarriorMath on held-out benchmarks that are temporally and topically disjoint from the training data, to confirm that the gains are from reasoning rather than from overfitting to benchmark-style problem distributions.","The defect-aware principle could be combined with process supervision: critiques generated during synthesis might double as step-by-step rationales, further boosting the training signal beyond just the final answer.","The progressive training schedule implies that the order of training data matters beyond its content, which suggests that curriculum design for LLMs may be as important as data volume.","One implicit consequence is that the same defect-aware loop could be run iteratively after each fine-tuning round, creating a self-improving cycle where the model's new failures become the next training batch."],"forward_implications":["If the claimed 12.57% average improvement holds, defect-aware synthesis could become a standard component of LLM math training pipelines, reducing reliance on rephrasing-based augmentation.","Progressive training on defect-matched data may let smaller or weaker base models reach performance levels comparable to larger models on mathematical benchmarks.","The generate-critique-refine loop could be adapted to other reasoning domains where failure modes are identifiable, such as code generation or logical deduction.","The framework suggests that the quality of synthetic data is less about raw diversity and more about whether it targets the model's current limitations."],"supporting_citations":[],"fun_headline_variants":["Math LLMs level up by training on their failures","Defect-aware training sharpens LLM math skills","LLMs learn math by solving what they get wrong","WarriorMath boosts LLM math by 12.6% on average","Adversarial math problems: key to better LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the generated defect-aware problems are not leaked from or too similar to the six evaluation benchmarks, so that the measured gains reflect genuine mathematical reasoning rather than memorization of test-like questions.","fun_headline_variants_meta":{"raw":{"variants":["Math LLMs level up by training on their failures","Defect-aware training sharpens LLM math skills","LLMs learn math by solving what they get wrong","WarriorMath boosts LLM math by 12.6% on average","Adversarial math problems: key to better LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000556,"raw_usage":{"total_tokens":2613,"prompt_tokens":880,"completion_tokens":1733,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":1651}},"tokens_in":496,"tokens_out":1733,"duration_ms":16316,"temperature":1.0,"reasoning_tokens":1651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:42:37.948699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to train WarriorMath on problems that are deliberately disjoint from the evaluation benchmarks in style and topic, then measure whether the reported 12.57% average improvement persists; if the gain largely disappears on such held-out benchmarks, the central claim of improved mathematical ability would be undermined.","supporting_citations":[],"review_version":1}