{"id":"9f3b58aa-d16e-4e5e-b653-1bb7efaa5cc5","arxiv_id":"2505.22591","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Self-Error-Instruct clusters a model's math errors into types, synthesizes targeted practice data per type, and selects the best samples for fine-tuning, improving held-out math test accuracy.","lead":"This paper trains AI math models using their own mistakes: it groups wrong answers into error types, asks GPT-4o to write new practice problems for each type, then fine-tunes the model on the best examples. The approach improved math accuracy on several tests, especially for Qwen2.5-Math-7B.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Out-of-domain gains may be inflated by test-set leakage: synthetic data are filtered only against GSM8K/MATH, not against the OOD benchmarks used for evaluation.","rationale":"The reader correctly identified the 100-case dev set as a fragile component, but the dev set supports only the secondary claim that one-shot selection beats random/LESS. The primary claim is that error-type-guided synthesis generalizes across in-domain and out-of-domain math datasets. That evidence is insecure until the possibility of OOD test-set contamination is ruled out, because GPT-4o-generated synthetic problems are not filtered against those benchmarks. The paper deserves credit for releasing code and for a clear pipeline, and the in-domain gains may still hold; hence a conditional rather than reject verdict. However, the specific condition should include a leakage audit, not only seed variance or pre-registration.","tokens_in":18685,"tokens_out":6197,"duration_ms":72501,"concrete_test":"Compute the maximum Rouge-L score between each of the 1,500 selected SEI-ICL training samples and every question in TAL-SCQ, GaoKaoBench-Math, SAT-MATH, and CollegeMath test sets. Also compute an embedding-based cosine similarity (e.g., sentence-transformer) as a paraphrase-robust check. If any training sample exceeds Rouge-L >= 0.7 (the paper's own threshold) or high embedding similarity to an OOD test question, re-run the OOD evaluation after removing the affected training samples and/or excluding the affected test questions. If the average OOD gain drops by more than 3 points when overlap is controlled, the out-of-domain generalization claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 filters generated data with Rouge-L > 0.7 only against GSM8K and MATH train/test sets. The four out-of-domain test sets (TAL-SCQ, GaoKaoBench-Math, SAT-MATH, CollegeMath) are public and likely present in GPT-4o training data, yet no overlap check is reported between the 1,500 selected training samples and these OOD test sets. The OOD improvements for Qwen2.5-Math-7B are large (TAL +26.14, GaoKao +42.13, SAT +29.40, College +11.46 in Figure 3), and these are exactly the benchmarks where the base model is weakest, so memorized near-duplicates are hard to distinguish from genuine generalization. Additionally, the Rouge-L threshold of 0.7 is permissive even for the in-domain sets, so some test-set overlap could survive the stated filter. Because the central claim emphasizes both in-domain and out-of-domain generalization, this unmeasured leakage risk is the most load-bearing threat to the paper's conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Self-Error-Instruct (SEI), a four-stage framework for improving LLM mathematical reasoning by synthesizing training data that generalizes from a target model's error types rather than from individual bad cases. Bad cases are extracted from the GSM8K/MATH training sets; GPT-4o generates error keyphrases, they are clustered into error types, and targeted synthetic data are generated via a self-instruct approach. A one-shot learning score computed on a 100-example dev set selects the top 5% of generated data, and the target model is continued with LoRA over three iterations. Experiments on Llama-3-8B-Instruct, Qwen2.5-Math-7B, and Mathstral-7B-v0.1 report in-domain and out-of-domain gains, with Qwen improving substantially and Llama3/Mathstral gaining about 1.7% and 1.0% on average. The paper also compares selection strategies, training schemes, and data volumes.","tokens_in":18901,"tokens_out":6471,"duration_ms":80172,"significance":"If the results are robust, the proposed error-type generalization and one-shot selection are useful and practical contributions. Strengths include a clearly specified pipeline, public code, several baselines and ablations, out-of-domain evaluation, and analyses of fix rates and training strategies. The one-shot selection criterion that rewards both fixing bad cases and preserving good cases is a reasonable and potentially transferable idea. However, the strength of the empirical claims is weakened by the absence of uncertainty quantification and by an unmeasured risk of overlap between GPT-4o-generated data and the out-of-domain test sets.","major_comments":[{"comment":"The out-of-domain generalization claim is exposed to a test-set contamination risk that is not measured. Section 4.1 states that generated data are filtered with Rouge-L > 0.7 only against the GSM8K and MATH training and test sets; no overlap check is reported between the 1,500 selected training samples and the TAL-SCQ, GaoKaoBench-Math, SAT-MATH, or CollegeMath test sets. Because the instructor model GPT-4o is likely trained on these public benchmarks, and because the out-of-domain gains for Qwen2.5-Math-7B are large (TAL +26.14, GaoKao +42.13, SAT +29.40, College +11.46 in Figure 3), the results could reflect memorized near-duplicates rather than genuine generalization. The authors should report max Rouge-L or embedding similarity between each selected synthetic sample and each out-of-domain test item, and ideally show that the gains persist on the least-similar subset of out-of-domain examples.","section":"Section 4.1, Table 2, Figure 3"},{"comment":"No error bars, confidence intervals, or significance tests are reported, and the claimed improvements for two of the three models are small: Llama-3-8B-Instruct improves by 1.72% and Mathstral by 0.98% on average. These margins are likely within run-to-run noise for greedy-decoding evaluations of fine-tuned models, so the statement that SEI-ICL 'outperforms others by substantial margins in all math datasets' is not supported for all models. The authors should provide multiple seeds or bootstrap estimates for the main table, and at least clarify which comparisons are stable.","section":"Section 5.1, Table 2"},{"comment":"The one-shot selection method is evaluated on a single fixed dev set of 100 examples (50 bad and 50 good cases), and this same dev set is used to select 1,500 training samples. The claim that one-shot ICL selection outperforms random selection and LESS could be driven by overfitting to this small dev set, and no stability analysis is given. The authors should repeat the selection with multiple dev-set resamples, or report the variance of the selected data and of the downstream performance.","section":"Section 3.3, Section 5.2"}],"minor_comments":[{"comment":"The text attributes LLM2LLM to 'Tong et al., 2024a' in two places (Sections 2.1 and 4.4), but the reference list gives Tong et al. 2024a as a paper on backdoor attacks; the correct citation for LLM2LLM is Lee et al. 2024.","section":"Section 2.1, References"},{"comment":"The table contains formatting errors, including missing spaces and merged numbers such as '15.7935.59', '38.2415.79', and '25.19 TAL', which should be corrected.","section":"Table 2"},{"comment":"The notation in Equations (4) and (5) is ambiguous: the superscript and subscript on r_i^j are not clearly indexed to the synthetic sample and the dev-set query, and the symbol for the model-generated reasoning path is not introduced before use.","section":"Equations (4)-(5)"},{"comment":"The phrase 'For fair comparison, the generalized data sizes for the baselines are kept consistent with SEI-ICL' is unclear for the Training Data and Bad Cases baselines, which appear to use the full training set rather than 1,500 samples; the intended data sizes should be stated explicitly.","section":"Section 4.4, Table 2"},{"comment":"The x-axis label 'Synthetic Data Size' is ambiguous: it should be stated whether the plotted performance corresponds to the full generated set or to a subset selected by the one-shot criterion, and whether the same selection pipeline was applied at each size.","section":"Figure 4 and Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is sensible, but the empirical evidence needs strengthening before acceptance. The contamination risk for out-of-domain gains and the lack of uncertainty quantification are the main load-bearing concerns; both are addressable with additional experiments and analyses. If the authors can provide the overlap checks and variance estimates, I would be willing to support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea here is real and worth your time. Instead of synthesizing data from isolated bad cases, the authors cluster error keyphrases into error types, generate targeted data per type, and then select the top 5% via a one-shot scoring rule that rewards fixing bad cases while preserving good ones. That combination is new relative to LEMA, LLM2LLM, and LLMs-as-Instructors, and the ablations support the importance of both the error-type grouping and the selection step. The Qwen2.5-Math gains are large and consistent across in-domain benchmarks (GSM8K +29.7, MATH +10.8), which is not something you see from random synthetic data. The paper is also honest about its limitations: GPT-4o cost, restricted error scope, and the computational load of one-shot selection are all stated plainly.\n\nNow the soft spots, in proportion. The most load-bearing issue is the out-of-domain evaluation. The authors filter generated data with Rouge-L > 0.7 only against GSM8K and MATH, not against TAL-SCQ, GaoKaoBench-Math, SAT-MATH, or CollegeMath. Those four benchmarks are public, likely present in GPT-4o's training data, and the largest OOD gains appear exactly where the base model is weakest. Without an overlap check between the 1,500 selected training samples and those test sets, the OOD generalization claim is not yet supported. The in-domain gains do not suffer from this particular leak, so the core method survives, but the OOD numbers as presented are suspicious.\n\nSmaller issues: no error bars or seed variance are reported, and for Llama3 and Mathstral the gains (1.72% and 0.98% average) are within run-to-run noise. The one-shot selection relies on a 100-case dev set, which is small for scoring 30,000 candidates. The top-5% threshold appears to have been chosen after looking at results for 5%, 10%, and 20%, so a pre-registered threshold would strengthen the claim. Also, the Rouge-L threshold of 0.7 is permissive even for in-domain filtering.\n\nWho this is for: people building fine-tuning pipelines for math reasoning and researchers working on synthetic data selection. It deserves a serious referee; the method is clearly described, the baselines are reasonable, and the central in-domain finding for Qwen is likely real. But the referee should demand an OOD overlap analysis and variance reporting before accepting the generalization story.\n\nMy recommendation: send it to review, but flag the leakage risk as a major revision item. If the OOD results hold up after the overlap check, this is a solid paper worth citing.","headline":"A genuinely new data-synthesis pipeline for math reasoning, with large Qwen gains, but the out-of-domain results are threatened by an unchecked leakage channel.","tokens_in":19449,"tokens_out":1133,"would_cite":true,"duration_ms":14708,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper tries to establish that an LLM's mathematical reasoning can be improved by mining its own wrong answers, grouping the mistakes into error types, synthesizing new practice problems for each type, and keeping only the synthetic…","keywords":["mathematical reasoning","error-type generalization","synthetic training data","self-instruct","one-shot data selection","fine-tuning","bad case analysis"],"falsifier":"Take the 30,000 generated samples, resample the 100-case dev set multiple times, and rerun the one-shot selection; if the top-5% subset chosen under different dev sets does not consistently beat random selection on the six test sets, the selection mechanism is not the source of the reported gains.","tokens_in":18495,"feed_emoji":"🧮","tokens_out":7326,"duration_ms":81076,"temperature":0.7,"pith_summary":"Self-Error-Instruct (SEI) is a training-data pipeline for improving an LLM's mathematical reasoning by mining its own mistakes. The paper claims that grouping a model's incorrect answers into error types, synthesizing new practice problems for each type, and then keeping only the synthetic examples that fix bad cases without hurting good ones yields large gains: average accuracy increases of 24.94% for Qwen2.5-Math-7B, 1.72% for Llama-3-8B-Instruct, and 0.98% for Mathstral-7B-v0.1 across six math benchmarks. The central bet is that generalizing from error types, rather than from isolated bad cases, produces training data that transfers to unseen problems. If true, this means small, targeted synthetic datasets can beat full training-set fine-tuning for math reasoning.","feed_headline":"Grouping math errors by type beats full-dataset training","feed_subtitle":"A pipeline mines a model's own wrong answers, generates new problems per error type, and keeps only examples that fix mistakes.","key_machinery":"The load-bearing object is the SEI pipeline: bad-case extraction by answer mismatch; error-keyphrase generation by an instructor model; clustering keyphrases into error types; error-type-specific data synthesis in a self-instruct style using a few sampled bad cases as prompts; one-shot learning selection on a 100-case dev set; and iterative LoRA fine-tuning. The one-shot selector is the component doing the paper's main work: it ranks each synthetic sample by the number of dev cases (50 bad plus 50 good) whose answer is correct when the sample is prepended as an exemplar, so selected data must both repair weaknesses and preserve existing competence. The pipeline also filters synthetic data against GSM8K and MATH train/test sets using Rouge-L similarity to avoid leakage.","core_discovery":"The paper's central claim is that error-type generalization, not the individual bad cases themselves, drives the improvement. Starting from a target model's wrong answers on GSM8K and MATH training problems, SEI has an instructor model summarize each mistake as a keyphrase, clusters those keyphrases into named error types, and then asks the instructor model to generate new problems that exercise each type. The generated set is filtered and scored: each candidate synthetic example is used as a one-shot prompt over 100 validation cases so that a score counts how many previously wrong cases it fixes and how many originally correct cases it preserves. The top 5% of 30,000 generated examples is used to fine-tune the target model, and the loop repeats for three iterations. The paper reports that this curated training set outperforms training on the raw math corpus, on the bad cases alone, and on data from prior single-case error-generation baselines, while also beating the full synthesized dataset.","pith_inferences":["The same recipe likely transfers beyond math: any setting where a target model emits checkable outputs and errors can be clustered, such as code generation, logic puzzles, or structured extraction, could use error-type generalization.","The one-shot selection score is a cheap proxy for fine-tuning gain; a testable extension is to replace the fixed 100-case dev set with a model-adaptive or online dev set that updates as errors are fixed.","Because the instructor model does all keyphrase extraction, clustering, and synthesis, the approach inherits the instructor's ceiling; a promising experiment is to measure how performance degrades with weaker or open-weight instructors.","The reported advantage over gradient-based selection could be sensitive to dev-set size and sampling; replicating selection with multiple random 100-case dev sets and multiple fine-tuning seeds would tell whether the selection signal is stable."],"forward_implications":["Training on error-type-generalized data improves both in-domain (GSM8K, MATH) and out-of-domain (TAL, GaoKao, SAT, College) math tests, so the effect is not just memorizing benchmark formats.","Raw training data and bad-case-only training can degrade already instruction-tuned Llama3 and Mathstral, so the synthesized data, not the original corpus, carries the gain.","One-shot learning selection outperforms random selection, gradient-based LESS, and even the full synthesized dataset, implying active selection matters more than data volume.","From-scratch training on all selected data across three iterations beats iteratively training round by round, consistent with small per-round data causing overfitting.","The bad-case fix rate rises with iterations, indicating the method keeps finding addressable weaknesses rather than exhausting them after one pass."],"supporting_citations":[{"why":"Instructor model GPT-4o that performs error keyphrase extraction, clustering, and data synthesis.","marker":"(OpenAI, 2024b)"},{"why":"Self-instruct method that SEI adapts to synthesize data from sampled examples.","marker":"(Wang et al., 2023)"},{"why":"One-shot learning data selection idea that SEI repurposes from perplexity to math-answer fixing.","marker":"(Li et al., 2024b)"},{"why":"Learning-from-error baseline that SEI compares against and extends from single cases to error types.","marker":"(Ying et al., 2024)"},{"why":"LLM2LLM iterative data-enhancement baseline that SEI must beat.","marker":"(Tong et al., 2024a)"},{"why":"LESS gradient-based data selection baseline that SEI outperforms in its selection experiments.","marker":"(Xia et al., 2024)"},{"why":"GSM8K dataset provides the bad cases and the in-domain test set.","marker":"(Cobbe et al., 2021)"},{"why":"MATH dataset provides the bad cases and the in-domain test set.","marker":"(Hendrycks et al., 2021)"},{"why":"Qwen2.5-Math-7B, the target model with the largest reported gain.","marker":"(Yang et al., 2024)"},{"why":"Llama-3-8B-Instruct, the target model whose smaller gains still support the method's positive effect.","marker":"(Grattafiori et al., 2024)"}],"fun_headline_variants":["Clustered error types outperform full dataset training","Group math mistakes by type to improve reasoning","Error-type generalization boosts LLM math skills","Curating error patterns beats training on all data","Targeted error synthesis improves LLM reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The one-shot selection score computed on a fixed 100-case dev set, split evenly between good and bad cases, reliably predicts which synthetic examples will most improve the target model after fine-tuning.","fun_headline_variants_meta":{"raw":{"variants":["Clustered error types outperform full dataset training","Group math mistakes by type to improve reasoning","Error-type generalization boosts LLM math skills","Curating error patterns beats training on all data","Targeted error synthesis improves LLM reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1361,"prompt_tokens":978,"completion_tokens":383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":315}},"tokens_in":594,"tokens_out":383,"duration_ms":5355,"temperature":1.0,"reasoning_tokens":315,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:03:16.175610+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 30,000 generated samples, resample the 100-case dev set multiple times, and rerun the one-shot selection; if the top-5% subset chosen under different dev sets does not consistently beat random selection on the six test sets, the selection mechanism is not the source of the reported gains.","supporting_citations":[],"review_version":1}