{"id":"4057607d-261f-4434-b5a3-a6b39f470875","arxiv_id":"2508.19069","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Training on easy synthetic math data lowers accuracy on hard benchmarks, and the proposed SST framework, which teaches explicit procedural chains, aims to reverse that drop.","lead":"The authors report that fine-tuning a language model on large amounts of easy, template-generated math problems hurts its performance on hard math exams, and they propose a three-stage training framework that injects structured solution templates to improve reasoning. They claim a 'Scaling Law by Difficulty' and report gains on GSM8K and AIME24, but the paper's own tables contradict part of the claim.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'Scaling Law by Difficulty' is not demonstrated: Figure 1 shows a monotonic decline, not a U-shape, and the attribution to difficulty is confounded with data source, format, and repetition.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing weakness: the causal attribution to 'difficulty' is not controlled against repetition, surface-form overfitting, and distribution shift. My stress-test adds a further, arguably more fundamental point: the only controlled curve in the paper (Figure 1) is monotonic decreasing, which directly contradicts the claimed U-shaped law. There is no empirical curve with difficulty on the x-axis at all. Since the SST framework is motivated by and claims to validate this law, the unsupported law undermines the central contribution. This does not change the reader's REJECT verdict; it reinforces it. No ad hominem or overstatement is intended: the issue is that the evidence presented fails to establish the paper's central empirical claim.","tokens_in":12653,"tokens_out":3710,"duration_ms":41290,"concrete_test":"Re-run the Figure 1 experiment with a difficulty-controlled grid: take one template family, generate variations at 5 difficulty levels rated by independent judges or a strong LLM, keep token length/format/repetition constant, and train at fixed sizes (e.g., 20k/50k/100k) for each level, evaluating on AIME24 and held-out template problems. If accuracy versus difficulty is not U-shaped (or does not increase with difficulty when volume is fixed), the claimed law is unsupported. Also run a matched-size comparison between a high-difficulty Open-R1 subset and low-difficulty synthetic data with the same token count and format to rule out source confounds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is a 'Scaling Law by Difficulty' described in the abstract and the 'Scaling Law by Difficulty' section as a U-shaped curve of model performance versus training-data difficulty. The only quantitative evidence for this law is Figure 1, which plots AIME24 accuracy against training-set size for template-generated synthetic data. That figure is monotonically decreasing over 0–100k samples for both 1.5B and 7B models; there is no difficulty axis, no high-difficulty condition, and no U-shaped function anywhere in the paper. The high-difficulty comparison is a single unmatched dataset (Open-R1, used at roughly one-fifth the volume) that differs simultaneously in source, problem distribution, format, token length, and repetition rate—so the observed 30.48 score cannot isolate difficulty as the causal variable. Figure 3 reports a difficulty distribution for synthetic vs. MATH data, but it is not linked to any performance curve. The claim that 'excessive low-difficulty data impedes abstraction, while high-difficulty data enhances reasoning' is thus unsupported: the entire scaling law rests on a confounded two-point comparison, and the one controlled curve (Figure 1) has the wrong shape for the stated U-shaped law.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims to identify a 'Scaling Law by Difficulty': LLM performance on math reasoning is asserted to follow a U-shaped curve with respect to training-data difficulty, with excessive low-difficulty data hurting abstraction and high-difficulty data enhancing reasoning. Motivated by this, the authors propose the Structured Solution Template (SST) framework, comprising (1) fine-tuning with weighted loss on structured solution-template chains, (2) prompt-time injection of generated chains as cognitive scaffolds, and (3) integrated curriculum fine-tuning with GRPO. Experiments on GSM8K, MATH500, AIME24/25, AMC23, Gaokao English, and a new Dynamic En benchmark report accuracy and efficiency gains, especially on harder problems.","tokens_in":13029,"tokens_out":3639,"duration_ms":36641,"significance":"The paper presents a substantial amount of experimental work: a three-stage training pipeline with clear ablations, results averaged over eight seeds, comparison with multiple public baselines, and a new cross-domain benchmark. If the scaling-law claim were supported, it would be a noteworthy finding with implications for synthetic-data curation. However, the central novelty is not established: the controlled experiment in Figure 1 shows a monotone decline, not a U-shape, and the high-difficulty comparison is confounded and partly misreported. The method itself may have value, but the paper's framing and central claim are not currently justified.","major_comments":[{"comment":"The claimed U-shaped law is not supported by the evidence presented. Figure 1 plots AIME24 accuracy against training-set size for template-generated synthetic data and shows a monotone decrease from 0k to 100k for both model sizes. There is no difficulty axis, no high-difficulty condition in this controlled experiment, and no U-shaped function fitted or displayed. The Abstract and Conclusion repeat the 'U-shaped curve' claim, but the only controlled curve in the paper has the opposite shape. To support the law, the authors would need to vary difficulty within the same data source and show a non-monotone performance curve.","section":"Scaling Law by Difficulty, Fig. 1"},{"comment":"The one high-difficulty comparison is confounded and internally inconsistent. The text states that training on Open-R1 with 'only about one-fifth the data volume of the synthetic set' yields 30.48 on AIME24, but Table 1 reports 30.48 for the full SST-Framework, not for Open-R1 alone; Stage 1 on Open-R1 attains 27.71. The comparison also changes data source, problem distribution, format, token length, and repetition rate simultaneously, so difficulty cannot be isolated as the causal factor. This paragraph is load-bearing for the scaling-law claim.","section":"Scaling Law by Difficulty, Open-R1 paragraph"},{"comment":"The difficulty distribution in Figure 3 is not linked to any performance curve. The paper asserts a causal link between low difficulty and performance decline, but Fig. 3 only shows that synthetic problems are rated easier than MATH problems; it does not show that difficulty, rather than repetition, surface-form overfitting, or distribution shift, caused the Figure 1 decline. No quantitative difficulty-performance relationship or functional form is provided, so the term 'Scaling Law' is not justified.","section":"Scaling Law by Difficulty, Fig. 3"},{"comment":"The claim that SST results 'validate and extend the Scaling Law by Difficulty' is circular. The scaling law is inferred from experiments with template-generated synthetic data, independent of SST; the success of SST is then presented as corroboration of the law. This does not test the law. The paper should specify a falsifiable prediction and test it separately.","section":"Contributions and Main Results"}],"minor_comments":[{"comment":"Typos and formatting issues: 'focuse' in Stage 1, 'Soltion' in the section header, 'soss' in Table 5 heading, 'AN SW ER' in the Stage 3 trace, and inconsistent use of 'Step' vs 'Stage' between Table 1 and the text.","section":"Throughout"},{"comment":"The legend and axis are confusing: the labels '1.5B 100k Baseline' and '1.5B 0k Baseline' are unclear, and the Open-R1 point is not clearly distinguished. A table of the underlying numbers would improve reproducibility.","section":"Figure 1"},{"comment":"Ablation tables report point estimates without standard deviations, unlike the main results. This makes it difficult to assess whether the reported differences are significant.","section":"Tables 4-6"}],"recommendation":"reject","confidential_remarks":"The central claim is unsupported and, as stated, contradicted by Figure 1. The misattribution of the 30.48 AIME24 score to Open-R1 alone is a factual error in a key argument. A revised paper that removes the 'U-shaped scaling law' or replaces it with a properly controlled difficulty study could be reconsidered, but the current manuscript does not meet the bar."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: the SST framework is a real contribution. Combining weighted chain-token SFT, prompt-time injection of solution templates, and a GRPO curriculum with self-correction is not something I've seen in this exact form. The stage-wise breakdown in Table 2 is informative, and the empirical observation that more template-generated synthetic data monotonically hurts AIME24 accuracy is worth reporting, even if the interpretation is wrong.\n\nThe soft spots are in the framing. The 'Scaling Law by Difficulty' is not demonstrated. Figure 1 shows accuracy declining as dataset size grows; there is no difficulty axis and no high-difficulty arm in that experiment. The U-shape appears only in the abstract and conclusion. The comparison to Open-R1 is a single unmatched dataset that differs in source, format, volume, and token length, and the 30.48 score actually comes from the full three-stage SST, not from that dataset alone. So the law rests on a confounded two-point comparison.\n\nThe paper also overclaims: 'SST outperforms all previous methods' is contradicted by Table 1, where L1-Qwen variants beat SST on five of seven benchmarks. That needs correcting. The new Dynamic En benchmark is not released and no code is provided, which limits reproducibility.\n\nOverall, there is useful engineering and a new empirical observation here, but the scaling-law framing is not supportable. In peer review, I'd push the authors to either drop the 'law' or run a proper experiment that varies difficulty while holding other factors fixed. The SST method can stand on its own.\n\nWho is this for? People working on synthetic data curation and structured prompting for math reasoning. I'd cite the empirical finding about synthetic data degradation. It does deserve a serious referee, but with major revisions.","headline":"Useful method paper with an overclaimed 'Scaling Law by Difficulty' that is not supported by the evidence; the SST framework itself deserves attention.","tokens_in":13469,"tokens_out":2360,"would_cite":true,"duration_ms":23364,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a Scaling Law by Difficulty for LLM math reasoning: performance is U-shaped in training-data difficulty, so flooding a model with easy generated problems degrades hard-benchmark accuracy, while high-difficulty data builds p","keywords":["scaling law by difficulty","structured solution templates","procedural reasoning","mathematical reasoning","synthetic data","curriculum fine-tuning","chain-of-thought","GRPO"],"falsifier":"Fine-tune the same 1.5B base model on template-generated synthetic problems whose difficulty distribution has been artificially matched to the curated Open-R1 set (by filtering easy items or prompting the generator to escalate difficulty), scaling from 0k to 100k samples and measuring AIME24 accuracy at each size. If accuracy no longer declines with dataset size, difficulty is confirmed as the driver of the U-shape; if it still declines, some other property of synthetic data is the cause.","tokens_in":12601,"feed_emoji":"📉","tokens_out":12266,"duration_ms":104509,"temperature":0.7,"pith_summary":"This paper tries to establish a 'Scaling Law by Difficulty' for mathematical reasoning in LLMs: model performance follows a U-shaped curve in training-data complexity. The authors show that fine-tuning on large volumes of template-generated synthetic problems—which they demonstrate are much easier than their parent problems—steadily lowers accuracy on hard contests like AIME24, while a much smaller set of curated high-difficulty problems sharply raises it. This matters because the field's default remedy for weak reasoning is to generate more synthetic data; on the paper's account, the difficulty of that data, not its volume, is the controlling variable. Building on the law, the paper proposes the Structured Solution Template (SST) framework—explicit procedural chains, weighted loss emphasis, prompt-time chain injection, and a plan–execute–self-correct curriculum trained with GRPO—and reports accuracy and efficiency gains across seven benchmarks, largest on the hardest problems. If the law holds, data scaling for reasoning should be redesigned around difficulty curation rather than sheer quantity.","feed_headline":"Easy math training data actively weakens LLM reasoning","feed_subtitle":"Hard problems, not volume, decide LLM math skill: explicit templates beat baselines on GSM8K and AIME24.","key_machinery":"Two linked mechanisms carry the argument. The first is the named Scaling Law by Difficulty: a U-shaped relationship between training-data complexity and downstream reasoning performance. It is established empirically by the Figure 1 curve—AIME24 accuracy monotonically drops as template-generated synthetic data grows from 0k to 100k—and by a GPT-4 difficulty audit showing generated problems are far easier than their parents, while a smaller curated high-difficulty set outperforms the larger synthetic set. The second is the Structured Solution Template (SST): short abstract blueprints of a solution procedure written as <chain>...</chain> token sequences. The template does three jobs. In Stage","core_discovery":"The paper's central discovery is that for math reasoning, more training data is not automatically better: performance on challenging benchmarks follows a U-shaped curve as a function of training-data difficulty. When the authors fine-tuned a 1.5B reasoning model on up to 100k template-generated synthetic problems, AIME24 accuracy fell steadily compared with no fine-tuning at all, even though the generated problems share surface structure with the test set. The paper traces this to a generator default: asked to produce variations of a hard 'parent' problem, LLMs produce much easier variants, confirmed by GPT-4 difficulty ratings, so the model learns rote pattern imitation rather than procedur","pith_inferences":["If the law generalizes beyond mathematics, curriculum designers for code, formal logic, or scientific reasoning should treat accumulated easy synthetic mass as an active hazard; a direct test would apply difficulty-filtered generation to those domains and check for the same U-shape.","The paper's attribution of the AIME24 decline to difficulty alone is not yet isolated from confounds such as repetition, surface-form overfitting, and distribution shift; a decisive experiment would hold the generator and template fixed and vary only measured difficulty.","The linearly decaying chain-token weight hints at a within-sequence curriculum; an untested extension is to make the weight schedule adapt to the model's own error rate, turning token weighting into an online difficulty controller.","The new Dynamic En benchmark separates small models widely (8.75 to 63.73 accuracy), so it could be reused as a high-discrimination stress test for template-guided and RL-trained reasoning methods beyond this paper."],"forward_implications":["The default strategy of scaling reasoning data by volume flips: since 100k easy synthetic problems can hurt more than help, generation pipelines must escalate or filter for difficulty, or risk degrading the very skill they target.","Chain-token weighting with a decaying schedule lets a 1.5B model internalize abstract procedure early and generalize later, so any format where a few tokens carry the reasoning load becomes a candidate for the same loss shaping.","Prompt-time chain injection from a lightweight generator improves hard-benchmark accuracy (AIME24 rises 28.30 to 32.67 in Stage 2) while cutting output tokens by 62% on easy problems, making the method an efficiency win as well as an accuracy win.","The plan-then-execute format with self-reflection, trained by GRPO over a hard-mined curriculum, produces the largest single-stage gains (up to +6.72 on Dynamic En), showing the U-shaped curve can be reshaped by training design, not merely respected by data filtering.","SST beats the strongest length-control baseline (L1) on GSM8K and matches or exceeds prior RL-tuned models on AIME24/25, indicating structured procedural scaffolding and reasoning-effort control are complementary levers, not alternatives."],"supporting_citations":[{"why":"Supplies the base reasoning models (DeepSeek-R1-Distill-Qwen-1.5B/7B) that all stages fine-tune, and the DeepSeek-R1 API used to generate the Stage 3 plan–execute–self-correct traces.","marker":"Guo et al. 2025"},{"why":"Open-R1 dataset: the curated high-difficulty training set whose smaller volume delivers the AIME24 score that motivates the Scaling Law by Difficulty and is used for Stage 1 SFT.","marker":"Hugging Face 2025"},{"why":"PromptCoT: the synthetic problem-generation baseline representing the scale-up-easy-data approach the paper argues against, and a baseline SST outperforms.","marker":"Zhao et al. 2025"},{"why":"LoRA: the parameter-efficient adaptation used to train the lightweight 1.5B chain generator that produces prompt-time solution templates in Stage 2.","marker":"Hu et al. 2022"},{"why":"Chain-of-thought prompting: the base reasoning paradigm that the structured <chain> templates formalize and supervise.","marker":"Wei et al. 2022"},{"why":"GSM8K: the easy benchmark used to show the difficulty-dependent trade-off where chain injection cuts tokens by 62% and to report SST's largest accuracy gain.","marker":"Cobbe et al. 2021"},{"why":"MATH500: the benchmark where Stage 1's chain-weighted SFT reaches its highest accuracy, supporting the claim that procedural templates aid hard curated problems.","marker":"Lightman et al. 2023"},{"why":"L1: the reasoning-length-control baseline whose GSM8K score (82.44) SST beats (84.80) and which frames the efficiency comparison.","marker":"Aggarwal and Welleck 2025"}],"fun_headline_variants":["Training on easy math hurts LLM reasoning, U-shaped law","Easy data weakens LLM math, hard data boosts it","U-shaped scaling: hard problems teach LLMs better","More easy math data? It actually hurts LLM performance","Structured templates beat volume for LLM math reasoning"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper attributes the AIME24 performance decline specifically to the low difficulty of the template-generated synthetic data, without controlling for other differences from the curated comparison data—repetition, surface-form overfitting, or distribution shift away from AIME-style problems; if difficulty is not the real driver, the U-shaped law collapses.","fun_headline_variants_meta":{"raw":{"variants":["Training on easy math hurts LLM reasoning, U-shaped law","Easy data weakens LLM math, hard data boosts it","U-shaped scaling: hard problems teach LLMs better","More easy math data? It actually hurts LLM performance","Structured templates beat volume for LLM math reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000313,"raw_usage":{"total_tokens":1613,"prompt_tokens":736,"completion_tokens":877,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":796}},"tokens_in":480,"tokens_out":877,"duration_ms":6590,"temperature":1.0,"reasoning_tokens":796,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:57:53.517529+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same 1.5B base model on template-generated synthetic problems whose difficulty distribution has been artificially matched to the curated Open-R1 set (by filtering easy items or prompting the generator to escalate difficulty), scaling from 0k to 100k samples and measuring AIME24 accuracy at each size. If accuracy no longer declines with dataset size, difficulty is confirmed as the driver of the U-shape; if it still declines, some other property of synthetic data is the cause.","supporting_citations":[],"review_version":1}