{"id":"121499b3-be88-4b3e-a19f-8492b86dbe9b","arxiv_id":"2502.06453","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Hard perturbations that change the required solution method cause 10-25% accuracy drops across 18 LLMs on MATH, revealing limits in reasoning robustness.","lead":"This paper builds MATH-P-Simple and MATH-P-Hard, 279 pairs of perturbed versions of the hardest MATH benchmark problems, where the hard versions require different solution methods from the originals. It shows all 18 tested language models drop 10 to 25 percent on the hard perturbed set, and argues the failures reveal a form of memorized problem-solving rather than genuine reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Performance drops on MATH-P-Hard are clear, but the paper's central attribution to a 'new form of memorization' is underdetermined: the only direct evidence is a small, non-blinded manual classification of error cases, with no difficulty-matched same-method control to rule out generic difficulty.","rationale":"The reader's weakest assumption identifies the annotation criterion for hard perturbations as the load-bearing premise: if hard perturbations are merely harder versions of the same method, then the accuracy drop is explained by difficulty rather than memorization. I agree that this is the core risk, but I see it as even broader than the annotation criterion. The aggregate accuracy gap between MATH-P-Simple and MATH-P-Hard is expected by construction, since hard perturbations are explicitly designed to require different and more difficult skills. To attribute the gap to memorization, the paper needs a control that separates difficulty from method-change, and it needs reliable, quantified evidence of the proposed failure mechanism. The only such evidence is the manual inspection of 20 error cases per model, with no inter-annotator reliability or blinding. This is a genuine soft spot, but it is addressable: the benchmark itself and the observed performance drops remain useful, and the memorization claim can be tested by independent annotation or by a difficulty-matched control. Therefore I do not move the verdict away from the reader's CONDITIONAL; the conditions should include releasing the data and providing stronger support for the memorization attribution.","tokens_in":23118,"tokens_out":4691,"duration_ms":43873,"concrete_test":"Release the dataset and error traces. Have two independent math-qualified annotators, blind to the authors' labels and to the original/simple solutions, classify a random sample of 100 MATH-P-Hard errors (50 from o1-mini and 50 from Claude-3.5-Sonnet) into 'misapplies original method', 'general math error', or 'other', and compute Cohen's kappa against the authors' classifications. If kappa is below about 0.6 or the memorization rate drops substantially, the central claim fails; if kappa is high, the memorization finding is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Sections 3.2 and 5 is that models exhibit a 'new form of memorization'—blindly applying learned techniques to modified settings. This claim requires that the MATH-P-Hard accuracy drops be attributable to method-change, not just to increased problem difficulty. Table 1 and Figure 2 establish only that MATH-P-Hard is harder; MATH-P-Simple is not a difficulty-matched control because it is defined as same-method and is intentionally close to the original. The direct evidence for the memorization mechanism is Section 3.2's manual failure-mode analysis: 20 error cases per model, with estimates of 40% (o1-mini) and 25% (Claude-3.5-Sonnet) labeled as memorization. No inter-annotator agreement, blinded protocol, or comparison against errors on a hard-but-same-method baseline is reported. Figures 5 and 6 are illustrative case studies; they do not quantify how prevalent the mechanism is. If the manual labels are unreliable, the paper still demonstrates robustness failure on a harder benchmark, but its stated novel contribution—memorization of problem-solving techniques—is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MATH-Perturb, a benchmark consisting of 279 problems derived from level-5 MATH problems, each with a simple perturbation (MATH-P-Simple, same solution method) and a hard perturbation (MATH-P-Hard, intended to require a different solution method). The authors evaluate 18 LLMs under zero-shot chain-of-thought and report consistent accuracy drops on MATH-P-Hard (10–25% for the strongest models), with smaller drops on MATH-P-Simple. They further analyze failure modes and argue that a substantial fraction of errors stem from a 'new form of memorization' in which models blindly apply solution techniques from the original problem without adapting to the modified conditions. They also study the effect of using the original problem as an in-context learning demonstration, showing that the misleading effect partially offsets the ICL benefit on MATH-P-Hard.","tokens_in":23366,"tokens_out":5791,"duration_ms":48765,"significance":"If the benchmark and its interpretation hold, the paper makes a useful contribution: a curated, human-validated stress test that goes beyond prior simple-perturbation benchmarks (e.g., Functional MATH) by requiring a genuine change of solution method. The accuracy drops on frontier models are striking and likely to motivate further work on out-of-distribution reasoning. The paper also ships useful auxiliary analyses: the train/test split of the seed problems, the retrieval-based MRR check showing that original problems are easily retrieved, and the ICL ablation with random examples. However, the central mechanistic claim—that failures indicate a new form of memorization of problem-solving techniques—is not yet adequately supported by the evidence presented.","major_comments":[{"comment":"The quantitative support for the paper's central mechanistic claim is thin. The memorization estimates (40% for o1-mini, 25% for Claude-3.5-Sonnet) come from manual inspection of only 20 error cases per model, for two models, with no inter-annotator agreement, no blinded protocol, and no confidence intervals. The sentence in §3.2 that these issues are 'often coupled with other types of errors' further weakens the attribution. To claim that memorization is 'pervasive among the models we evaluated,' the authors should either expand the manual analysis to more models and more errors, or report a validated coding scheme with agreement statistics.","section":"§3.2"},{"comment":"The fundamental distinction between MATH-P-Simple and MATH-P-Hard rests on the annotators' expert judgment that a hard perturbation 'cannot be solved using the same method as the original problem.' Yet the paper reports no inter-annotator reliability for this criterion, and Appendix C.1 itself notes a 'misalignment between the models' capabilities and the annotators' perception of the difficulties of math problems.' Because this criterion is the premise for interpreting the accuracy drop as evidence of method-change rather than generic difficulty, the paper needs an explicit validation study (e.g., independent re-annotation, or a check that the original solution method fails on MATH-P-Hard items).","section":"§2, Annotation Criterion"},{"comment":"The headline accuracy drops on MATH-P-Hard are clear, but the interpretation that they reflect bias toward the original reasoning patterns is underdetermined. MATH-P-Simple is a same-method control, not a difficulty-matched control: by construction it is close to the original and easy. A harder but same-method perturbation baseline is needed to attribute the 10–25% drops to the method change per se rather than to the increased difficulty of MATH-P-Hard. Without such a control, the accuracy data alone cannot distinguish the authors' memorization hypothesis from the generic claim that these models are weaker on harder problems.","section":"§3.1, Table 1"},{"comment":"The paper reports single-run accuracies without confidence intervals or significance tests. With only 279 items, a 5% difference corresponds to roughly 14 problems, and many train/test or model-to-model differences in Table 1 are of this size; the claim of 'significant performance drops' is therefore not statistically established. Similar caution applies to the ICL decomposition in Table 7, where the n(correct→wrong) and n(wrong→correct) counts are small; a bootstrap or per-item significance analysis would be appropriate.","section":"§3.1, Table 1; §3.4, Table 7"}],"minor_comments":[{"comment":"Fix the citation typo 'Hendrycksmath et. al.' to 'Hendrycks et al.'.","section":"Abstract"},{"comment":"The sentence 'in Table 7 and Figure 7, and we calculate' contains a stray 'and'; it should be 'in Table 7 and Figure 7, we calculate'.","section":"§3.4"},{"comment":"The model name 'Mathtral' should be spelled 'Mathstral'.","section":"Appendix A"},{"comment":"For reproducibility, the paper should state the exact zero-shot CoT prompt used and the decoding settings (temperature, top-p, max tokens), as these details are not currently specified.","section":"§3, Evaluation Setting"},{"comment":"The paper should include a link to the released benchmark (data, annotation guidelines, and evaluation code) if available, as the benchmark is the primary artifact of the manuscript.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is a valuable resource and the accuracy-drop results are likely to be of broad interest. The revision should focus on strengthening the evidence for the memorization interpretation, either by adding a difficulty-matched same-method control, validating the annotation criterion through inter-annotator statistics, or substantially expanding the manual error analysis. If the authors prefer to emphasize the benchmark rather than the mechanistic claim, they should soften the 'new form of memorization' claim accordingly. Given the strong initial results, I see this as a major-revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time. The core artifact is MATH-P-Hard, the first perturbation set I know that changes the solution method, not just the numbers, while keeping edits small. That is a real conceptual step over Functional MATH, GSM-Symbolic, and GSM-Plus. The 18-model evaluation is thorough, and the headline drop (10–25% across the board, including long-CoT models) is clearly present in Table 1. The ICL experiment is also a nice touch: showing that feeding the original problem and solution can hurt hard-perturbed performance, with a decomposition into helpful and misleading effects, is a concrete and non-obvious finding.\n\nThe soft spot is the interpretation, not the benchmark. The paper calls the failures a 'new form of memorization' and attributes 40% (o1-mini) and 25% (Claude-3.5-Sonnet) of errors to it based on 20 manually inspected cases per model. That sample is small, not blinded, and lacks inter-annotator agreement. More importantly, MATH-P-Simple is not a difficulty-matched control: it is intentionally same-method and close to the original, so the gap between Simple and Hard could partly be generic difficulty. The Case IV analysis (fails Hard while solving Original or Simple) somewhat mitigates this, but it still does not separate method-change from difficulty. I would not press the 'memorization of techniques' claim as established fact; I would frame it as a suggestive hypothesis that motivates future work.\n\nAlso: no error bars anywhere, and the dataset/code are not released. Both are fixable. For a benchmark paper, releasing the curated 558 problems is almost a requirement.\n\nWho is this for? Anyone working on math reasoning robustness, benchmark design, or the memorization-vs-reasoning debate. It deserves a serious referee, but I'd ask for the release, error bars, a difficulty-matched hard-but-same-method control, and a more rigorous manual coding protocol before I'd trust the mechanism claim. The benchmark itself will likely be reused regardless.","headline":"A genuinely new benchmark for hard perturbations with a real failure mode, but the memorization label outruns the evidence.","tokens_in":23900,"tokens_out":2492,"would_cite":true,"duration_ms":21423,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"All 18 tested LLMs drop 10-25% on minimal math edits that force a different solution method; many failures are a new form of memorization—applying learned techniques without checking if they apply.","keywords":["MATH-Perturb","hard perturbations","simple perturbations","mathematical reasoning","memorization","LLM evaluation","chain-of-thought","out-of-distribution generalization"],"falsifier":"Ask a fresh panel of experts to independently re-classify all 279 MATH-P-Hard items as either 'same method' or 'different method' relative to the original, and check whether the MATH-P-Hard accuracy drop concentrates on items with high agreement; low agreement would make difficulty, not memorization, the more parsimonious explanation.","tokens_in":22942,"feed_emoji":"🧮","tokens_out":12768,"duration_ms":89820,"temperature":0.7,"pith_summary":"This paper asks whether high LLM scores on math benchmarks reflect genuine reasoning or memorization of solution patterns. To test this, the authors construct two perturbed versions of 279 hardest (level-5) MATH problems: MATH-P-Simple, where surface details change but the solution method is identical, and MATH-P-Hard, where a minimal edit forces a different solution method. Every one of the 18 models evaluated, including o1-mini and Gemini-2.0-flash-thinking, scores substantially lower on MATH-P-Hard, with drops around 10-25%. Manual error analysis leads the authors to a new form of memorization: models apply trained solution techniques without checking whether the modified setting still permits them, and giving the original problem as an in-context example can make this worse. If this is right, current benchmark scores overstate reasoning robustness, and hard-perturbation generalization will be the next bottleneck.","feed_headline":"Minimal edit that changes the method drops every tested LLM by 10-25%","feed_subtitle":"New benchmark changes the required solution method, not just numbers; strong reasoning models stumble.","key_machinery":"The load-bearing mechanism is the paired perturbation design: for each seed problem, MATH-P-Simple and MATH-P-Hard are generated from the same original with minimal edits and guaranteed changed answers, and expert annotators enforce that a hard perturbation cannot be solved by the original method and instead requires deeper understanding. This pairing makes the Original-versus-Simple-versus-Hard comparison a controlled probe: Simple controls for surface-form sensitivity, while Hard isolates whether the model detects that the solution pattern has changed. The failure-mode analysis and the in-context-learning experiment are the instruments that convert the accuracy drop into evidence of method memorization, with the ICL decomposition into helpful versus misleading effects quantifying how demonstrations can entrench the memorized pattern.","core_discovery":"The paper's central claim is that LLMs' mathematical reasoning is substantially biased toward the distribution of solution patterns seen in training, and that this bias shows up as a new form of memorization. On MATH-P-Hard, problems that differ from the original by a minimal edit but require a different, often deeper solution method, all evaluated models show large accuracy drops even though the same models solve the simple-perturbation versions nearly as well as the originals. Manual inspection of failures shows models ignoring the modified assumptions, blindly applying the original technique, and sometimes even outputting the original answer rather than the changed one. The authors estimate that for strong models such as o1-mini and Claude-3.5-Sonnet, memorization-type errors account for 40% and 25% of the errors respectively. They also find that one-shot in-context learning with the original problem and solution has a misleading effect on MATH-P-Hard that cancels much of its benefit, and they conclude that robustness to hard perturbations, not average accuracy on familiar problems, is the key missing capability.","pith_inferences":["A direct testable extension is that an explicit verification step, asking the model to check whether the original solution method still applies before solving, should recover a large fraction of the MATH-P-Hard losses; this can be measured on the released benchmark without new annotation.","The paired simple/hard design transfers to code and science reasoning: changing one condition that invalidates a standard algorithm should produce a similar accuracy cliff, which would show the phenomenon is not specific to math word problems.","A stronger memorization test would compare the hard-perturbation drop against difficulty-matched control problems that share no training-distribution ancestor; if the drop disappears, the bias is toward learned solution patterns rather than toward easy problems.","If hard perturbations enter routine evaluation, published accuracies on MATH-style benchmarks would fall substantially, reframing how much 'reasoning' current models display."],"forward_implications":["Every tested model, including the strongest reasoning models, loses roughly 10-25% on MATH-P-Hard, so high average accuracy on the original MATH set coexists with systematic fragility when the required method changes.","Robustness to simple perturbations has improved since earlier functional benchmarks, but that improvement does not transfer to hard perturbations, meaning progress on average-case math is not the same as method-shift generalization.","One-shot in-context learning with the original problem and solution helps on MATH-P-Simple but can mislead models on MATH-P-Hard, and for several large models the misleading effect nearly cancels the demonstration's benefit.","Because in-context learning is a form of test-time training, the paper predicts that naive fine-tuning on a narrow distribution of problem settings will hurt hard-perturbation generalization.","Hard-perturbation robustness should be tracked as a separate evaluation axis rather than folded into average accuracy."],"supporting_citations":[{"why":"Supplies the 279 level-5 MATH seed problems and the string-normalization plus sympy equivalence checker used to score answers.","marker":"Hendrycks et al., 2021"},{"why":"Functional MATH is the simple-perturbation benchmark that this work extends and the reference whose reported 58%-80% drops motivate the robustness comparison.","marker":"Srivastava et al., 2024"},{"why":"GSM-Symbolic's numeric and template perturbations define the simple-perturbation category that MATH-P-Hard is contrasted with.","marker":"Mirzadeh et al., 2024"},{"why":"Zero-shot chain-of-thought is the evaluation protocol used for all models on the three problem sets.","marker":"Kojima et al., 2022"},{"why":"Chain-of-thought prompting grounds the CoT evaluation and the in-context-learning experiments.","marker":"Wei et al., 2022"},{"why":"The answer-equivalence checker (with Hendrycks et al.) and DeepSeekMath are part of the evaluation pipeline.","marker":"Shao et al., 2024"},{"why":"MATH2 is the comparison benchmark that creates harder problems without natural original references, used to justify keeping original problems as anchors in MATH-P-Hard.","marker":"Shah et al., 2024"},{"why":"o1-preview and o1-mini are central test subjects, and the observation that state-of-the-art models exceed 90% on MATH motivates focusing on level-5 problems.","marker":"OpenAI, 2024"}],"fun_headline_variants":["Same problem, new method: LLMs tumble 10-25% on hard edits","Hard perturbation benchmark exposes LLM math memorization","Minimal edits that change approach break LLM math reasoning","New benchmark: LLMs fail when solution steps change, not just digits","MATH-Perturb: LLMs' math masks memorized steps, not reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the expert annotators were right that every MATH-P-Hard item truly requires a different solution method than the original, because if a hard item is just a harder instance of the same method the observed drop could be plain difficulty rather than memorization.","fun_headline_variants_meta":{"raw":{"variants":["Same problem, new method: LLMs tumble 10-25% on hard edits","Hard perturbation benchmark exposes LLM math memorization","Minimal edits that change approach break LLM math reasoning","New benchmark: LLMs fail when solution steps change, not just digits","MATH-Perturb: LLMs' math masks memorized steps, not reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1341,"prompt_tokens":990,"completion_tokens":351,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":256}},"tokens_in":606,"tokens_out":351,"duration_ms":15025,"temperature":1.0,"reasoning_tokens":256,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:21:58.226772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a fresh panel of experts to independently re-classify all 279 MATH-P-Hard items as either 'same method' or 'different method' relative to the original, and check whether the MATH-P-Hard accuracy drop concentrates on items with high agreement; low agreement would make difficulty, not memorization, the more parsimonious explanation.","supporting_citations":[],"review_version":1}