Pith. sign in

REVIEW 4 major objections 5 minor 13 cited by

MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read All 18 tested LLMs drop 10-25% on minimal math edits that force a different solution method; many failures are a new form of memorization—applying learned techniques without checking if they apply.

desk verdict A genuinely new benchmark for hard perturbations with a real failure mode, but the memorization label outruns the evidence. read the letter →

arxiv 2502.06453 v2 pith:7MRDW5W4 submitted 2025-02-10 cs.LG cs.AIcs.CL

Kaixuan Huang , Jiacheng Guo , Zihao Li , Xiang Ji , Jiawei Ge , Wenzhe Li , Yingqing Guo , Tianle Cai
show 10 more authors
Hui Yuan Runzhe Wang Yue Wu Ming Yin Shange Tang Yangsibo Huang Chi Jin Xinyun Chen Chiyuan Zhang Mengdi Wang
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords MATH-PerturbhardperturbationssimplemathematicalreasoningmemorizationLLMevaluationchain-of-thoughtout-of-distributiongeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether high LLM scores on math benchmarks reflect genuine reasoning or memorization of solution patterns. To test this, the authors construct two perturbed versions of 279 hardest (level-5) MATH problems: MATH-P-Simple, where surface details change but the solution method is identical, and MATH-P-Hard, where a minimal edit forces a different solution method. Every one of the 18 models evaluated, including o1-mini and Gemini-2.0-flash-thinking, scores substantially lower on MATH-P-Hard, with drops around 10-25%. Manual error analysis leads the authors to a new form of memorization: models apply trained solution techniques without checking whether the modified setting still permits them, and giving the original problem as an in-context example can make this worse. If this is right, current benchmark scores overstate reasoning robustness, and hard-perturbation generalization will be the next bottleneck.

What carries the argument

The load-bearing mechanism is the paired perturbation design: for each seed problem, MATH-P-Simple and MATH-P-Hard are generated from the same original with minimal edits and guaranteed changed answers, and expert annotators enforce that a hard perturbation cannot be solved by the original method and instead requires deeper understanding. This pairing makes the Original-versus-Simple-versus-Hard comparison a controlled probe: Simple controls for surface-form sensitivity, while Hard isolates whether the model detects that the solution pattern has changed. The failure-mode analysis and the in-context-learning experiment are the instruments that convert the accuracy drop into evidence of method memorization, with the ICL decomposition into helpful versus misleading effects quantifying how demonstrations can entrench the memorized pattern.

What would settle it

Ask a fresh panel of experts to independently re-classify all 279 MATH-P-Hard items as either 'same method' or 'different method' relative to the original, and check whether the MATH-P-Hard accuracy drop concentrates on items with high agreement; low agreement would make difficulty, not memorization, the more parsimonious explanation.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs' mathematical reasoning is substantially biased toward the distribution of solution patterns seen in training, and that this bias shows up as a new form of memorization. On MATH-P-Hard, problems that differ from the original by a minimal edit but require a different, often deeper solution method, all evaluated models show large accuracy drops even though the same models solve the simple-perturbation versions nearly as well as the originals. Manual inspection of failures shows models ignoring the modified assumptions, blindly applying the original technique, and sometimes even outputting the original answer rather than the changed one. The authors estimate that for strong models such as o1-mini and Claude-3.5-Sonnet, memorization-type errors account for 40% and 25% of the errors respectively. They also find that one-shot in-context learning with the original problem and solution has a misleading effect on MATH-P-Hard that cancels much of its benefit, and they conclude that robustness to hard perturbations, not average accuracy on familiar problems, is the key missing capability.

Load-bearing premise

The load-bearing premise is that the expert annotators were right that every MATH-P-Hard item truly requires a different solution method than the original, because if a hard item is just a harder instance of the same method the observed drop could be plain difficulty rather than memorization.

Editorial extensions

If this is right

  • Every tested model, including the strongest reasoning models, loses roughly 10-25% on MATH-P-Hard, so high average accuracy on the original MATH set coexists with systematic fragility when the required method changes.
  • Robustness to simple perturbations has improved since earlier functional benchmarks, but that improvement does not transfer to hard perturbations, meaning progress on average-case math is not the same as method-shift generalization.
  • One-shot in-context learning with the original problem and solution helps on MATH-P-Simple but can mislead models on MATH-P-Hard, and for several large models the misleading effect nearly cancels the demonstration's benefit.
  • Because in-context learning is a form of test-time training, the paper predicts that naive fine-tuning on a narrow distribution of problem settings will hurt hard-perturbation generalization.
  • Hard-perturbation robustness should be tracked as a separate evaluation axis rather than folded into average accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension is that an explicit verification step, asking the model to check whether the original solution method still applies before solving, should recover a large fraction of the MATH-P-Hard losses; this can be measured on the released benchmark without new annotation.
  • The paired simple/hard design transfers to code and science reasoning: changing one condition that invalidates a standard algorithm should produce a similar accuracy cliff, which would show the phenomenon is not specific to math word problems.
  • A stronger memorization test would compare the hard-perturbation drop against difficulty-matched control problems that share no training-distribution ancestor; if the drop disappears, the bias is toward learned solution patterns rather than toward easy problems.
  • If hard perturbations enter routine evaluation, published accuracies on MATH-style benchmarks would fall substantially, reframing how much 'reasoning' current models display.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MATH-Perturb, a benchmark consisting of 279 problems derived from level-5 MATH problems, each with a simple perturbation (MATH-P-Simple, same solution method) and a hard perturbation (MATH-P-Hard, intended to require a different solution method). The authors evaluate 18 LLMs under zero-shot chain-of-thought and report consistent accuracy drops on MATH-P-Hard (10–25% for the strongest models), with smaller drops on MATH-P-Simple. They further analyze failure modes and argue that a substantial fraction of errors stem from a 'new form of memorization' in which models blindly apply solution techniques from the original problem without adapting to the modified conditions. They also study the effect of using the original problem as an in-context learning demonstration, showing that the misleading effect partially offsets the ICL benefit on MATH-P-Hard.

Significance. If the benchmark and its interpretation hold, the paper makes a useful contribution: a curated, human-validated stress test that goes beyond prior simple-perturbation benchmarks (e.g., Functional MATH) by requiring a genuine change of solution method. The accuracy drops on frontier models are striking and likely to motivate further work on out-of-distribution reasoning. The paper also ships useful auxiliary analyses: the train/test split of the seed problems, the retrieval-based MRR check showing that original problems are easily retrieved, and the ICL ablation with random examples. However, the central mechanistic claim—that failures indicate a new form of memorization of problem-solving techniques—is not yet adequately supported by the evidence presented.

major comments (4)
  1. [§3.2] The quantitative support for the paper's central mechanistic claim is thin. The memorization estimates (40% for o1-mini, 25% for Claude-3.5-Sonnet) come from manual inspection of only 20 error cases per model, for two models, with no inter-annotator agreement, no blinded protocol, and no confidence intervals. The sentence in §3.2 that these issues are 'often coupled with other types of errors' further weakens the attribution. To claim that memorization is 'pervasive among the models we evaluated,' the authors should either expand the manual analysis to more models and more errors, or report a validated coding scheme with agreement statistics.
  2. [§2, Annotation Criterion] The fundamental distinction between MATH-P-Simple and MATH-P-Hard rests on the annotators' expert judgment that a hard perturbation 'cannot be solved using the same method as the original problem.' Yet the paper reports no inter-annotator reliability for this criterion, and Appendix C.1 itself notes a 'misalignment between the models' capabilities and the annotators' perception of the difficulties of math problems.' Because this criterion is the premise for interpreting the accuracy drop as evidence of method-change rather than generic difficulty, the paper needs an explicit validation study (e.g., independent re-annotation, or a check that the original solution method fails on MATH-P-Hard items).
  3. [§3.1, Table 1] The headline accuracy drops on MATH-P-Hard are clear, but the interpretation that they reflect bias toward the original reasoning patterns is underdetermined. MATH-P-Simple is a same-method control, not a difficulty-matched control: by construction it is close to the original and easy. A harder but same-method perturbation baseline is needed to attribute the 10–25% drops to the method change per se rather than to the increased difficulty of MATH-P-Hard. Without such a control, the accuracy data alone cannot distinguish the authors' memorization hypothesis from the generic claim that these models are weaker on harder problems.
  4. [§3.1, Table 1; §3.4, Table 7] The paper reports single-run accuracies without confidence intervals or significance tests. With only 279 items, a 5% difference corresponds to roughly 14 problems, and many train/test or model-to-model differences in Table 1 are of this size; the claim of 'significant performance drops' is therefore not statistically established. Similar caution applies to the ICL decomposition in Table 7, where the n(correct→wrong) and n(wrong→correct) counts are small; a bootstrap or per-item significance analysis would be appropriate.
minor comments (5)
  1. [Abstract] Fix the citation typo 'Hendrycksmath et. al.' to 'Hendrycks et al.'.
  2. [§3.4] The sentence 'in Table 7 and Figure 7, and we calculate' contains a stray 'and'; it should be 'in Table 7 and Figure 7, we calculate'.
  3. [Appendix A] The model name 'Mathtral' should be spelled 'Mathstral'.
  4. [§3, Evaluation Setting] For reproducibility, the paper should state the exact zero-shot CoT prompt used and the decoding settings (temperature, top-p, max tokens), as these details are not currently specified.
  5. [General] The paper should include a link to the released benchmark (data, annotation guidelines, and evaluation code) if available, as the benchmark is the primary artifact of the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark accuracies are measured against external human-annotated ground truth, and the memorization interpretation is an empirical claim rather than a fitted or self-referential derivation.

full rationale

The paper's central measurements—accuracy on Original, MATH-P-Simple, and MATH-P-Hard—are direct evaluations against held-out, externally curated ground-truth answers from the MATH dataset, with answers checked by an independent equivalence checker. No parameter is fitted to the test outcomes and then renamed as a prediction; the only 'prediction' is the observed performance drop, which is measured, not derived from the benchmark's construction. The hard-perturbation criterion ('the modified problem cannot be solved using the same method as the original problem') is an annotator judgment used to design the benchmark, not a constraint imposed on the models' outputs, so the accuracy drop cannot reduce to the definition of the benchmark by construction. The failure-mode analysis that identifies 'memorization' is a manual, interpretive labeling of error cases; whether that interpretation is fully supported is a validity or correctness concern, not a circularity, because the labels are not derived from the benchmark's definitions. The self-citations in the related-work section (e.g., prior memorization work by some of the same authors) are background references and are not load-bearing for any argument in the paper. The paper is self-contained in its evaluation against external benchmarks, and no circular step can be exhibited from the text.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No parameters were fitted and no entities were introduced. The benchmark depends on judgement-based categorization and on the evaluation pipeline.

assumptions (3)
  • domain assumption The MATH dataset's level-5 problems and their official answers are correct.
    Used as seeds and ground truth in Section 2; if the source answers are wrong, the benchmark answers inherit the error.
  • ad hoc to paper Annotators can reliably determine whether a modified problem can be solved by the same method as the original.
    This is the defining criterion for MATH-P-Hard versus MATH-P-Simple in Section 2, and it is not formalized or externally validated.
  • domain assumption Zero-shot chain-of-thought with the sympy equivalence checker is a valid measure of model accuracy on these problems.
    Section 3 Evaluation Setting; no reliability analysis, repeated sampling, or human verification of all judged answers is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations." pith.science (2026). https://pith.science/paper/7MRDW5W4

@misc{pith2026250206453,
  author       = {Pith},
  title        = {Pith review of: MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7MRDW5W4}},
  note         = {Machine review of arXiv:2502.06453}
}
read the original abstract

Large language models have demonstrated impressive performance on challenging mathematical reasoning tasks, which has triggered the discussion of whether the performance is achieved by true reasoning capability or memorization. To investigate this question, prior work has constructed mathematical benchmarks when questions undergo simple perturbations -- modifications that still preserve the underlying reasoning patterns of the solutions. However, no work has explored hard perturbations, which fundamentally change the nature of the problem so that the original solution steps do not apply. To bridge the gap, we construct MATH-P-Simple and MATH-P-Hard via simple perturbation and hard perturbation, respectively. Each consists of 279 perturbed math problems derived from level-5 (hardest) problems in the MATH dataset (Hendrycksmath et. al., 2021). We observe significant performance drops on MATH-P-Hard across various models, including o1-mini (-16.49%) and gemini-2.0-flash-thinking (-12.9%). We also raise concerns about a novel form of memorization where models blindly apply learned problem-solving skills without assessing their applicability to modified contexts. This issue is amplified when using original problems for in-context learning. We call for research efforts to address this challenge, which is critical for developing more robust and reliable reasoning models.

Figures

Figures reproduced from arXiv: 2502.06453 by the authors.

Figure 1
Figure 1. Left: The overview of MATH-Perturb Benchmark. Right: An example of the original problem, its simple perturbation, its hard perturbation, and the corresponding model responses that overfit the short-cut solution. The simple perturbation to the problem is non-essential, so the modified problem can be solved using the same method as the original problem. The hard perturbation changes the problem fundamentally and it re… view at source ↗
Figure 2
Figure 2. Performance on MATH-P-Simple, MATH-P-Hard, and the corresponding Original problems. We observe performance degrada￾tions across all models on MATH-P-Hard. 2. Dataset Curation Origin of the Dataset. We choose the popular MATH benchmark (Hendrycks et al., 2021), which contains chal￾lenging mathematical reasoning problems sourced from American high school mathematics competitions such as the AMC 10, AMC 12, and AIME. E… view at source ↗
Figure 3
Figure 3. Illustration of the annotation process for MATH-P-Simple and MATH-P-Hard. text-embedding-3-large embedding model. The distribu￾tions of the normalized edit distance and the cosine similari￾ties are shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The distributions of edit distances and cosine similarities of embeddings between the perturbed problems and the origi￾nal problems. The edit distances are normalized by the lengths of the original problems. The embedding model is OpenAI’s text-embedding-3-large. We al…
Figure 5
Figure 5. Figure 5: An example of memorization coupled with incorrect reasoning: The model incorrectly reduces the modified condition to the original condition, and then follows the original reasoning pattern. The correct answer is 1/36 . We manually performed 20 repeated trials and found…
Figure 6
Figure 6. Figure 6: An example of memorizing the desired outcome. The model outputs all integer values instead of the smallest integer value. The correct answer is 10 . We manually performed 20 repeated trials and found that o1-mini has a pass rate of 75%. All the 25% errors are due to th…
Figure 7
Figure 7. Figure 7: The error rates (%) of the models without and with the original problem and solution as the in-context learning (ICL) example. For MATH-P-Hard, we decompose the influences of in-context learning into ICL effect (the down arrow ↓), which reduces the error rates, and mis…
Figure 8
Figure 8. Figure 8: The error rates (%) of the models without and with the original problem and solution as the in-context learning (ICL) example. For MATH-P-Hard, we decompose the influences of in-context learning into ICL effect (the down arrow ↓), which reduces the error rates, and mis…
Figure 9
Figure 9. Figure 9: The effect of scaling up inference-time compute. We report pass@k and self-consistency (SC) accuracies for different numbers of solutions k. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. New Lower and Upper Bounds for the Grothendieck Constant

    cs.CC 2026-08 conditional novelty 8.0 of 10

    New rigorous bounds pin the Grothendieck constant to [6pi/11, pi/(2 log(1+sqrt 2)) - 10^-4], improving both known lower and upper bounds.

  2. Robust Reasoning Benchmark

    cs.LG 2026-03 unverdicted novelty 7.0 of 10

    The Robust Reasoning Benchmark shows frontier LLMs are mostly resilient to textual perturbations on AIME problems while open-weight models suffer up to 54% accuracy drops and exhibit accuracy decay on later problems d...

  3. UQ: Assessing Language Models on Unsolved Questions

    cs.CL 2025-08 unverdicted novelty 7.0 of 10

    Unsolved Stack Exchange questions can serve as a dynamic benchmark: the best tested model passes validator screening on only 15% of 500 questions.

  4. Long-Horizon AI Research for Grothendieck Constant: A Case Study in Human-AI Mathematical Collaboration

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A human-AI team reports new bounds on the Grothendieck constant, 6π/11 ≤ K_G ≤ π/(2 log(1+√2)) - 3.47e-4, crediting an AI model with the core idea for the lower bound.

  5. Intern-S1-MO: Long-horizon Reasoning Agent for Olympiad?Level Mathematical Problem Solving

    cs.CL 2025-12 conditional novelty 6.0 of 10

    A multi-round, lemma-memory reasoning agent with hierarchical RL reaches reported gold-medal-level scores on Olympiad math benchmarks, though the proof-based scores are self-graded.

  6. Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility

    cs.LG 2026-08 conditional novelty 5.0 of 10

    The paper formalizes test-time scaling into three regimes, introduces a discovery-stability profile for repeated-sampling evaluation, and releases nearly two million reasoning traces.

  7. Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A 235-item multimodal stress-test shows frontier closed models outpace open-weight peers by ~10% and leaves shared failures on counting, spatial, and character-level tasks.

  8. InfoSynth: Information-Guided Benchmark Synthesis for LLMs

    cs.CL 2026-01 conditional novelty 5.0 of 10

    Using KL divergence and entropy on embeddings, InfoSynth scores benchmark novelty/diversity and guides a genetic pipeline that generates new, code-verified Python problems from seeds.

  9. Decoupling Knowledge and Reasoning in LLMs: An Exploration Using Cognitive Dual-System Theory

    cs.AI 2025-07 conditional novelty 5.0 of 10

    The accuracy gap between fast and slow thinking is proposed as a measure of reasoning contribution, and is used to show that reasoning is domain-specific, scaling mainly reduces overthinking, and knowledge and reasoni...

  10. Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Across ten LLMs, masking the final answer inside a complete reasoning chain causes a 26.9-point accuracy drop, evidence that models anchor to answers, not reasoning templates.

  11. SciDA: Scientific Dynamic Assessor of LLMs

    cs.CL 2025-06 conditional novelty 5.0 of 10

    SciDA is a dynamically initialized, multi-discipline olympiad benchmark that shows LLMs perform substantially worse when problem variables are randomized, which the authors attribute to memorization of fixed numerical...

  12. GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    The paper releases GSM-Plus-BN, a human-verified Bengali translation of the GSM-Plus perturbed math benchmark, and reports accuracy baselines for six open LLMs under standard and CoT prompting.

  13. A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models

    cs.CL 2025-09 conditional novelty 4.0 of 10

    A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.

Reference graph

Works this paper leans on

34 extracted references · 6 canonical work pages · cited by 13 Pith papers

  1. [2]

    Benchmark Statistics Table

    Version information of the models Model Provider Version/Link Gemini-2.0-flash-thinking-exp Google DeepMind 2024-12-19 o1-preview OpenAI 2024-09-12 o1-mini OpenAI 2024-09-12 Gemini-2.0-flash-exp Google DeepMind 2024-12-11 Gemini-1.5-pro Google DeepMind gemini-1.5-pro-002 GPT-4o OpenAI 2024-08-06 GPT-4-turbo OpenAI 2024-04-09 Claude-3.5-sonnet Anthropic 20...

    work page 2024

  2. [3]

    What does it mean for a language model to preserve privacy? InProceedings of the 2022 ACM conference on fairness, accountability, and transparency, pages 2280–2292,

    Hannah Brown, Katherine Lee, Fatemehsadat Mireshghal- lah, Reza Shokri, and Florian Tramèr. What does it mean for a language model to preserve privacy? InProceedings of the 2022 ACM conference on fairness, accountability, and transparency, pages 2280–2292,

    work page 2022

  3. [6]

    Training ver- ifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plap- pert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training ver- ifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021a. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukas...

    arXiv

  4. [8]

    Scaling laws and interpretability of learning from re- peated data

    Danny Hernandez, Tom Brown, Tom Conerly, Nova Das- Sarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from re- peated data. arXiv preprint arXiv:2205.10487,

    arXiv

  5. [9]

    Privacy Implications of Retrieval-Based Language Models

    Yangsibo Huang, Samyak Gupta, Zexuan Zhong, Kai Li, and Danqi Chen. Privacy implications of retrieval-based language models. arXiv preprint arXiv:2305.14888 ,

    work page Pith review arXiv

  6. [10]

    Oth- ellogpt learned a bag of heuristics

    jylin04, JackS, Adam Karvonen, and Can. Oth- ellogpt learned a bag of heuristics. https: //www.lesswrong.com/posts/gcpNuEZnxAPayaKBY/ othellogpt-learned-a-bag-of-heuristics-1 . Accessed on Date (2025-01-28). Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models. arXiv preprint arXiv:2310.13771,

    arXiv 2025

  7. [11]

    Deduplicating training data makes language mod- els better

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language mod- els better. arXiv preprint arXiv:2107.06499,

    arXiv

  8. [12]

    Numina- math

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zi- han Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numina- math. [https://github.com/project-numina/ aimo-progress-prize](https://github.com/ project-numina/aimo-progress-prize/blob...

    arXiv

Show all 34 references
  1. [13]

    Gsm- symbolic: Understanding the limitations of mathemati- cal reasoning in large language models

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, On- cel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm- symbolic: Understanding the limitations of mathemati- cal reasoning in large language models. arXiv preprint arXiv:2410.05229,

    arXiv

  2. [14]

    Scalable extraction of training data from (production) language models

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christo- pher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035,

    arXiv

  3. [15]

    Arithmetic without algorithms: Language mod- els solve math with a bag of heuristics

    Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algorithms: Language mod- els solve math with a bag of heuristics. arXiv preprint arXiv:2410.21272,

    arXiv

  4. [16]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal

    URL https://openai.com/ index/openai-o1-system-card/ . Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan...

    2021

  5. [17]

    doi: 10.18653/ v1/2021.naacl-main.168

    Asso- ciation for Computational Linguistics. doi: 10.18653/ v1/2021.naacl-main.168. URL https://aclanthology. org/2021.naacl-main.168/. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gp...

    2021 arXiv

  6. [18]

    Ai-assisted generation of difficult math questions

    Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Ji- atong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, et al. Ai-assisted generation of difficult math questions. arXiv preprint arXiv:2407.21009,

    arXiv

  7. [19]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junx- iao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  8. [20]

    Large language models can be easily distracted by irrelevant context

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR, 2023a. 11 MATH-Pertu...

    arXiv

  9. [21]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024a. Gemma ...

    arXiv

  10. [22]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171,

    arXiv

  11. [23]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574,

    arXiv

  12. [24]

    Evaluating copyright takedown methods for lan- guage models

    Boyi Wei, Weijia Shi, Yangsibo Huang, Noah A Smith, Chiyuan Zhang, Luke Zettlemoyer, Kai Li, and Peter Hen- derson. Evaluating copyright takedown methods for lan- guage models. arXiv preprint arXiv:2406.18664,

    arXiv

  13. [25]

    Reasoning or reciting? ex- ploring the capabilities and limitations of language models through counterfactual tasks

    Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob An- dreas, and Yoon Kim. Reasoning or reciting? ex- ploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477,

    arXiv

  14. [26]

    On memorization of large language models in logical reasoning

    Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123,

    arXiv

  15. [27]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122,

    arXiv

  16. [28]

    A careful examination of large 12 MATH-Perturb: Benchmarking LLMs’ Math Reasoning Abilities against Hard Perturbations language model performance on grade school arithmetic

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Cather- ine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. A careful examination of large 12 MATH-Perturb: Benchmarking LLMs’ Math Reasoning Abilities against Hard Perturbations language model performance ...

    arXiv

  17. [29]

    On large language models’ se- lection bias in multi-choice questions

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. On large language models’ se- lection bias in multi-choice questions. arXiv preprint arXiv:2309.03882,

    arXiv

  18. [30]

    Instruction-following evaluation for large language mod- els

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language mod- els. arXiv preprint arXiv:2311.07911,

    arXiv

  19. [31]

    Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning ro- bustness of vision language models

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning ro- bustness of vision language models. arXiv preprint arXiv:2411.00836,

    arXiv

  20. [32]

    Version Information of the Models We consider the following models in the paper

    13 MATH-Perturb: Benchmarking LLMs’ Math Reasoning Abilities against Hard Perturbations A. Version Information of the Models We consider the following models in the paper. • long-CoT models: o1-preview, o1-mini (OpenAI, 2024), Gemini 2.0 flash thinking • closed-source models: ...

    2024

  21. [34]

    Model MATH-P-Simple MATH-P-Hard ICL w

    Performance comparisons without and with the original problem and solution as the in-context learning example. Model MATH-P-Simple MATH-P-Hard ICL w. original ICL (random) ICL w. original ICL (random) o1-mini 94.98 92.83 78.49 75.99 Gemini-1.5-pro 88.17 75.99 60.57 51.97 GPT-4...

    2021

  22. [2021]

    Copy- bench: Measuring literal and non-literal reproduction of copyright-protected text in language model generation

    Tong Chen, Akari Asai, Niloofar Mireshghallah, Sewon Min, James Grimmelmann, Yejin Choi, Hannaneh Ha- jishirzi, Luke Zettlemoyer, and Pang Wei Koh. Copy- bench: Measuring literal and non-literal reproduction of copyright-protected text in language model generation. arXiv prepr...

    arXiv

  23. [2022]

    Sparks of artificial general intelligence: Early experiments with gpt-4

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712,

    arXiv

  24. [2023]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787,

    arXiv

  25. [2024]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al

    URL https://arxiv.org/abs/2404.14219. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  26. [2025]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al

    URL https://arxiv.org/abs/2501.12948. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.