Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current reasoning LLMs, even after RL fine-tuning, master exploratory generalization but fail at compositional and transformative math reasoning.

desk verdict OMEGA is a useful new benchmark with solid exploratory results, but its compositional and transformative claims are undercut by test items that don't force the intended skills. read the letter →

arxiv 2506.18880 v1 pith:4F3AOGJS submitted 2025-06-23 cs.CL cs.AI

Yiyou Sun , Shawn Hu , Georgia Zhou , Ken Zheng , Hannaneh Hajishirzi , Nouha Dziri , Dawn Song This is my paper · ORCID
classification cs.CLcs.AI
keywords OMEGAout-of-distributiongeneralizationmathematicalreasoningexploratorycompositionaltransformativereinforcementlearningchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces OMEGA, a benchmark of programmatically generated math problems designed to isolate three kinds of out-of-distribution generalization: exploratory (applying a known skill to harder instances), compositional (combining skills learned separately), and transformative (abandoning a familiar tactic for a genuinely new one). Its central claim is that current reasoning models, including reinforcement-learning-tuned ones, perform well only close to the training distribution. As complexity rises, accuracy on exploratory tasks degrades sharply, and RL fine-tuning mostly reinforces in-distribution patterns: gains on exploratory tasks are real but plateau, compositional gains are modest, and transformative gains are near zero. A sympathetic reader should care because the benchmark turns vague talk of "mathematical creativity" into measurable, per-skill failure modes.

What carries the argument

The carrying object is OMEGA itself: a collection of 40 templated problem generators across arithmetic, algebra, combinatorics, number theory, geometry, and logic & puzzles, each with a scalar complexity measure $\delta(\theta)$ that ranks instances. Training-test pairs are constructed by fixing the template and varying the parameter region: low-complexity instances for training and higher-complexity instances for exploratory tests; two single-skill families in isolation for training and their integrated combination for compositional tests; familiar-tactic problems for training and problems whose familiar tactic is intractable for transformative tests. Solutions are generated and verified programmatically, so the benchmark can scale indefinitely. This design is what lets the authors attribute a performance drop to a specific missing reasoning skill rather than to unseen problem content.

What would settle it

Take the transformative test set (for example, the 10×10 blocked-grid path-counting problem) and solve every item using the familiar tactic from the corresponding training problems, with unlimited compute and a simulator. If many items yield to the familiar tactic, the tests do not force a paradigm shift and the near-zero accuracy cannot be attributed to missing transformative reasoning; if nearly all items resist the familiar tactic but fall to the intended alternative strategy, the paper's isolation claim is supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a tripartite diagnosis. First, frontier long-chain-of-thought models show near-ceiling performance at low complexity but decline to near zero as task complexity increases; trace analysis attributes this to overthinking, error spirals, and a growing preference for guesses over computation rather than arithmetic error alone. Second, RL fine-tuning on easy instances produces substantial gains on in-distribution and moderately harder exploratory tasks, but the gains plateau on high-complexity instances. Third, models trained separately on two skills fail to integrate them on compositional tests, and RL training that improves the isolated skills does not transfer to the composed task. Fourth, on transformative tests that require a different solution paradigm, RL shows negligible improvement and can even reverse prior ability. The paper concludes that current RL amplifies learned strategies but does not induce new reasoning patterns.

Load-bearing premise

The load-bearing premise is that each OMEGA problem family and complexity level isolates exactly the intended skill: that $\delta(\theta)$ measures genuine reasoning difficulty, that compositional tests cannot be solved by applying the two skills one after another, and that transformative tests cannot be solved by the familiar tactic; if any test admits an unintended shortcut, the observed failures would not measure the axis the paper claims.

Editorial extensions

If this is right

  • If the OMEGA findings hold, current reasoning LLMs cannot be treated as reliable solvers on high-complexity mathematics even with long chain-of-thought and test-time search.
  • RL fine-tuning is a reliable way to broaden competence within a learned strategy, but it should not be expected to produce compositional integration by itself.
  • Training setups that isolate skills need to be paired with explicit composition training, because mastering the components does not yield the composite.
  • Benchmarks that measure only final accuracy on static problem sets will miss the plateau: performance on exploratory generalizations can improve while transformative ability stays at zero.
  • The three-axis typology provides a reusable diagnostic grid for evaluating future training methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to use OMEGA-style train/test splits to test whether test-time search (e.g., sampling many candidates and selecting by verifier) rescues compositional tasks; the paper's Pass@k data suggest the benefit vanishes at high complexity, but the interaction with compositional and transformative tests is untested.
  • The failure of RL on transformative tasks implies that reward signals based on verifiable answers may be too sparse to discover a new solution paradigm; process rewards or demonstrations of alternate strategies would be a direct training change the paper's framework could evaluate.
  • If the isolation assumption holds, the same template-pair design could be moved to code generation or scientific reasoning, where "exploratory-compositional-transformative" maps onto refactoring, library composition, and novel algorithm design.
  • The near-zero transformative results also suggest a testable prediction: models pretrained on diverse solution strategies across many templates should show larger transformative transfer than RL-finetuned models, since transformation requires exposure to paradigm shifts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces OMEGA, a programmatically generated math benchmark with matched training-test pairs along three axes of out-of-distribution generalization: exploratory, compositional, and transformative. The authors evaluate four frontier LLMs and fine-tune Qwen2.5-7B-Instruct and Qwen2.5-Math-7B with GRPO, reporting that accuracy degrades sharply with problem complexity, that RL improves explorative generalization but yields only modest compositional gains, and that transformative reasoning remains near zero. The paper additionally provides chain-of-thought error-pattern analyses, Pass@k scaling experiments, and an ablation of compositional skill pairings.

Significance. If its construct validity holds, OMEGA is a useful and unusually controlled testbed: the problems are templated with programmatic verification, the training distributions are explicit, and the RL results replicate across two Qwen variants and several domains. This makes the paper's negative result about RL and compositional/transformative generalization falsifiable and reproducible, and the release of code and dataset is a concrete asset. The main caveat is that the central claim depends on test items actually forcing the intended composition or transformation; this premise is currently asserted rather than verified, and at least two representative items admit simpler solution routes.

major comments (3)
  1. [§2.4–2.5, Tables 7 and 9] The load-bearing premise of the compositional and transformative experiments is that each test item can only be solved by the specified composed or reframed strategy, but the manuscript provides no independent check of this premise: the 'Decomposition' column in Table 7 and the 'Needed insight' column in Table 9 are post hoc descriptions of intended routes, not demonstrations that no simpler route exists. Two of the paper's own examples illustrate the risk. In Transformative Setting 1 (Table 9), rank(E_n) can be found by observing that the rows alternate between (1,0,1,0,...) and (0,1,0,1,...), so rank(E_n) ≤ 2 for n ≥ 2, without the named outer-product decomposition; a model failing this item may be failing symbolic-n out-of-distribution generalization rather than 'shifting to a novel paradigm.' In Compositional Setting 2 (Table 7), the number of rectangles in a regular 12-gon has the closed form C(6,2)=15 obtained by choosing two of the six diameters, and this route does not require either the trained polygon-rotation skill or the pattern-matching skill. Because the near-zero OOD accuracies in Figures 10–11 are the main evidence for the paper's central qualitative conclusions, the authors should add a validation step—for example, human or solver-generated solution protocols for every OOD item showing that the intended route is necessary, or an automated detector verifying that the trained skills are genuinely used.
  2. [§B.1, Figures 10–11] The headline quantitative claims rest on very small OOD test sets and single-seed RL runs. Section B.1 states that compositional and transformative OOD sets contain 20–50 samples per setting, and the GRPO runs use seed 3 only. With 20 items, the difference between 0% and 10% post-RL accuracy is within binomial sampling noise, and with no seed variance reported, the conclusion that RL yields 'little to no improvement' is not statistically supported as stated. Report per-setting test sizes, confidence intervals, or at least 3–5 seeds for the headline settings, and ideally the per-checkpoint trajectories rather than the average over the last five checkpoints.
  3. [§3.2.3, Figure 11] The matrix-rank result, reported as 'base model achieved decent OOD performance (70%)' with post-RL performance dropping 30 points, is an important exception to the general pattern and is currently supported by a single unseeded observation. Before drawing the conclusion that RL 'entrenches brittle heuristics,' the authors should verify that this drop is reproducible across seeds and is not an artifact of checkpoint selection or evaluation noise, and should report the per-instance trajectories that show the hypothesized entrenchment. This matters because the result is used to support the paper's characterization of how RL fails on transformative tasks.
minor comments (5)
  1. [Figure 7 and text] Figure 7's caption refers to 'geometry rotation problems' while the accompanying text reports Pass@k on 'letter distribution' problems; the rotation results appear in Figure 13, so the caption and cross-reference should be corrected.
  2. [Throughout] The paper uses 'exploratory' and 'explorative' interchangeably (e.g., Section 2.3 versus Figure 8), and similarly 'transformative' and 'transformational'; a single term per axis would improve precision.
  3. [Tables 11–12] Tables 11 and 12 are captioned as 'corresponding to Figure 11,' but the compositional-setting ablation tracks the settings reported in Figure 10; the cross-reference appears to be mislabeled.
  4. [Appendix D] Appendix D's token-cost estimate assumes 'each backtrack call translates to 20 tokens' without justification; since this estimate is used to argue that context length is not the bottleneck, the constant should be justified or replaced by an upper-bound accounting based on actual rollout lengths.
  5. [Table 1] The entry 'OMEGA (Ours)' in Table 1 lists verification as 'N/A; Correct by Construction,' which is undercut by the appendix's use of OpenCV, grid search, and DP solvers; the verification column should match the actual pipeline.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: OMEGA's claims are empirical measurements from independently generated test problems, not derivations that reduce to their inputs.

full rationale

The paper is an evaluation benchmark, not a derivation. The three generalization axes are operational definitions (Table 2, §§2.3–2.5), and the reported accuracies are measured on programmatically generated instances with independent symbolic, numerical, or graphical verification. The qualitative conclusions—that RL improves exploratory but not compositional or transformative reasoning—are empirical summaries of those measurements, not consequences of the definitions by construction. No fitted parameter is renamed as a prediction, and no uniqueness theorem or prior-work ansatz is invoked to force the central result. The only self-citations (e.g., [32] for the observation that LLMs rely on narrow strategies) support background motivation and are independently reinforced by the paper's own CoT analyses, so they are not load-bearing. The reviewer's construct-validity concern—that some transformative/compositional items (e.g., rank(E_n) via row alternation, or the 12-gon rectangle count via diameter selection) may admit familiar-tactic shortcuts—bears on whether the benchmark actually isolates the intended skills, not on whether the paper's claims reduce to its benchmark definitions by construction. A benchmark can be imperfectly valid without being circular, and no equation, fitted parameter, or self-citation chain makes the stated conclusions equivalent to the inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on benchmark design assumptions rather than on fitted constants. The main ingredients are the hand-chosen complexity cutoffs, the assumed validity of the three generalization axes, and the assumption that test problems force the intended strategy. No new physical or mathematical entities are postulated.

free parameters (3)
  • Complexity cutoff delta_0 = Per template, chosen so base model accuracy is below 50% on training data
    Determines the train/test split for exploratory generalization; hand-set in Section 2.3.
  • Test-set sizes = 100 per complexity level; 20-50 per compositional/transformative setting
    Chosen by convention, with no power analysis; documented in Appendix B.1.
  • Token-per-backtrack multiplier = 20
    Used in Appendix D to estimate token feasibility; an arbitrary conversion factor.
assumptions (5)
  • domain assumption Boden's typology of creativity (exploratory, combinatorial, transformative) is a valid decomposition of mathematical generalization.
    The whole benchmark structure rests on these three axes being separable; Sections 1 and 2.
  • domain assumption Template complexity measure delta(theta) monotonically corresponds to reasoning difficulty.
    Exploratory results assume higher delta means genuinely harder reasoning; Sections 2.2 and 2.3.
  • domain assumption Compositional test problems require true integration of the two trained skills and cannot be solved by applying them sequentially.
    Stated as a design principle in Section 2.4, with no formal or empirical check that no shortcut solution exists.
  • domain assumption Transformative test problems are intractable under the familiar trained tactic.
    Test items are engineered to make conventional strategies fail (Section 2.5); this is an assertion about the problem space, not a proven property.
  • domain assumption Programmatically generated answers are correct.
    Verification is symbolic, numerical, or graphical (Section 2.1); no independent audit is supplied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization." pith.science (2026). https://pith.science/paper/4F3AOGJS

@misc{pith2026250618880,
  author       = {Pith},
  title        = {Pith review of: OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4F3AOGJS}},
  note         = {Machine review of arXiv:2506.18880}
}
read the original abstract

Recent large-scale language models (LLMs) with long Chain-of-Thought reasoning-such as DeepSeek-R1-have achieved impressive results on Olympiad-level mathematics benchmarks. However, they often rely on a narrow set of strategies and struggle with problems that require a novel way of thinking. To systematically investigate these limitations, we introduce OMEGA-Out-of-distribution Math Problems Evaluation with 3 Generalization Axes-a controlled yet diverse benchmark designed to evaluate three axes of out-of-distribution generalization, inspired by Boden's typology of creativity: (1) Exploratory-applying known problem solving skills to more complex instances within the same problem domain; (2) Compositional-combining distinct reasoning skills, previously learned in isolation, to solve novel problems that require integrating these skills in new and coherent ways; and (3) Transformative-adopting novel, often unconventional strategies by moving beyond familiar approaches to solve problems more effectively. OMEGA consists of programmatically generated training-test pairs derived from templated problem generators across geometry, number theory, algebra, combinatorics, logic, and puzzles, with solutions verified using symbolic, numerical, or graphical methods. We evaluate frontier (or top-tier) LLMs and observe sharp performance degradation as problem complexity increases. Moreover, we fine-tune the Qwen-series models across all generalization settings and observe notable improvements in exploratory generalization, while compositional generalization remains limited and transformative reasoning shows little to no improvement. By isolating and quantifying these fine-grained failures, OMEGA lays the groundwork for advancing LLMs toward genuine mathematical creativity beyond mechanical proficiency.

Figures

Figures reproduced from arXiv: 2506.18880 by the authors.

Figure 1
Figure 1. Examples of training-test pairs designed to test distinct generalization capabilities: (a) Explorative Generalization increases complexity within the same frame of thinking (e.g., extending geometric reasoning from an octagon to a dodecagon). (b) Compositional Generalization requires integrating multiple learned strategies (e.g., combining GCD and root-finding for polynomials). (c) Transformative Generalization dema… view at source ↗
Figure 2
Figure 2. Two examples of compositional generalization in our training/test setup. Each case presents training problems from two separate templates that exercise particular reasoning skills that the model must master, and a test problem that composes the skills. More examples can be found at Appendix A. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Exact-match accuracy of four top-tier LLMs on OMEGA, plotted against increasing complexity levels. As the complexity increases, performance degrades and goes to zero. We provide complexity analysis to typical problems to ensure they are within the models’ output length as detailed in §D. scaling limitations. CoT reasoning remains effective only below a critical complexity threshold, beyond which performance rapidly … view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 4
Figure 4. Figure 4: The percentage of incorrect responses exhibiting two distinct error patterns: correct → incorrect shift (blue bars) where models initially provided correct answers but changed to incorrect ones through overthinking, and reasoning spirals (red bars) where models remaine…
Figure 5
Figure 5. Figure 5: Performance and reasoning patterns across six mathematical task domains showing accuracy degradation and verification behavior as problem complexity increases. Models often reach the correct answer early in the response but continue generating unnecessary verification …
Figure 6
Figure 6. Figure 6: Reasoning trace analysis with distribu￾tion of two specific types of reasoning steps and correctness for the computation step, tested on Ma￾trix Rank problem family. As problem difficulty increases, the model spends less of its CoT on explicit calculations (gold square…
Figure 7
Figure 7. Figure 7: Pass@k performance of the advanced LLMs across complexity levels for geometry rotation problems. 3.1.3 Can More Inference-Time Compute Solve Harder Problems? Helps at Moderate Complexity, but Gains Plateau at Higher Levels To investigate how inference-time compute cont…
Figure 8
Figure 8. Figure 8: Performance comparison of Qwen2.5-7B-Instruct before and after RL on OMEGA under the exploratory generalization setting (Section 2.1). Each problem setting is represented by concatenated bars: In-distribution (ID) accuracy (blue) and Out-of-distribution (OOD) accuracy …
Figure 9
Figure 9. Figure 9: Generalization across complexity levels. Models were trained with data up to a certain complexity level (y-axis) and evaluated on problems from levels 1 to 5 (x-axis). Cells marked ‘ID’ represent in-distribution evaluations where the test complexity level was included …
Figure 10
Figure 10. Figure 10: Performance comparison of Qwen2.5-7B-Instruct on OMEGA under the compositional generaliza￾tion setting. The model’s ability to integrate reasoning strategies from two problem families is assessed. For each setting, accuracies are reported on the individual in-distribu…
Figure 11
Figure 11. Figure 11: Performance comparison of Qwen2.5-7B-Instruct on OMEGA under the transformational gen￾eralization setting. The model’s ability to adopt qualitatively new reasoning strategies is evaluated. For each setting, we report accuracies on the source problem family and on the …
Figure 12
Figure 12. Figure 12: Performance comparison of state-of-the-art LLMs on mathematical reasoning tasks in [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Pass@k performance of the advanced LLMs across complexity levels for geometry rotation problems. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: Comparison of RL fine-tuning effectiveness in the explorative generalization setting [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Ask-E: An Environment for Calibrated Question Generation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A language model trained only to write questions that split two weaker solvers improves at solving math problems, while even frontier models calibrate less than half the time.

  2. The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Trace-grounded parametric profiling of three synthetic counting tasks shows current video-language models only count reliably at low event counts and low rates, and final-answer accuracy masks poor timestamp-level eve...

  3. PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages

    cs.CL 2026-07 conditional novelty 6.0 of 10

    PluraMath extends PolyMath with human-validated math problems in 18 mid-to-extreme low-resource languages and benchmarks 27 reasoning LLMs, finding a persistent high- vs low-resource performance gap.

  4. RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments

    cs.CL 2025-11 conditional novelty 6.0 of 10

    Training a 1.5B language model on 400 adaptively-difficult, procedurally generated, verifiable environments improved average reasoning-benchmark performance by 3.37% absolute, compared with 0.49% from continuing origi...

  5. Re:Form -- Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A minimal-prior pipeline with automated data curation and verifier-driven RL lets small LLMs generate verifiable Dafny specifications and beat larger proprietary models on a synthetic compositional benchmark.

  6. Rethinking the Illusion of Thinking

    cs.AI 2025-07 conditional novelty 5.0 of 10

    Reasoning models' Towers of Hanoi failures persist under stepwise prompting, while River Crossing failures mostly vanish when tests are restricted to solvable configurations.

Reference graph

Works this paper leans on

46 extracted references · 22 canonical work pages · cited by 6 Pith papers

  1. [1]

    Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025

    Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models, 2025

    2025

  2. [2]

    The mathematics of deepmind models.The Mathematics of DeepMind Models (November 01, 2024), 2024

    Noguer I Alonso et al. The mathematics of deepmind models.The Mathematics of DeepMind Models (November 01, 2024), 2024

    work page 2024

  3. [3]

    2024 aime ii problems/problem 1, 2024

    Art of Problem Solving. 2024 aime ii problems/problem 1, 2024. Accessed: 2025-03-27

    work page 2024

  4. [4]

    Creativity and artificial intelligence.Artificial intelligence, 103(1-2):347–356, 1998

    Margaret A Boden. Creativity and artificial intelligence.Artificial intelligence, 103(1-2):347–356, 1998

    work page 1998

  5. [5]

    Compositionality and generalization in emergent languages.arXiv preprint arXiv:2004.09124, 2020

    Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages.arXiv preprint arXiv:2004.09124, 2020

    arXiv 2004

  6. [6]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  7. [7]

    Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality, 2023

    2023

  8. [8]

    Metamathqa, 2023

    Hugging Face. Metamathqa, 2023

    work page 2023

Show all 46 references
  1. [9]

    Improving text-to-sql evaluation methodology.arXiv preprint arXiv:1806.09029, 2018

    Catherine Finegan-Dollak, Jonathan K Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasi- vam, Rui Zhang, and Dragomir Radev. Improving text-to-sql evaluation methodology.arXiv preprint arXiv:1806.09029, 2018

    2018 arXiv

  2. [10]

    Deep learning with long short-term memory networks for financial market predictions

    Thomas Fischer and Christopher Krauss. Deep learning with long short-term memory networks for financial market predictions. FAU Discussion Papers in Economics 11/2017, 2017

    2017

  3. [11]

    Omni-math: A universal olympiad level mathematic benchmark for large language models.arXiv preprint arXiv:2410.07985, 2024

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models.arXiv preprint arXiv:2410.07985, 2024

    arXiv

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scienti...

    2024

  6. [14]

    Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456, 2025

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456, 2025

    2025 arXiv

  7. [15]

    Compositionality decomposed: How do neural networks generalise?Journal of Artificial Intelligence Research, 67:757–795, 2020

    Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise?Journal of Artificial Intelligence Research, 67:757–795, 2020. 15

    2020

  8. [16]

    Measuring compositional generalization: A comprehensive method on realistic data.arXiv preprint arXiv:1912.09713, 2019

    Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measuring compositional generalization: A comprehensive method on realistic data.arXiv preprint arXiv:191...

    1912 arXiv

  9. [17]

    Measuring compositional generalization: A comprehensive method on realistic data, 2020

    Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A compreh...

    2020

  10. [18]

    Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks

    Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. InInternational conference on machine learning, pages 2873–2882. PMLR, 2018

    2018

  11. [19]

    Solving quantitative reasoning problems with language models, 2022

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models, 2022

    2022

  12. [20]

    Numinamath

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Ra- sul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https://huggingface.co/AI-MO/NuminaMa...

    2024

  13. [21]

    Probing out-of-distribution generalization in machine learning for materials, 2024

    Kangming Li, Andre Niyongabo Rubungo, Xiangyun Lei, Daniel Persaud, Kamal Choudhary, Brian DeCost, Adji Bousso Dieng, and Jason Hattrick-Simpers. Probing out-of-distribution generalization in machine learning for materials, 2024

    2024

  14. [22]

    Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers

    Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2024

  15. [23]

    Towards out-of-distribution generalization: A survey, 2023

    Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey, 2023

    2023

  16. [24]

    Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.arXiv preprint arXiv:2505.24864, 2025

    Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.arXiv preprint arXiv:2505.24864, 2025

    2025 arXiv

  17. [25]

    Compositional generalization by learning analytical expressions.Advances in Neural Information Processing Systems, 33:11416–11427, 2020

    Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. Compositional generalization by learning analytical expressions.Advances in Neural Information Processing Systems, 33:11416–11427, 2020

    2020

  18. [26]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229, 2024

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229, 2024

    arXiv

  19. [27]

    Beyond lines and circles: Unveiling the geometric reasoning gap in large language models.arXiv preprint arXiv:2402.03877, 2024

    Spyridon Mouselinos, Henryk Michalewski, and Mateusz Malinowski. Beyond lines and circles: Unveiling the geometric reasoning gap in large language models.arXiv preprint arXiv:2402.03877, 2024

    2024 arXiv

  20. [28]

    Gsm8k, April 2022

    OpenAI. Gsm8k, April 2022

    2022

  21. [29]

    Learning to reason with llms, September 2024

    OpenAI. Learning to reason with llms, September 2024

    2024

  22. [30]

    Math 500, November 2024

    OpenAI. Math 500, November 2024

    2024

  23. [31]

    The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity.arXiv preprint arXiv:2506.06941, 2025

    Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity.arXiv preprint arXiv:2506.06941, 2025

    2025 arXiv

  24. [32]

    Climbing the ladder of reasoning: What llms can-and still can’t-solve after sft?arXiv preprint arXiv:2504.11741, 2025

    Yiyou Sun, Georgia Zhou, Hao Wang, Dacheng Li, Nouha Dziri, and Dawn Song. Climbing the ladder of reasoning: What llms can-and still can’t-solve after sft?arXiv preprint arXiv:2504.11741, 2025

    2025

  25. [33]

    Mathscale: Scaling instruction tuning for mathematical reasoning, 2024

    Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning, 2024

    2024

  26. [34]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic Team. The claude 3 model family: Opus, sonnet, haiku. 16

  27. [35]

    Openmathinstruct-1: A 1.8 million math instruction tuning dataset, 2024

    Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-1: A 1.8 million math instruction tuning dataset, 2024

    2024

  28. [36]

    Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization, 2024

    Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization, 2024

    2024

  29. [37]

    Towards a theoretical framework of out-of-distribution generalization, 2021

    Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards a theoretical framework of out-of-distribution generalization, 2021

    2021

  30. [38]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  31. [39]

    Scaling and beyond: Advancing spatial reasoning in mllms requires new recipes.arXiv preprint arXiv:2504.15037, 2025

    Huanyu Zhang, Chengzu Li, Wenshan Wu, Shaoguang Mao, Yifan Zhang, Haochen Tian, Ivan Vuli ´c, Zhang Zhang, Liang Wang, Tieniu Tan, et al. Scaling and beyond: Advancing spatial reasoning in mllms requires new recipes.arXiv preprint arXiv:2504.15037, 2025

    2025 arXiv

  32. [40]

    Evaluating the performance of large language models on gaokao benchmark, 2024

    Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark, 2024

    2024

  33. [41]

    Can models learn skill composition from examples?Advances in Neural Information Processing Systems, 37:102393–102427, 2024

    Haoyu Zhao, Simran Kaur, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Can models learn skill composition from examples?Advances in Neural Information Processing Systems, 37:102393–102427, 2024

    2024

  34. [42]

    Gsm-infinite: How do your llms behave over infinitely increasing context length and reasoning complexity?arXiv preprint arXiv:2502.05252, 2025

    Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. Gsm-infinite: How do your llms behave over infinitely increasing context length and reasoning complexity?arXiv preprint arXiv:2502.05252, 2025. 17 A Dataset Details A.1 Details of Problem Families To provide ...

    2025 arXiv

  35. [43]

    visualizing

    + 2 + 5is increasing. Composed function num- ber 2 3 4 5 6 algebra/func_ext_coordsAverage of allx-coordinates of local min- ima off(x) =−3(−2 sin(πx−2)+2)+2 2(−2 sin(πx−2)+2)+1. Composed function num- ber 2 3 4 5 6 algebra/func_extremaNumber of local maxima off(x) = 2 cos3π(|x...

  36. [44]

    conjecture

    "conjecture" - The sentence makes a hypothesis or conjecture about the final answer. Typical examples include "Alternatively, maybe the matrix is singular.", "Wait, let’s check if the determinant is zero or not.", "Alternatively, maybe the problem is from a source where the an...

  37. [45]

    computation

    "computation" - The sentence performs a mathematical computation or calculation

  38. [46]

    computation

    "other" - The sentence is explanation, setup, conclusion, or another type of reasoning. Original math problem: {original_question} Correct answer: {correct_answer} Sentence to classify: {sentence} If you classify it as "computation", also verify if the computation is correct b...

    2014

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.