Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Scaling Reasoning without Attention

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A pure attention-free model trained on curriculum data beats similarly sized Transformers on hard math and code reasoning benchmarks.

desk verdict A plausible and potentially useful result about attention-free reasoning models, but the headline comparisons rest on unverified eval hygiene and unmatched evaluation protocols. read the letter →

arxiv 2505.22425 v1 pith:CFOUDLSF submitted 2025-05-28 cs.LG cs.AIcs.CL

Xueliang Zhao , Wei Wu , Lingpeng Kong This is my paper
classification cs.LGcs.AIcs.CL
keywords Mamba-2statespacemodelsattention-freereasoningcurriculumfine-tuningPromptCoTAIMELiveCodeBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to show that self-attention is not required for state-of-the-art reasoning in language models. The authors build PromptCoT-Mamba-7B, a pure attention-free model based on Mamba-2's state space dual layers, and train it with a two-phase curriculum that uses the PromptCoT synthesis paradigm to generate pedagogically structured math and code problems. On AIME 24, AIME 25, and LiveCodeBench the model beats all compared Transformer and hybrid baselines of similar size and surpasses the much larger Gemma3-27B. If true, this means attention-free architectures can be competitive on the hardest reasoning benchmarks while offering fixed-memory, constant-time inference.

What carries the argument

The load-bearing mechanism is the state space dual (SSD) layer from Mamba-2, a recurrence that keeps a fixed-size hidden state updated by a low-rank outer product with exponential decay, giving constant memory and per-token compute. Training uses a parallel contraction form over structured matrices. Around this sits the PromptCoT curriculum: first a foundational stage on OpenCodeReasoning and OpenThoughts 2, then an advanced stage on 256k PromptCoT-generated problems, with rationales generated before problems so examples carry expert-style intermediate steps.

What would settle it

Search the OpenThoughts 2, OpenCodeReasoning, and PromptCoT-generated training sets for near-duplicates of the AIME 24, AIME 25, and LiveCodeBench-v5 problems; if any test problem appears in training, the benchmark advantage could be memorization rather than reasoning. A cleaner test would rerun the evaluation on a newly released, contamination-free competition set.

Watch

Extended reading notes

Core claim

PromptCoT-Mamba-7B, with no attention mechanism and no KV cache, achieves 35.2 percent on AIME 24, 24.6 percent on AIME 25, and 29.9 percent on LiveCodeBench-v5, outperforming every Transformer and hybrid baseline in the comparison set and beating Gemma3-27B on those three tasks. The paper frames this as the first instance in which a pure Mamba-based architecture exceeds the performance of similarly sized attention-based models across this range of benchmarks. The result is attributed to the combination of Mamba-2's SSD layers for efficient recurrent inference and a curriculum that starts with foundational reasoning data and advances to PromptCoT-synthesized high-difficulty problems.

Load-bearing premise

The paper's headline gains depend on the assumption that the AIME, LiveCodeBench, and HumanEval test problems never appeared in the fine-tuning corpora, since the paper reports no deduplication check between the training data and these benchmarks.

Editorial extensions

If this is right

  • Pure attention-free models can reach or exceed similarly sized Transformer performance on hard math and code reasoning, so attention is not a prerequisite for state-of-the-art reasoning.
  • Deployments under memory constraints get the accuracy gains with up to 3.66x higher throughput on 24GB GPUs and 1.69x on 72GB, making long reasoning traces cheaper to serve.
  • The PromptCoT curriculum stage is the main driver: removing it drops AIME 24 from 35.2 to 11.7 and LiveCodeBench from 29.9 to 5.3.
  • A math-specialized variant further raises AIME 24 to 42.9, suggesting domain-focused adaptation works on this architecture at the cost of code performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the result replicates under strict contamination control, it suggests reasoning capability can scale in recurrent, non-attention architectures and may redirect some scaling investment away from attention and KV-cache engineering.
  • The architecture's fixed memory makes it a natural candidate for very long reasoning chains and for test-time compute budgets, areas the paper profiles but does not fully explore.
  • A decisive test would be to train the same curriculum on a Transformer of identical size: if the Mamba model still wins, the architectural advantage is real; if not, the data pipeline explains the result.
  • One could extend by ablating the quality of synthetic rationales versus problem-only data to isolate whether rationale-guided generation is the ingredient that closes the attention gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PromptCoT-Mamba-7B, a 7B-parameter pure Mamba-2 (SSD) language model initialized from Mamba Codestral and fine-tuned with a two-phase curriculum: first on OpenCodeReasoning/OpenThoughts2, then on PromptCoT-synthesized problems plus additional reasoning corpora. The authors report pass@1 results on seven math/code benchmarks, claiming that the model outperforms same-scale Transformer and hybrid baselines and surpasses Gemma3-27B by 2.6/0.6/3.0 points on AIME 24/AIME 25/LiveCodeBench v5, making it 'the first instance' where a pure Mamba model exceeds similarly sized attention models. A math-specialized variant and an inference-throughput comparison are also presented.

Significance. If the headline comparisons were statistically and contamination-wise secure, this would be a noteworthy result: it would show that a state-space architecture can reach leading reasoning performance with fixed-memory, constant-per-token inference, and it would provide a reproducible recipe, since model weights and code are released, the ablation in Table 2 clearly attributes the gains to PromptCoT and the curriculum, and the SSD formulation is described in enough detail to reimplement. The paper is honest about its training sources and the evaluation protocol. However, the central empirical claims currently rest on benchmark differences that are within sampling noise and on an unverified assumption that the fine-tuning corpora do not overlap with the evaluation problems; these issues must be resolved before the 'first pure Mamba beats attention' and 'surpasses Gemma3-27B' claims can be considered established.

major comments (4)
  1. [Section 2.2 / Section 3.1 / Section 3.4] The contamination concern raised by the stress-test note lands directly. Section 2.2 states that the initialization corpora combine 'expert-authored problems from competitions such as Codeforces and the AIME mathematics contest,' and Section 3.4 reports training on 735k OpenCodeReasoning and 1.14M OpenThoughts2 samples, with 246k additional OpenCodeReasoning samples in the advanced stage. The paper does not report any deduplication or overlap check against AIME 24/25 or LiveCodeBench v5, and Section 3.1's only defense for LiveCodeBench is the dataset release cutoff, which is insufficient because OpenCodeReasoning and OpenThoughts2 are released in 2025 and can contain problems from the May 2023-May 2024 LiveCodeBench window. Since each AIME problem is worth approximately 3.3 points on a 30-problem benchmark, and the reported margins over Gemma3-27B are 2.6 and 0.6 points on AIME 24/25 and 3.0 points on LiveCodeBench, one or two memorized problems could produce the headline results. Please add a rigorous overlap analysis (e.g., n-gram plus embedding-level checks) against all training corpora, and report results after removing any overlapping examples.
  2. [Section 3.1 / Section 3.2 / Table 1] The headline comparisons are not supported by any measure of statistical uncertainty. AIME 24 and AIME 25 each contain 30 problems, yet Table 1 reports only point estimates (35.2 vs. 32.6 on AIME 24 and 24.6 vs. 24.0 on AIME 25) with no standard errors, confidence intervals, bootstrap estimates, or per-problem pass rates. For a 30-problem benchmark, a one-problem difference is 3.3 points, and the binomial standard error at p=0.35 is about 8.7 points, so the 2.6-point and 0.6-point margins over Gemma3-27B are within sampling noise. The avg@16 protocol reduces per-problem sampling variance but does not address between-problem variance. Please provide confidence intervals or bootstrap estimates, and state the effective number of problems for every reported average.
  3. [Section 3.3 / Section 3.5] The comparison with baselines is not shown to be controlled. Section 3.3 lists public Transformer and hybrid models, and Section 3.5 draws strong comparative conclusions, but the paper never states whether Gemma3-27B, Sky-T1-7B, S1.1-7B, and the other baselines were re-run under the same decoding protocol (temperature, top-p, max generation length, prompt format, quantization, and sampler version) or whether the numbers are taken from external sources. Because avg@k estimates are sensitive to decoding settings, a 2.6-point or 0.6-point margin can easily be a protocol artifact. Please either re-run all baselines in the same harness or clearly state the provenance of each number and justify that the protocols are comparable.
  4. [Section 4.3 / Figure 2] The efficiency claims are under-specified. Figure 2 reports throughput ratios of 3.66x on 24GB and 1.69x on 72GB, but the text does not report the batch size, input sequence length, output sampling parameters, or the software and optimization settings for the s1.1-7B baseline beyond 'vLLM 0.7.3.' Without these details, the reader cannot determine whether the comparison is apples-to-apples or whether the Transformer baseline was run in a configuration that underuses its available memory. In addition, the phrase 'constant-time inference' in Section 2.1 should be qualified: the per-token step is O(NP), but generating T tokens takes O(TNP) total time, so inference time still grows linearly with the number of generated tokens.
minor comments (5)
  1. [Throughout] The benchmark name is inconsistently capitalized ('Livecodebench' in the Abstract and Table captions vs. 'LiveCodeBench' elsewhere); please standardize.
  2. [Section 3.4] The advanced-stage data budget is stated as '256k prompt-completion pairs constructed via the PROMPT COT pipeline, augmented with 246k from OpenCodeReasoning and 232k from OpenMathReasoning,' but the total and how these parts combine are not made explicit; please clarify whether these are additive budgets and report the final per-stage token counts.
  3. [Table 3 / Section 4.2] The claim that the math-specialized variant achieves 'state-of-the-art results on all math benchmarks' is not supported by comparisons to other math-specialized models; either add such baselines or soften the wording to 'improves over the general variant'.
  4. [References] Several references are informal URLs (e.g., [AIME-2024], [Team, 2025b], [Team, 2025a]) without version identifiers or access dates; for reproducibility, please cite specific dataset versions, commits, or arXiv identifiers where available.
  5. [Section 2.1, Eq. (2)] The contraction notation in Eq. (2) is terse; please explicitly define the shapes of M, Z, H, and Y and state which indices are contracted, since this is the formal core of the training-time formulation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical comparisons against external benchmarks, and the author-self-cited PromptCoT component is independently validated by the paper's own ablations.

full rationale

The claimed derivation chain is architecture (Mamba-2 SSD layers, Section 2.1) plus a two-phase curriculum fine-tuning pipeline (Sections 2.2 and 3.4), evaluated on external benchmarks (MATH-500, AIME 24/25, OlympiadBench, HumanEval/HumanEval+, LiveCodeBench v5, Section 3.1). No prediction is defined in terms of a fitted parameter or a quantity constructed from the training pipeline; the headline results are pass@1 accuracies on fixed, external test sets, so the central comparison does not reduce by construction. The only self-citation with methodological weight is PromptCoT [Zhao et al., 2025], adopted in Sections 2.2 and 5.1, but the paper tests this component directly in Table 2 (removing PromptCoT drops AIME 24 from 35.2 to 11.7), so the citation is not the sole evidence for the claim. The related-work self-citations (Sego, SubgoalXL) are not load-bearing. One data-integrity concern is flagged: Section 2.2 states OpenCodeReasoning and OpenThoughts2 include a small number of expert-authored competition problems, and Section 3.4 trains on those corpora, while Section 3.1's contamination defense for LiveCodeBench v5 relies on release cutoffs; no explicit deduplication against AIME 24/25 or LiveCodeBench v5 is reported. That concern bears on whether the observed gains are memorization or reasoning, but it is a soundness/contamination issue, not a circularity of the derivation: the evaluation metrics are external and are not defined as functions of the training corpora. Accordingly, no circular step can be quoted, and the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on unverified assumptions about data/benchmark separation and comparable baseline measurements. No free parameters are fitted in the mathematical sense; the hand-chosen items are data budgets and evaluation sampling choices. No invented entities are introduced.

free parameters (2)
  • Advanced-stage data budgets = 256k PromptCoT + 246k OpenCodeReasoning + 232k OpenMathReasoning samples
    Hand-chosen mixture sizes, with no sensitivity analysis; the headline results depend on this allocation.
  • Evaluation sampling sizes = avg@16 for AIME, avg@8 for LiveCodeBench
    Sampling hyperparameters chosen for evaluation; different k or temperature could change scores and the ranking versus baselines.
assumptions (4)
  • standard math The Mamba-2 SSD layer recurrence (Eq. 1) and its parallel form (Eq. 2) are equivalent as implemented.
    Adopted from Dao and Gu 2024; the paper relies on this equivalence without proof.
  • ad hoc to paper PromptCoT-generated problems do not duplicate or leak the evaluation benchmarks.
    The advanced stage uses PromptCoT synthesis, yet no overlap analysis with AIME 24/25 or LiveCodeBench is provided.
  • domain assumption The public training corpora (OpenThoughts 2, OpenCodeReasoning, OpenMathReasoning) do not overlap the test sets.
    These large corpora may include competition problems from the same sources as AIME and Codeforces-based LiveCodeBench; no deduplication is reported.
  • domain assumption Baseline benchmark scores are comparable across models despite coming from different sources.
    Table 1 mixes numbers from prior papers and model reports; comparability requires identical decoding protocols, which is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Reasoning without Attention." pith.science (2026). https://pith.science/paper/CFOUDLSF

@misc{pith2026250522425,
  author       = {Pith},
  title        = {Pith review of: Scaling Reasoning without Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CFOUDLSF}},
  note         = {Machine review of arXiv:2505.22425}
}
read the original abstract

Large language models (LLMs) have made significant advances in complex reasoning tasks, yet they remain bottlenecked by two core challenges: architectural inefficiency due to reliance on Transformers, and a lack of structured fine-tuning for high-difficulty domains. We introduce \ourmodel, an attention-free language model that addresses both issues through architectural and data-centric innovations. Built on the state space dual (SSD) layers of Mamba-2, our model eliminates the need for self-attention and key-value caching, enabling fixed-memory, constant-time inference. To train it for complex reasoning, we propose a two-phase curriculum fine-tuning strategy based on the \textsc{PromptCoT} synthesis paradigm, which generates pedagogically structured problems via abstract concept selection and rationale-guided generation. On benchmark evaluations, \ourmodel-7B outperforms strong Transformer and hybrid models of comparable scale, and even surpasses the much larger Gemma3-27B by 2.6\% on AIME 24, 0.6\% on AIME 25, and 3.0\% on Livecodebench. These results highlight the potential of state space models as efficient and scalable alternatives to attention-based architectures for high-capacity reasoning.

Figures

Figures reproduced from arXiv: 2505.22425 by the authors.

Figure 1
Figure 1. Comparison of benchmark performance across models of varying architectures. PROMPTCOT￾MAMBA-7B, a pure attention-free Mamba model (orange), outperforms all Transformer (blue) and hybrid Mamba-Transformer (hatched) baselines of similar or larger scale on AIME 24, AIME 25, and Livecodebench, while remaining competitive across other math and code reasoning benchmarks. Bars represent pass@1 accuracy. ∗ This work was don… view at source ↗
Figure 2
Figure 2. Token generation throughput (tokens/sec) of PROMPTCOT-MAMBA-7B and S1.1-7B under different GPU memory configurations. Performance is measured across a range of generation lengths. We compare the inference efficiency of PROMPTCOT-MAMBA-7B against a strong Transformer baseline, s1.1-7B [Muennighoff et al., 2025], using vLLM 0.7.3 on a single NVIDIA A100 80GB GPU. For each generation length interval [a, b), we set min_… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 2 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    1901

  3. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  4. [8]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008,

    arXiv

  5. [9]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,

    arXiv

  6. [11]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974,

    arXiv

  7. [12]

    9 Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al

    Accessed: 2025-01-22. 9 Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857,

    work page 2025

  8. [13]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050,

    arXiv

Show all 29 references
  1. [14]

    Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset

    Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schif- ferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.16891,

    arXiv

  2. [15]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393,

    arXiv

  3. [16]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048,

    arXiv

  4. [17]

    Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,

    arXiv

  5. [18]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786,

    arXiv

  6. [19]

    NovaSky Team

    Accessed: 2025-05-24. NovaSky Team. Sky-t1: Train your own o1 preview model within $450. https://novasky- ai.github.io/posts/sky-t1, 2025a. Accessed: 2025-01-09. OpenThoughts Team. Open Thoughts. https://open-thoughts.ai, January 2025b. Qwen Team. Qwq-32b: Embracing the power ...

    2025 arXiv

  7. [20]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171,

    arXiv

  8. [22]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484,

    arXiv

  9. [23]

    Mammoth: Building math generalist models through hybrid instruction tuning

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653,

    arXiv

  10. [24]

    Mammoth2: Scaling instructions from the web

    Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548,

    arXiv

  11. [25]

    Sego: Sequential subgoal optimization for mathematical problem-solving

    Xueliang Zhao, Xinting Huang, Wei Bi, and Lingpeng Kong. Sego: Sequential subgoal optimization for mathematical problem-solving. arXiv preprint arXiv:2310.12960,

    arXiv

  12. [26]

    Subgoalxl: Subgoal-based expert learning for theorem proving

    Xueliang Zhao, Lin Zheng, Haige Bo, Changran Hu, Urmish Thakker, and Lingpeng Kong. Subgoalxl: Subgoal-based expert learning for theorem proving. arXiv preprint arXiv:2408.11172,

    arXiv

  13. [27]

    Promptcot: Synthesizing olympiad-level problems for mathematical reasoning in large language models

    Xueliang Zhao, Wei Wu, Jian Guan, and Lingpeng Kong. Promptcot: Synthesizing olympiad-level problems for mathematical reasoning in large language models. arXiv preprint arXiv:2503.02324,

  14. [28]

    Efficient attention via control variates

    Lin Zheng, Jianbo Yuan, Chong Wang, and Lingpeng Kong. Efficient attention via control variates. arXiv preprint arXiv:2302.04542,

    arXiv

  15. [29]

    Least-to-most prompting enables complex reasoning in large language models

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625,

    arXiv

  16. [2020]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

  17. [2021]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    arXiv

  18. [2022]

    Gated linear attention transformers with hardware-efficient training

    10 Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635,

    arXiv

  19. [2023]

    Opencodereasoning: Advancing data distillation for competitive coding

    Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jo- celyn Huang, Vahid Noroozi, and Boris Ginsburg. Opencodereasoning: Advancing data distillation for competitive coding. arXiv preprint arXiv:2504.01943,

    arXiv

  20. [2024]

    mlr.press/v235/dao24a.html

    URL https://proceedings. mlr.press/v235/dao24a.html. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

    arXiv

  21. [2025]

    https://huggingface.co/datasets/ai-mo/aimo-validation-aime

    AIME-2024. https://huggingface.co/datasets/ai-mo/aimo-validation-aime. Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Alek- sander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Maha- baleshwarkar, et al. Ne...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.