Pith. sign in

REVIEW 1 major objections 6 minor 56 references

Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code

T0 review · 1 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that training models to predict test-case outputs from algorithmic problems—without showing code—improves reasoning across math, knowledge, code, and logic benchmarks at 1.5B–32B scale.

desk verdict TeaR is a sensible, data-efficient method for improving general reasoning via RL on code-derived test-case prediction, but the MATH result is inflated by using MATH-test as the checkpoint-selection set. read the letter →

arxiv 2507.07498 v2 pith:ES6FQUF3 submitted 2025-07-10 cs.CL cs.LG

Keqin Bao , Nuo Chen , Xiaoyuan Li , Binyuan Hui , Bowen Yu , Fuli Feng , Xiangnan He , Dayiheng Liu This is my paper
classification cs.CLcs.LG
keywords reinforcementlearningLLMreasoningtest-casepredictioncode-freetrainingGRPOalgorithmicproblemsgeneralizationlongchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes TeaR, a reinforcement-learning recipe meant to improve general reasoning in LLMs by training on algorithmic tasks without ever showing code. The model is given a problem statement and one concrete test input and must predict the output; exact match with the true output is the reward, so the model has to discover its own step-by-step chain of reasoning. From 2.4K seed problems the authors build 200K such instances, then train with GRPO on base and long-CoT distilled models spanning 1.5B to 32B parameters. On 17 benchmarks across math, knowledge, code, and logical reasoning, the paper reports consistent gains over the base models, including a 35.9% average improvement on Qwen2.5-7B and 5.9% on R1-Distilled-7B. If the claim holds, it matters because it points to a cheap, verifiable training signal that transfers across domains instead of overfitting to algorithm patterns.

What carries the argument

The load-bearing object is the test-case prediction instance: a query that pairs an algorithmic problem statement with a concrete input, whose target is the exact output, with code excluded from both prompt and reward. The pipeline has three components. First, a curation filter removes easy-to-guess answers (low output cardinality or majority-vote judgment by a large auxiliary model), keeps only executable samples, and rejects cases with runtime over one second or input/output longer than 200 characters. Second, GRPO trains the model to explore its own reasoning paths toward the deterministic output. Third, an exact-match rule gives a 0/1 reward. Together these parts turn deterministic code execution into a natural-language reasoning task with a verifiable signal.

What would settle it

Run the same TeaR pipeline but pick checkpoints on a validation split that is disjoint from all reported evaluation sets, for example a held-out slice of AMC/AIME or a non-math reasoning set, then re-evaluate MATH, AMC, AIME24/25, and the logic and code suites. If the math and competition-math gains shrink to noise while non-math gains remain, the general-reasoning mechanism stands and the MATH gain was selection; if all gains vanish, the transfer claim itself fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the reasoning value of code lives in the task, not in the code's syntax or in its optimized algorithms. TeaR presents a model with an algorithmic problem in natural language and one concrete test input, and asks it to predict the exact output; no reference code appears, and the reward is simply whether the predicted output matches the true one. Because such outputs are deterministic, GRPO can use exact match as a dense, unambiguous signal, and the model must construct its own step-by-step derivation. The reported consequence is that models trained this way improve on benchmarks across math, general knowledge, code, and logical reasoning, including on long-CoT distilled models, and that TeaR outperforms the code-supervision baseline CodeI/O on several of the comparable tasks while using roughly an order of magnitude less data.

Load-bearing premise

The paper's headline math gains depend on selecting checkpoints by accuracy on the MATH test set and then reporting MATH as an evaluation benchmark; if that same selection also inflated AMC and AIME scores, the broad math-transfer claim would be partly a selection artifact.

Editorial extensions

If this is right

  • Training on roughly 200K code-free test-case instances drawn from only 2.4K seed tasks is enough to move all four evaluated reasoning domains, so reasoning supervision need not be matched to each target benchmark.
  • The reported comparison with CodeI/O shows that a 10x smaller task set can match or beat code-supervision training, making verification-based RL a more data-efficient route to the same end.
  • Because gains appear on R1-Distilled models that were already optimized for math and code, the recipe is complementary to long-chain-of-thought distillation rather than a substitute for it.
  • The largest relative gains on logic-heavy BBH subtasks and on an unseen movie-recommendation task support the claim that what transfers is a general reasoning skill, not knowledge of algorithm patterns.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not run is to keep the exact same pipeline but select checkpoints on a validation set disjoint from all reported benchmarks; that experiment would isolate how much of the math gain is genuine transfer rather than selection.
  • Because any executable function can generate test-case pairs, the approach implies a scalable data engine: verified programs in any language could be turned into reasoning-supervision instances without human annotation, extending far beyond the current 2.4K seed problems.
  • The paper's own limitations section notes modest dataset scale, limited difficulty, and a 32B ceiling; a natural next test is whether the TeaR recipe still transfers when the seed problems are harder and the model is larger.
  • The 'with reference code' comparison hints that supplying code during RL can suppress the model's own reasoning; this predicts that other code-augmented RL recipes would improve by discarding the code and keeping only the verifiable input-output signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper proposes TeaR, a method that trains LLMs with GRPO on test-case prediction for algorithmic problems (from LeetCode) without showing code, using an exact-match reward. The data is filtered to remove easy-to-guess answers, non-executable code, and overly complex test cases. The authors evaluate on 17 benchmarks spanning math, knowledge, code, and logic, using two base models and three R1-distilled models (1.5B to 32B), and report broad improvements, claiming that code-free algorithmic reasoning transfers to general reasoning tasks. They also compare against CodeI/O and analyze data-format choices and fine-grained BBH subsets.

Significance. If the central claim holds, the result is significant: it suggests that reinforcement learning on a relatively small set of code-derived but code-free test-case-prediction tasks can improve general reasoning across domains, using less data than methods like CodeI/O. The reward design (exact match against executed code output) is clean and verifiable, and the preliminary observation that models write code but fail at the corresponding test-case reasoning is an interesting motivation. However, the current evidence is undermined by a circular validation protocol and the absence of statistical or reproducibility support, so the paper's broad claims are not yet established.

major comments (1)
  1. [Section 4, Implementation Details; Table 1] Section 4.3 (Table 3) compares data formats under the heading 'Data source (Math and Code) and format (w/ Ref Code) comparison,' but the comparison is not controlled. The text says that for the 'Code' setting, the reward remained zero when starting from Qwen2.5-7B-base, so training was instead initialized from R1-distilled-7B, while the other rows (Math, w/ Ref Code, Ours) use Qwen2.5-7B-base. Consequently, the conclusion that 'training with raw code provides almost no performance gain' confounds model initialization with data format. Please provide same-initialization comparisons or explicitly reframe this as a separate, non-comparable setting.
minor comments (6)
  1. [Abstract; Section 4.1] The abstract reports a '35.9% improvement on Qwen2.5-7B' without specifying the benchmark or whether the percentage is relative or absolute. In the main text, the MATH accuracy increases from 49.8% to 75.2%, which is a 51% relative change, so the abstract number appears inconsistent or at least undefined.
  2. [Section 3.2, Easy-to-Guess Question Filtering] The phrase 'potential answers less than or equal to 8' is ambiguous: please specify what quantity is being counted (e.g., the number of distinct output values in the test cases) and how the threshold of 8 was chosen.
  3. [Section 3.2, Complex Case Filtering] The thresholds of 1 second execution time and 200 characters for input/output are presented without sensitivity analysis. Since these choices determine the training distribution, a brief justification or an ablation would help assess their impact.
  4. [Reproducibility] No code, data, or training configuration is released, and the Limitations section does not mention the lack of public artifacts. Releasing the filtered dataset and training/evaluation scripts would substantially improve reproducibility.
  5. [Limitations] The Limitations section acknowledges dataset scale and model-size restrictions, but it does not disclose the MATH-test validation circularity, the absence of multiple seeds, or the lack of error bars, all of which materially affect the strength of the reported conclusions.
  6. [Throughout] There are several typos and formatting issues, e.g., 'mathmatics' (Section 4), 'extact match' (Section 3.1), 'conder' (Section 4.1), 'CTRUXEval' (Related Work), and the garbled '4.09.1 8.14' in the LCB-O row of Table 1. These should be corrected.

Circularity Check

1 steps flagged · score 6.0 of 10

MATH evaluation is circular because the MATH-test set is used for checkpoint selection and then reported as an evaluation benchmark.

  1. fitted input called prediction [Section 4, Implementation Details; Table 1 (Math row)]
    "Throughout the training process, we use the MATH-test set as the validation dataset to help us select appropriate checkpoints."

    The checkpoint is chosen using MATH-test accuracy, yet Table 1 reports MATH accuracy as evidence of TeaR's mathematical improvement. The reported MATH gain (e.g., 49.8 to 75.2 for Qwen2.5-7B) is the selection criterion itself: the model is selected to perform well on this exact set, so reporting it as a held-out evaluation measures fit to the validation set, not generalization. The abstract's headline '35.9% improvement' is this validation-set number. The MATH claim therefore reduces by construction to the checkpoint-selection objective; independent evidence for math transfer would require a validation split disjoint from all reported test sets.

full rationale

The training reward itself is external and code-grounded, so the core RL mechanism is not circular. Cross-domain claims rest on many benchmarks not used for selection. However, the paper explicitly uses MATH-test for checkpoint selection and then reports MATH in Table 1 as an evaluation result, making the headline MATH improvement a fitted selection metric rather than a prediction. This is a genuine, quotable circularity in the evaluation protocol, affecting the central 'consistent improvements' claim on the mathematics dimension even though most other benchmarks remain independent.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method's contribution is a training recipe, not a new theory. It relies on several ad hoc filtering thresholds and domain assumptions about transfer, reward cleanliness, and data contamination.

free parameters (4)
  • Easy-to-guess answer threshold = <=8 answer options
    Chosen by hand to filter questions with easily guessable outputs, potentially affecting the difficulty and diversity of training data (Section 3.2).
  • Code execution time threshold = 1 second
    Manually determined to remove slow or unstable test cases, which changes the distribution of training examples (Section 3.2).
  • Input/output character count threshold = 200 characters
    Ad hoc cutoff to filter overly long contexts, affecting the reasoning complexity of the training set (Section 3.2).
  • Training hyperparameters = batch 256, rollout 8, temp 1.1, lr 1e-6, max tokens 4096/16384
    Set without a reported sweep; these choices affect learning dynamics and final performance (Section 4, Implementation Details).
assumptions (4)
  • domain assumption Test-case prediction on algorithmic problems is a good proxy for general reasoning skill.
    The entire method rests on this transfer claim, stated in Section 1 and supported only by the benchmark results.
  • domain assumption Exact-match reward on executed code outputs is a clean, hack-resistant signal.
    Used as the sole reward in Section 3.1; filtering is meant to prevent guessing but does not guarantee resistance to reward hacking.
  • domain assumption RL can elicit latent reasoning abilities in base models.
    Borrowed from cited work (Yue et al., 2025) and central to the method's motivation in Section 1.
  • domain assumption LeetCodeDataset is a suitable seed and does not contaminate evaluation benchmarks.
    Training data from LeetCode and evaluation benchmarks like HumanEval, MBPP, and LiveCodeBench share code reasoning tasks; no contamination check is reported in Sections 3.2 or A.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code." pith.science (2026). https://pith.science/paper/ES6FQUF3

@misc{pith2026250707498,
  author       = {Pith},
  title        = {Pith review of: Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ES6FQUF3}},
  note         = {Machine review of arXiv:2507.07498}
}
read the original abstract

Enhancing reasoning capabilities remains a central focus in the LLM reasearch community. A promising direction involves requiring models to simulate code execution step-by-step to derive outputs for given inputs. However, as code is often designed for large-scale systems, direct application leads to over-reliance on complex data structures and algorithms, even for simple cases, resulting in overfitting to algorithmic patterns rather than core reasoning structures. To address this, we propose TeaR, which aims at teaching LLMs to reason better. TeaR leverages careful data curation and reinforcement learning to guide models in discovering optimal reasoning paths through code-related tasks, thereby improving general reasoning abilities. We conduct extensive experiments using two base models and three long-CoT distillation models, with model sizes ranging from 1.5 billion to 32 billion parameters, and across 17 benchmarks spanning Math, Knowledge, Code, and Logical Reasoning. The results consistently show significant performance improvements. Notably, TeaR achieves a 35.9% improvement on Qwen2.5-7B and 5.9% on R1-Distilled-7B.

Figures

Figures reproduced from arXiv: 2507.07498 by the authors.

Figure 1
Figure 1. The right shows the difference in code generation and test case prediction accuracy between Qwen2.5-7B [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of our TeaR framework soning strategies that have already been internal￾ized during pretraining (Chu et al., 2025a). To tap into this self-activated reasoning poten￾tial, we employ GRPO, an advanced RL algorithm tailored specifically for LLMs. In detail, during the training phase, we provide the model with an algorithmic question and its corresponding correct output, allowing the model to explore the so… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 13 canonical work pages

  1. [1]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  2. [2]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  3. [3]

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025. https://doi.org/10.48550/ARXIV.2503.09567 Towards reasoning era: A survey of long chain-of-thought for reasoning large language models . CoRR, abs/2503.09567

    arXiv doi:10.48550/arxiv.2503.09567 2025

  4. [4]

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Sergey Levine, and Yi Ma. 2025 a . https://openreview.net/forum?id=d3E3LWmTar SFT memorizes, RL generalizes: A comparative study of foundation model post-training . In The Second Conference on Parsimony and Learning (Recent Spotlight Track)

    work page 2025

  5. [5]

    Le, Sergey Levine, and Yi Ma

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. 2025 b . https://doi.org/10.48550/ARXIV.2501.17161 SFT memorizes, RL generalizes: A comparative study of foundation model post-training . CoRR, abs/2501.17161

    arXiv doi:10.48550/arxiv.2501.17161 2025

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

  7. [7]

    Min, Gail E

    Yangruibo Ding, Marcus J. Min, Gail E. Kaiser, and Baishakhi Ray. 2024 a . https://doi.org/10.1145/3649825 CYCLE: learning to self-refine the code generation . Proc. ACM Program. Lang. , 8( OOPSLA1 ):392--418

    doi:10.1145/3649825 2024

  8. [8]

    Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray

    Yangruibo Ding, Jinjun Peng, Marcus J. Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. 2024 b . https://arxiv.org/abs/2406.01006 Semcoder: Training code language models with comprehensive semantics reasoning . Preprint, arXiv:2406.01006

    arXiv 2024

Show all 56 references
  1. [9]

    Min, Gail E

    Yangruibo Ding, Jinjun Peng, Marcus J. Min, Gail E. Kaiser, Junfeng Yang, and Baishakhi Ray. 2024 c . http://papers.nips.cc/paper\_files/paper/2024/hash/6efcc7fd8efeee29a050a79c843c90e0-Abstract-Conference.html Semcoder: Training code language models with comprehensive semanti...

    2024

  2. [10]

    Kaiser, Wei Le, and Baishakhi Ray

    Yangruibo Ding, Benjamin Steenhoek, Kexin Pei, Gail E. Kaiser, Wei Le, and Baishakhi Ray. 2024 d . https://doi.org/10.1145/3597503.3608140 TRACED: execution-aware pre-training for source code . In Proceedings of the 46th IEEE/ACM International Conference on Software Engineerin...

    2024

  3. [11]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. https://doi.org/10.18653/v1/N19-1246 DROP : A reading comprehension benchmark requiring discrete reasoning over paragraphs . In Proceedings of the 2019 Conference of the North A ...

    2019 doi

  4. [12]

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D'Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. 2024. https://openreview.net/forum?id=5u1GpUkKtG H...

    2024

  5. [13]

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, and 1 others. 2024. Are we done with mmlu? arXiv preprint arXiv:2406.04127

    2024 arXiv

  6. [14]

    Chang Gong, Wanrui Bian, Zhijie Zhang, and Weiguo Zheng. 2025. Pseudocode-injection magic: Enabling llms to tackle graph computational tasks. arXiv preprint arXiv:2501.13731

    2025 arXiv

  7. [15]

    Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024 a . https://proceedings.mlr.press/v235/gu24c.html CRUXE val: A benchmark for code reasoning, understanding and execution . In Proceedings of the 41st International Confer...

    2024

  8. [16]

    Alex Gu, Baptiste Rozi \` e re, Hugh James Leather, Armando Solar - Lezama, Gabriel Synnaeve, and Sida Wang. 2024 b . https://openreview.net/forum?id=Ffpg52swvg Cruxeval: A benchmark for code reasoning, understanding and execution . In Forty-first International Conference on M...

    2024

  9. [17]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  10. [18]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. https://arxiv.org/abs/2401.14196 Deepseek-coder: When the large language model meets programming -- the rise of code...

    2024 arXiv

  11. [19]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the math dataset . Preprint, arXiv:2103.03874

    2021 arXiv

  12. [20]

    Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.67 Large language models can self-improve . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP...

    2023 doi

  13. [21]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024. https://arxiv.org/abs/2409.12186 Qwe...

    2024 arXiv

  14. [22]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  15. [23]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  16. [24]

    Naman Jain, King Han, Alex Gu, Wen - Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar - Lezama, Koushik Sen, and Ion Stoica. 2025. https://openreview.net/forum?id=chfJJYC3iL Livecodebench: Holistic and contamination free evaluation of large language models for code...

    2025

  17. [25]

    Prashant Jayannavar, Anjali Narayan - Chen, and Julia Hockenmaier. 2020. https://doi.org/10.18653/V1/2020.ACL-MAIN.232 Learning to execute instructions in a minecraft dialogue . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 202...

    2020 doi

  18. [26]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2024 doi arXiv

  19. [27]

    Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. 2025 a . https://arxiv.org/abs/2502.07316 Codei/o: Condensing reasoning patterns via code input-output prediction . Preprint, arXiv:2502.07316

    2025 arXiv

  20. [28]

    Xiaoyuan Li, Keqin Bao, Yubo Ma, Moxin Li, Wenjie Wang, Rui Men, Yichang Zhang, Fuli Feng, Dayiheng Liu, and Junyang Lin. 2025 b . Mtr-bench: A comprehensive benchmark for multi-turn reasoning evaluation. arXiv preprint arXiv:2505.17123

    2025 arXiv

  21. [29]

    Xiaoyuan Li, Moxin Li, Rui Men, Yichang Zhang, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu, and Junyang Lin. 2025 c . Hellaswag-pro: A large-scale bilingual benchmark for evaluating the robustness of llms in commonsense reasoning. arXiv preprint arXiv:2502.11393

    2025 arXiv

  22. [30]

    Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, and Fuli Feng. 2024. Evaluating mathematical reasoning of large language models: A focus on error identification and correction. In Findings of the Association for Computational Linguistics ACL 2024, pages 11316--11360

    2024

  23. [31]

    Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. 2025. https://arxiv.org/abs/2502.01100 Zebralogic: On the scaling limits of llms for logical reasoning . Preprint, arXiv:2502.01100

    2025 arXiv

  24. [32]

    Lobo, Chirag Agarwal, and Himabindu Lakkaraju

    Elita A. Lobo, Chirag Agarwal, and Himabindu Lakkaraju. 2024. https://doi.org/10.48550/ARXIV.2411.15382 On the impact of fine-tuning on chain-of-thought reasoning . CoRR, abs/2411.15382

    2024 doi arXiv

  25. [33]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai - Wei Chang, Song - Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/11332b6b6cf4485b84afadb1352d3a9a-Abstract-Conference.html Learn to explain: Multimodal rea...

    2022

  26. [34]

    Kaijing Ma, Xinrun Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, Wenhao Huang, and Ge Zhang. 2025. https://arxiv.org/abs/2410.06526 Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks . Prep...

    2025 arXiv

  27. [35]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwe...

    2025 arXiv

  28. [36]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. https://openreview.net/forum?id=Ti67584b98 GPQA : A graduate-level google-proof q&a benchmark . In First Conference on Language Modeling

    2024

  29. [37]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  30. [38]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  31. [39]

    Max Simchowitz and Aleksandrs Slivkins. 2024. Exploration and incentives in reinforcement learning. Operations Research, 72(3):983--998

    2024

  32. [40]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. https://doi.org/10.18653/v1/2023.findings-acl.824 Challenging BIG -bench tasks and whether chain-of-thought c...

    2023 doi

  33. [41]

    M. - A - P. Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Kang Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, and 77 others. 2025. https://doi.org/10.4...

    2025 doi arXiv

  34. [42]

    Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. 2024. https://api.semanticscholar.org/CorpusID:267681752 Openmathinstruct-1: A 1.8 million math instruction tuning dataset . ArXiv, abs/2402.10176

    2024 arXiv

  35. [43]

    Hemish Veeraboina. 2025. https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024 Aime problem set 1983-2024

    2025

  36. [44]

    Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023 a . Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731

    2023 arXiv

  37. [45]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 b . https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Confe...

    2023

  38. [46]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural I...

    2024

  39. [47]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022 a . https://openreview.net/forum?id=yzkSU...

    2022

  40. [48]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022 b . Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824--24837

    2022

  41. [49]

    Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. 2025. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms. arXiv preprint arXiv:2504.14655

    2025 arXiv

  42. [50]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. https://arxiv.org/abs/2409.12122 Qwen2.5-math technical report: ...

    2024 arXiv

  43. [51]

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023. https://doi.org/10.48550/ARXIV.2308.01825 Scaling relationship on learning mathematical reasoning with large language models . CoRR, abs/2308.01825

    2023 doi arXiv

  44. [52]

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837

    2025 arXiv

  45. [53]

    Wojciech Zaremba and Ilya Sutskever. 2014. https://arxiv.org/abs/1410.4615 Learning to execute . CoRR, abs/1410.4615

    2014 arXiv

  46. [54]

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. 2023. https://doi.org/10.48550/ARXIV.2308.10792 Instruction tuning for large language models: A survey . CoRR, abs/2308.10792

    2023 doi

  47. [55]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.